This bash tip can be useful when trying to extract all HTTP requests from PCAP generated traces.
First, use this command to generate the pcap file :
# tcpdump -s 0 -w trace.pcap
The option -s 0
enables capture of the whole packets and not only the first 64 bytes of each. With the -w trace.pcap
parameter, raw captured data are written to the trace.pcap
file.
FIELDS=( tcp.stream http.request.method http.request.uri http.request.version http.request.line http.response.version http.response.code http.response.phrase http.response.line http.file_data ) tshark -r trace.pcap -T fields -Y http ${FIELDS[@]/#/-e$IFS} | awk -v FS=$'\t' ' { output = $1 ".http"; n = $2 ? 2 : 6 if (OUTPUTS[output]) printf("") >> output; else {printf("") > output; OUTPUTS[output] = 1; } printf("%s %s %s\n", $n, $(n+1), $(n+2)) >> output; printf("%s\n", gensub("(\\\\r\\\\n,?)+", "\n", "g", $(n+3))) >> output; if (substr($10,1,1) == "<") { fflush(output); close(output); xmlstarlet = "xmlstarlet fo - >> "output; printf("%s\n", gensub("\\\\n", "\n", "g", $10)) | xmlstarlet; close(xmlstarlet); printf("") >> output; } else printf("%s\n", $10) >> output; printf("\n--\n\n") >> output; close(output); } '
This script is using :
- tshark, the wireshark cli (https://www.wireshark.org/docs/man-pages/tshark.html)
- xmlstartlet, command line tool to work with XML (http://xmlstar.sourceforge.net/)
- awk, the famous (https://en.wikipedia.org/wiki/AWK)
- and just a little Bash (https://en.wikipedia.org/wiki/Bash_(Unix_shell))
With a sample downloaded at https://www.cloudshark.org/captures/74a6deb7aa4e, the result is :
$ ls -l *.http -rw-rw-r-- 1 john dev 78407 Jan 26 13:55 0.http -rw-rw-r-- 1 john dev 9089 Jan 26 13:55 1.http -rw-rw-r-- 1 john dev 8307 Jan 26 13:55 2.http -rw-rw-r-- 1 john dev 82888 Jan 26 13:55 3.http -rw-rw-r-- 1 john dev 91637 Jan 26 13:55 4.http -rw-rw-r-- 1 john dev 6733 Jan 26 13:55 5.http
$ cat 5.http POST /ReportingWebService/ReportingWebService.asmx HTTP/1.1 Accept: text/xml Content-Type: text/xml SOAPAction: "http://www.microsoft.com/SoftwareDistribution/ReportEventBatch" User-Agent: Windows-Update-Agent Host: asupdt Content-Length: 4318 Connection: Keep-Alive <?xml version="1.0"?> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body> <ReportEventBatch xmlns="http://www.microsoft.com/SoftwareDistribution"> <cookie> <Expiration>2012-06-26T21:26:32.995Z</Expiration> ... </ReportEventBatch> </soap:Body> </soap:Envelope> -- HTTP/1.1 200 OK Cache-Control: private, max-age=0 Content-Type: text/xml; charset=utf-8 Server: Microsoft-IIS/7.0 X-AspNet-Version: 2.0.50727 X-Powered-By: ASP.NET Date: Tue, 26 Jun 2012 20:32:05 GMT Content-Length: 406 <?xml version="1.0" encoding="utf-8"?> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <soap:Body> <ReportEventBatchResponse xmlns="http://www.microsoft.com/SoftwareDistribution"> <ReportEventBatchResult>true</ReportEventBatchResult> </ReportEventBatchResponse> </soap:Body> </soap:Envelope>
It is now easy to use grep
, wc
and sort
on data.
$ grep "500 Error" *.http | wc -l 3
Now, try to do the same with REST requests !