Tips & Tricks : Extract HTTP/XML/SOAP requests from PCAP traces

This bash tip can be useful when trying to extract all HTTP requests from PCAP generated traces.

First, use this command to generate the pcap file :

# tcpdump -s 0 -w trace.pcap

The option -s 0 enables capture of the whole packets and not only the first 64 bytes of each. With the -w trace.pcap parameter, raw captured data are written to the trace.pcap file.

FIELDS=(
  tcp.stream
  http.request.method http.request.uri http.request.version
  http.request.line
  http.response.version http.response.code http.response.phrase
  http.response.line
  http.file_data
)
tshark -r trace.pcap -T fields -Y http ${FIELDS[@]/#/-e$IFS} |
  awk -v FS=$'\t' '
{
  output = $1 ".http";
  n = $2 ? 2 : 6
  if (OUTPUTS[output]) printf("") >> output;
  else {printf("") > output; OUTPUTS[output] = 1; }
  printf("%s %s %s\n", $n, $(n+1), $(n+2)) >> output;
  printf("%s\n", gensub("(\\\\r\\\\n,?)+", "\n", "g", $(n+3))) >> output;
  if (substr($10,1,1) == "<") {
    fflush(output);
    close(output);
    xmlstarlet = "xmlstarlet fo - >> "output;
    printf("%s\n", gensub("\\\\n", "\n", "g", $10)) | xmlstarlet;
    close(xmlstarlet);
    printf("") >> output;
  }
  else
    printf("%s\n", $10) >> output;
  printf("\n--\n\n") >> output;
  close(output);
}
'

This script is using :

• tshark, the wireshark cli (https://www.wireshark.org/docs/man-pages/tshark.html)
• xmlstartlet, command line tool to work with XML (http://xmlstar.sourceforge.net/)
• awk, the famous (https://en.wikipedia.org/wiki/AWK)
• and just a little Bash (https://en.wikipedia.org/wiki/Bash_(Unix_shell))

With a sample downloaded at https://www.cloudshark.org/captures/74a6deb7aa4e, the result is :

$ ls -l *.http
-rw-rw-r-- 1 john dev 78407 Jan 26 13:55 0.http
-rw-rw-r-- 1 john dev  9089 Jan 26 13:55 1.http
-rw-rw-r-- 1 john dev  8307 Jan 26 13:55 2.http
-rw-rw-r-- 1 john dev 82888 Jan 26 13:55 3.http
-rw-rw-r-- 1 john dev 91637 Jan 26 13:55 4.http
-rw-rw-r-- 1 john dev  6733 Jan 26 13:55 5.http

$ cat 5.http
POST /ReportingWebService/ReportingWebService.asmx HTTP/1.1
Accept: text/xml
Content-Type: text/xml
SOAPAction: "http://www.microsoft.com/SoftwareDistribution/ReportEventBatch"
User-Agent: Windows-Update-Agent
Host: asupdt
Content-Length: 4318
Connection: Keep-Alive

<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/">
  <soap:Body>
    <ReportEventBatch xmlns="http://www.microsoft.com/SoftwareDistribution">
      <cookie>
        <Expiration>2012-06-26T21:26:32.995Z</Expiration>
...

    </ReportEventBatch>
  </soap:Body>
</soap:Envelope>

--

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Tue, 26 Jun 2012 20:32:05 GMT
Content-Length: 406

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Body>
    <ReportEventBatchResponse xmlns="http://www.microsoft.com/SoftwareDistribution">
      <ReportEventBatchResult>true</ReportEventBatchResult>
    </ReportEventBatchResponse>
  </soap:Body>
</soap:Envelope>

It is now easy to use grep, wc and sort on data.

$ grep "500 Error" *.http | wc -l
3

Now, try to do the same with REST requests !