Putting PCAP trace in Elasticsearch is a very good option to find patterns and troubleshoot network issues.
Lots of very good articles explain how to convert PCAP to Elastic using tshark :
- Analyzing Network Packets with Wireshark, Elasticsearch, and Kibana, from Elastic.co blog (2019-02-15)
- tshark + Elasticsearch, from H21 LAB (update 2020-06-02)
- …
But it doesn’t work out-of-the-box in recent versions of Elasticsearch!
First, download a capture sample from wireshark.org:
$ curl -o http.cap -s https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=http.cap $ tcpdump -r http.cap -nn reading from file http.cap, link-type EN10MB (Ethernet) 12:17:07.311224 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [S], seq 951057939, win 8760, options [mss 1460,nop,nop,sackOK], length 0 12:17:08.222534 IP 65.208.228.223.80 > 145.254.160.237.3372: Flags [S.], seq 290218379, ack 951057940, win 5840, options [mss 1380,nop,nop,sackOK], length 0 12:17:08.222534 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [.], ack 1, win 9660, length 0 12:17:08.222534 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [P.], seq 1:480, ack 1, win 9660, length 479: HTTP: GET /download.html HTTP/1.1 12:17:08.783340 IP 65.208.228.223.80 > 145.254.160.237.3372: Flags [.], ack 480, win 6432, length 0 12:17:08.993643 IP 65.208.228.223.80 > 145.254.160.237.3372: Flags [.], seq 1:1381, ack 480, win 6432, length 1380: HTTP: HTTP/1.1 200 OK 12:17:09.123830 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [.], ack 1381, win 9660, length 0 (...) 12:17:12.328438 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [.], ack 18365, win 9236, length 0 12:17:25.216971 IP 65.208.228.223.80 > 145.254.160.237.3372: Flags [F.], seq 18365, ack 480, win 6432, length 0 12:17:25.216971 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [.], ack 18366, win 9236, length 0 12:17:37.374452 IP 145.254.160.237.3372 > 65.208.228.223.80: Flags [F.], seq 480, ack 18366, win 9236, length 0 12:17:37.704928 IP 65.208.228.223.80 > 145.254.160.237.3372: Flags [.], ack 481, win 6432, length 0
Create an alias for curl + bulk command:
function es-bulk() {
curl -H content-type:application/x-ndjson -X POST \
"http://127.0.0.1:9200/_bulk?filter_path=took,errors,items.*.error" \
-s -w "\n" --data-binary "@-"
}
function es-put() {
curl -H content-type:application/x-ndjson -X PUT \
"http://127.0.0.1:9200/${1#/}?filter_path=took,errors,items.*.error" \
-s -w "\n" --data-binary "@-"
}
$ tshark -r http.cap -T ek | es-bulk
{
"took": 27,
"errors": true,
"items": [
{
"index": {
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'eth_eth_addr'\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 988]"
}
(...)
In order to fix this problem, just pipe tshark with jq command which removes duplicate attributes. The -c flag makes output compact and compatible with bulk input format:
$ tshark -r http.cap -T ek | jq -c . | es-bulk
{
"took": 362,
"errors": true,
"items": [
{
"index": {
"error": {
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [packets-2004-05-13] as the final mapping would have more than 1 type: [_doc, doc]"
}
This error is caused by a change in Elasticsearch API (removal-of-types) that is not supported in EK format (-T ek) of tshark. The latest release of Wireshark to date (3.2.5) still uses a "_type: doc", which was required before Elasticsearch 7.x:
https://github.com/wireshark/wireshark/blob/a71e225c40c8a383dcb462ab077552b673b686ee/epan/print.c#L365 361 json_dumper_begin_object(&dumper); 362 json_dumper_set_member_name(&dumper, "index"); 363 json_dumper_begin_object(&dumper); 364 write_json_index(&dumper, edt); 365 json_dumper_set_member_name(&dumper, "_type"); 366 json_dumper_value_string(&dumper, "doc"); 367 json_dumper_end_object(&dumper); 368 json_dumper_end_object(&dumper); 369 json_dumper_finish(&dumper);
Then try this to fix the problem and make the bulk output format compatible with Elasticsearch 7.x:
$ tshark -r http.cap -T ek | jq -c 'del(.index._type)' | es-bulk
Et voilà!
Tshark is very powerful but it can be very CPU intensive when used improperly. Don’t forget to define which fields are really important to index in Elasticsearch. This sample only dumps full HTTP requests without payload and data:
$ tshark -r http.cap -T ek -J "frame ip tcp http" -Y http | \
jq -c 'del(.layers.tcp.tcp_tcp_payload) |
del(.layers.tcp.tcp_tcp_segment_data) |
del(.layers.http.http_http_file_data)'
When you try to use tshark-provided elastic mapping, you also have an error:
$ tshark -G elastic-mapping | es-put /_template/packets
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to parse content to map"
}
],
"type": "parse_exception",
"reason": "Failed to parse content to map",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'ieee17221_ieee17221_message_type'\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 171, column: 51]"
}
},
"status": 400
}
Now you known how to fix this error: just use jq as a normalizer:
$ tshark -G elastic-mapping | jq . | es-put /_template/packets
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [doc : { ....
There is still a mapping error in recent versions of Elasticsearch, caused by the document type. To fix it, it is a bit more difficult:
$ tshark -G elastic-mapping | \
jq '{ "index_patterns": .index_patterns, "settings": .settings, "mappings": .mappings.doc }' | \
es-put /_template/packets
{}
You may also filter mapping to needed values:
$ tshark -G elastic-mapping --elastic-mapping-filter frame,ip,tcp,http | \
jq '{ "index_patterns": .index_patterns, "settings": .settings, "mappings": .mappings.doc }' | \
es-put /_template/packets
{}
