Forcepoint NGFW log indexing in Elasticsearch using Filebeat
If you have been playing with Forcepoint™ firewalls, you know that traffic logs can be browsed and searched through their central management console (SMC).
If you have been playing with Forcepoint (s'ouvre dans un nouvel onglet)™ firewalls, you know that traffic logs can be browsed and searched through their central management console (SMC). You’ve probably noticed that searching for old records can sometimes be very slow, even with simple filters like source or destination IP, port, etc.
Computing aggregations or displaying tops n with the SMC is not really something you can do on the fly. I believe it was just not designed to do that, at least for now. This is due to the way traffic logs are stored on the SMC, they are only indexed by year/month/day/hour through a filesystem tree, so when you add filters to your searches, every single file matching your time period criteria has to be read. This is extremely frustrating when you want to cleanup old firewall policies and need to check what sources or destinations have matched a rule.
Here at Zenetys it’s been a while we have been working with Elasticsearch (s'ouvre dans un nouvel onglet), and naturally we started to think about indexing Forcepoint™ firewall logs in it.
Forwarding logs from the SMC
On the SMC, go to Network Elements, right click on the object corresponding to the Log Server, open the Properties window and go to the Log Forwarding tab. From there, you can define several forwarding rules for logs, each may have different options, in particular the target, the format or a filter on the events to send (just like when browsing logs).
Unfortunately there is no JSON format available, it would have been too easy. However there is CEF (s'ouvre dans un nouvel onglet), that’s the one we’ve chosen because it is readable (mostly key/value), includes by default the fields interesting for major use cases, and is pretty common, thus can be parsed by many logging tools.
We chose TCP over UDP to get ACKs in transport, allowing short disconnects with the target. The logs are sent as \n delimited plain text lines, with a syslog PRI prefix in front of each record. Following is an example of what’s sent, with lines truncated to 100 bytes:
[zuser@log6 ~]$ sudo ngrep -d bond0 -W byline -n 2 -S 100 port 3514
interface: bond0 (10.1.2.0/255.255.255.0)
filter: ( port 3514 ) and (ip or ip6)
#
T 10.4.5.6:36760 -> 10.1.2.3:3514 [AP]
<6>CEF:0|FORCEPOINT|Firewall|6.5.8|70018|Connection_Allowed|0|cat=System Situations app=Squid HTTP p
#
T 10.4.5.6:36760 -> 10.1.2.3:3514 [AP]
<6>CEF:0|FORCEPOINT|Firewall|6.5.8|70021|Connection_Closed|0|in=9084 out=987 app=Squid HTTP proxy (3
exit
4 received, 0 dropped
Note : in order to get in/out bytes on connection closed or progress events, accounting information must be logged. We usually do it with a continue rule on top of the policy; the option Connection Closing: Log Accounting Information must be set:
Parsing to JSON for Elasticsearch indexing
We normally use RSyslog (s'ouvre dans un nouvel onglet) to receive logs at a central point because it scales well. When it comes to parsing, RSyslog generally fits for simple cases and it has an output module for Elasticsearch. In the context of this experiment, we reused an old server, with an old version of RSyslog (not updatable). Unfortunately this build of RSyslog did not include the required modules; we also know that the CEF decoder in liblognorm (s'ouvre dans un nouvel onglet) current stable version do not parse well the first extension key name of Forcepoint™ logs (see Github PR#331 (s'ouvre dans un nouvel onglet)).
Since we’ve always tried to avoid using Logstash — because RAM may be cheap, but come on! —, we decided to see what could come up of experimenting with Filebeat (s'ouvre dans un nouvel onglet).
It is light, has a configurable TCP input, several processors for dissecting and manipulating data, including a CEF parser. Above all it has decent scripting capabilities for complex cases through an embedded Javascript interpreter. This is something RSyslog is really really really missing IMHO.
In the end, the only difficulty was to setup a proper timestamp field, because of some limitations with the date format used in Forcepoint™ logs:
- the date is in the Log Server’s local time but it does not include timezone information;
- it has only precision to the second;
- by default the date in the log is the reception time on the Log Server, not the creation time of the event as reported by the firewall.
The precision issue is logged at Forcepoint™ as a feature request. The timestamp reported by the firewall can be added to the list of CEF fields with SMC > 6.5.2. We managed to workaround the timezone issue in Filebeat by forcing it, the method is absolutely not ideal and it was not straight forward (see comments in filebeat.yml (s'ouvre dans un nouvel onglet)). However, this is were it’s important to have decent scripting capabilities when manipulating logs, to be able to quickly switch to plan B and not get stuck.
Filebeat configuration:
⇨ /etc/filebeat/filebeat.yml (s'ouvre dans un nouvel onglet)
⇨ /etc/filebeat/scripts/event-created.js (s'ouvre dans un nouvel onglet)
⇨ /etc/filebeat/scripts/iana-proto-num2name.js (s'ouvre dans un nouvel onglet)
⇨ /etc/filebeat/scripts/set-cef-rt-to-tmp-timestamp.js (s'ouvre dans un nouvel onglet)
⇨ /etc/filebeat/scripts/set-index-rj.js (s'ouvre dans un nouvel onglet)
Elasticsearch indexing
The indexes holding the data were named ecs-forcepoint-firewall-*, but we should have swapped the firewall and forcepoint terms to be able to create a vendor-independent ecs-firewall-* index pattern in Kibana. In order to get a proper mapping, the following templates were applied to these indexes:
name index_patterns order version
ecs [ecs-*] 1
ecs-forcepoint-firewall [ecs-forcepoint-firewall-*] 500
ecs-forcepoint-firewall-rj [ecs-forcepoint-firewall-rj-*] 700
- ecs (s'ouvre dans un nouvel onglet)
- ecs-forcepoint-firewall (s'ouvre dans un nouvel onglet)
- ecs-forcepoint-firewall-rj (s'ouvre dans un nouvel onglet)
The base template is a snapshot from the ECS project (s'ouvre dans un nouvel onglet), it defines all the fields standardized in the Elastic Common Schema (s'ouvre dans un nouvel onglet). Here we’ve added a settings.index.default_pipeline: ecs property, but we should have kept it vanilla and added another base template matching ecs-*.
The second template defines the non-standard forcepoint.* fields we’ve added because they have no ECS equivalent. We also extend a few ECS fields to index them both as keyword and text, or to set a custom analyzer on them. This is also were we set the shards, replicas and refresh interval for the indexes. The third template is only tuning for indexes created when reinjecting manually old logs exported from the SMC.
Filebeat did not offer a processor for geocoding of IP addresses against MaxMind databases, so we did that part on Elasticsearch side with a pipeline (s'ouvre dans un nouvel onglet) at ingest time. This is also were we drop the internal _ field we needed in Filebeat (see notes in filebeat.yml (s'ouvre dans un nouvel onglet)).
Elasticsearch configuration:
⇨ template ecs (s'ouvre dans un nouvel onglet)
⇨ template ecs-forcepoint-firewall (s'ouvre dans un nouvel onglet)
⇨ template ecs-forcepoint-firewall-rj (s'ouvre dans un nouvel onglet)
⇨ pipeline ecs (s'ouvre dans un nouvel onglet)
Et voilà!
We are now able to search and aggregate way faster than from the SMC. Here is a sample document (s'ouvre dans un nouvel onglet) to give an idea of the result.
Although the original purpose of this experiment was to ease cleaning of firewall rules, it is now possible to create Kibana dashboards. Here is an example:
Greetings
Elasticsearch, https://www.elastic.co/elasticsearch/ (s'ouvre dans un nouvel onglet)
ECS fields reference, https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html (s'ouvre dans un nouvel onglet)
Filebeat, https://www.elastic.co/beats/filebeat (s'ouvre dans un nouvel onglet)
RSyslog, https://www.rsyslog.com/ (s'ouvre dans un nouvel onglet)
Forcepoint™ NGFW, https://www.forcepoint.com/product/ngfw-next-generation-firewall (s'ouvre dans un nouvel onglet)
