Monday, April 6, 2015

Suricata IDPS - Application layer anomalies protocol detection




Suricata IDS/IPS/NSM also allows you to do application layer anomaly  detection.
I started talking to inliniac about protocol anomaly detection rules one day on the Suricata IRC chat room...which evolved more into a discussion resulting in us updating the rule sets with some examples of how to do that.

Below are a few examples for rules usage:

HTTP

alert tcp any any -> any ![80,8080] (msg:"SURICATA HTTP not tcp port 80, 8080"; flow:to_server; app-layer-protocol:http; sid:2271001; rev:1;)
The above rule finds http traffic that is not using dest port 80 or 8080.


alert tcp any any -> any 80 (msg:"SURICATA Port 80 but not HTTP"; flow:to_server; app-layer-protocol:!http; sid:2271002; rev:1;)
The above rule is kind of the reverse of the previous one - it will alert if the tcp traffic with destination port 80 is not http.

Here is another example

TLS

alert tcp any any -> any 443 (msg:"SURICATA Port 443 but not TLS"; flow:to_server; app-layer-protocol:!tls; sid:2271003; rev:1;)

HTTPS

Detecting HTTP traffic over HTTPS port -

alert http any any -> any 443 (msg:"SURICATA HTTP clear text on port 443"; flow:to_server; app-layer-protocol:http; sid:2271019; rev:1;)

You can find the full ruleset (open source and free to use) with examples for HTTP, HTTPS, TLS, FTP, SMTP, SSH, IMAP, SMB, DCERPC, DNS, MODBUS application layer anomaly  detection  here:

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Protocol_Anomalies_Detection







Thursday, February 19, 2015

Chasing MTUs


Setting up (configuring) the right MTU (maximum transmission unit) size when running Suricata IDS/IPS.

Sometimes you can end up in a situation as follows :


capture.kernel_packets    | AFPacketeth12              | 1143428204
decoder.pkts                    | AFPacketeth12             | 1143428143
decoder.invalid                | AFPacketeth12              | 416889536

a whole lot of  decoder.invalid. Not good. What could be the reason for that? One thing you should check right away is the MTU of the traffic that is being mirrored.

What does it mean? Well there is the MTU that you set up on the server that you run Suricata on and there is the MTU that is present in the "mirrored" traffic.

What is the difference?Why should it matter?
It matters because if not set correct  it will result in a lot of decoder.invalids (dropped by Suricata) and you will be missing on a lot of traffic inspection.
Example: if  on the sniffing interface that you run Suricata on has a MTU set as 1500  and in the traffic that you mirror you have jumbo frames (MTU 9000) - most likely your decoder.invalids will show a whole lotta love in your stats.log.

How can you adjust the MTU on the interface (NIC) ? (example)
First a have  look what is the current value:
ifconfig eth0
then adjust it
ifconfig eth0 mtu 1514

By the way - what could be the max size of the MTU (and what sizes there are in general)  -
(short answer - 9216)


This is the easy part :). There are situations where you do not know what is the MTU of the "mirrored" traffic. There is a few ways to find this  - ask the network team/guy, make a phone call or two, start manually testing and setting it on the NIC to find a middle ground ....however you can also make use of the procedure shown below (in order to get the byte size of the MTU):


On your Server/Sensor
1)
Stop Suricata.

2)
Change the MTU to 9216
(the interface that Suri is sniffing on)

example - ifconfig eth0 mtu 9216
(non boot persistent)

3)
install tcpstat - if you do not have it
apt-get install tcpstat

5)
run the following (substitute the interface name with yours - that Suri is sniffing on)
tcpstat -i eth0 -l -o "Time:%S\tn=%n\tavg=%a\tstddev=%d\tbps=%b\tMaxPacketSize=%M\n"  5
6)
Give it a minute or two
If there are Jumbo frames you should see that in the output (something like) -
"MaxPacketSize=9000", if not you should see whatever the max size is.

7)
Adjust your interface MTU accordingly  - the one that Suri is sniffing
on. -> Start Suri

8)
Let it run for  a while - lets say 1 hr. Have a look at the decoder.invalid stats in stats.log

NOTE: Do NOT just set the MTU to 9216 directly ("just to be on the safe side"). Only set it that high if needed !!

NOTE: This example below is not using the "-l" option of tcpstat as denoted in point 5) above - look at man tcpstat for more info



(tested on Ubuntu/Debian)
That's all ....feedback welcome.









Sunday, December 7, 2014

Suricata - disable detection mode


There is a trick for using Suricata IDS/IPS that has come in handy - in my experience that I thought  would share and might be useful to others.

My HW was CPU 1x E5-2680 with 64 GB RAM ,  NIC a  82599EB 10-Gigabit SFI/SFP+ and mirroring about 9,5Gbps:





So I wanted to get a better understanding of what is the max log output from Suricata in that particular environment without putting detection into consideration. Just pure event logging and profiling - DNS,HTTP,TLS,SSH, File transactions, SMTP all of these.

Suricata offers just that - with a really low need for HW (having in mind we are looking at 9,5Gbps). What i did was compile suricata with the "--disable-detection" switch. What it does it simply disables detection(alerts) in Suricata  - however every other logging/parsing capability is preserved ( DNS,HTTP,TLS,SSH, File transactions, SMTP). So i downloaded a fresh copy:

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

then

./autogen.sh && ./configure --disable-detection   &&  make clean && make && make install &&  ldconfig

enabled all JSON outputs in the eve-log section in suricata.yaml. I confirmed that detection was disabled:


and started Suricata.

Careful what you asked for :) - I was getting 10-15K logs per second :

 root@suricata:/var/log/suricata# tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
1
6531
13860
10704
10877
10389
10664
10205
9996
14798
15427
14223

And the HW(CPU/RAM/HDD) usage was not much at all:


As you can see - 30-40% CPU with a third of RAM  usage.

 I used this command to count logs per second  -
tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
more about similar commands and counting Suricata logs you could find here:


So this can came in handy in a few scenarious:
  • you can  actually do a  "profiling" run on that particular set up in order to size up a SIEM specification 
  • and/or you can size up your prod IDS/IPS deployment needs
  • you can also just feed all those logs info to an existing log analysis system

Thanks

Suricatasc unix socket interaction for Suricata IDS/IPS



Suricatasc is a unix socket interaction  script that is automatically installed when one compiles/installs Suricata IDS/IPS. An in depth description, prerequisites and how to documentation is located here - https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Interacting_via_Unix_Socket

However  lets look at a quick usage example - that can come very handy in certain situations.

Once you have unix socket command enabled in suricata.yaml :

unix-command:
    enabled: yes
    #filename: custom.socket # use this to specify an alternate file

the traditional way to use the script would be type suricatasc and hit Enter (on the machine running Suricata):




However you can also use it directly as a command line parameter for example :
root@suricata:~# suricatasc -c version

like so:


NOTE:
You need to quote commands involving interfaces:
root@debian64:~# suricatasc -c "iface-stat eth0"



Very handy when you want quick interaction and info from the currently running Suricata IDS/IPS.




Sunday, October 26, 2014

Suricata ids/ips - dropping privileges



This tutorial is intended for Linux (Debian/Ubuntu).

Install the prerequisite packages in order to compile Suricata. I add/enable some optional features so in my case I usually do:
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev \
build-essential autoconf automake libtool libpcap-dev libnet1-dev \
libyaml-0-2 libyaml-dev zlib1g zlib1g-dev make flex bison \
libmagic-dev

For Eve (all JSON output):
apt-get install libjansson-dev libjansson4
For MD5 support(file extraction):
apt-get install libnss3-dev libnspr4-dev
For GeoIP:
apt-get install libgeoip1 libgeoip-dev
For nfqueue(ips mode):
apt-get install libnetfilter-queue-dev libnetfilter-queue1 libnfnetlink-dev libnfnetlink0
For the dropping privileges part you can simply do:
apt-get install libcap-ng0 libcap-ng-dev

OR get the latest libcap-ng version form here:
http://people.redhat.com/sgrubb/libcap-ng/
like so:

wget http://people.redhat.com/sgrubb/libcap-ng/libcap-ng-0.7.4.tar.gz
tar -zxf libcap-ng-0.7.4.tar.gz
cd libcap-ng-0.7.4
./configure && make && make install
cd ..

Let's fetch and compile Suricata:
wget http://www.openinfosecfoundation.org/download/suricata-2.0.4.tar.gz
tar -xzf suricata-2.0.4.tar.gz 
cd suricata-2.0.4
 One liner... one of my favorite:

./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --disable-gccmarch-native \
--enable-geoip --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ \
--enable-nfqueue \
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include \
--with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr && \
make clean && make && make install-full && ldconfig


Above we enable some other features like :
(
you can do like this
root@IDS:~/suricata-2.0.4# ./configure --help
to see what each option is for
)

but this line -
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include
is the one you need to compile and enable dropping privileges with Suricata.


Then you can run Suri like so
/usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Make sure the log directory has the right permissions to allow the user "logstash" to write to it.
After you start Suricata  - you should see something similar:
root@IDS:~# ls -lh /var/log/suricata/
total 77M
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 certs
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 core
-rw-r----- 1 logstash logstash  18M Oct 26 10:48 eve.json
-rw-r----- 1 logstash logstash 806K Oct 26 10:48 fast.log
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 files
drwxr-xr-x 2 logstash logstash 4.0K Oct 26 06:26 StatsByDate
-rw-r--r-- 1 root     root      58M Oct 26 10:48 stats.log
-rw-r--r-- 1 root     root     1.1K Oct 26 09:15 suricata-start.log
root@IDS:~#
Notice the user logstash ownership.

root@IDS:~# ps aux |grep suricata
logstash  2189 11.0 10.6 420448 219972 ?       Ssl  09:15  13:04 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash
root@IDS:~#
Now you have the user logstash running (not as root) Suricata IDS/IPS.




Sunday, August 24, 2014

Suricata - more data for your alerts


As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of packet data and information in a standard JSON output logging capability supplementing further the alert logging output.

This guide makes use of Suricata and ELK - Elasticsearch, Logstash, Kibana.
You can install all of them following the guide HERE
 ...or you can download and try out SELKS  and use directly.


After everything is in place, we need to open the suricata.yaml and make the following editions in the eve.json section:

 # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert:

            payload: yes           # enable dumping payload in Base64
            payload-printable: yes # enable dumping payload in printable (lossy) format
            packet: yes            # enable dumping of packet (without stream segments)
            http: yes              # enable dumping of http fields
       
You can start Suricata and let it inspect traffic for some time in order to generate alert log data.
Then navigate to your Kibana web interface, find an alert record/log and you could see the usefulness of the extra data yourself.

Some examples though :) :






Lets kick it up  notch.....

We want to search through -
  1. all the generated alerts that have 
  2. a printable payload data 
  3. that have the following string: uid=0(root)
 Easy, here is the query:
 
payload_printable:"uid=0\(root\)"
You should enter it like this in Kibana:



Well what do you know - we got what we were looking for:




Some more useful reading on the Lucene Query Syntax (you should at least have a look :) ):
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

http://www.solrtutorial.com/solr-query-syntax.html

http://lucene.apache.org/core/2_9_4/queryparsersyntax.html






Suricata - Flows, Flow Managers and effect on performance



As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of high performance/advanced tuning for custom thread configuration for the IDS/IPS engine management threads.

Aka ..these
[27521] 20/7/2014 -- 01:46:19 - (tm-threads.c:2206) <Notice>  (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.


These 3 management threads initialized above are flow manager (1), counter/stats related threads (2x)

So ... in the default suricata.yaml setting we have:

flow:
  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  #managers: 1 # default to one flow manager
  #recyclers: 1 # default to one flow recycler thread

and we can choose accordingly of how many threads we would like to dedicate for the management tasks within the engine itself.
The recyclers threads offload part of the flow managers work and if enabled do flow/netflow logging.

Good !
What does this has to do with performance?

Suricata IDS/IPS is powerful, flexible and scalable - so be careful what you wish for.
The examples below demonstrate the effect on a 10Gbps Suricata IDS sensor.

Example 1


suricata.yaml config - >
flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30
  prune-flows: 50000
  managers: 2 # default is 1

CPU usage ->

 2 flow management threads use 8% CPU each

 Example 2



suricata.yaml config - >
flow:
  memcap: 4gb
  hash-size: 15728640
  prealloc: 8000000
  emergency-recovery: 30
  managers: 2 # default is 1

 CPU usage ->

2 flow management threads use 39% CPU each as compared to Example 1 !!


So a 4 fold increase in memcap, 8 fold increase in prealloc and 15 fold increase on hash-size settings leads to about 3 fold increase in RAM consumption and 5 fold on CPU consumption  - in terms of flow management thread usage.

It would be very rare that you would need the settings in Example 2 - you need huge traffic for that...

So how would you know when to tune/adjust those settings in suricata.yaml? It is recommended that you always keep an eye on your stats.log and make sure you do not enter emergency clean up mode:



it should always be 0

Some additional reading on flows and flow managers -
http://blog.inliniac.net/2014/07/28/suricata-flow-logging/