Friday, May 22, 2015

Suricata - wildcard rule loading


Recently (few hrs ago as of writing this blog) there was  a new feature (thanks to gozzy) introduced in Suricata IDS/IPS/NSM  - wildcard rule loading capability.

As of the moment the feature is available in our git master. If you are wondering how to get that up and running or do not have the latest Suricata from git master - here is a quick tutorial (Debian/Ubuntu):

1)
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev build-essential \
autoconf automake libtool libpcap-dev libnet1-dev libyaml-0-2 \
libyaml-dev zlib1g zlib1g-dev libmagic-dev libcap-ng-dev \
libjansson-dev pkg-config libnss3-dev libnspr4-dev git-core

2)
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

3)
 ./autogen.sh && \
 ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ \
 --enable-geoip --enable-unix-socket \
 --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ \
 --with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr \
 && make clean && make && make install-full && ldconfig

To confirm -
suricata --build-info

Now that you have latest Suricta up and running - here it is what this blog post is all about  - wildcard rule loading for Suricata IDPS. Some possible scenarios of use are loading wildcarded rules form the :

Command line


Please note the "quotes" !
suricata -c /etc/suricata/suricata.yaml  -v -i eth0 -S "/etc/suricata/rules/*.rules"

Pretty self explanatory. The command above will load all .rules files from /etc/suricata/rules/
suricata -c /etc/suricata/suricata.yaml  -v -i eth0 -S "/etc/suricata/rules/emerging*"
The command above will load all emerging* rules files from /etc/suricata/rules/

Config file


You can also set that up in the suricata.yaml config file. Here is how (please note the "quotes").

In your rules section in the suricata.yaml:

# Set the default rule path here to search for the files.
# if not set, it will look at the current working dir
default-rule-path: /etc/suricata/rules
rule-files:
 #- "*.rules"
 - "emerging*"
 #- botcc.rules
 #- ciarmy.rules
 #- compromised.rules
 #- drop.rules
 #- dshield.rules
 #- emerging-activex.rules
 #- emerging-attack_response.rules
The set up above will load all emerging* files and the rules residing in those. Then you can start Suricata anyway you would like, examples:

suricata -c /etc/suricata/suricata.yaml  -v -i eth0
suricata -c /etc/suricata/suricata.yaml  -v --af-packet=eth0

 and in suricata.log you should see all emerging* rule files being loaded:

......
[13558] 22/5/2015 -- 17:19:39 - (reputation.c:620) <Info> (SRepInit) -- IP reputation disabled
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-activex.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-attack_response.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-chat.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-current_events.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-deleted.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:420) <Warning> (ProcessSigFiles) -- [ERRCODE: SC_ERR_NO_RULES(42)] - No rules loaded from /etc/suricata/rules/emerging-deleted.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-dns.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-dos.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-exploit.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-ftp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-games.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-icmp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:420) <Warning> (ProcessSigFiles) -- [ERRCODE: SC_ERR_NO_RULES(42)] - No rules loaded from /etc/suricata/rules/emerging-icmp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-icmp_info.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-imap.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-inappropriate.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-info.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-malware.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-misc.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-mobile_malware.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-netbios.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-p2p.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-policy.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-pop3.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-rpc.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-scada.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-scan.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-shellcode.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-smtp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-snmp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-sql.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-telnet.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-tftp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-trojan.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-user_agents.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-voip.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_client.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_server.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_specific_apps.rules
[13558] 22/5/2015 -- 17:19:43 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-worm.rules
.......

You can also use it like that :

# Set the default rule path here to search for the files.
# if not set, it will look at the current working dir
default-rule-path: /etc/suricata/rules
rule-files:
 #- "*.rules"
 - "*web*"
 #- "emerging*"
 #- botcc.rules
 #- ciarmy.rules
 #- compromised.rules
 #- drop.rules


That's it.


Thursday, May 21, 2015

Suricata - multiple interface configuration with af-packet



Suricata is a very flexible and powerful multithreading  IDS/IPS/NSM.

Here is a simple tutorial (tested on Debian/Ubuntu) of how to configure multiple interfaces for af-packet mode with Suricata (af-packet mode works by default/out of the box on kernels 3.2 and above). Lets say you would like to start simple IDSing with Suricata on eth1, eth2 and eth3 on a particular machine/server.


In your suricata.yaml config (usually located in /etc/suricata/) find the af-packet section and do the following:


af-packet:
  - interface: eth2
    threads: 16
    cluster-id: 98
    cluster-type: cluster_cpu
    defrag: no
    use-mmap: yes
    ring-size: 200000
    checksum-checks: kernel
  - interface: eth1
    threads: 2
    cluster-id: 97
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    ring-size: 30000
  - interface: eth3
    threads: 2
    cluster-id: 96
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    ring-size: 20000
Of course feel free to adjust the ring-sizes (packet buffers) as you see fit for your particular set up.
NOTE:  do not forget to use a different cluster-id

so now you can start suricata like so:

suricata -c /etc/suricata/suricata.yaml -v --af-packet 

That above will start Suricata which will listen on eth2 with 16 threads with cluster_type: cluster_cpu and on eth1,eth3 with 2 threads each with cluster_type: cluster_flow. Have a look in your suricata.log file for more info.

If you would like to just test and see how it goes for eth2 only:
suricata -c /etc/suricata/suricata.yaml -v --af-packet=eth2

...easy and flexible.







Saturday, April 25, 2015

Suricata - check loaded yaml config settings with --dump-config



There is a very useful command available to Suricata IDS/IPS/NSM :
suricata --dump-config

The command above will dump all the config parameters and their respective values that are loaded by Suricata from the config file. You can run the command in any case - it does not matter if Suricata is running or not.

There is a peculiarity however. Sometimes people would think that the command(above) would dump the currently loaded config values by Suricata.... in some case it will and in some cases it will not.

So what does it depend on?.... simple:
suricata --dump-config

will dump the config settings that are loaded (or will be loaded) by Suricata by default from
/etc/suricata/suricata.yaml

So if you are running Suricata with a config file called suricata-test.yaml (or suricata.yaml located in a different directory) - you will not see those settings...unless you specify that config file in particular:
suricata --dump-config -c /etc/suricata/suricata-test.yaml
Here is a real case example.
I run Suricata for a specific test where I had specified the defrag memcap to be 512mb :
defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep (higher than trackers)
  prealloc: yes
  timeout: 60

Suricata up and running:
root@LTS-64-1:~/Work # ps aux |grep suricata
root      8109  2.3  7.6 878444 308372 pts/6   Sl+  12:45   1:02 suricata -c /etc/suricata/suricata-test.yaml --af-packet=eth0 -v
root@LTS-64-1:~/Work #

And the peculiarity that this blogpost is trying to emphasize on about :
root@LTS-64-1:~/Work # suricata --dump-config  |grep defrag.memcap
defrag.memcap = 32mb
root@LTS-64-1:~/Work # suricata --dump-config -c /etc/suricata/suricata-test.yaml |grep defrag.memcap
defrag.memcap = 512mb
root@LTS-64-1:~/Work #



suricata --dump-config dumps the settings loaded(or to be loaded) from the default location /etc/suricata/suricata.yaml if you are running suricata with a yaml config with a different name than the default or with a different location that the default - in order to get those settings - you need to specify that particular yaml location, like so:

suricata --dump-config -c /etc/local/some_test_dir/suricata/suricata-test.yaml


Thanks

related article:
http://pevma.blogspot.se/2014/02/suricata-override-config-parameters-on.html


Monday, April 6, 2015

Suricata IDPS - Application layer anomalies protocol detection




Suricata IDS/IPS/NSM also allows you to do application layer anomaly  detection.
I started talking to inliniac about protocol anomaly detection rules one day on the Suricata IRC chat room...which evolved more into a discussion resulting in us updating the rule sets with some examples of how to do that.

Below are a few examples for rules usage:

HTTP

alert tcp any any -> any ![80,8080] (msg:"SURICATA HTTP not tcp port 80, 8080"; flow:to_server; app-layer-protocol:http; sid:2271001; rev:1;)
The above rule finds http traffic that is not using dest port 80 or 8080.


alert tcp any any -> any 80 (msg:"SURICATA Port 80 but not HTTP"; flow:to_server; app-layer-protocol:!http; sid:2271002; rev:1;)
The above rule is kind of the reverse of the previous one - it will alert if the tcp traffic with destination port 80 is not http.

Here is another example

TLS

alert tcp any any -> any 443 (msg:"SURICATA Port 443 but not TLS"; flow:to_server; app-layer-protocol:!tls; sid:2271003; rev:1;)

HTTPS

Detecting HTTP traffic over HTTPS port -

alert http any any -> any 443 (msg:"SURICATA HTTP clear text on port 443"; flow:to_server; app-layer-protocol:http; sid:2271019; rev:1;)

You can find the full ruleset (open source and free to use) with examples for HTTP, HTTPS, TLS, FTP, SMTP, SSH, IMAP, SMB, DCERPC, DNS, MODBUS application layer anomaly  detection  here:

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Protocol_Anomalies_Detection







Thursday, February 19, 2015

Chasing MTUs


Setting up (configuring) the right MTU (maximum transmission unit) size when running Suricata IDS/IPS.

Sometimes you can end up in a situation as follows :


capture.kernel_packets    | AFPacketeth12              | 1143428204
decoder.pkts                    | AFPacketeth12             | 1143428143
decoder.invalid                | AFPacketeth12              | 416889536

a whole lot of  decoder.invalid. Not good. What could be the reason for that? One thing you should check right away is the MTU of the traffic that is being mirrored.

What does it mean? Well there is the MTU that you set up on the server that you run Suricata on and there is the MTU that is present in the "mirrored" traffic.

What is the difference?Why should it matter?
It matters because if not set correct  it will result in a lot of decoder.invalids (dropped by Suricata) and you will be missing on a lot of traffic inspection.
Example: if  on the sniffing interface that you run Suricata on has a MTU set as 1500  and in the traffic that you mirror you have jumbo frames (MTU 9000) - most likely your decoder.invalids will show a whole lotta love in your stats.log.

How can you adjust the MTU on the interface (NIC) ? (example)
First a have  look what is the current value:
ifconfig eth0
then adjust it
ifconfig eth0 mtu 1514

By the way - what could be the max size of the MTU (and what sizes there are in general)  -
(short answer - 9216)


This is the easy part :). There are situations where you do not know what is the MTU of the "mirrored" traffic. There is a few ways to find this  - ask the network team/guy, make a phone call or two, start manually testing and setting it on the NIC to find a middle ground ....however you can also make use of the procedure shown below (in order to get the byte size of the MTU):


On your Server/Sensor
1)
Stop Suricata.

2)
Change the MTU to 9216
(the interface that Suri is sniffing on)

example - ifconfig eth0 mtu 9216
(non boot persistent)

3)
install tcpstat - if you do not have it
apt-get install tcpstat

5)
run the following (substitute the interface name with yours - that Suri is sniffing on)
tcpstat -i eth0 -l -o "Time:%S\tn=%n\tavg=%a\tstddev=%d\tbps=%b\tMaxPacketSize=%M\n"  5
6)
Give it a minute or two
If there are Jumbo frames you should see that in the output (something like) -
"MaxPacketSize=9000", if not you should see whatever the max size is.

7)
Adjust your interface MTU accordingly  - the one that Suri is sniffing
on. -> Start Suri

8)
Let it run for  a while - lets say 1 hr. Have a look at the decoder.invalid stats in stats.log

NOTE: Do NOT just set the MTU to 9216 directly ("just to be on the safe side"). Only set it that high if needed !!

NOTE: This example below is not using the "-l" option of tcpstat as denoted in point 5) above - look at man tcpstat for more info



(tested on Ubuntu/Debian)
That's all ....feedback welcome.









Sunday, December 7, 2014

Suricata - disable detection mode


There is a trick for using Suricata IDS/IPS that has come in handy - in my experience that I thought  would share and might be useful to others.

My HW was CPU 1x E5-2680 with 64 GB RAM ,  NIC a  82599EB 10-Gigabit SFI/SFP+ and mirroring about 9,5Gbps:





So I wanted to get a better understanding of what is the max log output from Suricata in that particular environment without putting detection into consideration. Just pure event logging and profiling - DNS,HTTP,TLS,SSH, File transactions, SMTP all of these.

Suricata offers just that - with a really low need for HW (having in mind we are looking at 9,5Gbps). What i did was compile suricata with the "--disable-detection" switch. What it does it simply disables detection(alerts) in Suricata  - however every other logging/parsing capability is preserved ( DNS,HTTP,TLS,SSH, File transactions, SMTP). So i downloaded a fresh copy:

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

then

./autogen.sh && ./configure --disable-detection   &&  make clean && make && make install &&  ldconfig

enabled all JSON outputs in the eve-log section in suricata.yaml. I confirmed that detection was disabled:


and started Suricata.

Careful what you asked for :) - I was getting 10-15K logs per second :

 root@suricata:/var/log/suricata# tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
1
6531
13860
10704
10877
10389
10664
10205
9996
14798
15427
14223

And the HW(CPU/RAM/HDD) usage was not much at all:


As you can see - 30-40% CPU with a third of RAM  usage.

 I used this command to count logs per second  -
tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
more about similar commands and counting Suricata logs you could find here:


So this can came in handy in a few scenarious:
  • you can  actually do a  "profiling" run on that particular set up in order to size up a SIEM specification 
  • and/or you can size up your prod IDS/IPS deployment needs
  • you can also just feed all those logs info to an existing log analysis system

Thanks

Suricatasc unix socket interaction for Suricata IDS/IPS



Suricatasc is a unix socket interaction  script that is automatically installed when one compiles/installs Suricata IDS/IPS. An in depth description, prerequisites and how to documentation is located here - https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Interacting_via_Unix_Socket

However  lets look at a quick usage example - that can come very handy in certain situations.

Once you have unix socket command enabled in suricata.yaml :

unix-command:
    enabled: yes
    #filename: custom.socket # use this to specify an alternate file

the traditional way to use the script would be type suricatasc and hit Enter (on the machine running Suricata):




However you can also use it directly as a command line parameter for example :
root@suricata:~# suricatasc -c version

like so:


NOTE:
You need to quote commands involving interfaces:
root@debian64:~# suricatasc -c "iface-stat eth0"



Very handy when you want quick interaction and info from the currently running Suricata IDS/IPS.