Sunday, October 26, 2014

Suricata ids/ips - dropping privileges

This tutorial is intended for Linux (Debian/Ubuntu).

Install the prerequisite packages in order to compile Suricata. I add/enable some optional features so in my case I usually do:
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev \
build-essential autoconf automake libtool libpcap-dev libnet1-dev \
libyaml-0-2 libyaml-dev zlib1g zlib1g-dev make flex bison \

For Eve (all JSON output):
apt-get install libjansson-dev libjansson4
For MD5 support(file extraction):
apt-get install libnss3-dev libnspr4-dev
For GeoIP:
apt-get install libgeoip1 libgeoip-dev
For nfqueue(ips mode):
apt-get install libnetfilter-queue-dev libnetfilter-queue1 libnfnetlink-dev libnfnetlink0
For the dropping privileges part you can simply do:
apt-get install libcap-ng0 libcap-ng-dev

OR get the latest libcap-ng version form here:
like so:

tar -zxf libcap-ng-0.7.4.tar.gz
cd libcap-ng-0.7.4
./configure && make && make install
cd ..

Let's fetch and compile Suricata:
tar -xzf suricata-2.0.4.tar.gz 
cd suricata-2.0.4
 One liner... one of my favorite:

./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --disable-gccmarch-native \
--enable-geoip --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ \
--enable-nfqueue \
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include \
--with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr && \
make clean && make && make install-full && ldconfig

Above we enable some other features like :
you can do like this
root@IDS:~/suricata-2.0.4# ./configure --help
to see what each option is for

but this line -
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include
is the one you need to compile and enable dropping privileges with Suricata.

Then you can run Suri like so
/usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/ --af-packet -D -v --user=logstash

Make sure the log directory has the right permissions to allow the user "logstash" to write to it.
After you start Suricata  - you should see something similar:
root@IDS:~# ls -lh /var/log/suricata/
total 77M
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 certs
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 core
-rw-r----- 1 logstash logstash  18M Oct 26 10:48 eve.json
-rw-r----- 1 logstash logstash 806K Oct 26 10:48 fast.log
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 files
drwxr-xr-x 2 logstash logstash 4.0K Oct 26 06:26 StatsByDate
-rw-r--r-- 1 root     root      58M Oct 26 10:48 stats.log
-rw-r--r-- 1 root     root     1.1K Oct 26 09:15 suricata-start.log
Notice the user logstash ownership.

root@IDS:~# ps aux |grep suricata
logstash  2189 11.0 10.6 420448 219972 ?       Ssl  09:15  13:04 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/ --af-packet -D -v --user=logstash
Now you have the user logstash running (not as root) Suricata IDS/IPS.

Sunday, August 24, 2014

Suricata - more data for your alerts

As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of packet data and information in a standard JSON output logging capability supplementing further the alert logging output.

This guide makes use of Suricata and ELK - Elasticsearch, Logstash, Kibana.
You can install all of them following the guide HERE
 ...or you can download and try out SELKS  and use directly.

After everything is in place, we need to open the suricata.yaml and make the following editions in the eve.json section:

 # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
        - alert:

            payload: yes           # enable dumping payload in Base64
            payload-printable: yes # enable dumping payload in printable (lossy) format
            packet: yes            # enable dumping of packet (without stream segments)
            http: yes              # enable dumping of http fields
You can start Suricata and let it inspect traffic for some time in order to generate alert log data.
Then navigate to your Kibana web interface, find an alert record/log and you could see the usefulness of the extra data yourself.

Some examples though :) :

Lets kick it up  notch.....

We want to search through -
  1. all the generated alerts that have 
  2. a printable payload data 
  3. that have the following string: uid=0(root)
 Easy, here is the query:
You should enter it like this in Kibana:

Well what do you know - we got what we were looking for:

Some more useful reading on the Lucene Query Syntax (you should at least have a look :) ):

Suricata - Flows, Flow Managers and effect on performance

As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of high performance/advanced tuning for custom thread configuration for the IDS/IPS engine management threads.

Aka ..these
[27521] 20/7/2014 -- 01:46:19 - (tm-threads.c:2206) <Notice>  (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.

These 3 management threads initialized above are flow manager (1), counter/stats related threads (2x)

So ... in the default suricata.yaml setting we have:

  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  #managers: 1 # default to one flow manager
  #recyclers: 1 # default to one flow recycler thread

and we can choose accordingly of how many threads we would like to dedicate for the management tasks within the engine itself.
The recyclers threads offload part of the flow managers work and if enabled do flow/netflow logging.

Good !
What does this has to do with performance?

Suricata IDS/IPS is powerful, flexible and scalable - so be careful what you wish for.
The examples below demonstrate the effect on a 10Gbps Suricata IDS sensor.

Example 1

suricata.yaml config - >
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30
  prune-flows: 50000
  managers: 2 # default is 1

CPU usage ->

 2 flow management threads use 8% CPU each

 Example 2

suricata.yaml config - >
  memcap: 4gb
  hash-size: 15728640
  prealloc: 8000000
  emergency-recovery: 30
  managers: 2 # default is 1

 CPU usage ->

2 flow management threads use 39% CPU each as compared to Example 1 !!

So a 4 fold increase in memcap, 8 fold increase in prealloc and 15 fold increase on hash-size settings leads to about 3 fold increase in RAM consumption and 5 fold on CPU consumption  - in terms of flow management thread usage.

It would be very rare that you would need the settings in Example 2 - you need huge traffic for that...

So how would you know when to tune/adjust those settings in suricata.yaml? It is recommended that you always keep an eye on your stats.log and make sure you do not enter emergency clean up mode:

it should always be 0

Some additional reading on flows and flow managers -

Suricata - filtering tricks for the fileinfo output with eve.json

As of Suricata 2.0  - Suricata IDS/IPS provides the availability of a standard JSON output logging capability. This guide makes use of Suricata and ELK - Elasticsearch, Logstash, Kibana.

You can install all of them following the guide HERE
 ...or you can download and try out SELKS  and use directly.

Once you have the installation in place and have the Kibana web interface up and running you can make use of the following fileinfo filters (tricks :).
You can enter the queries like so:

 fileinfo.magic:"PE32" -fileinfo.filename:*exe
will show you all "PE32 executable" executables that were seen transferred that have no exe extension in their file name:

fileinfo.magic:"pdf" -fileinfo.filename:*pdf

will show you all "PDF document version......" files that were transferred that have no PDF extension in their file name.

You can explore further :)

Saturday, August 23, 2014

Suricata IDS/IPS - HTTP custom header logging

As a continuation of the article HERE- some more screenshots from the ready to use template....

For the Elasticsearch/Logstash/Kibana users there is a ready to use template that you could download from here - "HTTP-Extended-Custom"

Monday, July 28, 2014

Compiling your own perf utility tools

The perf utility is part of the linux-tools package and can be installed via :
apt-get install linux-tools

However..... if you have just installed a newer (and or custom) kernel version (as explained here for example)....let's say 3.14.0 on Ubuntu Precise ... the apt-get install would not find the appropriate linux-tools package for that kernel version.

So this is what yo can do:

apt-get install libdw-dev libnewt-dev binutils-dev git
git clone
cd linux
git tag -l

Make sure you checkout the same version as on the currently installed on the system kernel.

git checkout tags/v3.14
cd tools/perf
make -j `getconf _NPROCESSORS_ONLN` perf
make install

root@suricata:~/oisf# perf list

List of pre-defined events (to be used in -e):
  cpu-cycles OR cycles                               [Hardware event]
  instructions                                       [Hardware event]
  cache-references                                   [Hardware event]
  cache-misses                                       [Hardware event]
  branch-instructions OR branches                    [Hardware event]
  branch-misses                                      [Hardware event]
  bus-cycles                                         [Hardware event]
  stalled-cycles-frontend OR idle-cycles-frontend    [Hardware event]
  stalled-cycles-backend OR idle-cycles-backend      [Hardware event]
  ref-cycles                                         [Hardware event]

  cpu-clock                                          [Software event]
  task-clock                                         [Software event]
  page-faults OR faults                              [Software event]
  context-switches OR cs                             [Software event]
  cpu-migrations OR migrations                       [Software event]
  minor-faults                                       [Software event]
  major-faults                                       [Software event]
  alignment-faults                                   [Software event]
  emulation-faults                                   [Software event]
  dummy                                              [Software event]

  L1-dcache-loads                                    [Hardware cache event]
  L1-dcache-load-misses                              [Hardware cache event]
  L1-dcache-stores                                   [Hardware cache event]
  L1-dcache-store-misses                             [Hardware cache event]
  L1-dcache-prefetch-misses                          [Hardware cache event]
  L1-icache-load-misses                              [Hardware cache event]
  LLC-loads                                          [Hardware cache event]
  LLC-load-misses                                    [Hardware cache event]
  LLC-stores                                         [Hardware cache event]
  LLC-store-misses                                   [Hardware cache event]
  LLC-prefetches                                     [Hardware cache event]
  LLC-prefetch-misses                                [Hardware cache event]
  dTLB-loads                                         [Hardware cache event]
  dTLB-load-misses                                   [Hardware cache event]
  dTLB-stores                                        [Hardware cache event]
  dTLB-store-misses                                  [Hardware cache event]
  iTLB-loads                                         [Hardware cache event]
  iTLB-load-misses                                   [Hardware cache event]
  branch-loads                                       [Hardware cache event]
  branch-load-misses                                 [Hardware cache event]
  node-loads                                         [Hardware cache event]
  node-load-misses                                   [Hardware cache event]
  node-stores                                        [Hardware cache event]
  node-store-misses                                  [Hardware cache event]
  node-prefetches                                    [Hardware cache event]
  node-prefetch-misses                               [Hardware cache event]

  branch-instructions OR cpu/branch-instructions/    [Kernel PMU event]
  branch-misses OR cpu/branch-misses/                [Kernel PMU event]
  bus-cycles OR cpu/bus-cycles/                      [Kernel PMU event]
  cache-misses OR cpu/cache-misses/                  [Kernel PMU event]
  cache-references OR cpu/cache-references/          [Kernel PMU event]
  cpu-cycles OR cpu/cpu-cycles/                      [Kernel PMU event]
  instructions OR cpu/instructions/                  [Kernel PMU event]
  mem-loads OR cpu/mem-loads/                        [Kernel PMU event]
  mem-stores OR cpu/mem-stores/                      [Kernel PMU event]

root@suricata:~/oisf#perf top you are back in business.

Monday, July 7, 2014

Kernel upgrade for Debian

Kernel upgrade for Debian - a quick and useful tutorial. This guide will also show how to make kernel*.deb - debian packages ready for easy "dpkg -i ..* " installation of your desired kernel version.

Some packages you might want to have on the system prior to building the new kernel:

apt-get install wget fakeroot kernel-package gcc libncurses5-dev bc ca-certificates pkg-config make flex bison build-essential autoconf automake

Choose the kernel you want to build Debian packages for from here:

Please make sure you read here :
and here:

and know the difference between "Mainline,Stable and Longterm" with regards to kernel versions.

Then you could look up the kernel version changes in here:

For the purpose of this tutorial I have chosen kernel version 3.15.1

tar xfJ linux-3.15.1.tar.xz
cd linux-3.15.1

If you will be updating/installing the new kernel on the same machine that you will be building it, you could do:
yes "" | make oldconfig

If you will be updating/installing the new kernel on a different machine than the one that you are building it - it is better to do:
make defconfig

More about these differences and the implications of it you can read here:
and here

    make defconfig - Set all options to default values
    make allnoconfig - Set all yes/no options to "n"
    make allyesconfig - Set all yes/no options to "y"
    make allmodconfig - Set all yes/no options to "y" and all "yes/module/no" options to "m"
    make randconfig - Set each option randomly (for debugging purposes).
    make oldconfig - Update a .config file from a previous version of the kernel to work with the current version.

Then we do:

make clean && \
make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom KDEB_PKGVERSION=3.15.1
make -j `getconf _NPROCESSORS_ONLN`
means that if you have 4 CPUs it will start 4 kernel make jobs in parallel - provides for much faster building.
Now in the directory above
cd ..
you should have the new kernel.deb packages for the same architecture as the machine that you just build it on - example 64 bit

drwxrwxr-x 25 root root 4.0K Jul  7 14:40 linux-3.15.1
-rw-r--r--  1 root root  76M Jun 16 16:54 linux-3.15.1.tar.xz
-rw-r--r--  1 root root 9.9M Jul  7 14:40 linux-headers-3.15.1-custom_3.15.1_amd64.deb
-rw-r--r--  1 root root 6.1M Jul  7 14:40 linux-image-3.15.1-custom_3.15.1_amd64.deb
-rw-r--r--  1 root root 979K Jul  7 14:40 linux-libc-dev_3.15.1_amd64.deb

install as follows and then reboot:
dpkg -i linux-headers-3.15.1-custom_3.15.1_amd64.deb linux-image-3.15.1-custom_3.15.1_amd64.deb linux-libc-dev_3.15.1_amd64.deb

So only the commands themselves - as easy as one two three...:

apt-get install wget fakeroot kernel-package gcc libncurses5-dev bc ca-certificates pkg-config make flex bison build-essential autoconf automake


tar xfJ linux-3.15.1.tar.xz

cd linux-3.15.1

yes "" | make oldconfig

make clean && \
make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-NewKernel KDEB_PKGVERSION=3.15.1

cd ..

dpkg -i linux-headers-3.15.1-custom_3.15.1_amd64.deb linux-image-3.15.1-custom_3.15.1_amd64.deb linux-libc-dev_3.15.1_amd64.deb
    as easy as that...