Honeypot SSH with ipset and systemd journal

posted on 2013-09-08

After installing systemd and setting up the systemd-journal-gatewayd, I decided to set up an SSH root login honeypot: if somebody tries to log in as root, block their ip for 15 minutes.

Bring out the honey

First we open up root login attempts. For the rest of the users, we only allow public key authentication, so we have to add an User section to our /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
Match User root
    PasswordAuthentication yes

These lines make sure that root is not permitted to log in, passwords are not allowed by default, empty passwords are not allowed, and specially for root a password is allowed. Now you can try ssh root@149.210.136.127 and be asked for a password without being able to log in.

Any failed attempt will be written down in the journal of systemd. To make sure we block these hosts, we first set up ipset (sudo pacman -S ipset or sudo apt-get install ipset). If you are wondering why not just use a rule per host, that's because of performance reasons.

For our IPv4 and IPv6 entries we need a separate ipset based on the remote hash. One we call blockfour and the other blacksix. We give both a default timeout of 15 minutes (900 seconds):

sudo ipset create blacksix  hash:ip family inet6 timeout 900
sudo ipset create blackfour hash:ip family inet timeout 900

Then we make sure we block any IP in that set in the iptables.rules:

-A INPUT -m set --match-set blackfour src -j DROP

and the ip6tables.rules:

-A INPUT -m set --match-set blacksix src -j DROP

I decided to add them early, so my server will completely disappear when an authentication failure happens, even before the ICMP handling. And example of this is /etc/iptables/iptables.rules:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT 
-A INPUT -m set --set blackfour src -j DROP
-A INPUT -p icmp -j ACCEPT
-A INPUT -m tcp -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -m tcp -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT

-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 

COMMIT

If everything works, do not forget to make the sets persistent between reboots:

sudo ipset save -f /etc/ipset.conf
sudo systemctl enable ipset

Connecting to the systemd journal with a user script

Now that we have the firewall in place and the ipsets defined, we can manually add people to the block list with ipset -exist add blackfour someipaddress without having to worry about the ip possibly already being in the set (-exist). To view the list, you can use ipset list.

To connect up the authentication failure message from the systemd journal to these commands, I wrote a small Python script which you can download from the github repository. You should get the newest version there.

The script reads JSON (Accept: application/json in the request) from http://localhost:19531/entries?follow&_SYSTEMD_UNIT=sshd.service to follow all the output of sshd.service. When a message containing authentication failure comes by, the rhost field is extracted and tested for an IPv4 or IPv6 address. If it's IPv4 it's added to the blackfour set, if it's IPv6 it's added to blacksix.

An example JSON log record you will receive from systemd-journal-gatewayd is:

{ "__CURSOR" : "s=2e3057190fe443bc832d54d148684051;i=82e;b=10f8bb1a8af84c87a46db3ef394cbd36;m=c8f866a0f;t=4e5ca09e22ca1;x=77d0ab186fb3f57c"
, "__REALTIME_TIMESTAMP" : "1378555818814625"
, "__MONOTONIC_TIMESTAMP" : "53947558415"
, "_BOOT_ID" : "10f8bb1a8af84c87a46db3ef394cbd36"
, "_UID" : "0"
, "_GID" : "0"
, "_MACHINE_ID" : "7e81610459944ea8d3eb00001631a513"
, "PRIORITY" : "5"
, "_TRANSPORT" : "syslog"
, "SYSLOG_FACILITY" : "10"
, "SYSLOG_IDENTIFIER" : "sshd"
, "_COMM" : "sshd"
, "_EXE" : "/usr/bin/sshd"
, "_SYSTEMD_CGROUP" : "/system/sshd.service"
, "_SYSTEMD_UNIT" : "sshd.service"
, "_CMDLINE" : "sshd: unknown [priv"
, "_HOSTNAME" : "banana"
, "SYSLOG_PID" : "965"
, "_PID" : "965"
, "MESSAGE" : "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=114.214.163.110"
, "_SOURCE_REALTIME_TIMESTAMP" : "1378555818814023"
}