Protecting SSH port with Fail2ban on Ubuntu - Setup and configuration

Logo

Introduction

On Ubuntu servers, ufw (Uncomplicated Firewall) is a great tool to setup firewall rules on ports without having to use iptables commands which are very difficult.

root@vpsfrsqlpac2$ ufw status verbose

To                         Action      From
--                         ------      ----
22                         ALLOW IN    Anywhere                   # SSH
8600:8800/tcp              ALLOW IN    10.128.30.32               # MariaDB CS 
20501:20509/tcp            ALLOW IN    10.128.30.32               # PostgreSQL

But that’s not enough. For SSH services, one can define another port than the default one (port 22), but unfortunately some products, MariaDB ColumnStore for example, use ssh in their administration routines without allowing the ability to override the default port 22.

Thus, the port 22 must be opened and in the file /var/log/auth.log we notice many connections attempts coming from unknown machines, sometimes with very short time intervals between each attempt from an IP.

/var/log/auth.log
Feb  2 06:25:42 vpsfrsqlpac2 sshd[6764]: Disconnected from invalid user tester 186.4.184.218 port 42416 [preauth]
Feb  2 06:29:32 vpsfrsqlpac2 sshd[6830]: Invalid user tester from 123.108.35.186 port 41670
Feb  2 06:29:34 vpsfrsqlpac2 sshd[6830]: Failed password for invalid user tester from 123.108.35.186 port 41670 ssh2
Feb  2 06:29:34 vpsfrsqlpac2 sshd[6830]: Disconnected from invalid user tester 123.108.35.186 port 41670 [preauth]
Feb  2 06:35:50 vpsfrsqlpac2 sshd[6933]: Invalid user tester from 123.108.35.186 port 45178
Feb  2 06:35:52 vpsfrsqlpac2 sshd[6933]: Failed password for invalid user tester from 123.108.35.186 port 45178 ssh2

Fail2Ban is the complementary tool to ufw in order to protect an Ubuntu system from these attacks.

  • Its installation is easy.
  • Fail2Ban manages iptables rules in order to ban IP addresses during a period following configurable conditions.

Fail2Ban is a python package and can be installed in a Python virtual environment, that’s the case in this article.

The installation must be performed with root privileges as this tool manages iptables rules.

Installation

Preparing the Python virtual environment

Python 3.8 with virtualenv is installed in the directory /opt/python/python-3.8. The file /opt/python/.python-3.8 sources the needed environment variables : 

/opt/python/.python-3.8
export PYHOME=/opt/python/python-3.8
export PATH=$PYHOME/bin:$PATH
export LD_LIBRARY_PATH=$PYHOME/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/opt/python/packages

The Python virtual environment for fail2ban is installed in the directory /opt/monitoring/fail2ban : 

root@vpsfrsqlpac2$ source /opt/python/.python-3.8
          
root@vpsfrsqlpac2$ virtualenv --system-site-packages /opt/monitoring/fail2ban

The environment is activated : 

root@vpsfrsqlpac2$ source /opt/monitoring/fail2ban/bin/activate
(fail2ban) root@vpsfrsqlpac2$ which python3
/opt/monitoring/fail2ban/bin/python3

Fail2ban package

Download fail2ban from GitHub and run the installation : 

(fail2ban) root@vpsfrsqlpac2$ cd /opt/monitoring/setup
(fail2ban) root@vpsfrsqlpac2$ wget https://github.com/fail2ban/fail2ban/archive/0.11.1.tar.gz
(fail2ban) root@vpsfrsqlpac2$ tar -xvzf 0.11.1.tar.gz
(fail2ban) root@vpsfrsqlpac2$ cd fail2ban-0.11.1
(fail2ban) root@vpsfrsqlpac2$ python3 setup.py install
…
changing mode of /opt/monitoring/fail2ban/bin/fail2ban-testcases to 755

Please do not forget to update your configuration files.
They are in "/etc/fail2ban/".

You can also install systemd service-unit file from "build/fail2ban.service"
resp. corresponding init script from "files/*-initd".

The package fail2ban is installed successfully in the virtual environment /opt/monitoring/fail2ban.

Configuring Fail2ban

All configuration files are installed in the directory /etc/fail2ban.

fail2ban.conf

The main configuration file fail2ban.conf does not contain many parameters to configure. In this one, we can customize verbosity, directories and filenames for the log file, the pid file…

/etc/fail2ban/fail2ban.conf
[DEFAULT]
loglevel = INFO
logtarget = /var/log/fail2ban.log
socket = /var/run/fail2ban/fail2ban.sock
pidfile = /var/run/fail2ban/fail2ban.pid
dbfile = /var/lib/fail2ban/fail2ban.sqlite3
dbpurgeage = 15d

Fail2ban uses a sqlite3 database. When installing, update the retention parameter dbpurgeage to at least 8 days, it will be already done when configuring later the recidivists attacks.

jail.local

Fail2Ban rules are configured in the file /etc/fail2ban/jail.local. Copy the file /etc/fail2ban/jail.conf to /etc/fail2ban/jail.local. The local file prevents loosing any configuration when upgrading fail2ban : 

(fail2ban) root@vpsfrsqlpac2$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
/etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8 10.128.31.48

# "bantime" is the number of seconds that a host is banned.
bantime  = 30m

# A host is banned if it has generated "maxretry" during the last "findtime" seconds.
findtime  = 10m

# "maxretry" is the number of failures before a host get banned.
maxretry = 3
  • define the IPs to ignore in the DEFAULT section, especially its machines IPs.
  • the default banishment time may be adjusted : here it is set to 30 minutes instead of the default 10 minutes.

Now banishment rules for ssh are enabled : 

/etc/fail2ban/jail.local
[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Many sections are available and predefined in the file jail.local : sshd, apache, mysql…

All the parameters defined in the section DEFAULT (bantime, ignoreip…) can be customized in each section. Take care about the parameter ignoreip, if specific IP address is defined in the child, it will override the default value, no merge is performed.

In the "fail2ban language", a section in the configuration file is a jail. The jail sshd has been enabled in the example above.

Using fail2ban

iptables backup

On Ubuntu the IP tables iptables are not stored in a file, but in the kernel. Fail2ban will manage iptables, so a backup, in case of, should be performed before starting fail2ban :

(fail2ban) root@vpsfrsqlpac2$ iptables-save > /etc/iptables_rules.txt

If unexpected behaviours occur after Fail2ban activation, the restore to a stable state will then be easy :

(fail2ban) root@vpsfrsqlpac2$ iptables-restore < /etc/iptables_rules.txt

Managing fail2ban

Everything is ready, to start fail2ban : 

(fail2ban) root@vpsfrsqlpac2$ fail2ban-server -xf start
          2020-02-07 10:08:11,078 fail2ban.server         [26518]: INFO    --------------------------------------------------
2020-02-07 10:08:11,078 fail2ban.server         [26518]: INFO    Starting Fail2ban v0.11.1
2020-02-07 10:08:11,079 fail2ban.server         [26518]: INFO    Daemon started
2020-02-07 10:08:11,080 fail2ban.observer       [26518]: INFO    Observer start...
2020-02-07 10:08:11,137 fail2ban.database       [26518]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2020-02-07 10:08:11,140 fail2ban.database       [26518]: WARNING New database created. Version '4'

To stop, restart the server or to reload the configuration globally or just for a jail : 

(fail2ban) root@vpsfrsqlpac2$ fail2ban-client stop | restart | reload [jail]

fail2ban-client and jails

Use fail2ban-client to manage jails : status, unban IPs, start, stop…

Banned IPs appear in the jail sshd shortly after activation :

(fail2ban) root@vpsfrsqlpac2$ fail2ban-client status
Status
|- Number of jail:      2
`- Jail list:   recidive, sshd
(fail2ban) root@vpsfrsqlpac2$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  |- Total failed:     2022
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 4
   |- Total banned:     466
   `- Banned IP list:   209.141.57.211 218.92.0.145…

Jails configuration parameters can be listed and modified dynamically :

(fail2ban) root@vpsfrsqlpac2$ fail2ban-client get sshd bantime
600
(fail2ban) root@vpsfrsqlpac2$ fail2ban-client set sshd bantime 60m
3600

fail2ban-client set only updates the values for a running Fail2Ban server, do not forget to update the configuration file accordingly if they have to be persistent.

First command to know with Fail2ban (when you ban yourself 🙂) : how to unban an IP ? 

(fail2ban) root@vpsfrsqlpac2$ fail2ban-client set sshd unbanip 90.62.100.101

Many options exist : for help, run fail2ban-client --help.

fail2ban and IP Tables

How Fail2ban works with iptables ?

Fail2ban creates for each jail a user chain with the name f2b-[jail], f2b-sshd in the above example, then a rule is added in this chain for port 22 / INPUT :

(fail2ban) root@vpsfrsqlpac2$ iptables -S
-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd

Banned IPs are appended in the chain : 

(fail2ban) root@vpsfrsqlpac2$ iptables -L f2b-sshd
Chain f2b-sshd (1 references)
target     prot opt source               destination
REJECT     all  --  222.186.180.8        anywhere             reject-with icmp-port-unreachable
REJECT     all  --  222.186.173.142      anywhere             reject-with icmp-port-unreachable
REJECT     all  --  222.186.30.187       anywhere             reject-with icmp-port-unreachable
REJECT     all  --  222.186.173.215      anywhere             reject-with icmp-port-unreachable
REJECT     all  --  222.186.175.140      anywhere             reject-with icmp-port-unreachable

The chain is removed when stopping the Fail2ban server or stopping the jail (fail2ban-client stop sshd). As expected, the chain is not removed when Fail2ban server is killed (kill -9).

No loss when restarting the Fail2ban server, iptables chains are rebuilt based on the informations stored in the sqlite3 database.

(fail2ban) root@vpsfrsqlpac2$ sqlite3
sqlite> .open /var/lib/fail2ban/fail2ban.sqlite3
sqlite> .tables
bans        bips        fail2banDb  jails       logs
sqlite> select ip, jail, datetime(timeofban,'unixepochs'), bantime from bips;
222.186.173.215|sshd|2020-02-07 16:17:03|3600
222.186.30.187|sshd|2020-02-07 16:19:25|3600
222.186.173.142|sshd|2020-02-07 16:24:28|3600
…

Service fail2ban

fail2ban is installed in a Python virtual environment but it does not prevent defining a service for the automatic startup.

A file .fail2ban is created for sourcing and activating the Python virtual environment.

/opt/monitoring/fail2ban/.fail2ban
export F2BDIR=/opt/monitoring/fail2ban
source /opt/python/.python-3.8
source $F2BDIR/bin/activate

A template file for the service is available in the directory from which the installation has been performed (build/fail2ban.service).

The service is customized by integrating the call to the file .fail2ban : 

/lib/systemd/system/fail2ban.service
[Unit]
Description=Fail2Ban Service
Documentation=man:fail2ban(1)
After=network.target iptables.service firewalld.service ip6tables.service ipset.service nftables.service
PartOf=iptables.service firewalld.service ip6tables.service ipset.service nftables.service

[Service]
Type=simple
ExecStartPre=/bin/mkdir -p /run/fail2ban
ExecStart=/bin/bash -c "source /opt/monitoring/fail2ban/.fail2ban; /opt/monitoring/fail2ban/bin/fail2ban-server -xf start"
ExecStop=/bin/bash -c "source /opt/monitoring/fail2ban/.fail2ban; /opt/monitoring/fail2ban/bin/fail2ban-client stop"
ExecReload=/bin/bash -c "source /opt/monitoring/fail2ban/.fail2ban; /opt/monitoring/fail2ban/bin/fail2ban-client reload"
PIDFile=/run/fail2ban/fail2ban.pid
Restart=on-failure
RestartPreventExitStatus=0 255

[Install]
WantedBy=multi-user.target

The service is then enabled : 

root@vpsfrsqlpac2$ cd /etc/systemd/system
root@vpsfrsqlpac2$ ln -fs /lib/systemd/system/fail2ban.service fail2ban.service
root@vpsfrsqlpac2$ systemctl enable fail2ban
Created symlink /etc/systemd/system/multi-user.target.wants/fail2ban.service → /lib/systemd/system/fail2ban.service.
root@vpsfrsqlpac2$ systemctl start fail2ban
root@vpsfrsqlpac2$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
   Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-02-09 17:23:29 CET; 23h ago
     Docs: man:fail2ban(1)
  Process: 4240 ExecStop=/bin/bash -c source /opt/monitoring/fail2ban/.fail2ban; /opt/monitoring/fail2ban/bin/fail2ban-clien
  Process: 4250 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
 Main PID: 4259 (bash)
    Tasks: 8 (limit: 4587)
   CGroup: /system.slice/fail2ban.service
           ├─4259 /bin/bash -c source /opt/monitoring/fail2ban/.fail2ban; /opt/monitoring/fail2ban/bin/fail2ban-server -xf s
           └─4264 /opt/monitoring/fail2ban/bin/python3 /opt/monitoring/fail2ban/bin/fail2ban-server -xf start

Jail for recidivists

A very interesting jail to setup : the recidive jail

/etc/fail2ban/jail.local
[recidive]
enabled = true
logpath  = /var/log/fail2ban.log
banaction = %(banaction_allports)s
bantime  = 1w
findtime = 1d

When activating this jail, Fail2ban scans its own log file searching for recurrent IPs banned by the other defined jail rules.

The IPs are then banned for a longer period (here 1 week) for ALL input ports. That’s why the retention parameter dbpurgeage has been set to 15 days, otherwise the jail recidive would not properly work.

root@vpsfrsqlpac2$ iptables -S
…
-N f2b-recidive
-A INPUT -p tcp -j f2b-recidive
…
-A f2b-recidive -s 222.186.180.17/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-recidive -s 222.186.175.167/32 -j REJECT --reject-with icmp-port-unreachable
…

Conclusion

Some other jails seem to be of interest, especially jails detecting floods and injections over Apache / PHP, but it should be studied deeper.

The first opened jails (sshd, recidive) are largely enough for the moment when trying to protect better the sshd port of a Linux Ubuntu server located on the internet network.

About performances : fail2ban configured with sshd and recidive jails needs 450 - 650 Mb. Its CPU usage is very low (less than 0,2% on average on a 1 Core machine 2 GHz). Number of rules created in iptables should not degrade performance.