OSSEC and Fail2Ban Setup Tutorial

Keeping your Linux server secure is one of the most important steps in protecting your websites and applications. Attackers often try to break in using methods like brute-force logins, malware, or suspicious activity. That’s why adding extra layers of security tools can make a big difference. In this guide, you will learn a full OSSEC and Fail2Ban Setup Tutorial to keep your Linux server secure.

The OSSEC and Fail2Ban work together to monitor your server, block malicious attempts, and keep unwanted visitors out.

• OSSEC is a Host-based Intrusion Detection System (HIDS). It constantly monitors your server’s logs, files, and processes, and alerts you when something suspicious happens.

• Fail2Ban helps prevent brute-force attacks by automatically banning IP addresses that try to log in too many times with the wrong password.

With this step-by-step guide, you will learn to:

• Update OS and install prerequisites.
• Install OSSEC manager (server): /var/ossec
• Install OSSEC agents on hosts; register agents with the manager.
• Install Fail2Ban on each host.
• Create /etc/fail2ban/jail.local with sensible defaults and ignoreip.
• Test with simulated log events and verify iptables/nft rules.

We will show you the guide steps on a server from PerLod Web Hosting, which provides reliable hosting and easy server management.

Secure Linux Server: OSSEC and Fail2Ban Setup Tutorial

When it comes to protecting a Linux server, two tools that work really well together are OSSEC and Fail2Ban. They both focus on security but handle it in different ways.

OSSEC Roles: It acts like a security guard for your server. It checks logs, files, and activities to spot anything suspicious.

• Manager (server): It collects events from other machines, applies rules, sends alerts, and can even trigger automatic responses. It usually runs on a dedicated VM. By default, it listens on port UDP 1514, so make sure to allow it through your firewall.
• Agents: They are installed on each server or computer you want to monitor (Linux or Windows). These agents forward logs back to the Manager for analysis.

Fail2ban: Install Fail2Ban on each host that accepts logins, such as SSH, FTP, and web apps. It watches your server’s login attempts, and if someone keeps failing, it blocks their IP address quickly using the firewall.

You can use OSSEC for detection and auditing, and use Fail2Ban for immediate IP blocking.

Note: There’s also Wazuh, which is a modern fork of OSSEC. Wazuh comes with more features, APIs, and integration with the Elastic Stack. If you need those, Wazuh might be the better choice. But if you prefer to stick with classic OSSEC, you can simply follow the steps below.

1. Install Requirements for OSSEC Setup

The first step is to run the system update and install the required packages. Because OSSEC is compiled from source, it needs build-essential and the libraries listed.

On Ubuntu Linux, run the commands below:

sudo apt update && sudo apt upgrade -y
sudo apt install build-essential libssl-dev libpcre2-dev zlib1g-dev libevent-dev wget curl -y

On CentOS and RHEL, run:

sudo dnf update -y
sudo dnf groupinstall "Development Tools" -y
sudo dnf install epel-release -y
sudo dnf install pcre-devel zlib-devel openssl-devel libevent-devel wget curl -y

Once you are done with the requirements, proceed to the next steps to build OSSEC and install Fail2Ban.

2. Install OSSEC Manager (Server)

We start by installing the OSSEC Manager, which acts as the main server that collects and analyzes security events. OSSEC has an official installer script that guides you through the choice of server, agent, local, and hybrid. In this guide, we will focus on installing the manager.

First, navigate to the /tmp directory and download the latest version of OSSEC with the following command:

cd /tmp
sudo wget https://github.com/ossec/ossec-hids/archive/refs/tags/3.8.0.tar.gz -O ossec-hids.tar.gz

Extract your downloaded file, navigate to it, and run the installer:

sudo tar -xzf ossec-hids.tar.gz
cd ossec-hids-*
sudo ./install.sh

During ./install.sh, you will be prompted:

• Installation type: Choose server.
• Language: Choose (en).
• Install location: Default /var/ossec.
• Enable email notifications: Configure as needed.
• Enable active responses: You can allow iptables to block if desired.
• Other settings that you can configure depending on your needs.

Note: OSSEC Manager listens on UDP port 1514 for agents. Allow it through your firewall.

After the installation is completed, start and check the OSSEC status with the commands below:

sudo /var/ossec/bin/ossec-control start
sudo /var/ossec/bin/ossec-control status

Also, you can verify the OSSEC manager process and the listening port:

ps aux | grep ossec
ss -ulpn | grep 1514

OSSEC common post-install hardening and common configs

• Enable active responses if you want OSSEC to trigger firewall rules (OSSEC can call iptables or custom scripts). Active responses live in /var/ossec/active-response/. Example script names: firewall-drop.sh. Configure in /var/ossec/etc/ossec.conf.

• Rules & decoders: Custom rules go into /var/ossec/etc/rules/local_rules.xml. Example: create a rule that raises severity for repeated authfail entries.

• Alerts & email: Configure settings inside /var/ossec/etc/ossec.conf.

• Log forwarding: Consider central syslog aggregation (rsyslog) if you want additional logs to be parsed.

3. Install OSSEC Agent on Monitored Hosts

At this point, you must set up OSSEC Agents on the servers we want to monitor. The manager and agents will talk to each other so you can keep track of suspicious activity across all your systems.

On each host you want monitored, download and install the OSSEC agent with the following commands:

cd /tmp
sudo wget https://github.com/ossec/ossec-hids/archive/refs/tags/3.8.0.tar.gz -O ossec-hids-agent.tar.gz
sudo tar -xzf ossec-hids-agent.tar.gz
cd ossec-hids-*
sudo ./install.sh

During the installation:

• Choose the agent.
• When asked for the manager IP, provide your OSSEC manager (UDP 1514).

After the installation is completed, you must register the agent with the manager. Let’s see how to do it.

4. Register OSSEC Agent with Manager

Now, you must register the OSSEC agents with the manager.

On the manager, create a key for the agent, for example, the agent name or IP:

sudo /var/ossec/bin/manage_agents

Inside interactive tool:

• Press ‘A’ to add agent, supply name, and IP.
• Press ‘E’ to extract the key (shows ASCII key).
• Copy the key output to the agent.

On agent, run:

sudo /var/ossec/bin/manage_agents

Choose ‘I’ to import an agent key, and paste the key that the manager printed.

Then, restart the agent:

sudo /var/ossec/bin/ossec-control restart

Next, you can list the connected agents on the manager:

sudo /var/ossec/bin/agent_control -l

5. Installing Fail2Ban with Basic Configuration

Once OSSEC is running, you can add Fail2Ban. Unlike OSSEC, which focuses on detecting and reporting issues, Fail2Ban immediately blocks attackers by banning their IP addresses after repeated failed login attempts. With just a little configuration, it will automatically update your firewall to stop brute-force attacks on services like SSH.

To install Fail2Ban on Ubuntu, you can run:

sudo apt update
sudo apt install fail2ban -y

To install Fail2Ban on RHEL/CentOS, you can run:

sudo dnf install epel-release -y
sudo dnf install fail2ban -y

Start and enable Fail2Ban:

sudo systemctl enable --now fail2ban

For a basic Fail2Ban configuration, here is an SSH example. Remember not to edit jail.conf, create a local file.

sudo nano /etc/fail2ban/jail.local

Add the following SSH config example:

[DEFAULT]
ignoreip = 127.0.0.1/8 10.0.0.0/8      # add your management network/whitelisted IPs
bantime  = 3600                         # ban for 1 hour
findtime = 600                          # window to count failures (10min)
maxretry = 3                            # ban after 3 fails
backend = auto                          # auto choose e.g. systemd/journal/file

[sshd]
enabled = true
port    = ssh
filter  = sshd
logpath = /var/log/auth.log            # Ubuntu; CentOS may be /var/log/secure
maxretry = 3

Once you are done, save and close the file.

Restart the service and check bans and jails:

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd    # shows bans
sudo fail2ban-client status         # show all jails

To protect SSH and Nginx authentication, you can also add the following settings to the local jail file:

sudo nano /etc/fail2ban/jail.local

[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log

[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 3
bantime = 7200

After saving, restart the service:

sudo systemctl restart fail2ban

6. Testing OSSEC and Fail2Ban Setup

After you are done with the setup, you can test that your OSSEC and Fail2Ban are working correctly.

• Run OSSEC Tests:

Check the agent connection from the manager:

sudo /var/ossec/bin/agent_control -l

Force a test alert on the agent:

logger -p auth.info "TEST OSSEC ALERT from $(hostname)"

Then, check the manager alerts:

sudo tail -n 200 /var/ossec/logs/alerts/alerts.log

You can check the manager web UI if it is installed.

• Run Fail2Ban Tests:

You can simulate failed SSH logins (from a different host) or create a fake log line.

On the target host, simulate an attack:

sudo su -c 'for i in {1..4}; do logger -t sshd "Failed password for invalid user test from 198.51.100.5 port 22 ssh2"; sleep 1; done'

Then, check:

sudo fail2ban-client status sshd
sudo iptables -L -n | grep 198.51.100.5 || sudo nft list ruleset | grep 198.51.100.5

To unban the IP, you can run:

sudo fail2ban-client set sshd unbanip 198.51.100.5

By setting up these two strong security layers, your server will have:

• OSSEC for monitoring, auditing, and alerting.
• Fail2Ban for instant blocking of malicious login attempts.

Common Issues and Fixes for OSSEC and Fail2Ban

Here we provide common troubleshooting and tips related to the OSSEC and Fail2Ban setup tutorial.

1. Fail2Ban bans too aggressively: If Fail2Ban is blocking users too quickly, you can adjust its settings:

• Increase maxretry: How many times someone can fail before getting banned.
• Increase findtime: The time window used to count failures.
• Use ignoreip: Add your trusted IP addresses or networks so they never get banned.

2. OSSEC not showing file changes (missing FS events): If OSSEC isn’t reporting file changes, check these:

• Make sure the agent is set up correctly.
• Verify file permissions so OSSEC can read the files it’s supposed to monitor.
• Confirm the agent is connected to the manager.

3. OSSEC manager not receiving data from agents: If the manager isn’t getting logs from agents:

• Ensure UDP port 1514 is open and allowed through the firewall.
• Check if SELinux is blocking connections.
• Use the command ss -ulpn | grep 1514 to see if the manager is really listening on that port.

4. Fail2Ban with systemd systems: On servers that use systemd, set the backend to systemd (or leave it as auto). This makes sure Fail2Ban reads logs properly from the system journal (journald).

5. Fail2Ban with nftables instead of iptables: Some newer Linux systems use nftables instead of iptables for firewalls. Make sure Fail2Ban is configured to use nftables actions. If not, you may need to update its settings so it can block IPs correctly.

FAQs

Conclusion

By following these steps for the OSSEC and Fail2Ban Setup Tutorial, you will have a strong host-based defense strategy. OSSEC provides visibility, and Fail2Ban provides immediate action. This ensures you are not only detecting intrusions but also mitigating them in real time.

Whether you deploy OSSEC and Fail2Ban on your own hardware or with a hosting provider, make sure the server is stable and secure. Our dedicated hosting services and VPS hosting plans are designed for projects like this, giving you performance, flexibility, and reliable uptime.

Subscribe to X and Facebook channels to get the latest articles and setup guides.

For further reading:

Cloud vs dedicated servers: Cost predictability

Monitoring Linux with Prometheus and Node Exporter