In this tutorial, we want to discover how to set up an AIOps stack for server monitoring. An AIOps stack is a combination of tools that use AI and automation to monitor, analyze, and manage servers. It is a great way to detect issues, predict failures, and automate responses. In this guide, we will use the following tools for the AIOps stack:
Metrics: Prometheus, Node Exporter, and Blackbox Exporter.
Logs: Loki and Promtail.
Visualization and alerting UI: Grafana.
Alert routing: Alertmanager.
AIOps: A small Python service that fetches Prometheus metrics, detects anomalies, and triggers Alertmanager via webhook.
In this guide, we use Ubuntu 24.04 server from PerLod Hosting, where you can find support for server monitoring with AIOps.
Requirements for AIOps Stack for Server Monitoring
To complete the guide steps, you need a fresh Ubuntu 24.04 VM with 4 vCPU, 8 GB RAM, and 100 GB disk. Also, you must open the required firewall ports, including:
Set the correct ownership for the AIOps stack directory with:
Bash
sudochown-R$USER:$USER/opt/aiops
Switch to the AIOps stack directory:
Bash
cd/opt/aiops
Prometheus Configuration for AIOps Stack
Prometheus is the main monitoring engine. The first step is to configure Prometheus server monitoring by creating the Prometheus YAML file with the following command:
Loki and Promtail Log Configuration for AIOps Stack
Loki is a log aggregation system, and Promtail is its log collector. Promtail gathers logs from your server and sends them to Loki, which makes it easy to search logs in Grafana alongside metrics.
Create the Loki YAML file with the following command:
Blackbox Exporter checks external endpoints like APIs or websites using HTTP probes, which helps you monitor the availability and response of your web services.
At this point, you can create the Blackbox Exporter YAML file with the following command:
This custom Python microservice analyzes Prometheus metrics in real time using STL decomposition and z-score detection to identify anomalies. When it detects unusual behavior, it sends a synthetic alert to Alertmanager automatically.
Create the requirements file with the following command:
Check if the service is up and running with the command below:
Bash
dockercomposeps
Install Node Exporter on Each Server To Monitor
At this point, you must set up Node Exporter on each server target you want to monitor. It collects CPU, memory, disk, and network metrics and exposes them on port 9100, and then Prometheus scrapes these metrics for analysis and alerting.
You must be sure the Prometheus server can reach server_ip:9100.
Verifying AIOps Stack and Creating Dashboards
It is recommended to confirm that every component of the AIOps stack is running correctly. To verify your setup, open the Prometheus Web UI by navigating to the URL below:
http://your-server-ip:9090
From there, go to Status and then Targets. Make sure all targets show as UP. Also, check the Blackbox Exporter to confirm that your target pages are listed and also show as UP.
For Loki, navigate to the following URL:
http://your-server-ip:3100/ready
It must return ready, which means the service is active.
Then, navigate to the Grafana UI and log in with the default credentials:
http://your-server-ip:3000
In Grafana’s Explore section, select Prometheus and run the query up to confirm metrics are available, then switch to Loki and run the query {job="varlogs"} to verify logs are being collected correctly.
To quickly visualize your metrics, you can create a dashboard in Grafana.
Open Grafana, go to Dashboards and then Import, and enter the dashboard ID 1860 from Grafana.com to import Node Exporter Full and get a detailed view of your servers’ performance.
Alternatively, you can place your own JSON dashboard files in the /opt/aiops/grafana-provisioning/dashboards/ directory, and they’ll automatically appear under the AIOps folder when Grafana starts.
Test Alerts and Validate AIOps Anomalies
After setting up the AIOps stack, it’s important to test that alerts and anomaly detection are working correctly. You can simulate different scenarios to confirm that Prometheus, Alertmanager, and the AIOps anomaly detector respond as expected.
For example, stop the Node Exporter service on a monitored server to test a NodeDown alert after about two minutes.
You can also manually test the AIOps anomaly detection by running the command below:
Bash
curl-XPOSThttp://your-server-ip:8080/run
If the last metric point is an outlier, it sends an AIOpsDetectedAnomaly alert to Alertmanager via webhook.
You can automate this process with a cron job, sidecar container, or Grafana alert rule to call /run periodically and keep continuous anomaly checks active.
That's it, you are done with setting up the AIOps stack for server monitoring.
Conclusion
Setting up an AIOps stack for server monitoring provides proactive insights, automatic anomaly detection, and clear visibility across your entire infrastructure. By combining Prometheus, Grafana, Loki, Alertmanager, and a lightweight Python anomaly detector, you can monitor system health and performance in real time and receive smart alerts.
We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest server monitoring guides and tips.
AIOps uses machine learning and data analysis to automatically detect problems, find their causes, and connect related events. It helps reduce unnecessary alerts, fix issues faster, and predict potential system problems before they happen.
The detector regularly checks Prometheus metrics, uses STL and z-score analysis to find unusual patterns, and flags any outliers. When it detects an anomaly, it automatically sends an alert to Alertmanager, which can notify your team through email, Slack, or a webhook.
You can turn this stack into Ansible roles to automate deployments, Helm charts to run it on Kubernetes, and Terraform modules to manage infrastructure automatically with code.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.