Monitoring basics for a single server

Monitor from outside first

A check that runs on the server cannot tell you the server is down. Start with an external HTTP check from any uptime service, pointed at a real page rather than the home page cache. A cheap health endpoint that touches the database is ideal:

# nginx: expose the app's health route, skip logging it
location = /healthz {
    access_log off;
    proxy_pass http://127.0.0.1:8080/healthz;
}

Have it return 200 only when the app can reach its database and any queue. Check every minute, alert after two consecutive failures to avoid paging on a single dropped packet, and include a certificate expiry check while you are there.

The portal's CPU and bandwidth graphs are the other outside view. They keep working when the OS is wedged, which makes them the first place to look when the external check fires and SSH hangs.

Disk space: the classic outage

A full root filesystem breaks things in strange ways: databases refuse writes, sessions vanish, package upgrades half-apply. Look:

$ df -h -x tmpfs -x devtmpfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        79G   71G  4.6G  94% /

$ sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -12

-x keeps du on one filesystem, so it does not wander into /proc. Usual suspects:

• /var/log: an application writing debug logs with no rotation. Add a logrotate rule.
• /var/lib/docker: old images and build cache. docker system df, then docker image prune -a with care.
• /var/log/journal: cap it with SystemMaxUse=.
• Local backup dumps that never get pruned.
• Inodes, not bytes: df -i. Millions of tiny session or cache files can exhaust inodes with gigabytes free.

If df says full but du disagrees, a process is holding a deleted file open:

sudo lsof +L1 | sort -k7 -n | tail

Restart that process and the space comes back.

A minimal disk alert with no extra software

Before installing anything, a twenty-line script on a systemd timer covers the most common failure:

#!/bin/sh
# /usr/local/bin/disk-alert
LIMIT=85
df -P -x tmpfs -x devtmpfs | awk 'NR>1 {gsub("%","",$5); print $5, $6}' |
while read use mount; do
  if [ "$use" -ge "$LIMIT" ]; then
    /usr/local/bin/alert "$(hostname): $mount at ${use}%"
  fi
done

#!/bin/sh
# /usr/local/bin/alert  -- send to a chat webhook
curl -fsS -m 10 -H 'Content-Type: application/json' \
  -d "{\"text\": \"$*\"}" "$ALERT_WEBHOOK_URL" >/dev/null

Run it every ten minutes. It is crude, and it has caught more real incidents than many dashboards.

Option A: Netdata for one or two servers

Netdata collects thousands of metrics per second with sane default alarms, including disk fill rate, which predicts the full disk hours ahead. Install from your distribution's repository or the vendor's script after reading it, then keep the dashboard off the public internet:

# /etc/netdata/netdata.conf
[web]
    bind to = 127.0.0.1

View it through an SSH tunnel: ssh -L 19999:127.0.0.1:19999 you@server, then open http://localhost:19999. Expect it to use a few hundred MB of RAM; on a 2 GB Spark lower the history or use option B.

Option B: node_exporter and Prometheus

node_exporter is a single small binary that exposes host metrics. Prometheus scrapes it and evaluates alert rules; Alertmanager routes the alerts. Put Prometheus on a separate small server so it survives the box it is watching.

# on each monitored server (Debian/Ubuntu)
sudo apt install prometheus-node-exporter
# listens on :9100; allow only the Prometheus server's IP in the portal firewall

# prometheus rules: alert before the disk fills
groups:
- name: host
  rules:
  - alert: DiskWillFillIn6h
    expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[2h], 6*3600) < 0
    for: 15m
  - alert: DiskAbove85
    expr: 1 - node_filesystem_avail_bytes / node_filesystem_size_bytes > 0.85
    for: 10m
  - alert: MemoryPressure
    expr: rate(node_pressure_memory_waiting_seconds_total[5m]) > 0.1
    for: 10m

predict_linear is the useful one. A disk at 60 percent filling at 2 GB an hour is more urgent than one parked at 88 percent.

Services and the out-of-memory killer

Two quick checks belong in any routine:

systemctl --failed
journalctl -k -g 'Out of memory|oom-kill' --since -1d

If the OOM killer appears, the fix is less RAM use, a small swap file as a buffer, or a bigger plan. Performance triage covers how to tell which.

Keep alerts honest

• Every alert should have an action. If the response is "ignore it", delete the alert.
• Send to a channel people read, plus a phone for the external down check.
• Test alerts quarterly: fill a scratch file with fallocate -l 10G /root/fill, watch the alert arrive, delete the file.
• Monitor the backup job's age too; a stopped backup is an outage you only notice later.