Skip to content

guide

Performance triage for a slow server

Something is slow. Do not restart it yet. Spend five minutes finding out which resource is saturated, because the restart will destroy the evidence.

The first 60 seconds

Run these in order. Together they tell you which of CPU, memory, disk or network is the problem.

uptime                 # load averages
vmstat 1 5             # run queue, swap, iowait, steal
free -m                # memory, really available
cat /proc/pressure/{cpu,memory,io}   # PSI: who is waiting
df -h /                # is the disk simply full?
journalctl -p warning --since -30min --no-pager | tail -40

Read load average correctly

$ uptime
 14:02:11 up 12 days,  3:10,  1 user,  load average: 7.84, 6.10, 2.95
$ nproc
4

The three numbers are 1, 5 and 15-minute averages of tasks running or waiting. Compare them to nproc. Here, 7.84 on four vCPUs means about four tasks queued at any moment, and the rising trend says it started within the last ten minutes. On Linux, load includes tasks in uninterruptible disk sleep, so a high load with idle CPUs points at I/O, not compute.

vmstat: CPU, iowait and steal in one view

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  0      0 412044  81220 3011456    0    0     4   120 2210 4410 88 10  2  0  0
 5  1      0 409876  81220 3011802    0    0     0   340 2302 4598 90  9  1  0  0
ColumnMeaningWorry when
rRunnable tasksConsistently above vCPU count
bBlocked on I/OConsistently above 0
si/soSwap in/out per secondNon-zero every second
us+syUser and kernel CPUNear 100 with r high
waIdle waiting on diskAbove 10 sustained
stStolen by the hypervisorAbove a few percent sustained

The example is plain CPU saturation: high us, no iowait, no steal. Find the process with top -o %CPU or pidstat 1 5 from the sysstat package.

Memory: available, not free

$ free -m
               total        used        free      shared  buff/cache   available
Mem:            7940        6820         210          95         910         820
Swap:           2047        1460         587

Ignore free; Linux uses spare RAM as cache on purpose. available is what matters. Here, 820 MB available with 1.4 GB in swap on an 8 GB Turbo means the working set no longer fits. Sort by resident memory to see who is responsible:

ps -eo pid,rss,comm --sort=-rss | head

Check whether the kernel has already killed something:

journalctl -k -g 'oom-kill|Out of memory' --since today

PSI: the most honest metric

Pressure Stall Information, in every kernel our images ship, reports the share of time tasks were stalled waiting for a resource:

$ cat /proc/pressure/memory
some avg10=18.40 avg60=12.07 avg300=4.11 total=982341201
full avg10=6.02 avg60=3.88 avg300=1.20 total=301220988

some is the percentage of time at least one task waited; full is when all non-idle tasks waited, which is lost throughput. Sustained full above a few percent on memory or I/O means users feel it. CPU some going up with CPU usage is normal; memory and io full going up is not.

Disk: iowait and what causes it

$ iostat -xz 1 3
Device   r/s   w/s  rkB/s   wkB/s  r_await w_await aqu-sz %util
vda     12.0 840.0  96.0 52480.0     0.41    3.90   3.28  97.6

NVMe-backed storage handles a lot of random I/O, so a disk at high utilisation usually has a single loud writer. Find it:

sudo iotop -oPa       # accumulated I/O per process
sudo pidstat -d 1 5

Usual causes: a database checkpoint after a bulk import, a backup job running during peak, log files written synchronously, or the system swapping, which shows up as I/O but is really a memory problem. Fix the cause; for backups, the timers guide shows how to run them at idle I/O priority.

Steal time

On a virtual server, st in vmstat or top shows CPU time the hypervisor gave elsewhere while your vCPU wanted to run. Small blips are normal. If it stays high, record the values and timestamps with sar -u 1 60 and send them to support; that data makes the conversation short.

Logs: what changed?

Slowness usually has a start time. Find it, then look at what happened just before:

journalctl --since "14:00" --until "14:10" -p info --no-pager
journalctl -u nginx -u php8.3-fpm --since -1h | grep -iE 'error|timeout|max_children'
sudo tail -n 200 /var/log/nginx/error.log
grep -E 'upgrade|install' /var/log/dpkg.log | tail       # Debian/Ubuntu
sudo dnf history | head                                # AlmaLinux/Rocky

Deploys, package upgrades, a cron job, a crawler suddenly hitting search pages, a certificate renewal restarting a service: one of these explains most incidents.

Network last, not first

If CPU, memory and disk are calm but requests are slow, time a request from outside with curl -w and run mtr; the latency guide shows how to read both. Check connection counts too: ss -s and ss -tan state syn-recv | wc -l for a flood of half-open connections.

Then decide

  • One process misbehaving: fix or limit it.
  • Database queries: see database tuning.
  • PHP workers exhausted: see PHP-FPM tuning.
  • Honest, sustained saturation at normal traffic: it is time for the next plan up on the pricing table.

Questions

Is a load average of 4 bad?

It depends on the CPU count. On a four-vCPU Turbo, 4 means the CPUs are fully used with nothing waiting. On a two-vCPU Boost, it means work is queuing. Load also counts tasks stuck waiting on disk, so check iowait before concluding it is CPU.

What is CPU steal and what can I do about it?

Steal is time your virtual CPU was ready to run but the hypervisor ran something else. Brief small values are normal on any virtual server. If it stays high, note the times and values and contact support with them.

Should I add swap?

A small swap file, 1 to 2 GB, gives the kernel room to move idle pages out and avoids sudden OOM kills. It is not a substitute for RAM: if the system is constantly swapping, you need a bigger plan or a lighter workload.

When is it time to upgrade the plan?

When triage shows a resource is genuinely saturated by legitimate work at normal traffic, and tuning has run out. Ask support to move you to the next plan; the table on the pricing page shows what each step adds.