# OOM & Server Load Forensics Tooling for the production outages tracked in `evolution/audits/MARIADB_OOM_INVESTIGATION.md`. **Run all of these ON PROD, as root.** Everything is read-only — nothing is killed, restarted or reconfigured. --- ## RUN THIS NOW — deploy watcher v2.1 (2026-07-30) v2's **triggers** worked: they caught a 2,320 MB burst on 07-29 21:51 that v1 would have slept through. v2's **vhost tables did not** — every row of the ranking read exactly `200` and each vhost appeared 2–4×, so the one remaining question (*which URL*) still could not be answered. v2.1 fixes the tables and retunes the triggers off 8,319 real samples. Full analysis: Round 3 and Round 3.1 in `MARIADB_OOM_INVESTIGATION.md`. ```bash cd /root/forensics # 1. Stop v2, drop in v2.1, restart. Sample format is UNCHANGED (still 26 columns), # so today's TSV is appended to, not rolled. ./forensic-watch.sh -d /var/log/erp-forensics --stop # chmod +x forensic-watch.sh nohup ./forensic-watch.sh -i 10 -d /var/log/erp-forensics > /dev/null 2>&1 & ./forensic-watch.sh -d /var/log/erp-forensics --status # confirm RUNNING # 2. Confirm the dedupe actually collapses the duplicates on THIS box. The first # number should be 2-4x the second; if they're equal there were no dupes. for d in /etc/apache2/logs/domlogs /var/log/apache2/domlogs /usr/local/apache/domlogs; do [ -d "$d" ] && find "$d" -maxdepth 2 -type f -mmin -3 -size +0 \ ! -name '*.gz' ! -name '*bytes_log' ! -name '*.offset' ! -name '*.bkup' \ ! -name '*.localhost' ! -name 'proxy-subdomains-vhost.*' -printf '%i\n' done | wc -l # raw paths for d in /etc/apache2/logs/domlogs /var/log/apache2/domlogs /usr/local/apache/domlogs; do [ -d "$d" ] && find "$d" -maxdepth 2 -type f -mmin -3 -size +0 \ ! -name '*.gz' ! -name '*bytes_log' ! -name '*.offset' ! -name '*.bkup' \ ! -name '*.localhost' ! -name 'proxy-subdomains-vhost.*' -printf '%i\n' done | sort -u | wc -l # distinct physical files # 3. The @reboot line is unchanged (same path/flags) — but confirm it's still there. crontab -l | grep forensic-watch ``` v2.1 trips on `php_procs>=25`, `php_rss>=1500MB` or `mysql_conn>=15`. **`tcp_estab` is no longer a trigger** — all 28 apache-spawn storms observed on 07-29/07-30 carried near-zero PHP (`apache=153` at `php_rss=350MB`), because bots hitting nonexistent `.php` are 404'd by Apache without ever starting an interpreter. Apache child count is not the mechanism and it produced 3 empty dumps. Send `/var/log/erp-forensics/incident-*.txt` after the next trip. The tables to read first are **`top request paths across all vhosts`** (names the URL) and **`backup / dump / scan processes`** (for the separate 22:18-class page-cache event). ### Original bring-up (kept for reference) ```bash cd /root/forensics # or wherever you drop these chmod +x forensic-*.sh # 1. Start the black-box recorder FIRST — the next outage is the one we catch. nohup ./forensic-watch.sh -i 10 -d /var/log/erp-forensics > /dev/null 2>&1 & ./forensic-watch.sh -d /var/log/erp-forensics --status # confirm RUNNING # 2. Make it survive a reboot ( crontab -l 2>/dev/null; echo "@reboot /root/forensics/forensic-watch.sh -i 10 -d /var/log/erp-forensics" ) | crontab - # 3. Post-mortem of today. sar may ALREADY have recorded both outages. ./forensic-host.sh # 4. Web-side: no args = whole of today, auto-finds the spikes ./forensic-web.sh ``` Send me `forensic-host_*.txt` and `forensic-web_*.txt`. If step 3 says **sysstat/sar NOT INSTALLED**, install it immediately — it is the cheapest possible win and would already have answered today's question: ```bash yum install -y sysstat && systemctl enable --now sysstat # cPanel/EL sed -i 's|5,15,25,35,45,55|*/1|' /etc/cron.d/sysstat # 10min -> 1min systemctl restart crond ``` --- ## The scripts ### `forensic-watch.sh` — always-on black-box recorder The outages take the **whole OS** down, SSH included. You cannot log in and capture in-flight state; by the time you have a shell the evidence is gone. This daemon samples cheap `/proc` counters to disk every 10s so the next outage is recorded whether or not anyone can get in. If the OS hangs hard the recorder stops being scheduled too — **that gap is the signal**. `--gaps` finds each gap, prints the last sample before it, and classifies the failure mode: | Last sample before the gap | Verdict | |---|---| | `mem_avail` collapsing, swap churn | Memory exhaustion — application/DB is the cause | | `blocked(D)` spiking, memory fine | IO / storage stall — disk or host storage | | `steal%` high, memory + IO fine | Hypervisor contention — provider / noisy neighbour | | No degradation at all, then a clean gap | Host-level event — **not the ERP** | That distinction decides whether the OOM investigation is even the right investigation. ```bash ./forensic-watch.sh --status # today's peaks ./forensic-watch.sh --gaps # detected outage windows + verdict ./forensic-watch.sh --stop ``` Writes `samples-YYYY-MM-DD.tsv` (23 columns) plus `incident-*.txt` dumps (processlist, D-state procs, top RSS, recent requests) when a threshold trips. 14-day retention. MariaDB is queried only every 6th sample, 2s timeout, so the recorder cannot contribute to the problem. ### `forensic-host.sh` — host + MariaDB state ```bash ./forensic-host.sh # post-mortem ./forensic-host.sh --live # run WHILE the box is sick, if you can get in ``` Sections: OOM-kill records · **whole-OS hang evidence** (hung_task, IO errors, reboots, steal time, PID exhaustion, disk full) · **sar history** · MariaDB error log · memory config with a worst-case peak-RAM calculation · runtime counters (temp-disk-table and lock-wait ratios) · slow query log · engine/size audit · cron inventory · Apache/PHP limits. The last three close **Phase 0** items that have blocked the OOM project since 2026-07-10. ### `forensic-web.sh` — Apache access + error log ```bash ./forensic-web.sh # whole of today, finds the spikes ./forensic-web.sh -f 17:45 -t 18:05 # zoom into a window ./forensic-web.sh -d 26/Jul/2026 -f 14:00 -t 14:30 --full ``` 12 sections: traffic shape (per-minute request *and* distinct-IP histograms) · status codes and 5xx clustering · **real page loads** · screens users had open · top endpoints · expensive endpoint classes · retry amplification · per-document-id breakdown · heaviest responses by bytes · client IPs and bots · **Apache error log** (PHP memory exhaustion by script, CGI timeouts by script) · heuristic read. --- ## What was wrong with the old scripts `erp_forensic.sh` and `erp_forensic_oom.sh` are kept for provenance. Superseded. 1. **`page=` was read from the Referer, not the request.** Both used `sed -n 's/.*page=\([^& ]*\).*/\1/p'`. `.*` is greedy, so it matched the **last** `page=` on the line — which on a heartbeat POST is the referring screen, not the requested one. Verified on the 10-Jul lines: | | jobedit | quotedit | invedit | |---|---|---|---| | old method (referer) | **128** | 59 | 45 | | actual page loads | **4** | 2 | 1 | The "jobedit 128 hits" that drives the round-2 verdict in `MARIADB_OOM_INVESTIGATION.md` is a parsing artefact. See below. 2. **The time window matched nothing.** `START_REGEX="27/Jul/2026:10:[00-10]:"` — `[00-10]` is a character class (`0`, `1`, `-`), not a range, and matches a single character where the log has two-digit minutes. 3. **Wrong log path**, differently wrong in each script. `erp_forensic_oom.sh` pointed at `/usr/local/apache/logs/access_log`, which holds only localhost noise — hence `ERP Requests: 0` in `erp_forensics.txt`. Now auto-discovered. 4. **No status codes, no bytes, no error log, no timing, no concurrency measure.** 5xx responses, PHP `Allowed memory size exhausted`, and CGI `Timeout waiting for output` — the things that actually say *what broke* — were never looked at. 5. **15 sequential greps over the whole log.** Now one parse pass into a temp TSV; every section is a sort on that. 6. Output was unbounded. Now a capped digest that fits in a Claude session, with `--full` when you want everything. --- ## Finding that changes the investigation The 10-Jul window (11:10–11:17) had **511 requests from 11 distinct IPs** — about 9 users. Of those 511, **163 were `call=heartbeat` POSTs**, which fire on a timer from every open tab regardless of what anyone is doing. That is not a concurrency storm. Nine users on an ERP is a quiet Thursday morning, and the "128 jobedit loads" that the round-2 verdict was built on were 4 real loads plus ~50 background heartbeats from 3 people who had a job open on screen. Combined with today's symptom — **the entire OS unreachable, SSH included** — the application-load hypothesis looks much weaker than the audit doc currently records. A MariaDB OOM-kill leaves SSH working; that is precisely what the OOM killer is for. A box that goes dark on every service for five minutes and then returns is more consistent with memory exhaustion *plus swap thrash*, a storage stall, or hypervisor contention. `forensic-watch.sh --gaps` is designed to separate those four cases. Until it has caught one outage, that stays a hypothesis — the indexing and query work in the audit doc is still worth doing on its own merits, but it should not be assumed to be the fix for these outages. --- ## 2026-08-27 — read these first | File | What it is | |---|---| | `PHP-HANDLER-CONCURRENCY-CAP-HOWTO.md` | How to put a per-account cap on PHP. The handler is `cgi`, so `FcgidMaxProcesses` does **not** apply until it changes. Route A (per-account PHP-FPM) is recommended and already proven on this box. | | `MARIADB-CONFIG-PERSIST-HOWTO.md` | The two settings that drifted (`max_connections` 110→90, slow log OFF→ON) plus `OOMScoreAdjust=-500`. Phase 1's other my.cnf values all persisted — do not re-tune them. | | `forensic-watch.sh` (v2.2) | Adds `/proc//environ` request attribution, always-expand the burst user's vhost, `RETAIN_DAYS` 30, and a flock singleton. **Not yet deployed.** | | `forensic-watch.v2.1.sh` | The version currently running on prod. | **Two watcher instances were running 08-12 → 08-27.** Every sample in `erp-forensics/` is recorded twice. Dedupe before counting anything: ```bash awk -F'\t' '!seen[$1]++' erp-forensics/samples-2026-08-*.tsv ``` **Deploying v2.2** — stop *both* instances first, or the flock will simply refuse to start: ```bash pkill -f forensic-watch.sh ; sleep 2 ; pgrep -fa forensic-watch.sh # expect nothing cp forensic-watch.sh /root/forensic-watch.sh && chmod +x /root/forensic-watch.sh nohup /root/forensic-watch.sh -d /var/log/erp-forensics > /dev/null 2>&1 & ``` Full context: `evolution/audits/MARIADB_OOM_INVESTIGATION.md` → **Round 5**.