EVOLUTION ERP — HOST / MARIADB FORENSICS ====================================================================== Mode : post-mortem Host : server2.coscom.com.au Time : 2026-07-27 18:13:23 AEST Uptime : 18:13:23 up 28 days, 8:03, 2 users, load average: 1.22, 4.03, 22.67 Running as: root ====================================================================== 1. OOM KILLER EVENTS ====================================================================== -- via dmesg: [Thu Jul 23 07:08:07 2026] systemd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [Thu Jul 23 07:08:07 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=1481043,uid=986 [Thu Jul 23 07:08:07 2026] Out of memory: Killed process 1481043 (mariadbd) total-vm:8011708kB, anon-rss:875248kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:8452kB oom_score_adj:0 [Mon Jul 27 10:06:03 2026] PD[go.d] invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [Mon Jul 27 10:06:03 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=2265294,uid=986 [Mon Jul 27 10:06:03 2026] Out of memory: Killed process 2265294 (mariadbd) total-vm:8948316kB, anon-rss:2087408kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:12192kB oom_score_adj:0 [Mon Jul 27 17:53:03 2026] exim invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [Mon Jul 27 17:53:03 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=489952,uid=986 [Mon Jul 27 17:53:03 2026] Out of memory: Killed process 489952 (mariadbd) total-vm:6537952kB, anon-rss:1176232kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:6216kB oom_score_adj:0 -- via journalctl: Jul 27 10:05:53 server2.coscom.com.au kernel: PD[go.d] invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 27 10:05:53 server2.coscom.com.au kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=2265294,uid=986 Jul 27 10:05:53 server2.coscom.com.au kernel: Out of memory: Killed process 2265294 (mariadbd) total-vm:8948316kB, anon-rss:2087408kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:12192kB oom_score_adj:0 Jul 27 17:52:54 server2.coscom.com.au kernel: exim invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 27 17:52:54 server2.coscom.com.au kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=489952,uid=986 Jul 27 17:52:54 server2.coscom.com.au kernel: Out of memory: Killed process 489952 (mariadbd) total-vm:6537952kB, anon-rss:1176232kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:6216kB oom_score_adj:0 -- via /var/log/messages: Jul 27 10:05:56 server2.coscom.com.au kernel: [2418961.223130] PD[go.d] invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 27 10:05:56 server2.coscom.com.au kernel: [2418961.224126] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=2265294,uid=986 Jul 27 10:05:56 server2.coscom.com.au kernel: [2418961.224210] Out of memory: Killed process 2265294 (mariadbd) total-vm:8948316kB, anon-rss:2087408kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:12192kB oom_score_adj:0 Jul 27 17:52:59 server2.coscom.com.au kernel: [2446981.372413] exim invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 27 17:52:59 server2.coscom.com.au kernel: [2446981.373446] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=489952,uid=986 Jul 27 17:52:59 server2.coscom.com.au kernel: [2446981.373524] Out of memory: Killed process 489952 (mariadbd) total-vm:6537952kB, anon-rss:1176232kB, file-rss:0kB, shmem-rss:0kB, UID:986 pgtables:6216kB oom_score_adj:0 ====================================================================== 1b. WHOLE-OS HANG EVIDENCE (no-SSH outages are NOT a plain MariaDB OOM) ====================================================================== A MariaDB OOM-kill leaves SSH working — that is the OOM killer's entire job. If sshd was unreachable too, the whole OS stalled. Discriminators below. -- kernel hung-task / soft-lockup / stall warnings: [Thu Jul 23 07:08:07 2026] Call Trace: [Mon Jul 27 10:06:03 2026] Call Trace: [Mon Jul 27 17:53:03 2026] Call Trace: Jul 27 10:05:53 server2.coscom.com.au kernel: Call Trace: Jul 27 17:52:54 server2.coscom.com.au kernel: Call Trace: -- storage / IO errors (a stalled disk hangs every process incl. sshd): none found -- reboots / unclean shutdowns (did the box actually reset?): reboot system boot 4.18.0-553.94.1. Mon Jun 29 10:10 still running reboot system boot 4.18.0-553.89.1. Thu Dec 11 17:34 still running shutdown system down 4.18.0-553.63.1. Thu Dec 11 17:34 - 17:34 (00:00) reboot system boot 4.18.0-553.63.1. Mon Jul 28 22:07 - 17:34 (135+18:27) shutdown system down 4.18.0-553.63.1. Mon Jul 28 22:06 - 22:07 (00:00) reboot system boot 4.18.0-553.63.1. Mon Jul 28 19:42 - 22:06 (02:24) shutdown system down 4.18.0-553.63.1. Mon Jul 28 19:42 - 19:42 (00:00) reboot system boot 4.18.0-553.63.1. Mon Jul 28 17:40 - 19:42 (02:02) shutdown system down 4.18.0-553.62.1. Mon Jul 28 17:39 - 17:40 (00:00) reboot system boot 4.18.0-553.62.1. Mon Jul 21 19:32 - 17:39 (6+22:07) shutdown system down 4.18.0-553.44.1. Mon Jul 21 19:31 - 19:32 (00:00) reboot system boot 4.18.0-553.44.1. Thu Apr 10 23:47 - 19:31 (101+19:43) current uptime: up 4 weeks, 8 hours, 3 minutes -- virtualisation / steal time (hypervisor contention starves the whole VM): virt: vmware model name : AMD EPYC 7443 24-Core Processor cpus: 10 vmstat 1 5 (watch 'st' column — sustained >10 = hypervisor is starving us, 'b' column = procs blocked on IO, 'si/so' = swap thrash): procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 1360928 3974500 333660 3531952 4 5 115 38 2 1 6 3 91 0 0 1 0 1360928 3792896 333660 3531996 4 0 8 0 3392 2508 8 2 90 0 0 1 0 1360928 3952536 333668 3534732 12 0 12 264 3051 2789 4 2 94 0 0 1 0 1360928 3767808 333668 3532008 0 0 4 52 3318 2669 9 2 89 0 0 0 0 1360672 3981944 333668 3532012 4 0 4 0 2197 2236 2 1 97 0 0 -- fork/PID exhaustion (can't spawn sshd = looks identical to a hang): pids in use : 319 pid_max : 4194304 threads-max : 77172 nofile limit: 1024 -- disk full (a full / takes down every service): Filesystem Size Used Avail Use% Mounted on inodes: Filesystem Inodes IUsed IFree IUse% Mounted on ====================================================================== 1c. SYSSTAT / sar HISTORY *** RUN THIS FIRST AFTER AN OUTAGE *** ====================================================================== sar samples every 10 min by default and PERSISTS TO DISK, so it has almost certainly already recorded today's outages — retroactively, with no prep. -- memory (kbmemfree / %memused / kbswpused) today: 11:50:00 561888 6163796 9352136 94.33 397564 5387972 7824164 55.46 4280092 4308044 300 12:00:00 587544 6212232 9326480 94.07 404492 5395160 7586320 53.77 4273448 4283536 844 12:10:00 774340 5999708 9139684 92.19 400732 5010356 8499436 60.24 4504644 3854324 924 12:20:01 1765056 5885552 8148968 82.20 349044 3943228 8380256 59.40 3811620 3554120 692 12:30:01 1541760 5932076 8372264 84.45 370140 4205404 8299600 58.83 4133460 3469820 1432 12:40:01 896660 5953952 9017364 90.96 393248 4810540 7751996 54.95 4198840 4011412 1292 12:50:01 795636 5639376 9118388 91.97 350672 4638552 8542924 60.55 4190328 4116764 4208 13:00:00 872948 5980144 9041076 91.19 361028 4884668 7620632 54.02 4159192 4078572 328 13:10:01 1339856 5621696 8574168 86.49 292936 4390128 8345960 59.16 4371436 3404184 1212 13:20:01 1142788 5810144 8771236 88.47 306328 4515228 8238396 58.39 4836348 3115876 1628 13:30:01 1756300 5865112 8157724 82.28 284020 4009540 8467228 60.02 4071716 3286508 1656 13:40:01 1645648 5994188 8268376 83.40 312784 4246968 7900640 56.00 4660236 2823316 856 13:50:01 1576308 6051244 8337716 84.10 324500 4358880 7843348 55.59 4691348 2866916 752 14:00:01 1234236 5748612 8679788 87.55 354692 4345408 8290780 58.77 4625892 3254780 1840 14:10:01 1198068 6019796 8715956 87.92 353168 4651968 7845856 55.61 4978436 2953860 1204 14:20:01 755532 5850936 9158492 92.38 366256 4910184 7995744 56.67 4956424 3408352 440 14:30:01 1637400 5200864 8276624 83.48 323128 3497644 9571660 67.84 3528924 3937556 784 14:40:01 1619632 5789904 8294392 83.66 340560 4038592 8409936 59.61 4222328 3309532 740 14:50:01 1810060 6058132 8103964 81.74 354320 4090160 7681880 54.45 4291136 3054584 292 15:00:01 1522764 5991136 8391260 84.64 362976 4295888 7741916 54.87 4383816 3229984 1496 15:10:01 919020 6039696 8995004 90.73 392316 4906284 7546340 53.49 5147416 3071608 880 15:20:01 1044524 6176184 8869500 89.46 392512 4922164 7097984 50.31 5201276 2897868 296 15:30:01 708396 5943248 9205628 92.85 410392 5048572 7636820 54.13 5284868 3149736 908 15:40:02 1030364 6107836 8883660 89.61 386372 4876408 7349056 52.09 4904980 3200356 648 15:50:01 932848 6073524 8981176 90.59 392932 4930316 7253168 51.41 5109296 3097096 2148 16:00:01 974868 6132332 8939156 90.17 396632 4945580 7228284 51.23 5124880 3042972 1628 16:10:01 725972 5788968 9188052 92.68 393144 4849296 7733600 54.82 5194132 3194576 252 16:20:01 1613584 6132620 8300440 83.72 340028 4361184 7288912 51.66 4784436 2751604 180 16:30:01 1649556 6233740 8264468 83.36 344644 4418800 6933504 49.14 5032316 2464576 308 16:40:01 1331560 6130000 8582464 86.57 361228 4622808 7062100 50.06 5172180 2625720 796 16:50:01 1313796 6097844 8600228 86.75 359832 4601552 7145740 50.65 5159316 2652968 1364 17:00:01 1252492 6089728 8661532 87.37 365392 4666536 7349056 52.09 5221544 2653380 1008 17:10:01 1146128 6065652 8767896 88.44 410808 4653520 7004872 49.65 5268032 2592172 632 17:20:01 1759168 6172508 8154856 82.26 378780 4088004 7019524 49.75 4685760 2575632 28 17:30:01 1295028 6119256 8618996 86.94 399944 4481272 7172072 50.84 4943748 2773732 848 17:40:01 1225192 6177580 8688832 87.64 405976 4600696 6947056 49.24 5042976 2739192 484 17:52:57 844360 948440 9069664 91.48 31040 404524 16534576 117.20 6791100 1509476 704 18:00:01 5194592 7522348 4719432 47.60 273276 2264876 7050076 49.97 1763780 2318256 2388 18:10:00 4182376 7455528 5731648 57.81 329672 3198620 6903016 48.93 2627004 2444988 88 Average: 1136908 5616639 8777116 88.53 344123 4308185 7630698 54.09 4442888 3486361 1004 -- load & run queue (runq-sz, blocked) today: 11:50:00 3 617 1.03 1.18 1.16 0 12:00:00 0 611 2.03 1.51 1.27 0 12:10:00 3 623 1.42 1.45 1.37 0 12:20:01 1 645 3.50 3.08 2.28 0 12:30:01 4 626 2.47 2.28 2.19 0 12:40:01 1 612 1.03 1.34 1.77 0 12:50:01 5 645 1.40 1.60 1.71 0 13:00:00 1 612 0.93 1.26 1.50 0 13:10:01 6 635 2.72 2.03 1.69 0 13:20:01 3 651 1.69 1.56 1.60 0 13:30:01 8 656 1.07 1.32 1.50 0 13:40:01 1 633 1.44 1.67 1.64 0 13:50:01 4 639 1.53 1.46 1.52 0 14:00:01 3 657 2.22 2.53 2.12 0 14:10:01 5 626 1.13 1.58 1.84 0 14:20:01 5 638 1.51 1.68 1.77 0 14:30:01 8 705 1.26 1.66 1.80 0 14:40:01 5 640 1.66 1.33 1.54 0 14:50:01 5 629 1.20 1.35 1.47 0 15:00:01 3 646 2.54 1.53 1.41 0 15:10:01 5 624 1.24 1.25 1.34 0 15:20:01 3 604 1.05 1.18 1.22 0 15:30:01 4 622 1.31 1.38 1.30 0 15:40:02 2 611 1.17 1.18 1.25 0 15:50:01 3 606 0.22 0.63 0.98 0 16:00:01 6 609 0.31 0.45 0.72 1 16:10:01 11 630 0.54 0.66 0.76 0 16:20:01 3 606 1.43 1.27 0.99 0 16:30:01 2 588 0.68 0.80 0.89 0 16:40:01 2 601 0.84 0.83 0.87 0 16:50:01 3 598 0.73 0.61 0.70 0 17:00:01 4 605 0.76 0.88 0.82 0 17:10:01 5 592 0.58 0.82 0.80 0 17:20:01 3 594 0.29 0.53 0.64 0 17:30:01 5 598 0.78 0.60 0.62 0 17:40:01 1 593 0.44 0.39 0.47 0 17:52:57 5 738 284.47 180.17 81.56 182 18:00:01 5 596 1.26 45.28 52.56 0 18:10:00 1 571 0.76 6.93 28.02 0 Average: 4 627 3.72 3.70 3.18 2 -- cpu (%iowait, %steal) today: 11:50:00 all 7.00 0.00 4.66 0.06 0.00 88.29 12:00:00 all 8.02 0.01 4.33 0.05 0.00 87.59 12:10:00 all 7.73 0.00 4.31 0.22 0.00 87.73 12:20:01 all 16.49 0.01 6.62 0.25 0.00 76.64 12:30:01 all 12.60 0.01 5.71 0.16 0.00 81.53 12:40:01 all 9.42 0.00 4.96 0.45 0.00 85.16 12:50:01 all 8.54 0.00 4.87 0.07 0.00 86.52 13:00:00 all 8.42 0.00 4.33 0.05 0.00 87.19 13:10:01 all 11.71 0.02 5.08 0.18 0.00 83.01 13:20:01 all 9.74 0.00 5.26 0.12 0.00 84.87 13:30:01 all 8.94 0.01 4.42 0.11 0.00 86.51 13:40:01 all 7.65 0.00 4.56 0.14 0.00 87.65 13:50:01 all 6.98 0.00 3.99 0.08 0.00 88.94 14:00:01 all 15.99 0.01 7.75 0.35 0.00 75.91 14:10:01 all 9.90 0.00 5.29 0.13 0.00 84.67 14:20:01 all 9.81 0.00 5.35 0.17 0.00 84.67 14:30:01 all 10.55 0.01 5.49 0.54 0.00 83.42 14:40:01 all 7.71 0.00 4.65 0.17 0.00 87.46 14:50:01 all 8.74 0.01 4.69 0.07 0.00 86.49 15:00:01 all 5.27 0.01 3.64 0.06 0.00 91.03 15:10:01 all 6.19 0.01 4.16 0.18 0.00 89.45 15:20:01 all 7.90 0.00 3.92 0.08 0.00 88.09 15:30:01 all 8.20 0.00 3.85 0.10 0.00 87.84 15:40:02 all 8.70 0.01 4.48 0.10 0.00 86.72 15:50:01 all 5.27 0.00 3.56 0.06 0.00 91.10 16:00:01 all 3.69 0.00 2.40 0.05 0.00 93.86 16:10:01 all 5.00 0.01 2.89 0.06 0.00 92.05 16:20:01 all 6.02 0.00 3.60 0.11 0.00 90.27 16:30:01 all 3.45 0.01 2.22 0.06 0.00 94.26 16:40:01 all 4.49 0.01 2.63 0.23 0.00 92.65 16:50:01 all 4.23 0.00 2.21 0.06 0.00 93.50 17:00:01 all 6.14 0.00 2.81 0.04 0.00 91.00 17:10:01 all 5.72 0.01 2.37 0.13 0.00 91.77 17:20:01 all 3.84 0.01 2.11 0.10 0.00 93.94 17:30:01 all 3.87 0.00 2.16 0.10 0.00 93.87 17:40:01 all 2.61 0.01 1.42 0.05 0.00 95.91 17:52:57 all 7.92 0.00 42.68 4.75 0.00 44.65 18:00:01 all 5.65 0.01 2.80 2.28 0.00 89.27 18:10:00 all 6.79 0.01 2.53 0.29 0.00 90.38 Average: all 6.56 0.03 3.96 0.26 0.00 89.18 -- swapping (pswpin/s, pswpout/s) today: 14:20:01 7.89 1.90 14:30:01 62.06 87.26 14:40:01 1.25 0.02 14:50:01 0.33 0.02 15:00:01 0.41 0.01 15:10:01 0.65 0.02 15:20:01 0.46 0.01 15:30:01 0.27 0.01 15:40:02 5.56 9.50 15:50:01 0.32 0.00 16:00:01 0.18 0.00 16:10:01 0.45 0.86 16:20:01 2.69 14.33 16:30:01 0.52 0.06 16:40:01 4.41 0.04 16:50:01 0.42 0.74 17:00:01 0.22 0.06 17:10:01 0.73 3.94 17:20:01 5.30 12.93 17:30:01 1.41 0.12 17:40:01 0.17 0.02 17:52:57 155.20 1066.16 18:00:01 529.09 77.19 18:10:00 9.49 0.03 Average: 19.30 29.02 -- paging / page faults today: 14:20:01 674.41 527.09 67046.14 8.86 61544.85 128.98 9.88 126.30 90.96 14:30:01 2447.18 888.25 73013.02 42.23 70227.53 1383.80 159.93 2131.64 138.08 14:40:01 1107.89 189.89 47141.46 1.52 51230.45 4.15 3.56 12.89 167.33 14:50:01 145.13 264.80 52421.28 0.49 50621.30 0.00 3.69 6.18 167.25 15:00:01 164.74 512.81 42222.15 0.97 41430.81 0.00 3.01 5.05 167.72 15:10:01 934.67 610.25 43630.41 5.71 45597.24 0.00 6.04 10.51 174.03 15:20:01 169.82 224.23 47080.33 0.78 53385.12 33.73 6.78 50.67 125.05 15:30:01 257.06 188.96 47940.40 0.59 53320.16 0.00 4.23 7.16 169.12 15:40:02 321.05 253.97 47824.79 5.58 54047.25 357.55 7.09 273.31 74.96 15:50:01 174.09 441.43 34510.01 0.85 37908.57 0.00 2.73 4.90 179.37 16:00:01 158.78 125.22 24318.07 0.24 28878.43 0.00 2.76 4.85 175.95 16:10:01 215.70 188.22 34937.27 0.66 35054.61 118.72 5.04 173.20 139.95 16:20:01 547.64 271.02 36718.62 3.43 35811.78 468.58 9.98 560.93 117.21 16:30:01 299.53 140.61 24511.34 0.73 23465.54 0.00 5.46 9.80 179.32 16:40:01 496.04 166.13 29946.81 4.79 28582.56 0.00 6.24 10.95 175.57 16:50:01 153.89 138.32 26042.94 0.53 25180.42 38.13 15.64 70.40 130.93 17:00:01 104.66 155.65 36262.36 0.42 40691.28 0.00 4.25 7.46 175.58 17:10:01 416.69 272.37 35355.35 1.03 38546.76 175.21 4.77 173.73 96.53 17:20:01 388.97 257.48 31749.50 7.88 30269.66 451.15 14.68 576.67 123.79 17:30:01 522.26 720.88 30698.52 2.72 29245.04 31.54 4.86 71.03 195.16 17:40:01 289.06 129.68 17114.49 0.42 18440.44 0.00 0.67 1.18 176.56 17:52:57 190882.82 4348.18 36396.97 453.88 86116.02 79885.58 130324.31 99623.11 47.39 18:00:01 8508.33 494.63 36534.86 490.40 38769.88 93.45 211.58 264.43 86.69 18:10:00 2492.91 229.36 43175.22 11.49 41548.26 196.28 4.11 349.68 174.51 Average: 4339.11 456.14 41727.82 22.01 42298.41 1592.85 2382.35 2132.42 53.64 -- block IO (tps, await) today: 16:00:01 sda 17.73 158.76 125.21 16.02 0.02 0.94 0.33 0.58 16:00:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16:10:01 sda 25.76 215.72 188.23 15.68 0.03 1.27 0.26 0.68 16:10:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16:20:01 sda 44.39 547.64 271.02 18.44 0.06 1.34 0.28 1.26 16:20:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16:30:01 sda 21.44 299.53 140.61 20.53 0.02 1.03 0.33 0.70 16:30:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16:40:01 sda 66.61 496.04 166.13 9.94 0.04 0.66 0.37 2.44 16:40:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16:50:01 sda 20.46 153.89 138.32 14.28 0.02 0.93 0.32 0.65 16:50:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:00:01 sda 19.47 104.66 155.65 13.37 0.02 1.07 0.22 0.43 17:00:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:10:01 sda 58.13 416.69 272.37 11.85 0.05 0.82 0.25 1.43 17:10:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:20:01 sda 42.75 388.97 257.48 15.12 0.04 0.89 0.26 1.10 17:20:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:30:01 sda 47.04 522.26 720.88 26.43 0.07 1.53 0.23 1.08 17:30:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:40:01 sda 23.19 289.06 129.68 18.06 0.02 0.88 0.26 0.59 17:40:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 17:52:57 sda 1895.80 190882.70 4347.19 102.98 20.31 10.71 0.15 27.86 17:52:57 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18:00:01 sda 923.56 8508.55 496.44 9.75 1.43 1.54 0.11 10.11 18:00:01 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18:10:00 sda 94.02 2492.91 229.36 28.95 0.08 0.83 0.33 3.13 18:10:00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: sda 105.64 4339.11 456.14 45.39 0.47 4.40 0.22 2.34 Average: sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NOTE: gaps in the sar output = the box was too wedged to even sample. The last line BEFORE a gap is the diagnostic one. Yesterday / specific day: sar -r -f /var/log/sa/sa$(date -d yesterday +%d) Narrow to a window: sar -r -s 17:45:00 -e 18:05:00 ====================================================================== 2. MARIADB ERROR LOG (crashes, restarts, aborted connections) ====================================================================== File: /var/lib/mysql/118-127-40-55.cprapid.com.err -- startup/crash/recovery markers (last 60): 2026-07-27 13:28:58 1462029 [Warning] Aborted connection 1462029 to db: 'evolution_wp_nv3qp' user: 'evolution_wp_dhuho' host: 'localhost' (Got an error reading communication packets) 2026-07-27 13:29:00 1462053 [Warning] Aborted connection 1462053 to db: 'evolution_wp_nv3qp' user: 'evolution_wp_dhuho' host: 'localhost' (Got an error reading communication packets) 2026-07-27 13:29:05 1462373 [Warning] Aborted connection 1462373 to db: 'evolution_wp_nv3qp' user: 'evolution_wp_dhuho' host: 'localhost' (Got an error reading communication packets) 2026-07-27 13:29:06 1462392 [Warning] Aborted connection 1462392 to db: 'evolution_wp_nv3qp' user: 'evolution_wp_dhuho' host: 'localhost' (Got an error reading communication packets) 2026-07-27 17:53:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=30882451355,30882451355 2026-07-27 17:53:03 0 [Note] InnoDB: To recover: 2753 pages 2026-07-27 17:53:03 0 [Note] /usr/sbin/mariadbd: ready for connections. 2026-07-27 17:53:03 5 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_options' is marked as crashed and should be repaired 2026-07-27 17:53:03 4 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_options' is marked as crashed and should be repaired 2026-07-27 17:53:03 8 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_postmeta' is marked as crashed and should be repaired 2026-07-27 17:53:03 7 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_postmeta' is marked as crashed and should be repaired 2026-07-27 17:53:03 13 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfconfig' is marked as crashed and should be repaired 2026-07-27 17:53:04 13 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfblocks7' is marked as crashed and should be repaired 2026-07-27 17:53:04 12 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_redirects' is marked as crashed and should be repaired 2026-07-27 17:53:05 13 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfhits' is marked as crashed and should be repaired 2026-07-27 17:53:07 12 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wftrafficrates' is marked as crashed and should be repaired 2026-07-27 17:53:08 28 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_options' is marked as crashed and should be repaired 2026-07-27 17:53:11 28 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_posts' is marked as crashed and should be repaired 2026-07-27 17:53:11 28 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_postmeta' is marked as crashed and should be repaired 2026-07-27 17:53:12 28 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_wc_webhooks' is marked as crashed and should be repaired 2026-07-27 17:53:12 28 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_redirects' is marked as crashed and should be repaired 2026-07-27 17:53:14 35 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 17:53:14 35 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_actionscheduler_logs' is marked as crashed and should be repaired 2026-07-27 17:53:21 40 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_options' is marked as crashed and should be repaired 2026-07-27 17:53:24 40 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_posts' is marked as crashed and should be repaired 2026-07-27 17:53:24 40 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_postmeta' is marked as crashed and should be repaired 2026-07-27 17:53:25 41 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_jetpack_sync_queue' is marked as crashed and should be repaired 2026-07-27 17:53:26 44 [ERROR] mariadbd: Table './alison_wp_z9crm/zNZUUL_options' is marked as crashed and should be repaired 2026-07-27 17:53:27 44 [ERROR] mariadbd: Table './alison_wp_z9crm/zNZUUL_wfconfig' is marked as crashed and should be repaired 2026-07-27 17:53:36 46 [ERROR] mariadbd: Table './85@002d1559715131/jobs' is marked as crashed and should be repaired 2026-07-27 17:53:47 52 [ERROR] mariadbd: Table './108@002d1481181613/jobs' is marked as crashed and should be repaired 2026-07-27 17:53:47 52 [ERROR] mariadbd: Table './108@002d1481181613/inventory' is marked as crashed and should be repaired 2026-07-27 17:53:51 55 [ERROR] mariadbd: Table './infinium_wp754/wpnb_options' is marked as crashed and should be repaired 2026-07-27 17:53:51 56 [ERROR] mariadbd: Table './evolution/users' is marked as crashed and should be repaired 2026-07-27 17:53:51 55 [ERROR] mariadbd: Table './infinium_wp754/wpnb_wfconfig' is marked as crashed and should be repaired 2026-07-27 17:53:52 55 [ERROR] mariadbd: Table './infinium_wp754/wpnb_actionscheduler_logs' is marked as crashed and should be repaired 2026-07-27 17:53:53 64 [ERROR] mariadbd: Table './85@002d1559715131/modules' is marked as crashed and should be repaired 2026-07-27 17:53:53 61 [ERROR] mariadbd: Table './infinium_wp754/wpnb_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 17:53:53 100 [ERROR] mariadbd: Table './85@002d1559715131/quotes' is marked as crashed and should be repaired 2026-07-27 17:54:01 154 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_woocommerce_sessions' is marked as crashed and should be repaired 2026-07-27 17:54:02 484 [ERROR] mariadbd: Table './85@002d1701079776/jobs' is marked as crashed and should be repaired 2026-07-27 17:54:02 484 [ERROR] mariadbd: Table './85@002d1701079776/inventory' is marked as crashed and should be repaired 2026-07-27 17:54:03 501 [ERROR] mariadbd: Table './85@002d1559715131/quoteitems' is marked as crashed and should be repaired 2026-07-27 17:54:04 503 [ERROR] mariadbd: Table './85@002d1559715131/uploads' is marked as crashed and should be repaired 2026-07-27 17:54:07 699 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_frmt_form_views' is marked as crashed and should be repaired 2026-07-27 17:54:10 1092 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_actionscheduler_logs' is marked as crashed and should be repaired 2026-07-27 17:54:10 1092 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 17:54:15 1431 [ERROR] mariadbd: Table './vinesit_wp/7EykARYhU_options' is marked as crashed and should be repaired 2026-07-27 17:55:08 2706 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_redirection_404' is marked as crashed and should be repaired 2026-07-27 17:55:54 3398 [ERROR] mariadbd: Table './85@002d1539867819/quotes' is marked as crashed and should be repaired 2026-07-27 17:55:54 3400 [ERROR] mariadbd: Table './85@002d1539867819/quoteitems' is marked as crashed and should be repaired 2026-07-27 17:55:54 3402 [ERROR] mariadbd: Table './85@002d1539867819/uploads' is marked as crashed and should be repaired 2026-07-27 18:00:13 5392 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wflogins' is marked as crashed and should be repaired 2026-07-27 18:00:13 5392 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfreversecache' is marked as crashed and should be repaired 2026-07-27 18:00:14 5404 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfnotifications' is marked as crashed and should be repaired 2026-07-27 18:00:14 5404 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfblockediplog' is marked as crashed and should be repaired 2026-07-27 18:04:11 7045 [ERROR] mariadbd: Table './aaa_wp/WnNM0_options' is marked as crashed and should be repaired 2026-07-27 18:04:12 7049 [ERROR] mariadbd: Table './aaa_wp/WnNM0_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 18:04:53 7389 [ERROR] mariadbd: Table './85@002d1559715131/opportunities' is marked as crashed and should be repaired 2026-07-27 18:06:34 8875 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_usermeta' is marked as crashed and should be repaired -- last 40 lines: 2026-07-27 17:54:02 484 [ERROR] mariadbd: Table './85@002d1701079776/jobs' is marked as crashed and should be repaired 2026-07-27 17:54:02 484 [Warning] Checking table: './85@002d1701079776/jobs' 2026-07-27 17:54:02 484 [ERROR] mariadbd: Table './85@002d1701079776/inventory' is marked as crashed and should be repaired 2026-07-27 17:54:02 484 [Warning] Checking table: './85@002d1701079776/inventory' 2026-07-27 17:54:03 501 [ERROR] mariadbd: Table './85@002d1559715131/quoteitems' is marked as crashed and should be repaired 2026-07-27 17:54:03 501 [Warning] Checking table: './85@002d1559715131/quoteitems' 2026-07-27 17:54:04 503 [ERROR] mariadbd: Table './85@002d1559715131/uploads' is marked as crashed and should be repaired 2026-07-27 17:54:04 503 [Warning] Checking table: './85@002d1559715131/uploads' 2026-07-27 17:54:07 699 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_frmt_form_views' is marked as crashed and should be repaired 2026-07-27 17:54:07 699 [Warning] Checking table: './madalice_wp/AMuqf6qZ_frmt_form_views' 2026-07-27 17:54:10 1092 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_actionscheduler_logs' is marked as crashed and should be repaired 2026-07-27 17:54:10 1092 [Warning] Checking table: './madalice_wp/AMuqf6qZ_actionscheduler_logs' 2026-07-27 17:54:10 1092 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 17:54:10 1092 [Warning] Checking table: './madalice_wp/AMuqf6qZ_actionscheduler_claims' 2026-07-27 17:54:15 1431 [ERROR] mariadbd: Table './vinesit_wp/7EykARYhU_options' is marked as crashed and should be repaired 2026-07-27 17:54:15 1431 [Warning] Checking table: './vinesit_wp/7EykARYhU_options' 2026-07-27 17:55:08 2706 [ERROR] mariadbd: Table './evolution_wp_nv3qp/QpCIPREUS_redirection_404' is marked as crashed and should be repaired 2026-07-27 17:55:08 2706 [Warning] Checking table: './evolution_wp_nv3qp/QpCIPREUS_redirection_404' 2026-07-27 17:55:54 3398 [ERROR] mariadbd: Table './85@002d1539867819/quotes' is marked as crashed and should be repaired 2026-07-27 17:55:54 3398 [Warning] Checking table: './85@002d1539867819/quotes' 2026-07-27 17:55:54 3400 [ERROR] mariadbd: Table './85@002d1539867819/quoteitems' is marked as crashed and should be repaired 2026-07-27 17:55:54 3400 [Warning] Checking table: './85@002d1539867819/quoteitems' 2026-07-27 17:55:54 3402 [ERROR] mariadbd: Table './85@002d1539867819/uploads' is marked as crashed and should be repaired 2026-07-27 17:55:54 3402 [Warning] Checking table: './85@002d1539867819/uploads' 2026-07-27 18:00:13 5392 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wflogins' is marked as crashed and should be repaired 2026-07-27 18:00:13 5392 [Warning] Checking table: './madalice_wp/AMuqf6qZ_wflogins' 2026-07-27 18:00:13 5392 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfreversecache' is marked as crashed and should be repaired 2026-07-27 18:00:13 5392 [Warning] Checking table: './madalice_wp/AMuqf6qZ_wfreversecache' 2026-07-27 18:00:14 5404 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfnotifications' is marked as crashed and should be repaired 2026-07-27 18:00:14 5404 [Warning] Checking table: './madalice_wp/AMuqf6qZ_wfnotifications' 2026-07-27 18:00:14 5404 [ERROR] mariadbd: Table './madalice_wp/AMuqf6qZ_wfblockediplog' is marked as crashed and should be repaired 2026-07-27 18:00:14 5404 [Warning] Checking table: './madalice_wp/AMuqf6qZ_wfblockediplog' 2026-07-27 18:04:11 7045 [ERROR] mariadbd: Table './aaa_wp/WnNM0_options' is marked as crashed and should be repaired 2026-07-27 18:04:11 7045 [Warning] Checking table: './aaa_wp/WnNM0_options' 2026-07-27 18:04:12 7049 [ERROR] mariadbd: Table './aaa_wp/WnNM0_actionscheduler_claims' is marked as crashed and should be repaired 2026-07-27 18:04:12 7049 [Warning] Checking table: './aaa_wp/WnNM0_actionscheduler_claims' 2026-07-27 18:04:53 7389 [ERROR] mariadbd: Table './85@002d1559715131/opportunities' is marked as crashed and should be repaired 2026-07-27 18:04:53 7389 [Warning] Checking table: './85@002d1559715131/opportunities' 2026-07-27 18:06:34 8875 [ERROR] mariadbd: Table './corpkicks_wp_baolr/MPHOrV_usermeta' is marked as crashed and should be repaired 2026-07-27 18:06:34 8875 [Warning] Checking table: './corpkicks_wp_baolr/MPHOrV_usermeta' ====================================================================== 3. MEMORY / RESOURCE BASELINE ====================================================================== total used free shared buff/cache available Mem: 9681 2105 2175 80 5400 7110 Swap: 4095 1328 2767 -- MemTotal / swap: MemTotal: 9914024 kB MemAvailable: 7280492 kB SwapTotal: 4194300 kB SwapFree: 2833628 kB Committed_AS: 7000008 kB -- top 25 by RSS: PID USER RSS %CPU ELAPSED COMMAND 856299 mysql 840540 5.4 20:30 mariadbd 1220 netdata 176404 2.4 28-08:03:22 netdata 640860 root 145244 0.0 04:28:14 spamd child 864426 madalice 113328 0.0 00:00 php-cgi 851423 cpanell+ 48848 0.0 39:58 show_template.s 4922 netdata 46256 0.1 28-08:03:17 go.d.plugin 159820 root 40992 0.0 7-08:26:30 systemd-journal 1469 root 40740 0.1 28-08:03:21 cpdavd - accept 2223 root 25532 0.0 28-08:03:21 /opt/saltstack/ 861743 nobody 22292 0.1 05:38 httpd 862978 nobody 21340 0.1 02:59 httpd 863281 nobody 20872 0.0 02:12 httpd 863511 nobody 20732 0.4 01:48 httpd 862977 nobody 20588 0.0 02:59 httpd 863225 nobody 19536 0.0 02:14 httpd 856445 root 18884 0.0 20:25 spamd child 863786 nobody 18180 0.0 01:06 httpd 863279 nobody 18120 0.0 02:12 httpd 1481367 netdata 17768 2.2 5-09:28:06 apps.plugin 863784 nobody 16500 0.0 01:06 httpd 863785 nobody 16424 0.0 01:06 httpd 863783 nobody 16144 0.0 01:06 httpd 1353308 root 15516 0.1 5-15:31:52 cpsrvd (SSL) - 863776 nobody 15496 0.0 01:07 httpd 254302 root 15200 0.0 15:50:39 spamd ====================================================================== 4. MARIADB MEMORY CONFIG (the OOM amplifier) ====================================================================== +-------------------------------+------------------+ | Variable_name | Value | +-------------------------------+------------------+ | default_storage_engine | InnoDB | | innodb_buffer_pool_size | 1073741824 | | innodb_log_buffer_size | 16777216 | | join_buffer_size | 262144 | | key_buffer_size | 1000341504 | | log_queries_not_using_indexes | OFF | | long_query_time | 10.000000 | | max_allowed_packet | 268435456 | | max_connections | 151 | | max_heap_table_size | 101711872 | | myisam_sort_buffer_size | 134216704 | | net_buffer_length | 16384 | | open_files_limit | 40000 | | performance_schema | OFF | | query_cache_size | 1048576 | | read_buffer_size | 131072 | | read_rnd_buffer_size | 262144 | | slow_query_log | OFF | | slow_query_log_file | server2-slow.log | | sort_buffer_size | 2097152 | | table_open_cache | 2000 | | thread_cache_size | 151 | | tmp_table_size | 101711872 | | version | 10.6.24-MariaDB | +-------------------------------+------------------+ -- WORST-CASE MEMORY MATH (per-connection buffers x max_connections + global): global : 1994 MB\n per-conn : 100 MB\n x max_connections (151) = 15043 MB\n THEORETICAL PEAK = 17037 MB ^ compare THEORETICAL PEAK against MemTotal above. If peak > RAM, the box is one concurrency spike away from an OOM kill by design. ====================================================================== 5. MARIADB RUNTIME COUNTERS (scan + temp-table fingerprint) ====================================================================== +----------------------------------+----------+ | Variable_name | Value | +----------------------------------+----------+ | Aborted_clients | 0 | | Aborted_connects | 0 | | Connections | 14852 | | Created_tmp_disk_tables | 2087 | | Created_tmp_files | 29 | | Created_tmp_tables | 8058 | | Handler_read_first | 2174 | | Handler_read_key | 946575 | | Handler_read_rnd_next | 76049717 | | Innodb_buffer_pool_read_requests | 725123 | | Innodb_buffer_pool_reads | 26801 | | Key_read_requests | 3183858 | | Key_reads | 42846 | | Max_used_connections | 21 | | Open_tables | 546 | | Opened_tables | 606 | | Select_full_join | 66 | | Select_scan | 11923 | | Slow_queries | 0 | | Sort_merge_passes | 12 | | Sort_scan | 6180 | | Table_locks_immediate | 167937 | | Table_locks_waited | 42 | | Threads_connected | 2 | | Threads_running | 1 | | Uptime | 1231 | +----------------------------------+----------+ -- derived: disk tmp table ratio : 25.9% (>25% = bad)\n table lock wait ratio: 0.03% (MyISAM pileup) ====================================================================== 6. SLOW QUERY LOG (Phase 0 blocker) ====================================================================== slow_query_log : 0 slow_query_log_file : server2-slow.log long_query_time : 10.000000 !! NOT ENABLED (or unreadable). This is why we still cannot confirm H1/H2/H3. Enable at runtime, no restart needed: SET GLOBAL slow_query_log = 1; SET GLOBAL long_query_time = 2; SET GLOBAL log_queries_not_using_indexes = 1; (add to my.cnf to survive restart; watch disk with log_queries_not_using_indexes on) ====================================================================== 7. STORAGE ENGINE / TABLE SIZE AUDIT (top 25 tables, all tenants) ====================================================================== +------------------------+-----------------+--------+---------+--------+----------+ | db | tbl | ENGINE | data_mb | idx_mb | rows_est | +------------------------+-----------------+--------+---------+--------+----------+ | 85-1525752154 | log | InnoDB | 514.0 | 181.4 | 1950495 | | 108-1481181613 | log | InnoDB | 490.0 | 130.2 | 1972627 | | 85-1384776560 | log | InnoDB | 427.9 | 141.3 | 1631892 | | 85-1559715131-restored | log | InnoDB | 395.9 | 135.3 | 1445494 | | 85-1559715131 | log | InnoDB | 379.9 | 96.2 | 1323875 | | lecsafec_det | asset_results | MyISAM | 355.2 | 90.6 | 7935733 | | lecsafec_icatv2 | results | MyISAM | 270.9 | 209.0 | 3983655 | | vinesit_westtankers | schedule | MyISAM | 268.4 | 54.6 | 4592352 | | lecsafec_icatv2 | assets | MyISAM | 265.2 | 88.6 | 2671421 | | lecsafec_patguard | asset_results | MyISAM | 228.3 | 58.5 | 5187712 | | 108-1481181613 | quotes | MyISAM | 116.8 | 5.0 | 75675 | | 85-1525752154 | quotes | MyISAM | 113.2 | 4.9 | 74403 | | 108-1481181613 | quoteitems | MyISAM | 111.9 | 10.2 | 601851 | | 85-1525752154 | quoteitems | MyISAM | 110.0 | 9.6 | 590373 | | lecsafec_main | assets | MyISAM | 107.7 | 75.8 | 1586461 | | 85-1559715131 | quoteitems | MyISAM | 102.9 | 9.9 | 573089 | | 85-1384776560 | quoteitems | MyISAM | 102.9 | 9.4 | 572829 | | lecsafec_main | results | MyISAM | 100.1 | 36.3 | 1596240 | | 85-1559715131-restored | quoteitems | MyISAM | 90.8 | 7.2 | 549163 | | lecsafec_det | assets | MyISAM | 89.6 | 54.5 | 1196735 | | lecsafec_det | results | MyISAM | 70.9 | 10.0 | 1205087 | | 108-1481181613 | dispatch | MyISAM | 67.6 | 1.0 | 31077 | | corpkicks_wp_baolr | MPHOrV_postmeta | MyISAM | 62.2 | 2.7 | 67821 | | 85-1559715131 | dispatch | MyISAM | 59.9 | 0.5 | 15178 | | 85-1384776560 | dispatch | MyISAM | 59.9 | 0.5 | 15170 | +------------------------+-----------------+--------+---------+--------+----------+ -- engine mix: +--------+--------+---------+ | ENGINE | tables | data_mb | +--------+--------+---------+ | MyISAM | 4549 | 4726 | | InnoDB | 6479 | 4096 | | MEMORY | 14 | 0 | +--------+--------+---------+ ====================================================================== 8. SCHEDULER / CRON INVENTORY (Phase 0 blocker) ====================================================================== -- system crontab: SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root -- /etc/cron.d: PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin 25 1 * * 6 root /usr/bin/tmpwatch 168 /var/imunify360/tmp > /dev/null 2>&1 || : 25 2 * * 6 root /usr/bin/tmpwatch 168 /var/imunify360/tmp_cleanup > /dev/null 2>&1 || : */5 * * * * root bash -c "sleep $((RANDOM \% 60))" ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || : 17 4 * * * root /usr/libexec/report-command-error /opt/imunify360/venv/share/imunify360/scripts/update_components_versions.py > /dev/null 2>&1 0 * * * * root bash -c "sleep $((RANDOM \% 1800))" ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 32 0,3,6,9,12,15,18,21 * * * root /usr/local/cpanel/bin/autossl_check --all 55 21 * * 0 root /usr/local/prospamfilter/bin/checkUpdate.php SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root 01 * * * * root run-parts /etc/cron.hourly 11 1 * * * root /usr/local/cpanel/scripts/gather_update_log_stats --logfile /var/cpanel/updatelogs/last --upload > /dev/null 2>&1 SHELL=/bin/bash MAILTO=postmaster 0 8 * * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/checkdbs 0 9 * * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/disabled 0 12 * * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/senddigests 0 5 1 * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/mailpasswds 27 3 * * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/nightly_gzip 30 4 * * * mailman /usr/local/cpanel/scripts/restartsrv_mailman --status &> /dev/null && /usr/bin/python3 -S /usr/local/cpanel/3rdparty/mailman/cron/cull_bad_shunt 0 1 * * * root sleep $((1 + RANDOM \% 5))h $((1 + RANDOM \% 60))m; /usr/local/bin/wp-toolkit update-configuration > /dev/null 2> /dev/null || /usr/local/cpanel/3rdparty/wp-toolkit/bin/wp-toolkit-installer.sh --generate-configs > /dev/null 2> /dev/null; /usr/bin/yum -y update wp-toolkit-cpanel > /dev/null 2> /dev/null 49 2 * * * root /usr/local/cpanel/bin/process_ssl_reissue SHELL=/bin/sh 0 0 * * * root /usr/sbin/csf --lfd restart > /dev/null 2>&1 SHELL=/bin/bash MAILTO="" * * * * * root /usr/sbin/imunify-notifier -update-cron SHELL=/bin/sh -- per-user crontabs: [aaa] [alison] [coscom] [evolution] MAILTO="" SHELL="/bin/bash" */15 * * * * /home/evolution/my.evolutionerp.com.au/cron/xero2.php SHELL="/bin/bash" */30 * * * * /home/evolution/my.evolutionerp.com.au/cron/invoicing.php SHELL="/bin/bash" 0 2 * * * /home/evolution/my.evolutionerp.com.au/cron/daily.php SHELL="/bin/bash" 0 * * * * /home/evolution/my.evolutionerp.com.au/cron/hourly.php SHELL="/bin/bash" 0 */4 * * * /home/evolution/my.evolutionerp.com.au/cron/xero_payment_cron.php SHELL="/bin/bash" */5 * * * * /home/evolution/pull_staging.sh SHELL="/bin/bash" */5 * * * * /home/evolution/pull_portal_staging.sh SHELL="/bin/bash" 0,30 * * * * /home/evolution/pull_production.sh [fencetech] [infinium] 52 23 27 * * /usr/local/cpanel/3rdparty/bin/php -d disable_functions="" "/usr/local/cpanel/whostmgr/docroot/cgi/softaculous"/cli.php --backup --auto=1 --insid=26_46576 [kamcool] [keans] [kurri] [lecsafec] [lowrange] [madalice] MAILTO="" SHELL="/bin/bash" 0 4 * * * wget -q -O - https://madalice.com.au/wp-cron.php?doing_wp_cron >/dev/null 2>&1 [nrg] SHELL="/usr/local/cpanel/bin/jailshell" MAILTO="" [olympic] [olytis] [otter2] [otterfencing] [root] 22 2 * * * (/usr/local/cpanel/scripts/fix-cpanel-perl; /usr/local/cpanel/scripts/upcp --cron > /dev/null) 0 2 * * * /usr/local/cpanel/bin/backup 35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check 0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1 30 5 * * * /usr/local/cpanel/scripts/optimize_eximstats > /dev/null 2>&1 5,20,35,50 * * * * /usr/local/cpanel/scripts/eximstats_spam_check 2>&1 45 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_mailman_cache && /usr/local/cpanel/scripts/update_mailman_cache 19 3 * * * /usr/local/cpanel/scripts/autorepair recoverymgmt >/dev/null 2>&1 */5 * * * * /usr/local/cpanel/scripts/dcpumon-wrapper >/dev/null 2>&1 30 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache 25 */2 * * * /usr/local/cpanel/bin/mysqluserstore >/dev/null 2>&1 15 */2 * * * /usr/local/cpanel/bin/dbindex >/dev/null 2>&1 0 */2 * * * /usr/local/cpanel/scripts/shrink_modsec_ip_database -x 2>&1 9,24,39,54 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1 0 0 * * * /usr/local/cpanel/3rdparty/bin/perl /root/scripts/r1softprep.pl >/dev/null 2>&1 */5 * * * * /root/dnsexport.pl >/dev/null 2>&1 * * * * * /bin/bash -c 'top -b -n1 > /root/toplogs/top_snapshot_$(date +\%F_\%H-\%M-\%S).txt' >/dev/null 2>&1 * * * * * ls -1t /root/toplogs/top_snapshot_*.txt 2>/dev/null | tail -n +31 | xargs -r rm -f 09,39 * * * * /usr/local/cpanel/scripts/clean_user_php_sessions > /dev/null 2>&1 4 * * * * /usr/local/cpanel/bin/process_team_queue > /dev/null 2>&1 13 * * * * /usr/local/cpanel/scripts/upcp --security [speedlin] [vinesit] [whiz] -- systemd timers: NEXT LEFT LAST PASSED UNIT ACTIVATES Mon 2026-07-27 18:20:00 AEST 6min left Mon 2026-07-27 18:10:00 AEST 3min 53s ago sysstat-collect.timer sysstat-collect.service Mon 2026-07-27 19:07:38 AEST 53min left Mon 2026-07-27 17:36:05 AEST 37min ago dnf-makecache.timer dnf-makecache.service Tue 2026-07-28 00:00:00 AEST 5h 46min left Mon 2026-07-27 00:00:00 AEST 18h ago initialize_monitoring_plugin.timer initialize_monitoring_plugin.service Tue 2026-07-28 00:00:00 AEST 5h 46min left Mon 2026-07-27 00:00:00 AEST 18h ago initialize_socialbee_plugin.timer initialize_socialbee_plugin.service Tue 2026-07-28 00:00:00 AEST 5h 46min left Mon 2026-07-27 00:00:00 AEST 18h ago initialize_xovi_plugin.timer initialize_xovi_plugin.service Tue 2026-07-28 00:00:00 AEST 5h 46min left Mon 2026-07-27 00:00:00 AEST 18h ago unbound-anchor.timer unbound-anchor.service Tue 2026-07-28 00:07:00 AEST 5h 53min left Mon 2026-07-27 00:07:00 AEST 18h ago sysstat-summary.timer sysstat-summary.service Tue 2026-07-28 10:25:21 AEST 16h left Mon 2026-07-27 10:25:21 AEST 7h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service 8 timers listed. NOTE: MARIADB_OOM_INVESTIGATION.md records the production scheduler as living OFF-BOX. If the above is empty, the external scheduler's job list is still an outstanding Phase 0 item. ====================================================================== 9. APACHE / PHP LIMITS ====================================================================== Server version: Apache/2.4.66 (cPanel) Server MPM: prefork ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 Timeout 300 ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 Timeout 300 ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 Timeout 300 ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 Timeout 300 ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 Timeout 300 -- PHP limits: memory_limit : 128M max_execution_time: 0 post_max_size : 8M -- disk: Filesystem Size Used Avail Use% Mounted on /dev/sda2 641G 485G 130G 79% / /dev/sda1 974M 294M 613M 33% /boot ====================================================================== Report written to: forensic-host_20260727-181323.txt ======================================================================