INCIDENT 2026-08-13 23:15:43 trigger: load=140.57/10cpu ================================================================== load: 140.57 67.21 26.22 2/678 1339327 ncpu:10 mem_avail: 9081MB/11729MB (77.4%) swap_used:1785MB swapout:511/s blocked(D):1 running:3 iowait:5.9% steal:0.0% php: 2 procs / 276MB heaviest user: madalice (1 procs, 212MB) apache: 99 procs tcp_estab: 24 --- RSS by user (all processes) --- USER PROCS RSS_MB nobody 98 1010 mysql 1 456 root 247 347 netdata 8 218 madalice 1 218 alison 1 67 mailnull 7 39 wp-toolkit 6 16 lowrange 2 16 kurri 2 16 dovenull 6 14 vinesit 1 13 mailman 20 13 cpanellogin 1 10 zabbix 1 9 cpanelphpmyadmin 1 8 dovecot 3 7 named 1 2 evolution 4 2 dbus 1 2 --- php workers by user --- USER PROCS RSS_MB madalice 1 220 alison 1 68 evolution 1 32 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 1052 MB Active(file): 450 MB Inactive(file): 389 MB Dirty: 3 MB Writeback: 0 MB SReclaimable: 178 MB CommitLimit: 9960 MB Committed_AS: 8562 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 342014 root 22:04:26 984 /usr/bin/imunify-agent-proxy 1331798 root 15:42 76 /bin/sh -c bash -c "sleep $((RANDOM % 1800))" ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 1338878 root 00:08 3132 /bin/sh -c bash -c "sleep $((RANDOM % 60))" ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || : --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 1339177 1 mysql 466984 27.0 Ssl 00:05 /usr/sbin/mariadbd 1339218 1337557 madalice 235536 66.0 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1220 1 netdata 170044 2.5 Ssl 45-13:05:44 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 382990 1 root 83880 0.0 Ss 20:52:56 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 1339296 1338740 alison 72352 0.0 D 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 388652 382990 root 69492 0.0 S 20:41:44 spamd child 1339332 1338746 evolution 40880 0.0 S 00:00 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 4922 2431 netdata 37860 0.1 Sl 45-13:05:39 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2223 1201 root 21960 0.0 Sl 45-13:05:43 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 159820 1 root 21688 0.0 Ss 24-13:28:52 /usr/lib/systemd/systemd-journald 1338558 1276780 nobody 18772 0.0 S 02:37 /usr/sbin/httpd -k start 1338560 1276780 nobody 16720 0.0 S 02:37 /usr/sbin/httpd -k start 382254 1 root 16376 0.0 Ss 20:53:30 lfd - sleeping 1338526 1276780 nobody 15684 0.0 S 02:38 /usr/sbin/httpd -k start 1338557 1276780 nobody 15608 0.0 S 02:37 /usr/sbin/httpd -k start 1338556 1276780 nobody 15552 0.0 S 02:37 /usr/sbin/httpd -k start 1338523 1276780 nobody 15480 0.0 S 02:38 /usr/sbin/httpd -k start 1338524 1276780 nobody 15348 0.0 S 02:38 /usr/sbin/httpd -k start 1338746 1276780 nobody 15272 1.0 S 02:13 /usr/sbin/httpd -k start 1338527 1276780 nobody 15232 0.0 S 02:38 /usr/sbin/httpd -k start 1338571 1276780 nobody 15148 0.0 S 02:37 /usr/sbin/httpd -k start 1338567 1276780 nobody 15036 0.0 S 02:37 /usr/sbin/httpd -k start 1338740 1276780 nobody 15020 1.1 S 02:16 /usr/sbin/httpd -k start 1338522 1276780 nobody 14468 0.0 S 02:38 /usr/sbin/httpd -k start 1338866 3828812 lowrange 14340 0.0 S 00:08 dovecot/imap 1339220 3828812 kurri 14072 0.0 S 00:02 dovecot/imap 1338856 1471 root 13980 0.3 S 00:09 cPhulkd - processor - http socket 1338392 1276780 nobody 13780 0.0 S 02:39 /usr/sbin/httpd -k start 1338739 1276780 nobody 13772 1.1 S 02:16 /usr/sbin/httpd -k start 1338521 1276780 nobody 13676 0.0 S 02:38 /usr/sbin/httpd -k start 1338520 1276780 nobody 13624 0.0 S 02:38 /usr/sbin/httpd -k start 1338867 3828812 vinesit 13604 0.0 S 00:08 dovecot/imap 1338230 1276780 nobody 13480 0.5 S 02:43 /usr/sbin/httpd -k start 185104 2431 netdata 13460 2.0 Sl 1-02:42:02 /usr/libexec/netdata/plugins.d/apps.plugin 1 1338215 1276780 nobody 13148 0.0 S 02:43 /usr/sbin/httpd -k start 1353308 1 root 13096 0.0 Ss 22-20:34:14 cpsrvd (SSL) - waiting for connections 1338225 1276780 nobody 12952 0.0 S 02:43 /usr/sbin/httpd -k start 1338565 1276780 nobody 12924 0.0 S 02:37 /usr/sbin/httpd -k start 1338214 1276780 nobody 12912 0.0 S 02:43 /usr/sbin/httpd -k start 1337557 1276780 nobody 12836 0.0 S 04:05 /usr/sbin/httpd -k start 1338390 1276780 nobody 12616 0.0 S 02:39 /usr/sbin/httpd -k start 1337526 1276780 nobody 12588 0.0 S 04:07 /usr/sbin/httpd -k start 1338223 1276780 nobody 12196 0.0 S 02:43 /usr/sbin/httpd -k start 1338849 3828812 root 11872 0.0 S 00:09 dovecot/auth -w 1463 1 root 10948 0.0 Ss 45-13:05:43 tailwatchd 1338227 1276780 nobody 10688 0.0 S 02:43 /usr/sbin/httpd -k start 1338190 1276780 nobody 10560 0.0 S 02:44 /usr/sbin/httpd -k start 1337559 1276780 nobody 10424 0.0 S 04:05 /usr/sbin/httpd -k start 1318469 1353308 cpanellogin 10396 0.0 S 42:16 /usr/local/cpanel/base/show_template.stor default_login_theme cpanel server_locale en docroot /usr/local/cpanel/base cpanel_locale 1335545 1471 root 10392 0.2 S 08:19 cPhulkd - dbprocessor 1338195 1276780 nobody 10376 0.6 S 02:44 /usr/sbin/httpd -k start 1337681 1276780 nobody 10328 0.0 S 03:53 /usr/sbin/httpd -k start 1338160 1276780 nobody 10284 0.0 S 02:45 /usr/sbin/httpd -k start 1338239 1276780 nobody 10284 0.0 S 02:43 /usr/sbin/httpd -k start 1204 1 zabbix 10172 0.2 Ssl 45-13:05:44 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 1338140 1276780 nobody 10172 0.0 S 02:47 /usr/sbin/httpd -k start 1338150 1276780 nobody 10160 0.0 S 02:46 /usr/sbin/httpd -k start 1338750 1276780 nobody 10148 0.3 S 02:13 /usr/sbin/httpd -k start 1336803 1276780 nobody 10080 0.0 S 05:33 /usr/sbin/httpd -k start 1337533 1276780 nobody 10072 0.0 S 04:06 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- 1339296 D - /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1339297 D - /sbin/exim -bpc --- mariadb processlist (non-sleeping) --- +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 59 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min madalice/madalice.com.au-ssl_log 94 evolution/portal.evolutionerp.com.au-ssl_log 5 (total vhosts taking traffic: 2) --- top client IPs across all vhosts (last 3 min) --- 5 203.54.180.238 2 85.85.26.210 2 45.184.143.58 2 190.61.45.115 2 187.250.167.100 1 94.143.197.197 1 93.157.180.30 1 89.151.189.159 1 80.233.45.71 1 77.81.171.13 1 77.79.148.23 1 62.217.157.214 1 62.122.60.48 1 5.134.233.51 1 51.186.14.163 --- top request paths across all vhosts (last 3 min) --- 29 GET / 26 POST / 5 GET /heartbeat.php 2 GET /store/rabbitohs-custom-shoes-us9-5/ 2 GET /store/premium-wrapping/ 2 GET /store/croc-charms-stitch/ 2 GET /store/choc-donut/ 2 GET /product-category/custom-shoes/ 2 GET /customise-shoes-online/ 2 GET /category/uncategorized/ 2 GET /category/custom-sneakers/ 1 POST /wp-login.php 1 GET /wp-json/wc/store/v1/cart 1 GET /terms-and-conditions/ 1 GET /store/tripleblack-thick/ 1 GET /store/swarovski-crystal-am90/ 1 GET /store/pink-frangipani/ 1 GET /store/pink-butterfly-blossom-af1-shoes/ 1 GET /store/hairyscotsman/ 1 GET /store/felix_sneakers/ --- busiest 3 vhosts: top paths + last 10 requests --- == madalice/madalice.com.au-ssl_log == -- top paths -- 29 GET / 26 POST / 2 GET /store/rabbitohs-custom-shoes-us9-5/ 2 GET /store/premium-wrapping/ 2 GET /store/croc-charms-stitch/ 2 GET /store/choc-donut/ 2 GET /product-category/custom-shoes/ 2 GET /customise-shoes-online/ 2 GET /category/uncategorized/ 2 GET /category/custom-sneakers/ 1 POST /wp-login.php 1 GET /wp-json/wc/store/v1/cart -- last 10 requests -- 43.119.100.235 - - [13/Aug/2026:23:13:05 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/brand/adidas/?srsltid=AfmBOoo2XudNNT5_ZchdNmsyr-Fi8GM2_Zn_Pu0z62pplqnZFK5GF7JO" "Moz 114.119.131.88 - - [13/Aug/2026:23:13:02 +1000] "GET /store/swarovski-crystal-am90/?verified=1&rating=3&image=1&wcpr_page=2 HTTP/1.1" 200 45173 "https://madalice.com.au/store/swarovski-crystal-am90/?verified=1&rating=3&i 124.106.91.150 - - [13/Aug/2026:23:13:02 +1000] "GET /size-charts/?srsltid=AfmBOoqc8uEAK8RF_ul8K-NLKoIYY-V9apXuc0o6_EQSB5EkerPwWmOA HTTP/1.1" 200 31775 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537 102.223.58.112 - - [13/Aug/2026:23:13:02 +1000] "GET /category/custom-sneakers/?srsltid=AfmBOoqg0iP7qrUiml3h9uKjC6wXEZYwQBDRTydC_fxPhl7GoN7uoOTG HTTP/1.1" 200 39917 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Ap 47.82.201.82 - - [13/Aug/2026:23:13:02 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/product-tag/swarovski/?srsltid=AfmBOopdX4aoHHsR7MGycOBxbYGMoxJon-CBm5ELQBSr963Xnj5N8Kw 179.125.165.30 - - [13/Aug/2026:23:13:02 +1000] "GET /store/pink-frangipani/?srsltid=AfmBOor6qsLODamLfX-0F5NTp6lH3_WCaf64JkhEhK07jergEdzjxaEy HTTP/1.1" 500 861 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWe 192.141.112.136 - - [13/Aug/2026:23:13:02 +1000] "GET / HTTP/1.1" 200 36544 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" 177.234.190.220 - - [13/Aug/2026:23:13:09 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/store/swarovski-crystal-am90/?srsltid=AfmBOorPnKuvYW6I8SVKyUJMsaEoPXd0KIy9sceTWwlSDLXZLL 27.79.234.218 - - [13/Aug/2026:23:13:03 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/product-tag/swarovski/?srsltid=AfmBOopdX4aoHHsR7MGycOBxbYGMoxJon-CBm5ELQBSr963Xnj5N8KwU" " 62.122.60.48 - - [13/Aug/2026:23:13:01 +1000] "GET /product-category/custom-shoes/?srsltid=AfmBOorQX8ELUonhv75UsmAi9vS_pqh3527Bde02YxlvbjvoM5k-ZI8K HTTP/1.1" 200 36135 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 5 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [13/Aug/2026:23:13:02 +1000] "GET /heartbeat.php?sentAt=1786626780&bytes=106,125,194,220&_=1786602369786 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:23:13:04 +1000] "GET /heartbeat.php?sentAt=1786626783&bytes=106,125,194,223&_=1786596792036 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:23:13:10 +1000] "GET /heartbeat.php?sentAt=1786626790&bytes=106,125,194,230&_=1786602369787 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:23:13:20 +1000] "GET /heartbeat.php?sentAt=1786626800&bytes=106,125,194,240&_=1786602369788 HTTP/1.1" 500 20 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:23:13:13 +1000] "GET /heartbeat.php?sentAt=1786626793&bytes=106,125,194,233&_=1786596792037 HTTP/1.1" 500 20 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 --- last 30 samples (run-up to this incident) --- 2026-08-13 23:11:07 L1=1.09 run=1 blk=0 avail=7304 (62.3%) swpO=0 conn=3 myRSS=2245 php=0 phpRSS=0 ap=14 tcp=18 topuser=- 2026-08-13 23:11:13 L1=1.09 run=1 blk=0 avail=7288 (62.1%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=40 ap=14 tcp=22 topuser=evolution 2026-08-13 23:11:18 L1=1.08 run=1 blk=0 avail=7279 (62.1%) swpO=0 conn=3 myRSS=2245 php=0 phpRSS=0 ap=17 tcp=24 topuser=- 2026-08-13 23:11:23 L1=1.07 run=2 blk=0 avail=7152 (61.0%) swpO=0 conn=2 myRSS=2245 php=2 phpRSS=189 ap=17 tcp=21 topuser=madalice 2026-08-13 23:11:28 L1=0.99 run=1 blk=0 avail=7280 (62.1%) swpO=0 conn=3 myRSS=2245 php=0 phpRSS=0 ap=17 tcp=21 topuser=- 2026-08-13 23:11:33 L1=1.07 run=3 blk=0 avail=6996 (59.6%) swpO=0 conn=2 myRSS=2245 php=3 phpRSS=377 ap=16 tcp=54 topuser=tubesupplies 2026-08-13 23:11:38 L1=1.06 run=4 blk=0 avail=6988 (59.6%) swpO=0 conn=3 myRSS=2245 php=2 phpRSS=142 ap=76 tcp=77 topuser=alison 2026-08-13 23:11:43 L1=1.06 run=4 blk=0 avail=6875 (58.6%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=41 ap=74 tcp=46 topuser=evolution 2026-08-13 23:11:48 L1=0.97 run=3 blk=0 avail=6673 (56.9%) swpO=0 conn=4 myRSS=2245 php=3 phpRSS=330 ap=69 tcp=58 topuser=customc 2026-08-13 23:11:53 L1=0.89 run=1 blk=0 avail=6796 (57.9%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=41 ap=83 tcp=62 topuser=evolution 2026-08-13 23:11:58 L1=0.82 run=1 blk=0 avail=6842 (58.3%) swpO=0 conn=4 myRSS=2245 php=0 phpRSS=0 ap=78 tcp=24 topuser=- 2026-08-13 23:12:03 L1=0.76 run=1 blk=0 avail=6864 (58.5%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=40 ap=73 tcp=21 topuser=evolution 2026-08-13 23:12:08 L1=0.78 run=2 blk=0 avail=6911 (58.9%) swpO=0 conn=4 myRSS=2245 php=0 phpRSS=0 ap=68 tcp=22 topuser=- 2026-08-13 23:12:13 L1=0.71 run=1 blk=0 avail=6942 (59.2%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=41 ap=63 tcp=22 topuser=evolution 2026-08-13 23:12:18 L1=0.66 run=1 blk=0 avail=6987 (59.6%) swpO=0 conn=4 myRSS=2245 php=0 phpRSS=0 ap=58 tcp=18 topuser=- 2026-08-13 23:12:23 L1=0.60 run=1 blk=0 avail=7008 (59.8%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=40 ap=53 tcp=20 topuser=evolution 2026-08-13 23:12:28 L1=0.64 run=2 blk=0 avail=6880 (58.7%) swpO=0 conn=4 myRSS=2245 php=2 phpRSS=236 ap=48 tcp=27 topuser=tubesupplies 2026-08-13 23:12:33 L1=0.82 run=3 blk=0 avail=6966 (59.4%) swpO=0 conn=2 myRSS=2245 php=2 phpRSS=184 ap=43 tcp=23 topuser=madalice 2026-08-13 23:12:38 L1=0.76 run=1 blk=0 avail=7127 (60.8%) swpO=0 conn=4 myRSS=2245 php=0 phpRSS=0 ap=38 tcp=22 topuser=- 2026-08-13 23:12:43 L1=0.70 run=1 blk=0 avail=7152 (61.0%) swpO=0 conn=2 myRSS=2245 php=1 phpRSS=41 ap=33 tcp=20 topuser=evolution 2026-08-13 23:12:48 L1=0.64 run=1 blk=0 avail=7200 (61.4%) swpO=0 conn=1 myRSS=2245 php=0 phpRSS=0 ap=28 tcp=21 topuser=- 2026-08-13 23:12:53 L1=0.67 run=5 blk=0 avail=7088 (60.4%) swpO=0 conn=2 myRSS=2245 php=4 phpRSS=281 ap=23 tcp=26 topuser=olympic 2026-08-13 23:12:58 L1=1.82 run=14 blk=0 avail=6233 (53.1%) swpO=0 conn=1 myRSS=2245 php=7 phpRSS=1129 ap=37 tcp=55 topuser=madalice 2026-08-13 23:13:03 L1=3.35 run=27 blk=0 avail=5129 (43.7%) swpO=215 conn=23 myRSS=2239 php=24 phpRSS=2244 ap=87 tcp=114 topuser=madalice 2026-08-13 23:13:08 L1=8.13 run=63 blk=1 avail=1628 (13.9%) swpO=7010 conn=60 myRSS=2104 php=60 phpRSS=7115 ap=114 tcp=110 topuser=madalice 2026-08-13 23:13:16 L1=14.53 run=38 blk=66 avail=383 (3.3%) swpO=41279 conn=66 myRSS=1667 php=67 phpRSS=9212 ap=123 tcp=81 topuser=madalice 2026-08-13 23:13:50 L1=72.30 run=134 blk=8 avail=132 (1.1%) swpO=6537 conn=- myRSS=0 php=87 phpRSS=5819 ap=110 tcp=41 topuser=madalice 2026-08-13 23:13:20 L1=22.09 run=87 blk=12 avail=243 (2.1%) swpO=55688 conn=- myRSS=1499 php=73 phpRSS=6781 ap=119 tcp=69 topuser=madalice 2026-08-13 23:15:43 L1=140.57 run=3 blk=1 avail=9081 (77.4%) swpO=511 conn=3 myRSS=456 php=2 phpRSS=276 ap=99 tcp=24 topuser=madalice 2026-08-13 23:15:43 L1=140.57 run=3 blk=2 avail=9042 (77.1%) swpO=478 conn=4 myRSS=456 php=3 phpRSS=341 ap=99 tcp=25 topuser=madalice