INCIDENT 2026-08-15 07:01:02 trigger: load=132.09/10cpu ================================================================== load: 132.09 59.32 23.02 1/685 2544977 ncpu:10 mem_avail: 9325MB/11729MB (79.5%) swap_used:1899MB swapout:788/s blocked(D):0 running:1 iowait:10.5% steal:0.0% php: 0 procs / 0MB heaviest user: - (0 procs, 0MB) apache: 103 procs tcp_estab: 21 --- RSS by user (all processes) --- USER PROCS RSS_MB nobody 102 1043 mysql 1 436 root 252 323 netdata 8 194 mailnull 7 45 cpanellogin 1 23 wp-toolkit 6 18 mailman 20 18 corpkicks 1 12 zabbix 1 11 dovenull 6 7 dovecot 3 6 kurri 4 5 cpanelconnecttrack 1 3 evolution 4 2 dbus 1 2 polkitd 1 1 nscd 1 1 named 1 1 rpc 1 0 --- php workers by user --- USER PROCS RSS_MB --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 752 MB Active(file): 311 MB Inactive(file): 361 MB Dirty: 1 MB Writeback: 0 MB SReclaimable: 144 MB CommitLimit: 9960 MB Committed_AS: 8285 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2342983 root 05:49:47 208 /usr/bin/imunify-agent-proxy 2544311 root 01:00 0 /bin/sh -c bash -c "sleep $((RANDOM % 1800))" ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 2544314 root 01:00 0 /bin/sh -c bash -c "sleep $((RANDOM % 60))" ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || : --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2544865 1 mysql 447336 28.0 Ssl 00:03 /usr/sbin/mariadbd 1220 1 netdata 150400 2.5 Ssl 46-20:51:05 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2382613 1 root 82984 0.0 Ss 04:37:51 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 159820 1 root 38588 0.0 Ss 25-21:14:13 /usr/lib/systemd/systemd-journald 4922 2431 netdata 30888 0.1 Sl 46-20:51:00 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2544948 1469 root 27556 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 2525318 1353308 cpanellogin 24216 0.1 S 32:50 /usr/local/cpanel/base/show_template.stor docroot /usr/local/cpanel/base cpanel_locale default_login_theme cpanel server_locale en 2390234 2382613 root 22324 0.0 S 04:25:25 spamd child 2223 1201 root 20184 0.0 Sl 46-20:51:04 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 2381704 1 root 18972 0.0 Ss 04:38:50 lfd - sleeping 2544189 1276780 nobody 18228 1.2 S 02:23 /usr/sbin/httpd -k start 2544240 1276780 nobody 18144 1.0 S 01:43 /usr/sbin/httpd -k start 2544232 1276780 nobody 17924 1.0 S 01:48 /usr/sbin/httpd -k start 2544142 1276780 nobody 17504 1.1 S 02:27 /usr/sbin/httpd -k start 185104 2431 netdata 16040 2.0 Sl 2-10:27:23 /usr/libexec/netdata/plugins.d/apps.plugin 1 2544219 1276780 nobody 15480 1.7 S 02:07 /usr/sbin/httpd -k start 2544258 1276780 nobody 15468 3.2 S 01:24 /usr/sbin/httpd -k start 2544246 1276780 nobody 15244 2.6 S 01:27 /usr/sbin/httpd -k start 2543967 1276780 nobody 15204 0.0 S 02:38 /usr/sbin/httpd -k start 2543968 1276780 nobody 15172 0.0 S 02:38 /usr/sbin/httpd -k start 2544010 1276780 nobody 15172 0.0 S 02:37 /usr/sbin/httpd -k start 2544008 1276780 nobody 15100 0.0 S 02:37 /usr/sbin/httpd -k start 2544009 1276780 nobody 14984 0.0 S 02:37 /usr/sbin/httpd -k start 2544186 1276780 nobody 14956 1.4 S 02:24 /usr/sbin/httpd -k start 2543957 1276780 nobody 14904 0.0 S 02:39 /usr/sbin/httpd -k start 2544046 1276780 nobody 14788 0.0 S 02:36 /usr/sbin/httpd -k start 2544049 1276780 nobody 14752 0.0 S 02:36 /usr/sbin/httpd -k start 2544011 1276780 nobody 14732 0.0 S 02:37 /usr/sbin/httpd -k start 2544047 1276780 nobody 14668 0.0 S 02:36 /usr/sbin/httpd -k start 2544052 1276780 nobody 14644 0.0 S 02:36 /usr/sbin/httpd -k start 2544048 1276780 nobody 14616 0.0 S 02:36 /usr/sbin/httpd -k start 2544051 1276780 nobody 14580 0.1 S 02:36 /usr/sbin/httpd -k start 2544050 1276780 nobody 14572 0.0 S 02:36 /usr/sbin/httpd -k start 2541800 1276780 nobody 14564 0.9 S 05:46 /usr/sbin/httpd -k start 1469 1 root 14312 0.1 Ss 46-20:51:04 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 2543876 1276780 nobody 14100 0.0 S 02:46 /usr/sbin/httpd -k start 2544163 1276780 nobody 14056 1.3 S 02:26 /usr/sbin/httpd -k start 2544162 1276780 nobody 14048 1.6 S 02:26 /usr/sbin/httpd -k start 2544187 1276780 nobody 13592 1.2 S 02:24 /usr/sbin/httpd -k start 2544045 1276780 nobody 13468 0.0 S 02:36 /usr/sbin/httpd -k start 2543523 1276780 nobody 13240 1.3 S 02:52 /usr/sbin/httpd -k start 2544791 3828812 corpkicks 12580 0.0 S 00:06 dovecot/imap 2543917 1276780 nobody 12384 0.0 S 02:43 /usr/sbin/httpd -k start 2544792 1471 root 12348 0.3 S 00:06 cPhulkd - processor - http socket 2544266 1276780 nobody 12064 1.5 S 01:20 /usr/sbin/httpd -k start 2543557 1276780 nobody 12052 0.0 S 02:50 /usr/sbin/httpd -k start 2543553 1276780 nobody 11984 0.0 S 02:50 /usr/sbin/httpd -k start 1204 1 zabbix 11892 0.2 Ssl 46-20:51:05 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 2544329 1276780 nobody 11160 1.8 S 00:31 /usr/sbin/httpd -k start 2543524 1276780 nobody 10652 0.0 S 02:52 /usr/sbin/httpd -k start 1601939 1471 root 10452 0.0 S 23:48:05 cPhulkd - dbprocessor 2540706 1276780 nobody 10044 0.0 S 06:23 /usr/sbin/httpd -k start 2539124 1276780 nobody 9932 0.0 S 09:07 /usr/sbin/httpd -k start 2543855 1276780 nobody 9860 0.0 S 02:47 /usr/sbin/httpd -k start 2543898 1276780 nobody 9776 0.0 S 02:44 /usr/sbin/httpd -k start 1463 1 root 9700 0.0 Ss 46-20:51:04 tailwatchd 2543922 1276780 nobody 9616 0.0 S 02:43 /usr/sbin/httpd -k start 2543914 1276780 nobody 9612 0.0 S 02:43 /usr/sbin/httpd -k start 2543567 1276780 nobody 9544 0.0 S 02:50 /usr/sbin/httpd -k start 2543559 1276780 nobody 9512 0.0 S 02:50 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 47 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min madalice/madalice.com.au-ssl_log 14 (total vhosts taking traffic: 1) --- top client IPs across all vhosts (last 3 min) --- 1 54.39.203.37 1 54.156.248.117 1 51.39.16.149 1 43.119.100.73 1 41.90.136.245 1 40.77.178.142 1 37.220.124.5 1 212.112.119.233 1 201.80.98.22 1 197.121.140.64 1 191.52.219.19 1 181.110.53.183 1 170.233.199.46 1 103.85.117.1 --- top request paths across all vhosts (last 3 min) --- 4 POST / 1 GET /wp-content/plugins/woo-custom-product-addons-pro/assets/js/backend/ 1 GET /tag/flowes/ 1 GET /tag/children/ 1 GET /store/personalised-boot-bows-blue/ 1 GET /store/hand-dyed-crew-socks-gift-pack-festive-christmas-edition/ 1 GET /store/baby-crocs/ 1 GET /store/af1-pink-hearts/ 1 GET /custom-sneakers-australia/ 1 GET /custom-hand-painted-shoes-blog/page/2/ 1 GET / --- busiest 3 vhosts: top paths + last 10 requests --- == madalice/madalice.com.au-ssl_log == -- top paths -- 4 POST / 1 GET /wp-content/plugins/woo-custom-product-addons-pro/assets/js/backend/ 1 GET /tag/flowes/ 1 GET /tag/children/ 1 GET /store/personalised-boot-bows-blue/ 1 GET /store/hand-dyed-crew-socks-gift-pack-festive-christmas-edition/ 1 GET /store/baby-crocs/ 1 GET /store/af1-pink-hearts/ 1 GET /custom-sneakers-australia/ 1 GET /custom-hand-painted-shoes-blog/page/2/ 1 GET / -- last 10 requests -- 43.119.100.73 - - [15/Aug/2026:06:59:46 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 500 833 "https://madalice.com.au/store/custom-jordans-turtles/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Appl 54.156.248.117 - - [15/Aug/2026:07:00:52 +1000] "GET /wp-content/plugins/woo-custom-product-addons-pro/assets/js/backend/?C=N;O%3DD HTTP/1.1" 200 431 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Am 54.39.203.37 - - [15/Aug/2026:07:00:51 +1000] "GET /store/personalised-boot-bows-blue/ HTTP/1.1" 500 833 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 197.121.140.64 - - [15/Aug/2026:06:59:40 +1000] "GET /custom-sneakers-australia/ HTTP/1.1" 500 833 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.3 191.52.219.19 - - [15/Aug/2026:06:59:57 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 400 - "https://madalice.com.au/store/custom-jordans-turtles/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/ 170.233.199.46 - - [15/Aug/2026:06:59:40 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 400 - "https://madalice.com.au/product-category/process/crystals/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleW 181.110.53.183 - - [15/Aug/2026:06:59:46 +1000] "GET /custom-hand-painted-shoes-blog/page/2/ HTTP/1.1" 500 833 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 37.220.124.5 - - [15/Aug/2026:06:59:29 +1000] "GET /tag/children/ HTTP/1.1" 500 833 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" 201.80.98.22 - - [15/Aug/2026:06:59:17 +1000] "GET /store/baby-crocs/?srsltid=AfmBOorgG8xEUs4g7LNHZEY3KD-5760qigTR2kxh8mo2H7XUvSnVTPp1 HTTP/1.1" 500 833 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/53 40.77.178.142 - - [15/Aug/2026:07:00:58 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 500 833 "https://madalice.com.au/custom-sneakers-australia-2/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatib --- last 30 samples (run-up to this incident) --- 2026-08-15 06:56:25 L1=3.09 run=1 blk=0 avail=6734 (57.4%) swpO=0 conn=2 myRSS=2162 php=0 phpRSS=0 ap=86 tcp=33 topuser=- 2026-08-15 06:56:33 L1=2.84 run=2 blk=0 avail=6749 (57.5%) swpO=0 conn=2 myRSS=2162 php=1 phpRSS=67 ap=78 tcp=32 topuser=alison 2026-08-15 06:56:35 L1=2.69 run=1 blk=0 avail=6804 (58.0%) swpO=0 conn=2 myRSS=2162 php=0 phpRSS=0 ap=76 tcp=33 topuser=- 2026-08-15 06:56:43 L1=2.48 run=1 blk=0 avail=6857 (58.5%) swpO=0 conn=2 myRSS=2162 php=0 phpRSS=0 ap=68 tcp=30 topuser=- 2026-08-15 06:56:45 L1=2.28 run=1 blk=0 avail=6873 (58.6%) swpO=0 conn=2 myRSS=2162 php=0 phpRSS=0 ap=66 tcp=29 topuser=- 2026-08-15 06:56:53 L1=2.10 run=1 blk=0 avail=6932 (59.1%) swpO=0 conn=1 myRSS=2162 php=0 phpRSS=0 ap=58 tcp=31 topuser=- 2026-08-15 06:56:55 L1=2.01 run=2 blk=0 avail=6715 (57.3%) swpO=0 conn=2 myRSS=2162 php=1 phpRSS=260 ap=56 tcp=36 topuser=madalice 2026-08-15 06:57:03 L1=1.85 run=3 blk=0 avail=6998 (59.7%) swpO=0 conn=1 myRSS=2162 php=1 phpRSS=40 ap=48 tcp=37 topuser=evolution 2026-08-15 06:57:05 L1=1.86 run=1 blk=0 avail=6861 (58.5%) swpO=0 conn=3 myRSS=2162 php=2 phpRSS=218 ap=46 tcp=39 topuser=tubesupplies 2026-08-15 06:57:13 L1=1.71 run=3 blk=1 avail=6683 (57.0%) swpO=0 conn=1 myRSS=2162 php=5 phpRSS=531 ap=38 tcp=37 topuser=infinium 2026-08-15 06:57:15 L1=1.73 run=1 blk=0 avail=6992 (59.6%) swpO=0 conn=3 myRSS=2162 php=1 phpRSS=80 ap=36 tcp=34 topuser=infinium 2026-08-15 06:57:23 L1=1.60 run=1 blk=0 avail=7105 (60.6%) swpO=0 conn=1 myRSS=2162 php=1 phpRSS=40 ap=30 tcp=39 topuser=evolution 2026-08-15 06:57:25 L1=1.47 run=1 blk=0 avail=7116 (60.7%) swpO=0 conn=3 myRSS=2162 php=0 phpRSS=0 ap=30 tcp=36 topuser=- 2026-08-15 06:57:33 L1=1.35 run=1 blk=0 avail=7139 (60.9%) swpO=0 conn=1 myRSS=2162 php=1 phpRSS=39 ap=28 tcp=32 topuser=evolution 2026-08-15 06:57:35 L1=1.24 run=1 blk=0 avail=7166 (61.1%) swpO=0 conn=3 myRSS=2162 php=0 phpRSS=0 ap=26 tcp=31 topuser=- 2026-08-15 06:57:43 L1=1.14 run=1 blk=0 avail=7157 (61.0%) swpO=0 conn=1 myRSS=2162 php=1 phpRSS=40 ap=25 tcp=31 topuser=evolution 2026-08-15 06:57:45 L1=1.05 run=2 blk=0 avail=7176 (61.2%) swpO=0 conn=3 myRSS=2162 php=1 phpRSS=10 ap=23 tcp=32 topuser=evolution 2026-08-15 06:57:53 L1=1.05 run=3 blk=0 avail=7162 (61.1%) swpO=0 conn=2 myRSS=2162 php=1 phpRSS=40 ap=23 tcp=32 topuser=evolution 2026-08-15 06:57:55 L1=1.04 run=3 blk=0 avail=6966 (59.4%) swpO=0 conn=3 myRSS=2162 php=2 phpRSS=270 ap=23 tcp=32 topuser=madalice 2026-08-15 06:58:03 L1=0.96 run=1 blk=0 avail=7185 (61.3%) swpO=0 conn=2 myRSS=2162 php=1 phpRSS=40 ap=20 tcp=29 topuser=evolution 2026-08-15 06:58:05 L1=0.88 run=1 blk=1 avail=7183 (61.2%) swpO=0 conn=2 myRSS=2162 php=1 phpRSS=40 ap=20 tcp=29 topuser=evolution 2026-08-15 06:58:13 L1=1.45 run=16 blk=0 avail=5856 (49.9%) swpO=0 conn=12 myRSS=2162 php=12 phpRSS=1627 ap=51 tcp=66 topuser=madalice 2026-08-15 06:58:16 L1=2.38 run=25 blk=0 avail=6084 (51.9%) swpO=0 conn=15 myRSS=2162 php=18 phpRSS=1505 ap=51 tcp=98 topuser=madalice 2026-08-15 06:58:25 L1=7.44 run=47 blk=0 avail=3254 (27.7%) swpO=892 conn=39 myRSS=2156 php=42 phpRSS=4781 ap=81 tcp=117 topuser=madalice 2026-08-15 06:58:26 L1=7.44 run=54 blk=0 avail=2647 (22.6%) swpO=1070 conn=47 myRSS=2157 php=48 phpRSS=5442 ap=84 tcp=120 topuser=madalice 2026-08-15 06:58:35 L1=18.02 run=49 blk=66 avail=380 (3.2%) swpO=37984 conn=72 myRSS=1653 php=73 phpRSS=9365 ap=102 tcp=77 topuser=madalice 2026-08-15 06:58:37 L1=18.02 run=46 blk=93 avail=385 (3.3%) swpO=38323 conn=- myRSS=1697 php=73 phpRSS=9307 ap=102 tcp=75 topuser=madalice 2026-08-15 06:59:52 L1=114.71 run=98 blk=4 avail=60 (0.5%) swpO=4667 conn=- myRSS=0 php=94 phpRSS=5899 ap=114 tcp=43 topuser=madalice 2026-08-15 07:00:00 L1=118.42 run=159 blk=4 avail=51 (0.4%) swpO=4421 conn=- myRSS=0 php=76 phpRSS=5895 ap=112 tcp=20 topuser=madalice 2026-08-15 07:01:02 L1=132.09 run=1 blk=0 avail=9325 (79.5%) swpO=788 conn=1 myRSS=436 php=0 phpRSS=0 ap=103 tcp=21 topuser=-