INCIDENT 2026-08-15 07:51:52 trigger: load=97.42/10cpu ================================================================== load: 97.42 167.37 93.99 1/694 2576901 ncpu:10 mem_avail: 9323MB/11729MB (79.5%) swap_used:1726MB swapout:0/s blocked(D):0 running:1 iowait:0.3% steal:0.0% php: 0 procs / 0MB heaviest user: - (0 procs, 0MB) apache: 74 procs tcp_estab: 13 --- RSS by user (all processes) --- USER PROCS RSS_MB nobody 73 936 mysql 1 518 root 247 263 netdata 8 201 mailman 20 25 wp-toolkit 6 18 dovenull 6 13 zabbix 1 10 dovecot 3 7 evolution 4 3 dbus 1 3 cpanelconnecttrack 1 3 polkitd 1 2 nscd 1 1 named 1 1 mailnull 1 1 kurri 4 1 cpanellogin 1 1 rpc 1 0 chrony 1 0 --- php workers by user --- USER PROCS RSS_MB --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 947 MB Active(file): 681 MB Inactive(file): 385 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 156 MB CommitLimit: 9960 MB Committed_AS: 7438 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2342983 root 06:40:36 236 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2575700 1 mysql 531080 7.7 Ssl 01:12 /usr/sbin/mariadbd 1220 1 netdata 157160 2.5 Ssl 46-21:41:54 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2390233 2382613 root 107512 0.0 S 05:16:14 spamd child 4922 2431 netdata 29808 0.1 Sl 46-21:41:49 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2382613 1 root 23232 0.1 Ss 05:28:40 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 2575287 1276780 nobody 18908 0.1 S 01:18 /usr/sbin/httpd -k start 2575409 1276780 nobody 18492 0.0 S 01:15 /usr/sbin/httpd -k start 2574887 1276780 nobody 18180 1.6 S 08:07 /usr/sbin/httpd -k start 2575407 1276780 nobody 18092 0.0 S 01:15 /usr/sbin/httpd -k start 2574830 1276780 nobody 18056 0.3 S 08:13 /usr/sbin/httpd -k start 2575562 1276780 nobody 18040 0.0 S 01:13 /usr/sbin/httpd -k start 2575557 1276780 nobody 18020 0.0 S 01:13 /usr/sbin/httpd -k start 2575340 1276780 nobody 17988 0.0 S 01:16 /usr/sbin/httpd -k start 2574831 1276780 nobody 17864 0.3 S 08:13 /usr/sbin/httpd -k start 2574865 1276780 nobody 17708 0.4 S 08:09 /usr/sbin/httpd -k start 2574904 1276780 nobody 17680 1.8 S 08:05 /usr/sbin/httpd -k start 2575410 1276780 nobody 17552 0.0 S 01:15 /usr/sbin/httpd -k start 2575055 1276780 nobody 17516 1.3 S 06:40 /usr/sbin/httpd -k start 2574631 1276780 nobody 17424 0.5 S 08:19 /usr/sbin/httpd -k start 2574630 1276780 nobody 17156 0.3 S 08:19 /usr/sbin/httpd -k start 2575408 1276780 nobody 17080 0.0 S 01:15 /usr/sbin/httpd -k start 2574829 1276780 nobody 16936 0.0 S 08:13 /usr/sbin/httpd -k start 2574747 1276780 nobody 16896 0.3 S 08:15 /usr/sbin/httpd -k start 185104 2431 netdata 16652 2.0 Sl 2-11:18:12 /usr/libexec/netdata/plugins.d/apps.plugin 1 2574629 1276780 nobody 16220 0.4 S 08:19 /usr/sbin/httpd -k start 2575050 1276780 nobody 16024 1.0 S 06:57 /usr/sbin/httpd -k start 2575542 1276780 nobody 15884 0.0 S 01:13 /usr/sbin/httpd -k start 2574709 1276780 nobody 15796 0.5 S 08:16 /usr/sbin/httpd -k start 2574626 1276780 nobody 15712 0.2 S 08:19 /usr/sbin/httpd -k start 2574627 1276780 nobody 15676 0.2 S 08:19 /usr/sbin/httpd -k start 2575561 1276780 nobody 15644 0.0 S 01:13 /usr/sbin/httpd -k start 2575541 1276780 nobody 15488 0.0 S 01:13 /usr/sbin/httpd -k start 2574899 1276780 nobody 15428 1.4 S 08:05 /usr/sbin/httpd -k start 2574799 1276780 nobody 15164 0.2 S 08:14 /usr/sbin/httpd -k start 2574800 1276780 nobody 14988 0.3 S 08:14 /usr/sbin/httpd -k start 2575054 1276780 nobody 14620 1.2 S 06:49 /usr/sbin/httpd -k start 2574900 1276780 nobody 14564 2.2 S 08:05 /usr/sbin/httpd -k start 2223 1201 root 14312 0.0 Sl 46-21:41:53 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 2575339 1276780 nobody 14244 0.0 S 01:16 /usr/sbin/httpd -k start 2575341 1276780 nobody 14244 0.0 S 01:16 /usr/sbin/httpd -k start 2575528 1276780 nobody 13636 0.0 S 01:13 /usr/sbin/httpd -k start 2575558 1276780 nobody 13568 0.0 S 01:13 /usr/sbin/httpd -k start 2573990 1276780 nobody 12840 0.4 S 08:40 /usr/sbin/httpd -k start 2574108 1276780 nobody 12636 0.4 S 08:32 /usr/sbin/httpd -k start 2574067 1276780 nobody 12076 0.5 S 08:35 /usr/sbin/httpd -k start 1353308 1 root 12072 0.0 Ss 24-05:10:24 cpsrvd (SSL) - waiting for connections 2574016 1276780 nobody 11876 0.2 S 08:37 /usr/sbin/httpd -k start 2571805 1276780 nobody 11808 0.4 S 12:05 /usr/sbin/httpd -k start 2574297 1276780 nobody 11752 0.5 S 08:29 /usr/sbin/httpd -k start 2573946 1276780 nobody 11508 0.4 S 08:42 /usr/sbin/httpd -k start 2574062 1276780 nobody 11456 0.4 S 08:35 /usr/sbin/httpd -k start 2571044 1276780 nobody 11332 0.3 S 13:06 /usr/sbin/httpd -k start 2573665 1276780 nobody 11232 0.4 S 09:12 /usr/sbin/httpd -k start 2574299 1276780 nobody 11212 0.1 S 08:29 /usr/sbin/httpd -k start 2574063 1276780 nobody 11200 0.6 S 08:35 /usr/sbin/httpd -k start 2574294 1276780 nobody 11200 0.5 S 08:29 /usr/sbin/httpd -k start 2574068 1276780 nobody 11188 0.2 S 08:35 /usr/sbin/httpd -k start 2573940 1276780 nobody 11176 0.0 S 08:43 /usr/sbin/httpd -k start 2574598 1276780 nobody 11044 0.5 S 08:20 /usr/sbin/httpd -k start 1204 1 zabbix 11012 0.2 Ssl 46-21:41:54 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +-----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +-----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 154 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +-----+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min olympic/olympicfencing.com.au-ssl_log 64 customkidscom/customkids.com.au 43 madalice/madalice.com.au-ssl_log 29 evolution/portal.evolutionerp.com.au-ssl_log 17 evolution/my.evolutionerp.com.au-ssl_log 3 (total vhosts taking traffic: 5) --- top client IPs across all vhosts (last 3 min) --- 64 135.225.75.187 43 172.213.18.162 17 203.54.180.238 3 157.55.39.52 2 42.116.206.165 2 191.7.94.223 2 138.255.238.139 2 1.10.247.160 1 93.215.147.8 1 84.104.44.110 1 5.199.239.199 1 47.82.202.197 1 47.82.201.80 1 45.163.75.189 1 43.119.104.190 --- top request paths across all vhosts (last 3 min) --- 17 GET /heartbeat.php 5 GET / 4 POST / 3 GET /index.php 2 GET /wp-configs.php 1 GET /yellow.php 1 GET /xxxx.php 1 GET /xxx.php 1 GET /xxw.php 1 GET /xx.php 1 GET /x.php 1 GET /xeqdlun.php 1 GET /x7.php 1 GET /wsd.php 1 GET /ws36.php 1 GET /--wp-lgj.php 1 GET /wp-l0gin.php 1 GET /wp-kd4xalrg7m.php 1 GET /wp-kaori.php 1 GET /wp-is.php --- busiest 3 vhosts: top paths + last 10 requests --- == olympic/olympicfencing.com.au-ssl_log == -- top paths -- 1 GET /yellow.php 1 GET /xxxx.php 1 GET /xxx.php 1 GET /xxw.php 1 GET /xx.php 1 GET /x.php 1 GET /xeqdlun.php 1 GET /x7.php 1 GET /wsd.php 1 GET /ws36.php 1 GET /wp-is.php 1 GET /wp-configs.php -- last 10 requests -- 135.225.75.187 - - [15/Aug/2026:07:51:18 +1000] "GET /0x.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:18 +1000] "GET /file1.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:19 +1000] "GET /file8.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:20 +1000] "GET /100.kb.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:20 +1000] "GET /ppp.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:21 +1000] "GET /tox.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:21 +1000] "GET /bless11.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:22 +1000] "GET /030.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:23 +1000] "GET /v4.php HTTP/1.1" 401 36 "-" "-" 135.225.75.187 - - [15/Aug/2026:07:51:23 +1000] "GET /xxx.php HTTP/1.1" 401 36 "-" "-" == customkidscom/customkids.com.au == -- top paths -- 1 GET /--wp-lgj.php 1 GET /wp-l0gin.php 1 GET /wp-kd4xalrg7m.php 1 GET /wp-kaori.php 1 GET /wp-home.php 1 GET /wp-head.php 1 GET /wp-header.php 1 GET /wpgum.php 1 GET /wp-good3.php 1 GET /wp-god.Php 1 GET /wp-fmfile.php 1 GET /wp-files.php -- last 10 requests -- 172.213.18.162 - - [15/Aug/2026:07:51:39 +1000] "GET /wp-god.Php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:40 +1000] "GET /wp-good3.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:40 +1000] "GET /wpgum.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:40 +1000] "GET /wp-head.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:40 +1000] "GET /wp-header.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:41 +1000] "GET /wp-home.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:41 +1000] "GET /wp-kaori.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:41 +1000] "GET /wp-kd4xalrg7m.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:42 +1000] "GET /wp-l0gin.php HTTP/1.1" 404 - "-" "-" 172.213.18.162 - - [15/Aug/2026:07:51:42 +1000] "GET /--wp-lgj.php HTTP/1.1" 404 - "-" "-" == madalice/madalice.com.au-ssl_log == -- top paths -- 5 GET / 4 POST / 1 GET /tag/packs/ 1 GET /tag/brown/ 1 GET /tag/blocks/ 1 GET /tag/beige/ 1 GET /store/sprinkles-kids-custom-shoes/ 1 GET /store/nike-gold-eyelets/ 1 GET /store/dont-be-a-goose-custom-nike-air-force-1/ 1 GET /store/colour-choice-cartoon-drippy-custom-shoes/ 1 GET /store/choc-donut/ 1 GET /store/af1-blue-glitter-swoosh/ -- last 10 requests -- 42.116.206.165 - - [15/Aug/2026:07:50:53 +1000] "GET / HTTP/1.1" 200 36567 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" 138.255.238.139 - - [15/Aug/2026:07:50:54 +1000] "GET /tag/beige/ HTTP/1.1" 200 28893 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36" 47.82.202.197 - - [15/Aug/2026:07:50:59 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/tag/blocks/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHT 1.10.247.160 - - [15/Aug/2026:07:50:59 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/tag/blocks/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, li 43.119.104.190 - - [15/Aug/2026:07:51:00 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/tag/brown/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHT 191.7.94.223 - - [15/Aug/2026:07:51:00 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/tag/brown/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, lik 42.116.206.165 - - [15/Aug/2026:07:51:00 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) 43.119.104.156 - - [15/Aug/2026:07:51:00 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/tag/beige/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHT 47.82.201.80 - - [15/Aug/2026:07:51:00 +1000] "GET /?commercekit-ajax=commercekit_get_nonce HTTP/1.1" 200 37 "https://madalice.com.au/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gec 138.255.238.139 - - [15/Aug/2026:07:51:01 +1000] "POST /?wc-ajax=get_refreshed_fragments HTTP/1.1" 200 578 "https://madalice.com.au/tag/beige/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, --- last 30 samples (run-up to this incident) --- 2026-08-15 07:42:36 L1=1.17 run=2 blk=0 avail=9429 (80.4%) swpO=0 conn=2 myRSS=546 php=2 phpRSS=117 ap=17 tcp=21 topuser=olympic 2026-08-15 07:42:40 L1=1.23 run=2 blk=0 avail=9484 (80.9%) swpO=0 conn=1 myRSS=546 php=0 phpRSS=0 ap=18 tcp=19 topuser=- 2026-08-15 07:42:46 L1=1.13 run=1 blk=0 avail=9460 (80.7%) swpO=0 conn=2 myRSS=546 php=2 phpRSS=69 ap=21 tcp=20 topuser=evolution 2026-08-15 07:42:50 L1=1.04 run=1 blk=0 avail=9467 (80.7%) swpO=0 conn=1 myRSS=546 php=0 phpRSS=0 ap=20 tcp=24 topuser=- 2026-08-15 07:42:56 L1=1.04 run=2 blk=0 avail=9420 (80.3%) swpO=0 conn=2 myRSS=547 php=1 phpRSS=40 ap=20 tcp=24 topuser=evolution 2026-08-15 07:43:00 L1=0.96 run=1 blk=0 avail=9430 (80.4%) swpO=0 conn=1 myRSS=547 php=0 phpRSS=0 ap=20 tcp=24 topuser=- 2026-08-15 07:43:06 L1=0.88 run=1 blk=0 avail=9424 (80.3%) swpO=0 conn=2 myRSS=547 php=1 phpRSS=40 ap=19 tcp=24 topuser=evolution 2026-08-15 07:43:10 L1=1.37 run=10 blk=0 avail=8623 (73.5%) swpO=0 conn=1 myRSS=547 php=7 phpRSS=988 ap=26 tcp=41 topuser=madalice 2026-08-15 07:43:16 L1=1.98 run=12 blk=0 avail=8535 (72.8%) swpO=0 conn=11 myRSS=549 php=11 phpRSS=1078 ap=41 tcp=85 topuser=madalice 2026-08-15 07:43:21 L1=3.10 run=30 blk=0 avail=7095 (60.5%) swpO=0 conn=22 myRSS=549 php=21 phpRSS=2763 ap=52 tcp=117 topuser=madalice 2026-08-15 07:43:27 L1=5.50 run=54 blk=0 avail=4744 (40.4%) swpO=0 conn=40 myRSS=551 php=44 phpRSS=5517 ap=71 tcp=155 topuser=madalice 2026-08-15 07:43:34 L1=10.34 run=68 blk=0 avail=2706 (23.1%) swpO=38 conn=58 myRSS=558 php=59 phpRSS=7958 ap=87 tcp=186 topuser=madalice 2026-08-15 07:43:37 L1=15.36 run=79 blk=0 avail=1634 (13.9%) swpO=505 conn=67 myRSS=561 php=74 phpRSS=9131 ap=89 tcp=172 topuser=madalice 2026-08-15 07:43:44 L1=27.28 run=69 blk=24 avail=383 (3.3%) swpO=15506 conn=79 myRSS=563 php=85 phpRSS=11291 ap=108 tcp=156 topuser=madalice 2026-08-15 07:43:48 L1=27.28 run=20 blk=81 avail=259 (2.2%) swpO=42058 conn=85 myRSS=532 php=90 phpRSS=11353 ap=119 tcp=154 topuser=madalice 2026-08-15 07:44:28 L1=104.09 run=226 blk=2 avail=54 (0.5%) swpO=8286 conn=- myRSS=0 php=101 phpRSS=4346 ap=149 tcp=96 topuser=madalice 2026-08-15 07:44:26 L1=95.66 run=224 blk=30 avail=41 (0.4%) swpO=16065 conn=- myRSS=0 php=102 phpRSS=4390 ap=149 tcp=96 topuser=madalice 2026-08-15 07:50:51 L1=260.46 run=46 blk=2 avail=2932 (25.0%) swpO=166 conn=42 myRSS=475 php=40 phpRSS=7151 ap=135 tcp=19 topuser=madalice 2026-08-15 07:50:51 L1=260.46 run=42 blk=1 avail=2833 (24.2%) swpO=165 conn=41 myRSS=475 php=40 phpRSS=7250 ap=135 tcp=19 topuser=madalice 2026-08-15 07:51:01 L1=223.88 run=12 blk=0 avail=8006 (68.3%) swpO=62 conn=9 myRSS=501 php=8 phpRSS=1300 ap=125 tcp=29 topuser=madalice 2026-08-15 07:51:04 L1=223.88 run=4 blk=0 avail=9069 (77.3%) swpO=47 conn=1 myRSS=503 php=0 phpRSS=0 ap=122 tcp=27 topuser=- 2026-08-15 07:51:11 L1=189.62 run=5 blk=1 avail=8498 (72.5%) swpO=0 conn=6 myRSS=515 php=5 phpRSS=739 ap=115 tcp=22 topuser=corpkicks 2026-08-15 07:51:14 L1=189.62 run=2 blk=1 avail=9096 (77.6%) swpO=0 conn=1 myRSS=515 php=0 phpRSS=0 ap=112 tcp=21 topuser=- 2026-08-15 07:51:21 L1=160.46 run=1 blk=0 avail=9132 (77.9%) swpO=0 conn=1 myRSS=515 php=0 phpRSS=0 ap=105 tcp=14 topuser=- 2026-08-15 07:51:24 L1=160.46 run=1 blk=0 avail=9142 (77.9%) swpO=0 conn=1 myRSS=515 php=0 phpRSS=0 ap=102 tcp=18 topuser=- 2026-08-15 07:51:32 L1=136.03 run=2 blk=0 avail=9155 (78.1%) swpO=0 conn=2 myRSS=516 php=1 phpRSS=68 ap=95 tcp=17 topuser=alison 2026-08-15 07:51:34 L1=136.03 run=1 blk=1 avail=9076 (77.4%) swpO=0 conn=2 myRSS=516 php=1 phpRSS=155 ap=92 tcp=16 topuser=alison 2026-08-15 07:51:42 L1=115.12 run=2 blk=0 avail=9248 (78.8%) swpO=0 conn=1 myRSS=518 php=0 phpRSS=0 ap=84 tcp=17 topuser=- 2026-08-15 07:51:44 L1=115.12 run=1 blk=0 avail=9265 (79.0%) swpO=0 conn=1 myRSS=518 php=0 phpRSS=0 ap=82 tcp=17 topuser=- 2026-08-15 07:51:52 L1=97.42 run=1 blk=0 avail=9323 (79.5%) swpO=0 conn=1 myRSS=518 php=0 phpRSS=0 ap=74 tcp=13 topuser=-