INCIDENT 2026-09-03 02:39:04 trigger: php_rss=1626MB ================================================================== load: 2.61 1.63 1.29 4/744 1597554 ncpu:10 mem_avail: 7883MB/13777MB (57.2%) swap_used:1558MB swapout:0/s blocked(D):0 running:2 iowait:1.1% steal:0.0% php: 12 procs / 1626MB heaviest user: madalice (10 procs, 1589MB) apache: 78 procs tcp_estab: 37 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2119 madalice 10 1589 nobody 77 1297 root 251 1063 netdata 8 306 wp-toolkit 2 127 kurri 3 27 mailman 20 26 dovenull 6 26 lowrange 2 24 infinium 2 23 polkitd 1 22 cpanellogin 1 18 zabbix 1 13 aaa 1 13 named 1 7 dovecot 3 6 nscd 1 4 evolution 8 4 cpanelconnecttrack 1 3 --- php workers by user --- USER PROCS RSS_MB madalice 10 1589 root 1 36 evolution 1 0 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5578 MB Active(file): 3767 MB Inactive(file): 2257 MB Dirty: 2 MB Writeback: 0 MB SReclaimable: 606 MB CommitLimit: 10984 MB Committed_AS: 9214 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 3926885 root 2-00:16:37 5216 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 2169956 8.8 Ssl 12:24:52 /usr/sbin/mariadbd 1543747 1 root 175084 0.4 Ss 16:12 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 1542443 1 netdata 169440 2.7 Ssl 16:42 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 1596047 1543747 root 169288 0.0 S 01:17 spamd child 1596048 1543747 root 167752 0.0 S 01:17 spamd child 1595994 1542510 madalice 163404 14.5 S 01:22 php-fpm: pool madalice_com_au 1597376 1542510 madalice 163068 32.4 S 00:12 php-fpm: pool madalice_com_au 1597371 1542510 madalice 162964 31.5 S 00:12 php-fpm: pool madalice_com_au 1597374 1542510 madalice 162824 30.8 S 00:12 php-fpm: pool madalice_com_au 1597375 1542510 madalice 162792 24.3 R 00:12 php-fpm: pool madalice_com_au 1597378 1542510 madalice 162784 25.5 S 00:12 php-fpm: pool madalice_com_au 1597370 1542510 madalice 162744 25.4 S 00:12 php-fpm: pool madalice_com_au 1597373 1542510 madalice 162692 22.1 S 00:12 php-fpm: pool madalice_com_au 1597372 1542510 madalice 162424 24.6 S 00:12 php-fpm: pool madalice_com_au 1597377 1542510 madalice 162412 26.4 S 00:12 php-fpm: pool madalice_com_au 1542981 1542551 netdata 95176 0.1 Sl 16:38 /usr/libexec/netdata/plugins.d/go.d.plugin 1 1542487 1 wp-toolkit 77412 0.0 Ss 16:42 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 509236 509098 root 65784 0.0 Sl 6-00:19:23 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 509266 1 root 56316 0.0 Ss 6-00:19:23 /usr/lib/systemd/systemd-journald 1542531 1 wp-toolkit 52764 0.0 Ss 16:41 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script scheduled-tasks-executor.php 1542141 1 root 43208 0.0 Ss 16:45 sw-engine-fpm: master process (/etc/sw-engine/sw-engine-fpm.conf) 1353308 1 root 41728 0.1 Ss 42-23:57:49 cpsrvd (SSL) - waiting for connections 1469 1 root 40364 0.1 Ss 65-16:29:18 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1542510 1 root 37636 0.0 Ss 16:41 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 1542072 1 root 36152 0.0 Ss 16:46 lfd - sleeping 1542188 1 root 29284 0.1 Ssl 16:44 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P 1463 1 root 25380 0.0 Ss 65-16:29:18 tailwatchd 1542555 1 polkitd 23520 0.0 Ssl 16:41 /usr/lib/polkit-1/polkitd --no-debug 1597492 1597489 root 22680 3.0 D 00:04 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/clean_user_php_sessions 1577629 506832 nobody 22028 0.0 S 08:22 /usr/sbin/httpd -k start 1594759 506832 nobody 21972 0.0 S 03:21 /usr/sbin/httpd -k start 1577627 506832 nobody 20196 0.0 S 08:22 /usr/sbin/httpd -k start 1594752 506832 nobody 19652 0.2 S 03:22 /usr/sbin/httpd -k start 1581560 506832 nobody 19644 0.0 S 04:02 /usr/sbin/httpd -k start 1596205 506832 nobody 19488 0.0 S 01:15 /usr/sbin/httpd -k start 1596322 506832 nobody 19436 0.0 S 01:13 /usr/sbin/httpd -k start 1596303 506832 nobody 19428 0.0 S 01:13 /usr/sbin/httpd -k start 1507769 1353308 cpanellogin 19368 0.0 S 25:49 /usr/local/cpanel/base/show_template.stor default_login_theme cpanel server_locale en docroot /usr/local/cpanel/base cpanel_locale 1596107 506832 nobody 19364 0.0 S 01:17 /usr/sbin/httpd -k start 1596220 506832 nobody 19124 0.0 S 01:15 /usr/sbin/httpd -k start 1596044 506832 nobody 19104 0.0 S 01:18 /usr/sbin/httpd -k start 1596138 506832 nobody 19040 0.0 S 01:16 /usr/sbin/httpd -k start 1596106 506832 nobody 18948 0.0 S 01:17 /usr/sbin/httpd -k start 1596254 506832 nobody 18884 0.0 S 01:14 /usr/sbin/httpd -k start 1596128 506832 nobody 18820 0.0 S 01:16 /usr/sbin/httpd -k start 1595854 506832 nobody 18716 0.0 S 01:35 /usr/sbin/httpd -k start 1596139 506832 nobody 18704 0.0 S 01:16 /usr/sbin/httpd -k start 1595393 506832 nobody 18644 0.0 S 02:27 /usr/sbin/httpd -k start 1596236 506832 nobody 18620 0.0 S 01:14 /usr/sbin/httpd -k start 1596108 506832 nobody 18604 0.0 S 01:17 /usr/sbin/httpd -k start 1596110 506832 nobody 18556 0.0 S 01:17 /usr/sbin/httpd -k start 1596137 506832 nobody 18508 0.0 S 01:16 /usr/sbin/httpd -k start 1596041 506832 nobody 18484 0.0 S 01:18 /usr/sbin/httpd -k start 1596301 506832 nobody 18468 0.0 S 01:13 /usr/sbin/httpd -k start 1596256 506832 nobody 18432 0.0 S 01:14 /usr/sbin/httpd -k start 1596299 506832 nobody 18428 0.0 S 01:13 /usr/sbin/httpd -k start 1596235 506832 nobody 18400 0.0 S 01:14 /usr/sbin/httpd -k start 1596208 506832 nobody 18396 0.0 S 01:15 /usr/sbin/httpd -k start 1596239 506832 nobody 18396 0.0 S 01:14 /usr/sbin/httpd -k start 1596308 506832 nobody 18372 0.0 S 01:13 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- 1597492 D - /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/clean_user_php_sessions --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1317864 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min vinesit/westpanel.vinesit.com.au 89 vinesit/westpanel.vinesit.com.au-ssl_log 87 evolution/app.evolutionerp.com.au-ssl_log 84 speedlin/speedlinefencing.com.au 75 speedlin/speedlinefencing.com.au-ssl_log 74 evolution/evolutionerp.com.au 67 infinium/infinium.net.au-ssl_log 64 evolution/evolutionerp.com.au-ssl_log 56 madalice/madalice.com.au-ssl_log 35 evolution/portal.evolutionerp.com.au-ssl_log 22 aaa/aaadownunder.com.au-ssl_log 5 corpkicks/corporatekicks.com.au-ssl_log 2 alison/alisonarts.com.au-ssl_log 1 (total vhosts taking traffic: 13) --- top client IPs across all vhosts (last 3 min) --- 175 20.151.10.161 149 192.253.248.173 123 84.17.60.251 84 40.115.94.245 58 4.204.224.164 22 203.54.180.238 21 38.133.189.67 7 2001:8003:709d:2100:e120:3b27:5d2a:1c87 5 118.127.40.55 2 91.92.243.234 2 85.59.105.234 1 94.130.237.182 1 91.92.243.251 1 91.92.243.246 1 66.249.68.67 --- top request paths across all vhosts (last 3 min) --- 58 GET /autodiscover/autodiscover.xml 54 GET /api/getItem/LWPG2001KG 30 GET /api/getItem/LWFB2001KG 22 GET /heartbeat.php 8 GET / 5 POST /wp-login.php 5 POST /wp-cron.php 4 GET /admin.php 2 POST / 2 GET /.yarnrc.yml 2 GET /xiugai.php 2 GET /xamp.php 2 GET /wp.php 2 GET /wp-login.php 2 GET /wp-includes/sodium_compat/src/Core/SecretStream/ 2 GET /wp-includes/sodium_compat/src/Core/Salsa20.php 2 GET /wp-includes/sodium_compat/src/Core/Ristretto255.php 2 GET /wp-includes/sodium_compat/src/Core/Poly1305/State.php 2 GET /wp-includes/sodium_compat/src/Core/Poly1305.php 2 GET /wp-includes/sodium_compat/src/Core/Poly1305/ --- busiest 3 vhosts: top paths + last 10 requests --- == vinesit/westpanel.vinesit.com.au == -- top paths -- 1 GET /xiugai.php 1 GET /xamp.php 1 GET /wp.php 1 GET /wp-mails.php 1 GET /w.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /verox.php 1 GET /this_is_a_new_hello_world.php 1 GET /sm.php 1 GET /sky.php 1 GET /site.php 1 GET /simple.php -- last 10 requests -- 20.151.10.161 - - [03/Sep/2026:02:38:30 +1000] "GET /dex.php HTTP/1.1" 301 292 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:31 +1000] "GET /key.php HTTP/1.1" 301 292 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:32 +1000] "GET /kir.php HTTP/1.1" 301 292 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:33 +1000] "GET /nofile.php HTTP/1.1" 301 295 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:34 +1000] "GET /fling.php?p= HTTP/1.1" 301 297 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:34 +1000] "GET /btyuio.php HTTP/1.1" 301 295 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:35 +1000] "GET /00zip.php HTTP/1.1" 301 294 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:36 +1000] "GET /silver.php HTTP/1.1" 301 295 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:37 +1000] "GET /wp-mails.php HTTP/1.1" 301 297 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:38 +1000] "GET /sm.php HTTP/1.1" 301 291 "-" "-" == vinesit/westpanel.vinesit.com.au-ssl_log == -- top paths -- 3 GET /admin.php 1 GET /xiugai.php 1 GET /xamp.php 1 GET /wp.php 1 GET /w.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /verox.php 1 GET /this_is_a_new_hello_world.php 1 GET /sky.php 1 GET /site.php 1 GET /simple.php 1 GET /shiny.php -- last 10 requests -- 20.151.10.161 - - [03/Sep/2026:02:38:28 +1000] "GET /cabs.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:29 +1000] "GET /insc.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:30 +1000] "GET /file.php? HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:31 +1000] "GET /dex.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:32 +1000] "GET /key.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:32 +1000] "GET /kir.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:33 +1000] "GET /nofile.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:34 +1000] "GET /fling.php?p= HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:35 +1000] "GET /btyuio.php HTTP/1.1" 500 718 "-" "-" 20.151.10.161 - - [03/Sep/2026:02:38:36 +1000] "GET /00zip.php HTTP/1.1" 500 718 "-" "-" == evolution/app.evolutionerp.com.au-ssl_log == -- top paths -- 54 GET /api/getItem/LWPG2001KG 30 GET /api/getItem/LWFB2001KG -- last 10 requests -- 40.115.94.245 - - [03/Sep/2026:02:38:17 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:17 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:17 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:18 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:18 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:19 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:19 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:19 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:20 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [03/Sep/2026:02:38:20 +1000] "GET /api/getItem/LWPG2001KG HTTP/1.1" 200 1334 "-" "RestSharp/106.13.0.0" --- last 30 samples (run-up to this incident) --- 2026-09-03 02:36:35 L1=0.63 run=3 blk=0 avail=9476 (68.8%) swpO=0 conn=1 myRSS=2175 php=5 phpRSS=465 ap=16 tcp=33 topuser=evolution 2026-09-03 02:36:41 L1=0.58 run=2 blk=0 avail=9625 (69.9%) swpO=0 conn=1 myRSS=2175 php=4 phpRSS=238 ap=23 tcp=33 topuser=madalice 2026-09-03 02:36:45 L1=0.53 run=2 blk=0 avail=9481 (68.8%) swpO=0 conn=1 myRSS=2175 php=4 phpRSS=383 ap=22 tcp=35 topuser=corpkicks 2026-09-03 02:36:51 L1=0.57 run=1 blk=0 avail=9657 (70.1%) swpO=0 conn=1 myRSS=2175 php=3 phpRSS=198 ap=20 tcp=28 topuser=madalice 2026-09-03 02:36:55 L1=0.52 run=3 blk=0 avail=9576 (69.5%) swpO=0 conn=2 myRSS=2175 php=4 phpRSS=317 ap=18 tcp=31 topuser=madalice 2026-09-03 02:37:01 L1=0.48 run=1 blk=0 avail=9668 (70.2%) swpO=0 conn=1 myRSS=2175 php=3 phpRSS=198 ap=18 tcp=29 topuser=madalice 2026-09-03 02:37:05 L1=0.44 run=1 blk=0 avail=9800 (71.1%) swpO=0 conn=2 myRSS=2175 php=2 phpRSS=38 ap=17 tcp=32 topuser=root 2026-09-03 02:37:11 L1=0.49 run=3 blk=0 avail=9603 (69.7%) swpO=0 conn=1 myRSS=2175 php=4 phpRSS=297 ap=17 tcp=32 topuser=madalice 2026-09-03 02:37:15 L1=0.61 run=1 blk=0 avail=9684 (70.3%) swpO=0 conn=2 myRSS=2175 php=3 phpRSS=175 ap=18 tcp=33 topuser=madalice 2026-09-03 02:37:21 L1=0.64 run=2 blk=0 avail=9574 (69.5%) swpO=0 conn=1 myRSS=2175 php=4 phpRSS=319 ap=18 tcp=32 topuser=alison 2026-09-03 02:37:25 L1=0.59 run=1 blk=0 avail=9700 (70.4%) swpO=2 conn=2 myRSS=2175 php=3 phpRSS=175 ap=18 tcp=32 topuser=madalice 2026-09-03 02:37:31 L1=0.70 run=1 blk=0 avail=9571 (69.5%) swpO=2 conn=1 myRSS=2175 php=4 phpRSS=299 ap=21 tcp=38 topuser=madalice 2026-09-03 02:37:35 L1=0.65 run=4 blk=0 avail=9710 (70.5%) swpO=2 conn=2 myRSS=2175 php=3 phpRSS=150 ap=21 tcp=36 topuser=customc 2026-09-03 02:37:42 L1=0.59 run=2 blk=0 avail=9714 (70.5%) swpO=1 conn=1 myRSS=2175 php=2 phpRSS=38 ap=20 tcp=30 topuser=root 2026-09-03 02:37:45 L1=0.63 run=1 blk=0 avail=9342 (67.8%) swpO=0 conn=2 myRSS=2175 php=4 phpRSS=315 ap=19 tcp=140 topuser=madalice 2026-09-03 02:37:52 L1=0.58 run=3 blk=0 avail=8791 (63.8%) swpO=27 conn=1 myRSS=2174 php=5 phpRSS=353 ap=114 tcp=174 topuser=madalice 2026-09-03 02:37:55 L1=0.69 run=3 blk=0 avail=8425 (61.2%) swpO=641 conn=1 myRSS=2153 php=6 phpRSS=412 ap=146 tcp=122 topuser=madalice 2026-09-03 02:38:02 L1=0.72 run=37 blk=0 avail=6567 (47.7%) swpO=1470 conn=28 myRSS=2122 php=32 phpRSS=3202 ap=140 tcp=62 topuser=speedlin 2026-09-03 02:38:05 L1=2.98 run=30 blk=0 avail=7924 (57.5%) swpO=949 conn=23 myRSS=2119 php=26 phpRSS=1663 ap=137 tcp=90 topuser=speedlin 2026-09-03 02:38:14 L1=5.03 run=1 blk=0 avail=8661 (62.9%) swpO=76 conn=28 myRSS=2119 php=4 phpRSS=349 ap=128 tcp=33 topuser=madalice 2026-09-03 02:38:15 L1=5.03 run=6 blk=0 avail=8539 (62.0%) swpO=0 conn=23 myRSS=2119 php=6 phpRSS=540 ap=127 tcp=36 topuser=madalice 2026-09-03 02:38:24 L1=4.41 run=2 blk=0 avail=8722 (63.3%) swpO=0 conn=28 myRSS=2119 php=5 phpRSS=388 ap=118 tcp=30 topuser=madalice 2026-09-03 02:38:26 L1=4.41 run=2 blk=0 avail=8729 (63.4%) swpO=0 conn=23 myRSS=2119 php=5 phpRSS=365 ap=116 tcp=37 topuser=madalice 2026-09-03 02:38:34 L1=3.88 run=3 blk=0 avail=8788 (63.8%) swpO=0 conn=28 myRSS=2119 php=4 phpRSS=351 ap=108 tcp=32 topuser=madalice 2026-09-03 02:38:36 L1=3.88 run=1 blk=0 avail=8812 (64.0%) swpO=0 conn=23 myRSS=2119 php=4 phpRSS=351 ap=106 tcp=32 topuser=madalice 2026-09-03 02:38:44 L1=3.36 run=2 blk=0 avail=8950 (65.0%) swpO=0 conn=2 myRSS=2119 php=3 phpRSS=248 ap=98 tcp=39 topuser=madalice 2026-09-03 02:38:46 L1=3.36 run=1 blk=0 avail=9017 (65.4%) swpO=0 conn=23 myRSS=2119 php=3 phpRSS=192 ap=96 tcp=39 topuser=madalice 2026-09-03 02:38:54 L1=2.99 run=18 blk=0 avail=7354 (53.4%) swpO=0 conn=11 myRSS=2119 php=12 phpRSS=1940 ap=88 tcp=94 topuser=madalice 2026-09-03 02:38:56 L1=2.99 run=13 blk=0 avail=7360 (53.4%) swpO=0 conn=11 myRSS=2119 php=13 phpRSS=2112 ap=86 tcp=92 topuser=madalice 2026-09-03 02:39:04 L1=2.61 run=2 blk=0 avail=7883 (57.2%) swpO=0 conn=1 myRSS=2119 php=12 phpRSS=1626 ap=78 tcp=37 topuser=madalice