INCIDENT 2026-08-30 04:26:02 trigger: php_rss=1833MB ================================================================== load: 1.84 1.19 1.11 5/686 2701951 ncpu:10 mem_avail: 7032MB/13777MB (51.0%) swap_used:1835MB swapout:0/s blocked(D):0 running:5 iowait:0.0% steal:0.0% php: 14 procs / 1833MB heaviest user: madalice (10 procs, 1596MB) apache: 48 procs tcp_estab: 52 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2688 madalice 10 1602 root 252 1108 nobody 47 740 olympic 2 202 netdata 7 196 mailman 20 127 cpanellogin 1 62 dovenull 6 46 evolution 11 44 wp-toolkit 3 32 lowrange 2 29 kurri 2 26 mailnull 1 24 zabbix 1 21 dovecot 3 20 named 1 15 infinium 1 14 polkitd 1 5 nscd 1 4 --- php workers by user --- USER PROCS RSS_MB madalice 10 1605 olympic 2 203 root 1 22 evolution 1 13 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5524 MB Active(file): 4682 MB Inactive(file): 1155 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 647 MB CommitLimit: 10984 MB Committed_AS: 9016 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 489033 root 2-02:07:23 4236 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 506671 1 mysql 2752748 7.9 Ssl 2-02:06:52 /usr/sbin/mariadbd 2701827 510022 madalice 203636 61.8 R 00:05 php-fpm: pool madalice_com_au 2328310 2277352 root 183316 0.0 S 01:47:52 spamd child 508924 1 netdata 176512 1.9 Ssl 2-02:06:21 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2277352 1 root 175896 0.0 Ss 02:02:58 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 2328311 2277352 root 168452 0.0 S 01:47:52 spamd child 2701823 510022 madalice 162512 41.5 S 00:06 php-fpm: pool madalice_com_au 2701562 510022 madalice 162148 9.5 S 00:48 php-fpm: pool madalice_com_au 2701693 510022 madalice 161988 8.8 S 00:27 php-fpm: pool madalice_com_au 2701694 510022 madalice 161964 9.1 S 00:27 php-fpm: pool madalice_com_au 2701692 510022 madalice 161864 9.2 S 00:27 php-fpm: pool madalice_com_au 2701822 510022 madalice 161744 39.8 S 00:06 php-fpm: pool madalice_com_au 2701831 510022 madalice 159652 53.6 S 00:05 php-fpm: pool madalice_com_au 2701832 510022 madalice 159640 54.0 S 00:05 php-fpm: pool madalice_com_au 2701825 510022 madalice 156980 47.0 S 00:05 php-fpm: pool madalice_com_au 2701852 2701843 olympic 134144 52.5 S 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 509266 1 root 131868 0.0 Ss 2-02:06:18 /usr/lib/systemd/systemd-journald 2701896 2701879 olympic 78060 39.0 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2651982 1353308 cpanellogin 64000 0.0 S 13:23 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base server_locale en default_login_theme cpanel 509236 509098 root 58612 0.0 Sl 2-02:06:18 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 509640 508992 root 43992 0.0 Sl 2-02:06:16 /usr/libexec/netdata/plugins.d/ebpf.plugin 1 1353308 1 root 42692 0.1 Ss 39-01:44:44 cpsrvd (SSL) - waiting for connections 2275833 1 root 36856 0.0 Ss 02:03:48 lfd - sleeping 508720 1 mailnull 25564 0.0 Ss 2-02:06:25 /usr/sbin/exim -ps -bd -q15m -oP /var/spool/exim/exim-daemon.pid 1469 1 root 23756 0.1 Ss 61-18:16:13 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 510022 1 root 23532 0.0 Ss 2-02:06:14 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 2652660 506832 nobody 23144 0.0 S 12:24 /usr/sbin/httpd -k start 483898 1 zabbix 22064 0.2 Ssl 2-02:07:35 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 2686210 506832 nobody 21948 0.0 S 04:03 /usr/sbin/httpd -k start 2687075 506832 nobody 21844 0.0 S 02:18 /usr/sbin/httpd -k start 1463 1 root 21840 0.0 Ss 61-18:16:13 tailwatchd 2669466 506832 nobody 21544 0.0 S 09:26 /usr/sbin/httpd -k start 2669468 506832 nobody 21144 0.0 S 09:26 /usr/sbin/httpd -k start 2701697 506832 nobody 20468 0.0 S 00:23 /usr/sbin/httpd -k start 2670038 506832 nobody 19732 0.0 S 08:24 /usr/sbin/httpd -k start 2687384 506832 nobody 19512 0.0 S 01:52 /usr/sbin/httpd -k start 2687371 506832 nobody 19460 0.0 S 01:54 /usr/sbin/httpd -k start 2687081 506832 nobody 19340 0.0 S 02:14 /usr/sbin/httpd -k start 2701701 506832 nobody 19140 0.0 S 00:22 /usr/sbin/httpd -k start 2701838 506832 nobody 19048 0.6 S 00:03 /usr/sbin/httpd -k start 2701845 506832 nobody 19032 0.5 S 00:02 /usr/sbin/httpd -k start 2701835 506832 nobody 19000 0.2 S 00:04 /usr/sbin/httpd -k start 2701841 506832 nobody 18988 0.3 S 00:03 /usr/sbin/httpd -k start 2701702 506832 nobody 18968 0.0 S 00:22 /usr/sbin/httpd -k start 2701846 506832 nobody 18964 1.0 S 00:02 /usr/sbin/httpd -k start 2701844 506832 nobody 18960 0.5 S 00:02 /usr/sbin/httpd -k start 2701824 506832 nobody 18940 0.4 S 00:05 /usr/sbin/httpd -k start 2701839 506832 nobody 18924 0.3 S 00:03 /usr/sbin/httpd -k start 2701847 506832 nobody 18924 0.5 S 00:02 /usr/sbin/httpd -k start 2701836 506832 nobody 18912 0.2 S 00:04 /usr/sbin/httpd -k start 938582 1 wp-toolkit 18728 0.0 Ss 2-00:13:55 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 2701703 506832 nobody 17604 0.0 S 00:22 /usr/sbin/httpd -k start 2701704 506832 nobody 17604 0.0 S 00:22 /usr/sbin/httpd -k start 2701843 506832 nobody 16936 0.0 S 00:02 /usr/sbin/httpd -k start 1471 1 root 16728 0.0 Ss 61-18:16:13 cPhulkd - processor 2701842 506832 nobody 16720 0.5 S 00:02 /usr/sbin/httpd -k start 2653013 1471 root 16532 0.0 S 12:11 cPhulkd - processor - http socket 2701849 506832 nobody 16448 0.0 S 00:02 /usr/sbin/httpd -k start 2701879 506832 nobody 16416 0.0 S 00:01 /usr/sbin/httpd -k start 2327413 1471 root 16400 0.0 S 01:49:44 cPhulkd - dbprocessor --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4328176 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/my.evolutionerp.com.au-ssl_log 36 evolution/release.evolutionerp.com.au 29 evolution/portal.evolutionerp.com.au-ssl_log 16 madalice/madalice.com.au-ssl_log 16 corpkicks/corporatekicks.com.au-ssl_log 8 aaa/aaadownunder.com.au 7 alison/alisonarts.com.au-ssl_log 4 infinium/infinium.net.au-ssl_log 4 (total vhosts taking traffic: 8) --- top client IPs across all vhosts (last 3 min) --- 61 20.220.10.235 16 203.54.180.238 8 118.127.40.55 7 141.98.11.42 3 93.152.221.252 3 195.178.110.247 2 52.167.144.173 2 118.71.62.248 2 114.119.139.61 2 114.119.137.101 1 85.208.96.201 1 62.60.130.182 1 40.77.167.51 1 34.196.51.17 1 207.46.13.36 --- top request paths across all vhosts (last 3 min) --- 16 GET /heartbeat.php 6 POST / 6 GET /store/custom-cowboys-kids/ 4 GET /wp-login.php 3 POST /index.php 3 GET /index.php 2 POST /wp-cron.php 2 GET /yyy.php 2 GET /wp-content/plugins/hellopress/wp_filemanager.php 2 GET /this_is_a_new_hello_world.php 2 GET /tanjiro.php 2 GET /super.php 2 GET /shelp.php 2 GET /sf9.php 2 GET /Rk41.php 2 GET /mga.php 2 GET /lufix1.php 2 GET /lte7.php 2 GET /k8.php 2 GET /inwp.php --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/my.evolutionerp.com.au-ssl_log == -- top paths -- 3 GET /index.php 1 GET /yyy.php 1 GET /x23.php 1 GET /ws66.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /waw.php 1 GET /this_is_a_new_hello_world.php 1 GET /tanjiro.php 1 GET /super.php 1 GET /shelp.php 1 GET /sf9.php 1 GET /Rk41.php -- last 10 requests -- 20.220.10.235 - - [30/Aug/2026:04:25:57 +1000] "GET /7logs.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:58 +1000] "GET /super.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:58 +1000] "GET /lufix1.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:59 +1000] "GET /hack.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:59 +1000] "GET /155.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:59 +1000] "GET /mcd.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:26:00 +1000] "GET /waw.php HTTP/1.1" 404 - "-" "-" 20.220.10.235 - - [30/Aug/2026:04:26:00 +1000] "GET /x23.php HTTP/1.1" 404 - "-" "-" 207.46.13.36 - - [30/Aug/2026:04:26:00 +1000] "GET /app/ajax.php?cn=&furi=true&uri=page%3Djobedit%26id%3D27748&call=getUserSetting&caller=ajaxSubmitLocal&seq=10466497328 HTTP/1.1" 200 97 "https://my.evolutionerp.com.au/i 20.220.10.235 - - [30/Aug/2026:04:26:01 +1000] "GET /ws66.php HTTP/1.1" 404 - "-" "-" == evolution/release.evolutionerp.com.au == -- top paths -- 1 GET /yyy.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /this_is_a_new_hello_world.php 1 GET /tanjiro.php 1 GET /super.php 1 GET /shelp.php 1 GET /sf.php 1 GET /sf9.php 1 GET /Rk41.php 1 GET /mga.php 1 GET /lufix1.php 1 GET /lte7.php -- last 10 requests -- 20.220.10.235 - - [30/Aug/2026:04:25:55 +1000] "GET /sf9.php HTTP/1.1" 301 286 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:56 +1000] "GET /lte7.php HTTP/1.1" 301 287 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:56 +1000] "GET /tanjiro.php HTTP/1.1" 301 290 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:56 +1000] "GET /Rk41.php HTTP/1.1" 301 287 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:57 +1000] "GET /7logs.php HTTP/1.1" 301 288 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:57 +1000] "GET /sf.php HTTP/1.1" 403 13 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:57 +1000] "GET /super.php HTTP/1.1" 301 288 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:58 +1000] "GET /lufix1.php HTTP/1.1" 301 289 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:58 +1000] "GET /hack.php HTTP/1.1" 301 287 "-" "-" 20.220.10.235 - - [30/Aug/2026:04:25:59 +1000] "GET /155.php HTTP/1.1" 301 286 "-" "-" == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 16 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [30/Aug/2026:04:24:33 +1000] "GET /heartbeat.php?sentAt=1788027874&bytes=106,147,35,226&_=1787897053438 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:24:33 +1000] "GET /heartbeat.php?sentAt=1788027874&bytes=106,147,35,226&_=1787918091923 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:24:43 +1000] "GET /heartbeat.php?sentAt=1788027884&bytes=106,147,35,236&_=1787897053439 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:24:43 +1000] "GET /heartbeat.php?sentAt=1788027884&bytes=106,147,35,236&_=1787918091924 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:24:53 +1000] "GET /heartbeat.php?sentAt=1788027894&bytes=106,147,35,246&_=1787897053440 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:24:53 +1000] "GET /heartbeat.php?sentAt=1788027894&bytes=106,147,35,246&_=1787918091925 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:25:03 +1000] "GET /heartbeat.php?sentAt=1788027904&bytes=106,147,36,0&_=1787897053441 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" " 203.54.180.238 - - [30/Aug/2026:04:25:03 +1000] "GET /heartbeat.php?sentAt=1788027904&bytes=106,147,36,0&_=1787918091926 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" " 203.54.180.238 - - [30/Aug/2026:04:25:13 +1000] "GET /heartbeat.php?sentAt=1788027914&bytes=106,147,36,10&_=1787897053442 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [30/Aug/2026:04:25:13 +1000] "GET /heartbeat.php?sentAt=1788027914&bytes=106,147,36,10&_=1787918091927 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" --- last 30 samples (run-up to this incident) --- 2026-08-30 04:23:41 L1=0.94 run=2 blk=0 avail=8515 (61.8%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=270 ap=15 tcp=30 topuser=madalice 2026-08-30 04:23:41 L1=0.94 run=3 blk=0 avail=8515 (61.8%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=270 ap=15 tcp=30 topuser=madalice 2026-08-30 04:23:51 L1=0.88 run=1 blk=0 avail=8580 (62.3%) swpO=2 conn=2 myRSS=2688 php=3 phpRSS=196 ap=17 tcp=29 topuser=madalice 2026-08-30 04:23:51 L1=0.88 run=3 blk=0 avail=8578 (62.3%) swpO=2 conn=2 myRSS=2688 php=3 phpRSS=196 ap=17 tcp=29 topuser=madalice 2026-08-30 04:24:01 L1=0.74 run=1 blk=0 avail=8582 (62.3%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=196 ap=16 tcp=30 topuser=madalice 2026-08-30 04:24:01 L1=0.74 run=5 blk=0 avail=8584 (62.3%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=196 ap=16 tcp=30 topuser=madalice 2026-08-30 04:24:11 L1=0.79 run=1 blk=0 avail=8366 (60.7%) swpO=0 conn=2 myRSS=2688 php=5 phpRSS=490 ap=18 tcp=35 topuser=evolution 2026-08-30 04:24:11 L1=0.79 run=2 blk=0 avail=8366 (60.7%) swpO=0 conn=2 myRSS=2688 php=5 phpRSS=490 ap=18 tcp=35 topuser=evolution 2026-08-30 04:24:21 L1=0.98 run=4 blk=0 avail=8611 (62.5%) swpO=0 conn=1 myRSS=2688 php=4 phpRSS=237 ap=16 tcp=27 topuser=madalice 2026-08-30 04:24:22 L1=0.98 run=3 blk=0 avail=8603 (62.4%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=196 ap=16 tcp=27 topuser=madalice 2026-08-30 04:24:32 L1=0.83 run=1 blk=0 avail=8639 (62.7%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=196 ap=14 tcp=26 topuser=madalice 2026-08-30 04:24:32 L1=0.83 run=2 blk=0 avail=8639 (62.7%) swpO=0 conn=2 myRSS=2688 php=3 phpRSS=196 ap=14 tcp=26 topuser=madalice 2026-08-30 04:24:42 L1=0.70 run=2 blk=0 avail=8637 (62.7%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=196 ap=14 tcp=27 topuser=madalice 2026-08-30 04:24:42 L1=0.70 run=2 blk=0 avail=8637 (62.7%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=196 ap=14 tcp=27 topuser=madalice 2026-08-30 04:24:52 L1=0.67 run=3 blk=0 avail=8611 (62.5%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=222 ap=14 tcp=26 topuser=madalice 2026-08-30 04:24:52 L1=0.67 run=5 blk=0 avail=8609 (62.5%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=224 ap=14 tcp=26 topuser=madalice 2026-08-30 04:25:02 L1=0.64 run=3 blk=0 avail=8640 (62.7%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=25 topuser=madalice 2026-08-30 04:25:02 L1=0.64 run=3 blk=0 avail=8640 (62.7%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=25 topuser=madalice 2026-08-30 04:25:12 L1=0.70 run=2 blk=1 avail=8667 (62.9%) swpO=2 conn=1 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=24 topuser=madalice 2026-08-30 04:25:12 L1=0.70 run=2 blk=1 avail=8667 (62.9%) swpO=2 conn=1 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=24 topuser=madalice 2026-08-30 04:25:22 L1=0.99 run=4 blk=0 avail=8564 (62.2%) swpO=0 conn=1 myRSS=2688 php=5 phpRSS=345 ap=13 tcp=27 topuser=madalice 2026-08-30 04:25:22 L1=0.99 run=4 blk=0 avail=8564 (62.2%) swpO=0 conn=3 myRSS=2688 php=5 phpRSS=345 ap=13 tcp=27 topuser=madalice 2026-08-30 04:25:32 L1=1.22 run=2 blk=0 avail=8675 (63.0%) swpO=0 conn=1 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=25 topuser=madalice 2026-08-30 04:25:32 L1=1.22 run=2 blk=0 avail=8674 (63.0%) swpO=0 conn=3 myRSS=2688 php=3 phpRSS=169 ap=14 tcp=25 topuser=madalice 2026-08-30 04:25:42 L1=1.11 run=2 blk=0 avail=8589 (62.3%) swpO=0 conn=3 myRSS=2688 php=6 phpRSS=252 ap=21 tcp=28 topuser=madalice 2026-08-30 04:25:42 L1=1.11 run=1 blk=0 avail=8590 (62.4%) swpO=0 conn=1 myRSS=2688 php=6 phpRSS=252 ap=21 tcp=28 topuser=madalice 2026-08-30 04:25:52 L1=1.29 run=2 blk=0 avail=8597 (62.4%) swpO=0 conn=1 myRSS=2688 php=6 phpRSS=252 ap=18 tcp=28 topuser=madalice 2026-08-30 04:25:52 L1=1.29 run=2 blk=0 avail=8597 (62.4%) swpO=0 conn=3 myRSS=2688 php=6 phpRSS=252 ap=18 tcp=28 topuser=madalice 2026-08-30 04:26:02 L1=1.84 run=5 blk=0 avail=7032 (51.0%) swpO=0 conn=5 myRSS=2688 php=14 phpRSS=1833 ap=48 tcp=52 topuser=madalice 2026-08-30 04:26:02 L1=1.84 run=5 blk=0 avail=7030 (51.0%) swpO=0 conn=4 myRSS=2688 php=14 phpRSS=1833 ap=48 tcp=52 topuser=madalice