INCIDENT 2026-08-21 16:35:55 trigger: php_rss=1577MB ================================================================== load: 8.85 5.52 2.99 15/641 970038 ncpu:10 mem_avail: 5144MB/11729MB (43.9%) swap_used:2914MB swapout:0/s blocked(D):0 running:16 iowait:0.0% steal:0.0% php: 10 procs / 1577MB heaviest user: madalice (10 procs, 1577MB) apache: 29 procs tcp_estab: 29 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2969 madalice 9 1415 root 240 847 nobody 28 467 netdata 8 243 cpanellogin 1 104 mailman 20 95 evolution 8 30 dovenull 6 25 wp-toolkit 6 20 zabbix 1 14 named 1 14 aaa 1 13 kamcool 1 11 kurri 1 10 nscd 1 7 dovecot 3 7 mailnull 1 5 cpanelconnecttrack 1 3 dbus 1 2 --- php workers by user --- USER PROCS RSS_MB madalice 9 1420 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 4693 MB Active(file): 3657 MB Inactive(file): 571 MB Dirty: 1 MB Writeback: 0 MB SReclaimable: 543 MB CommitLimit: 9960 MB Committed_AS: 8282 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 267385 root 15:24:39 5408 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2575700 1 mysql 3040548 11.8 Ssl 6-08:45:20 /usr/sbin/mariadbd 969973 965431 madalice 264108 120 S 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 969976 969864 madalice 206344 105 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 969977 963511 madalice 198208 87.5 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1220 1 netdata 193440 2.5 Ssl 53-06:26:02 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 969978 967169 madalice 173160 83.5 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 969979 963904 madalice 168688 159 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 319521 311538 root 160664 0.0 S 13:58:11 spamd child 969980 966558 madalice 154064 164 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 969981 965105 madalice 153868 152 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 853881 311538 root 153336 0.0 S 02:45:06 spamd child 2575382 1 root 118048 0.0 Ss 6-08:45:23 /usr/lib/systemd/systemd-journald 969982 966708 madalice 116224 109 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 950837 1353308 cpanellogin 107448 0.1 S 29:32 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base server_locale en default_login_theme cpanel 311538 1 root 96588 0.0 Ss 14:13:10 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 969983 965046 madalice 71456 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2223 1201 root 42916 0.0 Sl 53-06:26:01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1353308 1 root 42292 0.0 Ss 30-13:54:32 cpsrvd (SSL) - waiting for connections 1469 1 root 37312 0.1 Ss 53-06:26:01 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 4922 2431 netdata 31160 0.1 Sl 53-06:25:57 /usr/libexec/netdata/plugins.d/go.d.plugin 1 963511 1276780 nobody 22116 0.1 S 09:35 /usr/sbin/httpd -k start 1463 1 root 21752 0.0 Ss 53-06:26:01 tailwatchd 310731 1 root 21704 0.0 Ss 14:13:42 lfd - sleeping 963904 1276780 nobody 21384 0.2 S 09:00 /usr/sbin/httpd -k start 965046 1276780 nobody 21236 0.2 S 06:53 /usr/sbin/httpd -k start 965105 1276780 nobody 21084 0.2 S 06:45 /usr/sbin/httpd -k start 965928 1276780 nobody 21060 0.0 S 05:33 /usr/sbin/httpd -k start 965431 1276780 nobody 21036 0.2 S 06:06 /usr/sbin/httpd -k start 965039 1276780 nobody 20980 0.2 S 06:54 /usr/sbin/httpd -k start 966708 1276780 nobody 20108 0.3 S 04:23 /usr/sbin/httpd -k start 967169 1276780 nobody 19504 0.3 S 03:44 /usr/sbin/httpd -k start 966558 1276780 nobody 19448 0.3 S 04:35 /usr/sbin/httpd -k start 968123 1276780 nobody 19328 0.0 S 02:14 /usr/sbin/httpd -k start 969864 1276780 nobody 19316 0.2 S 00:18 /usr/sbin/httpd -k start 965938 1276780 nobody 19312 0.0 S 05:32 /usr/sbin/httpd -k start 969793 1276780 nobody 19268 0.0 S 00:21 /usr/sbin/httpd -k start 968093 1276780 nobody 19040 0.0 S 02:19 /usr/sbin/httpd -k start 968599 1276780 nobody 18640 0.0 S 01:24 /usr/sbin/httpd -k start 968149 1276780 nobody 18612 0.0 S 02:13 /usr/sbin/httpd -k start 967819 1276780 nobody 18272 0.1 S 02:45 /usr/sbin/httpd -k start 967813 1276780 nobody 18196 0.0 S 02:46 /usr/sbin/httpd -k start 185104 2431 netdata 17848 2.1 Sl 8-20:02:20 /usr/libexec/netdata/plugins.d/apps.plugin 1 969843 1276780 nobody 16720 0.0 S 00:20 /usr/sbin/httpd -k start 969853 1276780 nobody 16688 0.0 S 00:19 /usr/sbin/httpd -k start 969859 1276780 nobody 16632 0.0 S 00:18 /usr/sbin/httpd -k start 969861 1276780 nobody 16456 0.0 S 00:18 /usr/sbin/httpd -k start 969844 1276780 nobody 16068 0.0 S 00:20 /usr/sbin/httpd -k start 1204 1 zabbix 15212 0.2 Ssl 53-06:26:02 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 2107664 1 named 14472 0.0 Ssl 2-14:13:16 /usr/sbin/pdns_server --socket-dir=/run/pdns --guardian=no --daemon=no --disable-syslog --log-timestamp=no --write-pid=no 946022 3828812 aaa 13832 0.0 S 37:34 dovecot/imap 1471 1 root 13048 0.0 Ss 53-06:26:01 cPhulkd - processor 405204 1471 root 12956 0.0 S 11:24:45 cPhulkd - dbprocessor 969852 1276780 nobody 12912 0.0 S 00:19 /usr/sbin/httpd -k start 1276780 1 root 12192 0.0 Ss 16-01:04:37 /usr/sbin/httpd -k start 963402 3828812 kamcool 11720 0.0 S 09:50 dovecot/pop3 3668761 3828812 kurri 10828 0.0 S 1-06:21:41 dovecot/imap 969277 3828812 root 10816 0.0 S 00:55 dovecot/auth -w 2107879 1 root 9808 0.0 Ss 2-14:13:15 queueprocd - waiting up to 60s to process a task 918817 27524 root 9116 0.0 Ss 01:21:58 sshd: evolution [priv] 1059098 3828812 dovenull 8020 0.0 S 8-01:03:07 dovecot/imap-login --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 22882860 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min madalice/madalice.com.au-ssl_log 477 bigdeals/bigdeals.com.au-ssl_log 123 evolution/my.evolutionerp.com.au-ssl_log 79 evolution/portal.evolutionerp.com.au-ssl_log 30 corpkicks/corporatekicks.com.au-ssl_log 5 aaa/aaadownunder.com.au-ssl_log 4 (total vhosts taking traffic: 6) --- top client IPs across all vhosts (last 3 min) --- 467 20.251.48.93 123 20.118.172.148 96 203.54.180.238 5 165.225.233.109 4 103.104.165.99 3 118.127.40.55 3 114.119.143.104 2 40.77.177.51 2 40.77.167.150 1 85.208.98.23 1 52.139.47.57 1 40.77.167.241 1 34.196.51.17 1 2001:4860:7:b01::ef 1 193.115.90.9 --- top request paths across all vhosts (last 3 min) --- 32 POST /functions.php 30 GET /heartbeat.php 18 GET /wp-ws68.php 18 GET /11.php 15 GET /aaa.php 12 GET /app/ajax.php 11 GET /admin.php 10 GET /about.php 10 GET /1.php 9 POST /quoteditsave.php 9 GET /yj09.php 9 GET /yedliner.php 9 GET /version.php 9 GET /upgrade.php 9 GET /uni.php 9 GET /styles.php 9 GET /source.php 9 GET /RxR.php 9 GET /media.php 9 GET /mc.php --- busiest 3 vhosts: top paths + last 10 requests --- == madalice/madalice.com.au-ssl_log == -- top paths -- 18 GET /wp-ws68.php 18 GET /11.php 15 GET /aaa.php 9 GET /yj09.php 9 GET /yedliner.php 9 GET /version.php 9 GET /upgrade.php 9 GET /uni.php 9 GET /styles.php 9 GET /source.php 9 GET /RxR.php 9 GET /media.php -- last 10 requests -- 20.251.48.93 - - [21/Aug/2026:16:35:44 +1000] "GET /blurbs.php HTTP/1.1" 404 30625 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:44 +1000] "GET /yj09.php HTTP/1.1" 404 30625 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:44 +1000] "GET /FWAZ.php HTTP/1.1" 404 30626 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:45 +1000] "GET /aaa.php?p= HTTP/1.1" 301 1 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:45 +1000] "GET /aaa.php?p= HTTP/1.1" 301 1 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:45 +1000] "GET /fpwch.php HTTP/1.1" 404 30625 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:46 +1000] "GET /ccc.php HTTP/1.1" 404 30624 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:47 +1000] "GET /bajah.php HTTP/1.1" 404 30625 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:47 +1000] "GET /adminner.php HTTP/1.1" 404 30630 "-" "-" 20.251.48.93 - - [21/Aug/2026:16:35:47 +1000] "GET /fpwch.php HTTP/1.1" 404 30625 "-" "-" == bigdeals/bigdeals.com.au-ssl_log == -- top paths -- 2 GET /css.php 2 GET /cong.php 2 GET /admin.php 2 GET /404.php 1 GET /z60.php 1 GET /xx.php 1 GET /xv.php 1 GET /x.php 1 GET /x56.php 1 GET /x50.php 1 GET /ww1.php 1 GET /wsd.php -- last 10 requests -- 20.118.172.148 - - [21/Aug/2026:16:35:12 +1000] "GET /wrt.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:12 +1000] "GET /bibil.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:12 +1000] "GET /radio.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:13 +1000] "GET /files.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:13 +1000] "GET /file7.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:13 +1000] "GET /gifclass.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:14 +1000] "GET /wp-admin/css/colors/midnight/about.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:14 +1000] "GET /nox.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:14 +1000] "GET /file48.php HTTP/1.1" 401 36 "-" "-" 20.118.172.148 - - [21/Aug/2026:16:35:14 +1000] "GET /info.php HTTP/1.1" 401 36 "-" "-" == evolution/my.evolutionerp.com.au-ssl_log == -- top paths -- 32 POST /functions.php 12 GET /app/ajax.php 9 POST /quoteditsave.php 8 GET /index.php 4 POST /itemModify.php 4 POST /app/ajax/selectFilter.php 3 POST /plugins/csvexporter/plugin.php 2 POST /app/ajax/selectFilterSingle.php 2 POST /app/ajax.php 2 GET /plugins/fensa/fensa-ajax.php 1 GET /logoff.php -- last 10 requests -- 203.54.180.238 - - [21/Aug/2026:16:35:10 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 236 "https://my.evolutionerp.com.au/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:153.0) Gecko/20100101 Firefox/153.0" 40.77.177.51 - - [21/Aug/2026:16:35:11 +1000] "GET /app/ajax.php?cn=&furi=true&uri=page%3Dquotedit%26quoteid%3D71692%26sectionid%3D0&call=getUserSetting&caller=ajaxSubmitLocal&seq=12652708319 HTTP/1.1" 200 111 "https://m 40.77.177.51 - - [21/Aug/2026:16:35:11 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 52 "https://my.evolutionerp.com.au/index.php?page=quotedit"eid=71692§ionid=0" "Mozilla/5.0 AppleWebKit/537.36 (KHTML 103.104.165.99 - - [21/Aug/2026:16:35:21 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 282 "https://my.evolutionerp.com.au/index.php?page=jobedit&id=11921&tab=generaltab" "Mozilla/5.0 (Windows NT 10.0; Win64; 203.54.180.238 - - [21/Aug/2026:16:35:28 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 237 "https://my.evolutionerp.com.au/index.php?page=quotedit"eid=78421" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl 203.54.180.238 - - [21/Aug/2026:16:35:28 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 235 "https://my.evolutionerp.com.au/index.php?page=quotedit"eid=78653§ionid=0" "Mozilla/5.0 (Windows NT 10.0; Win6 40.77.167.150 - - [21/Aug/2026:16:35:31 +1000] "GET /index.php?page=activities&keyword=1940+chevy+coupe+front+straight+axle+material HTTP/1.1" 200 5290 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; 165.225.233.109 - - [21/Aug/2026:16:35:34 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 238 "https://my.evolutionerp.com.au/index.php?page=jobregister" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/53 165.225.233.109 - - [21/Aug/2026:16:35:34 +1000] "GET /logoff.php?page=jobregister HTTP/1.1" 302 20 "https://my.evolutionerp.com.au/index.php?page=jobregister" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.3 165.225.233.109 - - [21/Aug/2026:16:35:34 +1000] "GET /index.php?page=jobregister HTTP/1.1" 200 5241 "https://my.evolutionerp.com.au/index.php?page=jobregister" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537. --- last 30 samples (run-up to this incident) --- 2026-08-21 16:33:32 L1=6.95 run=19 blk=0 avail=5548 (47.3%) swpO=0 conn=9 myRSS=2971 php=9 phpRSS=1095 ap=26 tcp=32 topuser=madalice 2026-08-21 16:33:34 L1=6.95 run=9 blk=0 avail=5519 (47.1%) swpO=0 conn=8 myRSS=2971 php=7 phpRSS=1059 ap=26 tcp=29 topuser=madalice 2026-08-21 16:33:42 L1=7.11 run=12 blk=0 avail=5390 (46.0%) swpO=0 conn=9 myRSS=2971 php=9 phpRSS=1223 ap=32 tcp=33 topuser=madalice 2026-08-21 16:33:44 L1=7.11 run=12 blk=0 avail=5228 (44.6%) swpO=0 conn=8 myRSS=2971 php=9 phpRSS=1395 ap=34 tcp=35 topuser=madalice 2026-08-21 16:33:52 L1=7.17 run=11 blk=0 avail=5685 (48.5%) swpO=0 conn=9 myRSS=2971 php=6 phpRSS=832 ap=32 tcp=30 topuser=madalice 2026-08-21 16:33:54 L1=7.17 run=8 blk=0 avail=5608 (47.8%) swpO=0 conn=8 myRSS=2971 php=7 phpRSS=951 ap=31 tcp=30 topuser=madalice 2026-08-21 16:34:03 L1=7.22 run=10 blk=0 avail=5385 (45.9%) swpO=0 conn=10 myRSS=2971 php=9 phpRSS=1288 ap=26 tcp=29 topuser=madalice 2026-08-21 16:34:04 L1=7.22 run=10 blk=0 avail=5367 (45.8%) swpO=0 conn=8 myRSS=2971 php=8 phpRSS=1253 ap=26 tcp=30 topuser=madalice 2026-08-21 16:34:13 L1=7.34 run=11 blk=0 avail=5672 (48.4%) swpO=0 conn=10 myRSS=2971 php=8 phpRSS=923 ap=26 tcp=31 topuser=madalice 2026-08-21 16:34:14 L1=7.34 run=7 blk=0 avail=5558 (47.4%) swpO=0 conn=8 myRSS=2971 php=6 phpRSS=998 ap=26 tcp=29 topuser=madalice 2026-08-21 16:34:23 L1=7.06 run=12 blk=0 avail=5326 (45.4%) swpO=0 conn=10 myRSS=2971 php=8 phpRSS=1254 ap=25 tcp=31 topuser=madalice 2026-08-21 16:34:24 L1=7.06 run=12 blk=0 avail=5491 (46.8%) swpO=0 conn=7 myRSS=2971 php=8 phpRSS=1036 ap=25 tcp=30 topuser=madalice 2026-08-21 16:34:33 L1=7.29 run=11 blk=0 avail=5187 (44.2%) swpO=0 conn=10 myRSS=2971 php=8 phpRSS=1430 ap=28 tcp=35 topuser=madalice 2026-08-21 16:34:34 L1=7.29 run=7 blk=0 avail=5792 (49.4%) swpO=0 conn=7 myRSS=2971 php=6 phpRSS=746 ap=28 tcp=30 topuser=madalice 2026-08-21 16:34:43 L1=7.46 run=13 blk=0 avail=5150 (43.9%) swpO=27 conn=10 myRSS=2970 php=8 phpRSS=1449 ap=28 tcp=30 topuser=madalice 2026-08-21 16:34:44 L1=7.46 run=7 blk=0 avail=5727 (48.8%) swpO=27 conn=7 myRSS=2970 php=6 phpRSS=734 ap=28 tcp=30 topuser=madalice 2026-08-21 16:34:53 L1=8.34 run=11 blk=0 avail=5125 (43.7%) swpO=0 conn=9 myRSS=2970 php=9 phpRSS=1509 ap=28 tcp=35 topuser=madalice 2026-08-21 16:34:54 L1=8.34 run=20 blk=0 avail=4987 (42.5%) swpO=0 conn=9 myRSS=2970 php=10 phpRSS=1657 ap=28 tcp=34 topuser=madalice 2026-08-21 16:35:05 L1=8.44 run=11 blk=0 avail=5208 (44.4%) swpO=0 conn=9 myRSS=2970 php=9 phpRSS=1409 ap=28 tcp=32 topuser=madalice 2026-08-21 16:35:05 L1=8.44 run=8 blk=0 avail=5482 (46.7%) swpO=0 conn=8 myRSS=2970 php=7 phpRSS=1063 ap=28 tcp=32 topuser=madalice 2026-08-21 16:35:15 L1=8.30 run=10 blk=0 avail=5336 (45.5%) swpO=30 conn=9 myRSS=2969 php=8 phpRSS=1225 ap=28 tcp=33 topuser=madalice 2026-08-21 16:35:15 L1=8.30 run=14 blk=0 avail=5363 (45.7%) swpO=30 conn=8 myRSS=2969 php=8 phpRSS=1237 ap=28 tcp=32 topuser=madalice 2026-08-21 16:35:25 L1=8.57 run=11 blk=0 avail=5246 (44.7%) swpO=0 conn=10 myRSS=2969 php=10 phpRSS=1512 ap=27 tcp=32 topuser=madalice 2026-08-21 16:35:25 L1=8.57 run=20 blk=0 avail=5283 (45.0%) swpO=0 conn=8 myRSS=2969 php=9 phpRSS=1378 ap=27 tcp=32 topuser=madalice 2026-08-21 16:35:35 L1=8.54 run=16 blk=2 avail=5475 (46.7%) swpO=0 conn=10 myRSS=2969 php=8 phpRSS=1141 ap=28 tcp=37 topuser=madalice 2026-08-21 16:35:35 L1=8.54 run=13 blk=0 avail=5434 (46.3%) swpO=0 conn=8 myRSS=2969 php=8 phpRSS=1194 ap=28 tcp=37 topuser=madalice 2026-08-21 16:35:45 L1=8.39 run=14 blk=0 avail=5315 (45.3%) swpO=0 conn=10 myRSS=2969 php=9 phpRSS=1277 ap=36 tcp=33 topuser=madalice 2026-08-21 16:35:45 L1=8.39 run=12 blk=0 avail=5288 (45.1%) swpO=0 conn=8 myRSS=2969 php=9 phpRSS=1306 ap=36 tcp=33 topuser=madalice 2026-08-21 16:35:55 L1=8.41 run=16 blk=0 avail=5144 (43.9%) swpO=0 conn=11 myRSS=2969 php=10 phpRSS=1577 ap=29 tcp=29 topuser=madalice 2026-08-21 16:35:55 L1=8.41 run=13 blk=0 avail=5097 (43.5%) swpO=0 conn=10 myRSS=2969 php=10 phpRSS=1603 ap=29 tcp=29 topuser=madalice