INCIDENT 2026-09-05 19:35:22 trigger: php_rss=1972MB mysql_conn=16 ================================================================== load: 1.40 1.44 1.56 23/714 2986619 ncpu:10 mem_avail: 8192MB/13777MB (59.5%) swap_used:4095MB swapout:3/s blocked(D):1 running:22 iowait:6.6% steal:0.0% php: 19 procs / 1972MB heaviest user: otterfencing (14 procs, 1842MB) apache: 28 procs tcp_estab: 40 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 1863 otterfencing 13 1816 root 256 923 nobody 27 406 netdata 8 264 cpanellogin 1 92 madalice 1 80 evolution 11 77 dovenull 6 64 wp-toolkit 3 52 dovecot 3 27 mailman 20 25 mailnull 4 17 kurri 1 13 zabbix 1 10 sshd 2 8 polkitd 1 7 named 1 4 nscd 1 3 dbus 1 2 --- php workers by user --- USER PROCS RSS_MB otterfencing 12 1664 madalice 1 81 evolution 3 59 root 1 27 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 8423 MB Active(file): 5029 MB Inactive(file): 2763 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 462 MB CommitLimit: 10984 MB Committed_AS: 9710 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2972159 root 00:22 1228 /bin/sh -c bash -c 'sleep $((RANDOM % 60))' ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || : 3926885 root 4-17:12:56 3704 /usr/bin/imunify-agent-proxy 4109079 root 14:50:34 1856 /opt/ai-bolit/wrapper /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --mem 4109105 root 14:50:34 178912 /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --memory 2048M --progress --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 1908140 10.4 Ssl 3-05:21:11 /usr/sbin/mariadbd 2986506 2960161 otterfencing 228036 94.2 R 00:10 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 1542443 1 netdata 199312 2.5 Ssl 2-17:13:01 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 4109105 4109079 root 178912 2.9 Sl 14:50:34 /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --memory 2141727 3626973 root 173832 0.1 S 04:00:24 spamd child 2986534 2986531 otterfencing 165172 53.3 R 00:03 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986536 2971151 otterfencing 158920 51.0 R 00:03 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986544 2986539 otterfencing 158100 75.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986548 2986532 otterfencing 150400 68.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986542 2971868 otterfencing 148944 64.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986551 2953143 otterfencing 146528 68.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986545 2986540 otterfencing 144640 65.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986543 2986538 otterfencing 140248 62.0 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986550 2971875 otterfencing 113032 42.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986549 2970709 otterfencing 101004 39.0 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2749918 1353308 cpanellogin 94544 0.0 S 01:06:51 /usr/local/cpanel/base/show_template.stor server_locale en default_login_theme cpanel cpanel_locale docroot /usr/local/cpanel/base 2986552 2971352 otterfencing 92340 29.5 R 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 2986562 2601907 madalice 91940 39.0 R 00:01 php-fpm: pool madalice_com_au 3626973 1 root 82548 0.0 Ss 17:12:08 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 509266 1 root 66412 0.0 Ss 8-17:15:42 /usr/lib/systemd/systemd-journald 1542981 1542551 netdata 41624 0.1 Sl 2-17:12:57 /usr/libexec/netdata/plugins.d/go.d.plugin 1 1469 1 root 41536 0.1 Ss 68-09:25:37 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1353308 1 root 40340 0.1 Ss 45-16:54:08 cpsrvd (SSL) - waiting for connections 3643625 3626973 root 37940 0.0 S 17:08:33 spamd child 509236 509098 root 34880 0.0 Sl 8-17:15:42 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 3127358 3127320 evolution 31324 0.0 Ss 19:35:22 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero2.php 3127357 3127312 evolution 29320 0.0 Ss 19:35:22 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero_payment_cron.php 1542487 1 wp-toolkit 28824 0.0 Ss 2-17:13:01 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 2601907 1 root 28044 0.0 Ss 1-17:11:57 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 2953151 506832 nobody 22632 0.1 S 09:05 /usr/sbin/httpd -k start 2958627 506832 nobody 21896 0.0 S 05:18 /usr/sbin/httpd -k start 2961769 506832 nobody 21484 0.0 S 05:16 /usr/sbin/httpd -k start 2953143 506832 nobody 21144 0.0 S 09:06 /usr/sbin/httpd -k start 2971352 506832 nobody 20760 0.0 S 02:06 /usr/sbin/httpd -k start 2971868 506832 nobody 20408 0.0 S 01:00 /usr/sbin/httpd -k start 2970709 506832 nobody 19736 0.0 S 03:07 /usr/sbin/httpd -k start 2960161 506832 nobody 19300 0.0 S 05:17 /usr/sbin/httpd -k start 2971151 506832 nobody 18916 0.0 S 02:24 /usr/sbin/httpd -k start 1542531 1 wp-toolkit 18844 0.0 Ss 2-17:13:00 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script scheduled-tasks-executor.php 1542982 1542551 netdata 18608 2.2 Sl 2-17:12:57 /usr/libexec/netdata/plugins.d/apps.plugin 1 2971875 506832 nobody 18488 0.0 S 00:59 /usr/sbin/httpd -k start 1463 1 root 18192 0.0 Ss 68-09:25:37 tailwatchd 2986519 506832 nobody 18004 0.0 S 00:05 /usr/sbin/httpd -k start 2986537 506832 nobody 17956 0.3 S 00:03 /usr/sbin/httpd -k start 2986532 506832 nobody 16568 0.2 S 00:04 /usr/sbin/httpd -k start 2986540 506832 nobody 16272 0.0 S 00:03 /usr/sbin/httpd -k start 2986531 506832 nobody 16208 0.2 S 00:04 /usr/sbin/httpd -k start 2986539 506832 nobody 16208 0.0 S 00:03 /usr/sbin/httpd -k start 2986538 506832 nobody 16144 0.3 S 00:03 /usr/sbin/httpd -k start 3626082 1 root 15568 0.0 Ss 17:13:08 lfd - sleeping 2986559 506832 nobody 15404 0.0 S 00:02 /usr/sbin/httpd -k start 1542141 1 root 15316 0.0 Ss 2-17:13:04 sw-engine-fpm: master process (/etc/sw-engine/sw-engine-fpm.conf) 2986554 506832 nobody 14244 0.0 S 00:02 /usr/sbin/httpd -k start 2986557 506832 nobody 14244 0.0 S 00:02 /usr/sbin/httpd -k start 2986558 506832 nobody 14244 0.0 S 00:02 /usr/sbin/httpd -k start 3141944 508511 kurri 13544 0.0 S 19:35:00 dovecot/imap 2602226 1 root 13508 0.1 Ssl 1-17:11:42 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P 690514 508511 dovecot 13236 0.0 S 1-07:02:33 dovecot/auth 2972087 508511 root 12860 0.0 S 00:33 dovecot/auth -w 690505 508511 dovenull 12824 0.0 S 1-07:02:33 dovecot/imap-login --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 11087544 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/app.evolutionerp.com.au-ssl_log 94 corpkicks/2025.corporatekicks.com.au 4 (total vhosts taking traffic: 2) --- top client IPs across all vhosts (last 3 min) --- 94 40.115.94.245 4 4.204.224.164 --- top request paths across all vhosts (last 3 min) --- 47 GET /api/getItem/LWFB2001KG 3 GET /api/getItem/SHSG252516-6.5 2 GET /api/getItem/SHSG252516-8 2 GET /api/getItem/RHSG1257530 2 GET /api/getItem/19R-8000 2 GET /api/getItem/19R-7320 1 GET /v4on3fxtoo5z5cq0yCdefault.php 1 GET /CAE-2.5.php 1 GET /api/getItem/SHSG2020308 1 GET /api/getItem/SHSG2020258 1 GET /api/getItem/SHSG2020168 1 GET /api/getItem/RHSP200100608 1 GET /api/getItem/RHSP200100508 1 GET /api/getItem/RHSP200100408 1 GET /api/getItem/RHSP15050608 1 GET /api/getItem/RHSP150100608 1 GET /api/getItem/RHSP150100508 1 GET /api/getItem/RHSP12575608 1 GET /api/getItem/RHSG15050508 1 GET /api/getItem/RHSG15050408 --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/app.evolutionerp.com.au-ssl_log == -- top paths -- 47 GET /api/getItem/LWFB2001KG 3 GET /api/getItem/SHSG252516-6.5 2 GET /api/getItem/SHSG252516-8 2 GET /api/getItem/RHSG1257530 2 GET /api/getItem/19R-8000 2 GET /api/getItem/19R-7320 1 GET /api/getItem/SHSG2020308 1 GET /api/getItem/SHSG2020258 1 GET /api/getItem/SHSG2020168 1 GET /api/getItem/RHSP200100608 1 GET /api/getItem/RHSP200100508 1 GET /api/getItem/RHSP200100408 -- last 10 requests -- 40.115.94.245 - - [05/Sep/2026:19:34:27 +1000] "GET /api/getItem/PS252512-2400G-DI HTTP/1.1" 200 1517 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:27 +1000] "GET /api/getItem/LWFB2001KG HTTP/1.1" 200 1309 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:27 +1000] "GET /api/getItem/RHSG1005020-DP25-8 HTTP/1.1" 200 1415 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:27 +1000] "GET /api/getItem/LWFB2001KG HTTP/1.1" 200 1309 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:27 +1000] "GET /api/getItem/RHSG1005020-SP25-8 HTTP/1.1" 200 1415 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:28 +1000] "GET /api/getItem/LWFB2001KG HTTP/1.1" 200 1309 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:28 +1000] "GET /api/getItem/RHSG10050208 HTTP/1.1" 200 1617 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:28 +1000] "GET /api/getItem/LWFB2001KG HTTP/1.1" 200 1309 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:28 +1000] "GET /api/getItem/RHSG10050258 HTTP/1.1" 200 1170 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:19:34:28 +1000] "GET /api/getItem/LWFB2001KG HTTP/1.1" 200 1309 "-" "RestSharp/106.13.0.0" == corpkicks/2025.corporatekicks.com.au == -- top paths -- 1 GET /v4on3fxtoo5z5cq0yCdefault.php 1 GET /CAE-2.5.php 1 GET /8xmlr9ommwdy3ro8dCdefault.php 1 GET /0.php -- last 10 requests -- 4.204.224.164 - - [05/Sep/2026:19:33:00 +1000] "GET /CAE-2.5.php HTTP/1.1" 301 294 "-" "-" 4.204.224.164 - - [05/Sep/2026:19:33:02 +1000] "GET /0.php HTTP/1.1" 301 288 "-" "-" 4.204.224.164 - - [05/Sep/2026:19:33:03 +1000] "GET /v4on3fxtoo5z5cq0yCdefault.php HTTP/1.1" 301 312 "-" "-" 4.204.224.164 - - [05/Sep/2026:19:33:05 +1000] "GET /8xmlr9ommwdy3ro8dCdefault.php HTTP/1.1" 301 312 "-" "-" --- last 30 samples (run-up to this incident) --- 2026-09-05 19:33:01 L1=1.43 run=2 blk=1 avail=9428 (68.4%) swpO=30 conn=1 myRSS=1863 php=5 phpRSS=242 ap=18 tcp=29 topuser=madalice 2026-09-05 19:33:01 L1=1.43 run=2 blk=1 avail=9524 (69.1%) swpO=30 conn=1 myRSS=1863 php=5 phpRSS=242 ap=18 tcp=29 topuser=madalice 2026-09-05 19:33:11 L1=1.36 run=2 blk=1 avail=9355 (67.9%) swpO=0 conn=1 myRSS=1863 php=6 phpRSS=473 ap=15 tcp=28 topuser=corpkicks 2026-09-05 19:33:11 L1=1.36 run=3 blk=1 avail=9350 (67.9%) swpO=0 conn=1 myRSS=1863 php=6 phpRSS=477 ap=15 tcp=28 topuser=corpkicks 2026-09-05 19:33:21 L1=1.31 run=2 blk=1 avail=9463 (68.7%) swpO=0 conn=1 myRSS=1863 php=6 phpRSS=355 ap=16 tcp=26 topuser=madalice 2026-09-05 19:33:21 L1=1.31 run=3 blk=1 avail=9460 (68.7%) swpO=0 conn=1 myRSS=1863 php=6 phpRSS=360 ap=16 tcp=26 topuser=madalice 2026-09-05 19:33:31 L1=1.33 run=1 blk=1 avail=9556 (69.4%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=242 ap=15 tcp=22 topuser=madalice 2026-09-05 19:33:31 L1=1.33 run=2 blk=0 avail=9555 (69.4%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=242 ap=15 tcp=22 topuser=madalice 2026-09-05 19:33:41 L1=1.28 run=2 blk=1 avail=9661 (70.1%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=155 ap=14 tcp=25 topuser=aaa 2026-09-05 19:33:41 L1=1.28 run=4 blk=1 avail=9656 (70.1%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=162 ap=14 tcp=25 topuser=aaa 2026-09-05 19:33:51 L1=1.32 run=1 blk=1 avail=9585 (69.6%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=220 ap=14 tcp=26 topuser=madalice 2026-09-05 19:33:51 L1=1.32 run=3 blk=1 avail=9585 (69.6%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=220 ap=14 tcp=26 topuser=madalice 2026-09-05 19:34:01 L1=1.27 run=1 blk=1 avail=9600 (69.7%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=220 ap=13 tcp=20 topuser=madalice 2026-09-05 19:34:02 L1=1.27 run=1 blk=1 avail=9600 (69.7%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=220 ap=13 tcp=20 topuser=madalice 2026-09-05 19:34:12 L1=1.23 run=1 blk=1 avail=9712 (70.5%) swpO=0 conn=1 myRSS=1863 php=4 phpRSS=86 ap=13 tcp=21 topuser=evolution 2026-09-05 19:34:12 L1=1.23 run=2 blk=1 avail=9711 (70.5%) swpO=0 conn=2 myRSS=1863 php=4 phpRSS=86 ap=13 tcp=21 topuser=evolution 2026-09-05 19:34:22 L1=1.27 run=1 blk=1 avail=9571 (69.5%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=245 ap=14 tcp=27 topuser=madalice 2026-09-05 19:34:22 L1=1.27 run=2 blk=1 avail=9569 (69.5%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=245 ap=14 tcp=27 topuser=madalice 2026-09-05 19:34:32 L1=1.31 run=2 blk=1 avail=9507 (69.0%) swpO=1 conn=2 myRSS=1863 php=6 phpRSS=320 ap=17 tcp=24 topuser=madalice 2026-09-05 19:34:32 L1=1.31 run=3 blk=1 avail=9504 (69.0%) swpO=1 conn=2 myRSS=1863 php=6 phpRSS=322 ap=17 tcp=24 topuser=madalice 2026-09-05 19:34:42 L1=1.26 run=2 blk=1 avail=9517 (69.1%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=245 ap=15 tcp=25 topuser=madalice 2026-09-05 19:34:42 L1=1.26 run=3 blk=1 avail=9563 (69.4%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=245 ap=15 tcp=25 topuser=madalice 2026-09-05 19:34:52 L1=1.22 run=2 blk=1 avail=9561 (69.4%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=250 ap=14 tcp=25 topuser=madalice 2026-09-05 19:34:52 L1=1.22 run=2 blk=1 avail=9560 (69.4%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=250 ap=14 tcp=25 topuser=madalice 2026-09-05 19:35:02 L1=1.26 run=3 blk=1 avail=9534 (69.2%) swpO=0 conn=2 myRSS=1863 php=5 phpRSS=246 ap=13 tcp=24 topuser=madalice 2026-09-05 19:35:02 L1=1.26 run=6 blk=1 avail=9535 (69.2%) swpO=0 conn=1 myRSS=1863 php=5 phpRSS=246 ap=13 tcp=24 topuser=madalice 2026-09-05 19:35:12 L1=1.29 run=4 blk=1 avail=9356 (67.9%) swpO=1 conn=2 myRSS=1863 php=6 phpRSS=388 ap=13 tcp=24 topuser=madalice 2026-09-05 19:35:12 L1=1.29 run=3 blk=1 avail=9355 (67.9%) swpO=1 conn=1 myRSS=1863 php=6 phpRSS=390 ap=13 tcp=24 topuser=madalice 2026-09-05 19:35:22 L1=1.40 run=22 blk=1 avail=8192 (59.5%) swpO=3 conn=16 myRSS=1863 php=19 phpRSS=1972 ap=28 tcp=40 topuser=otterfencing 2026-09-05 19:35:22 L1=1.40 run=20 blk=1 avail=7944 (57.7%) swpO=3 conn=15 myRSS=1863 php=19 phpRSS=1984 ap=28 tcp=40 topuser=otterfencing