INCIDENT 2026-09-05 07:46:04 trigger: php_rss=1557MB ================================================================== load: 1.94 2.06 2.14 2/690 551876 ncpu:10 mem_avail: 6730MB/13777MB (48.8%) swap_used:2199MB swapout:0/s blocked(D):1 running:2 iowait:9.6% steal:0.0% php: 13 procs / 1557MB heaviest user: madalice (9 procs, 1426MB) apache: 17 procs tcp_estab: 26 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2980 madalice 9 1432 root 247 1381 nobody 16 291 netdata 8 259 evolution 11 124 dovenull 6 65 wp-toolkit 3 53 cpanellogin 1 53 kurri 2 27 dovecot 3 27 mailman 20 26 lowrange 1 15 aaa 1 14 zabbix 1 10 polkitd 1 7 named 1 5 mailnull 2 5 nscd 1 3 dbus 1 2 --- php workers by user --- USER PROCS RSS_MB madalice 9 1435 evolution 3 103 root 1 28 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5971 MB Active(file): 4379 MB Inactive(file): 1663 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 547 MB CommitLimit: 10984 MB Committed_AS: 8706 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 3926885 root 4-05:23:38 4832 /usr/bin/imunify-agent-proxy 4109079 root 03:01:16 3136 /opt/ai-bolit/wrapper /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --mem 4109105 root 03:01:16 541516 /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --memory 2048M --progress --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 3052212 11.8 Ssl 2-17:31:53 /usr/sbin/mariadbd 4109105 4109079 root 541516 8.3 Sl 03:01:16 /opt/rustbolit/rustbolit --smart --deobfuscate --avdb /var/imunify360/files/sigs/v1/aibolit/ai-bolit-hoster-full.db --no-html --memory 536420 2601907 madalice 237096 7.8 R 01:53 php-fpm: pool madalice_com_au 1542443 1 netdata 193708 2.6 Ssl 2-05:23:43 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 3643624 3626973 root 168464 0.0 S 05:19:15 spamd child 495842 2601907 madalice 161052 7.2 S 15:49 php-fpm: pool madalice_com_au 536487 2601907 madalice 160992 8.5 S 01:45 php-fpm: pool madalice_com_au 536493 2601907 madalice 160820 8.0 S 01:43 php-fpm: pool madalice_com_au 515304 2601907 madalice 160652 6.9 S 09:38 php-fpm: pool madalice_com_au 536530 2601907 madalice 159060 6.9 S 01:39 php-fpm: pool madalice_com_au 536397 2601907 madalice 158936 6.8 S 01:57 php-fpm: pool madalice_com_au 496129 2601907 madalice 154584 7.3 S 15:26 php-fpm: pool madalice_com_au 3626973 1 root 127652 0.0 Ss 05:22:50 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 536452 2601907 madalice 124220 5.2 S 01:49 php-fpm: pool madalice_com_au 3643625 3626973 root 90276 0.0 S 05:19:15 spamd child 509236 509098 root 61236 0.0 Sl 8-05:26:24 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 3127357 3127312 evolution 54476 0.0 Ss 07:46:04 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero_payment_cron.php 498558 1353308 cpanellogin 54388 0.0 S 12:10 /usr/local/cpanel/base/show_template.stor server_locale en default_login_theme cpanel cpanel_locale docroot /usr/local/cpanel/base 3127358 3127320 evolution 51296 0.0 Ss 07:46:04 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero2.php 509266 1 root 46040 0.0 Ss 8-05:26:24 /usr/lib/systemd/systemd-journald 1353308 1 root 41880 0.1 Ss 45-05:04:50 cpsrvd (SSL) - waiting for connections 1542981 1542551 netdata 41144 0.1 Sl 2-05:23:39 /usr/libexec/netdata/plugins.d/go.d.plugin 1 1469 1 root 34496 0.1 Ss 67-21:36:19 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1542487 1 wp-toolkit 29348 0.0 Ss 2-05:23:43 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 2601907 1 root 28692 0.0 Ss 1-05:22:39 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 3626082 1 root 27832 0.0 Ss 05:23:50 lfd - sleeping 1463 1 root 24052 0.0 Ss 67-21:36:19 tailwatchd 516315 506832 nobody 23176 0.0 S 09:01 /usr/sbin/httpd -k start 516235 506832 nobody 22820 0.0 S 09:03 /usr/sbin/httpd -k start 518214 506832 nobody 22540 0.1 S 06:57 /usr/sbin/httpd -k start 536608 506832 nobody 22008 0.0 S 01:32 /usr/sbin/httpd -k start 536584 506832 nobody 21448 0.0 S 01:35 /usr/sbin/httpd -k start 536403 506832 nobody 21048 0.0 S 01:56 /usr/sbin/httpd -k start 536577 506832 nobody 21000 0.0 S 01:36 /usr/sbin/httpd -k start 536600 506832 nobody 20604 0.0 S 01:33 /usr/sbin/httpd -k start 549563 506832 nobody 20196 0.0 S 00:53 /usr/sbin/httpd -k start 536572 506832 nobody 20108 0.0 S 01:37 /usr/sbin/httpd -k start 546921 506832 nobody 19696 0.2 S 00:55 /usr/sbin/httpd -k start 548277 506832 nobody 19620 0.0 S 00:54 /usr/sbin/httpd -k start 548273 506832 nobody 19516 0.1 S 00:54 /usr/sbin/httpd -k start 2602226 1 root 19304 0.1 Ssl 1-05:22:24 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P 1542531 1 wp-toolkit 19264 0.0 Ss 2-05:23:42 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script scheduled-tasks-executor.php 549567 506832 nobody 19020 0.0 S 00:53 /usr/sbin/httpd -k start 1542982 1542551 netdata 18520 2.2 Sl 2-05:23:39 /usr/libexec/netdata/plugins.d/apps.plugin 1 306499 1471 root 16440 0.0 S 01:07:01 cPhulkd - dbprocessor 1471 1 root 16352 0.0 Ss 67-21:36:19 cPhulkd - processor 1542141 1 root 15420 0.0 Ss 2-05:23:46 sw-engine-fpm: master process (/etc/sw-engine/sw-engine-fpm.conf) 3539428 508511 lowrange 15372 0.0 S 05:49:57 dovecot/imap 534505 508511 aaa 14972 0.0 S 04:48 dovecot/imap 536903 508511 kurri 14588 0.0 S 01:08 dovecot/imap 3141944 508511 kurri 13864 0.0 S 07:45:42 dovecot/imap 690505 508511 dovenull 13276 0.0 S 19:13:15 dovecot/imap-login 690514 508511 dovecot 13276 0.0 S 19:13:15 dovecot/auth 690509 508511 dovenull 13068 0.0 S 19:13:15 dovecot/imap-login 506832 1 root 12356 0.0 Ss 8-05:26:57 /usr/sbin/httpd -k start 690511 508511 dovenull 11532 0.0 S 19:13:15 dovecot/pop3-login 690507 508511 dovenull 11024 0.0 S 19:13:15 dovecot/pop3-login 1 0 root 10980 0.1 Ss 67-21:36:28 /usr/lib/systemd/systemd --system --deserialize 22 483898 1 zabbix 10748 0.2 Ssl 8-05:27:41 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 690506 508511 dovenull 9476 0.0 S 19:13:15 dovecot/managesieve-login --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 10752914 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min madalice/madalice.com.au-ssl_log 311 evolution/app.evolutionerp.com.au-ssl_log 104 evolution/portal.evolutionerp.com.au-ssl_log 1 (total vhosts taking traffic: 3) --- top client IPs across all vhosts (last 3 min) --- 189 195.178.110.22 117 139.28.219.70 104 40.115.94.245 2 114.119.131.88 1 203.54.180.238 1 114.119.159.70 1 114.119.146.16 1 114.119.143.14 --- top request paths across all vhosts (last 3 min) --- 48 GET /api/getItem/LWFBW2001KG 35 GET /api/getItem/LWHG2001KG 16 GET /api/getItem/LWPB2001KG 6 GET /backup.zip 3 GET /www.zip 3 GET /www.tar.gz 3 GET /www.sql 3 GET /wwwroot.zip 3 GET /wwwroot.tar.gz 3 GET /wp-content/uploads/dump.sql 3 GET /wp-content/uploads/database.sql 3 GET /wp-content/uploads/backup.sql 3 GET /wp-content/mysql.sql 3 GET /wp-content/database.sql 3 GET /web.zip 3 GET /website.zip 3 GET /users.sql 3 GET /sql.sql 3 GET /site.zip 3 GET /site.sql --- busiest 3 vhosts: top paths + last 10 requests --- == madalice/madalice.com.au-ssl_log == -- top paths -- 6 GET /backup.zip 3 GET /www.zip 3 GET /www.tar.gz 3 GET /www.sql 3 GET /wwwroot.zip 3 GET /wwwroot.tar.gz 3 GET /wp-content/uploads/dump.sql 3 GET /wp-content/uploads/database.sql 3 GET /wp-content/uploads/backup.sql 3 GET /wp-content/mysql.sql 3 GET /wp-content/database.sql 3 GET /web.zip -- last 10 requests -- 139.28.219.70 - - [05/Sep/2026:07:45:39 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-image/wp-content/plugins/woocommerce/assets/client/blocks/ HTTP/1.1" 404 36659 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:41 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-meta/ HTTP/1.1" 200 322 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:41 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-meta//wp-content/plugins/woocommerce/assets/client/blocks/ HTTP/1.1" 301 1 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:42 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-meta/wp-content/plugins/woocommerce/assets/client/blocks/ HTTP/1.1" 404 36662 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:44 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-new-style.asset.php HTTP/1.1" 200 1 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:45 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-new.asset.php HTTP/1.1" 200 1 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:45 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-new/ HTTP/1.1" 200 322 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:45 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-new//wp-content/plugins/woocommerce/assets/client/blocks/ HTTP/1.1" 301 1 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:47 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-new/wp-content/plugins/woocommerce/assets/client/blocks/ HTTP/1.1" 404 36661 "-" "-" 139.28.219.70 - - [05/Sep/2026:07:45:48 +1000] "GET /wp-content/plugins/woocommerce/assets/client/blocks/product-on-sale-style.asset.php HTTP/1.1" 200 1 "-" "-" == evolution/app.evolutionerp.com.au-ssl_log == -- top paths -- 48 GET /api/getItem/LWFBW2001KG 35 GET /api/getItem/LWHG2001KG 16 GET /api/getItem/LWPB2001KG 1 GET /api/getItem/H4020 1 GET /api/getItem/H3235 1 GET /api/getItem/H3220 1 GET /api/getItem/H2525 1 GET /api/getItem/H2020 -- last 10 requests -- 40.115.94.245 - - [05/Sep/2026:07:45:17 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:17 +1000] "GET /api/getItem/H2020 HTTP/1.1" 200 1488 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:17 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:18 +1000] "GET /api/getItem/H2525 HTTP/1.1" 200 1332 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:18 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:18 +1000] "GET /api/getItem/H3220 HTTP/1.1" 200 1480 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:18 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:19 +1000] "GET /api/getItem/H3235 HTTP/1.1" 404 107 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:19 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:07:45:19 +1000] "GET /api/getItem/H4020 HTTP/1.1" 404 107 "-" "RestSharp/106.13.0.0" == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 1 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [05/Sep/2026:07:44:07 +1000] "GET /heartbeat.php?sentAt=1788558247&bytes=106,155,59,167&_=1788436625391 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" --- last 30 samples (run-up to this incident) --- 2026-09-05 07:43:36 L1=1.87 run=4 blk=1 avail=7023 (51.0%) swpO=0 conn=4 myRSS=2985 php=13 phpRSS=1251 ap=19 tcp=33 topuser=madalice 2026-09-05 07:43:44 L1=1.96 run=4 blk=1 avail=6795 (49.3%) swpO=0 conn=4 myRSS=2985 php=13 phpRSS=1478 ap=19 tcp=29 topuser=madalice 2026-09-05 07:43:46 L1=1.88 run=3 blk=1 avail=6798 (49.3%) swpO=0 conn=4 myRSS=2985 php=13 phpRSS=1484 ap=19 tcp=29 topuser=madalice 2026-09-05 07:43:54 L1=1.89 run=3 blk=1 avail=6884 (50.0%) swpO=0 conn=4 myRSS=2985 php=13 phpRSS=1396 ap=18 tcp=33 topuser=madalice 2026-09-05 07:43:56 L1=1.82 run=1 blk=1 avail=6962 (50.5%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1326 ap=17 tcp=34 topuser=madalice 2026-09-05 07:44:04 L1=1.99 run=1 blk=1 avail=6889 (50.0%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1396 ap=17 tcp=36 topuser=madalice 2026-09-05 07:44:06 L1=1.91 run=2 blk=1 avail=6899 (50.1%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1389 ap=18 tcp=37 topuser=madalice 2026-09-05 07:44:14 L1=1.92 run=3 blk=1 avail=7111 (51.6%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1124 ap=20 tcp=39 topuser=madalice 2026-09-05 07:44:16 L1=1.93 run=1 blk=1 avail=7363 (53.4%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=853 ap=20 tcp=37 topuser=madalice 2026-09-05 07:44:24 L1=1.85 run=1 blk=1 avail=7321 (53.1%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=884 ap=19 tcp=36 topuser=madalice 2026-09-05 07:44:26 L1=1.78 run=2 blk=1 avail=7298 (53.0%) swpO=2 conn=3 myRSS=2985 php=15 phpRSS=968 ap=19 tcp=43 topuser=madalice 2026-09-05 07:44:34 L1=1.96 run=2 blk=1 avail=6965 (50.6%) swpO=2 conn=3 myRSS=2985 php=13 phpRSS=1165 ap=37 tcp=52 topuser=madalice 2026-09-05 07:44:36 L1=1.97 run=1 blk=1 avail=6897 (50.1%) swpO=2 conn=3 myRSS=2985 php=13 phpRSS=1255 ap=37 tcp=49 topuser=madalice 2026-09-05 07:44:44 L1=1.97 run=3 blk=1 avail=6855 (49.8%) swpO=2 conn=3 myRSS=2985 php=13 phpRSS=1296 ap=33 tcp=50 topuser=madalice 2026-09-05 07:44:46 L1=1.89 run=2 blk=1 avail=6876 (49.9%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1278 ap=33 tcp=51 topuser=madalice 2026-09-05 07:44:54 L1=1.98 run=2 blk=1 avail=6802 (49.4%) swpO=0 conn=3 myRSS=2985 php=13 phpRSS=1398 ap=30 tcp=38 topuser=madalice 2026-09-05 07:44:56 L1=1.98 run=2 blk=1 avail=6847 (49.7%) swpO=0 conn=4 myRSS=2985 php=13 phpRSS=1347 ap=29 tcp=45 topuser=madalice 2026-09-05 07:45:04 L1=1.90 run=2 blk=1 avail=6770 (49.1%) swpO=0 conn=4 myRSS=2985 php=14 phpRSS=1453 ap=28 tcp=37 topuser=madalice 2026-09-05 07:45:06 L1=1.83 run=3 blk=1 avail=6799 (49.4%) swpO=0 conn=4 myRSS=2985 php=14 phpRSS=1449 ap=26 tcp=39 topuser=madalice 2026-09-05 07:45:14 L1=1.92 run=2 blk=1 avail=6778 (49.2%) swpO=163 conn=4 myRSS=2980 php=14 phpRSS=1394 ap=39 tcp=40 topuser=madalice 2026-09-05 07:45:16 L1=1.93 run=2 blk=1 avail=6716 (48.7%) swpO=163 conn=4 myRSS=2980 php=14 phpRSS=1471 ap=37 tcp=37 topuser=madalice 2026-09-05 07:45:24 L1=2.02 run=2 blk=1 avail=6779 (49.2%) swpO=0 conn=4 myRSS=2980 php=14 phpRSS=1442 ap=29 tcp=31 topuser=madalice 2026-09-05 07:45:26 L1=2.01 run=2 blk=1 avail=6860 (49.8%) swpO=0 conn=4 myRSS=2980 php=14 phpRSS=1372 ap=27 tcp=34 topuser=madalice 2026-09-05 07:45:34 L1=1.93 run=2 blk=1 avail=6864 (49.8%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1397 ap=19 tcp=30 topuser=madalice 2026-09-05 07:45:37 L1=2.02 run=1 blk=1 avail=6817 (49.5%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1451 ap=19 tcp=28 topuser=madalice 2026-09-05 07:45:44 L1=2.02 run=1 blk=1 avail=6813 (49.4%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1457 ap=18 tcp=29 topuser=madalice 2026-09-05 07:45:47 L1=2.02 run=1 blk=1 avail=6843 (49.7%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1429 ap=18 tcp=30 topuser=madalice 2026-09-05 07:45:54 L1=1.93 run=3 blk=1 avail=6923 (50.3%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1347 ap=18 tcp=26 topuser=madalice 2026-09-05 07:45:57 L1=1.93 run=1 blk=1 avail=6855 (49.8%) swpO=0 conn=3 myRSS=2980 php=13 phpRSS=1417 ap=18 tcp=28 topuser=madalice 2026-09-05 07:46:04 L1=1.94 run=2 blk=1 avail=6730 (48.8%) swpO=0 conn=4 myRSS=2980 php=13 phpRSS=1557 ap=17 tcp=26 topuser=madalice