INCIDENT 2026-08-15 02:31:26 trigger: php_rss=1818MB ================================================================== load: 3.32 2.00 1.85 16/634 2387940 ncpu:10 mem_avail: 5526MB/11729MB (47.1%) swap_used:1130MB swapout:10/s blocked(D):0 running:19 iowait:0.3% steal:0.0% php: 11 procs / 1818MB heaviest user: madalice (9 procs, 1687MB) apache: 45 procs tcp_estab: 45 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2242 madalice 9 1710 root 240 595 nobody 44 532 netdata 8 263 cpanellogin 1 60 evolution 5 43 kurri 3 37 mailnull 4 28 dovenull 6 28 wp-toolkit 6 20 mailman 20 15 lowrange 1 13 infinium 1 13 zabbix 1 11 dovecot 3 8 named 1 7 nscd 1 3 dbus 1 3 cpanelconnecttrack 1 3 --- php workers by user --- USER PROCS RSS_MB madalice 9 1539 root 1 90 evolution 1 39 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 4728 MB Active(file): 3634 MB Inactive(file): 813 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 1182 MB CommitLimit: 9960 MB Committed_AS: 8253 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2342983 root 01:20:10 9656 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 1420296 1 mysql 2296572 11.5 Ssl 1-00:52:25 /usr/sbin/mariadbd 2387883 2387876 madalice 268744 118 S 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2387882 2387875 madalice 253540 116 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2387879 2387872 madalice 246648 114 S 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2387885 2387862 madalice 245196 111 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2387881 2387874 madalice 230624 109 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1220 1 netdata 214400 2.5 Ssl 46-16:21:28 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2342760 2342758 root 179312 0.3 S 01:20:26 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/gather_update_log_stats --logfile /var/cpanel/updatelogs/last --upload 2387904 2387878 madalice 148432 121 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2387905 2387594 madalice 137308 110 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2370159 2342760 root 92692 98.0 R 31:10 /usr/local/cpanel/3rdparty/bin/php /opt/cpanel/ea-wappspector/wappspector.phar --max=1 --json --depth=9 /home/evolution 2387910 2387888 madalice 70652 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2357659 1353308 cpanellogin 61524 0.0 S 52:03 /usr/local/cpanel/base/show_template.stor default_login_theme cpanel server_locale en cpanel_locale docroot /usr/local/cpanel/base 2223 1201 root 55476 0.0 Sl 46-16:21:27 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 2387911 2387890 evolution 40800 0.0 S 00:00 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 1353308 1 root 40672 0.0 Ss 23-23:49:58 cpsrvd (SSL) - waiting for connections 1469 1 root 38144 0.1 Ss 46-16:21:27 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 2381704 1 root 33132 0.0 Ss 09:13 lfd - sleeping 4922 2431 netdata 32928 0.1 Sl 46-16:21:23 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2382132 1276780 nobody 19964 0.0 S 08:42 /usr/sbin/httpd -k start 2385506 1276780 nobody 19860 0.0 S 04:17 /usr/sbin/httpd -k start 2385903 1276780 nobody 19740 0.0 S 03:38 /usr/sbin/httpd -k start 2385505 1276780 nobody 19612 0.0 S 04:17 /usr/sbin/httpd -k start 2386040 1276780 nobody 19376 0.0 S 03:25 /usr/sbin/httpd -k start 2385564 1276780 nobody 19008 0.0 S 04:12 /usr/sbin/httpd -k start 2386039 1276780 nobody 18584 0.0 S 03:25 /usr/sbin/httpd -k start 2385191 1276780 nobody 17736 0.0 S 04:51 /usr/sbin/httpd -k start 2386933 1276780 nobody 17380 0.0 S 01:49 /usr/sbin/httpd -k start 2387873 1276780 nobody 17200 1.0 S 00:02 /usr/sbin/httpd -k start 2387599 1276780 nobody 17012 0.0 S 00:42 /usr/sbin/httpd -k start 2387594 1276780 nobody 16348 0.0 S 00:44 /usr/sbin/httpd -k start 159820 1 root 16136 0.0 Ss 25-16:44:36 /usr/lib/systemd/systemd-journald 2387866 1276780 nobody 15772 0.0 S 00:03 /usr/sbin/httpd -k start 2387863 1276780 nobody 15736 0.0 S 00:04 /usr/sbin/httpd -k start 2387854 1276780 nobody 15716 0.0 S 00:05 /usr/sbin/httpd -k start 2387876 1276780 nobody 15096 0.0 R 00:02 /usr/sbin/httpd -k start 2387888 1276780 nobody 15084 0.0 S 00:01 /usr/sbin/httpd -k start 2387872 1276780 nobody 15064 0.5 S 00:02 /usr/sbin/httpd -k start 2387878 1276780 nobody 15060 0.0 S 00:02 /usr/sbin/httpd -k start 2387875 1276780 nobody 15000 0.0 S 00:02 /usr/sbin/httpd -k start 2387862 1276780 nobody 14980 0.2 S 00:04 /usr/sbin/httpd -k start 2387871 1276780 nobody 14976 0.0 S 00:02 /usr/sbin/httpd -k start 2387877 1276780 nobody 14956 0.0 S 00:02 /usr/sbin/httpd -k start 2387867 1276780 nobody 14948 0.0 S 00:03 /usr/sbin/httpd -k start 2387868 1276780 nobody 14944 0.0 S 00:03 /usr/sbin/httpd -k start 2387874 1276780 nobody 14940 0.0 S 00:02 /usr/sbin/httpd -k start 185104 2431 netdata 14900 2.0 Rl 2-05:57:46 /usr/libexec/netdata/plugins.d/apps.plugin 1 2387890 1276780 nobody 14864 0.0 S 00:01 /usr/sbin/httpd -k start 2387865 1276780 nobody 14816 0.0 S 00:03 /usr/sbin/httpd -k start 1601939 1471 root 14424 0.0 S 19:18:28 cPhulkd - dbprocessor 1463 1 root 13908 0.0 Ss 46-16:21:27 tailwatchd 2347792 3828812 lowrange 13476 0.0 S 01:11:20 dovecot/imap 2337342 3828812 infinium 13448 0.0 S 01:25:37 dovecot/imap 2368218 3828812 kurri 12932 0.0 S 33:45 dovecot/imap 2385750 3828812 kurri 12796 0.0 S 03:53 dovecot/imap 2379224 3828812 kurri 12736 0.0 S 13:50 dovecot/imap 1471 1 root 12532 0.0 Ss 46-16:21:27 cPhulkd - processor 1204 1 zabbix 11600 0.2 Ssl 46-16:21:28 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 2385263 3828812 root 11348 0.0 S 04:39 dovecot/auth -w 2387886 1471 root 11180 0.0 S 00:02 cPhulkd - processor - http socket --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3983192 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/portal.evolutionerp.com.au-ssl_log 19 madalice/madalice.com.au-ssl_log 6 corpkicks/corporatekicks.com.au-ssl_log 4 alison/alisonarts.com.au-ssl_log 3 (total vhosts taking traffic: 4) --- top client IPs across all vhosts (last 3 min) --- 19 203.54.180.238 3 20.215.70.187 3 136.70.133.108 1 34.56.53.225 1 34.196.51.17 1 216.73.217.86 1 216.73.216.103 1 216.244.66.236 1 118.127.40.55 1 103.225.137.230 --- top request paths across all vhosts (last 3 min) --- 19 GET /heartbeat.php 1 POST /wp-cron.php 1 HEAD / 1 GET /Zoo.php 1 GET /sitemap_index.xml 1 GET /robots.txt 1 GET /product/untitled-product-2/ 1 GET /product-category/customised-shoes/wedding/ 1 GET /new/ 1 GET /explorer/index_.php 1 GET /buy.php 1 GET /.aws/credentials 1 GET /.aws/config 1 GET / --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 19 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [15/Aug/2026:02:29:45 +1000] "GET /heartbeat.php?sentAt=1786724986&bytes=106,127,66,122&_=1786712383516 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:29:53 +1000] "GET /heartbeat.php?sentAt=1786724993&bytes=106,127,66,129&_=1786683793549 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:29:55 +1000] "GET /heartbeat.php?sentAt=1786724996&bytes=106,127,66,132&_=1786712383517 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:03 +1000] "GET /heartbeat.php?sentAt=1786725003&bytes=106,127,66,139&_=1786683793550 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:05 +1000] "GET /heartbeat.php?sentAt=1786725006&bytes=106,127,66,142&_=1786712383518 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:13 +1000] "GET /heartbeat.php?sentAt=1786725013&bytes=106,127,66,149&_=1786683793551 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:15 +1000] "GET /heartbeat.php?sentAt=1786725016&bytes=106,127,66,152&_=1786712383519 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:23 +1000] "GET /heartbeat.php?sentAt=1786725023&bytes=106,127,66,159&_=1786683793552 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:25 +1000] "GET /heartbeat.php?sentAt=1786725026&bytes=106,127,66,162&_=1786712383520 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:02:30:33 +1000] "GET /heartbeat.php?sentAt=1786725033&bytes=106,127,66,169&_=1786683793553 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" == madalice/madalice.com.au-ssl_log == -- top paths -- 1 GET /sitemap_index.xml 1 GET /product-category/customised-shoes/wedding/ 1 GET /new/ 1 GET /.aws/credentials 1 GET /.aws/config 1 GET / -- last 10 requests -- 103.225.137.230 - - [15/Aug/2026:02:29:05 +1000] "GET /product-category/customised-shoes/wedding/?srsltid=AfmBOoqmTh0rYot_-uL9FksO1_ynAT-uJj958J_v_XLcc6QaKfK9WaHI HTTP/1.1" 200 33600 "https://madalice.com.au" "Mozilla/5. 216.244.66.236 - - [15/Aug/2026:02:30:30 +1000] "GET /new/ HTTP/1.1" 301 1 "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)" 216.73.217.86 - - [15/Aug/2026:02:30:55 +1000] "GET /sitemap_index.xml HTTP/1.1" 200 439 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" 136.70.133.108 - - [15/Aug/2026:02:31:15 +1000] "GET / HTTP/1.1" 301 1 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Mobile Safari/537.36" 136.70.133.108 - - [15/Aug/2026:02:31:22 +1000] "GET /.aws/config HTTP/1.1" 403 13 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 136.70.133.108 - - [15/Aug/2026:02:31:22 +1000] "GET /.aws/credentials HTTP/1.1" 403 13 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" == corpkicks/corporatekicks.com.au-ssl_log == -- top paths -- 1 POST /wp-cron.php 1 HEAD / 1 GET /robots.txt 1 GET /product/untitled-product-2/ -- last 10 requests -- 34.196.51.17 - - [15/Aug/2026:02:29:01 +1000] "HEAD / HTTP/1.1" 200 - "https://corporatekicks.com.au/" "WPMUDEV Uptime Monitor 5.0 (https://wpmudev.com)" 118.127.40.55 - - [15/Aug/2026:02:29:02 +1000] "POST /wp-cron.php?doing_wp_cron=1786724942.8529899120330810546875 HTTP/1.1" 200 1 "-" "WordPress/7.0.4; https://corporatekicks.com.au" 34.56.53.225 - - [15/Aug/2026:02:29:07 +1000] "GET /product/untitled-product-2/ HTTP/1.1" 200 37512 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537. 216.73.216.103 - - [15/Aug/2026:02:30:38 +1000] "GET /robots.txt HTTP/1.1" 200 146 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" --- last 30 samples (run-up to this incident) --- 2026-08-15 02:28:58 L1=1.41 run=2 blk=0 avail=7328 (62.5%) swpO=1 conn=1 myRSS=2243 php=1 phpRSS=90 ap=17 tcp=21 topuser=root 2026-08-15 02:29:05 L1=1.38 run=4 blk=0 avail=7146 (60.9%) swpO=1 conn=1 myRSS=2243 php=3 phpRSS=319 ap=18 tcp=23 topuser=alison 2026-08-15 02:29:08 L1=1.43 run=3 blk=0 avail=7096 (60.5%) swpO=1 conn=1 myRSS=2243 php=2 phpRSS=337 ap=18 tcp=23 topuser=corpkicks 2026-08-15 02:29:15 L1=1.39 run=2 blk=0 avail=7307 (62.3%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=18 tcp=21 topuser=root 2026-08-15 02:29:18 L1=1.36 run=2 blk=0 avail=7315 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=17 tcp=20 topuser=root 2026-08-15 02:29:25 L1=1.33 run=2 blk=0 avail=7321 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=16 topuser=root 2026-08-15 02:29:28 L1=1.31 run=2 blk=0 avail=7329 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=15 tcp=19 topuser=root 2026-08-15 02:29:35 L1=1.28 run=2 blk=0 avail=7330 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=15 tcp=21 topuser=root 2026-08-15 02:29:38 L1=1.26 run=2 blk=0 avail=7324 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=20 topuser=root 2026-08-15 02:29:45 L1=1.24 run=2 blk=0 avail=7322 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=21 topuser=root 2026-08-15 02:29:49 L1=1.22 run=3 blk=0 avail=7319 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=20 topuser=root 2026-08-15 02:29:55 L1=1.28 run=2 blk=0 avail=7334 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=15 tcp=15 topuser=root 2026-08-15 02:29:59 L1=1.26 run=2 blk=0 avail=7334 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=19 topuser=root 2026-08-15 02:30:05 L1=1.24 run=2 blk=0 avail=7300 (62.2%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=159 ap=14 tcp=19 topuser=root 2026-08-15 02:30:09 L1=1.22 run=2 blk=0 avail=7295 (62.2%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=159 ap=14 tcp=19 topuser=root 2026-08-15 02:30:15 L1=1.20 run=2 blk=0 avail=7308 (62.3%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=160 ap=14 tcp=16 topuser=root 2026-08-15 02:30:19 L1=1.18 run=4 blk=0 avail=7147 (60.9%) swpO=0 conn=1 myRSS=2243 php=3 phpRSS=347 ap=14 tcp=18 topuser=madalice 2026-08-15 02:30:25 L1=1.17 run=2 blk=0 avail=7343 (62.6%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=15 topuser=root 2026-08-15 02:30:29 L1=1.16 run=4 blk=0 avail=7332 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=15 topuser=root 2026-08-15 02:30:35 L1=1.22 run=2 blk=0 avail=7340 (62.6%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=18 topuser=root 2026-08-15 02:30:39 L1=1.20 run=3 blk=0 avail=7315 (62.4%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=146 ap=14 tcp=18 topuser=root 2026-08-15 02:30:45 L1=1.27 run=6 blk=0 avail=7318 (62.4%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=130 ap=16 tcp=23 topuser=root 2026-08-15 02:30:49 L1=1.25 run=3 blk=0 avail=7322 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=19 topuser=root 2026-08-15 02:30:55 L1=1.23 run=4 blk=0 avail=7274 (62.0%) swpO=0 conn=1 myRSS=2243 php=3 phpRSS=197 ap=16 tcp=22 topuser=root 2026-08-15 02:30:59 L1=1.45 run=3 blk=0 avail=7319 (62.4%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=16 tcp=19 topuser=root 2026-08-15 02:31:05 L1=1.54 run=2 blk=0 avail=7316 (62.4%) swpO=0 conn=1 myRSS=2243 php=2 phpRSS=130 ap=15 tcp=19 topuser=root 2026-08-15 02:31:09 L1=1.54 run=2 blk=0 avail=7330 (62.5%) swpO=0 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=16 topuser=root 2026-08-15 02:31:16 L1=1.70 run=4 blk=0 avail=7282 (62.1%) swpO=0 conn=1 myRSS=2243 php=3 phpRSS=199 ap=14 tcp=16 topuser=root 2026-08-15 02:31:19 L1=1.70 run=2 blk=0 avail=7344 (62.6%) swpO=1 conn=1 myRSS=2243 php=1 phpRSS=90 ap=14 tcp=17 topuser=root 2026-08-15 02:31:26 L1=3.32 run=19 blk=0 avail=5526 (47.1%) swpO=10 conn=11 myRSS=2242 php=11 phpRSS=1818 ap=45 tcp=45 topuser=madalice