INCIDENT 2026-08-15 01:29:15 trigger: php_rss=1664MB ================================================================== load: 3.79 2.04 1.67 18/654 2351915 ncpu:10 mem_avail: 5592MB/11729MB (47.7%) swap_used:1335MB swapout:664/s blocked(D):0 running:17 iowait:0.6% steal:0.0% php: 13 procs / 1664MB heaviest user: madalice (10 procs, 1467MB) apache: 75 procs tcp_estab: 59 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2217 madalice 12 1499 nobody 73 1086 root 239 996 netdata 8 264 vinesit 1 91 evolution 5 43 kurri 3 37 dovenull 6 28 lowrange 2 26 wp-toolkit 6 20 cpanellogin 1 20 mailman 20 14 infinium 1 13 zabbix 1 12 mailnull 2 10 dovecot 3 8 named 1 7 nscd 1 4 dbus 1 3 --- php workers by user --- USER PROCS RSS_MB madalice 12 1533 vinesit 1 91 root 1 68 evolution 1 39 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5005 MB Active(file): 3218 MB Inactive(file): 1245 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 800 MB CommitLimit: 9960 MB Committed_AS: 10041 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2342983 root 17:59 8104 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 1420296 1 mysql 2270696 11.9 Ssl 23:50:15 /usr/sbin/mariadbd 2351876 2351847 madalice 220424 70.3 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1220 1 netdata 214220 2.5 Ssl 46-15:19:18 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2351871 2351863 madalice 200680 47.2 R 00:04 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1451534 1444234 root 196000 0.0 S 22:52:09 spamd child 2351875 2351851 madalice 195120 63.6 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2351873 2351846 madalice 194212 64.3 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2342760 2342758 root 179428 1.4 S 18:16 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/gather_update_log_stats --logfile /var/cpanel/updatelogs/last --upload 2351877 2351841 madalice 178528 62.0 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2351880 2351862 madalice 156736 68.0 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2351879 2349406 madalice 151948 43.6 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2351878 2351849 madalice 124908 34.0 R 00:03 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1444234 1 root 97132 0.0 Ss 23:06:28 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 159820 1 root 89692 0.0 Ss 25-15:42:26 /usr/lib/systemd/systemd-journald 2351895 2348041 madalice 70448 27.0 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2343703 2342760 root 70268 98.6 R 16:52 /usr/local/cpanel/3rdparty/bin/php /opt/cpanel/ea-wappspector/wappspector.phar --max=1 --json --depth=9 /home/nrg 2351882 2350619 madalice 68236 33.0 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1910802 1444234 root 65324 0.0 S 12:00:42 spamd child 2351909 2351242 madalice 63856 19.0 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2351910 2351768 madalice 60544 17.0 S 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2223 1201 root 57956 0.0 Sl 46-15:19:17 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1353308 1 root 41892 0.0 Ss 23-22:47:48 cpsrvd (SSL) - waiting for connections 1469 1 root 41424 0.1 Ss 46-15:19:17 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 2351886 2350414 evolution 40624 3.0 S 00:01 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 4922 2431 netdata 34992 0.1 Sl 46-15:19:13 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2285972 1 root 31960 0.0 Ss 01:29:15 lfd - sleeping 2344241 1353308 cpanellogin 21108 0.0 S 15:53 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base server_locale en default_login_theme cpanel 2348041 1276780 nobody 20036 0.0 S 08:39 /usr/sbin/httpd -k start 2348975 1276780 nobody 19300 0.0 S 06:27 /usr/sbin/httpd -k start 2350196 1276780 nobody 19216 0.0 S 04:08 /usr/sbin/httpd -k start 2350411 1276780 nobody 17988 0.0 S 03:31 /usr/sbin/httpd -k start 2349406 1276780 nobody 17888 0.0 S 05:29 /usr/sbin/httpd -k start 2350414 1276780 nobody 17792 0.0 S 03:30 /usr/sbin/httpd -k start 2350619 1276780 nobody 17752 0.0 S 03:01 /usr/sbin/httpd -k start 2351811 1276780 nobody 17276 0.2 S 00:07 /usr/sbin/httpd -k start 2351810 1276780 nobody 17216 0.1 S 00:07 /usr/sbin/httpd -k start 2351813 1276780 nobody 17204 0.1 S 00:07 /usr/sbin/httpd -k start 2351834 1276780 nobody 17184 0.1 S 00:06 /usr/sbin/httpd -k start 2351814 1276780 nobody 17168 0.2 S 00:07 /usr/sbin/httpd -k start 2351812 1276780 nobody 17164 0.2 S 00:07 /usr/sbin/httpd -k start 2351856 1276780 nobody 17144 0.1 S 00:06 /usr/sbin/httpd -k start 2351809 1276780 nobody 17128 0.1 S 00:07 /usr/sbin/httpd -k start 2351855 1276780 nobody 17128 0.1 S 00:06 /usr/sbin/httpd -k start 2351816 1276780 nobody 17064 0.1 S 00:07 /usr/sbin/httpd -k start 2351836 1276780 nobody 17056 0.1 S 00:06 /usr/sbin/httpd -k start 2351860 1276780 nobody 17056 0.3 S 00:06 /usr/sbin/httpd -k start 2351242 1276780 nobody 17000 0.0 S 01:28 /usr/sbin/httpd -k start 2351150 1276780 nobody 16920 0.0 S 01:39 /usr/sbin/httpd -k start 2351850 1276780 nobody 16580 0.1 S 00:06 /usr/sbin/httpd -k start 2350202 1276780 nobody 16500 0.0 S 04:07 /usr/sbin/httpd -k start 2351861 1276780 nobody 15852 0.1 S 00:06 /usr/sbin/httpd -k start 2351765 1276780 nobody 15788 0.0 S 00:09 /usr/sbin/httpd -k start 2351757 1276780 nobody 15712 0.0 S 00:11 /usr/sbin/httpd -k start 2351761 1276780 nobody 15712 0.0 S 00:10 /usr/sbin/httpd -k start 2351767 1276780 nobody 15644 0.0 S 00:09 /usr/sbin/httpd -k start 2351808 1276780 nobody 15636 0.0 S 00:07 /usr/sbin/httpd -k start 2351837 1276780 nobody 15532 0.1 S 00:06 /usr/sbin/httpd -k start 2351766 1276780 nobody 15456 0.1 S 00:09 /usr/sbin/httpd -k start 2351841 1276780 nobody 15112 0.0 S 00:06 /usr/sbin/httpd -k start 2351862 1276780 nobody 15076 0.0 S 00:06 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3978586 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/portal.evolutionerp.com.au-ssl_log 22 madalice/madalice.com.au-ssl_log 15 (total vhosts taking traffic: 2) --- top client IPs across all vhosts (last 3 min) --- 22 203.54.180.238 13 34.169.205.61 1 2a03:2880:f802:7:: 1 18.215.112.101 --- top request paths across all vhosts (last 3 min) --- 22 GET /heartbeat.php 2 GET /z9x8c7v6b5-debug-trigger-www.madalice.com.au 2 GET /rclone.conf 2 GET /.git/config 1 GET /wp-json 1 GET /wp-content/plugins/ajax-search-for-woocommerce/includes/Integrations/Plugins/FilterEverything/ 1 GET /tag/xmas/ 1 GET /.git/HEAD 1 GET /.git-credentials 1 GET /.gitconfig 1 GET /.aws/credentials 1 GET /.aws/config 1 GET / --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 22 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [15/Aug/2026:01:28:03 +1000] "GET /heartbeat.php?sentAt=1786721283&bytes=106,127,52,3&_=1786683793178 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" " 203.54.180.238 - - [15/Aug/2026:01:28:05 +1000] "GET /heartbeat.php?sentAt=1786721286&bytes=106,127,52,6&_=1786712383146 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" " 203.54.180.238 - - [15/Aug/2026:01:28:13 +1000] "GET /heartbeat.php?sentAt=1786721293&bytes=106,127,52,13&_=1786683793179 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:15 +1000] "GET /heartbeat.php?sentAt=1786721296&bytes=106,127,52,16&_=1786712383147 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:23 +1000] "GET /heartbeat.php?sentAt=1786721303&bytes=106,127,52,23&_=1786683793180 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:25 +1000] "GET /heartbeat.php?sentAt=1786721306&bytes=106,127,52,26&_=1786712383148 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:33 +1000] "GET /heartbeat.php?sentAt=1786721313&bytes=106,127,52,33&_=1786683793181 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:35 +1000] "GET /heartbeat.php?sentAt=1786721316&bytes=106,127,52,36&_=1786712383149 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:43 +1000] "GET /heartbeat.php?sentAt=1786721323&bytes=106,127,52,43&_=1786683793182 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:01:28:45 +1000] "GET /heartbeat.php?sentAt=1786721326&bytes=106,127,52,46&_=1786712383150 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" == madalice/madalice.com.au-ssl_log == -- top paths -- 2 GET /z9x8c7v6b5-debug-trigger-www.madalice.com.au 2 GET /rclone.conf 2 GET /.git/config 1 GET /wp-json 1 GET /wp-content/plugins/ajax-search-for-woocommerce/includes/Integrations/Plugins/FilterEverything/ 1 GET /tag/xmas/ 1 GET /.git/HEAD 1 GET /.git-credentials 1 GET /.gitconfig 1 GET /.aws/credentials 1 GET /.aws/config 1 GET / -- last 10 requests -- 34.169.205.61 - - [15/Aug/2026:01:29:05 +1000] "GET /z9x8c7v6b5-debug-trigger-www.madalice.com.au HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:05 +1000] "GET /wp-json HTTP/1.1" 200 50289 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:06 +1000] "GET /.git/config HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:07 +1000] "GET /rclone.conf HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:07 +1000] "GET /.git-credentials HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:08 +1000] "GET /.git/HEAD HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:09 +1000] "GET /.gitconfig HTTP/1.1" 301 1 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:10 +1000] "GET /z9x8c7v6b5-debug-trigger-www.madalice.com.au HTTP/1.1" 404 30553 "https://www.madalice.com.au/z9x8c7v6b5-debug-trigger-www.madalice.com.au" "meta-externalagent/1.1 (+h 34.169.205.61 - - [15/Aug/2026:01:29:10 +1000] "GET /.git/config HTTP/1.1" 404 30541 "https://www.madalice.com.au/.git/config" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" 34.169.205.61 - - [15/Aug/2026:01:29:11 +1000] "GET /rclone.conf HTTP/1.1" 404 30539 "https://www.madalice.com.au/rclone.conf" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" --- last 30 samples (run-up to this incident) --- 2026-08-15 01:26:48 L1=1.34 run=2 blk=0 avail=6940 (59.2%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=18 tcp=28 topuser=root 2026-08-15 01:26:55 L1=1.31 run=2 blk=0 avail=6948 (59.2%) swpO=1 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=26 topuser=root 2026-08-15 01:26:58 L1=1.37 run=2 blk=0 avail=6952 (59.3%) swpO=1 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=20 topuser=root 2026-08-15 01:27:05 L1=1.34 run=2 blk=0 avail=6946 (59.2%) swpO=3 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=25 topuser=root 2026-08-15 01:27:08 L1=1.31 run=2 blk=0 avail=6944 (59.2%) swpO=3 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=27 topuser=root 2026-08-15 01:27:15 L1=1.36 run=3 blk=0 avail=6938 (59.2%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=18 tcp=25 topuser=root 2026-08-15 01:27:18 L1=1.42 run=4 blk=0 avail=6816 (58.1%) swpO=0 conn=2 myRSS=2236 php=2 phpRSS=219 ap=18 tcp=28 topuser=evolution 2026-08-15 01:27:25 L1=1.54 run=3 blk=0 avail=6954 (59.3%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=18 tcp=19 topuser=root 2026-08-15 01:27:28 L1=1.58 run=2 blk=0 avail=6953 (59.3%) swpO=0 conn=2 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=20 topuser=root 2026-08-15 01:27:35 L1=1.61 run=2 blk=0 avail=6953 (59.3%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=23 topuser=root 2026-08-15 01:27:38 L1=1.64 run=5 blk=0 avail=6768 (57.7%) swpO=0 conn=2 myRSS=2236 php=3 phpRSS=308 ap=18 tcp=25 topuser=alison 2026-08-15 01:27:45 L1=1.83 run=2 blk=0 avail=6934 (59.1%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=18 tcp=24 topuser=root 2026-08-15 01:27:48 L1=1.77 run=4 blk=0 avail=6821 (58.2%) swpO=0 conn=2 myRSS=2236 php=3 phpRSS=251 ap=18 tcp=22 topuser=tubesupplies 2026-08-15 01:27:55 L1=1.79 run=3 blk=0 avail=6935 (59.1%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=19 tcp=25 topuser=root 2026-08-15 01:27:58 L1=1.72 run=3 blk=0 avail=6836 (58.3%) swpO=0 conn=2 myRSS=2236 php=2 phpRSS=197 ap=19 tcp=28 topuser=alison 2026-08-15 01:28:05 L1=1.66 run=3 blk=0 avail=6898 (58.8%) swpO=1 conn=1 myRSS=2236 php=2 phpRSS=145 ap=19 tcp=24 topuser=alison 2026-08-15 01:28:09 L1=1.69 run=2 blk=0 avail=6940 (59.2%) swpO=1 conn=2 myRSS=2236 php=1 phpRSS=68 ap=19 tcp=23 topuser=root 2026-08-15 01:28:15 L1=1.64 run=2 blk=0 avail=6956 (59.3%) swpO=3 conn=1 myRSS=2236 php=1 phpRSS=68 ap=18 tcp=23 topuser=root 2026-08-15 01:28:19 L1=1.58 run=3 blk=0 avail=6888 (58.7%) swpO=3 conn=2 myRSS=2236 php=2 phpRSS=163 ap=18 tcp=21 topuser=alison 2026-08-15 01:28:25 L1=1.54 run=2 blk=0 avail=6953 (59.3%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=22 topuser=root 2026-08-15 01:28:29 L1=1.49 run=9 blk=0 avail=6877 (58.6%) swpO=0 conn=2 myRSS=2236 php=2 phpRSS=175 ap=17 tcp=22 topuser=evolution 2026-08-15 01:28:35 L1=1.45 run=2 blk=0 avail=7003 (59.7%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=17 tcp=17 topuser=root 2026-08-15 01:28:39 L1=1.42 run=2 blk=0 avail=7001 (59.7%) swpO=0 conn=2 myRSS=2236 php=1 phpRSS=68 ap=16 tcp=19 topuser=root 2026-08-15 01:28:45 L1=1.38 run=2 blk=0 avail=7011 (59.8%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=15 tcp=20 topuser=root 2026-08-15 01:28:49 L1=1.35 run=2 blk=0 avail=7008 (59.8%) swpO=0 conn=2 myRSS=2236 php=1 phpRSS=68 ap=15 tcp=20 topuser=root 2026-08-15 01:28:55 L1=1.49 run=2 blk=0 avail=7004 (59.7%) swpO=0 conn=1 myRSS=2236 php=1 phpRSS=68 ap=15 tcp=20 topuser=root 2026-08-15 01:28:59 L1=1.45 run=2 blk=0 avail=7005 (59.7%) swpO=0 conn=2 myRSS=2236 php=1 phpRSS=68 ap=15 tcp=18 topuser=root 2026-08-15 01:29:05 L1=1.41 run=7 blk=0 avail=6851 (58.4%) swpO=0 conn=1 myRSS=2236 php=6 phpRSS=376 ap=15 tcp=34 topuser=madalice 2026-08-15 01:29:09 L1=2.18 run=6 blk=1 avail=6516 (55.6%) swpO=119 conn=2 myRSS=2232 php=7 phpRSS=777 ap=30 tcp=54 topuser=madalice 2026-08-15 01:29:15 L1=2.72 run=17 blk=0 avail=5592 (47.7%) swpO=664 conn=12 myRSS=2217 php=13 phpRSS=1664 ap=75 tcp=59 topuser=madalice