INCIDENT 2026-08-30 16:46:52 trigger: php_rss=1525MB ================================================================== load: 0.63 0.94 1.14 4/676 1034078 ncpu:10 mem_avail: 6891MB/13777MB (50.0%) swap_used:1771MB swapout:0/s blocked(D):0 running:6 iowait:0.0% steal:0.0% php: 11 procs / 1525MB heaviest user: madalice (9 procs, 1489MB) apache: 48 procs tcp_estab: 48 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 3043 madalice 9 1497 root 244 949 nobody 47 741 netdata 7 200 mailman 20 119 cpanellogin 1 118 infinium 2 82 dovenull 6 47 wp-toolkit 3 30 mailnull 1 24 zabbix 1 22 evolution 5 22 dovecot 3 19 aaa 1 15 candyfabcom 1 14 alison 1 14 named 1 13 kurri 1 12 lowrange 1 11 --- php workers by user --- USER PROCS RSS_MB madalice 9 1499 root 1 22 evolution 1 13 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5845 MB Active(file): 4752 MB Inactive(file): 1535 MB Dirty: 1 MB Writeback: 0 MB SReclaimable: 595 MB CommitLimit: 10984 MB Committed_AS: 8591 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 489033 root 2-14:28:12 7780 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 506671 1 mysql 3116080 7.0 Ssl 2-14:27:41 /usr/sbin/mariadbd 1033971 510022 madalice 201684 43.0 R 00:05 php-fpm: pool madalice_com_au 889551 2277352 root 191368 0.0 S 41:09 spamd child 508924 1 netdata 181412 1.9 Ssl 2-14:27:10 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2277352 1 root 175548 0.0 Ss 14:23:47 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 1033881 510022 madalice 163496 34.2 S 00:14 php-fpm: pool madalice_com_au 1033884 510022 madalice 162772 27.0 S 00:13 php-fpm: pool madalice_com_au 1033962 510022 madalice 162328 42.0 S 00:06 php-fpm: pool madalice_com_au 1033965 510022 madalice 162164 51.0 S 00:05 php-fpm: pool madalice_com_au 1033968 510022 madalice 162000 49.0 S 00:05 php-fpm: pool madalice_com_au 1033970 510022 madalice 158948 50.4 S 00:05 php-fpm: pool madalice_com_au 2328311 2277352 root 156632 0.0 S 14:08:41 spamd child 1033972 510022 madalice 141924 36.0 R 00:05 php-fpm: pool madalice_com_au 1033964 510022 madalice 139308 26.8 S 00:05 php-fpm: pool madalice_com_au 4027958 1353308 cpanellogin 121528 0.0 S 05:48:38 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base server_locale en default_login_theme cpanel 1032943 1353308 infinium 55332 0.2 S 00:58 webmaild - serving 1.41.252.211 509236 509098 root 53628 0.0 Sl 2-14:27:07 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 509640 508992 root 51256 0.0 Sl 2-14:27:05 /usr/libexec/netdata/plugins.d/ebpf.plugin 1 1353308 1 root 42576 0.1 Ss 39-14:05:33 cpsrvd (SSL) - waiting for connections 2275833 1 root 35612 0.0 Ss 14:24:37 lfd - sleeping 892109 938407 infinium 29248 0.0 S 37:31 php-fpm: pool user_infinium 508720 1 mailnull 25284 0.0 Ss 2-14:27:14 /usr/sbin/exim -ps -bd -q15m -oP /var/spool/exim/exim-daemon.pid 1469 1 root 24040 0.1 Ss 62-06:37:02 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 510022 1 root 23336 0.0 Ss 2-14:27:03 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 997657 506832 nobody 22912 0.1 S 09:50 /usr/sbin/httpd -k start 1015946 506832 nobody 22740 0.0 S 05:08 /usr/sbin/httpd -k start 483898 1 zabbix 22628 0.2 Ssl 2-14:28:24 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 1015920 506832 nobody 22360 0.0 S 05:09 /usr/sbin/httpd -k start 1023469 506832 nobody 22056 0.0 S 01:45 /usr/sbin/httpd -k start 1463 1 root 21888 0.0 Ss 62-06:37:02 tailwatchd 1015846 506832 nobody 21468 0.0 S 05:11 /usr/sbin/httpd -k start 1015840 506832 nobody 21228 0.0 S 05:11 /usr/sbin/httpd -k start 1033326 506832 nobody 20736 0.1 S 00:44 /usr/sbin/httpd -k start 1015953 506832 nobody 20152 0.0 S 05:07 /usr/sbin/httpd -k start 1015886 506832 nobody 20136 0.1 S 05:10 /usr/sbin/httpd -k start 1017246 506832 nobody 20000 0.2 S 02:31 /usr/sbin/httpd -k start 1017245 506832 nobody 19944 0.2 S 02:31 /usr/sbin/httpd -k start 1032494 506832 nobody 19864 0.3 S 01:15 /usr/sbin/httpd -k start 1032450 506832 nobody 19712 0.2 S 01:17 /usr/sbin/httpd -k start 999429 506832 nobody 19424 0.1 S 08:08 /usr/sbin/httpd -k start 1033977 506832 nobody 18840 0.5 S 00:04 /usr/sbin/httpd -k start 938582 1 wp-toolkit 17952 0.0 Ss 2-12:34:44 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 1471 1 root 16852 0.0 Ss 62-06:37:02 cPhulkd - processor 3446928 1471 root 16684 0.0 S 08:41:02 cPhulkd - dbprocessor 1034010 506832 nobody 16556 0.0 S 00:01 /usr/sbin/httpd -k start 1033982 506832 nobody 16528 0.0 S 00:03 /usr/sbin/httpd -k start 1033995 506832 nobody 16496 0.0 S 00:02 /usr/sbin/httpd -k start 1033996 506832 nobody 16496 0.0 S 00:02 /usr/sbin/httpd -k start 1033997 506832 nobody 16496 0.0 S 00:02 /usr/sbin/httpd -k start 1033998 506832 nobody 16496 0.0 S 00:02 /usr/sbin/httpd -k start 1033999 506832 nobody 16496 0.0 S 00:02 /usr/sbin/httpd -k start 1034008 506832 nobody 16496 0.0 S 00:01 /usr/sbin/httpd -k start 1034009 506832 nobody 16496 0.0 S 00:01 /usr/sbin/httpd -k start 1033984 506832 nobody 16448 0.0 S 00:03 /usr/sbin/httpd -k start 1033992 506832 nobody 16448 0.0 S 00:02 /usr/sbin/httpd -k start 1033993 506832 nobody 16448 0.0 S 00:02 /usr/sbin/httpd -k start 1033994 506832 nobody 16448 0.0 S 00:02 /usr/sbin/httpd -k start 1034005 506832 nobody 16448 0.0 S 00:01 /usr/sbin/httpd -k start 1034006 506832 nobody 16448 0.0 S 00:01 /usr/sbin/httpd -k start 1034007 506832 nobody 16448 0.0 S 00:01 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4595451 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/app.evolutionerp.com.au-ssl_log 1075 kamcool/kamcool.com.au-ssl_log 307 tubesupplies/tubesupplies.com.au-ssl_log 84 bigdeals/bigdeals.com.au-ssl_log 62 lowrange/lowrange.com.au 35 evolution/portal.evolutionerp.com.au-ssl_log 28 madalice/madalice.com.au-ssl_log 24 (total vhosts taking traffic: 7) --- top client IPs across all vhosts (last 3 min) --- 1075 40.115.94.245 170 4.205.62.107 137 52.139.44.162 81 158.23.147.79 62 52.139.37.240 35 68.155.159.216 28 203.54.180.238 15 47.128.228.221 2 87.58.197.202 2 118.127.40.55 2 114.119.159.70 1 2a03:2880:f802:43:: 1 23.180.120.148 1 2001:8003:3b46:cb00:2164:d3a3:2a9a:40ed 1 114.119.143.14 --- top request paths across all vhosts (last 3 min) --- 137 GET /autodiscover/autodiscover.xml 135 GET /api/getItem/LWPG2001KG 135 GET /api/getItem/LWPB2001KG 135 GET /api/getItem/LWHG2001KG 135 GET /api/getItem/LWFBW2001KG 135 GET /api/getItem/LWFB2001KG 28 GET /heartbeat.php 9 GET /api/getItem/NA 4 GET /api/getItem/PWFBW3151KG 4 GET / 3 GET /wp-content/plugins/admin.php 3 GET /images/index.php 2 POST /wp-admin/admin-ajax.php 2 GET /xda.php 2 GET /wzy.php 2 GET /wp-mail.php 2 GET /wp-includes/widgets/index.php 2 GET /wp-includes/Text/Diff/index.php 2 GET /wp-includes/sitemaps/providers/index.php 2 GET /wp-includes/SimplePie/Content/index.php --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/app.evolutionerp.com.au-ssl_log == -- top paths -- 135 GET /api/getItem/LWPG2001KG 135 GET /api/getItem/LWPB2001KG 135 GET /api/getItem/LWHG2001KG 135 GET /api/getItem/LWFBW2001KG 135 GET /api/getItem/LWFB2001KG 9 GET /api/getItem/NA 4 GET /api/getItem/PWFBW3151KG 2 GET /api/getItem/SHSG8989508 2 GET /api/getItem/SHSG8989358 2 GET /api/getItem/SHSG8989258 2 GET /api/getItem/SHSG8989208 2 GET /api/getItem/SHSG7575508 -- last 10 requests -- 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW90 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW100 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW120 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW160 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW117T HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/RGW80 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:11 +1000] "GET /api/getItem/GP150x200x8 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:12 +1000] "GET /api/getItem/K125MSGB HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:12 +1000] "GET /api/getItem/253-40SP HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [30/Aug/2026:16:46:12 +1000] "GET /api/getItem/300558 HTTP/1.1" 429 62 "-" "RestSharp/106.13.0.0" == kamcool/kamcool.com.au-ssl_log == -- top paths -- 137 GET /autodiscover/autodiscover.xml 2 GET /thui.php 2 GET /ms-edit.php 2 GET /koiy.php 1 GET /zz.php 1 GET /zaza.php 1 GET /xza.php 1 GET /xve22.php 1 GET /xqq.php 1 GET /xp.php 1 GET /xmy.php 1 GET /xin1.php -- last 10 requests -- 4.205.62.107 - - [30/Aug/2026:16:46:24 +1000] "GET /public/rip.php.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:25 +1000] "GET /environment.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:25 +1000] "GET /aws_secret_config.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:26 +1000] "GET /snq.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:26 +1000] "GET /wp-access.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:27 +1000] "GET /otuz1.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:27 +1000] "GET /update.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:27 +1000] "GET /channels.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:28 +1000] "GET /zz.php HTTP/1.1" 404 5749 "-" "-" 4.205.62.107 - - [30/Aug/2026:16:46:28 +1000] "GET /test.php HTTP/1.1" 404 5749 "-" "-" == tubesupplies/tubesupplies.com.au-ssl_log == -- top paths -- 2 GET / 1 POST /wp-cron.php 1 GET /x.php 1 GET /wzy.php 1 GET /wp-mail.php 1 GET /wp-login.php 1 GET /wp-l0gin.php 1 GET /wp-includes/widgets/index.php 1 GET /wp-includes/Text/index.php 1 GET /wp-includes/Text/Diff/index.php 1 GET /wp-includes/sitemaps/providers/index.php 1 GET /wp-includes/SimplePie/Content/index.php -- last 10 requests -- 158.23.147.79 - - [30/Aug/2026:16:44:38 +1000] "GET /worksec.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:38 +1000] "GET /x.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:39 +1000] "GET /wp-content/x.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:39 +1000] "GET /wp-content/themes/wp-pridmag/init.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:40 +1000] "GET /log-mama/function.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:40 +1000] "GET /bk/index.php HTTP/1.1" 404 5749 "-" "-" 158.23.147.79 - - [30/Aug/2026:16:44:41 +1000] "GET /first.php HTTP/1.1" 404 5749 "-" "-" 87.58.197.202 - - [30/Aug/2026:16:44:53 +1000] "GET / HTTP/1.1" 301 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" 118.127.40.55 - - [30/Aug/2026:16:44:54 +1000] "POST /wp-cron.php?doing_wp_cron=1788072294.1217110157012939453125 HTTP/1.1" 200 1 "-" "WordPress/7.1; https://tubesupplies.com.au" 87.58.197.202 - - [30/Aug/2026:16:44:55 +1000] "GET / HTTP/1.1" 200 352500 "https://tubesupplies.com.au:443" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari --- last 30 samples (run-up to this incident) --- 2026-08-30 16:44:31 L1=0.69 run=3 blk=0 avail=8116 (58.9%) swpO=2 conn=2 myRSS=3042 php=4 phpRSS=323 ap=21 tcp=28 topuser=madalice 2026-08-30 16:44:31 L1=0.69 run=1 blk=0 avail=8117 (58.9%) swpO=2 conn=2 myRSS=3042 php=4 phpRSS=323 ap=21 tcp=26 topuser=madalice 2026-08-30 16:44:41 L1=0.58 run=1 blk=0 avail=8129 (59.0%) swpO=0 conn=2 myRSS=3042 php=4 phpRSS=323 ap=18 tcp=26 topuser=madalice 2026-08-30 16:44:41 L1=0.58 run=1 blk=0 avail=8124 (59.0%) swpO=0 conn=2 myRSS=3042 php=4 phpRSS=323 ap=18 tcp=27 topuser=madalice 2026-08-30 16:44:51 L1=0.49 run=1 blk=0 avail=8387 (60.9%) swpO=0 conn=2 myRSS=3042 php=2 phpRSS=35 ap=18 tcp=22 topuser=root 2026-08-30 16:44:51 L1=0.49 run=1 blk=0 avail=8387 (60.9%) swpO=0 conn=2 myRSS=3042 php=2 phpRSS=35 ap=18 tcp=21 topuser=root 2026-08-30 16:45:01 L1=0.49 run=3 blk=1 avail=8275 (60.1%) swpO=0 conn=2 myRSS=3042 php=4 phpRSS=206 ap=16 tcp=31 topuser=madalice 2026-08-30 16:45:01 L1=0.49 run=2 blk=0 avail=8271 (60.0%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=212 ap=16 tcp=32 topuser=madalice 2026-08-30 16:45:11 L1=0.72 run=3 blk=0 avail=8215 (59.6%) swpO=0 conn=2 myRSS=3042 php=4 phpRSS=241 ap=17 tcp=25 topuser=madalice 2026-08-30 16:45:11 L1=0.72 run=3 blk=0 avail=8223 (59.7%) swpO=0 conn=1 myRSS=3042 php=5 phpRSS=254 ap=17 tcp=25 topuser=madalice 2026-08-30 16:45:21 L1=0.75 run=2 blk=0 avail=8338 (60.5%) swpO=0 conn=2 myRSS=3042 php=4 phpRSS=140 ap=17 tcp=22 topuser=evolution 2026-08-30 16:45:21 L1=0.75 run=1 blk=0 avail=8338 (60.5%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=105 ap=17 tcp=24 topuser=evolution 2026-08-30 16:45:31 L1=0.80 run=3 blk=0 avail=8232 (59.7%) swpO=0 conn=2 myRSS=3042 php=3 phpRSS=201 ap=17 tcp=23 topuser=madalice 2026-08-30 16:45:31 L1=0.80 run=5 blk=0 avail=8208 (59.6%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=228 ap=17 tcp=23 topuser=madalice 2026-08-30 16:45:41 L1=0.76 run=2 blk=0 avail=8254 (59.9%) swpO=0 conn=2 myRSS=3042 php=3 phpRSS=169 ap=19 tcp=23 topuser=madalice 2026-08-30 16:45:41 L1=0.76 run=1 blk=0 avail=8262 (60.0%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=190 ap=19 tcp=22 topuser=madalice 2026-08-30 16:45:51 L1=0.79 run=3 blk=0 avail=8250 (59.9%) swpO=1 conn=2 myRSS=3042 php=3 phpRSS=193 ap=18 tcp=20 topuser=madalice 2026-08-30 16:45:51 L1=0.79 run=1 blk=0 avail=8262 (60.0%) swpO=1 conn=1 myRSS=3042 php=4 phpRSS=209 ap=18 tcp=21 topuser=madalice 2026-08-30 16:46:01 L1=0.75 run=2 blk=0 avail=8259 (59.9%) swpO=0 conn=2 myRSS=3042 php=3 phpRSS=193 ap=17 tcp=24 topuser=madalice 2026-08-30 16:46:01 L1=0.75 run=2 blk=0 avail=8260 (60.0%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=218 ap=17 tcp=23 topuser=madalice 2026-08-30 16:46:11 L1=0.71 run=1 blk=0 avail=8259 (60.0%) swpO=0 conn=2 myRSS=3042 php=3 phpRSS=192 ap=18 tcp=22 topuser=madalice 2026-08-30 16:46:11 L1=0.71 run=5 blk=0 avail=8258 (59.9%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=192 ap=18 tcp=20 topuser=madalice 2026-08-30 16:46:21 L1=0.60 run=1 blk=0 avail=8267 (60.0%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=192 ap=18 tcp=19 topuser=madalice 2026-08-30 16:46:22 L1=0.60 run=1 blk=0 avail=8266 (60.0%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=192 ap=18 tcp=21 topuser=madalice 2026-08-30 16:46:31 L1=0.67 run=2 blk=0 avail=8333 (60.5%) swpO=0 conn=1 myRSS=3042 php=3 phpRSS=127 ap=17 tcp=22 topuser=alison 2026-08-30 16:46:32 L1=0.67 run=3 blk=0 avail=8292 (60.2%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=197 ap=17 tcp=22 topuser=alison 2026-08-30 16:46:41 L1=0.56 run=1 blk=0 avail=8186 (59.4%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=293 ap=17 tcp=21 topuser=madalice 2026-08-30 16:46:42 L1=0.56 run=1 blk=0 avail=8185 (59.4%) swpO=0 conn=1 myRSS=3042 php=4 phpRSS=293 ap=17 tcp=21 topuser=madalice 2026-08-30 16:46:51 L1=0.63 run=4 blk=0 avail=6966 (50.6%) swpO=0 conn=1 myRSS=3043 php=11 phpRSS=1462 ap=48 tcp=48 topuser=madalice 2026-08-30 16:46:52 L1=0.63 run=6 blk=0 avail=6891 (50.0%) swpO=0 conn=4 myRSS=3043 php=11 phpRSS=1525 ap=48 tcp=48 topuser=madalice