INCIDENT 2026-08-15 03:42:36 trigger: php_rss=2333MB ================================================================== load: 1.40 0.72 0.80 11/675 2427772 ncpu:10 mem_avail: 4990MB/11729MB (42.5%) swap_used:1149MB swapout:314/s blocked(D):0 running:16 iowait:0.0% steal:0.0% php: 13 procs / 2333MB heaviest user: madalice (11 procs, 2195MB) apache: 83 procs tcp_estab: 58 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2273 nobody 82 949 root 240 901 madalice 7 886 netdata 8 264 speedlin 1 110 infinium 5 64 evolution 5 43 kurri 3 37 dovenull 6 28 wp-toolkit 6 20 cpanellogin 1 18 mailman 20 14 lowrange 1 12 zabbix 1 11 dovecot 3 8 named 1 7 nscd 1 4 dbus 1 3 cpanelconnecttrack 1 3 --- php workers by user --- USER PROCS RSS_MB madalice 7 758 speedlin 1 111 evolution 1 39 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 3724 MB Active(file): 3178 MB Inactive(file): 635 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 1020 MB CommitLimit: 9960 MB Committed_AS: 8660 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 2342983 root 02:31:21 6220 /usr/bin/imunify-agent-proxy 2425529 root 03:06 60944 /opt/imunify360/venv/bin/python3 -m imav.run --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 1420296 1 mysql 2328412 11.1 Ssl 1-02:03:36 /usr/sbin/mariadbd 2427693 2426897 madalice 260432 58.0 R 00:04 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1220 1 netdata 215860 2.5 Ssl 46-17:32:39 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2382613 1 root 175196 0.0 Ss 01:19:25 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 2390233 2382613 root 169012 0.0 S 01:06:59 spamd child 2390234 2382613 root 167448 0.0 S 01:06:59 spamd child 2427739 2425583 madalice 121680 53.5 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2427737 2426090 speedlin 117752 52.0 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2427744 2427655 madalice 70616 27.0 R 00:01 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2427751 2427675 madalice 67004 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 2425529 1 root 60944 0.7 Ssl 03:06 /opt/imunify360/venv/bin/python3 -m imav.run 2223 1201 root 54904 0.0 Sl 46-17:32:38 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 2427777 2426918 madalice 54388 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 1469 1 root 42028 0.1 Ss 46-17:32:38 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1353308 1 root 41728 0.0 Ss 24-01:01:09 cpsrvd (SSL) - waiting for connections 2427740 2427654 evolution 40884 3.0 S 00:01 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 159820 1 root 37960 0.0 Ss 25-17:55:47 /usr/lib/systemd/systemd-journald 2381704 1 root 33356 0.0 Ss 01:20:24 lfd - sleeping 4922 2431 netdata 32580 0.1 Sl 46-17:32:34 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2424927 1276780 nobody 20984 0.0 S 04:27 /usr/sbin/httpd -k start 2426085 1276780 nobody 20768 0.1 S 02:47 /usr/sbin/httpd -k start 2424161 1276780 nobody 19824 0.0 S 05:57 /usr/sbin/httpd -k start 2422581 1353308 cpanellogin 18672 0.0 S 08:11 /usr/local/cpanel/base/show_template.stor docroot /usr/local/cpanel/base cpanel_locale server_locale en default_login_theme cpanel 2425583 1276780 nobody 18664 0.0 S 03:03 /usr/sbin/httpd -k start 2427056 1276780 nobody 18364 0.0 S 01:17 /usr/sbin/httpd -k start 2426090 1276780 nobody 18060 0.0 S 02:46 /usr/sbin/httpd -k start 2426901 1276780 nobody 18060 0.0 S 01:27 /usr/sbin/httpd -k start 2426914 1276780 nobody 18060 0.0 S 01:26 /usr/sbin/httpd -k start 2426486 1276780 nobody 18024 0.0 S 02:10 /usr/sbin/httpd -k start 2426798 1276780 nobody 17464 0.0 S 01:29 /usr/sbin/httpd -k start 2426897 1276780 nobody 17392 0.0 S 01:27 /usr/sbin/httpd -k start 2426912 1276780 nobody 17344 0.0 S 01:26 /usr/sbin/httpd -k start 2427676 1276780 nobody 17304 0.5 S 00:04 /usr/sbin/httpd -k start 2427679 1276780 nobody 17200 0.2 S 00:04 /usr/sbin/httpd -k start 2427674 1276780 nobody 17196 0.2 S 00:04 /usr/sbin/httpd -k start 2427677 1276780 nobody 17144 0.5 S 00:04 /usr/sbin/httpd -k start 2427682 1276780 nobody 17132 0.5 S 00:04 /usr/sbin/httpd -k start 2427680 1276780 nobody 17124 0.5 S 00:04 /usr/sbin/httpd -k start 2427678 1276780 nobody 17088 0.2 S 00:04 /usr/sbin/httpd -k start 2427683 1276780 nobody 17064 0.2 S 00:04 /usr/sbin/httpd -k start 2427681 1276780 nobody 17060 0.5 S 00:04 /usr/sbin/httpd -k start 2426800 1276780 nobody 16976 0.0 S 01:28 /usr/sbin/httpd -k start 2427502 1276780 nobody 16752 0.0 S 00:21 /usr/sbin/httpd -k start 2426918 1276780 nobody 16460 0.1 S 01:26 /usr/sbin/httpd -k start 2426920 1276780 nobody 16012 0.0 S 01:26 /usr/sbin/httpd -k start 2426916 1276780 nobody 15916 0.0 S 01:26 /usr/sbin/httpd -k start 2427684 1276780 nobody 15860 0.0 S 00:04 /usr/sbin/httpd -k start 2427622 1276780 nobody 15784 0.0 S 00:08 /usr/sbin/httpd -k start 2427667 1276780 nobody 15716 0.0 S 00:05 /usr/sbin/httpd -k start 2427655 1276780 nobody 15448 0.1 S 00:07 /usr/sbin/httpd -k start 2427670 1276780 nobody 15260 0.0 S 00:05 /usr/sbin/httpd -k start 2427673 1276780 nobody 15036 0.0 S 00:04 /usr/sbin/httpd -k start 2427661 1276780 nobody 15016 0.0 R 00:06 /usr/sbin/httpd -k start 2427666 1276780 nobody 15004 0.0 S 00:05 /usr/sbin/httpd -k start 2427675 1276780 nobody 15004 0.0 S 00:04 /usr/sbin/httpd -k start 2427665 1276780 nobody 14996 0.0 S 00:05 /usr/sbin/httpd -k start 2427654 1276780 nobody 14960 0.0 S 00:07 /usr/sbin/httpd -k start 2427663 1276780 nobody 14960 0.0 S 00:05 /usr/sbin/httpd -k start 2427660 1276780 nobody 14956 0.0 S 00:06 /usr/sbin/httpd -k start 2427662 1276780 nobody 14956 0.0 S 00:06 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3987789 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min coscom/coscom.com.au 104 customkidscom/customkids.com.au-ssl_log 86 evolution/portal.evolutionerp.com.au-ssl_log 27 corpkicks/corporatekicks.com.au-ssl_log 24 madalice/madalice.com.au-ssl_log 13 evolution/my.evolutionerp.com.au-ssl_log 4 (total vhosts taking traffic: 6) --- top client IPs across all vhosts (last 3 min) --- 104 20.65.105.233 86 4.232.94.23 27 203.54.180.238 21 20.250.27.191 6 35.224.219.114 3 49.65.18.34 2 118.127.40.55 1 65.55.210.71 1 54.86.66.252 1 54.175.74.27 1 51.211.38.152 1 40.77.167.79 1 34.196.51.17 1 2a01:cb06:3d3:e800:f146:e5e6:6681:3e2a 1 184.72.115.35 --- top request paths across all vhosts (last 3 min) --- 86 GET /autodiscover/autodiscover.xml 27 GET /heartbeat.php 3 GET /index.php 2 POST /wp-cron.php 2 GET /wp-ws68.php 2 GET /wp-content/plugins/hellopress/wp_filemanager.php 2 GET /this_is_a_new_hello_world.php 2 GET /aa.php 2 GET /222.php 2 GET /1.php 2 GET / 1 POST / 1 HEAD / 1 GET /z9x8c7v6b5-debug-trigger-www.madalice.com.au 1 GET /yj09.php 1 GET /xxxx.php 1 GET /xxw.php 1 GET /xx.php 1 GET /x.php 1 GET /xeqdlun.php --- busiest 3 vhosts: top paths + last 10 requests --- == coscom/coscom.com.au == -- top paths -- 1 GET /xxxx.php 1 GET /xxw.php 1 GET /xx.php 1 GET /x.php 1 GET /xeqdlun.php 1 GET /x7.php 1 GET /wto.php 1 GET /wsd.php 1 GET /ws45.php 1 GET /ws38.php 1 GET /ws37.php 1 GET /ws36.php -- last 10 requests -- 20.65.105.233 - - [15/Aug/2026:03:40:44 +1000] "GET /haa.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:44 +1000] "GET /f6.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:45 +1000] "GET /asas.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:45 +1000] "GET /x.php HTTP/1.1" 403 13 "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:45 +1000] "GET /ava.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:45 +1000] "GET /ton.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:45 +1000] "GET /v1.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:46 +1000] "GET /111.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:46 +1000] "GET /an7.php HTTP/1.1" 404 - "-" "-" 20.65.105.233 - - [15/Aug/2026:03:40:46 +1000] "GET /xeqdlun.php HTTP/1.1" 404 - "-" "-" == customkidscom/customkids.com.au-ssl_log == -- top paths -- 86 GET /autodiscover/autodiscover.xml -- last 10 requests -- 4.232.94.23 - - [15/Aug/2026:03:42:01 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:02 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:04 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:05 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:06 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:07 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:09 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:10 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:11 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 4.232.94.23 - - [15/Aug/2026:03:42:13 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 27 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [15/Aug/2026:03:41:25 +1000] "GET /heartbeat.php?sentAt=1786729286&bytes=106,127,83,70&_=1786712383946 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:33 +1000] "GET /heartbeat.php?sentAt=1786729293&bytes=106,127,83,77&_=1786683793979 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:35 +1000] "GET /heartbeat.php?sentAt=1786729296&bytes=106,127,83,80&_=1786712383947 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:43 +1000] "GET /heartbeat.php?sentAt=1786729303&bytes=106,127,83,87&_=1786683793980 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:45 +1000] "GET /heartbeat.php?sentAt=1786729306&bytes=106,127,83,90&_=1786712383948 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:53 +1000] "GET /heartbeat.php?sentAt=1786729313&bytes=106,127,83,97&_=1786683793981 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:41:55 +1000] "GET /heartbeat.php?sentAt=1786729316&bytes=106,127,83,100&_=1786712383949 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:42:03 +1000] "GET /heartbeat.php?sentAt=1786729323&bytes=106,127,83,107&_=1786683793982 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:42:05 +1000] "GET /heartbeat.php?sentAt=1786729326&bytes=106,127,83,110&_=1786712383950 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [15/Aug/2026:03:42:13 +1000] "GET /heartbeat.php?sentAt=1786729333&bytes=106,127,83,117&_=1786683793983 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" --- last 30 samples (run-up to this incident) --- 2026-08-15 03:40:08 L1=0.38 run=3 blk=0 avail=7034 (60.0%) swpO=0 conn=3 myRSS=2283 php=2 phpRSS=118 ap=20 tcp=30 topuser=aaa 2026-08-15 03:40:15 L1=0.43 run=1 blk=0 avail=7095 (60.5%) swpO=0 conn=2 myRSS=2283 php=0 phpRSS=0 ap=20 tcp=22 topuser=- 2026-08-15 03:40:18 L1=0.39 run=1 blk=0 avail=7103 (60.6%) swpO=0 conn=3 myRSS=2283 php=0 phpRSS=0 ap=19 tcp=23 topuser=- 2026-08-15 03:40:25 L1=0.36 run=1 blk=0 avail=7122 (60.7%) swpO=0 conn=2 myRSS=2283 php=0 phpRSS=0 ap=16 tcp=23 topuser=- 2026-08-15 03:40:28 L1=0.33 run=2 blk=0 avail=6976 (59.5%) swpO=0 conn=3 myRSS=2283 php=2 phpRSS=188 ap=19 tcp=29 topuser=tubesupplies 2026-08-15 03:40:35 L1=0.31 run=1 blk=0 avail=7092 (60.5%) swpO=0 conn=2 myRSS=2283 php=0 phpRSS=0 ap=20 tcp=26 topuser=- 2026-08-15 03:40:39 L1=0.28 run=1 blk=0 avail=7092 (60.5%) swpO=0 conn=3 myRSS=2283 php=0 phpRSS=0 ap=20 tcp=29 topuser=- 2026-08-15 03:40:45 L1=0.26 run=3 blk=0 avail=7106 (60.6%) swpO=0 conn=2 myRSS=2283 php=0 phpRSS=0 ap=19 tcp=22 topuser=- 2026-08-15 03:40:49 L1=0.24 run=1 blk=0 avail=7107 (60.6%) swpO=0 conn=3 myRSS=2283 php=0 phpRSS=0 ap=18 tcp=25 topuser=- 2026-08-15 03:40:55 L1=0.38 run=1 blk=0 avail=7104 (60.6%) swpO=0 conn=3 myRSS=2283 php=1 phpRSS=22 ap=18 tcp=23 topuser=evolution 2026-08-15 03:40:59 L1=0.35 run=1 blk=0 avail=7105 (60.6%) swpO=0 conn=3 myRSS=2283 php=0 phpRSS=0 ap=18 tcp=19 topuser=- 2026-08-15 03:41:05 L1=0.32 run=4 blk=0 avail=7029 (59.9%) swpO=0 conn=3 myRSS=2283 php=2 phpRSS=153 ap=16 tcp=24 topuser=tubesupplies 2026-08-15 03:41:09 L1=0.45 run=2 blk=0 avail=6901 (58.8%) swpO=0 conn=2 myRSS=2283 php=1 phpRSS=182 ap=23 tcp=56 topuser=evolution 2026-08-15 03:41:15 L1=0.50 run=1 blk=0 avail=6931 (59.1%) swpO=0 conn=3 myRSS=2283 php=1 phpRSS=39 ap=43 tcp=36 topuser=evolution 2026-08-15 03:41:19 L1=0.46 run=11 blk=0 avail=6483 (55.3%) swpO=4 conn=2 myRSS=2283 php=4 phpRSS=589 ap=40 tcp=52 topuser=corpkicks 2026-08-15 03:41:26 L1=0.66 run=2 blk=0 avail=6838 (58.3%) swpO=96 conn=3 myRSS=2281 php=2 phpRSS=191 ap=41 tcp=35 topuser=corpkicks 2026-08-15 03:41:29 L1=0.69 run=2 blk=0 avail=6757 (57.6%) swpO=101 conn=2 myRSS=2281 php=1 phpRSS=267 ap=38 tcp=32 topuser=corpkicks 2026-08-15 03:41:36 L1=0.71 run=5 blk=0 avail=7002 (59.7%) swpO=0 conn=3 myRSS=2281 php=3 phpRSS=134 ap=31 tcp=34 topuser=corpkicks 2026-08-15 03:41:39 L1=0.90 run=3 blk=0 avail=6956 (59.3%) swpO=0 conn=2 myRSS=2281 php=1 phpRSS=140 ap=28 tcp=30 topuser=corpkicks 2026-08-15 03:41:46 L1=0.91 run=1 blk=0 avail=7086 (60.4%) swpO=0 conn=3 myRSS=2281 php=1 phpRSS=40 ap=23 tcp=28 topuser=evolution 2026-08-15 03:41:49 L1=0.99 run=5 blk=0 avail=7011 (59.8%) swpO=0 conn=2 myRSS=2281 php=1 phpRSS=118 ap=22 tcp=28 topuser=corpkicks 2026-08-15 03:41:56 L1=1.07 run=2 blk=0 avail=7094 (60.5%) swpO=0 conn=3 myRSS=2281 php=2 phpRSS=81 ap=20 tcp=27 topuser=evolution 2026-08-15 03:41:59 L1=1.07 run=3 blk=0 avail=6959 (59.3%) swpO=0 conn=2 myRSS=2281 php=2 phpRSS=214 ap=20 tcp=29 topuser=alison 2026-08-15 03:42:06 L1=1.06 run=1 blk=0 avail=7100 (60.5%) swpO=0 conn=3 myRSS=2281 php=1 phpRSS=39 ap=19 tcp=30 topuser=evolution 2026-08-15 03:42:09 L1=0.98 run=2 blk=0 avail=7109 (60.6%) swpO=0 conn=1 myRSS=2281 php=0 phpRSS=0 ap=19 tcp=29 topuser=- 2026-08-15 03:42:16 L1=0.90 run=2 blk=0 avail=7063 (60.2%) swpO=0 conn=3 myRSS=2281 php=2 phpRSS=109 ap=20 tcp=30 topuser=corpkicks 2026-08-15 03:42:19 L1=0.91 run=4 blk=0 avail=7025 (59.9%) swpO=0 conn=1 myRSS=2281 php=2 phpRSS=142 ap=21 tcp=31 topuser=corpkicks 2026-08-15 03:42:26 L1=0.91 run=2 blk=0 avail=7044 (60.1%) swpO=0 conn=3 myRSS=2281 php=2 phpRSS=133 ap=21 tcp=27 topuser=madalice 2026-08-15 03:42:29 L1=1.00 run=16 blk=0 avail=6195 (52.8%) swpO=19 conn=1 myRSS=2281 php=12 phpRSS=1252 ap=22 tcp=51 topuser=madalice 2026-08-15 03:42:36 L1=1.40 run=16 blk=0 avail=4990 (42.5%) swpO=314 conn=8 myRSS=2273 php=13 phpRSS=2333 ap=83 tcp=58 topuser=madalice