INCIDENT 2026-08-20 22:11:07 trigger: swapout=2278/s ================================================================== load: 1.76 0.92 0.81 4/644 163430 ncpu:10 mem_avail: 4665MB/11729MB (39.8%) swap_used:2890MB swapout:2278/s blocked(D):0 running:3 iowait:7.3% steal:0.0% php: 2 procs / 118MB heaviest user: vinesit (1 procs, 75MB) apache: 20 procs tcp_estab: 29 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2761 evolution 11 2105 root 246 970 nobody 19 326 netdata 8 261 mailman 20 97 vinesit 1 81 madalice 1 59 mailnull 4 36 dovenull 6 26 cpanellogin 1 18 wp-toolkit 6 17 zabbix 1 16 named 1 12 infinium 1 12 lowrange 1 11 corpkicks 1 11 candyfabcom 1 11 kurri 1 10 dovecot 3 7 --- php workers by user --- USER PROCS RSS_MB vinesit 1 83 madalice 1 53 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 4262 MB Active(file): 2266 MB Inactive(file): 2045 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 466 MB CommitLimit: 9960 MB Committed_AS: 8999 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 163123 root 00:13 61096 /opt/imunify360/venv/bin/python3 -m imav.run 3248642 root 20:59:51 2632 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2575700 1 mysql 2825152 11.6 Ssl 5-14:20:31 /usr/sbin/mariadbd 163290 163289 evolution 2101876 87.7 RNs 00:08 /usr/local/cpanel/3rdparty/bin/analog +CIMAGEDIR /images/ +CDOMAINSFILE /usr/local/cpanel/3rdparty/share/analog/usdom.tab +CCHARTDIR ./ 1220 1 netdata 211752 2.5 Ssl 52-12:01:13 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 3298115 3290507 root 180424 0.0 S 19:35:32 spamd child 2575382 1 root 132420 0.0 Ss 5-14:20:34 /usr/lib/systemd/systemd-journald 3864059 3290507 root 122340 0.0 S 09:11:32 spamd child 163397 162977 vinesit 89892 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 163123 1 root 61096 10.7 Ssl 00:13 /opt/imunify360/venv/bin/python3 -m imav.run 163401 159796 madalice 59332 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 3290507 1 root 48440 0.0 Ss 19:48:21 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 161733 2111368 root 44580 0.7 SN 02:03 cpanellogd - waiting for child to process logs for evolution 163127 161733 evolution 42600 0.3 SN 00:13 cpanellogd - http logs for evolution 2223 1201 root 42292 0.0 Sl 52-12:01:12 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1353308 1 root 42000 0.0 Ss 29-19:29:43 cpsrvd (SSL) - waiting for connections 1469 1 root 39388 0.1 Ss 52-12:01:12 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 163438 1469 root 37860 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 163398 1469 root 37832 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 4922 2431 netdata 32172 0.1 Sl 52-12:01:08 /usr/libexec/netdata/plugins.d/go.d.plugin 1 160995 160888 root 23452 0.0 S 03:02 /usr/sbin/exim -q 159796 1276780 nobody 22604 0.1 S 04:52 /usr/sbin/httpd -k start 161140 1276780 nobody 22132 0.0 S 02:52 /usr/sbin/httpd -k start 159380 1276780 nobody 22040 0.0 S 05:48 /usr/sbin/httpd -k start 159820 1276780 nobody 21932 0.0 S 04:52 /usr/sbin/httpd -k start 159818 1276780 nobody 21292 0.0 S 04:52 /usr/sbin/httpd -k start 159318 1276780 nobody 20928 0.0 S 05:57 /usr/sbin/httpd -k start 160997 160995 mailnull 20768 0.0 S 03:02 /usr/sbin/exim -q 158691 1276780 nobody 20096 0.0 S 07:11 /usr/sbin/httpd -k start 161386 1276780 nobody 19892 0.0 S 02:35 /usr/sbin/httpd -k start 161119 1276780 nobody 19476 0.1 S 02:54 /usr/sbin/httpd -k start 154202 1353308 cpanellogin 19324 0.0 S 15:33 /usr/local/cpanel/base/show_template.stor default_login_theme cpanel server_locale en docroot /usr/local/cpanel/base cpanel_locale 3289952 1 root 19236 0.0 Ss 19:48:54 lfd - sleeping 159808 1276780 nobody 19068 0.0 S 04:52 /usr/sbin/httpd -k start 160476 1276780 nobody 18904 0.0 S 03:32 /usr/sbin/httpd -k start 1463 1 root 18148 0.0 Ss 52-12:01:12 tailwatchd 162933 1276780 nobody 17796 0.0 S 00:25 /usr/sbin/httpd -k start 163100 1276780 nobody 17348 0.0 S 00:17 /usr/sbin/httpd -k start 185104 2431 netdata 17312 2.1 Sl 8-01:37:31 /usr/libexec/netdata/plugins.d/apps.plugin 1 162977 1276780 nobody 17288 0.0 S 00:23 /usr/sbin/httpd -k start 162986 1276780 nobody 17224 0.0 S 00:22 /usr/sbin/httpd -k start 1204 1 zabbix 16724 0.2 Ssl 52-12:01:13 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 163316 1276780 nobody 15776 0.1 S 00:06 /usr/sbin/httpd -k start 1276780 1 root 13456 0.0 Ss 15-06:39:48 /usr/sbin/httpd -k start 162987 1276780 nobody 13424 0.0 S 00:22 /usr/sbin/httpd -k start 162473 1471 root 13336 0.0 S 00:58 cPhulkd - processor - http socket 2107664 1 named 13004 0.0 Ssl 1-19:48:27 /usr/sbin/pdns_server --socket-dir=/run/pdns --guardian=no --daemon=no --disable-syslog --log-timestamp=no --write-pid=no 3416303 1471 root 12748 0.0 S 16:23:49 cPhulkd - dbprocessor 153281 3828812 infinium 12380 0.0 S 17:38 dovecot/imap 162737 3828812 candyfabcom 12272 0.0 S 00:35 dovecot/imap 150138 3828812 lowrange 12256 0.0 S 23:47 dovecot/imap 163337 3828812 corpkicks 12156 0.0 S 00:06 dovecot/imap 160888 2111716 root 11772 0.0 S 03:04 /usr/sbin/exim -q 1471 1 root 11480 0.0 Ss 52-12:01:12 cPhulkd - processor 3668761 3828812 kurri 10928 0.0 S 11:56:52 dovecot/imap 2107879 1 root 10196 0.0 Ss 1-19:48:26 queueprocd - waiting up to 60s to process a task 4028163 27524 root 9092 0.0 Ss 06:51:30 sshd: evolution [priv] 4161699 27524 root 8452 0.0 Ss 04:23:26 sshd: evolution [priv] 4028165 27524 root 8412 0.0 Ss 06:51:30 sshd: root [priv] 1059098 3828812 dovenull 8388 0.0 S 7-06:38:18 dovecot/imap-login 163162 2111716 mailnull 7804 0.0 S 00:11 /usr/sbin/exim -ps -bd -q15m -oP /var/spool/exim/exim-daemon.pid 1466 1 root 7752 0.0 Ss 52-12:01:12 dnsadmin - dormant mode --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 19391847 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min aaa/aaadownunder.com.au-ssl_log 99 whiz/whizengineering.com.au-ssl_log 45 customkidscom/customkids.com.au-ssl_log 44 evolution/evolutionerp.com.au-ssl_log 1 (total vhosts taking traffic: 4) --- top client IPs across all vhosts (last 3 min) --- 96 20.80.111.3 44 68.155.146.130 34 20.226.112.14 11 68.155.153.139 2 68.67.112.145 1 52.173.121.69 1 45.61.187.148 --- top request paths across all vhosts (last 3 min) --- 3 GET /css.php 3 GET /admin.php 2 GET /xx.php 2 GET /ws37.php 2 GET /wp-content/plugins/hellopress/wp_filemanager.php 2 GET /wp-admin/css/colors/blue/index.php 2 GET /this_is_a_new_hello_world.php 2 GET /themes.php 2 GET /install.php 2 GET /file.php 2 GET /cong.php 2 GET /class20.php 2 GET /about.php 2 GET /404.php 2 GET /403.php 1 GET /zoom1.php 1 GET /z60.php 1 GET /xxw.php 1 GET /xv.php 1 GET /xmlrpc.php --- busiest 3 vhosts: top paths + last 10 requests --- == aaa/aaadownunder.com.au-ssl_log == -- top paths -- 1 GET /zoom1.php 1 GET /xx.php 1 GET /xmlrpc.php 1 GET /xda.php 1 GET /ws.php 1 GET /ws81.php 1 GET /ws77.php 1 GET /ws37.php 1 GET /wp-trackback.php 1 GET /wp-the.php 1 GET /wp-themes.php 1 GET /wp-signin.php -- last 10 requests -- 20.80.111.3 - - [20/Aug/2026:22:09:50 +1000] "GET /ca5.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:50 +1000] "GET /install.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:50 +1000] "GET /radio.php HTTP/1.1" 403 13 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:50 +1000] "GET /wp-admin/css/colors/modern/ HTTP/1.1" 200 361 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:50 +1000] "GET /wp-signin.php HTTP/1.1" 403 13 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:51 +1000] "GET /Ov-Simple1.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:51 +1000] "GET /fm.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:51 +1000] "GET /ftde.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:51 +1000] "GET /hplfuns.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" 20.80.111.3 - - [20/Aug/2026:22:09:51 +1000] "GET /log.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" == whiz/whizengineering.com.au-ssl_log == -- top paths -- 2 GET /admin.php 2 GET /about.php 1 GET /xxw.php 1 GET /xx.php 1 GET /xeqdlun.php 1 GET /ws38.php 1 GET /ws37.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /wcas.php 1 GET /tx38.php 1 GET /tor.php 1 GET /this_is_a_new_hello_world.php -- last 10 requests -- 68.155.153.139 - - [20/Aug/2026:22:10:58 +1000] "GET /this_is_a_new_hello_world.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:10:59 +1000] "GET /adminfuns.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:10:59 +1000] "GET /goods.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:10:59 +1000] "GET /100.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:00 +1000] "GET /about.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:00 +1000] "GET /about.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:00 +1000] "GET /admin.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:01 +1000] "GET /admin.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:02 +1000] "GET /themes.php HTTP/1.1" 404 5749 "-" "-" 68.155.153.139 - - [20/Aug/2026:22:11:02 +1000] "GET /biufile.php HTTP/1.1" 404 5749 "-" "-" == customkidscom/customkids.com.au-ssl_log == -- top paths -- 2 GET /css.php 2 GET /cong.php 1 GET /z60.php 1 GET /xv.php 1 GET /x56.php 1 GET /x50.php 1 GET /wsd.php 1 GET /wp-includes/images/wp-login.php 1 GET /w.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /wp-content/index.php 1 GET /wp-admin/maint/about.php -- last 10 requests -- 68.155.146.130 - - [20/Aug/2026:22:10:52 +1000] "GET /cong.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:54 +1000] "GET /wp-includes/images/wp-login.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:54 +1000] "GET /cong.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:54 +1000] "GET /w.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:55 +1000] "GET /404.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:55 +1000] "GET /ioxi-o.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:55 +1000] "GET /0x.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:56 +1000] "GET /css.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:56 +1000] "GET /222.php HTTP/1.1" 401 36 "-" "-" 68.155.146.130 - - [20/Aug/2026:22:10:57 +1000] "GET /wp-content/index.php HTTP/1.1" 401 36 "-" "-" --- last 30 samples (run-up to this incident) --- 2026-08-20 22:08:37 L1=0.65 run=1 blk=0 avail=6674 (56.9%) swpO=0 conn=2 myRSS=2810 php=0 phpRSS=0 ap=22 tcp=21 topuser=- 2026-08-20 22:08:47 L1=0.63 run=3 blk=0 avail=6535 (55.7%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=174 ap=21 tcp=25 topuser=evolution 2026-08-20 22:08:48 L1=0.63 run=1 blk=0 avail=6680 (57.0%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=21 tcp=26 topuser=- 2026-08-20 22:08:57 L1=0.61 run=3 blk=0 avail=6522 (55.6%) swpO=0 conn=1 myRSS=2810 php=3 phpRSS=235 ap=21 tcp=26 topuser=aaa 2026-08-20 22:08:58 L1=0.61 run=1 blk=0 avail=6678 (56.9%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=21 tcp=25 topuser=- 2026-08-20 22:09:07 L1=0.59 run=2 blk=0 avail=6647 (56.7%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=21 tcp=22 topuser=- 2026-08-20 22:09:08 L1=0.59 run=3 blk=0 avail=6534 (55.7%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=144 ap=20 tcp=23 topuser=evolution 2026-08-20 22:09:17 L1=0.73 run=2 blk=0 avail=6532 (55.7%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=148 ap=20 tcp=21 topuser=evolution 2026-08-20 22:09:18 L1=0.73 run=2 blk=0 avail=6624 (56.5%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=20 tcp=21 topuser=- 2026-08-20 22:09:27 L1=0.77 run=6 blk=0 avail=6647 (56.7%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=18 tcp=24 topuser=- 2026-08-20 22:09:28 L1=0.77 run=2 blk=0 avail=6532 (55.7%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=150 ap=18 tcp=23 topuser=evolution 2026-08-20 22:09:37 L1=0.72 run=1 blk=0 avail=6653 (56.7%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=18 tcp=19 topuser=- 2026-08-20 22:09:38 L1=0.72 run=1 blk=0 avail=6651 (56.7%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=23 ap=18 tcp=23 topuser=madalice 2026-08-20 22:09:47 L1=0.69 run=2 blk=0 avail=6147 (52.4%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=17 tcp=21 topuser=- 2026-08-20 22:09:48 L1=0.80 run=3 blk=0 avail=5942 (50.7%) swpO=0 conn=2 myRSS=2810 php=1 phpRSS=89 ap=17 tcp=21 topuser=aaa 2026-08-20 22:09:57 L1=0.89 run=1 blk=0 avail=6701 (57.1%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=15 tcp=19 topuser=- 2026-08-20 22:09:58 L1=0.90 run=2 blk=0 avail=6681 (57.0%) swpO=0 conn=2 myRSS=2810 php=1 phpRSS=49 ap=15 tcp=21 topuser=corpkicks 2026-08-20 22:10:07 L1=0.83 run=1 blk=0 avail=6695 (57.1%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=15 tcp=17 topuser=- 2026-08-20 22:10:08 L1=0.84 run=2 blk=0 avail=6615 (56.4%) swpO=0 conn=2 myRSS=2810 php=1 phpRSS=110 ap=15 tcp=17 topuser=madalice 2026-08-20 22:10:17 L1=0.78 run=1 blk=0 avail=6698 (57.1%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=15 tcp=19 topuser=- 2026-08-20 22:10:18 L1=0.95 run=2 blk=0 avail=6710 (57.2%) swpO=0 conn=2 myRSS=2810 php=0 phpRSS=0 ap=15 tcp=19 topuser=- 2026-08-20 22:10:27 L1=0.88 run=3 blk=0 avail=6706 (57.2%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=14 tcp=22 topuser=- 2026-08-20 22:10:28 L1=0.89 run=2 blk=0 avail=6583 (56.1%) swpO=0 conn=2 myRSS=2810 php=0 phpRSS=0 ap=14 tcp=21 topuser=- 2026-08-20 22:10:37 L1=0.82 run=4 blk=0 avail=6386 (54.4%) swpO=0 conn=1 myRSS=2810 php=3 phpRSS=400 ap=14 tcp=25 topuser=customc 2026-08-20 22:10:38 L1=0.99 run=4 blk=0 avail=6468 (55.1%) swpO=0 conn=2 myRSS=2810 php=3 phpRSS=310 ap=14 tcp=26 topuser=tubesupplies 2026-08-20 22:10:47 L1=0.91 run=2 blk=0 avail=6657 (56.8%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=18 tcp=26 topuser=- 2026-08-20 22:10:48 L1=0.84 run=1 blk=0 avail=6656 (56.8%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=18 tcp=26 topuser=- 2026-08-20 22:10:57 L1=0.85 run=3 blk=0 avail=6308 (53.8%) swpO=0 conn=1 myRSS=2810 php=0 phpRSS=0 ap=19 tcp=28 topuser=- 2026-08-20 22:10:58 L1=0.86 run=2 blk=0 avail=6580 (56.1%) swpO=0 conn=1 myRSS=2810 php=1 phpRSS=41 ap=19 tcp=29 topuser=vinesit 2026-08-20 22:11:07 L1=1.76 run=3 blk=0 avail=4665 (39.8%) swpO=2278 conn=2 myRSS=2763 php=2 phpRSS=118 ap=20 tcp=29 topuser=vinesit