INCIDENT 2026-08-20 21:06:39 trigger: swapout=2147/s ================================================================== load: 0.91 0.76 0.90 4/738 124979 ncpu:10 mem_avail: 5883MB/11729MB (50.2%) swap_used:2891MB swapout:2147/s blocked(D):0 running:8 iowait:0.4% steal:0.0% php: 3 procs / 277MB heaviest user: madalice (1 procs, 160MB) apache: 120 procs tcp_estab: 118 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2802 nobody 119 2373 root 247 955 netdata 8 262 madalice 1 165 mailman 20 97 customc 1 86 evolution 9 71 cpanellogin 1 69 dovenull 6 26 wp-toolkit 6 17 zabbix 1 15 named 1 12 aaa 1 11 kurri 1 10 cpanelphpmyadmin 1 8 dovecot 3 7 nscd 1 6 mailnull 1 5 cpanelconnecttrack 1 3 --- php workers by user --- USER PROCS RSS_MB madalice 1 167 customc 1 89 evolution 1 39 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 3930 MB Active(file): 2505 MB Inactive(file): 1465 MB Dirty: 0 MB Writeback: 0 MB SReclaimable: 485 MB CommitLimit: 9960 MB Committed_AS: 8822 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 120758 root 06:38 1304 /bin/sh -c bash -c "sleep $((RANDOM % 1800))" ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 3248642 root 19:55:23 2652 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2575700 1 mysql 2870140 11.7 Ssl 5-13:16:03 /usr/sbin/mariadbd 1220 1 netdata 211360 2.5 Ssl 52-10:56:45 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 3298115 3290507 root 181136 0.0 S 18:31:04 spamd child 124903 124498 madalice 178296 72.0 R 00:02 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 3864059 3290507 root 134420 0.0 S 08:07:04 spamd child 2575382 1 root 111116 0.0 Ss 5-13:16:06 /usr/lib/systemd/systemd-journald 124926 124484 customc 105380 35.0 R 00:01 /opt/cpanel/ea-php81/root/usr/bin/php-cgi 4182872 1353308 cpanellogin 70752 0.0 S 03:03:58 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base default_login_theme cpanel server_locale en 3290507 1 root 48548 0.0 Ss 18:43:53 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 2223 1201 root 42148 0.0 Sl 52-10:56:44 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1469 1 root 40852 0.1 Ss 52-10:56:44 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 124945 124913 evolution 40492 0.0 S 00:00 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 124946 1469 root 38360 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 124963 1469 root 38356 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1353308 1 root 37984 0.0 Ss 29-18:25:15 cpsrvd (SSL) - waiting for connections 4922 2431 netdata 32844 0.1 Sl 52-10:56:40 /usr/libexec/netdata/plugins.d/go.d.plugin 1 123657 1276780 nobody 25660 0.0 S 01:23 /usr/sbin/httpd -k start 122952 1276780 nobody 25604 0.0 S 02:29 /usr/sbin/httpd -k start 123041 1276780 nobody 25392 0.0 S 02:17 /usr/sbin/httpd -k start 123664 1276780 nobody 25000 0.0 S 01:22 /usr/sbin/httpd -k start 122477 1276780 nobody 24956 0.1 S 03:27 /usr/sbin/httpd -k start 123288 1276780 nobody 24740 0.0 S 01:45 /usr/sbin/httpd -k start 123035 1276780 nobody 24736 0.0 S 02:18 /usr/sbin/httpd -k start 121394 1276780 nobody 24356 0.0 S 05:28 /usr/sbin/httpd -k start 124496 1276780 nobody 24296 0.0 S 00:13 /usr/sbin/httpd -k start 122484 1276780 nobody 24096 0.0 S 03:26 /usr/sbin/httpd -k start 121312 1276780 nobody 23988 0.0 S 05:32 /usr/sbin/httpd -k start 122950 1276780 nobody 23708 0.1 S 02:30 /usr/sbin/httpd -k start 124044 1276780 nobody 23604 0.0 S 00:41 /usr/sbin/httpd -k start 124393 1276780 nobody 23584 0.1 S 00:17 /usr/sbin/httpd -k start 121205 1276780 nobody 23568 0.0 S 05:47 /usr/sbin/httpd -k start 124471 1276780 nobody 23568 0.1 S 00:14 /usr/sbin/httpd -k start 124420 1276780 nobody 23536 0.1 S 00:16 /usr/sbin/httpd -k start 123199 1276780 nobody 23492 0.0 S 01:59 /usr/sbin/httpd -k start 124467 1276780 nobody 23480 0.0 S 00:14 /usr/sbin/httpd -k start 124451 1276780 nobody 23456 0.0 S 00:15 /usr/sbin/httpd -k start 121415 1276780 nobody 23440 0.0 S 05:28 /usr/sbin/httpd -k start 123695 1276780 nobody 23440 0.0 S 01:21 /usr/sbin/httpd -k start 124473 1276780 nobody 23428 0.0 S 00:14 /usr/sbin/httpd -k start 124497 1276780 nobody 23420 0.0 S 00:13 /usr/sbin/httpd -k start 124494 1276780 nobody 23392 0.0 S 00:13 /usr/sbin/httpd -k start 124440 1276780 nobody 23376 0.1 S 00:15 /usr/sbin/httpd -k start 124475 1276780 nobody 23332 0.0 S 00:14 /usr/sbin/httpd -k start 123700 1276780 nobody 23328 0.0 S 01:21 /usr/sbin/httpd -k start 124910 1276780 nobody 23176 0.0 S 00:01 /usr/sbin/httpd -k start 121360 1276780 nobody 23168 0.0 S 05:31 /usr/sbin/httpd -k start 124482 1276780 nobody 23148 0.2 S 00:13 /usr/sbin/httpd -k start 123334 1276780 nobody 22708 0.0 S 01:41 /usr/sbin/httpd -k start 124450 1276780 nobody 22148 0.0 S 00:15 /usr/sbin/httpd -k start 124445 1276780 nobody 21948 0.0 S 00:15 /usr/sbin/httpd -k start 124042 1276780 nobody 21688 0.0 S 00:41 /usr/sbin/httpd -k start 124244 1276780 nobody 21572 0.0 S 00:25 /usr/sbin/httpd -k start 121361 1276780 nobody 21536 0.0 S 05:31 /usr/sbin/httpd -k start 124439 1276780 nobody 21520 0.0 S 00:15 /usr/sbin/httpd -k start 124498 1276780 nobody 21516 0.0 S 00:13 /usr/sbin/httpd -k start 124449 1276780 nobody 21504 0.0 S 00:15 /usr/sbin/httpd -k start 124447 1276780 nobody 21460 0.0 S 00:15 /usr/sbin/httpd -k start 124444 1276780 nobody 21456 0.0 S 00:15 /usr/sbin/httpd -k start 124495 1276780 nobody 21396 0.0 S 00:13 /usr/sbin/httpd -k start 124021 1276780 nobody 21368 0.0 S 00:42 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 19388051 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min infinium/infinium.net.au-ssl_log 260 coscom/coscom.com.au 94 speedlin/speedlinefencing.com.au-ssl_log 89 evolution/evolutionerp.com.au-ssl_log 62 kurri/kurrimotorinn.com.au-ssl_log 47 lowrange/lowrange.com.au-ssl_log 44 evolution/portal.evolutionerp.com.au-ssl_log 15 madalice/madalice.com.au-ssl_log 11 alison/alisonarts.com.au-ssl_log 9 tubesupplies/tubesupplies.com.au-ssl_log 2 (total vhosts taking traffic: 10) --- top client IPs across all vhosts (last 3 min) --- 260 34.158.52.41 94 20.51.153.15 89 20.104.100.201 50 158.23.18.197 47 74.248.136.165 44 20.38.3.247 15 203.54.180.238 9 216.73.216.108 7 27.32.89.221 6 34.59.229.251 2 85.204.70.88 2 185.91.69.242 2 114.119.130.40 1 64.233.172.171 1 64.233.172.160 --- top request paths across all vhosts (last 3 min) --- 47 GET /autodiscover/autodiscover.xml 21 GET /fetch 6 GET /heartbeat.php 5 GET /store/highlife-custom-af1-shoes-copy/ 5 GET / 4 GET /wp-content/plugins/hellopress/wp_filemanager.php 4 GET /this_is_a_new_hello_world.php 3 GET /read 3 GET /index.php 3 GET /@fs/proc/self/environ 3 GET /download 3 GET /category/work/custom-airbrush-painted-hard-hats/ 3 GET /aa.php 2 POST /factory_functions.php 2 GET /wpxml.php 2 GET /webhook 2 GET /v2.php 2 GET /simtars-30-years-custom-hard-hat/ 2 GET /redirect 2 GET /proxy --- busiest 3 vhosts: top paths + last 10 requests --- == infinium/infinium.net.au-ssl_log == -- top paths -- 21 GET /fetch 3 GET /read 3 GET /@fs/proc/self/environ 3 GET /download 3 GET / 2 GET /webhook 2 GET /redirect 2 GET /proxy 2 GET /preview 2 GET /@fs/root/.aws/credentials 2 GET /@fs/proc/1/environ 2 GET /@fs/home/ec2-user/.aws/credentials -- last 10 requests -- 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /frontend/.env.production HTTP/1.1" 403 12 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /ci/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /phpinfo.php HTTP/1.1" 403 12 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /production/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /old/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /conf/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /html/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /.openclaw/.env HTTP/1.1" 403 12 "-" "Mozilla/5.0 (compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /api/v1/aws/credentials HTTP/1.1" 404 3421 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" 34.158.52.41 - - [20/Aug/2026:21:06:39 +1000] "GET /aws/ecs/task-credentials HTTP/1.1" 404 3421 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)" == coscom/coscom.com.au == -- top paths -- 1 GET /zk.php 1 GET /zews.php 1 GET /zaza.php 1 GET /xox.php 1 GET /xmrlpc.php 1 GET /xdmah.php 1 GET /ww.php 1 GET /wp-mt.php 1 GET /wp-mn.php 1 GET /wp-gr.php 1 GET /wp-crom.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php -- last 10 requests -- 20.51.153.15 - - [20/Aug/2026:21:05:43 +1000] "GET /release.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:43 +1000] "GET /validate.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:43 +1000] "GET /dummy.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:43 +1000] "GET /Mo0n.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:44 +1000] "GET /cron.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:44 +1000] "GET /uwu.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:44 +1000] "GET /ccc.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:44 +1000] "GET /test-env.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:44 +1000] "GET /789.php HTTP/1.1" 404 - "-" "-" 20.51.153.15 - - [20/Aug/2026:21:05:45 +1000] "GET /1975Team.php HTTP/1.1" 404 - "-" "-" == speedlin/speedlinefencing.com.au-ssl_log == -- top paths -- 1 GET /zxz.php 1 GET /zero.php 1 GET /z43agz.php 1 GET /yawa.php 1 GET /x1da.php 1 GET /ws77.php 1 GET /wpxml.php 1 GET /wp-temp.php 1 GET /wp-includes/Text/Diff/Engine/file_cdd9.php 1 GET /wp-css.php 1 GET /wp-content/replace.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php -- last 10 requests -- 20.104.100.201 - - [20/Aug/2026:21:06:14 +1000] "GET /bal.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:15 +1000] "GET /yawa.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:15 +1000] "GET /wp-admin/maint/max.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:16 +1000] "GET /7.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:16 +1000] "GET /ws77.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:17 +1000] "GET /read.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:17 +1000] "GET /albin.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:18 +1000] "GET /fw/34.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:18 +1000] "GET /wp9.php HTTP/1.1" 401 36 "-" "-" 20.104.100.201 - - [20/Aug/2026:21:06:18 +1000] "GET /save.php HTTP/1.1" 401 36 "-" "-" --- last 30 samples (run-up to this incident) --- 2026-08-20 21:04:18 L1=0.58 run=3 blk=0 avail=6361 (54.2%) swpO=0 conn=2 myRSS=2830 php=3 phpRSS=252 ap=34 tcp=37 topuser=madalice 2026-08-20 21:04:18 L1=0.58 run=6 blk=0 avail=6275 (53.5%) swpO=0 conn=2 myRSS=2830 php=4 phpRSS=369 ap=34 tcp=37 topuser=madalice 2026-08-20 21:04:28 L1=0.57 run=1 blk=0 avail=6481 (55.3%) swpO=1 conn=2 myRSS=2830 php=0 phpRSS=0 ap=39 tcp=34 topuser=- 2026-08-20 21:04:29 L1=0.57 run=1 blk=0 avail=6482 (55.3%) swpO=1 conn=1 myRSS=2830 php=0 phpRSS=0 ap=39 tcp=32 topuser=- 2026-08-20 21:04:38 L1=0.63 run=2 blk=0 avail=6431 (54.8%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=146 ap=30 tcp=34 topuser=alison 2026-08-20 21:04:39 L1=0.63 run=2 blk=0 avail=6514 (55.5%) swpO=0 conn=1 myRSS=2830 php=1 phpRSS=60 ap=30 tcp=39 topuser=evolution 2026-08-20 21:04:48 L1=0.60 run=1 blk=0 avail=6527 (55.7%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=32 tcp=34 topuser=- 2026-08-20 21:04:49 L1=0.60 run=1 blk=0 avail=6526 (55.6%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=32 tcp=34 topuser=- 2026-08-20 21:04:58 L1=0.51 run=2 blk=0 avail=6516 (55.6%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=33 tcp=36 topuser=- 2026-08-20 21:04:59 L1=0.51 run=1 blk=0 avail=6512 (55.5%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=34 tcp=36 topuser=- 2026-08-20 21:05:08 L1=0.51 run=2 blk=0 avail=6174 (52.6%) swpO=0 conn=1 myRSS=2830 php=2 phpRSS=413 ap=32 tcp=36 topuser=corpkicks 2026-08-20 21:05:09 L1=0.51 run=2 blk=0 avail=6334 (54.0%) swpO=0 conn=1 myRSS=2830 php=1 phpRSS=221 ap=32 tcp=33 topuser=corpkicks 2026-08-20 21:05:18 L1=0.74 run=3 blk=0 avail=6400 (54.6%) swpO=0 conn=1 myRSS=2830 php=2 phpRSS=138 ap=39 tcp=44 topuser=evolution 2026-08-20 21:05:19 L1=0.74 run=2 blk=0 avail=6323 (53.9%) swpO=0 conn=1 myRSS=2830 php=1 phpRSS=140 ap=47 tcp=45 topuser=evolution 2026-08-20 21:05:29 L1=0.70 run=1 blk=0 avail=6483 (55.3%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=39 tcp=32 topuser=- 2026-08-20 21:05:29 L1=0.70 run=3 blk=0 avail=6480 (55.2%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=38 tcp=32 topuser=evolution 2026-08-20 21:05:39 L1=0.67 run=1 blk=0 avail=6548 (55.8%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=31 tcp=33 topuser=- 2026-08-20 21:05:39 L1=0.67 run=2 blk=0 avail=6530 (55.7%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=30 tcp=32 topuser=evolution 2026-08-20 21:05:49 L1=0.65 run=1 blk=0 avail=6543 (55.8%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=30 tcp=32 topuser=- 2026-08-20 21:05:49 L1=0.65 run=1 blk=0 avail=6530 (55.7%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=30 tcp=33 topuser=evolution 2026-08-20 21:05:59 L1=0.63 run=1 blk=0 avail=6527 (55.7%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=31 tcp=36 topuser=- 2026-08-20 21:05:59 L1=0.63 run=2 blk=0 avail=6504 (55.5%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=33 tcp=37 topuser=evolution 2026-08-20 21:06:09 L1=0.77 run=4 blk=0 avail=6339 (54.0%) swpO=0 conn=1 myRSS=2830 php=2 phpRSS=245 ap=32 tcp=35 topuser=olympic 2026-08-20 21:06:09 L1=0.77 run=3 blk=0 avail=6286 (53.6%) swpO=0 conn=2 myRSS=2830 php=3 phpRSS=328 ap=32 tcp=34 topuser=evolution 2026-08-20 21:06:19 L1=0.80 run=1 blk=0 avail=6508 (55.5%) swpO=0 conn=1 myRSS=2830 php=0 phpRSS=0 ap=33 tcp=39 topuser=- 2026-08-20 21:06:19 L1=0.80 run=2 blk=0 avail=6495 (55.4%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=33 tcp=40 topuser=evolution 2026-08-20 21:06:29 L1=0.99 run=1 blk=0 avail=6090 (51.9%) swpO=0 conn=1 myRSS=2830 php=1 phpRSS=22 ap=94 tcp=96 topuser=evolution 2026-08-20 21:06:29 L1=0.99 run=1 blk=0 avail=6084 (51.9%) swpO=0 conn=2 myRSS=2830 php=1 phpRSS=39 ap=94 tcp=93 topuser=evolution 2026-08-20 21:06:39 L1=0.91 run=8 blk=0 avail=5883 (50.2%) swpO=2147 conn=4 myRSS=2802 php=3 phpRSS=277 ap=120 tcp=118 topuser=madalice 2026-08-20 21:06:39 L1=0.91 run=3 blk=1 avail=5734 (48.9%) swpO=2147 conn=4 myRSS=2802 php=3 phpRSS=360 ap=120 tcp=125 topuser=madalice