INCIDENT 2026-08-25 22:07:39 trigger: swapout=2060/s ================================================================== load: 1.84 1.10 0.99 4/612 2082464 ncpu:10 mem_avail: 4124MB/11729MB (35.2%) swap_used:1236MB swapout:2060/s blocked(D):2 running:2 iowait:12.8% steal:0.0% php: 1 procs / 166MB heaviest user: otterfencing (1 procs, 166MB) apache: 22 procs tcp_estab: 34 --- RSS by user (all processes) --- USER PROCS RSS_MB evolution 7 2819 mysql 1 2366 root 243 951 nobody 21 371 netdata 8 249 otterfencing 1 166 cpanellogin 1 86 infinium 4 56 dovenull 6 46 kurri 3 36 mailnull 3 29 mailman 20 22 wp-toolkit 6 18 dovecot 3 18 zabbix 1 13 named 1 6 nscd 1 5 cpanelconnecttrack 1 3 polkitd 1 2 dbus 1 2 --- php workers by user --- USER PROCS RSS_MB otterfencing 1 166 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 3931 MB Active(file): 1308 MB Inactive(file): 2485 MB Dirty: 366 MB Writeback: 24 MB SReclaimable: 409 MB CommitLimit: 9960 MB Committed_AS: 9680 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 1913717 root 20:56:21 1056 /usr/bin/imunify-agent-proxy 2068799 root 02:35 60264 /opt/imunify360/venv/bin/python3 -m imav.run --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 2082212 2082211 evolution 2839468 96.8 RNs 00:18 /usr/local/cpanel/3rdparty/bin/analog +CIMAGEDIR /images/ +CDOMAINSFILE /usr/local/cpanel/3rdparty/share/analog/usdom.tab +CCHARTDIR ./ 2362432 1 mysql 2422852 17.7 Ssl 18:36:21 /usr/sbin/mariadbd 1220 1 netdata 196344 2.5 Ssl 57-11:57:47 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 2082400 2081302 otterfencing 170144 87.0 S 00:02 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 83438 2145741 root 168912 0.1 S 09:33:38 spamd child 2160755 2145741 root 165232 0.0 S 19:42:14 spamd child 2575382 1 root 129716 0.0 Ss 10-14:17:08 /usr/lib/systemd/systemd-journald 656359 1353308 cpanellogin 88680 0.0 S 06:59:29 /usr/local/cpanel/base/show_template.stor server_locale en default_login_theme cpanel cpanel_locale docroot /usr/local/cpanel/base 2068799 1 root 60264 1.0 Ssl 02:35 /opt/imunify360/venv/bin/python3 -m imav.run 2145741 1 root 52532 0.0 Ss 19:44:36 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 2223 1201 root 47612 0.0 Sl 57-11:57:46 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 2080601 2111368 root 41512 0.7 SN 02:24 cpanellogd - waiting for child to process logs for evolution 2082119 2080601 evolution 39888 0.1 SN 00:23 cpanellogd - http logs for evolution 1469 1 root 38800 0.1 Ss 57-11:57:46 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 2082417 1469 root 37836 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1353308 1 root 33440 0.0 Ss 34-19:26:17 cpsrvd (SSL) - waiting for connections 4922 2431 netdata 33272 0.1 Sl 57-11:57:42 /usr/libexec/netdata/plugins.d/go.d.plugin 1 2080847 2080815 root 25000 0.0 S 02:08 /usr/sbin/exim -q 2064318 1276780 nobody 22416 0.0 S 06:37 /usr/sbin/httpd -k start 2064537 1276780 nobody 22052 0.0 S 06:03 /usr/sbin/httpd -k start 2049384 1276780 nobody 21948 0.0 S 09:10 /usr/sbin/httpd -k start 2064975 1276780 nobody 21680 0.1 S 05:25 /usr/sbin/httpd -k start 2064144 1276780 nobody 21572 0.1 S 06:59 /usr/sbin/httpd -k start 2065862 1276780 nobody 21416 0.0 S 04:02 /usr/sbin/httpd -k start 2069465 1276780 nobody 21416 0.0 S 02:35 /usr/sbin/httpd -k start 2080416 1276780 nobody 20864 0.1 S 02:27 /usr/sbin/httpd -k start 2080849 2080847 mailnull 20352 0.0 S 02:08 /usr/sbin/exim -q 2081297 1276780 nobody 19704 0.1 S 01:19 /usr/sbin/httpd -k start 2081019 1276780 nobody 19604 0.0 S 01:45 /usr/sbin/httpd -k start 2082236 1276780 nobody 19148 0.1 S 00:15 /usr/sbin/httpd -k start 2081303 1276780 nobody 19072 0.1 S 01:18 /usr/sbin/httpd -k start 2081302 1276780 nobody 18868 0.1 S 01:18 /usr/sbin/httpd -k start 2081709 1276780 nobody 18624 0.1 S 00:48 /usr/sbin/httpd -k start 185104 2431 netdata 18528 2.1 Sl 13-01:34:05 /usr/libexec/netdata/plugins.d/apps.plugin 1 2081304 1276780 nobody 17836 0.1 S 01:18 /usr/sbin/httpd -k start 2082237 1276780 nobody 17576 0.0 S 00:15 /usr/sbin/httpd -k start 2081296 1276780 nobody 17284 0.1 S 01:19 /usr/sbin/httpd -k start 2082226 1276780 nobody 17192 0.1 S 00:16 /usr/sbin/httpd -k start 2082397 1276780 nobody 16476 0.0 S 00:02 /usr/sbin/httpd -k start 1982516 3828812 kurri 15112 0.0 S 30:14 dovecot/imap 2000136 3828812 infinium 14648 0.0 S 24:25 dovecot/imap 1463 1 root 14620 0.0 Ss 57-11:57:46 tailwatchd 2033489 3828812 infinium 14512 0.0 S 13:12 dovecot/imap 2064151 3828812 infinium 14500 0.0 S 06:58 dovecot/imap 1951539 3828812 infinium 14380 0.0 S 38:38 dovecot/imap 2082379 3828812 kurri 14284 0.0 S 00:04 dovecot/imap 1204 1 zabbix 14016 0.2 Ssl 57-11:57:47 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 2969139 1471 root 13556 0.0 S 15:34:26 cPhulkd - dbprocessor 1471 1 root 13512 0.0 Ss 57-11:57:46 cPhulkd - processor 2064848 3828812 root 12944 0.0 S 05:37 dovecot/auth -w 2082378 1471 root 12220 0.0 S 00:04 cPhulkd - processor - http socket 2144580 1 root 11760 0.0 Ss 19:45:25 lfd - sleeping 2080815 2111716 root 11700 0.0 S 02:08 /usr/sbin/exim -q 1780156 3828812 dovenull 11348 0.0 S 21:34:47 dovecot/imap-login 1780152 3828812 dovenull 10656 0.0 S 21:34:47 dovecot/imap-login 1780162 3828812 dovecot 9776 0.0 S 21:34:47 dovecot/auth 1780159 3828812 dovenull 9432 0.0 S 21:34:47 dovecot/pop3-login 1780154 3828812 dovenull 9244 0.0 S 21:34:47 dovecot/pop3-login 1795692 3828812 kurri 8172 0.0 S 21:32:04 dovecot/imap 1466 1 root 7756 0.0 Ss 57-11:57:46 dnsadmin - dormant mode --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+-------------+-------------+---------+------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+-------------+-------------+---------+------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4581814 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | | 4581808 | otterfen_wp | otterfen_wp | Query | 0 | Executing | INSERT INTO RrBWkT6j_actionscheduler_actions ( `hook`, `status`, `scheduled_date_gmt`, `scheduled_date_local`, `schedule`, `group_id`, `priority`, `args` ) SELECT 'forminator_send_export', 'pending', '2026-08-25 12:08:39', '2026-08-25 12:08:39', 'O:32:\"ActionScheduler_IntervalSchedule\":5:{s:22:\"\ | +---------+-------------+-------------+---------+------+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/portal.evolutionerp.com.au-ssl_log 122 customkidscom/customkids.com.au-ssl_log 76 evolution/portal.evolutionerp.com.au 44 evolution/my.evolutionerp.com.au-ssl_log 22 otterfencing/otterfencing.com.au-ssl_log 16 madalice/madalice.com.au-ssl_log 9 (total vhosts taking traffic: 6) --- top client IPs across all vhosts (last 3 min) --- 99 20.104.50.220 76 20.48.163.91 67 203.54.180.238 14 20.104.104.62 9 175.36.227.212 4 52.167.144.187 4 40.77.167.17 2 40.77.167.25 2 114.119.140.234 2 114.119.131.88 1 45.148.76.45 1 40.77.189.115 1 40.77.167.76 1 40.77.167.52 1 209.242.198.155 --- top request paths across all vhosts (last 3 min) --- 76 GET /autodiscover/autodiscover.xml 27 GET /factory_session_checker.php 14 GET /heartbeat.php 13 GET /factory_session_processed_qty.php 13 GET /factory_session_joiner.php 10 POST /functions.php 10 GET /index.php 3 GET /wp-content/plugins/hellopress/wp_filemanager.php 3 GET /this_is_a_new_hello_world.php 3 GET /naxc.php 3 GET /LegioN_LeakeRs.php 3 GET / 2 GET /yellow.php 2 GET /wp-content/plugins/seoplugins/index.php/new.php 2 GET /wp-configs.php 2 GET /wp-admin/css/mghnhykio.php/new.php 2 GET /wfile.php 2 GET /v1.php 2 GET /ty.php 2 GET /ton.php --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 27 GET /factory_session_checker.php 14 GET /heartbeat.php 13 GET /factory_session_processed_qty.php 13 GET /factory_session_joiner.php 2 GET /naxc.php 2 GET /LegioN_LeakeRs.php 1 GET /yellow.php 1 GET /ws33.php 1 GET /wp-content/plugins/seoplugins/index.php/new.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /wp-configs.php 1 GET /wp-admin/css/mghnhykio.php/new.php -- last 10 requests -- 20.104.50.220 - - [25/Aug/2026:22:07:27 +1000] "GET /vee.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:27 +1000] "GET /ws33.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:28 +1000] "GET /menu.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:28 +1000] "GET /1.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:28 +1000] "GET /aa.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:29 +1000] "GET /bolt.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:29 +1000] "GET /v2.php HTTP/1.1" 404 - "-" "-" 203.54.180.238 - - [25/Aug/2026:22:07:29 +1000] "GET /factory_session_checker.php HTTP/1.1" 200 35 "https://portal.evolutionerp.com.au/index.php?page=factory_workbench" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537 20.104.50.220 - - [25/Aug/2026:22:07:30 +1000] "GET /class9.php HTTP/1.1" 404 - "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:30 +1000] "GET /abc.php HTTP/1.1" 404 - "-" "-" == customkidscom/customkids.com.au-ssl_log == -- top paths -- 76 GET /autodiscover/autodiscover.xml -- last 10 requests -- 20.48.163.91 - - [25/Aug/2026:22:07:19 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:19 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:20 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:21 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:22 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:23 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:24 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:25 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:26 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" 20.48.163.91 - - [25/Aug/2026:22:07:27 +1000] "GET /autodiscover/autodiscover.xml HTTP/1.1" 400 52 "-" "-" == evolution/portal.evolutionerp.com.au == -- top paths -- 1 GET /yellow.php 1 GET /wp-content/plugins/seoplugins/index.php/new.php 1 GET /wp-content/plugins/hellopress/wp_filemanager.php 1 GET /wp-configs.php 1 GET /wp-admin/css/mghnhykio.php/new.php 1 GET /wfile.php 1 GET /v1.php 1 GET /ty.php 1 GET /ton.php 1 GET /this_is_a_new_hello_world.php 1 GET /ppp.php 1 GET /p.php -- last 10 requests -- 20.104.50.220 - - [25/Aug/2026:22:07:22 +1000] "GET /p.php HTTP/1.1" 301 288 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:22 +1000] "GET /cc.php HTTP/1.1" 301 289 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:23 +1000] "GET /v1.php HTTP/1.1" 301 289 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:23 +1000] "GET /0x.php HTTP/1.1" 301 289 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:23 +1000] "GET /class629.php HTTP/1.1" 301 295 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:24 +1000] "GET /file21.php HTTP/1.1" 301 293 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:24 +1000] "GET /yellow.php HTTP/1.1" 301 293 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:25 +1000] "GET /777.php HTTP/1.1" 301 290 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:25 +1000] "GET /030.php HTTP/1.1" 301 290 "-" "-" 20.104.50.220 - - [25/Aug/2026:22:07:25 +1000] "GET /cfile.php HTTP/1.1" 301 292 "-" "-" --- last 30 samples (run-up to this incident) --- 2026-08-25 22:05:18 L1=1.55 run=8 blk=0 avail=6227 (53.1%) swpO=1 conn=5 myRSS=2418 php=5 phpRSS=836 ap=20 tcp=36 topuser=customc 2026-08-25 22:05:18 L1=1.55 run=1 blk=0 avail=6951 (59.3%) swpO=1 conn=2 myRSS=2418 php=0 phpRSS=0 ap=20 tcp=36 topuser=- 2026-08-25 22:05:28 L1=1.39 run=4 blk=0 avail=6604 (56.3%) swpO=0 conn=5 myRSS=2418 php=1 phpRSS=58 ap=21 tcp=30 topuser=alison 2026-08-25 22:05:28 L1=1.39 run=4 blk=0 avail=6581 (56.1%) swpO=0 conn=2 myRSS=2418 php=1 phpRSS=82 ap=21 tcp=30 topuser=alison 2026-08-25 22:05:38 L1=1.18 run=1 blk=0 avail=6948 (59.2%) swpO=0 conn=5 myRSS=2418 php=0 phpRSS=0 ap=18 tcp=31 topuser=- 2026-08-25 22:05:38 L1=1.18 run=2 blk=0 avail=6948 (59.2%) swpO=0 conn=2 myRSS=2418 php=0 phpRSS=0 ap=18 tcp=30 topuser=- 2026-08-25 22:05:48 L1=1.08 run=3 blk=0 avail=6952 (59.3%) swpO=0 conn=5 myRSS=2418 php=0 phpRSS=0 ap=16 tcp=28 topuser=- 2026-08-25 22:05:48 L1=1.08 run=3 blk=1 avail=6875 (58.6%) swpO=0 conn=2 myRSS=2418 php=0 phpRSS=0 ap=16 tcp=28 topuser=- 2026-08-25 22:05:58 L1=1.22 run=3 blk=0 avail=6916 (59.0%) swpO=0 conn=5 myRSS=2418 php=1 phpRSS=57 ap=17 tcp=29 topuser=alison 2026-08-25 22:05:58 L1=1.22 run=3 blk=0 avail=6872 (58.6%) swpO=0 conn=2 myRSS=2418 php=1 phpRSS=84 ap=17 tcp=29 topuser=alison 2026-08-25 22:06:08 L1=1.11 run=1 blk=0 avail=6946 (59.2%) swpO=0 conn=5 myRSS=2418 php=0 phpRSS=0 ap=16 tcp=25 topuser=- 2026-08-25 22:06:08 L1=1.11 run=1 blk=0 avail=6944 (59.2%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=16 tcp=25 topuser=- 2026-08-25 22:06:18 L1=1.18 run=7 blk=1 avail=6350 (54.1%) swpO=1 conn=4 myRSS=2418 php=4 phpRSS=717 ap=17 tcp=32 topuser=madalice 2026-08-25 22:06:18 L1=1.18 run=5 blk=0 avail=6542 (55.8%) swpO=1 conn=1 myRSS=2418 php=3 phpRSS=490 ap=17 tcp=33 topuser=otterfencing 2026-08-25 22:06:28 L1=1.00 run=2 blk=1 avail=6910 (58.9%) swpO=0 conn=4 myRSS=2418 php=0 phpRSS=0 ap=22 tcp=28 topuser=- 2026-08-25 22:06:28 L1=1.00 run=2 blk=0 avail=6905 (58.9%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=22 tcp=28 topuser=- 2026-08-25 22:06:38 L1=1.08 run=3 blk=0 avail=6632 (56.5%) swpO=1 conn=4 myRSS=2418 php=2 phpRSS=377 ap=18 tcp=30 topuser=corpkicks 2026-08-25 22:06:38 L1=1.08 run=2 blk=0 avail=6619 (56.4%) swpO=1 conn=1 myRSS=2418 php=2 phpRSS=388 ap=18 tcp=30 topuser=corpkicks 2026-08-25 22:06:48 L1=1.07 run=1 blk=1 avail=6950 (59.3%) swpO=0 conn=4 myRSS=2418 php=0 phpRSS=0 ap=18 tcp=25 topuser=- 2026-08-25 22:06:48 L1=1.07 run=1 blk=0 avail=6950 (59.3%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=18 tcp=25 topuser=- 2026-08-25 22:06:58 L1=1.06 run=2 blk=0 avail=6923 (59.0%) swpO=0 conn=4 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=30 topuser=- 2026-08-25 22:06:58 L1=1.06 run=2 blk=0 avail=6915 (59.0%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=30 topuser=- 2026-08-25 22:07:08 L1=1.13 run=5 blk=0 avail=6765 (57.7%) swpO=0 conn=4 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=32 topuser=- 2026-08-25 22:07:08 L1=1.13 run=2 blk=0 avail=6760 (57.6%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=32 topuser=- 2026-08-25 22:07:18 L1=1.11 run=2 blk=0 avail=6671 (56.9%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=31 topuser=- 2026-08-25 22:07:19 L1=1.11 run=3 blk=0 avail=6664 (56.8%) swpO=0 conn=1 myRSS=2418 php=0 phpRSS=0 ap=19 tcp=31 topuser=- 2026-08-25 22:07:29 L1=1.17 run=2 blk=0 avail=4735 (40.4%) swpO=520 conn=1 myRSS=2404 php=0 phpRSS=0 ap=22 tcp=32 topuser=- 2026-08-25 22:07:29 L1=1.17 run=6 blk=0 avail=4716 (40.2%) swpO=627 conn=1 myRSS=2403 php=0 phpRSS=0 ap=22 tcp=30 topuser=- 2026-08-25 22:07:39 L1=1.84 run=2 blk=2 avail=4124 (35.2%) swpO=2060 conn=2 myRSS=2366 php=1 phpRSS=166 ap=22 tcp=34 topuser=otterfencing 2026-08-25 22:07:39 L1=1.84 run=4 blk=2 avail=4118 (35.1%) swpO=2005 conn=2 myRSS=2366 php=1 phpRSS=166 ap=22 tcp=34 topuser=otterfencing