INCIDENT 2026-08-13 22:04:32 trigger: swapout=2607/s ================================================================== load: 1.19 0.63 0.53 2/590 1290241 ncpu:10 mem_avail: 4394MB/11729MB (37.5%) swap_used:1159MB swapout:2607/s blocked(D):1 running:3 iowait:1.1% steal:0.0% php: 0 procs / 0MB heaviest user: - (0 procs, 0MB) apache: 16 procs tcp_estab: 21 --- RSS by user (all processes) --- USER PROCS RSS_MB evolution 7 2755 mysql 1 2331 root 244 949 netdata 8 256 nobody 15 247 dovenull 6 66 cpanellogin 1 50 lowrange 3 45 mailman 20 29 aaa 2 29 kurri 2 28 dovecot 3 28 wp-toolkit 6 20 zabbix 1 14 named 1 7 nscd 1 3 dbus 1 3 cpanelconnecttrack 1 3 polkitd 1 2 mailnull 1 2 --- php workers by user --- USER PROCS RSS_MB --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 4151 MB Active(file): 2055 MB Inactive(file): 2122 MB Dirty: 190 MB Writeback: 0 MB SReclaimable: 386 MB CommitLimit: 9960 MB Committed_AS: 9060 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 342014 root 20:53:15 5036 /usr/bin/imunify-agent-proxy 1287769 root 04:30 3064 /bin/sh -c bash -c "sleep $((RANDOM % 1800))" ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 1290114 1290113 evolution 2766708 94.6 RNs 00:11 /usr/local/cpanel/3rdparty/bin/analog +CIMAGEDIR /images/ +CDOMAINSFILE /usr/local/cpanel/3rdparty/share/analog/usdom.tab +CCHARTDIR ./ 184574 1 mysql 2387048 15.1 Ssl 1-01:31:00 /usr/sbin/mariadbd 1220 1 netdata 204880 2.5 Ssl 45-11:54:33 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 388652 382990 root 174104 0.0 S 19:30:33 spamd child 955146 382990 root 126676 0.0 S 08:13:41 spamd child 382990 1 root 122896 0.0 Ss 19:41:45 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 159820 1 root 114600 0.0 Ss 24-12:17:41 /usr/lib/systemd/systemd-journald 1280741 1353308 cpanellogin 52212 0.0 S 19:00 /usr/local/cpanel/base/show_template.stor docroot /usr/local/cpanel/base cpanel_locale server_locale en default_login_theme cpanel 2223 1201 root 42660 0.0 Sl 45-11:54:32 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1353308 1 root 42020 0.0 Ss 22-19:23:03 cpsrvd (SSL) - waiting for connections 1469 1 root 36076 0.1 Ss 45-11:54:32 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1288598 3651816 root 34496 0.4 SN 02:54 cpanellogd - waiting for child to process logs for evolution 1290061 1288598 evolution 32652 0.4 SN 00:15 cpanellogd - http logs for evolution 4922 2431 netdata 32492 0.1 Sl 45-11:54:28 /usr/libexec/netdata/plugins.d/go.d.plugin 1 382254 1 root 25792 0.0 Ss 19:42:19 lfd - sleeping 1287074 1276780 nobody 21664 0.0 S 05:55 /usr/sbin/httpd -k start 1287274 1276780 nobody 20640 0.0 S 05:27 /usr/sbin/httpd -k start 1287992 1276780 nobody 20640 0.0 S 04:20 /usr/sbin/httpd -k start 1288908 1276780 nobody 20584 0.0 S 02:19 /usr/sbin/httpd -k start 1288904 1276780 nobody 19544 0.0 S 02:20 /usr/sbin/httpd -k start 1287271 1276780 nobody 19312 0.0 S 05:27 /usr/sbin/httpd -k start 1287993 1276780 nobody 19176 0.0 S 04:20 /usr/sbin/httpd -k start 1287998 1276780 nobody 18820 0.0 S 04:19 /usr/sbin/httpd -k start 1288879 1276780 nobody 18544 0.0 S 02:21 /usr/sbin/httpd -k start 1288000 1276780 nobody 18444 0.0 S 04:19 /usr/sbin/httpd -k start 185104 2431 netdata 18368 2.0 Sl 1-01:30:51 /usr/libexec/netdata/plugins.d/apps.plugin 1 1288909 1276780 nobody 17688 0.0 S 02:19 /usr/sbin/httpd -k start 1287272 1276780 nobody 16900 0.0 S 05:27 /usr/sbin/httpd -k start 1282951 1471 root 16508 0.0 S 14:44 cPhulkd - dbprocessor 1059106 3828812 root 16264 0.0 S 06:31:38 dovecot/config 1290057 1276780 nobody 16212 0.0 S 00:19 /usr/sbin/httpd -k start 1289358 3828812 lowrange 15952 0.0 S 01:19 dovecot/imap 1194962 3828812 aaa 15836 0.0 S 02:22:24 dovecot/imap 1289357 3828812 lowrange 15636 0.0 S 01:19 dovecot/imap 1471 1 root 15392 0.0 Ss 45-11:54:32 cPhulkd - processor 1289073 3828812 aaa 14812 0.0 S 02:00 dovecot/imap 1290178 3828812 kurri 14804 0.0 S 00:05 dovecot/imap 1289361 3828812 lowrange 14596 0.0 S 01:19 dovecot/imap 1204 1 zabbix 14428 0.2 Ssl 45-11:54:33 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 1463 1 root 14428 0.0 Ss 45-11:54:32 tailwatchd 1258646 3828812 kurri 14368 0.0 S 38:37 dovecot/imap 1288166 1471 root 14124 0.0 S 03:54 cPhulkd - processor - http socket 1059107 3828812 dovecot 13956 0.0 S 06:31:37 dovecot/auth 1059098 3828812 dovenull 13812 0.0 S 06:31:38 dovecot/imap-login 1288068 3828812 root 13308 0.0 S 04:08 dovecot/auth -w 1288074 3828812 root 13100 0.0 S 04:08 dovecot/auth -w 1287466 3828812 root 13004 0.0 S 04:59 dovecot/auth -w 1059102 3828812 dovenull 12912 0.0 S 06:31:38 dovecot/imap-login 1288069 3828812 root 12888 0.0 S 04:08 dovecot/auth -w 1276780 1 root 12884 0.0 Ss 8-06:33:08 /usr/sbin/httpd -k start 1059104 3828812 dovenull 10504 0.0 S 06:31:38 dovecot/pop3-login 1059100 3828812 dovenull 10452 0.0 S 06:31:38 dovecot/pop3-login 1059099 3828812 dovenull 10172 0.0 S 06:31:38 dovecot/managesieve-login 1059103 3828812 dovenull 10076 0.0 S 06:31:38 dovecot/managesieve-login 908514 1 evolution 9592 0.0 Ss 08:55:14 /usr/lib/systemd/systemd --user 908482 27524 root 9280 0.0 Ss 08:55:14 sshd: evolution [priv] 1059105 3828812 dovecot 7980 0.0 S 06:31:38 dovecot/stats 3651102 1 root 7976 0.0 Ss 5-19:41:46 queueprocd - waiting up to 60s to process a task 1059878 3828812 dovecot 7748 0.0 S 06:30:34 dovecot/imap-hibernate 1 0 root 7628 0.2 Ss 45-11:54:41 /usr/lib/systemd/systemd --switched-root --system --deserialize 17 --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 5375604 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/portal.evolutionerp.com.au-ssl_log 20 (total vhosts taking traffic: 1) --- top client IPs across all vhosts (last 3 min) --- 20 203.54.180.238 --- top request paths across all vhosts (last 3 min) --- 20 GET /heartbeat.php --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 20 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [13/Aug/2026:22:02:50 +1000] "GET /heartbeat.php?sentAt=1786622570&bytes=106,125,178,106&_=1786602369365 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:02:53 +1000] "GET /heartbeat.php?sentAt=1786622573&bytes=106,125,178,109&_=1786596791615 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:00 +1000] "GET /heartbeat.php?sentAt=1786622580&bytes=106,125,178,116&_=1786602369366 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:03 +1000] "GET /heartbeat.php?sentAt=1786622583&bytes=106,125,178,119&_=1786596791616 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:10 +1000] "GET /heartbeat.php?sentAt=1786622590&bytes=106,125,178,126&_=1786602369367 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:13 +1000] "GET /heartbeat.php?sentAt=1786622593&bytes=106,125,178,129&_=1786596791617 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:20 +1000] "GET /heartbeat.php?sentAt=1786622600&bytes=106,125,178,136&_=1786602369368 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:23 +1000] "GET /heartbeat.php?sentAt=1786622603&bytes=106,125,178,139&_=1786596791618 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:30 +1000] "GET /heartbeat.php?sentAt=1786622610&bytes=106,125,178,146&_=1786602369369 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [13/Aug/2026:22:03:33 +1000] "GET /heartbeat.php?sentAt=1786622613&bytes=106,125,178,149&_=1786596791619 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 --- last 30 samples (run-up to this incident) --- 2026-08-13 22:02:08 L1=0.43 run=1 blk=0 avail=6958 (59.3%) swpO=0 conn=2 myRSS=2367 php=0 phpRSS=0 ap=18 tcp=19 topuser=- 2026-08-13 22:02:11 L1=0.43 run=1 blk=0 avail=6956 (59.3%) swpO=0 conn=1 myRSS=2367 php=0 phpRSS=0 ap=18 tcp=29 topuser=- 2026-08-13 22:02:18 L1=0.52 run=3 blk=0 avail=6540 (55.8%) swpO=0 conn=2 myRSS=2367 php=1 phpRSS=174 ap=22 tcp=22 topuser=madalice 2026-08-13 22:02:21 L1=0.52 run=2 blk=0 avail=6638 (56.6%) swpO=0 conn=1 myRSS=2367 php=0 phpRSS=0 ap=21 tcp=21 topuser=- 2026-08-13 22:02:28 L1=0.59 run=2 blk=0 avail=6170 (52.6%) swpO=167 conn=1 myRSS=2365 php=0 phpRSS=0 ap=20 tcp=22 topuser=- 2026-08-13 22:02:31 L1=0.59 run=1 blk=0 avail=6964 (59.4%) swpO=167 conn=1 myRSS=2365 php=0 phpRSS=0 ap=19 tcp=21 topuser=- 2026-08-13 22:02:38 L1=0.50 run=1 blk=0 avail=6970 (59.4%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=17 tcp=19 topuser=- 2026-08-13 22:02:41 L1=0.50 run=2 blk=0 avail=6774 (57.8%) swpO=0 conn=2 myRSS=2365 php=1 phpRSS=241 ap=16 tcp=22 topuser=madalice 2026-08-13 22:02:48 L1=0.57 run=1 blk=0 avail=6982 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=18 topuser=- 2026-08-13 22:02:51 L1=0.57 run=1 blk=0 avail=6980 (59.5%) swpO=0 conn=2 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=19 topuser=- 2026-08-13 22:02:58 L1=0.48 run=1 blk=0 avail=6980 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=20 topuser=- 2026-08-13 22:03:01 L1=0.48 run=1 blk=0 avail=6992 (59.6%) swpO=0 conn=2 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=18 topuser=- 2026-08-13 22:03:08 L1=0.49 run=2 blk=0 avail=6797 (58.0%) swpO=0 conn=1 myRSS=2365 php=1 phpRSS=225 ap=16 tcp=20 topuser=madalice 2026-08-13 22:03:11 L1=0.49 run=2 blk=0 avail=6631 (56.5%) swpO=0 conn=2 myRSS=2365 php=2 phpRSS=416 ap=16 tcp=23 topuser=corpkicks 2026-08-13 22:03:18 L1=0.49 run=1 blk=0 avail=6970 (59.4%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=26 topuser=- 2026-08-13 22:03:21 L1=0.49 run=4 blk=0 avail=6882 (58.7%) swpO=0 conn=2 myRSS=2365 php=1 phpRSS=89 ap=16 tcp=22 topuser=alison 2026-08-13 22:03:28 L1=0.79 run=2 blk=0 avail=6923 (59.0%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=20 topuser=- 2026-08-13 22:03:31 L1=0.80 run=2 blk=0 avail=6865 (58.5%) swpO=0 conn=2 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=20 topuser=- 2026-08-13 22:03:38 L1=0.74 run=1 blk=0 avail=6975 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=23 topuser=- 2026-08-13 22:03:41 L1=0.68 run=1 blk=0 avail=6973 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=22 topuser=- 2026-08-13 22:03:48 L1=0.63 run=2 blk=0 avail=6978 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=21 topuser=- 2026-08-13 22:03:51 L1=0.66 run=3 blk=0 avail=6944 (59.2%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=21 topuser=- 2026-08-13 22:03:58 L1=0.60 run=2 blk=0 avail=6868 (58.6%) swpO=0 conn=1 myRSS=2365 php=1 phpRSS=141 ap=15 tcp=22 topuser=madalice 2026-08-13 22:04:01 L1=0.55 run=1 blk=0 avail=6974 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=23 topuser=- 2026-08-13 22:04:08 L1=0.59 run=1 blk=0 avail=6977 (59.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=15 tcp=18 topuser=- 2026-08-13 22:04:12 L1=0.70 run=3 blk=0 avail=6908 (58.9%) swpO=0 conn=1 myRSS=2365 php=2 phpRSS=123 ap=15 tcp=23 topuser=aaa 2026-08-13 22:04:19 L1=0.65 run=2 blk=0 avail=6743 (57.5%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=20 topuser=- 2026-08-13 22:04:22 L1=0.76 run=2 blk=0 avail=6887 (58.7%) swpO=0 conn=1 myRSS=2365 php=0 phpRSS=0 ap=16 tcp=20 topuser=- 2026-08-13 22:04:29 L1=0.86 run=6 blk=0 avail=4804 (41.0%) swpO=1548 conn=2 myRSS=2346 php=1 phpRSS=161 ap=16 tcp=22 topuser=madalice 2026-08-13 22:04:32 L1=1.19 run=3 blk=1 avail=4394 (37.5%) swpO=2607 conn=1 myRSS=2331 php=0 phpRSS=0 ap=16 tcp=21 topuser=-