INCIDENT 2026-09-05 02:12:17 trigger: php_rss=1602MB ================================================================== load: 1.74 1.84 1.70 5/682 3590959 ncpu:10 mem_avail: 6446MB/13777MB (46.8%) swap_used:1824MB swapout:0/s blocked(D):0 running:3 iowait:0.8% steal:0.0% php: 15 procs / 1602MB heaviest user: madalice (8 procs, 1158MB) apache: 15 procs tcp_estab: 35 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 3356 madalice 8 1158 root 257 1146 netdata 8 259 nobody 14 236 customc 1 186 evolution 12 184 dovenull 6 65 cpanellogin 1 57 wp-toolkit 3 56 kurri 3 42 lowrange 2 29 dovecot 3 28 mailman 20 25 infinium 1 14 zabbix 1 11 polkitd 1 7 named 1 6 nscd 1 4 mailnull 1 2 --- php workers by user --- USER PROCS RSS_MB madalice 8 1158 customc 1 186 evolution 4 161 root 2 97 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 4225 MB Active(file): 3589 MB Inactive(file): 1192 MB Dirty: 5 MB Writeback: 0 MB SReclaimable: 1559 MB CommitLimit: 10984 MB Committed_AS: 8246 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 3541449 root 12:16 3028 /bin/sh -c bash -c 'sleep $((RANDOM % 1800))' ;imunify-antivirus imunify-patch subscriptions refresh > /dev/null 2>&1 3926885 root 3-23:49:50 3896 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 3436960 12.6 Ssl 2-11:58:05 /usr/sbin/mariadbd 1043892 2603077 root 197472 0.1 S 12:07:21 spamd child 1542443 1 netdata 192876 2.6 Ssl 1-23:49:55 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 3590927 3573529 customc 191344 80.0 S 00:01 /opt/cpanel/ea-php81/root/usr/bin/php-cgi 3385916 3385914 root 178636 0.4 S 01:01:16 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/scripts/gather_update_log_stats --logfile /var/cpanel/updatelogs/last --upload 3590406 2601907 madalice 161232 8.5 S 00:59 php-fpm: pool madalice_com_au 3590652 2601907 madalice 159840 10.3 S 00:25 php-fpm: pool madalice_com_au 3590655 2601907 madalice 158908 10.3 S 00:25 php-fpm: pool madalice_com_au 3590653 2601907 madalice 158728 9.0 S 00:25 php-fpm: pool madalice_com_au 3590659 2601907 madalice 137228 5.7 S 00:24 php-fpm: pool madalice_com_au 3590657 2601907 madalice 137144 6.3 S 00:24 php-fpm: pool madalice_com_au 3590658 2601907 madalice 137140 6.5 S 00:24 php-fpm: pool madalice_com_au 3590664 2601907 madalice 136164 6.2 S 00:24 php-fpm: pool madalice_com_au 2603077 1 root 132932 0.0 Ss 23:48:06 /usr/local/cpanel/3rdparty/perl/542/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 -- 3590888 3385916 root 71096 98.0 R 00:07 /usr/local/cpanel/3rdparty/bin/php /opt/cpanel/ea-wappspector/wappspector.phar --max=1 --json --depth=9 /home/nrg 2654376 2603077 root 68300 0.0 S 23:33:10 spamd child 3127357 3127312 evolution 63176 0.0 Ss 02:12:16 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero_payment_cron.php 3127358 3127320 evolution 61664 0.0 Ss 02:12:16 /opt/cpanel/ea-php83/root/usr/bin/php-cgi /home/evolution/my.evolutionerp.com.au/cron/xero2.php 3454439 1353308 cpanellogin 59120 0.0 S 40:56 /usr/local/cpanel/base/show_template.stor cpanel_locale docroot /usr/local/cpanel/base server_locale en default_login_theme cpanel 3590969 3590668 customc 51160 0.0 R 00:00 /opt/cpanel/ea-php81/root/usr/bin/php-cgi 509236 509098 root 42788 0.0 Sl 7-23:52:36 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 1353308 1 root 42280 0.1 Ss 44-23:31:02 cpsrvd (SSL) - waiting for connections 1469 1 root 42096 0.1 Ss 67-16:02:31 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1542981 1542551 netdata 40880 0.1 Sl 1-23:49:51 /usr/libexec/netdata/plugins.d/go.d.plugin 1 3590928 3558373 evolution 40868 3.0 S 00:01 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 3127515 1 root 36252 0.0 Ss 02:12:15 lfd - sleeping 1542487 1 wp-toolkit 30656 0.0 Ss 1-23:49:55 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script background-tasks-executor.php 2601907 1 root 28876 0.0 Rs 23:48:51 php-fpm: master process (/opt/cpanel/ea-php83/root/etc/php-fpm.conf) 1463 1 root 23260 0.0 Ss 67-16:02:31 tailwatchd 2602226 1 root 23088 0.1 Ssl 23:48:36 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P 3558373 506832 nobody 22792 0.0 S 07:31 /usr/sbin/httpd -k start 3573529 506832 nobody 22044 0.0 S 05:55 /usr/sbin/httpd -k start 3558370 506832 nobody 21988 0.1 S 07:31 /usr/sbin/httpd -k start 3558369 506832 nobody 21384 0.0 S 07:31 /usr/sbin/httpd -k start 3573669 506832 nobody 21324 0.1 S 05:37 /usr/sbin/httpd -k start 1542531 1 wp-toolkit 20596 0.0 Ss 1-23:49:54 /usr/bin/sw-engine /usr/local/cpanel/3rdparty/wp-toolkit/bin/run-script scheduled-tasks-executor.php 3590494 506832 nobody 20264 0.0 S 00:45 /usr/sbin/httpd -k start 3590410 506832 nobody 19304 0.0 S 00:57 /usr/sbin/httpd -k start 3590484 506832 nobody 19212 0.0 S 00:47 /usr/sbin/httpd -k start 1542982 1542551 netdata 18632 2.1 Sl 1-23:49:51 /usr/libexec/netdata/plugins.d/apps.plugin 1 3590492 506832 nobody 18032 0.0 S 00:45 /usr/sbin/httpd -k start 3590678 506832 nobody 17228 0.0 S 00:21 /usr/sbin/httpd -k start 1542141 1 root 16952 0.0 Ss 1-23:49:58 sw-engine-fpm: master process (/etc/sw-engine/sw-engine-fpm.conf) 3590668 506832 nobody 16696 0.0 S 00:23 /usr/sbin/httpd -k start 3590663 506832 nobody 16572 0.0 S 00:24 /usr/sbin/httpd -k start 3212102 508511 lowrange 15796 0.0 S 01:50:14 dovecot/imap 3160309 508511 kurri 15160 0.0 S 02:05:03 dovecot/imap 690513 508511 root 14908 0.0 S 13:39:27 dovecot/config 3539428 508511 lowrange 14804 0.0 S 16:09 dovecot/imap 3539094 508511 infinium 14684 0.0 S 16:39 dovecot/imap 3590276 508511 kurri 14476 0.0 S 01:11 dovecot/imap 3141944 508511 kurri 14352 0.0 S 02:11:54 dovecot/imap 509266 1 root 14148 0.0 Ss 7-23:52:36 /usr/lib/systemd/systemd-journald 690514 508511 dovecot 13912 0.0 S 13:39:27 dovecot/auth 2508826 1471 root 13580 0.0 S 1-00:15:20 cPhulkd - dbprocessor 690505 508511 dovenull 13412 0.0 S 13:39:27 dovecot/imap-login 3590337 508511 root 13012 0.0 S 01:06 dovecot/auth -w 506832 1 root 12980 0.0 Ss 7-23:53:09 /usr/sbin/httpd -k start 690509 508511 dovenull 12948 0.0 S 13:39:27 dovecot/imap-login 3590649 1471 root 12908 0.0 S 00:25 cPhulkd - processor - http socket --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 10607808 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +----------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/app.evolutionerp.com.au-ssl_log 122 madalice/madalice.com.au-ssl_log 7 evolution/portal.evolutionerp.com.au-ssl_log 6 evolution/my.evolutionerp.com.au-ssl_log 3 (total vhosts taking traffic: 4) --- top client IPs across all vhosts (last 3 min) --- 122 40.115.94.245 6 203.54.180.238 2 207.46.13.127 2 114.119.143.104 2 114.119.130.40 1 52.167.144.25 1 38.190.4.84 1 114.119.140.234 1 103.29.124.25 --- top request paths across all vhosts (last 3 min) --- 75 GET /api/getItem/LWHG2001KG 47 GET /api/getItem/LWPB2001KG 6 GET /heartbeat.php 3 GET /index.php 1 GET /store/mad-alice-custom-shoe-magnetic-drop-side-display-cases/ 1 GET /store/leopard-print-custom-kids-kicks/ 1 GET /store/kids-swarovski-crystal-airmax90/ 1 GET /store/kids-swarovski-blinged-diamond-kicks/ 1 GET /store/kids-hand-painted-af1-shoes/ 1 GET /product-category/customised-shoes/wedding 1 GET /custom-wedding-sneakers-australia --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/app.evolutionerp.com.au-ssl_log == -- top paths -- 75 GET /api/getItem/LWHG2001KG 47 GET /api/getItem/LWPB2001KG -- last 10 requests -- 40.115.94.245 - - [05/Sep/2026:02:12:05 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:06 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:06 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:07 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:07 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:07 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:08 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:08 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:08 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" 40.115.94.245 - - [05/Sep/2026:02:12:09 +1000] "GET /api/getItem/LWPB2001KG HTTP/1.1" 200 1336 "-" "RestSharp/106.13.0.0" == madalice/madalice.com.au-ssl_log == -- top paths -- 1 GET /store/mad-alice-custom-shoe-magnetic-drop-side-display-cases/ 1 GET /store/leopard-print-custom-kids-kicks/ 1 GET /store/kids-swarovski-crystal-airmax90/ 1 GET /store/kids-swarovski-blinged-diamond-kicks/ 1 GET /store/kids-hand-painted-af1-shoes/ 1 GET /product-category/customised-shoes/wedding 1 GET /custom-wedding-sneakers-australia -- last 10 requests -- 38.190.4.84 - - [05/Sep/2026:02:10:05 +1000] "GET /custom-wedding-sneakers-australia HTTP/1.1" 301 1 "https://madalice.com.au" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrom 103.29.124.25 - - [05/Sep/2026:02:10:06 +1000] "GET /product-category/customised-shoes/wedding HTTP/1.1" 301 1 "https://madalice.com.au" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) C 114.119.130.40 - - [05/Sep/2026:02:10:12 +1000] "GET /store/mad-alice-custom-shoe-magnetic-drop-side-display-cases/?amp%3Brating=2&wcpr_rating=3&wcpr_page=2&wcpr_verified=1 HTTP/1.1" 200 52726 "https://madalice.com.au/st 114.119.143.104 - - [05/Sep/2026:02:10:42 +1000] "GET /store/leopard-print-custom-kids-kicks/?amp%3Bwcpr_image=1&wcpr_rating=3&image=1&wcpr_page=4 HTTP/1.1" 200 54649 "https://madalice.com.au/store/leopard-print-custom-k 114.119.140.234 - - [05/Sep/2026:02:10:56 +1000] "GET /store/kids-swarovski-crystal-airmax90/?rating=5&image=1&wcpr_rating=3 HTTP/1.1" 200 51227 "https://madalice.com.au/store/kids-swarovski-crystal-airmax90/?rating=5&im 114.119.130.40 - - [05/Sep/2026:02:11:19 +1000] "GET /store/kids-swarovski-blinged-diamond-kicks/?verified=1&rating=2&wcpr_image=1 HTTP/1.1" 200 54696 "https://madalice.com.au/store/kids-swarovski-blinged-diamond-kicks/? 114.119.143.104 - - [05/Sep/2026:02:11:41 +1000] "GET /store/kids-hand-painted-af1-shoes/?rating=2&image=1&wcpr_image=1&wcpr_page=2 HTTP/1.1" 404 36657 "https://madalice.com.au/store/kids-hand-painted-af1-shoes/?rating=2 == evolution/portal.evolutionerp.com.au-ssl_log == -- top paths -- 6 GET /heartbeat.php -- last 10 requests -- 203.54.180.238 - - [05/Sep/2026:02:10:07 +1000] "GET /heartbeat.php?sentAt=1788538207&bytes=106,154,237,95&_=1788436623387 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6" 203.54.180.238 - - [05/Sep/2026:02:10:17 +1000] "GET /heartbeat.php?sentAt=1788538217&bytes=106,154,237,105&_=1788436623388 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [05/Sep/2026:02:10:27 +1000] "GET /heartbeat.php?sentAt=1788538227&bytes=106,154,237,115&_=1788436623389 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [05/Sep/2026:02:10:37 +1000] "GET /heartbeat.php?sentAt=1788538237&bytes=106,154,237,125&_=1788436623390 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [05/Sep/2026:02:10:47 +1000] "GET /heartbeat.php?sentAt=1788538247&bytes=106,154,237,135&_=1788436623391 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 203.54.180.238 - - [05/Sep/2026:02:10:57 +1000] "GET /heartbeat.php?sentAt=1788538257&bytes=106,154,237,145&_=1788436623392 HTTP/1.1" 200 10 "https://portal.evolutionerp.com.au/index.php?page=factory_proc_queue&process=6 --- last 30 samples (run-up to this incident) --- 2026-09-05 02:09:47 L1=1.63 run=2 blk=0 avail=7489 (54.4%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=442 ap=14 tcp=25 topuser=evolution 2026-09-05 02:09:57 L1=1.91 run=3 blk=0 avail=7506 (54.5%) swpO=1 conn=3 myRSS=3356 php=6 phpRSS=402 ap=13 tcp=27 topuser=madalice 2026-09-05 02:09:57 L1=1.91 run=2 blk=0 avail=7491 (54.4%) swpO=1 conn=4 myRSS=3356 php=7 phpRSS=442 ap=13 tcp=27 topuser=evolution 2026-09-05 02:10:07 L1=2.62 run=4 blk=0 avail=7326 (53.2%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=592 ap=13 tcp=26 topuser=madalice 2026-09-05 02:10:07 L1=2.62 run=3 blk=0 avail=7379 (53.6%) swpO=0 conn=4 myRSS=3356 php=8 phpRSS=566 ap=13 tcp=28 topuser=madalice 2026-09-05 02:10:17 L1=2.68 run=2 blk=0 avail=7396 (53.7%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=526 ap=13 tcp=27 topuser=madalice 2026-09-05 02:10:17 L1=2.68 run=2 blk=0 avail=7386 (53.6%) swpO=0 conn=4 myRSS=3356 php=8 phpRSS=566 ap=13 tcp=27 topuser=madalice 2026-09-05 02:10:27 L1=2.81 run=4 blk=0 avail=7393 (53.7%) swpO=2 conn=4 myRSS=3356 php=8 phpRSS=566 ap=13 tcp=30 topuser=madalice 2026-09-05 02:10:28 L1=2.81 run=2 blk=1 avail=7384 (53.6%) swpO=1 conn=4 myRSS=3356 php=8 phpRSS=566 ap=13 tcp=30 topuser=madalice 2026-09-05 02:10:37 L1=2.91 run=2 blk=0 avail=7625 (55.3%) swpO=0 conn=4 myRSS=3356 php=6 phpRSS=285 ap=13 tcp=27 topuser=evolution 2026-09-05 02:10:38 L1=2.91 run=2 blk=0 avail=7624 (55.3%) swpO=0 conn=4 myRSS=3356 php=6 phpRSS=285 ap=13 tcp=27 topuser=evolution 2026-09-05 02:10:47 L1=2.78 run=2 blk=0 avail=7514 (54.5%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=419 ap=13 tcp=27 topuser=evolution 2026-09-05 02:10:48 L1=2.78 run=3 blk=0 avail=7514 (54.5%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=419 ap=13 tcp=27 topuser=evolution 2026-09-05 02:10:57 L1=2.74 run=5 blk=0 avail=7481 (54.3%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=454 ap=13 tcp=26 topuser=madalice 2026-09-05 02:10:58 L1=2.74 run=2 blk=0 avail=7491 (54.4%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=440 ap=13 tcp=26 topuser=evolution 2026-09-05 02:11:07 L1=2.47 run=2 blk=0 avail=7490 (54.4%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=440 ap=13 tcp=26 topuser=evolution 2026-09-05 02:11:08 L1=2.47 run=3 blk=0 avail=7488 (54.4%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=440 ap=13 tcp=26 topuser=evolution 2026-09-05 02:11:17 L1=2.25 run=2 blk=0 avail=7485 (54.3%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=440 ap=13 tcp=27 topuser=evolution 2026-09-05 02:11:18 L1=2.25 run=2 blk=0 avail=7484 (54.3%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=440 ap=13 tcp=29 topuser=evolution 2026-09-05 02:11:27 L1=2.05 run=2 blk=0 avail=7505 (54.5%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=419 ap=14 tcp=25 topuser=evolution 2026-09-05 02:11:28 L1=2.05 run=2 blk=0 avail=7502 (54.5%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=419 ap=14 tcp=25 topuser=evolution 2026-09-05 02:11:37 L1=1.89 run=2 blk=0 avail=7474 (54.2%) swpO=2 conn=4 myRSS=3356 php=7 phpRSS=419 ap=19 tcp=35 topuser=evolution 2026-09-05 02:11:38 L1=1.89 run=2 blk=0 avail=7481 (54.3%) swpO=2 conn=4 myRSS=3356 php=6 phpRSS=379 ap=18 tcp=34 topuser=madalice 2026-09-05 02:11:47 L1=1.75 run=2 blk=0 avail=7467 (54.2%) swpO=0 conn=4 myRSS=3356 php=7 phpRSS=442 ap=16 tcp=31 topuser=evolution 2026-09-05 02:11:48 L1=1.75 run=4 blk=0 avail=7485 (54.3%) swpO=0 conn=4 myRSS=3356 php=6 phpRSS=402 ap=16 tcp=30 topuser=madalice 2026-09-05 02:11:57 L1=1.72 run=2 blk=0 avail=6595 (47.9%) swpO=0 conn=4 myRSS=3356 php=14 phpRSS=1402 ap=19 tcp=41 topuser=madalice 2026-09-05 02:11:58 L1=1.72 run=4 blk=0 avail=6594 (47.9%) swpO=0 conn=4 myRSS=3356 php=13 phpRSS=1364 ap=21 tcp=42 topuser=madalice 2026-09-05 02:12:07 L1=1.68 run=2 blk=0 avail=6625 (48.1%) swpO=0 conn=4 myRSS=3356 php=14 phpRSS=1401 ap=16 tcp=32 topuser=madalice 2026-09-05 02:12:08 L1=1.68 run=2 blk=0 avail=6636 (48.2%) swpO=0 conn=4 myRSS=3356 php=13 phpRSS=1362 ap=15 tcp=32 topuser=madalice 2026-09-05 02:12:17 L1=1.74 run=3 blk=0 avail=6446 (46.8%) swpO=0 conn=5 myRSS=3356 php=15 phpRSS=1602 ap=15 tcp=35 topuser=madalice