INCIDENT 2026-09-02 14:29:26 trigger: php_rss=1500MB ================================================================== load: 2.70 3.72 7.46 2/624 3295022 ncpu:10 mem_avail: 8841MB/13777MB (64.2%) swap_used:1396MB swapout:0/s blocked(D):0 running:3 iowait:0.1% steal:0.0% php: 12 procs / 1500MB heaviest user: madalice (9 procs, 1457MB) apache: 43 procs tcp_estab: 53 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 1682 madalice 9 1462 nobody 42 686 root 246 441 netdata 7 177 cpanellogin 1 47 evolution 9 43 mailman 20 36 dovenull 6 19 zabbix 1 17 mailnull 3 16 wp-toolkit 3 14 lowrange 1 11 dovecot 3 9 kurri 1 4 polkitd 1 3 dbus 1 3 cpanelconnecttrack 1 3 nscd 1 2 named 1 2 --- php workers by user --- USER PROCS RSS_MB madalice 9 1464 evolution 2 40 root 1 2 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 5029 MB Active(file): 3934 MB Inactive(file): 1422 MB Dirty: 3 MB Writeback: 0 MB SReclaimable: 267 MB CommitLimit: 10984 MB Committed_AS: 8544 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 3926885 root 1-12:06:57 4596 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 1722384 36.4 Ssl 15:12 /usr/sbin/mariadbd 3274009 3929444 madalice 224988 16.0 R 06:37 php-fpm: pool madalice_com_au 3272326 3929444 madalice 164756 16.6 S 08:46 php-fpm: pool madalice_com_au 508924 1 netdata 161804 2.0 Ssl 5-12:09:46 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 3292380 3929444 madalice 161428 8.3 S 02:08 php-fpm: pool madalice_com_au 3273285 3929444 madalice 160660 17.4 S 07:41 php-fpm: pool madalice_com_au 3292363 3929444 madalice 160252 9.2 S 02:09 php-fpm: pool madalice_com_au 3292379 3929444 madalice 159444 8.2 S 02:08 php-fpm: pool madalice_com_au 3292368 3929444 madalice 158936 9.3 S 02:09 php-fpm: pool madalice_com_au 3292362 3929444 madalice 158668 9.3 S 02:09 php-fpm: pool madalice_com_au 3292374 3929444 madalice 157396 8.3 S 02:08 php-fpm: pool madalice_com_au 706071 655564 root 149620 0.0 S 11:51:27 spamd child 3239542 1353308 cpanellogin 48744 0.0 S 14:58 /usr/local/cpanel/base/show_template.stor default_login_theme cpanel server_locale en docroot /usr/local/cpanel/base cpanel_locale 2900038 655564 root 43520 0.0 S 01:47:23 spamd child 1353308 1 root 42032 0.1 Ss 42-11:48:09 cpsrvd (SSL) - waiting for connections 3294962 3294262 evolution 41784 0.0 S 00:00 /opt/cpanel/ea-php74/root/usr/bin/php-cgi 3295045 3292428 olympic 41224 0.0 R 00:00 /opt/cpanel/ea-php82/root/usr/bin/php-cgi 509236 509098 root 27028 0.0 Sl 5-12:09:43 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 3275524 506832 nobody 22824 0.0 S 04:59 /usr/sbin/httpd -k start 3294991 1469 root 21008 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 3274753 506832 nobody 20980 0.0 S 05:46 /usr/sbin/httpd -k start 3294262 506832 nobody 20848 0.4 S 00:35 /usr/sbin/httpd -k start 3291490 506832 nobody 20620 0.0 S 03:02 /usr/sbin/httpd -k start 3292399 506832 nobody 20316 0.0 S 02:07 /usr/sbin/httpd -k start 3292367 506832 nobody 20136 0.0 S 02:09 /usr/sbin/httpd -k start 3284623 506832 nobody 19964 0.0 S 04:17 /usr/sbin/httpd -k start 3293618 506832 nobody 19612 0.0 S 01:13 /usr/sbin/httpd -k start 3291566 506832 nobody 19524 0.0 S 03:01 /usr/sbin/httpd -k start 3286055 506832 nobody 18852 0.2 S 04:16 /usr/sbin/httpd -k start 3293188 506832 nobody 18244 0.0 S 01:24 /usr/sbin/httpd -k start 3294264 506832 nobody 18216 0.0 S 00:35 /usr/sbin/httpd -k start 3293577 506832 nobody 18140 0.1 S 01:15 /usr/sbin/httpd -k start 3294069 506832 nobody 18016 0.0 S 00:47 /usr/sbin/httpd -k start 509671 508992 netdata 18000 2.1 Sl 5-12:09:41 /usr/libexec/netdata/plugins.d/apps.plugin 1 3293619 506832 nobody 17980 0.0 S 01:13 /usr/sbin/httpd -k start 3294033 506832 nobody 17936 0.2 S 00:50 /usr/sbin/httpd -k start 3294265 506832 nobody 17908 0.0 S 00:35 /usr/sbin/httpd -k start 3292428 506832 nobody 17860 0.0 S 02:06 /usr/sbin/httpd -k start 3292356 506832 nobody 17776 0.1 S 02:10 /usr/sbin/httpd -k start 483898 1 zabbix 17732 0.2 Ssl 5-12:11:00 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf 3294289 506832 nobody 17676 0.0 S 00:34 /usr/sbin/httpd -k start 3293563 506832 nobody 17660 0.0 S 01:16 /usr/sbin/httpd -k start 3294317 506832 nobody 17472 0.0 S 00:32 /usr/sbin/httpd -k start 3293576 506832 nobody 17420 0.1 S 01:15 /usr/sbin/httpd -k start 3294314 506832 nobody 17204 0.0 S 00:33 /usr/sbin/httpd -k start 3283276 506832 nobody 16836 0.0 S 04:18 /usr/sbin/httpd -k start 3293623 506832 nobody 16368 0.0 S 01:13 /usr/sbin/httpd -k start 3294307 506832 nobody 16204 0.0 S 00:33 /usr/sbin/httpd -k start 3294283 506832 nobody 16144 0.0 S 00:34 /usr/sbin/httpd -k start 3294286 506832 nobody 15972 0.0 S 00:34 /usr/sbin/httpd -k start 3294306 506832 nobody 15656 0.0 S 00:33 /usr/sbin/httpd -k start 3294287 506832 nobody 15596 0.0 S 00:34 /usr/sbin/httpd -k start 3294244 506832 nobody 15524 0.0 S 00:36 /usr/sbin/httpd -k start 3290554 506832 nobody 15472 0.0 S 04:05 /usr/sbin/httpd -k start 3294315 506832 nobody 15436 0.1 S 00:33 /usr/sbin/httpd -k start 3294243 506832 nobody 15424 0.0 S 00:36 /usr/sbin/httpd -k start 3294312 506832 nobody 15384 0.0 S 00:33 /usr/sbin/httpd -k start 3294316 506832 nobody 15300 0.2 S 00:33 /usr/sbin/httpd -k start 3294284 506832 nobody 15212 0.1 S 00:34 /usr/sbin/httpd -k start 3294319 506832 nobody 13204 0.0 S 00:32 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +--------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +--------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 103240 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +--------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min evolution/my.evolutionerp.com.au-ssl_log 658 bigdeals/bigdeals.com.au-ssl_log 149 alison/testing.alisonarts.com.au-ssl_log 138 alison/testing.alisonarts.com.au 134 evolution/portal.evolutionerp.com.au-ssl_log 127 evolution/app.evolutionerp.com.au-ssl_log 123 madalice/madalice.com.au-ssl_log 82 kurri/kurrimotorinn.com.au-ssl_log 64 madalice/madalice.com.au 32 infinium/infinium.net.au-ssl_log 6 (total vhosts taking traffic: 10) --- top client IPs across all vhosts (last 3 min) --- 405 4.205.62.107 393 203.54.180.238 252 103.104.165.99 123 40.115.94.245 84 20.48.251.3 64 20.48.179.39 42 101.182.108.127 20 150.228.151.47 18 2001:8004:14b3:854f:95d1:bdfe:1050:32b5 17 101.166.96.210 16 195.178.110.137 16 103.104.164.114 13 1.144.107.82 7 182.239.240.254 6 180.181.236.49 --- top request paths across all vhosts (last 3 min) --- 209 POST /functions.php 129 GET /app/ajax.php 82 GET /index.php 68 POST /plugins/csvexporter/plugin.php 62 GET /api/getItem/LWFBW2001KG 61 GET /api/getItem/LWHG2001KG 56 GET /factory_session_checker.php 34 POST /app/ajax/selectFilterSingle.php 28 GET /factory_session_processed_qty.php 28 GET /factory_session_joiner.php 24 POST /clienteditsave.php 19 POST /inveditsave.php 13 POST /wp-admin/admin-ajax.php 13 GET /heartbeat.php 12 POST /app/ajax.php 11 POST /quoteditsave.php 11 GET //app/ajax.php 10 POST /jobeditsave.php 8 POST //functions.php 6 GET /ms-edit.php --- busiest 3 vhosts: top paths + last 10 requests --- == evolution/my.evolutionerp.com.au-ssl_log == -- top paths -- 209 POST /functions.php 129 GET /app/ajax.php 80 GET /index.php 68 POST /plugins/csvexporter/plugin.php 34 POST /app/ajax/selectFilterSingle.php 24 POST /clienteditsave.php 19 POST /inveditsave.php 12 POST /app/ajax.php 11 POST /quoteditsave.php 11 GET //app/ajax.php 10 POST /jobeditsave.php 8 POST //functions.php -- last 10 requests -- 1.144.107.82 - - [02/Sep/2026:14:29:24 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 51 "https://my.evolutionerp.com.au/index.php?page=quotedit"eid=2315" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0 1.144.107.82 - - [02/Sep/2026:14:29:23 +1000] "POST /inv_expired_pricing.php?call=getInv HTTP/1.1" 200 6881 "https://my.evolutionerp.com.au/index.php?page=inv_expired_pricing&catid=66" "Mozilla/5.0 (Windows NT 10.0; Win6 101.182.108.127 - - [02/Sep/2026:14:29:24 +1000] "GET /index.php?page=opportunityedit&opportunityid=8546 HTTP/1.1" 200 116721 "https://my.evolutionerp.com.au/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWeb 101.182.108.127 - - [02/Sep/2026:14:29:24 +1000] "GET /app/ajax.php?cn=tsSimpleViewCB&furi=true&uri=%2F&call=getUserSetting&caller=ajaxSubmitLocal&seq=15520623866 HTTP/1.1" 200 90 "https://my.evolutionerp.com.au/index.ph 101.182.108.127 - - [02/Sep/2026:14:29:24 +1000] "GET /app/ajax.php?cn=tsDisabledViewCB&furi=false&uri=&call=getUserSetting&caller=ajaxSubmitLocal&seq=15288443457 HTTP/1.1" 200 90 "https://my.evolutionerp.com.au/index.ph 101.182.108.127 - - [02/Sep/2026:14:29:24 +1000] "POST /plugins/csvexporter/plugin.php HTTP/1.1" 200 20 "https://my.evolutionerp.com.au/index.php?page=opportunityedit&opportunityid=8546" "Mozilla/5.0 (Windows NT 10.0; Wi 101.182.108.127 - - [02/Sep/2026:14:29:24 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 235 "https://my.evolutionerp.com.au/index.php?page=quotedit"eid=62603" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) App 1.144.107.82 - - [02/Sep/2026:14:29:25 +1000] "POST /functions.php?call=heartbeat HTTP/1.1" 200 232 "https://my.evolutionerp.com.au/index.php?page=inv_expired_pricing&catid=66" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; 101.182.108.127 - - [02/Sep/2026:14:29:25 +1000] "GET /app/ajax.php?cn=attachmentListSize&furi=true&uri=job&call=getUserSetting&caller=ajaxSubmitLocal&seq=12829700643 HTTP/1.1" 200 95 "https://my.evolutionerp.com.au/inde 101.182.108.127 - - [02/Sep/2026:14:29:25 +1000] "GET /app/ajax.php?cn=attachmentThumbSize&furi=true&uri=job&call=getUserSetting&caller=ajaxSubmitLocal&seq=14395898025 HTTP/1.1" 200 96 "https://my.evolutionerp.com.au/ind == bigdeals/bigdeals.com.au-ssl_log == -- top paths -- 2 GET /ms-edit.php 2 GET /koiy.php 2 GET /doc.php 1 GET /zz.php 1 GET /zaza.php 1 GET /xza.php 1 GET /xve22.php 1 GET /xqq.php 1 GET /xp.php 1 GET /xmy.php 1 GET /xin1.php 1 GET /xa.php -- last 10 requests -- 4.205.62.107 - - [02/Sep/2026:14:29:10 +1000] "GET /queue.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:11 +1000] "GET /admin-ajax.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:11 +1000] "GET /btx25.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:12 +1000] "GET /goods.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:12 +1000] "GET /umgebung.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:13 +1000] "GET /124.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:13 +1000] "GET /file59.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:14 +1000] "GET /thui.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:14 +1000] "GET /server-info.php HTTP/1.1" 401 36 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:15 +1000] "GET /ws79.php HTTP/1.1" 401 36 "-" "-" == alison/testing.alisonarts.com.au-ssl_log == -- top paths -- 2 GET /ms-edit.php 2 GET /koiy.php 2 GET /doc.php 1 GET /zz.php 1 GET /zaza.php 1 GET /xza.php 1 GET /xve22.php 1 GET /xqq.php 1 GET /xp.php 1 GET /xmy.php 1 GET /xin1.php 1 GET /xa.php -- last 10 requests -- 4.205.62.107 - - [02/Sep/2026:14:29:12 +1000] "GET /old_phpinfo.php HTTP/1.1" 403 12 "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:12 +1000] "GET /adminfus.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:12 +1000] "GET /session.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:13 +1000] "GET /otuz1.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:13 +1000] "GET /time.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:14 +1000] "GET /adminer-4.7.6-en.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:14 +1000] "GET /ah25.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:15 +1000] "GET /ccou.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:15 +1000] "GET /xin1.php HTTP/1.1" 404 - "-" "-" 4.205.62.107 - - [02/Sep/2026:14:29:15 +1000] "GET /xmy.php HTTP/1.1" 404 - "-" "-" --- last 30 samples (run-up to this incident) --- 2026-09-02 14:27:04 L1=3.11 run=3 blk=0 avail=9816 (71.3%) swpO=0 conn=2 myRSS=1614 php=5 phpRSS=475 ap=40 tcp=40 topuser=madalice 2026-09-02 14:27:04 L1=3.11 run=4 blk=0 avail=9821 (71.3%) swpO=0 conn=1 myRSS=1614 php=5 phpRSS=475 ap=40 tcp=40 topuser=madalice 2026-09-02 14:27:14 L1=3.34 run=3 blk=1 avail=9642 (70.0%) swpO=0 conn=2 myRSS=1633 php=10 phpRSS=800 ap=35 tcp=49 topuser=madalice 2026-09-02 14:27:14 L1=3.34 run=5 blk=0 avail=9641 (70.0%) swpO=0 conn=1 myRSS=1634 php=10 phpRSS=800 ap=35 tcp=49 topuser=madalice 2026-09-02 14:27:25 L1=3.19 run=2 blk=0 avail=8952 (65.0%) swpO=0 conn=2 myRSS=1643 php=11 phpRSS=1270 ap=63 tcp=48 topuser=madalice 2026-09-02 14:27:25 L1=3.19 run=2 blk=0 avail=8951 (65.0%) swpO=0 conn=1 myRSS=1643 php=11 phpRSS=1270 ap=63 tcp=48 topuser=madalice 2026-09-02 14:27:35 L1=2.86 run=2 blk=0 avail=8834 (64.1%) swpO=0 conn=2 myRSS=1645 php=12 phpRSS=1490 ap=53 tcp=44 topuser=madalice 2026-09-02 14:27:35 L1=2.86 run=3 blk=0 avail=8833 (64.1%) swpO=0 conn=1 myRSS=1645 php=12 phpRSS=1490 ap=53 tcp=44 topuser=madalice 2026-09-02 14:27:45 L1=2.79 run=2 blk=0 avail=8911 (64.7%) swpO=0 conn=2 myRSS=1648 php=11 phpRSS=1407 ap=46 tcp=53 topuser=madalice 2026-09-02 14:27:45 L1=2.79 run=3 blk=0 avail=8922 (64.8%) swpO=0 conn=1 myRSS=1648 php=11 phpRSS=1407 ap=46 tcp=53 topuser=madalice 2026-09-02 14:27:55 L1=2.52 run=4 blk=0 avail=8859 (64.3%) swpO=0 conn=2 myRSS=1649 php=11 phpRSS=1408 ap=44 tcp=55 topuser=madalice 2026-09-02 14:27:55 L1=2.52 run=5 blk=0 avail=8863 (64.3%) swpO=0 conn=2 myRSS=1649 php=11 phpRSS=1408 ap=44 tcp=55 topuser=madalice 2026-09-02 14:28:05 L1=2.66 run=3 blk=0 avail=8730 (63.4%) swpO=0 conn=2 myRSS=1651 php=16 phpRSS=1673 ap=41 tcp=53 topuser=madalice 2026-09-02 14:28:05 L1=2.66 run=4 blk=0 avail=8730 (63.4%) swpO=0 conn=2 myRSS=1651 php=16 phpRSS=1673 ap=41 tcp=53 topuser=madalice 2026-09-02 14:28:15 L1=2.25 run=3 blk=0 avail=8686 (63.0%) swpO=0 conn=5 myRSS=1654 php=17 phpRSS=1656 ap=52 tcp=62 topuser=madalice 2026-09-02 14:28:15 L1=2.25 run=2 blk=0 avail=8703 (63.2%) swpO=0 conn=3 myRSS=1654 php=12 phpRSS=1508 ap=52 tcp=62 topuser=madalice 2026-09-02 14:28:25 L1=1.98 run=3 blk=0 avail=8828 (64.1%) swpO=0 conn=5 myRSS=1660 php=12 phpRSS=1456 ap=43 tcp=52 topuser=madalice 2026-09-02 14:28:25 L1=1.98 run=3 blk=0 avail=8777 (63.7%) swpO=0 conn=3 myRSS=1660 php=13 phpRSS=1538 ap=43 tcp=53 topuser=madalice 2026-09-02 14:28:35 L1=1.67 run=3 blk=0 avail=8863 (64.3%) swpO=0 conn=5 myRSS=1661 php=12 phpRSS=1495 ap=35 tcp=51 topuser=madalice 2026-09-02 14:28:35 L1=1.67 run=3 blk=0 avail=8769 (63.7%) swpO=0 conn=3 myRSS=1661 php=13 phpRSS=1613 ap=35 tcp=52 topuser=madalice 2026-09-02 14:28:45 L1=1.80 run=2 blk=0 avail=8886 (64.5%) swpO=0 conn=5 myRSS=1661 php=13 phpRSS=1485 ap=38 tcp=43 topuser=madalice 2026-09-02 14:28:45 L1=1.80 run=1 blk=0 avail=8892 (64.5%) swpO=0 conn=3 myRSS=1661 php=12 phpRSS=1455 ap=38 tcp=43 topuser=madalice 2026-09-02 14:28:55 L1=2.08 run=1 blk=0 avail=8738 (63.4%) swpO=0 conn=2 myRSS=1662 php=12 phpRSS=1453 ap=65 tcp=65 topuser=madalice 2026-09-02 14:28:55 L1=2.08 run=2 blk=0 avail=8734 (63.4%) swpO=0 conn=2 myRSS=1662 php=13 phpRSS=1490 ap=65 tcp=65 topuser=madalice 2026-09-02 14:29:05 L1=2.29 run=3 blk=0 avail=8804 (63.9%) swpO=0 conn=2 myRSS=1665 php=12 phpRSS=1457 ap=55 tcp=62 topuser=madalice 2026-09-02 14:29:06 L1=2.29 run=2 blk=0 avail=8794 (63.8%) swpO=0 conn=2 myRSS=1665 php=12 phpRSS=1464 ap=55 tcp=62 topuser=madalice 2026-09-02 14:29:15 L1=2.65 run=1 blk=0 avail=8831 (64.1%) swpO=0 conn=2 myRSS=1681 php=12 phpRSS=1456 ap=48 tcp=62 topuser=madalice 2026-09-02 14:29:16 L1=2.65 run=2 blk=0 avail=8831 (64.1%) swpO=0 conn=2 myRSS=1681 php=12 phpRSS=1456 ap=48 tcp=62 topuser=madalice 2026-09-02 14:29:25 L1=2.70 run=3 blk=0 avail=8884 (64.5%) swpO=0 conn=2 myRSS=1682 php=12 phpRSS=1456 ap=43 tcp=52 topuser=madalice 2026-09-02 14:29:26 L1=2.70 run=3 blk=0 avail=8841 (64.2%) swpO=0 conn=3 myRSS=1682 php=12 phpRSS=1500 ap=43 tcp=53 topuser=madalice