INCIDENT 2026-09-03 01:38:47 trigger: php_rss=1552MB ================================================================== load: 1.59 1.10 0.91 4/726 1389516 ncpu:10 mem_avail: 7780MB/13777MB (56.5%) swap_used:1826MB swapout:13/s blocked(D):0 running:4 iowait:0.0% steal:0.0% php: 11 procs / 1552MB heaviest user: madalice (9 procs, 1550MB) apache: 81 procs tcp_estab: 84 --- RSS by user (all processes) --- USER PROCS RSS_MB mysql 1 2150 madalice 9 1560 nobody 80 1207 root 249 665 netdata 7 195 infinium 3 35 kurri 3 27 dovenull 6 26 lowrange 2 24 mailman 20 22 cpanellogin 1 19 wp-toolkit 3 18 zabbix 1 13 aaa 1 13 candyfabcom 1 11 mailnull 2 10 named 1 7 dovecot 3 6 nscd 1 4 evolution 8 4 --- php workers by user --- USER PROCS RSS_MB madalice 9 1564 root 1 1 evolution 1 0 --- vm tuning / memory pressure --- vm.swappiness 30 vm.vfs_cache_pressure 100 vm.min_free_kbytes 67584 vm.overcommit_memory 0 vm.overcommit_ratio 50 Cached: 6405 MB Active(file): 3800 MB Inactive(file): 3089 MB Dirty: 1 MB Writeback: 0 MB SReclaimable: 606 MB CommitLimit: 10984 MB Committed_AS: 9269 MB --- backup / dump / scan processes (page-cache evictors) --- PID USER ELAPSED RSS_KB COMMAND 3926885 root 1-23:16:19 5708 /usr/bin/imunify-agent-proxy --- top 60 by RSS --- PID PPID USER RSS %CPU STAT ELAPSED COMMAND 3238812 1 mysql 2202604 9.2 Ssl 11:24:34 /usr/sbin/mariadbd 1389292 3929444 madalice 252320 43.1 S 00:09 php-fpm: pool madalice_com_au 1389472 3929444 madalice 251620 142 S 00:01 php-fpm: pool madalice_com_au 1388694 3929444 madalice 188672 19.6 R 00:52 php-fpm: pool madalice_com_au 508924 1 netdata 180816 2.0 Ssl 5-23:19:08 /usr/sbin/netdata -P /run/netdata/netdata.pid -D 1389296 3929444 madalice 165512 34.1 R 00:08 php-fpm: pool madalice_com_au 1389294 3929444 madalice 162680 34.0 S 00:08 php-fpm: pool madalice_com_au 1389295 3929444 madalice 162216 45.6 R 00:08 php-fpm: pool madalice_com_au 1389291 3929444 madalice 159308 42.2 S 00:09 php-fpm: pool madalice_com_au 1119526 655564 root 145732 0.0 S 01:19:16 spamd child 1389470 3929444 madalice 136672 66.0 S 00:02 php-fpm: pool madalice_com_au 1389468 3929444 madalice 136432 67.0 S 00:02 php-fpm: pool madalice_com_au 2900038 655564 root 66288 0.0 S 12:56:45 spamd child 509236 509098 root 65948 0.0 Sl 5-23:19:05 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager 509266 1 root 44776 0.0 Ss 5-23:19:05 /usr/lib/systemd/systemd-journald 1353308 1 root 41904 0.1 Ss 42-22:57:31 cpsrvd (SSL) - waiting for connections 1469 1 root 41604 0.1 Ss 65-15:29:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1389485 1469 root 39096 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1389487 1469 root 39092 0.0 S 00:00 cpdavd - accepting connections on: 2079, 2080, 2091, 2077, 2078 1052629 1 root 36616 0.0 Ss 01:38:45 lfd - sleeping 1371285 506832 nobody 20896 0.0 S 06:37 /usr/sbin/httpd -k start 1387956 506832 nobody 20636 0.0 S 02:10 /usr/sbin/httpd -k start 1387954 506832 nobody 20136 0.0 S 02:10 /usr/sbin/httpd -k start 1387953 506832 nobody 19604 0.1 S 02:10 /usr/sbin/httpd -k start 1388772 506832 nobody 19556 0.1 S 00:47 /usr/sbin/httpd -k start 1337167 1353308 cpanellogin 19540 0.0 S 17:22 /usr/local/cpanel/base/show_template.stor server_locale en default_login_theme cpanel cpanel_locale docroot /usr/local/cpanel/base 1388825 506832 nobody 19060 0.0 S 00:46 /usr/sbin/httpd -k start 1388821 506832 nobody 19032 0.0 S 00:46 /usr/sbin/httpd -k start 1387603 506832 nobody 18516 0.1 S 02:42 /usr/sbin/httpd -k start 1388824 506832 nobody 18424 0.0 S 00:46 /usr/sbin/httpd -k start 1389068 506832 nobody 18372 0.1 S 00:30 /usr/sbin/httpd -k start 1388771 506832 nobody 18256 0.0 S 00:47 /usr/sbin/httpd -k start 1371345 506832 nobody 18220 0.0 S 06:30 /usr/sbin/httpd -k start 1389094 506832 nobody 18160 0.1 S 00:26 /usr/sbin/httpd -k start 1389310 506832 nobody 17948 0.4 S 00:07 /usr/sbin/httpd -k start 1388822 506832 nobody 17680 0.0 S 00:46 /usr/sbin/httpd -k start 509671 508992 netdata 17584 2.1 Sl 5-23:19:03 /usr/libexec/netdata/plugins.d/apps.plugin 1 1389392 506832 nobody 17428 0.3 S 00:03 /usr/sbin/httpd -k start 1389385 506832 nobody 17384 0.6 S 00:03 /usr/sbin/httpd -k start 1389335 506832 nobody 17368 0.2 S 00:05 /usr/sbin/httpd -k start 1389395 506832 nobody 17356 0.6 S 00:03 /usr/sbin/httpd -k start 1389393 506832 nobody 17352 0.3 S 00:03 /usr/sbin/httpd -k start 1389127 506832 nobody 17244 0.0 S 00:23 /usr/sbin/httpd -k start 1389110 506832 nobody 17160 0.0 S 00:24 /usr/sbin/httpd -k start 1389401 506832 nobody 16976 0.0 S 00:03 /usr/sbin/httpd -k start 1389301 506832 nobody 16884 0.1 S 00:08 /usr/sbin/httpd -k start 1218759 1471 root 16800 0.0 S 50:26 cPhulkd - dbprocessor 1389387 506832 nobody 16544 0.3 S 00:03 /usr/sbin/httpd -k start 1389358 506832 nobody 16468 0.2 S 00:04 /usr/sbin/httpd -k start 1389380 506832 nobody 16404 0.3 S 00:03 /usr/sbin/httpd -k start 1389318 506832 nobody 16204 0.1 S 00:06 /usr/sbin/httpd -k start 1463 1 root 16032 0.0 Ss 65-15:29:00 tailwatchd 1389396 506832 nobody 15756 0.0 S 00:03 /usr/sbin/httpd -k start 1389398 506832 nobody 15712 0.0 S 00:03 /usr/sbin/httpd -k start 1389391 506832 nobody 15708 0.0 S 00:03 /usr/sbin/httpd -k start 1389394 506832 nobody 15704 0.0 S 00:03 /usr/sbin/httpd -k start 1471 1 root 15696 0.0 Ss 65-15:29:00 cPhulkd - processor 1389328 506832 nobody 15260 0.2 S 00:05 /usr/sbin/httpd -k start 1389347 506832 nobody 15260 0.0 S 00:04 /usr/sbin/httpd -k start 1389350 506832 nobody 15216 0.0 S 00:04 /usr/sbin/httpd -k start --- processes in D state (uninterruptible IO) --- --- mariadb processlist (non-sleeping) --- +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | USER | DB | COMMAND | TIME | STATE | q | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1291404 | root | NULL | Query | 0 | Filling schema table | SELECT ID,USER,DB,COMMAND,TIME,STATE,LEFT(INFO,300) q FROM information_schema.PROCESSLIST WHERE COMMAND<>'Sleep' ORDER BY TIME DESC LIMIT 40 | +---------+------+------+---------+------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ --- lock waits --- --- vhost traffic RIGHT NOW (last 3 min, deduped by inode) --- VHOST LOG REQ/3min madalice/madalice.com.au-ssl_log 167 cosgrovecreative/cosgrovecreative.com.au-ssl_log 153 coscom/coscom.com.au-ssl_log 151 evolution/app.evolutionerp.com.au-ssl_log 126 infinium/infinium.net.au-ssl_log 121 lowrange/lowrange.com.au-ssl_log 103 kurri/kurrimotorinn.com.au-ssl_log 75 candyfabcom/candyfab.com.au-ssl_log 50 otterfencing/otterfencing.com.au-ssl_log 44 evolution/portal.evolutionerp.com.au-ssl_log 22 alison/alisonarts.com.au-ssl_log 4 olympic/olympicfencing.com.au-ssl_log 3 corpkicks/corporatekicks.com.au-ssl_log 2 tubesupplies/tubesupplies.com.au-ssl_log 2 whiz/whizengineering.com.au 2 (total vhosts taking traffic: 20) --- top client IPs across all vhosts (last 3 min) --- 200 91.148.244.131 179 81.171.72.135 143 81.171.72.93 126 40.115.94.245 118 20.104.49.130 106 91.148.245.81 40 190.107.177.233 39 81.171.74.60 22 203.54.180.238 19 169.58.197.251 8 118.127.40.55 4 94.130.237.182 2 66.249.68.66 2 60.53.156.148 2 39.36.2.244 --- top request paths across all vhosts (last 3 min) --- 67 GET / 63 GET /api/getItem/LWPG2001KG 25 GET /storage/logs/laravel.log 25 GET /.ssh/id_ed25519 25 GET /config.xml 25 GET /backup.sql 24 POST / 24 GET /wp-config.php 24 GET /database.sql 24 GET /database_backup.sql 24 GET /actuator/heapdump 23 GET /_vti_pvt/service.pwd 23 GET /.ssh/id_rsa 23 GET /.env 23 GET /backup.zip 23 GET /backup.tar.gz 22 HEAD / 22 GET /wp-admin/setup-config.php 22 GET /.svn/wc.db 22 GET /heartbeat.php --- busiest 3 vhosts: top paths + last 10 requests --- == madalice/madalice.com.au-ssl_log == -- top paths -- 8 POST / 8 GET / 7 GET /storage/logs/laravel.log 6 HEAD / 6 GET /config.xml 5 GET /wp-config.php 5 GET /wp-admin/setup-config.php 5 GET /_vti_pvt/service.pwd 5 GET /.ssh/id_rsa 5 GET /.ssh/id_ed25519 5 GET /.git/HEAD 5 GET /.env -- last 10 requests -- 91.148.244.131 - - [03/Sep/2026:01:38:44 +1000] "GET /storage/logs/laravel.log HTTP/1.1" 404 215105 "-" "Go-http-client/1.1" 81.171.72.93 - - [03/Sep/2026:01:38:46 +1000] "GET /backup.sql HTTP/1.1" 404 3421 "-" "Go-http-client/1.1" 81.171.74.60 - - [03/Sep/2026:01:38:46 +1000] "GET /backup.sql HTTP/1.1" 401 53 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:38:44 +1000] "GET /config.xml HTTP/1.1" 404 147440 "-" "Go-http-client/1.1" 91.148.245.81 - - [03/Sep/2026:01:38:46 +1000] "GET /.ssh/id_ed25519 HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 91.148.245.81 - - [03/Sep/2026:01:38:47 +1000] "GET /backup.tar.gz HTTP/1.1" 400 44 "-" "Go-http-client/1.1" 81.171.72.93 - - [03/Sep/2026:01:38:47 +1000] "GET /.ssh/id_ed25519 HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 81.171.72.93 - - [03/Sep/2026:01:38:47 +1000] "GET /backup.tar.gz HTTP/1.1" 404 3166 "-" "Go-http-client/1.1" 81.171.74.60 - - [03/Sep/2026:01:38:47 +1000] "GET /database_backup.sql HTTP/1.1" 401 53 "-" "Go-http-client/1.1" 81.171.74.60 - - [03/Sep/2026:01:38:47 +1000] "GET /database.sql HTTP/1.1" 401 53 "-" "Go-http-client/1.1" == cosgrovecreative/cosgrovecreative.com.au-ssl_log == -- top paths -- 5 POST / 5 GET /wp-config.php 5 GET /wp-admin/setup-config.php 5 GET /_vti_pvt/service.pwd 5 GET /user_secrets.yml 5 GET /.svn/wc.db 5 GET /storage/logs/laravel.log 5 GET /.ssh/id_rsa 5 GET /.ssh/id_ed25519 5 GET /server.key 5 GET /phpinfo.php 5 GET /.npmrc -- last 10 requests -- 81.171.72.135 - - [03/Sep/2026:01:36:39 +1000] "GET /phpinfo.php HTTP/1.1" 403 12 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:39 +1000] "GET /dump.sql HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:39 +1000] "GET /backup.tar.gz HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:41 +1000] "GET /.env.production HTTP/1.1" 403 12 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:41 +1000] "GET /config.php HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:41 +1000] "GET /user_secrets.yml HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:41 +1000] "GET /.npmrc HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:41 +1000] "GET /server.key HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:44 +1000] "GET /.bash_history HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 81.171.72.135 - - [03/Sep/2026:01:36:44 +1000] "GET /storage/logs/laravel.log HTTP/1.1" 400 52 "-" "Go-http-client/1.1" == coscom/coscom.com.au-ssl_log == -- top paths -- 5 POST / 5 GET /wp-config.php 5 GET /wp-admin/setup-config.php 5 GET /_vti_pvt/service.pwd 5 GET /user_secrets.yml 5 GET /.svn/wc.db 5 GET /storage/logs/laravel.log 5 GET /.ssh/id_rsa 5 GET /.ssh/id_ed25519 5 GET /server.key 5 GET /phpinfo.php 5 GET /.npmrc -- last 10 requests -- 91.148.244.131 - - [03/Sep/2026:01:36:13 +1000] "GET /user_secrets.yml HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 81.171.72.93 - - [03/Sep/2026:01:36:13 +1000] "GET /backup.sql HTTP/1.1" 404 - "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:14 +1000] "GET /server.key HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:14 +1000] "GET /.env.production HTTP/1.1" 403 12 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:14 +1000] "GET /.npmrc HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:15 +1000] "GET /.bash_history HTTP/1.1" 403 13 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:15 +1000] "GET /config.xml HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:15 +1000] "GET /dump.sql HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:15 +1000] "GET /backup.sql HTTP/1.1" 400 52 "-" "Go-http-client/1.1" 91.148.244.131 - - [03/Sep/2026:01:36:15 +1000] "GET /config.php HTTP/1.1" 400 52 "-" "Go-http-client/1.1" --- last 30 samples (run-up to this incident) --- 2026-09-03 01:36:24 L1=1.09 run=2 blk=0 avail=9154 (66.4%) swpO=0 conn=3 myRSS=2150 php=7 phpRSS=529 ap=18 tcp=34 topuser=madalice 2026-09-03 01:36:27 L1=1.09 run=1 blk=0 avail=9329 (67.7%) swpO=0 conn=1 myRSS=2150 php=5 phpRSS=291 ap=18 tcp=32 topuser=madalice 2026-09-03 01:36:34 L1=0.99 run=1 blk=0 avail=9325 (67.7%) swpO=0 conn=3 myRSS=2150 php=5 phpRSS=291 ap=17 tcp=55 topuser=madalice 2026-09-03 01:36:37 L1=0.99 run=4 blk=0 avail=9274 (67.3%) swpO=0 conn=2 myRSS=2150 php=5 phpRSS=314 ap=24 tcp=41 topuser=madalice 2026-09-03 01:36:44 L1=1.07 run=2 blk=0 avail=8948 (64.9%) swpO=0 conn=3 myRSS=2150 php=7 phpRSS=692 ap=27 tcp=34 topuser=madalice 2026-09-03 01:36:47 L1=1.07 run=3 blk=0 avail=9061 (65.8%) swpO=0 conn=2 myRSS=2150 php=6 phpRSS=560 ap=24 tcp=34 topuser=madalice 2026-09-03 01:36:54 L1=1.06 run=3 blk=0 avail=9186 (66.7%) swpO=0 conn=3 myRSS=2150 php=7 phpRSS=535 ap=18 tcp=32 topuser=madalice 2026-09-03 01:36:57 L1=1.06 run=1 blk=0 avail=9188 (66.7%) swpO=0 conn=2 myRSS=2150 php=5 phpRSS=466 ap=18 tcp=34 topuser=madalice 2026-09-03 01:37:04 L1=0.89 run=1 blk=0 avail=9183 (66.7%) swpO=0 conn=3 myRSS=2150 php=5 phpRSS=467 ap=18 tcp=33 topuser=madalice 2026-09-03 01:37:07 L1=0.89 run=2 blk=0 avail=9181 (66.6%) swpO=0 conn=2 myRSS=2150 php=5 phpRSS=467 ap=18 tcp=31 topuser=madalice 2026-09-03 01:37:14 L1=1.00 run=2 blk=0 avail=9059 (65.8%) swpO=0 conn=3 myRSS=2150 php=6 phpRSS=617 ap=17 tcp=32 topuser=madalice 2026-09-03 01:37:17 L1=1.00 run=3 blk=0 avail=9152 (66.4%) swpO=0 conn=2 myRSS=2150 php=6 phpRSS=524 ap=17 tcp=33 topuser=madalice 2026-09-03 01:37:24 L1=1.08 run=3 blk=0 avail=9315 (67.6%) swpO=0 conn=3 myRSS=2150 php=4 phpRSS=313 ap=17 tcp=34 topuser=madalice 2026-09-03 01:37:27 L1=1.08 run=2 blk=0 avail=9363 (68.0%) swpO=0 conn=2 myRSS=2150 php=4 phpRSS=265 ap=17 tcp=33 topuser=madalice 2026-09-03 01:37:34 L1=1.14 run=2 blk=0 avail=9404 (68.3%) swpO=0 conn=3 myRSS=2150 php=4 phpRSS=227 ap=17 tcp=33 topuser=madalice 2026-09-03 01:37:37 L1=1.13 run=3 blk=0 avail=9339 (67.8%) swpO=0 conn=2 myRSS=2150 php=5 phpRSS=317 ap=17 tcp=33 topuser=madalice 2026-09-03 01:37:45 L1=1.12 run=3 blk=0 avail=9347 (67.8%) swpO=0 conn=3 myRSS=2150 php=4 phpRSS=278 ap=17 tcp=33 topuser=madalice 2026-09-03 01:37:47 L1=1.19 run=3 blk=0 avail=9567 (69.4%) swpO=0 conn=2 myRSS=2150 php=2 phpRSS=2 ap=17 tcp=33 topuser=root 2026-09-03 01:37:55 L1=1.17 run=3 blk=0 avail=9197 (66.8%) swpO=0 conn=3 myRSS=2150 php=5 phpRSS=467 ap=17 tcp=37 topuser=customc 2026-09-03 01:37:57 L1=1.24 run=4 blk=0 avail=9460 (68.7%) swpO=0 conn=2 myRSS=2150 php=3 phpRSS=133 ap=17 tcp=36 topuser=madalice 2026-09-03 01:38:05 L1=1.30 run=5 blk=0 avail=9212 (66.9%) swpO=0 conn=3 myRSS=2150 php=4 phpRSS=237 ap=30 tcp=46 topuser=madalice 2026-09-03 01:38:07 L1=1.28 run=3 blk=0 avail=9257 (67.2%) swpO=0 conn=2 myRSS=2150 php=4 phpRSS=288 ap=30 tcp=35 topuser=otterfencing 2026-09-03 01:38:15 L1=1.33 run=4 blk=0 avail=9400 (68.2%) swpO=0 conn=3 myRSS=2150 php=3 phpRSS=156 ap=23 tcp=38 topuser=madalice 2026-09-03 01:38:17 L1=1.31 run=2 blk=0 avail=9353 (67.9%) swpO=0 conn=2 myRSS=2150 php=4 phpRSS=234 ap=23 tcp=41 topuser=madalice 2026-09-03 01:38:25 L1=1.28 run=2 blk=0 avail=9272 (67.3%) swpO=0 conn=2 myRSS=2150 php=4 phpRSS=246 ap=32 tcp=52 topuser=madalice 2026-09-03 01:38:27 L1=1.26 run=2 blk=0 avail=9233 (67.0%) swpO=0 conn=2 myRSS=2150 php=4 phpRSS=300 ap=32 tcp=33 topuser=madalice 2026-09-03 01:38:35 L1=1.24 run=1 blk=0 avail=9396 (68.2%) swpO=0 conn=2 myRSS=2150 php=3 phpRSS=157 ap=24 tcp=35 topuser=madalice 2026-09-03 01:38:37 L1=1.14 run=1 blk=0 avail=9411 (68.3%) swpO=0 conn=1 myRSS=2150 php=3 phpRSS=157 ap=22 tcp=34 topuser=madalice 2026-09-03 01:38:45 L1=1.21 run=9 blk=0 avail=8270 (60.0%) swpO=1 conn=2 myRSS=2150 php=10 phpRSS=1128 ap=82 tcp=70 topuser=madalice 2026-09-03 01:38:47 L1=1.59 run=4 blk=0 avail=7780 (56.5%) swpO=13 conn=4 myRSS=2150 php=11 phpRSS=1552 ap=81 tcp=84 topuser=madalice