[03:00:51] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 27 seconds old and 93 bytes [03:08:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb [03:09:01] PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:18] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [03:09:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb [03:09:48] PROBLEM - Host cp51 is DOWN: PING CRITICAL - Packet loss = 100% [03:19:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:49:22] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:52:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.88, 19.16, 16.95 [04:54:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.53, 17.28, 16.55 [04:58:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 21.92, 18.65 [05:10:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.09, 23.94, 21.83 [05:16:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.24, 18.97, 20.18 [05:28:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.68, 20.76, 20.02 [05:30:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 22.44, 20.71 [05:39:16] [02dns] 07BlankEclair synchronize pull request 03#520: T12156: Add requested subdomains for rainverse.wiki - 13https://github.com/miraheze/dns/pull/520 [05:40:19] [02dns] 07BlankEclair edited pull request 03#520: T12156: Add requested subdomains for rainverse.wiki - 13https://github.com/miraheze/dns/pull/520 [05:40:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.25, 23.54, 22.57 [05:43:59] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:44:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 24.23, 23.08 [05:46:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.62, 23.56, 22.95 [05:54:15] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [05:54:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.99, 22.12, 22.12 [06:05:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.64, 20.48, 17.73 [06:07:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 21.47, 18.39 [06:09:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.53, 21.74, 18.88 [06:09:44] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 21.99, 17.35 [06:09:58] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:11:43] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.31, 20.20, 17.28 [06:11:45] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 23.08, 18.65 [06:15:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.75, 20.08, 19.10 [06:16:44] PROBLEM - wiki.mhe.my.id - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [06:17:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.94, 20.09, 17.70 [06:19:22] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.78, 23.99, 21.29 [06:19:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.02, 17.73, 17.13 [06:23:10] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.66, 24.97, 22.10 [06:24:00] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:25:04] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.33, 23.05, 21.79 [06:25:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:27:36] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.59, 19.92, 17.98 [06:31:34] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.75, 18.93, 18.16 [06:36:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.91, 18.82, 20.15 [06:50:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:08:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.28, 19.92, 22.69 [07:14:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.81, 16.78, 20.28 [07:48:18] RECOVERY - Host cp51 is UP: PING OK - Packet loss = 0%, RTA = 162.68 ms [07:48:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:48:52] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [07:49:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:50:32] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3777 bytes in 1.067 second response time [08:33:22] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 20.57, 17.46 [08:37:15] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.96, 23.53, 19.47 [08:41:08] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.38, 20.07, 18.94 [08:45:04] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.11, 23.28, 20.49 [08:49:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.85, 16.66, 13.06 [08:52:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 19.99, 16.12 [08:53:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.92, 16.61, 14.00 [08:54:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.07, 21.35, 17.14 [08:58:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.58, 17.29, 16.52 [09:05:27] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.38, 18.95, 15.68 [09:06:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.06, 21.18, 18.44 [09:07:26] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.47, 19.12, 16.21 [09:08:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.15, 19.08, 18.05 [09:56:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.99, 20.38, 17.87 [09:58:43] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.10, 19.17, 17.69 [10:06:19] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 21.85, 19.34 [10:08:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.87, 21.06, 19.41 [10:10:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.15, 19.53, 19.05 [10:17:07] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [10:17:39] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [10:18:15] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [10:27:13] !log cf: restore HSTS [10:27:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.67, 19.93, 16.80 [10:33:48] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.36, 18.01, 16.44 [10:39:07] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [10:39:39] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19090 bytes in 0.062 second response time [10:40:15] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.043 second response time [10:46:26] PROBLEM - wiki.thelastdimension.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thelastdimension.net' expires in 15 day(s) (Mon 10 Jun 2024 10:35:45 AM GMT +0000). [10:46:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7519f7a858d7...e93711a0d16d [10:46:38] [02ssl] 07WikiTideSSLBot 03e93711a - Bot: Update SSL cert for wiki.thelastdimension.net [11:07:07] PROBLEM - drachronicles.voxel.gay - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drachronicles.voxel.gay' expires in 15 day(s) (Mon 10 Jun 2024 10:35:30 AM GMT +0000). [11:09:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.12, 19.89, 16.23 [11:11:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.27, 17.64, 15.88 [11:34:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.82, 21.74, 19.05 [11:36:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.42, 20.28, 18.88 [11:45:17] RECOVERY - wiki.thelastdimension.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thelastdimension.net' will expire on Fri 23 Aug 2024 09:46:31 AM GMT +0000. [12:00:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.44, 18.58, 17.12 [12:02:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.75, 17.95, 17.05 [12:07:06] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.75, 20.09, 16.81 [12:08:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.26, 22.35, 23.98 [12:09:05] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.56, 17.41, 16.21 [12:10:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.97, 24.88, 24.69 [12:15:54] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:16:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.85, 22.87, 23.98 [12:18:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 23.35, 23.97 [12:20:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.98, 21.91, 23.35 [12:34:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.54, 17.60, 19.89 [12:40:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.18, 21.15 [12:44:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.33, 21.82, 21.37 [12:44:43] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [12:54:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.09, 18.20, 20.08 [13:02:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 22.35, 21.13 [13:06:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.39, 22.80, 21.72 [13:12:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 22.76, 21.72 [13:15:54] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:20:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 13.52, 20.87, 21.96 [13:24:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.35, 16.47, 19.83 [13:42:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.46, 24.25, 20.81 [13:44:41] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [13:44:56] PROBLEM - pso2ngs.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pso2ngs.wiki' expires in 15 day(s) (Mon 10 Jun 2024 01:22:59 PM GMT +0000). [13:45:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e93711a0d16d...f53dbe0716ef [13:45:09] [02ssl] 07WikiTideSSLBot 03f53dbe0 - Bot: Update SSL cert for pso2ngs.wiki [13:46:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.51, 23.92, 21.56 [13:48:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.93, 25.87, 22.56 [13:54:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 21.02, 16.94 [13:56:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.74, 21.10, 17.41 [14:04:03] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/pso2ngs.wiki.crt] [14:04:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.09, 19.65, 18.30 [14:08:25] [02CreateWiki] 07pixDeVl synchronize pull request 03#511: T12075: Notify Wiki Creators about requesters with over X(5) request - 13https://github.com/miraheze/CreateWiki/pull/511 [14:08:28] [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#511: T12075: Notify Wiki Creators about requesters with over X(5) request - 13https://github.com/miraheze/CreateWiki/pull/511#issuecomment-2118166172 [14:18:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:21:37] miraheze/CreateWiki - pixDeVl the build passed. [14:24:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.19, 21.31, 18.48 [14:26:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.92, 19.33, 18.08 [14:32:03] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:37:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.45, 20.53, 19.23 [14:40:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.01, 19.72, 19.21 [14:43:47] RECOVERY - pso2ngs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pso2ngs.wiki' will expire on Fri 23 Aug 2024 12:45:00 PM GMT +0000. [14:44:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.66, 20.76, 23.26 [14:48:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:52:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.10, 16.57, 20.06 [14:58:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.65, 23.22, 21.61 [15:17:21] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.43 ms [15:19:23] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 162.10 ms [15:22:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.85, 22.25, 23.54 [15:24:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.03, 21.18, 19.48 [15:26:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 22.94, 20.34 [15:28:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.88, 20.20, 19.65 [15:28:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.34, 23.06, 23.20 [15:42:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.02, 22.52, 23.73 [15:42:47] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.36 ms [15:44:49] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 171.51 ms [15:46:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 6.02, 13.44, 19.71 [16:05:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:09:22] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.11, 19.43, 16.01 [16:09:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.87, 18.93, 12.99 [16:10:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 20.76, 15.55 [16:12:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.31, 19.21, 15.64 [16:13:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.64, 19.45, 13.87 [16:15:33] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.97, 22.78, 16.68 [16:15:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.28, 18.10, 14.01 [16:16:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.14, 24.17, 18.44 [16:17:32] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.81, 24.15, 17.92 [16:19:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.88, 21.88, 15.27 [16:21:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.35, 21.13, 15.79 [16:21:30] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.55, 23.19, 19.07 [16:21:40] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.05, 21.29, 16.62 [16:22:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.99, 20.41, 15.65 [16:23:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.49, 19.92, 15.96 [16:25:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.81, 19.82, 17.33 [16:26:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.94, 23.74, 21.79 [16:26:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 11.31, 22.28, 22.43 [16:26:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 8.09, 16.58, 15.43 [16:27:28] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 8.03, 16.84, 18.01 [16:28:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 6.58, 17.50, 19.72 [16:30:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 4.86, 13.24, 18.73 [17:08:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 21.58, 16.61 [17:09:16] !log [oa@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [17:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:31] !log [oa@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 15s [17:09:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.62, 22.03, 16.93 [17:24:54] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 18.40, 13.55 [17:26:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.39, 19.70, 16.68 [17:26:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.49, 19.12, 14.42 [17:27:01] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 21.43, 17.06 [17:27:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 20.27, 15.23 [17:28:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.96, 18.82, 14.88 [17:29:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.56, 21.98, 16.51 [17:30:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.53, 23.47, 18.82 [17:31:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.37, 18.22, 13.54 [17:33:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.82, 18.31, 14.19 [17:33:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.28, 23.44, 18.25 [17:33:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 22.48, 17.03 [17:35:24] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.25, 20.26, 15.44 [17:35:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.13, 22.34, 17.65 [17:36:54] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 22.25, 17.78 [17:37:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.86, 19.58, 17.21 [17:38:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.61, 19.29, 17.27 [17:39:14] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.76, 18.10, 15.78 [17:41:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.29, 22.30, 20.34 [17:43:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.67, 21.10, 18.50 [17:45:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:45:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.05, 18.54, 17.86 [17:46:53] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.10, 23.57, 23.70 [17:47:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.24, 19.80, 19.97 [17:50:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 9.78, 19.86, 21.83 [17:52:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 6.51, 15.22, 19.88 [17:52:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 6.37, 16.41, 23.47 [17:52:51] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 7.14, 13.82, 19.54 [17:55:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:56:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 5.97, 10.68, 19.50 [18:05:19] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [20:33:20] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 57% [20:37:21] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 4% [20:42:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:44:57] [02ImportDump] 07Universal-Omega pushed 031 commit to 03path-traversal [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/c9405e5a9814 [21:44:58] [02ImportDump] 07Universal-Omega 03c9405e5 - Add extra path traversal check [21:45:01] [02ImportDump] 07Universal-Omega created branch 03path-traversal - 13https://github.com/miraheze/ImportDump [21:45:03] [02ImportDump] 07Universal-Omega opened pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109 [21:45:22] [02ImportDump] 07coderabbitai[bot] commented on pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109#issuecomment-2131500353 [21:45:54] [02ImportDump] 07coderabbitai[bot] edited pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109 [21:47:48] [02ImportDump] 07coderabbitai[bot] edited a comment on pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109#issuecomment-2131500353 [21:47:56] miraheze/ImportDump - Universal-Omega the build has errored. [21:53:57] [02ImportDump] 07Universal-Omega synchronize pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109 [21:53:58] [02ImportDump] 07Universal-Omega pushed 031 commit to 03path-traversal [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/c9405e5a9814...3466c64cc5c4 [21:53:59] [02ImportDump] 07Universal-Omega 033466c64 - Try again [21:54:02] [02ImportDump] 07coderabbitai[bot] edited a comment on pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109#issuecomment-2131500353 [21:54:36] [02ImportDump] 07coderabbitai[bot] edited pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109 [21:56:10] [02ImportDump] 07coderabbitai[bot] edited a comment on pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109#issuecomment-2131500353 [21:57:02] miraheze/ImportDump - Universal-Omega the build has errored. [22:03:24] [02ImportDump] 07Universal-Omega closed pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109 [22:03:25] [02ImportDump] 07Universal-Omega commented on pull request 03#109: Add extra path traversal check - 13https://github.com/miraheze/ImportDump/pull/109#issuecomment-2131530422 [22:03:27] [02ImportDump] 07Universal-Omega deleted branch 03path-traversal [22:03:30] [02ImportDump] 07Universal-Omega deleted branch 03path-traversal - 13https://github.com/miraheze/ImportDump [22:32:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.13, 19.49, 13.26 [22:37:18] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=trollpastawiki (START) [22:37:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 19.41, 12.91 [22:38:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.08, 19.58, 13.12 [22:38:54] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.71, 20.20, 13.20 [22:39:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.27, 20.44, 13.33 [22:39:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.30, 19.88, 12.53 [22:40:32] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.77, 20.14, 13.11 [22:40:47] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.94, 21.75, 14.51 [22:45:49] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 10.12, 6.34, 3.00 [22:47:00] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 4 backends are down. mw161 mw162 mw181 mw182 [22:47:27] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw171 mw181 [22:47:49] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.75, 4.41, 2.68 [22:47:50] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [22:48:58] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [22:49:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.19, 23.81, 19.28 [22:49:27] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [22:49:47] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [22:50:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 13.54, 20.79, 18.02 [22:50:47] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 13.73, 22.17, 19.25 [22:51:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 8.71, 19.25, 18.21 [22:51:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 8.90, 19.63, 19.19 [22:51:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 7.30, 19.45, 18.17 [22:52:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 7.97, 21.21, 20.19 [22:52:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 8.49, 21.79, 22.50 [22:52:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.15, 15.10, 16.26 [22:52:47] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 6.23, 16.69, 17.60 [22:52:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 6.39, 17.79, 18.55 [22:54:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 5.42, 15.77, 18.32 [22:54:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 5.71, 16.25, 20.37 [22:55:17] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [22:55:18] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [22:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:20] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_IRC-Discord-Relay-relaybot2] [23:17:06] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=trollpastawiki (END - exit=0) [23:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:20] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures