[00:00:30] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.51, 23.96, 19.01 [00:00:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.25, 22.86, 17.89 [00:00:44] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.73, 24.43, 18.47 [00:01:03] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw152 mw161 mw181 [00:01:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 21.02, 16.66 [00:01:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 23.80, 18.10 [00:01:35] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw162 mw182 [00:01:36] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [00:02:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.52, 23.19, 18.69 [00:02:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.36, 21.93, 18.28 [00:02:59] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [00:03:08] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.40, 19.33, 16.62 [00:03:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 15.81, 20.90, 18.57 [00:03:30] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [00:03:32] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.37, 21.78, 18.03 [00:03:34] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [00:04:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.52, 22.87, 19.85 [00:05:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.65, 19.89, 18.49 [00:05:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.58, 19.88, 17.75 [00:06:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.76, 24.00, 21.94 [00:06:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.11, 19.82, 18.36 [00:07:23] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:08:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.57, 18.50, 18.79 [00:08:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.19, 19.18, 18.30 [00:12:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 22.34, 21.73 [00:18:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.26, 22.34, 22.08 [00:20:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 23.31, 22.44 [00:22:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.34, 21.52, 21.90 [00:24:48] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.55, 23.65, 22.66 [00:26:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.11, 20.58, 19.14 [00:26:47] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [00:30:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.44, 23.64, 22.89 [00:30:18] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.06, 20.04, 19.29 [00:34:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 23.95, 24.04, 23.21 [00:42:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.76, 22.50, 22.92 [00:42:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:43:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.11, 3.46, 3.96 [00:51:37] [02mw-config] 07hurohukidaikon commented on pull request 03#5643: [WIP, Need help] Update UploadWizard settings for kagagawiki - 13https://github.com/miraheze/mw-config/pull/5643#issuecomment-2305980417 [00:51:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.01, 3.74, 3.81 [00:53:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.23, 3.52, 3.75 [00:57:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.08, 3.79, 3.78 [01:02:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 22.17, 22.20 [01:02:17] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:02:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:03:23] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 21.22, 18.24 [01:04:30] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.65, 23.77, 19.61 [01:04:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.96, 20.78, 18.58 [01:05:22] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.74, 20.08, 18.20 [01:05:35] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 21.27, 18.13 [01:07:17] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [01:07:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:08:44] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.77, 23.41, 19.99 [01:09:17] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.54, 24.85, 20.52 [01:09:19] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw161 mw172 [01:09:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 30.13, 24.32, 19.53 [01:09:40] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw152 mw161 mw181 [01:10:14] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [01:10:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.56, 23.23, 20.37 [01:11:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.34, 21.18, 16.62 [01:11:16] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.09, 23.22, 20.48 [01:11:19] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [01:11:32] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.59, 23.73, 19.97 [01:11:37] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [01:11:42] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:12:14] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [01:13:08] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.68, 18.87, 16.30 [01:13:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 15.85, 21.98, 20.35 [01:14:14] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:14:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.07, 23.71, 22.00 [01:14:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.84, 19.42, 19.47 [01:15:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.34, 19.49, 19.65 [01:15:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.79, 19.69, 19.13 [01:16:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.32, 24.00, 23.99 [01:17:49] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.103 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:18:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.65, 18.13, 20.17 [01:19:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.30, 19.16, 19.67 [01:19:15] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [01:20:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.57, 23.89, 23.82 [01:22:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 23.41, 23.68 [01:24:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.38, 24.53, 24.02 [01:32:29] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:33:23] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:24] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:35:29] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [01:40:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.67, 22.59, 23.91 [01:58:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.97, 23.23, 22.68 [02:00:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.40, 22.28, 22.42 [02:02:27] PROBLEM - wiki.overwood.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.overwood.xyz All nameservers failed to answer the query. [02:04:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.55, 24.09, 22.96 [02:04:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 21.89, 19.14 [02:05:19] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [02:05:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.03, 3.31, 3.95 [02:06:19] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.75, 20.64, 18.98 [02:07:19] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [02:08:14] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.76, 23.20, 20.10 [02:09:12] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.56, 23.23, 19.58 [02:09:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.72, 22.34, 18.77 [02:09:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.33, 23.78, 19.62 [02:09:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.13, 3.85, 4.00 [02:10:14] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [02:10:35] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182 [02:10:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.46, 23.73, 19.48 [02:11:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.59, 3.38, 3.79 [02:11:59] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [02:12:14] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [02:12:35] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [02:13:07] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.53, 22.16, 19.54 [02:13:32] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.66, 23.51, 20.63 [02:13:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 4.22, 4.05 [02:13:58] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [02:14:01] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.22, 22.57, 21.15 [02:14:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.83, 20.17, 19.14 [02:15:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.05, 22.81, 20.81 [02:15:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.84, 3.67, 3.88 [02:16:55] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.14, 19.04, 18.88 [02:17:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.00, 20.27, 19.88 [02:23:40] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.53, 21.81, 21.14 [02:25:36] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.66, 21.22, 20.98 [02:25:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.38, 2.49, 3.25 [02:29:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.97, 19.29, 20.31 [02:31:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.67, 18.25, 19.80 [02:31:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.62, 3.92, 3.64 [02:32:23] PROBLEM - wiki.overwood.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.overwood.xyz could not be found [02:33:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.49, 4.10, 3.74 [02:35:36] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:37:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:37:44] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:39:38] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.222 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:39:45] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [02:42:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:43:48] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.71, 3.96 [02:44:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.86, 21.43, 23.94 [02:45:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 3.99, 4.03 [03:01:07] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:02:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:06:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.26, 16.51, 19.43 [03:07:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:11:25] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.061 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:19:40] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:21:36] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:32:14] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:34:05] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:34:37] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.16, 23.18, 20.43 [03:36:34] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.69, 22.75, 20.57 [03:36:42] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [03:38:31] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.89, 24.82, 21.57 [03:40:27] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.106 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:40:28] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.54, 23.65, 21.59 [03:42:24] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 25.56, 22.54 [03:46:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.43, 22.72, 22.02 [03:58:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 21.49, 21.21 [03:58:20] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:58:34] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:00:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.24, 21.98, 21.46 [04:00:20] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 4.525 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:02:42] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:04:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.53, 23.97, 22.24 [04:04:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:06:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.33, 23.53, 22.29 [04:08:44] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.076 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:12:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.70, 24.83, 22.99 [04:12:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:13:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 21.06, 17.49 [04:13:28] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 21.07, 17.73 [04:13:37] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 22.47, 18.35 [04:14:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.83, 19.75, 16.60 [04:15:13] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.84, 21.79, 18.15 [04:18:10] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:18:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.67, 23.55, 18.85 [04:19:01] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.97, 23.20, 19.44 [04:20:16] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.82, 21.30, 18.67 [04:22:11] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 22.60, 19.42 [04:22:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.46, 23.93, 20.14 [04:24:07] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.50, 21.83, 19.50 [04:24:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 23.97, 20.57 [04:26:38] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.55, 23.48, 21.27 [04:26:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.91, 23.60, 20.84 [04:28:28] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:28:32] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 23.99, 21.70 [04:28:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.63, 24.03, 21.34 [04:30:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.77, 23.81, 21.92 [04:30:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.26, 23.66, 21.58 [04:30:50] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.08, 23.75, 22.66 [04:31:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.07, 22.46, 22.39 [04:32:33] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 24.82, 22.27 [04:33:12] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 24.72, 23.23 [04:34:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.33, 23.38, 22.02 [04:35:41] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.34, 20.35, 20.12 [04:36:38] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:37:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.08, 22.34, 22.75 [04:37:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:42:23] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:42:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.26, 18.42, 20.16 [04:42:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.32, 17.36, 19.71 [04:44:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.41, 17.07, 19.49 [04:45:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 10.68, 16.05, 19.81 [04:51:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.44, 3.04, 3.86 [04:52:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.49, 18.78, 19.04 [04:54:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.68, 17.12, 18.41 [04:57:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.50, 2.97, 3.48 [05:02:23] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:02:28] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:04:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.61, 20.81, 23.55 [05:04:24] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.423 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:07:23] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:08:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.16, 22.72, 23.48 [05:10:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:12:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.07, 23.00, 23.54 [05:12:39] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 23.63, 23.62 [05:18:58] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:20:54] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:22:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.40, 23.00, 23.53 [05:23:26] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [05:24:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.17, 24.68, 24.09 [05:33:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.41, 2.81, 3.89 [05:35:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.52, 3.96, 4.18 [05:37:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.91, 3.36, 3.95 [05:39:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.52, 4.45, 4.27 [05:42:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:53:26] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [05:55:48] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.55, 3.32, 3.95 [05:57:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.89, 4.09 [05:59:17] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:59:21] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:03:18] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:03:26] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [06:03:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 7.639 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:06:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.90, 21.83, 23.93 [06:08:18] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [06:12:23] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 23.58, 23.64 [06:20:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.03, 22.95, 23.46 [06:23:11] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.92, 19.62, 18.22 [06:24:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 24.62, 23.97 [06:24:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.29, 20.00, 18.01 [06:25:07] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.74, 18.32, 17.91 [06:25:30] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:30] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.08, 22.01, 18.99 [06:27:31] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 6.893277168e-05 secs [06:28:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.65, 20.70, 18.87 [06:32:31] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.82, 19.11, 18.68 [06:34:30] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [06:48:08] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:50:06] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.125 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:54:53] [02puppet] 07BlankEclair opened pull request 03#3899: T12502: Add calendar.google.com to CSP - 13https://github.com/miraheze/puppet/pull/3899 [06:54:58] [02puppet] 07coderabbitai[bot] commented on pull request 03#3899: T12502: Add calendar.google.com to CSP - 13https://github.com/miraheze/puppet/pull/3899#issuecomment-2306417766 [07:01:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:01:50] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:42] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:03:59] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:05:43] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [07:05:48] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [07:05:54] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [07:06:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.29, 22.16, 23.90 [07:09:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.09, 2.43, 3.87 [07:10:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 22.30, 23.44 [07:13:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.11, 1.14, 3.00 [07:14:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.10, 22.41, 23.26 [07:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 23.24, 23.46 [07:22:23] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:25:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.56, 20.18, 17.93 [07:26:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.07, 20.18, 17.68 [07:27:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.76, 19.59, 17.99 [07:28:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.16, 19.32, 17.66 [07:36:23] [02puppet] 07Universal-Omega closed pull request 03#3899: T12502: Add calendar.google.com to CSP - 13https://github.com/miraheze/puppet/pull/3899 [07:36:26] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a2763ee90fa8...e8d82e8076d0 [07:36:27] [02puppet] 07BlankEclair 03e8d82e8 - T12502: Add calendar.google.com to CSP (#3899) [07:36:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.98, 21.51, 19.11 [07:38:20] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.45, 20.06, 17.71 [07:40:16] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.24, 18.95, 17.61 [07:40:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.48, 20.29, 19.18 [07:48:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.53, 19.99, 18.41 [07:49:29] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.49, 20.88, 19.03 [07:49:56] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.93, 20.12, 18.51 [07:50:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.36, 18.35, 18.00 [07:51:27] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.57, 17.34, 17.97 [07:51:51] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.53, 18.52, 18.13 [07:58:46] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [07:58:54] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [08:00:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 13.78, 19.36, 23.11 [08:00:45] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:00:54] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [08:02:40] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:08:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.01, 16.22, 20.10 [08:27:23] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:36:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.93, 19.86, 18.74 [08:38:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.54, 18.35, 18.32 [08:46:50] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 21.81, 19.73 [08:48:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.47, 21.93, 20.01 [08:58:31] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.30, 19.62, 19.90 [09:01:01] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:09:28] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.96, 4.43, 2.04 [09:11:27] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [09:11:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.51, 3.61, 2.01 [09:13:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.77, 3.10, 2.03 [09:17:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.32, 4.49, 2.83 [09:23:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.69, 3.82, 3.21 [09:27:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.67, 3.09, 3.09 [09:40:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.78, 20.52, 19.19 [09:42:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.06, 20.01, 19.13 [09:48:13] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:48:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.28, 5.34, 3.85 [09:50:09] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [09:52:29] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.42, 3.83, 3.55 [09:52:53] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.67, 21.10, 20.04 [09:58:24] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.18, 2.95, 3.29 [09:58:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.41, 20.05, 20.05 [10:00:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:02:22] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.47, 3.89, 3.55 [10:02:37] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:03:04] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:20] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.57, 3.71, 3.56 [10:04:33] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.111 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:07:08] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [10:10:16] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.71, 4.30, 3.76 [10:10:52] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:11:31] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:50] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:13:32] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [10:16:11] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.73, 3.89, 3.83 [10:22:07] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.24, 3.90, 3.78 [10:22:45] [02MirahezeMagic] 07redbluegreenhat pushed 031 commit to 03replacetext-script [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3c6a8c0eda98...ffa64459296d [10:22:47] [02MirahezeMagic] 07redbluegreenhat 03ffa6445 - Fix getting the name of the deleted page [10:22:50] [02MirahezeMagic] 07redbluegreenhat synchronize pull request 03#500: Add a script to check if a wiki is OK for enabling ReplaceText - 13https://github.com/miraheze/MirahezeMagic/pull/500 [10:25:27] miraheze/MirahezeMagic - redbluegreenhat the build has errored. [10:26:22] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:30:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:30:04] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.80, 3.87, 3.88 [10:30:23] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.101 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:34:01] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.02, 3.62, 3.71 [10:35:59] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.55, 3.39, 3.61 [10:37:59] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.72, 4.32, 3.93 [10:38:46] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:38:58] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:42] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:40:55] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [10:45:05] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:47:06] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.402 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:47:39] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:49:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.22, 21.70, 19.22 [11:00:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:02:47] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 22.51, 20.92 [11:04:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.44, 23.16, 21.38 [11:07:40] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:08:37] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.03, 19.72, 20.39 [11:11:14] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728798672 [11:11:17] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728794594 [11:11:19] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728799834 [11:11:20] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728801814 [11:12:40] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:13:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.72, 3.28, 3.77 [11:16:51] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728794594 [11:18:49] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728799834 [11:20:34] [02MirahezeMagic] 07redbluegreenhat reviewed pull request 03#500 commit - 13https://github.com/miraheze/MirahezeMagic/pull/500#discussion_r1728813464 [11:21:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.53, 3.60 [11:23:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.77, 3.04, 3.39 [11:27:29] [02MirahezeMagic] 07redbluegreenhat pushed 031 commit to 03replacetext-script [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ffa64459296d...eaf55d94819d [11:27:31] [02MirahezeMagic] 07redbluegreenhat 03eaf55d9 - fix indent [11:27:33] [02MirahezeMagic] 07redbluegreenhat synchronize pull request 03#500: Add a script to check if a wiki is OK for enabling ReplaceText - 13https://github.com/miraheze/MirahezeMagic/pull/500 [11:29:46] miraheze/MirahezeMagic - redbluegreenhat the build has errored. [11:30:55] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.91, 22.24, 20.33 [11:32:52] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.52, 21.95, 20.44 [11:34:49] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.58, 23.15, 21.05 [11:36:46] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.88, 22.65, 21.07 [11:38:42] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.05, 23.69, 21.62 [11:40:39] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.51, 23.21, 21.70 [11:42:36] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 23.73, 22.04 [11:44:43] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.59, 4.86, 3.84 [11:45:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:51:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:52:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.63, 23.23, 22.95 [11:52:22] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:54:21] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.991 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:56:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:58:37] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.32, 3.78, 3.99 [12:00:36] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.03, 4.51, 4.23 [12:05:35] [02mediawiki-repos] 07OAuthority closed pull request 03#32: T12020: Add UnusedRedirects - 13https://github.com/miraheze/mediawiki-repos/pull/32 [12:05:37] [02mediawiki-repos] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/3d65b9b1d804...e4454ea74bfb [12:05:39] [02mediawiki-repos] 07BlankEclair 03e4454ea - T12020: Add UnusedRedirects (#32) [12:06:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.02, 17.44, 19.98 [12:07:01] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/245eb70a3b11...84350d787b8e [12:07:02] [02mw-config] 07BlankEclair 0384350d7 - T12020: Add UnusedRedirects (#5644) [12:07:04] [02mw-config] 07OAuthority closed pull request 03#5644: T12020: Add UnusedRedirects - 13https://github.com/miraheze/mw-config/pull/5644 [12:07:56] miraheze/mw-config - OAuthority the build passed. [12:12:29] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.37, 3.20, 3.92 [12:14:28] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.07, 4.28, 4.23 [12:15:33] [02puppet] 07redbluegreenhat commented on pull request 03#3898: build(deps-dev): bump rexml from 3.3.3 to 3.3.6 in /modules/graylog - 13https://github.com/miraheze/puppet/pull/3898#issuecomment-2306970247 [12:15:36] [02puppet] 07dependabot[bot] closed pull request 03#3898: build(deps-dev): bump rexml from 3.3.3 to 3.3.6 in /modules/graylog - 13https://github.com/miraheze/puppet/pull/3898 [12:15:39] [02puppet] 07dependabot[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e8d82e8076d0...da3f5925aed2 [12:15:40] [02puppet] 07dependabot[bot] 03da3f592 - build(deps-dev): bump rexml from 3.3.3 to 3.3.6 in /modules/graylog (#3898) [12:15:42] [02puppet] 07dependabot[bot] deleted branch 03dependabot/bundler/modules/graylog/rexml-3.3.6 - 13https://github.com/miraheze/puppet [12:15:43] [02puppet] 07dependabot[bot] deleted branch 03dependabot/bundler/modules/graylog/rexml-3.3.6 [12:18:25] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.88, 3.36, 3.87 [12:18:34] [02MirahezeMagic] 07redbluegreenhat pushed 031 commit to 03replacetext-script [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/eaf55d94819d...c2b655d5c304 [12:18:37] [02MirahezeMagic] 07redbluegreenhat 03c2b655d - oops [12:18:38]