[00:00:17] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:00:29] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181
[00:04:25] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.307 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:06:40] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181
[00:07:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:08:15] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[00:08:31] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181
[00:08:40] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[00:10:20] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[00:11:20] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw151 mw161 mw162
[00:12:08] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw151 mw161
[00:12:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:13:22] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures
[00:13:35] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181
[00:13:56] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:14:15] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw171 mw182
[00:14:27] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[00:15:16] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[00:15:34] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[00:16:00] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[00:16:14] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[00:20:14] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[00:22:12] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.156 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:22:17] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures
[00:22:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:27:01] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:27:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:29:00] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[00:32:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.05, 22.34, 23.87
[00:39:50] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:41:47] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.221 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:50:59] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.29, 17.45, 20.06
[00:56:46] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:00:49] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:07:10] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:07:43] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:09:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:12:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:13:32] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:14:02] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[01:14:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.61, 20.98, 19.42
[01:16:59] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.80, 20.18, 19.30
[01:17:26] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:17:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.081 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:22:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 22.80, 20.65
[01:24:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.90, 21.84, 20.55
[01:26:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 21.93, 20.68
[01:27:03] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:28:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.073 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:28:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.90, 21.91, 20.80
[01:32:59] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.87, 19.99, 20.30
[01:34:02] <botifico>	 [02puppet] 07The-Voidwalker closed pull request 03#3900: port some cloudflare rules to varnish nginx - 13https://github.com/miraheze/puppet/pull/3900
[01:34:04] <botifico>	 [02puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/25f7486bbe1d...5ee29f5218ec
[01:34:05] <botifico>	 [02puppet] 07The-Voidwalker 035ee29f5 - port some cloudflare rules to varnish nginx (#3900)
[01:34:07] <botifico>	 [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet
[01:34:09] <botifico>	 [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1
[01:37:41] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:38:28] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:39:37] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.085 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:40:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 14.48, 18.64, 23.90
[01:40:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 10.90, 17.84, 23.08
[01:40:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 11.44, 18.90, 23.66
[01:40:22] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 12.06, 17.27, 22.90
[01:41:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 12.43, 17.91, 23.35
[01:41:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.80, 16.89, 22.67
[01:42:35] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[01:44:22] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 11.53, 13.54, 20.04
[01:46:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.63, 15.08, 20.00
[01:47:08] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:47:35] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.52, 14.90, 19.81
[01:48:10] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.31, 15.65, 20.12
[01:48:13] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 11.09, 13.98, 19.27
[01:49:05] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:51:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.61, 16.52, 19.92
[01:52:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:04:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.04, 3.27, 3.93
[02:07:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:08:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.85, 3.85, 3.98
[02:09:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.31, 19.97, 23.45
[02:10:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.72, 3.15, 3.69
[02:13:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.90, 22.41, 23.63
[02:16:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.51, 3.66
[02:18:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[02:20:45] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.742 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[02:22:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:22:28] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[02:25:10] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[02:27:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.076 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[02:27:24] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[02:27:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:30:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 19.79, 16.11
[02:30:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 19.85, 15.76
[02:30:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.53, 19.94, 16.24
[02:30:59] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.50, 18.62, 14.19
[02:32:04] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.31, 23.75, 19.38
[02:32:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.47, 21.11, 16.74
[02:32:22] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 20.57, 16.19
[02:32:30] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.90, 22.53, 17.60
[02:32:59] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.12, 16.94, 14.12
[02:34:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.14, 19.72, 16.76
[02:34:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.77, 20.29, 17.20
[02:34:22] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.03, 19.83, 16.41
[02:36:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 22.04, 18.20
[02:36:22] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.58, 18.90, 16.52
[02:36:27] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.26, 21.17, 18.23
[02:38:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.05, 21.57, 18.82
[02:38:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.01, 22.67, 18.93
[02:40:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.60, 21.84, 18.57
[02:40:24] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.11, 22.95, 19.58
[02:42:22] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.17, 22.23, 19.74
[02:42:26] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:44:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 22.47, 20.05
[02:44:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 23.15, 20.35
[02:44:21] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 23.39, 20.50
[02:45:15] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.37, 21.54, 18.76
[02:46:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.26, 22.01, 20.18
[02:46:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 22.39, 19.80
[02:46:19] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.07, 22.53, 20.51
[02:48:10] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.97, 20.20, 19.72
[02:48:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.96, 20.90, 19.56
[02:48:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.34, 23.04, 20.99
[02:49:11] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.68, 19.95, 18.75
[02:50:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 22.48, 20.30
[02:52:10] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.70, 21.03, 20.36
[02:52:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.37, 22.46, 20.59
[02:52:14] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.69, 20.38, 20.37
[02:52:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:54:10] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 20.13, 20.34, 20.16
[02:54:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.66, 20.34, 20.02
[02:55:39] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.64, 22.42, 23.39
[02:56:13] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.58, 18.95, 20.02
[03:07:27] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.18, 14.92, 19.25
[03:08:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.59, 3.01, 3.82
[03:09:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.76, 20.52, 23.90
[03:12:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.67, 3.51, 3.83
[03:12:26] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:14:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.96, 3.00, 3.59
[03:18:11] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.98, 2.65, 3.31
[03:21:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.76, 21.71, 21.94
[03:28:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.78, 4.90, 4.01
[03:31:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 23.44, 23.33
[03:32:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.63, 3.56, 3.71
[03:34:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 4.05, 3.88
[03:38:34] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:40:30] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.073 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:46:23] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[03:49:19] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:49:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:49:35] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.49, 18.62, 20.12
[03:51:04] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:53:02] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[03:53:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.26, 20.32, 20.59
[03:54:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:59:35] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.91, 17.73, 19.59
[04:04:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.69, 3.11, 3.81
[04:08:10] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.35, 2.58, 3.38
[04:12:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.17, 4.72, 4.06
[04:12:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.40, 20.93, 19.77
[04:12:50] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:13:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:14:29] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 22.51, 20.48
[04:14:49] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures
[04:18:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:18:35] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:20:25] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.97, 23.15, 21.73
[04:20:39] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.227 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:27:45] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:30:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:31:52] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[04:36:16] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.73, 18.62, 20.10
[04:36:31] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:38:27] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.062 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:40:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:42:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:42:33] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:44:10] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 20.66, 20.05
[04:46:24] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:46:40] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[04:47:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:48:07] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.90, 22.21, 20.95
[04:52:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:54:04] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.17, 20.12, 20.35
[04:59:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 22.24, 21.07
[05:02:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.01, 3.29, 3.96
[05:04:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.35, 4.67, 4.37
[05:05:25] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:05:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.68, 22.73, 21.98
[05:07:22] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.098 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[05:07:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:07:54] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 23.74, 22.45
[05:11:51] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.47, 23.21, 22.56
[05:13:50] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.49, 24.91, 23.29
[05:14:49] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:15:49] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.87, 23.58, 23.01
[05:17:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 24.01, 23.20
[05:23:20] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:23:45] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.90, 23.14, 23.39
[05:25:16] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[05:27:42] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 23.98, 23.65
[05:29:41] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.07, 23.48, 23.53
[05:31:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:31:40] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 23.88, 24.25, 23.84
[05:33:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.36, 23.00, 23.45
[05:41:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:42:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.31, 3.17, 3.78
[05:46:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.54, 3.76
[05:47:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:48:54] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[05:51:00] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found
[05:51:35] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.05, 18.40, 20.09
[05:52:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:55:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.32, 21.56, 20.99
[05:57:38] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:02:34] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures
[06:05:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.42, 21.38
[06:06:16] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[06:07:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:08:13] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.069 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:12:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:25:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 19.93, 17.65
[06:27:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.21, 18.19, 17.28
[06:28:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:30:13] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[06:31:58] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:33:57] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures
[06:34:15] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.110 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:35:29] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[06:38:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:39:04] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:49:04] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:54:04] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:54:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[06:54:35] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5646: T12515: Reduce file upload limit to 2 MB for irisstationwiki - 13https://github.com/miraheze/mw-config/pull/5646
[06:55:37] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[06:58:12] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.233 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:59:04] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:59:46] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:02:37] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:03:53] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[07:04:04] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:04:33] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.091 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:05:07] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[07:12:52] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:13:25] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:14:48] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.062 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:15:30] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[07:16:47] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:19:04] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:21:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:23:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:25:34] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.073 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:26:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:32:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:33:11] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[07:35:19] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:35:50] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:37:14] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.077 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:37:19] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures
[07:37:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:41:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.18, 21.96, 23.54
[07:42:05] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[07:43:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 23.10, 23.74
[07:45:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.26, 22.28, 23.38
[07:45:55] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:49:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 23.77, 23.77
[07:50:27] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:51:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.14, 22.47, 23.32
[07:53:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 24.15, 23.85
[07:53:42] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:55:10] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:56:33] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5646: T12515: Reduce file upload limit to 2 MB for irisstationwiki - 13https://github.com/miraheze/mw-config/pull/5646
[07:56:36] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f7a3816f003e...baa4202c3c7d
[07:56:37] <botifico>	 [02mw-config] 07BlankEclair 03baa4202 - T12515: Reduce file upload limit to 2 MB for irisstationwiki (#5646)
[07:56:42] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[07:57:09] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures
[07:57:30] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[07:58:42] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:59:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.35, 23.87, 23.91
[08:00:54] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[08:01:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.21, 24.84, 24.24
[08:02:51] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[08:02:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:03:04] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s
[08:03:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:03:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:05:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.34, 23.28, 23.94
[08:07:03] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.124 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:07:13] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[08:07:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:07:22] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s
[08:07:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:15:37] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 258.83 ms
[08:19:45] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 197.18 ms
[08:20:59] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[08:21:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 21.96, 21.37
[08:22:57] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.098 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:23:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:23:40] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[08:23:41] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[08:23:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:23:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:25:32] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[08:28:02] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 183.26 ms
[08:30:04] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 190.60 ms
[08:32:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:33:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.51, 22.55, 22.83
[08:34:18] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.36 ms
[08:36:20] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 181.11 ms
[08:38:26] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[08:42:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:42:30] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.373 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:45:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.35, 22.57, 22.12
[08:45:39] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 181.26 ms
[08:45:46] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[08:46:55] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[08:47:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.47, 22.72, 22.26
[08:47:41] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 179.53 ms
[08:48:13] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures
[08:48:51] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.093 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:52:50] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:53:17] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[08:53:32] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[08:57:23] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.840 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:58:04] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.56 ms
[08:58:21] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:59:05] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[09:00:20] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001181960106 secs
[09:02:10] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 179.86 ms
[09:05:35] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.38, 17.01, 19.71
[09:16:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.69, 3.30, 3.99
[09:17:41] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.71 ms
[09:19:43] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 179.91 ms
[09:20:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.48, 4.17, 4.14
[09:26:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.23, 3.43, 3.94
[09:26:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.52, 24.07, 20.25
[09:28:29] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.31, 23.60, 20.55
[09:34:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.74, 3.49, 3.70
[09:36:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.72, 3.22, 3.56
[09:40:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.47, 4.25, 3.87
[09:44:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.66, 3.97, 3.86
[09:44:19] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.35, 20.24, 20.40
[09:46:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.72, 4.70, 4.15
[09:49:55] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:50:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.41, 20.70, 20.54
[09:52:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:54:02] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.741 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:54:12] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 21.76, 20.94
[09:56:11] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.27, 21.81, 21.06
[09:58:01] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:58:10] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.90, 19.42, 20.27
[09:58:27] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:00:00] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:00:08] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 179.82 ms
[10:02:06] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 23.99, 21.93
[10:02:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:04:05] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.79, 23.00, 21.83
[10:04:41] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 4.877 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:06:04] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 23.68, 22.20
[10:06:16] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 179.77 ms
[10:07:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:08:02] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.29, 22.86, 22.08
[10:13:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 23.98, 22.61
[10:19:50] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:22:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:23:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.556 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:26:50] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:26:54] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.56 ms
[10:31:31] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:33:07] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:33:28] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.079 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:35:42] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:37:51] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:37:53] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:39:50] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:39:50] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004993677139 secs
[10:46:41] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:47:29] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.34 ms
[10:49:44] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:51:42] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:52:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:53:38] <botifico>	 [02puppet] 07redbluegreenhat created branch 03theresnotime-bastion - 13https://github.com/miraheze/puppet
[10:53:39] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03theresnotime-bastion [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d246d8ee25ad
[10:53:40] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.48 ms
[10:53:40] <botifico>	 [02puppet] 07redbluegreenhat 03d246d8e - Add theresnotime to the bastion group
[10:53:48] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3902: Add theresnotime to the bastion group - 13https://github.com/miraheze/puppet/pull/3902
[10:53:55] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3902: Add theresnotime to the bastion group - 13https://github.com/miraheze/puppet/pull/3902#issuecomment-2317314497
[10:54:26] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3902: Add theresnotime to the bastion group - 13https://github.com/miraheze/puppet/pull/3902
[10:54:28] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5ee29f5218ec...8abae6937f68
[10:54:31] <botifico>	 [02puppet] 07redbluegreenhat 038abae69 - Add theresnotime to the bastion group (#3902)
[10:54:32] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03theresnotime-bastion
[10:54:34] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03theresnotime-bastion - 13https://github.com/miraheze/puppet
[10:57:48] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.73 ms
[11:03:51] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:05:05] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.17 ms
[11:05:54] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.280 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:06:09] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:08:07] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004674196243 secs
[11:09:12] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 179.77 ms
[11:10:19] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:11:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:16:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:16:41] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:18:28] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.90 ms
[11:18:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:18:50] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:19:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.12, 20.55, 23.05
[11:20:49] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003999769688 secs
[11:21:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.06, 23.96, 24.00
[11:23:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.53, 21.85, 23.17
[11:24:37] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.69 ms
[11:25:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.31, 23.57, 23.68
[11:27:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.28, 23.07, 23.51
[11:29:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 24.02, 23.79
[11:32:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.14, 2.88, 3.80
[11:36:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 3.57, 3.84
[11:37:41] <icinga-miraheze>	 PROBLEM - wiki.sequoiafabrica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sequoiafabrica.org' expires in 15 day(s) (Sat 14 Sep 2024 11:26:16 AM GMT +0000).
[11:37:51] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/90a8e0265ecc...70b6dc3eb585
[11:37:52] <botifico>	 [02ssl] 07WikiTideSSLBot 0370b6dc3 - Bot: Update SSL cert for wiki.sequoiafabrica.org
[11:38:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.93, 3.62, 3.85
[11:39:05] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 233.46 ms
[11:41:07] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.60 ms
[11:42:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.49, 3.88, 3.85
[11:44:18] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[11:52:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.63, 3.64, 3.98
[11:54:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 4.25, 4.17
[11:55:18] <icinga-miraheze>	 PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864177 seconds old and 146034 bytes
[11:55:32] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[11:57:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.33, 21.21, 18.20
[11:57:17] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[11:59:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.24, 19.91, 18.09
[12:00:13] <icinga-miraheze>	 PROBLEM - wiki.showtapeselection.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.showtapeselection.com' expires in 15 day(s) (Sat 14 Sep 2024 11:28:31 AM GMT +0000).
[12:00:22] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/70b6dc3eb585...e57299ca7adb
[12:00:24] <botifico>	 [02ssl] 07WikiTideSSLBot 03e57299c - Bot: Update SSL cert for wiki.showtapeselection.com
[12:00:57] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:01:37] <icinga-miraheze>	 PROBLEM - wiki.cubestudios.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cubestudios.xyz' expires in 15 day(s) (Sat 14 Sep 2024 11:36:09 AM GMT +0000).
[12:01:49] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e57299ca7adb...6cfa214a6908
[12:01:52] <botifico>	 [02ssl] 07WikiTideSSLBot 036cfa214 - Bot: Update SSL cert for wiki.cubestudios.xyz
[12:02:16] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures
[12:02:44] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:03:01] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0006178319454 secs
[12:07:21] <icinga-miraheze>	 RECOVERY - wiki.sequoiafabrica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sequoiafabrica.org' will expire on Wed 27 Nov 2024 10:39:15 AM GMT +0000.
[12:07:44] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:09:41] <botifico>	 [02GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/9c25dbc1228b...daa58539f67d
[12:09:44] <botifico>	 [02GlobalNewFiles] 07translatewiki 03daa5853 - Localisation updates from https://translatewiki.net.
[12:09:46] <botifico>	 [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/72f5e319781f...a735eeb1ecb2
[12:09:47] <botifico>	 [02ImportDump] 07translatewiki 03a735eeb - Localisation updates from https://translatewiki.net.
[12:09:49] <botifico>	 [02IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/a00e6ce885c4...d0e08a34b7c5
[12:09:52] <botifico>	 [02IncidentReporting] 07translatewiki 03d0e08a3 - Localisation updates from https://translatewiki.net.
[12:09:52] <botifico>	 [02landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/689aa04ca146...67cf8c3efd32
[12:09:55] <botifico>	 [02landing] 07translatewiki 0367cf8c3 - Localisation updates from https://translatewiki.net.
[12:09:57] <botifico>	 [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/DataDump/compare/2c01a6e29aba...b01f0c1281dc
[12:09:58] <botifico>	 [02DataDump] 07translatewiki 03b01f0c1 - Localisation updates from https://translatewiki.net.
[12:10:01] <botifico>	 [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/cf78aa814103...b954480d12a9
[12:10:04] <botifico>	 [02MirahezeMagic] 07translatewiki 03b954480 - Localisation updates from https://translatewiki.net.
[12:10:47] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:11:35] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:12:10] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:13:26] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:13:39] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0007171034813 secs
[12:14:34] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:14:39] <miraheze-github_>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[12:14:55] <miraheze-github>	 miraheze/IncidentReporting - translatewiki the build passed.
[12:15:25] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.886 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:20:24] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.66 ms
[12:20:37] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:21:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:23:54] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:55] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:07] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[12:24:08] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:26:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:28:37] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.77 ms
[12:30:07] <icinga-miraheze>	 RECOVERY - wiki.showtapeselection.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.showtapeselection.com' will expire on Wed 27 Nov 2024 11:01:47 AM GMT +0000.
[12:31:13] <icinga-miraheze>	 RECOVERY - wiki.cubestudios.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cubestudios.xyz' will expire on Wed 27 Nov 2024 11:03:13 AM GMT +0000.
[12:33:02] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:33:11] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:33:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:26] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 15s
[12:33:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:34:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:36:54] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.19 ms
[12:38:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:38:03] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:38:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:21] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 18s
[12:38:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:43:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:43:03] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.98 ms
[12:46:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.70, 19.52, 16.83
[12:46:17] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 22.01, 19.47
[12:48:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:48:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.11, 19.65, 17.21
[12:49:19] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 178.02 ms
[12:52:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.38, 23.11, 20.97
[12:53:24] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.61 ms
[12:53:35] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:56:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 24.04, 21.82
[12:56:14] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:58:14] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0008434653282 secs
[13:03:58] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.33, 23.03, 22.46
[13:04:46] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 178.47 ms
[13:08:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:08:08] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[13:08:51] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.81 ms
[13:10:05] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.337 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[13:13:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:13:48] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.17, 18.05, 20.07
[13:15:08] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.25 ms
[13:16:42] <botifico>	 [02mw-config] 07theresnotime assigned pull request 03#5589: T12195: Enable $wgCirrusSearchPrefixSearchStartsWithAnyWord on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5589
[13:17:09] <icinga-miraheze>	 PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864151 seconds old and 146264 bytes
[13:17:42] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.76, 20.16, 20.66
[13:19:15] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.63 ms
[13:19:40] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.66, 18.91, 20.11
[13:23:00] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:23:33] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.29, 20.24, 20.45
[13:24:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.61, 2.81, 3.94
[13:25:31] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.36, 19.97, 20.35
[13:28:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 3.45, 3.93
[13:29:25] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.40, 20.72, 20.61
[13:30:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.24, 3.06, 3.71
[13:32:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.78, 4.15, 4.03
[13:33:09] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:35:56] <botifico>	 [02mw-config] 07theresnotime assigned pull request 03#5612: GlobalSettings: remove version checks - 13https://github.com/miraheze/mw-config/pull/5612
[13:37:15] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:37:17] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 21.72, 21.18
[13:38:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.59, 3.79, 3.97
[13:38:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.54, 19.41, 17.01
[13:39:15] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.86, 22.49, 21.53
[13:39:57] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.28 ms
[13:40:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.05, 18.94, 17.10
[13:41:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 23.98, 22.17
[13:42:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:43:05] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.88, 19.69, 17.61
[13:43:52] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.77, 20.64, 18.80
[13:45:03] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.29, 18.63, 17.47
[13:45:47] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.65, 18.71, 18.32
[13:46:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 4.07, 3.91
[13:47:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.57, 23.13, 22.63
[13:48:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.98, 3.60, 3.75
[13:50:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.50, 3.67
[13:52:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.56, 3.02, 3.47
[13:52:14] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.80 ms
[13:54:10] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.05, 2.92, 3.40
[13:57:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 21.78, 21.83
[13:58:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.98, 3.74, 3.66
[13:59:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.65, 20.36, 21.32
[14:00:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 3.57, 3.58
[14:00:34] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 179.60 ms
[14:02:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:02:12] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.38, 2.98, 3.36
[14:02:36] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 178.05 ms
[14:03:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.22, 18.36, 20.23
[14:05:25] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:05:52] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:06:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.52, 6.00, 4.47
[14:07:22] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.083 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:09:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.35, 21.41, 20.92
[14:10:51] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 179.21 ms
[14:12:11] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[14:12:53] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.44 ms
[14:13:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 23.12, 21.59
[14:15:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.50, 23.06, 21.78
[14:20:09] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.11 ms
[14:23:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.82, 18.30, 20.31
[14:26:24] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:28:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.48, 2.94, 3.79
[14:30:29] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.05 ms
[14:31:24] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:32:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.05, 3.89, 3.94
[14:32:21] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:34:52] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[14:36:24] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.101 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:36:52] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures
[14:38:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.33, 3.65, 3.93
[14:40:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 3.99, 4.01
[14:42:55] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.81 ms
[14:43:46] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[14:45:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:47:00] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.57 ms
[14:48:53] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 33 minutes ago with 0 failures
[14:49:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.14, 21.41, 23.85
[14:50:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:53:51] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/createAndPromote.php --wiki=metawikibeta TheresNoTime --bureaucrat --sysop --custom-groups=steward (END - exit=256)
[14:53:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:54:55] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:55:13] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/createAndPromote.php --wiki=metawikibeta TheresNoTime --bureaucrat --sysop --custom-groups=steward --force (END - exit=0)
[14:55:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:55:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.06, 22.36, 23.26
[14:57:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.38, 21.51, 22.83
[14:59:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 22.03, 22.82
[15:01:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.37, 21.33, 22.46
[15:03:33] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 178.54 ms
[15:03:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.59, 23.43, 23.07
[15:06:36] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:07:35] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:08:38] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.444 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:09:41] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.47 ms
[15:15:55] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 177.37 ms
[15:16:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.89, 3.53, 3.99
[15:17:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.18, 22.22, 23.55
[15:17:39] <MirahezeRelay>	 <originalauthority, replying to MirahezeLSBot> @theresnotime you need to run this on test151
[15:18:16] <MirahezeRelay>	 <theresnotime> oh?!
[15:18:24] <MirahezeRelay>	 <originalauthority> Yeah its all a bit of a mess
[15:18:54] <MirahezeLSBot>	 !log [theresnotime@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/createAndPromote.php --wiki=metawikibeta TheresNoTime --bureaucrat --sysop --custom-groups=steward --force (END - exit=0)
[15:18:55] <MirahezeRelay>	 <originalauthority> I think anyway (ive never ran a command for beta on prod)
[15:19:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:20:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.61, 4.19, 4.10
[15:20:17] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:22:04] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.11 ms
[15:22:15] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:22:51] <MirahezeRelay>	 <pixldev, replying to originalauthority> Didn’t rhinos blow a brain stem when you tried
[15:25:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 22.30, 22.57
[15:26:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.56, 3.91
[15:26:40] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[15:30:45] <MirahezeLSBot>	 !log [alex@ldap171] created theresnotime LDAP account
[15:30:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:31:47] <MirahezeRelay>	 <originalauthority, replying to pixldev> Rip
[15:32:05] <MirahezeRelay>	 <pixldev> That was very funny
[15:32:58] <MirahezeRelay>	 <pixldev> https://cdn.discordapp.com/attachments/808001911868489748/1278739179798401095/image0.jpg?ex=66d1e62a&is=66d094aa&hm=de243d508190eaf0d5a99e1a3a0dadf651e28ae731fbdd10af0719f90c3eb267&
[15:33:23] <MirahezeLSBot>	 !log [alex@ldap171] Added theresnotime to the sre-mediawiki LDAP group
[15:33:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:36:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 3.50, 3.59
[15:37:16] <icinga-miraheze>	 PROBLEM - cp26 APT on cp26 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[15:38:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.16, 3.10, 3.42
[15:38:34] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.69 ms
[15:39:16] <icinga-miraheze>	 PROBLEM - cp26 APT on cp26 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (5 critical updates).
[15:40:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 3.64, 3.59
[15:40:36] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.75 ms
[15:41:05] <MirahezeLSBot>	 !log [alex@ldap171] Added theresnotime to the sre LDAP group
[15:41:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:43:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:44:50] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 177.32 ms
[15:46:52] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.41 ms
[15:47:08] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php createAndPromote --wiki=ldapwikiwiki alex --bureaucrat (END - exit=256)
[15:47:14] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php createAndPromote --wiki=ldapwikiwiki alex --bureaucrat --force (END - exit=0)
[15:47:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:47:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:50:21] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php createAndPromote --wiki=ldapwikiwiki alex --sysop --force (END - exit=0)
[15:50:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:55:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.03, 20.57, 23.68
[15:58:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:01:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 20.11, 22.06
[16:09:55] <MirahezeRelay>	 <songngu.xyz> that reminds me of Shaun...
[16:15:23] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:18:20] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:20:44] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[16:23:06] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:24:53] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.038 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[16:25:39] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:27:52] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 20.85, 17.66
[16:28:20] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:29:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:29:49] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.84, 21.48, 18.29
[16:31:14] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[16:35:27] <MirahezeRelay>	 <rhinosf1> Whose Shaun
[16:35:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 22.13, 19.44
[16:37:24] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.096 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[16:38:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 21.14, 16.58
[16:40:13] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.21, 20.77, 17.00
[16:43:35] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.31, 21.74, 20.72
[16:44:12] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[16:44:13] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.92, 19.16, 17.28
[16:45:33] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.09, 20.03, 20.26
[16:48:40] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 54%
[16:49:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:49:44] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:49:45] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[16:51:42] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.278 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[16:51:43] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[16:52:40] <icinga-miraheze>	 RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 34%
[16:54:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:54:10] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.68, 0.15, 0.05
[16:54:17] <CosmicAlpha>	 !log reboot prometheus151
[16:54:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:54:53] <MirahezeRelay>	 <songngu.xyz, replying to rhinosf1> Shaun Murphy :)
[16:59:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:59:13] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[16:59:50] <MirahezeRelay>	 <rhinosf1, replying to songngu.xyz> Which one?
[17:00:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.17, 4.45, 1.95
[17:00:16] <MirahezeRelay>	 <songngu.xyz, replying to rhinosf1> ah right, you haven't seen The Good Doctor did you
[17:00:33] <MirahezeRelay>	 <rhinosf1, replying to songngu.xyz> I have seen it
[17:01:00] <MirahezeRelay>	 <rhinosf1> But I'm also doing multiple things at once
[17:01:11] <MirahezeRelay>	 <rhinosf1> That kinda makes sense though
[17:01:45] <MirahezeRelay>	 <songngu.xyz> never the less I would like to watch it again...
[17:02:01] <MirahezeRelay>	 <rhinosf1> It is good
[17:02:09] <MirahezeRelay>	 <songngu.xyz> yes
[17:02:57] <MirahezeRelay>	 <songngu.xyz> btw I may ask for access, but just the mw-config stuffs
[17:03:16] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:05:39] <MirahezeRelay>	 <rhinosf1> That would be mw-admins
[17:05:50] <MirahezeRelay>	 <rhinosf1> It's a bit wider access than just config
[17:06:10] <MirahezeRelay>	 <songngu.xyz, replying to rhinosf1> does it shell related?
[17:06:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.43, 2.29
[17:06:22] <MirahezeRelay>	 <rhinosf1> Yes
[17:06:34] <MirahezeRelay>	 <rhinosf1> You need shell to deploy config
[17:07:37] <MirahezeRelay>	 <songngu.xyz> I have done shell executions and mw config change for quite sometimes before I have to make this my hosting service
[17:07:58] <MirahezeRelay>	 <songngu.xyz> and I got not 1, but 2 yubikeys, FYI
[17:08:06] <MirahezeRelay>	 <rhinosf1> Good
[17:08:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.58, 4.67, 2.89
[17:08:16] <MirahezeRelay>	 <rhinosf1> You should never have just 1 yubikey
[17:08:40] <MirahezeRelay>	 <songngu.xyz> I noticed that Reception only have 1 sk SSH key
[17:09:07] <MirahezeRelay>	 <rhinosf1> That's an old style ssh key I think
[17:09:10] <MirahezeRelay>	 <rhinosf1> Not a hardware one
[17:09:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:11:18] <MirahezeRelay>	 <songngu.xyz> interesting, and... I did have a question about SSH stuffs, but you see, making an non-discoverable key mean't that there would be no private key file?
[17:13:03] <MirahezeRelay>	 <pixldev, replying to rhinosf1> Is it better to have one or zero
[17:13:14] <MirahezeRelay>	 <rhinosf1, replying to pixldev> 1
[17:13:22] <MirahezeRelay>	 <rhinosf1> But if you loose it, you're fucked
[17:13:24] <MirahezeRelay>	 <rhinosf1> Just get 2
[17:13:44] <MirahezeRelay>	 <rhinosf1> We provided them heavily discounted to a few volunteers once
[17:13:52] <MirahezeRelay>	 <rhinosf1> That's how a lot of ours got it
[17:13:58] <MirahezeRelay>	 <rhinosf1> That was a cloudflare promotion
[17:14:56] <MirahezeRelay>	 <songngu.xyz> FYI, Yubikeys cannot be shipped to Vietnam as our country require a company to import them for us, which is really costly
[17:16:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.49, 3.91, 3.45
[17:16:17] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a9f9e440737b
[17:16:18] <botifico>	 [02puppet] 07Universal-Omega 03a9f9e44 - prometheus: use systemd service
[17:16:21] <botifico>	 [02puppet] 07Universal-Omega created branch 03prometheus-systemd - 13https://github.com/miraheze/puppet
[17:16:23] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:16:27] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903#issuecomment-2318419461
[17:17:11] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9f9e440737b...8d2c5b9d0d25
[17:17:14] <botifico>	 [02puppet] 07Universal-Omega 038d2c5b9 - Fix
[17:17:16] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:18:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 4.43, 3.71
[17:18:28] <MirahezeRelay>	 <pixldev, replying to songngu.xyz> Dang
[17:18:32] <MirahezeRelay>	 <songngu.xyz> How I get my Yubikey you may ask? It cost almost 2mil in vietnamese dong to get 2 differences key
[17:18:53] <MirahezeRelay>	 <pixldev> Do I wanna know How much is that in USD
[17:19:14] <MirahezeRelay>	 <songngu.xyz> 80,39 USD if we round it to exactly 2mil
[17:19:26] <MirahezeRelay>	 <pixldev> Oh
[17:19:36] <MirahezeRelay>	 <pixldev> You almost gave me a stroke
[17:19:52] <MirahezeRelay>	 <pixldev> Here in the states we use the period (.) to indicate decimal ls
[17:20:02] <MirahezeRelay>	 <pixldev> And commas for thousands
[17:20:38] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:20:59] <MirahezeRelay>	 <songngu.xyz> we uses difference system, no wonder why
[17:21:08] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:21:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.79, 21.39, 18.22
[17:21:15] <MirahezeRelay>	 <rhinosf1> @cosmicalpha can I apply for more cheap yubikeys to hand out
[17:21:31] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.08 ms
[17:22:10] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 4.004 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:22:19] <MirahezeRelay>	 <rhinosf1> Yubico can ship to @songngu.xyz
[17:22:32] <MirahezeRelay>	 <songngu.xyz> I already have keys for that
[17:22:37] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[17:23:07] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[17:23:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.50, 20.54, 18.36
[17:23:32] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.12 ms
[17:24:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:24:13] <MirahezeRelay>	 <rhinosf1> Good
[17:25:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 21.92, 19.10
[17:25:40] <MirahezeRelay>	 <originalauthority> Ship me 2
[17:25:55] <MirahezeRelay>	 <pixldev> I just keep my MFA gens on my phone and a USB
[17:26:00] <MirahezeRelay>	 <pixldev> Plus recov codes
[17:26:16] <MirahezeRelay>	 <pixldev> Should probably keep recov codes in more then just a usb
[17:26:27] <MirahezeRelay>	 <rhinosf1> @originalauthority if @cosmicalpha says I can go ask for more
[17:26:29] <MirahezeRelay>	 <rhinosf1> I will
[17:26:36] <MirahezeRelay>	 <rhinosf1> I don't have any left from the last round
[17:27:09] <MirahezeRelay>	 <originalauthority> Rip
[17:27:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.94, 22.04, 19.50
[17:29:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 22.60, 20.00
[17:30:06] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/8d2c5b9d0d25...2546b0b7f67a
[17:30:09] <botifico>	 [02puppet] 07Universal-Omega 032546b0b - Add
[17:30:11] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:30:59] <MirahezeRelay>	 <bluemoon0332> I'm the only user of FIDO-backed keys in the tech team
[17:31:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.27, 21.40, 19.85
[17:31:39] <MirahezeRelay>	 <originalauthority> I just use regular ssh keys because i live life on the edge
[17:32:04] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2546b0b7f67a...d9780fbe88e6
[17:32:06] <botifico>	 [02puppet] 07Universal-Omega 03d9780fb - Fix
[17:32:08] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:32:11] <MirahezeRelay>	 <bluemoon0332> probably because Putty doesn't support it
[17:32:28] <MirahezeRelay>	 <originalauthority> Why who uses putty
[17:32:46] <MirahezeRelay>	 <bluemoon0332> I think at least Reception does
[17:32:52] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:32:53] <MirahezeRelay>	 <bluemoon0332> probably OU too
[17:32:53] <MirahezeRelay>	 <pixldev> I just use the terminal
[17:32:54] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d9780fbe88e6...76a5d1a0369c
[17:32:56] <MirahezeRelay>	 <originalauthority> Surprised thats still going
[17:32:57] <botifico>	 [02puppet] 07Universal-Omega 0376a5d1a - Remove
[17:33:04] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> UO)
[17:33:24] <MirahezeRelay>	 <bluemoon0332, replying to bluemoon0332> I'm guessing from the fact they both wrote the guide for using PuTTY on Miraheze
[17:33:39] <MirahezeRelay>	 <originalauthority> I think UO uses mobile most of the time tbf
[17:33:52] <MirahezeRelay>	 <originalauthority> All hail terminus
[17:33:55] <MirahezeRelay>	 <pixldev> I don’t understand how he develops on mobile
[17:34:06] <MirahezeRelay>	 <pixldev, replying to originalauthority> I can barely use it turn on a docker container
[17:34:07] <MirahezeRelay>	 <bluemoon0332, replying to pixldev> text to speech
[17:34:08] <MirahezeRelay>	 <originalauthority> The power of samsung
[17:34:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 3.01, 3.77
[17:34:20] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> ☠️
[17:34:28] <MirahezeRelay>	 <pixldev> Is he just a masochist
[17:34:32] <MirahezeRelay>	 <pixldev> Like
[17:34:33] <MirahezeRelay>	 <pixldev> How
[17:34:35] <MirahezeRelay>	 <pixldev> Do you dev
[17:34:37] <MirahezeRelay>	 <pixldev> On a phone
[17:34:55] <MirahezeRelay>	 <originalauthority, replying to pixldev> Erm not sure that term means what you think it does
[17:35:10] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.65, 20.31, 19.76
[17:35:12] <MirahezeRelay>	 <bluemoon0332, replying to pixldev> some phones allow you to plugin a keyboard/connect one over Bluetooth don't they
[17:35:15] <MirahezeRelay>	 <originalauthority> Or maybe it doesnt mean what i think it means
[17:35:26] <MirahezeRelay>	 <bluemoon0332> maybe whenever he wants to code on mobile he pulls out a BT keyboard
[17:35:28] <MirahezeRelay>	 <rhinosf1> @originalauthority I do have some R Pi Zeros for the cost of postage
[17:35:29] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> Okay that’s fair ig?
[17:35:31] <MirahezeRelay>	 <bluemoon0332> just guessing here
[17:35:56] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 176.13 ms
[17:36:02] <MirahezeRelay>	 <pixldev> Screen would be a pain but without a keyboard on screen understandable
[17:36:09] <MirahezeRelay>	 <bluemoon0332> maybe he really does use the on-screen keyboard 😳
[17:36:28] <MirahezeRelay>	 <pixldev, replying to rhinosf1> Can I you one as a yubi key
[17:36:41] <MirahezeRelay>	 <rhinosf1, replying to pixldev> No
[17:37:02] <MirahezeRelay>	 <rhinosf1> I will see if our next volunteer donations can be yubikeys
[17:37:31] <MirahezeRelay>	 <rhinosf1> Up to a @Board Members whether I can use the foundations name to apply for them
[17:37:40] <MirahezeRelay>	 <bluemoon0332, replying to rhinosf1> can they be GameCubes 🥺?
[17:37:58] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.06 ms
[17:38:05] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> If I can get hold of some cheap ones
[17:38:15] <MirahezeRelay>	 <bluemoon0332> with the broadband adapter and some homebrew I'm sure I can use them for SSH'ing in the servers
[17:39:55] <MirahezeRelay>	 <rhinosf1> If anyone can find tech going on the cheap, I will buy it up
[17:39:57] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/76a5d1a0369c...642ad2c8c42e
[17:40:00] <botifico>	 [02puppet] 07Universal-Omega 03642ad2c - Delete
[17:40:02] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:40:07] <MirahezeRelay>	 <rhinosf1> I got 10 Raspberry Pi Zeros for £50
[17:40:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.97, 3.50, 3.73
[17:40:24] <MirahezeRelay>	 <rhinosf1> I think the yubikeys were like half off
[17:41:15] <MirahezeRelay>	 <rhinosf1> If you can find stuff enough volunteers will buy for less than like £50-100 then I'm happy to buy and ship out
[17:41:44] <MirahezeRelay>	 <rhinosf1> Only to volunteers though
[17:41:47] <MirahezeRelay>	 <rhinosf1> Not anyone
[17:41:55] <MirahezeRelay>	 <rhinosf1> As I need to put my address on customs labels
[17:44:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:44:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.68, 3.91, 3.91
[17:44:52] <MirahezeRelay>	 <songngu.xyz> finally got my ed-sk keys working
[17:45:08] <MirahezeRelay>	 <songngu.xyz> I would kill the VM anyway do meh
[17:46:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.70, 4.15, 3.99
[17:50:21] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.68 ms
[17:50:46] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:52:45] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures
[17:54:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:54:27] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.66 ms
[17:55:07] <icinga-miraheze>	 PROBLEM - prometheus151 Prometheus on prometheus151 is CRITICAL: connect to address 10.0.15.116 and port 9090: Connection refused
[17:55:52] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/642ad2c8c42e...240d3858d09d
[17:55:53] <botifico>	 [02puppet] 07Universal-Omega 03240d385 - Adjust
[17:55:54] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[17:56:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.10, 2.70, 3.79
[17:58:10] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.20, 1.89, 3.36
[18:00:14] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[18:00:15] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/240d3858d09d...9c675557bcd5
[18:00:18] <botifico>	 [02puppet] 07Universal-Omega 039c67555 - -
[18:03:07] <icinga-miraheze>	 RECOVERY - prometheus151 Prometheus on prometheus151 is OK: TCP OK - 0.000 second response time on 10.0.15.116 port 9090
[18:03:20] <botifico>	 [02ssl] 07redbluegreenhat pushed 031 commit to 03themanaworldwiki-miraheze-wildcard [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/6bbcfb7b57d5
[18:03:22] <botifico>	 [02ssl] 07redbluegreenhat 036bbcfb7 - Serve the miraheze.org wildcard on wiki.themanaworld.org
[18:03:24] <botifico>	 [02ssl] 07redbluegreenhat created branch 03themanaworldwiki-miraheze-wildcard - 13https://github.com/miraheze/ssl
[18:03:51] <botifico>	 [02ssl] 07redbluegreenhat opened pull request 03#792: Serve the miraheze.org wildcard on wiki.themanaworld.org - 13https://github.com/miraheze/ssl/pull/792
[18:03:58] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#792: Serve the miraheze.org wildcard on wiki.themanaworld.org - 13https://github.com/miraheze/ssl/pull/792#issuecomment-2318523488
[18:04:00] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:04:45] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.18 ms
[18:05:11] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#792: Serve the miraheze.org wildcard on wiki.themanaworld.org - 13https://github.com/miraheze/ssl/pull/792#issuecomment-2318523488
[18:06:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.45, 4.33, 3.60
[18:06:12] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:07:33] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:07:50] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[18:07:53] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9c675557bcd5...5dd5de5f1064
[18:07:55] <botifico>	 [02puppet] 07Universal-Omega 035dd5de5 - Update
[18:08:50] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.72 ms
[18:09:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:12:26] <botifico>	 [02ssl] 07redbluegreenhat closed pull request 03#792: Serve the miraheze.org wildcard on wiki.themanaworld.org - 13https://github.com/miraheze/ssl/pull/792
[18:12:28] <botifico>	 [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6cfa214a6908...62b91ce4a7a0
[18:12:31] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.984 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:12:31] <botifico>	 [02ssl] 07redbluegreenhat 0362b91ce - Serve the miraheze.org wildcard on wiki.themanaworld.org (#792)
[18:12:32] <botifico>	 [02ssl] 07redbluegreenhat deleted branch 03themanaworldwiki-miraheze-wildcard - 13https://github.com/miraheze/ssl
[18:12:34] <botifico>	 [02ssl] 07redbluegreenhat deleted branch 03themanaworldwiki-miraheze-wildcard
[18:12:34] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures
[18:15:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.02, 20.82, 23.20
[18:15:55] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[18:21:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.01, 22.87, 23.06
[18:24:11] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.75, 0.53, 0.19
[18:24:19] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.68 ms
[18:25:32] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[18:27:05] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:27:17] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:28:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.87, 5.75, 2.41
[18:29:13] <icinga-miraheze>	 PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host
[18:29:15] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[18:29:28] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:29:33] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:29:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.14, 23.62, 23.82
[18:30:10] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.31, 0.09, 0.03
[18:30:53] <MirahezeLSBot>	 !log [alex@cp26.wikitide.net] restarted nginx
[18:30:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:31:44] <Orange_Star>	 I saw puppet fail in cp26 due to nginx, error was however temporary
[18:31:59] <icinga-miraheze>	 RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full
[18:32:01] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[18:33:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 23.85, 23.73
[18:34:12] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[18:34:13] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03prometheus-systemd [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5dd5de5f1064...92f459e7c796
[18:34:14] <botifico>	 [02puppet] 07Universal-Omega 0392f459e - Update
[18:34:34] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.71 ms
[18:35:16] <CosmicAlpha>	 !log increased cpu cores and memory on prometheus151
[18:35:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:35:25] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:45:58] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.67, 21.89, 18.64
[18:47:11] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.64, 21.01, 18.57
[18:49:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 22.65, 19.42
[18:50:07] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.63, 18.99, 15.82
[18:52:03] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.53, 18.67, 16.06
[18:54:55] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.58 ms
[18:56:45] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.05, 22.74, 21.27
[18:58:40] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.36, 19.85, 20.40
[19:00:43] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.06, 20.22, 23.29
[19:04:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 21.41, 23.00
[19:06:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.11, 21.53, 22.85
[19:08:26] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 22.64, 23.08
[19:09:09] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.23 ms
[19:14:25] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:16:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.54, 22.46, 23.02
[19:18:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 24.95, 23.87
[19:23:08] <botifico>	 [02mw-config] 07theresnotime closed pull request 03#5589: T12195: Enable $wgCirrusSearchPrefixSearchStartsWithAnyWord on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5589
[19:23:10] <botifico>	 [02mw-config] 07theresnotime pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/baa4202c3c7d...ddb376f2ee28
[19:23:11] <botifico>	 [02mw-config] 07BlankEclair 03ddb376f - T12195: Enable $wgCirrusSearchPrefixSearchStartsWithAnyWord on rainversewiki (#5589)
[19:23:35] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.10, 18.47, 16.76
[19:23:36] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[19:23:37] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[19:23:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:23:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:24:01] <miraheze-github>	 miraheze/mw-config - theresnotime the build passed.
[19:24:23] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[19:24:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:24:38] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 15s
[19:24:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:25:30] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 21.09, 17.90
[19:29:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.75, 21.88, 18.94
[19:29:21] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/maintenance --wiki=rainversewiki --reindexAndRemoveOk --indexIdentifier now (END - exit=256)
[19:29:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:29:55] <RhinosF1>	 TheresNoTime: you managed to understand mwscript ?
[19:31:34] <TheresNoTime>	 RhinosF1: initially messed that one up ^ but working now ^^'
[19:32:04] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=rainversewiki --reindexAndRemoveOk --indexIdentifier now (END - exit=0)
[19:32:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:43] <RhinosF1>	 TheresNoTime: yipeee!
[19:33:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.95, 24.03, 20.45
[19:33:36] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki=rainversewiki --from 2024-08-29T19:08:19Z --deletes (END - exit=0)
[19:33:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:33:48] <MirahezeLSBot>	 !log [theresnotime@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki=rainversewiki --from 2024-08-29T19:08:19Z (END - exit=0)
[19:34:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:34:24] <RhinosF1>	 TheresNoTime: can I give you a software task?
[19:35:53] <TheresNoTime>	 RhinosF1: sure! no promises though ^^
[19:37:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.91, 22.89, 20.84
[19:37:17] <RhinosF1>	 TheresNoTime: can you make a mediawiki extension for editing some cloudflare features
[19:37:31] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.73 ms
[19:37:33] <RhinosF1>	 So wiki crats can turn off like the auto webp stuff
[19:37:39] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[19:37:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:59] <RhinosF1>	 Or build it into ManageWiki
[19:38:00] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 21s
[19:38:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:38:16] <TheresNoTime>	 RhinosF1: seems interesting.. the cloudflare API is pretty nice too.. hmmmm
[19:38:24] <RhinosF1>	 So if you turn something on as a setting it can auto tell cloudflare
[19:38:34] <TheresNoTime>	 does a task for that exist? might be a weekend project
[19:38:48] <RhinosF1>	 TheresNoTime: nope, I came up with the idea 10 seconds ago
[19:39:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.09, 25.21, 21.94
[19:39:31] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.98 ms
[19:40:02] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.87, 19.25, 16.51
[19:40:30] <RhinosF1>	 TheresNoTime: feel free to create a task
[19:43:44] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 176.62 ms
[19:44:00] <TheresNoTime>	 RhinosF1: could you? ^^' just so I know exactly what the requirements are..
[19:45:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.62, 23.29, 22.23
[19:46:02] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.05, 17.63, 16.87
[19:47:42] <RhinosF1>	 TheresNoTime: https://issue-tracker.miraheze.org/T12519
[19:47:45] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.51 ms
[19:47:51] <TheresNoTime>	 ta muchly!
[19:49:13] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.42, 17.14, 19.88
[19:54:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 22.70, 23.88
[19:58:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 23.79, 24.06
[19:58:33] <MirahezeRelay>	 <originalauthority> Seems like a handy feature
[19:58:40] <MirahezeRelay>	 <originalauthority> ManageWiki needs a huge overhaul
[19:58:47] <RhinosF1>	 Yep
[19:58:54] <RhinosF1>	 And now we have another software dev
[19:59:11] <MirahezeRelay>	 <originalauthority> It just really doesnt follow OO Programming and it annoys the life out of me
[20:00:00] <RhinosF1>	 I mean it's like how many years old
[20:00:10] <MirahezeRelay>	 <pixldev> all of them
[20:00:11] <MirahezeRelay>	 <originalauthority> Yeah true
[20:00:12] <RhinosF1>	 It's functionally not that different to when it was first made
[20:00:19] <RhinosF1>	 When Miraheze was first starting out
[20:00:40] <MirahezeRelay>	 <originalauthority> The problem is that its so intertwined with CreateWiki theres no real way to modernise ManageWiki without doing the same to createwiki at the same time
[20:04:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.58, 23.66, 23.98
[20:06:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.30, 23.78, 23.97
[20:07:22] <RhinosF1>	 Forget it and build new?
[20:08:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.68, 22.33, 23.48
[20:12:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.08, 23.65, 23.67
[20:14:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.28, 23.33, 23.54
[20:16:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 24.08, 23.76
[20:17:26] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 175.61 ms
[20:19:26] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.99 ms
[20:23:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 20.54, 17.49
[20:25:13] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.81, 19.51, 17.51
[20:31:17] <MirahezeRelay>	 <originalauthority> I guess yeah
[20:40:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.41, 20.59, 23.64
[20:42:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:45:43] <botifico>	 [02puppet] 07Universal-Omega commented on pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903#issuecomment-2318951519
[20:45:49] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903#issuecomment-2318951751
[20:45:51] <botifico>	 [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903#issuecomment-2318419461
[20:48:14] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.57, 16.72, 20.32
[20:48:25] <botifico>	 [02puppet] 07coderabbitai[bot] edited pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[20:48:52] <botifico>	 [02puppet] 07Universal-Omega reviewed pull request 03#3885 commit - 13https://github.com/miraheze/puppet/pull/3885#discussion_r1737166080
[20:48:56] <botifico>	 [02puppet] 07coderabbitai[bot] reviewed pull request 03#3903 commit - 13https://github.com/miraheze/puppet/pull/3903#discussion_r1737166168
[20:48:58] <botifico>	 [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903#issuecomment-2318419461
[20:52:26] <botifico>	 [02puppet] 07Universal-Omega reviewed pull request 03#3903 commit - 13https://github.com/miraheze/puppet/pull/3903#discussion_r1737172101
[20:52:31] <botifico>	 [02puppet] 07coderabbitai[bot] reviewed pull request 03#3903 commit - 13https://github.com/miraheze/puppet/pull/3903#discussion_r1737166168
[20:52:41] <botifico>	 [02puppet] 07coderabbitai[bot] reviewed pull request 03#3903 commit - 13https://github.com/miraheze/puppet/pull/3903#discussion_r1737172416
[20:53:49] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3903: prometheus: use systemd service - 13https://github.com/miraheze/puppet/pull/3903
[20:53:51] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/puppet/compare/8abae6937f68...23f4f9a618bf
[20:53:54] <botifico>	 [02puppet] 07Universal-Omega 0323f4f9a - prometheus: use systemd service (#3903)
[20:53:56] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03prometheus-systemd
[20:53:57] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03prometheus-systemd - 13https://github.com/miraheze/puppet
[20:54:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.60, 21.53, 21.14
[20:54:58] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:58:40] <MirahezeRelay>	 <originalauthority> this should be a task
[20:58:58] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[21:00:08] <MirahezeRelay>	 <cosmicalpha, replying to originalauthority> I did once have ManageWiki version independent of CreateWiki but it wasn't fully done and I'm not sure I do have it anymore. But I wish they were independent lol
[21:02:28] <MirahezeRelay>	 <cosmicalpha> It basically extracted CreateWiki and ManageWiki shared code to a package that loads the necessary remote code only when necessary and allows ManageWiki to work without CreateWiki. But I didn't 100% like my method either which may be why I procrastinated pushing lol
[21:06:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.12, 22.91, 22.98
[21:10:27] <MirahezeRelay>	 <originalauthority, replying to cosmicalpha> https://issue-tracker.miraheze.org/T12521 I created and invite everyone who has thoughts to share them on the task (I gave some initial/rough thoughts I have, not saying there aren't any better ideas, I haven't given it  a proper thought)
[21:11:02] <MirahezeRelay>	 <originalauthority> If the WMF goes ahead with their plans to move `wgConf` as a service that would be great for us
[21:11:47] <MirahezeRelay>	 <originalauthority> i thought there was a wishlist section on phab for #managewiki and #createwiki, did it get removed or did it never exist?
[21:12:34] <MirahezeRelay>	 <originalauthority> oh its "goals" on #technology-team
[21:12:48] <MirahezeRelay>	 <cosmicalpha> That and there is Community Wishlists that were done annually on meta but we haven't for a couple years I think.
[21:14:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 21.01, 21.63
[21:14:42] <MirahezeRelay>	 <rhinosf1> @theresnotime haven't you come up with a new idea instead of wishlist?
[21:16:12] <MirahezeRelay>	 <theresnotime> oh you mean wmf wishlist? Yeah ^^
[21:16:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.63, 20.72, 21.45
[21:16:26] <MirahezeRelay>	 <theresnotime> But I saw miraheze has/had a wishlist
[21:17:02] <MirahezeRelay>	 <rhinosf1> @theresnotime we do but we haven't done one for a while
[21:17:10] <MirahezeRelay>	 <rhinosf1> So maybe you want to come up with a fancy new idea
[21:17:19] <MirahezeRelay>	 <rhinosf1> Community Wishlist was stole from you anyway
[21:19:03] <MirahezeRelay>	 <songngu.xyz> I did prefer an alternative to SocialProfile, that is until Luke made UPV2
[21:20:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.33, 22.69, 21.88
[21:21:15] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.44 ms
[21:24:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.69, 21.97, 21.96
[21:27:17] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.23 ms
[21:28:14] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.53, 16.88, 19.87
[21:29:54] <MirahezeRelay>	 <pixldev, replying to rhinosf1> i mean idk why we would do a wishlist when we already have a backlog to do
[21:31:25] <MirahezeRelay>	 <rhinosf1> Exactly @pixldev
[21:31:36] <MirahezeRelay>	 <rhinosf1> You are on the money there
[21:31:38] <MacFan4000>	 we used to have goal projects, but we stopped creating new ones
[21:32:20] <MacFan4000>	 https://issue-tracker.miraheze.org/project/profile/80/ was the latest
[21:34:30] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.17 ms
[21:36:30] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.55 ms
[21:39:45] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/62b91ce4a7a0...5c1d63086b8a
[21:39:47] <botifico>	 [02ssl] 07WikiTideSSLBot 035c1d630 - Bot: Add SSL cert for wiki.irisstation.lol
[21:42:04] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#543: add geeu.org zone - 13https://github.com/miraheze/dns/pull/543
[21:46:08] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03nginx-csp [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e3d5244d95e7
[21:46:10] <botifico>	 [02puppet] 07Universal-Omega 03e3d5244 - nginx: add default Content-Security-Policy header
[21:46:11] <botifico>	 [02puppet] 07Universal-Omega created branch 03nginx-csp - 13https://github.com/miraheze/puppet
[21:46:13] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3904: nginx: add default Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3904
[21:46:22] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3904: nginx: add default Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3904#issuecomment-2319084789
[21:46:27] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (START)
[21:46:30] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (END - exit=256)
[21:46:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:46:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:52:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.26, 21.85, 18.39
[21:56:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.49, 23.47, 19.92
[21:58:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.07, 24.29, 20.61
[22:00:04] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.53, 19.45, 15.65
[22:01:59] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.10, 22.49, 17.21
[22:03:00] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[22:03:48] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#543: add geeu.org zone - 13https://github.com/miraheze/dns/pull/543
[22:03:49] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/de48a844dc0f...b0a975c6927f
[22:03:51] <botifico>	 [02dns] 07MacFan4000 03b0a975c - add geeu.org zone (#543)
[22:06:53] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (START)
[22:06:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:07:08] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (END - exit=2)
[22:07:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:08:24] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=unitedworldcuwiki --no-updates --username-prefix=wikia:united-world-cu /home/macfan/unitedworldcuwiki-20240829164207.xml (START)
[22:08:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:11:33] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.61, 22.24, 19.90
[22:12:20] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=unitedworldcuwiki --no-updates --username-prefix=wikia:united-world-cu /home/macfan/unitedworldcuwiki-20240829164207.xml (END - exit=0)
[22:12:21] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (START)
[22:12:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:12:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:18] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=unitedworldcuwiki (END - exit=0)
[22:13:20] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=unitedworldcuwiki --active --update (END - exit=0)
[22:13:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:28] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.00, 20.09, 19.35
[22:13:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:44] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on bots171
[22:13:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:02] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp36
[22:14:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:20] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp37
[22:14:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:41] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on eventgate181
[22:14:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:15:00] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on jobchron171
[22:15:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:15:18] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on matomo151
[22:15:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:15:29] <icinga-miraheze>	 RECOVERY - bots171 APT on bots171 is OK: APT OK: 55 packages available for upgrade (0 critical updates).
[22:15:39] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on db161
[22:15:44] <icinga-miraheze>	 RECOVERY - cp37 APT on cp37 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:15:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:15:47] <icinga-miraheze>	 RECOVERY - cp36 APT on cp36 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:15:57] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on graylog161
[22:16:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:16:05] <icinga-miraheze>	 RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 39 packages available for upgrade (0 critical updates).
[22:16:16] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on db151
[22:16:18] <icinga-miraheze>	 RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:16:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:16:34] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cloud16
[22:16:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:16:46] <icinga-miraheze>	 RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 56 packages available for upgrade (0 critical updates).
[22:16:54] <icinga-miraheze>	 RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 97 packages available for upgrade (0 critical updates).
[22:16:58] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on graphite151
[22:16:59] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[22:17:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:15] <icinga-miraheze>	 RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 32 packages available for upgrade (0 critical updates).
[22:17:16] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cloud15
[22:17:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:35] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on db182
[22:17:36] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.95, 23.18, 23.56
[22:17:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:48] <icinga-miraheze>	 RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 50 packages available for upgrade (0 critical updates).
[22:17:54] <icinga-miraheze>	 RECOVERY - db161 APT on db161 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[22:17:54] <icinga-miraheze>	 RECOVERY - db151 APT on db151 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[22:17:55] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on db171
[22:17:56] <icinga-miraheze>	 RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 105 packages available for upgrade (0 critical updates).
[22:18:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:16] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on bast181
[22:18:17] <icinga-miraheze>	 RECOVERY - db182 APT on db182 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[22:18:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:36] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cloud17
[22:18:41] <icinga-miraheze>	 RECOVERY - bast181 APT on bast181 is OK: APT OK: 47 packages available for upgrade (0 critical updates).
[22:18:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:45] <icinga-miraheze>	 RECOVERY - db171 APT on db171 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[22:18:59] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5c1d63086b8a...a97a553f1222
[22:18:59] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on bast161
[22:19:00] <botifico>	 [02ssl] 07WikiTideSSLBot 03a97a553 - Bot: Update SSL cert for o.kaoyu666.cn
[22:19:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:19:19] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on ldap171
[22:19:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:19:35] <icinga-miraheze>	 RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 97 packages available for upgrade (0 critical updates).
[22:19:39] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on kafka181
[22:19:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:19:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a97a553f1222...087b0b1b2e76
[22:19:53] <icinga-miraheze>	 RECOVERY - bast161 APT on bast161 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:19:54] <botifico>	 [02ssl] 07WikiTideSSLBot 03087b0b1 - Bot: Add SSL cert for geeu.org
[22:19:59] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on db181
[22:20:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:20:18] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cloud18
[22:20:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:20:25] <icinga-miraheze>	 RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 32 packages available for upgrade (0 critical updates).
[22:20:27] <icinga-miraheze>	 RECOVERY - db181 APT on db181 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[22:20:36] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp27
[22:20:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:20:42] <icinga-miraheze>	 RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:20:43] <icinga-miraheze>	 RECOVERY - cp27 APT on cp27 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:20:53] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp41
[22:20:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:21:05] <icinga-miraheze>	 RECOVERY - cp41 APT on cp41 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:21:14] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on mem151
[22:21:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:21:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.20, 25.11, 24.16
[22:21:31] <icinga-miraheze>	 RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 98 packages available for upgrade (0 critical updates).
[22:21:44] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw151
[22:21:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:22:02] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp51
[22:22:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:22:17] <icinga-miraheze>	 RECOVERY - mw151 APT on mw151 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:22:23] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on mem161
[22:22:25] <icinga-miraheze>	 RECOVERY - mem151 APT on mem151 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:22:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:22:44] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on mon181
[22:22:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:06] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on os151
[22:23:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:18] <icinga-miraheze>	 RECOVERY - mon181 APT on mon181 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[22:23:22] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on prometheus151
[22:23:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:46] <icinga-miraheze>	 RECOVERY - cp51 APT on cp51 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:23:50] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw171
[22:23:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:24:07] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:24:07] <icinga-miraheze>	 RECOVERY - mem161 APT on mem161 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:24:11] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on rdb151
[22:24:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:24:29] <icinga-miraheze>	 RECOVERY - mw171 APT on mw171 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:24:30] <icinga-miraheze>	 RECOVERY - os151 APT on os151 is OK: APT OK: 48 packages available for upgrade (0 critical updates).
[22:24:38] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw161
[22:24:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:24:59] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on os161
[22:25:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:25:06] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[22:25:27] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw172
[22:25:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:25:56] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mwtask171
[22:25:56] <icinga-miraheze>	 RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:26:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:26:02] <icinga-miraheze>	 RECOVERY - mw161 APT on mw161 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:26:16] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on os162
[22:26:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:26:31] <icinga-miraheze>	 RECOVERY - mw172 APT on mw172 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:26:32] <icinga-miraheze>	 RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 55 packages available for upgrade (0 critical updates).
[22:26:39] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on phorge171
[22:26:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:26:55] <icinga-miraheze>	 RECOVERY - os161 APT on os161 is OK: APT OK: 47 packages available for upgrade (0 critical updates).
[22:27:09] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw162
[22:27:10] <icinga-miraheze>	 RECOVERY - mw162 APT on mw162 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:27:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:27:19] <icinga-miraheze>	 RECOVERY - os162 APT on os162 is OK: APT OK: 48 packages available for upgrade (0 critical updates).
[22:27:35] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on puppet181
[22:27:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:28:05] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mwtask181
[22:28:05] <icinga-miraheze>	 RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[22:28:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:28:21] <icinga-miraheze>	 RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 48 packages available for upgrade (0 critical updates).
[22:28:24] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.41 ms
[22:28:26] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on ns1
[22:28:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:28:55] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw152
[22:29:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:29:25] <icinga-miraheze>	 RECOVERY - mw152 APT on mw152 is OK: APT OK: 57 packages available for upgrade (0 critical updates).
[22:29:27] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw182
[22:29:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.37, 21.32, 23.25
[22:29:34] <icinga-miraheze>	 RECOVERY - ns1 APT on ns1 is OK: APT OK: 45 packages available for upgrade (0 critical updates).
[22:29:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:29:50] <icinga-miraheze>	 RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 55 packages available for upgrade (0 critical updates).
[22:30:01] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on mw181
[22:30:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:30:22] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on reports171
[22:30:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:30:32] <icinga-miraheze>	 RECOVERY - mw182 APT on mw182 is OK: APT OK: 58 packages available for upgrade (0 critical updates).
[22:31:00] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, ghostscript, libgs10, libgs10-common, and libgs-common on test151
[22:31:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:31:08] <icinga-miraheze>	 RECOVERY - mw181 APT on mw181 is OK: APT OK: 58 packages available for upgrade (0 critical updates).
[22:31:36] <icinga-miraheze>	 RECOVERY - reports171 APT on reports171 is OK: APT OK: 58 packages available for upgrade (0 critical updates).
[22:31:58] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on cp26
[22:32:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:32:07] <icinga-miraheze>	 RECOVERY - cp26 APT on cp26 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:32:25] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 67 packages available for upgrade (0 critical updates).
[22:32:32] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftac171
[22:32:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:32:52] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftobject161
[22:32:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:15] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftobject151
[22:33:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:37] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftobject171
[22:33:41] <icinga-miraheze>	 RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:33:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:00] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftproxy161
[22:34:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:23] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.34 ms
[22:34:24] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftobject181
[22:34:27] <icinga-miraheze>	 RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 33 packages available for upgrade (0 critical updates).
[22:34:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:41] <icinga-miraheze>	 RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 33 packages available for upgrade (0 critical updates).
[22:34:43] <icinga-miraheze>	 RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:34:49] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages python3.11-dev, libpython3.11-dev, libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, and libpython3.11-minimal on swiftproxy171
[22:34:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:35:19] <icinga-miraheze>	 RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 47 packages available for upgrade (0 critical updates).
[22:35:23] <icinga-miraheze>	 RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[22:35:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 22.05, 22.87
[22:35:46] <icinga-miraheze>	 RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 47 packages available for upgrade (0 critical updates).
[22:37:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.28, 21.92, 22.78
[22:45:23] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.76 ms
[22:47:29] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 176.08 ms
[22:47:36] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 21.19, 21.54
[22:49:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.00, 21.93, 21.79
[22:54:11] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[22:55:13] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.29, 24.63, 22.88
[22:57:07] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.04, 23.79, 22.78
[22:59:58] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.73, 17.33, 13.53
[23:00:16] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 20.08, 16.06
[23:00:20] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.88, 16.97, 13.65
[23:01:07] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 18.65, 14.03
[23:01:28] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.17, 18.87, 13.24
[23:01:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 19.37, 14.64
[23:01:57] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 20.70, 15.25
[23:02:20] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 20.65, 15.42
[23:03:26] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw171
[23:03:38] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.25, 20.51, 15.94
[23:03:56] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.76, 21.22, 16.11
[23:04:19] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.90, 20.40, 15.90
[23:04:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 23.93, 23.28
[23:04:57] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.45, 21.18, 16.04
[23:05:23] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[23:05:24] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.77, 21.50, 15.81
[23:05:29] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.80, 21.35, 16.52
[23:06:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.07, 21.55, 18.12
[23:06:06] <icinga-miraheze>	 RECOVERY - o.kaoyu666.cn - LetsEncrypt on sslhost is OK: OK - Certificate 'o.kaoyu666.cn' will expire on Wed 27 Nov 2024 09:20:22 PM GMT +0000.
[23:06:19] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.72, 19.32, 16.09
[23:07:22] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.91, 18.11, 15.24
[23:07:59] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 22.91, 19.01
[23:08:47] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.22, 20.04, 16.70
[23:11:04] <icinga-miraheze>	 PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:11:32] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.30, 19.61, 17.60
[23:11:40] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 36.15, 17.67, 7.24
[23:11:42] <icinga-miraheze>	 PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:13:01] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19644 bytes in 0.063 second response time
[23:13:26] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.37, 19.09, 17.50
[23:13:38] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.037 second response time
[23:13:46] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.19, 22.86, 20.48
[23:13:52] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.44, 18.39, 17.56
[23:17:35] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.33, 5.93, 5.28
[23:19:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 22.62, 20.88
[23:26:21] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[23:27:18] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.11, 23.04, 22.07
[23:32:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:39:02] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.80, 18.72, 20.28
[23:50:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.13, 21.60, 23.96
[23:52:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 23.20, 24.25
[23:56:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.37, 21.55, 23.36
[23:58:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 23.29, 23.82
[23:59:55] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 246.10 ms