[00:04:18] <miraheze-github>	 miraheze/ManageWiki - AgentIsai the build passed.
[00:15:21] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 51778MiB (11% inode=99%);
[00:20:58] <MacFanBot>	 [02ManageWiki] 07AgentIsai merged pull request #528: Fix CI issues with 1.43 (03master...03143-compat) 13https://github.com/miraheze/ManageWiki/pull/528
[00:20:58] <MacFanBot>	 [02ManageWiki] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/ManageWiki/commit/879a8a6a99231f83a22786faa5390c6e1f17557b
[00:20:59] <MacFanBot>	 02ManageWiki/03master 07Agent Isai 03879a8a6 Fix CI issues with 1.43 (#528)
[00:21:01] <MacFanBot>	 [02ManageWiki] 07AgentIsai 04deleted 03143-compat at 035fe1630 13https://github.com/miraheze/ManageWiki/commit/5fe1630
[00:24:51] <MacFanBot>	 [02MatomoAnalytics] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/c67c8d448ab6b5b55f081a01dcb7367fbdf128ac
[00:24:52] <MacFanBot>	 02MatomoAnalytics/03master 07Agent Isai 03c67c8d4 Update to clarify only 1.43+ is supported…
[00:25:32] <miraheze-github>	 miraheze/ManageWiki - AgentIsai the build passed.
[00:29:56] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build passed.
[01:13:04] <icinga-miraheze>	 RECOVERY - tech.exputra.com - Cloudflare on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Wed 26 Mar 2025 06:03:13 PM GMT +0000.
[01:23:21] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49793MiB (10% inode=99%);
[03:00:30] <icinga-miraheze>	 RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 1 seconds old and 1988567 bytes
[03:01:05] <icinga-miraheze>	 RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 4 seconds old and 1984291 bytes
[03:03:16] <icinga-miraheze>	 PROBLEM - db172 Backups SQL on db172 is WARNING: FILE_AGE WARNING: /var/log/db-backups/db-backups/db-backups.log is 1382585 seconds old and 1247 bytes
[03:06:42] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.57 ms
[03:08:41] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.56 ms
[03:15:40] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=battlenationswiki battlenations_pages_full.xml --no-updates (END - exit=0)
[03:15:41] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=battlenationswiki (START)
[03:15:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:15:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:57:19] <icinga-miraheze>	 PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:57:23] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:57:31] <icinga-miraheze>	 PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:57:33] <icinga-miraheze>	 PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:57:33] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:57:35] <icinga-miraheze>	 PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[03:57:42] <icinga-miraheze>	 PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.015 second response time
[03:57:48] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.019 second response time
[03:57:50] <icinga-miraheze>	 PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[03:57:51] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[03:57:55] <icinga-miraheze>	 PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[03:57:58] <icinga-miraheze>	 PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:58:01] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:58:05] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:58:10] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:11] <icinga-miraheze>	 PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[03:58:15] <icinga-miraheze>	 PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[03:58:16] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 122.81, 61.65, 24.88
[03:58:30] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:58:35] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:58:35] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[03:58:36] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[03:58:40] <icinga-miraheze>	 PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[03:58:41] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:58:45] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:48] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki
[03:58:49] <icinga-miraheze>	 PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:51] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[03:58:52] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:55] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:58:56] <icinga-miraheze>	 PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[03:58:57] <icinga-miraheze>	 PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:59:03] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:59:10] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki
[03:59:11] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:59:15] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:59:24] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:01:50] <icinga-miraheze>	 RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.641 second response time
[04:01:51] <icinga-miraheze>	 RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time
[04:01:51] <icinga-miraheze>	 RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time
[04:01:51] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time
[04:01:56] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time
[04:01:58] <icinga-miraheze>	 RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.058 second response time
[04:02:00] <icinga-miraheze>	 RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time
[04:02:06] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.065 second response time
[04:02:07] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.061 second response time
[04:02:19] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time
[04:02:32] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.056 second response time
[04:02:36] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4109 bytes in 0.043 second response time
[04:02:39] <icinga-miraheze>	 RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.062 second response time
[04:02:40] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.049 second response time
[04:02:44] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.051 second response time
[04:02:44] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.055 second response time
[04:02:48] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy
[04:02:51] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time
[04:02:51] <icinga-miraheze>	 RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time
[04:02:53] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time
[04:02:57] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.098 second response time
[04:02:58] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.131 second response time
[04:03:01] <icinga-miraheze>	 RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time
[04:03:03] <icinga-miraheze>	 RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.045 second response time
[04:03:10] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy
[04:03:10] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.464 second response time
[04:03:13] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.745 second response time
[04:03:15] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.735 second response time
[04:03:15] <icinga-miraheze>	 PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[04:03:15] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 25
[04:03:22] <icinga-miraheze>	 PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:03:26] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.150 second response time
[04:03:27] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.056 second response time
[04:03:31] <icinga-miraheze>	 RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.053 second response time
[04:03:35] <icinga-miraheze>	 RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.051 second response time
[04:03:37] <icinga-miraheze>	 RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.064 second response time
[04:03:37] <icinga-miraheze>	 RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.055 second response time
[04:03:50] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:03:51] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is CRITICAL: Too many connections
[04:04:13] <icinga-miraheze>	 RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.052 second response time
[04:04:16] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4129 bytes in 0.060 second response time
[04:11:09] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'config': True, 'force': True} to all
[04:11:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:11:29] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s
[04:11:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:12:22] <MirahezeRelay>	 <pixldev> @agentisai what was the change
[04:12:40] <MirahezeRelay>	 <agentisai> set cluster 4 to ro
[04:13:56] <MirahezeRelay>	 <pixldev, replying to agentisai> Oh
[04:14:06] <MirahezeRelay>	 <pixldev> should we put up a notice
[04:14:13] <MirahezeRelay>	 <pixldev> Also is this the backups script again
[04:16:11] <MirahezeRelay>	 <agentisai> appear to be
[04:17:02] <MirahezeRelay>	 <pixldev, replying to agentisai> What in the hell is going on
[04:17:24] <MirahezeRelay>	 <pixldev> When did it start doing this? May be worth looking for previous config or setuo changes
[04:17:44] <MirahezeRelay>	 <pixldev> Can we just find a way to limit open connections from the backup process
[04:20:00] <MirahezeRelay>	 <agentisai> wikitide-backup is the culprit
[04:20:11] <MirahezeRelay>	 <pixldev, replying to agentisai> Any idea why?
[04:20:24] <MirahezeRelay>	 <pixldev> It must be put on trial
[04:20:34] <MirahezeRelay>	 <pixldev> ⚖️
[04:24:41] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 54 minutes ago with 0 failures
[04:26:06] <icinga-miraheze>	 RECOVERY - db181 APT on db181 is OK: APT OK: 9 packages available for upgrade (0 critical updates).
[04:28:39] <icinga-miraheze>	 RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.6%Current connections: 6
[04:29:08] <icinga-miraheze>	 RECOVERY - db181 MariaDB on db181 is OK: Uptime: 107 Threads: 6 Questions: 70766 Slow queries: 0 Opens: 4556 Open tables: 4501 Queries per second avg: 661.364
[04:57:39] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server
[05:03:11] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 45%
[05:05:11] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 38%
[05:20:15] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.24, 0.29, 11.72
[05:22:51] <MacFan4000>	 !log (nfswiki) UPDATE echo_notification SET notification_read_timestamp=20240415090051 WHERE notification_user=9;
[05:22:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:24:15] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.37, 0.34, 9.13
[05:25:39] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.230 second response time
[05:34:41] <MirahezeRelay>	 <cosmicalpha, replying to pixldev> We need more RAM
[06:34:20] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.59 ms
[06:36:20] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.60 ms
[07:10:09] <icinga-miraheze>	 PROBLEM - worldless.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldless.wiki' expires in 14 day(s) (Thu 20 Feb 2025 07:02:16 AM GMT +0000).
[07:10:20] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/29eaf87632297f2dad8647283098df8b418fc5be
[07:10:20] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 0329eaf87 Bot: Update SSL cert for worldless.wiki
[07:39:48] <icinga-miraheze>	 RECOVERY - worldless.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Tue 06 May 2025 06:11:44 AM GMT +0000.
[09:32:19] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=battlenationswiki (END - exit=0)
[09:32:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:32:20] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=battlenationswiki --update (END - exit=0)
[09:32:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:33:27] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:35:27] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms
[12:21:01] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.73, 11.39, 6.79
[12:22:55] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.14, 10.70, 7.09
[12:24:49] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.25, 9.05, 6.90
[12:36:24] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.82, 10.28, 7.81
[12:46:25] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.95, 9.07, 9.33
[13:00:58] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> Download some
[13:34:54] <BlankEclair>	 girl who plays a game, and then finds rng behaviour frustrating enough that she proceeds to debug it
[14:18:38] <MacFan4000>	 !log (metawiki) DELETE FROM echo_unread_wikis WHERE euw_user=431746;
[14:18:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:30:15] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 31.58 ms
[14:32:15] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms
[14:35:21] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27198MiB (5% inode=99%);
[15:19:54] <MacFanBot>	 [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/95b433eb86562d62121937378d4f59648c8ff2da
[15:19:54] <MacFanBot>	 02ssl/03master 07MacFan4000 0395b433e Add shatteredrealmsrpg.com and pb.maciejcreates.com
[15:32:44] <MacFanBot>	 [02mw-config] 07Friskygote opened pull request #5807: T13091: Change license of swordartonlinezhwiki to CC BY-NC-SA 3.0 CN (07miraheze:03master...07Friskygote:03master) 13https://github.com/miraheze/mw-config/pull/5807
[15:52:44] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw174
[15:53:20] <icinga-miraheze>	 PROBLEM - mw174 Current Load on mw174 is WARNING: LOAD WARNING - total load average: 22.26, 23.62, 11.83
[15:54:43] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy
[15:55:17] <icinga-miraheze>	 RECOVERY - mw174 Current Load on mw174 is OK: LOAD OK - total load average: 5.15, 16.60, 10.68
[16:04:32] <MacFanBot>	 [02mw-config] 07Friskygote opened pull request #5808: T13125: Fix typo in protection level name for googlewiki (07miraheze:03master...07Friskygote:03T13125) 13https://github.com/miraheze/mw-config/pull/5808
[16:06:48] <MacFanBot>	 [02mw-config] 07AgentIsai merged 07Friskygote's pull request #5808: T13125: Fix typo in protection level name for googlewiki (07miraheze:03master...07Friskygote:03T13125) 13https://github.com/miraheze/mw-config/pull/5808
[16:06:48] <MacFanBot>	 [02mw-config] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/45cfedc85c5b48836c75eb6a13cb8a17094df846
[16:06:48] <MacFanBot>	 02mw-config/03master 07Frisk 0345cfedc T13125: Fix typo in protection level name for googlewiki (#5808)…
[16:07:08] <MacFanBot>	 [02mw-config] 07AgentIsai merged 07Friskygote's pull request #5807: T13091: Change license of swordartonlinezhwiki to CC BY-NC-SA 3.0 CN (07miraheze:03master...07Friskygote:03master) 13https://github.com/miraheze/mw-config/pull/5807
[16:07:10] <MacFanBot>	 [02mw-config] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/d31899b03f8391196fe2f88c9c90c4fb2444a2a7
[16:07:12] <MacFanBot>	 02mw-config/03master 07Frisk 03d31899b T13091: Change license of swordartonlinezhwiki to CC BY-NC-SA 3.0 CN (#5807)…
[16:07:24] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=battlenationswiki images --search-recursively (START)
[16:07:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:07:54] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[16:08:04] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[16:08:10] <MacFanBot>	 [02mw-config] 07AgentIsai merged 07robkam's pull request #5722: Open interwiki links in new tab (07miraheze:03master...07robkam:03patch-1) 13https://github.com/miraheze/mw-config/pull/5722
[16:08:11] <MacFanBot>	 [02mw-config] 07AgentIsai pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/08c377b084ff8b86ac826506b8cc10e18ab5d046
[16:08:11] <MacFanBot>	 02mw-config/03master 07Rob Kam 0308c377b Open interwiki links in new tab (#5722)…
[16:09:09] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[16:37:08] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=battlenationswiki images --search-recursively (END - exit=0)
[16:37:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log