[00:09:46] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7f4a61368e37...3df4d6a7dbde [00:09:47] [02puppet] 07paladox 033df4d6a - changeprop: set worker_heartbeat_timeout [00:13:14] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3df4d6a7dbde...fc5e97934007 [00:13:16] [02puppet] 07paladox 03fc5e979 - Changeprop: adjust logger [00:22:40] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fc5e97934007...25c55ef8f8c6 [00:22:42] [02puppet] 07paladox 0325c55ef - changeprop: lower kafka concurrency [00:32:15] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mariopedia.org' expires in 15 day(s) (Fri 11 Oct 2024 12:06:46 AM GMT +0000). [00:32:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/846b30a94a50...a9f19131c744 [00:32:28] [02ssl] 07WikiTideSSLBot 03a9f1913 - Bot: Update SSL cert for mariopedia.org [00:33:22] [02puppet] 07Universal-Omega created branch 03mwscript-php - 13https://github.com/miraheze/puppet [00:33:25] [02puppet] 07Universal-Omega pushed 031 commit to 03mwscript-php [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b8cb2f8dd54b [00:33:27] [02puppet] 07Universal-Omega 03b8cb2f8 - mwscript: use .php lists by default [00:33:28] [02puppet] 07Universal-Omega opened pull request 03#3966: mwscript: use .php lists by default - 13https://github.com/miraheze/puppet/pull/3966 [00:34:04] [02puppet] 07coderabbitai[bot] commented on pull request 03#3966: mwscript: use .php lists by default - 13https://github.com/miraheze/puppet/pull/3966#issuecomment-2372630346 [00:36:43] [02puppet] 07Universal-Omega pushed 031 commit to 03mwscript-php [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b8cb2f8dd54b...463affc630a5 [00:36:44] [02puppet] 07Universal-Omega 03463affc - Use .php in tests [00:36:46] [02puppet] 07Universal-Omega synchronize pull request 03#3966: mwscript: use .php lists by default - 13https://github.com/miraheze/puppet/pull/3966 [00:38:57] [02puppet] 07Universal-Omega pushed 031 commit to 03mwscript-php [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/463affc630a5...1027309b56cc [00:38:59] [02puppet] 07Universal-Omega 031027309 - -TODO [00:39:00] [02puppet] 07Universal-Omega synchronize pull request 03#3966: mwscript: use .php lists by default - 13https://github.com/miraheze/puppet/pull/3966 [00:40:50] miraheze/puppet - Universal-Omega the build passed. [00:50:49] PROBLEM - persist.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'persist.wiki' expires in 15 day(s) (Fri 11 Oct 2024 12:21:52 AM GMT +0000). [00:50:59] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a9f19131c744...70cccb672bb0 [00:51:00] [02ssl] 07WikiTideSSLBot 0370cccb6 - Bot: Update SSL cert for persist.wiki [01:01:00] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Mon 23 Dec 2024 11:33:51 PM GMT +0000. [01:05:32] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knifepointhorror.wiki' expires in 15 day(s) (Fri 11 Oct 2024 12:38:56 AM GMT +0000). [01:05:48] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/70cccb672bb0...5bfb2cd63f78 [01:05:51] [02ssl] 07WikiTideSSLBot 035bfb2cd - Bot: Update SSL cert for knifepointhorror.wiki [01:14:15] [02mw-config] 07Universal-Omega created branch 03no-config - 13https://github.com/miraheze/mw-config [01:14:16] [02mw-config] 07Universal-Omega pushed 031 commit to 03no-config [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/ba1c8d39537e [01:14:17] [02mw-config] 07Universal-Omega 03ba1c8d3 - Try without config cache files [01:14:19] [02mw-config] 07Universal-Omega opened pull request 03#5685: Try without config cache files - 13https://github.com/miraheze/mw-config/pull/5685 [01:15:10] miraheze/mw-config - Universal-Omega the build passed. [01:20:07] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Mon 23 Dec 2024 11:52:24 PM GMT +0000. [01:20:17] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'buildabearwiki.info' expires in 15 day(s) (Fri 11 Oct 2024 01:07:26 AM GMT +0000). [01:20:31] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5bfb2cd63f78...e623a150bcd1 [01:20:32] [02ssl] 07WikiTideSSLBot 03e623a15 - Bot: Update SSL cert for buildabearwiki.info [01:22:55] [02mw-config] 07Universal-Omega edited pull request 03#5685: Try without config cache files - 13https://github.com/miraheze/mw-config/pull/5685 [01:26:29] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [01:29:05] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'roll-of-arms.com' expires in 15 day(s) (Fri 11 Oct 2024 12:58:02 AM GMT +0000). [01:29:32] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e623a150bcd1...95c2d35f54f1 [01:29:35] [02ssl] 07WikiTideSSLBot 0395c2d35 - Bot: Update SSL cert for roll-of-arms.com [01:35:11] RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Tue 24 Dec 2024 12:07:10 AM GMT +0000. [01:38:17] PROBLEM - iol.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'iol.wiki' expires in 15 day(s) (Fri 11 Oct 2024 01:25:12 AM GMT +0000). [01:38:29] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:38:29] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/95c2d35f54f1...a0d591e4f04a [01:38:32] [02ssl] 07WikiTideSSLBot 03a0d591e - Bot: Update SSL cert for iol.wiki [01:43:09] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'royal-wiki.org' expires in 15 day(s) (Fri 11 Oct 2024 01:33:31 AM GMT +0000). [01:43:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a0d591e4f04a...c97a557b2caa [01:43:22] [02ssl] 07WikiTideSSLBot 03c97a557 - Bot: Update SSL cert for royal-wiki.org [01:49:24] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Tue 24 Dec 2024 12:21:56 AM GMT +0000. [01:58:57] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Tue 24 Dec 2024 12:30:41 AM GMT +0000. [02:05:15] [02mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/34e6b632e31f...7bacc004859f [02:05:17] [02mw-config] 07MacFan4000 037bacc00 - use new footer button on beta [02:05:55] !log [macfan@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [02:05:57] !log [macfan@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [02:05:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:06] miraheze/mw-config - MacFan4000 the build passed. [02:06:56] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:18] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [02:07:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:54] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Tue 24 Dec 2024 12:39:53 AM GMT +0000. [02:18:12] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'squareenixwiki.com' expires in 15 day(s) (Fri 11 Oct 2024 01:52:06 AM GMT +0000). [02:18:25] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c97a557b2caa...ca364df59acb [02:18:26] [02ssl] 07WikiTideSSLBot 03ca364df - Bot: Update SSL cert for squareenixwiki.com [02:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:38:53] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:40:53] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.87 ms [02:41:28] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Tue 24 Dec 2024 12:44:45 AM GMT +0000. [03:01:47] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=lostepisodecreepypastawiki dump.xml --no-updates (END - exit=0) [03:01:48] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=lostepisodecreepypastawiki (START) [03:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:11] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Tue 24 Dec 2024 01:19:48 AM GMT +0000. [03:21:25] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:23:25] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.93 ms [03:41:14] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ca364df59acb...4801c5b59bb0 [03:41:16] [02ssl] 07MacFan4000 034801c5b - Update certs.yaml [03:57:45] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=lostepisodecreepypastawiki (END - exit=0) [03:57:46] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=lostepisodecreepypastawiki --update (END - exit=0) [03:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:31] [02mw-config] 07silentgamer568 opened pull request 03#5686: set wgUploadNavigationUrl for Abyssal Wiki - 13https://github.com/miraheze/mw-config/pull/5686 [04:20:50] miraheze/mw-config - silentgamer568 the build passed. [04:21:01] [02mw-config] 07Universal-Omega closed pull request 03#5686: set wgUploadNavigationUrl for Abyssal Wiki - 13https://github.com/miraheze/mw-config/pull/5686 [04:21:03] [02mw-config] 07Universal-Omega commented on pull request 03#5686: set wgUploadNavigationUrl for Abyssal Wiki - 13https://github.com/miraheze/mw-config/pull/5686#issuecomment-2372885799 [04:41:39] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:45:57] miraheze/MirahezeMagic - colleirose the build has errored. [04:55:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:59:16] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.80 ms [05:15:02] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2373038347 [06:09:03] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --procs 64 (START) [06:09:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:06] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 69.80, 39.85, 19.36 [06:23:29] @cosmicalpha seems you're right! [06:25:07] Hopefully we can get jobs fixed. I am flying out of Australia tomorrow, so won't be around much for the next couple days. [06:25:45] I'll be semi around for a little while but not with a ton of access. [06:27:04] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 8.14, 22.28, 16.81 [06:27:21] @cosmicalpha shouldn't this be just if $beta? https://github.com/miraheze/mw-config/blob/master/GlobalCache.php#L152 [06:27:36] Oh yes the logic is inversed [06:27:42] will fix then [06:27:52] That means it only uses changeprop on redis lol [06:27:56] lop [06:28:30] Hopefully it works because wants redis stops runJobs won't work anymore [06:28:33] [02mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7bacc004859f...41bd87fb006e [06:28:35] [02mw-config] 07AgentIsai 0341bd87f - Only use Redis if beta [06:29:02] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:29:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:25] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [06:29:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:31] miraheze/mw-config - AgentIsai the build passed. [06:30:30] seems to be working so far... [06:30:36] ugh I just realized I pushed my change to stop the possibility for root to run maintenance scripts which causes database lists to be broken and fatals to happen to my own (private) fork earlier rather than to the main mw-config lol [06:31:03] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.04, 17.68, 16.30 [06:31:06] oh lol, I was meaning to ask about that [06:31:12] !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --procs 64 (END - exit=2) [06:31:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:22] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=metawiki --procs 64 (START) [06:31:23] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=metawiki --procs 64 (END - exit=0) [06:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:54] Yeah it just adds a wrapper around any maintenance scripts with a check for certain users. [06:32:02] There is a convenient hook for it lol [06:32:28] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:32:43] powerdns? [06:32:53] don't like those warnings lol [06:32:58] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 64.79, 29.40, 11.53 [06:33:11] what? [06:33:18] oh bother [06:34:05] I can't ssh in [06:34:58] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:12] did something break [06:35:17] I'm getting 502/503s [06:35:37] send a pic please [06:35:47] https://cdn.discordapp.com/attachments/808001911868489748/1288388464827433004/thoAOBF.png?ex=66f500c3&is=66f3af43&hm=9adefed14a12012597ff0c71a04880d2c94a7b8e468bfb5d88e2ba903ee11628& [06:35:54] danke [06:36:05] oh it's loading now after it wasn't for like a minute lol [06:36:59] [1/2] that's not good [06:36:59] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1288388764762247169/jVQirdS.png?ex=66f5010a&is=66f3af8a&hm=6f0545ea8c6524697dff9377df9a88f5cea25b352e1aeed18e4743c6af574568& [06:37:10] now it changed to "No route to host" [06:37:11] I restarted the server as it hung [06:37:17] oh okay [06:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:37:38] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [06:37:38] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [06:37:45] more than 2000 is a bit of an understatement [06:38:25] PROBLEM - db181 MariaDB on db181 is UNKNOWN: [06:39:08] PROBLEM - db181 Backups SQL on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:39:47] PROBLEM - db181 SSH on db181 is CRITICAL: connect to address 10.0.18.102 and port 22: No route to host [06:39:56] PROBLEM - db181 ferm_active on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:39:56] PROBLEM - db181 conntrack_table_size on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:39:59] PROBLEM - ping on db181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.102) [06:40:07] PROBLEM - db181 Backups SQL mhglobal on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:40:07] PROBLEM - db181 Disk Space on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:40:10] PROBLEM - db181 NTP time on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host [06:41:01] PROBLEM - Host db181 is DOWN: CRITICAL - Host Unreachable (10.0.18.102) [06:41:20] ugh [06:42:54] [1/2] I'm getting this error when trying to access https://turbooverkill.miraheze.org [06:42:54] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1288390254641811477/image.png?ex=66f5026d&is=66f3b0ed&hm=5df076b2b54a6049e0d57bbff4f70b822722cf0545f357293961418be445548f& [06:43:01] seems to be related to this [06:43:19] since my wiki also is unavailable for the same reason [06:43:29] I can access some other wikis tho [06:43:34] like https://tribalhunter.miraheze.org [06:43:51] yeah it's just db181 [06:44:46] [1/2] that is not what I expected to see [06:44:46] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1288390724064116799/ZWyW3gz.png?ex=66f502dd&is=66f3b15d&hm=d5e05d1c315eeaa4284a31283121c1e2dd0ab8e6b56a49ae4c6111492d2d045b& [06:45:06] RECOVERY - Host db181 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [06:45:11] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [06:45:26] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 2%Current connections: 10 [06:45:27] the fuck? [06:45:45] RECOVERY - db181 conntrack_table_size on db181 is OK: OK: nf_conntrack is 0 % full [06:45:49] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [06:45:53] RECOVERY - db181 ferm_active on db181 is OK: OK ferm input default policy is set [06:45:57] and just like that it's all back to normal [06:46:02] RECOVERY - ping on db181 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms [06:46:03] RECOVERY - db181 Disk Space on db181 is OK: DISK OK - free space: / 650890MiB (73% inode=99%); [06:46:05] RECOVERY - db181 Backups SQL mhglobal on db181 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 265556 seconds old and 207 bytes [06:46:12] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 72 Threads: 4 Questions: 65361 Slow queries: 0 Opens: 6540 Open tables: 6532 Queries per second avg: 907.791 [06:46:13] PROBLEM - db181 NTP time on db181 is WARNING: NTP WARNING: Offset -0.2220479846 secs [06:46:49] nvm it's not [06:47:03] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.11, 0.52, 0.20 [06:47:03] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.029 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:47:03] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 522090 seconds old and 40849 bytes [06:47:35] it was up for a little bit and now it's saying connection refused again [06:47:45] yeah i think its gonna be a while [06:47:54] should be a few minutes [06:47:59] RECOVERY - db181 APT on db181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:48:17] RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset -0.0001221895218 secs [06:49:02] I figured out that it faliled to reb [06:50:23] should be up now [06:50:47] it is [06:51:31] oh ok cool! i was literally just about to ask [06:53:23] !log [@test151] starting deploy of {'config': True} to test151 [06:53:24] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:53:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:50] [02puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/25c55ef8f8c6...f81ac8788392 [06:53:51] [02puppet] 07AgentIsai 03f81ac87 - Install qemu-guest-agent on all servers [06:59:21] Wait, what happened? [07:01:10] most vms on cloud18 still had the debian install image attached but that image was deleted a while ago but Proxmox didn't remove it from being mounted on the vms so it failed to start as the image was missin [07:01:24] ah [07:01:29] I also need to file some incident reports maybe I will use my long layovers during my trip back for that lol [07:06:03] [1/2] Anything that leads to downtime will likely go back to needing an incident report probably since it is much less common now. When we went down at least once a day yeah there was no way we were gonna file IRs every day lol. I also like to keep records of outages that way also. But the change is we will likely not use IncidentReporting, and go back to the way old fanished method of [07:06:03] [2/2] on-wiki IRs lol. I don't like the extension. Also IRs can serve as a guideline and to learn from any past potentially avoidable mistakes. [08:02:05] !log [agent@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'AutoCreateCategoryPages'} to all [08:02:06] !log [agent@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'AutoCreateCategoryPages'} to all - SUCCESS in 1s [08:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:02:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:37] !log [agent@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'ExternalData'} to all [08:09:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:59] !log [agent@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'ExternalData'} to all - SUCCESS in 22s [08:10:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:51:34] [02puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f81ac8788392...23959dcf8591 [08:51:35] [02puppet] 07AgentIsai 0323959dc - T12229: Run script to purge expired blocks every week [08:51:50] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 27.89 ms [08:53:53] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.88 ms [08:57:02] [02mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/41bd87fb006e...410ce489b17d [08:57:04] [02mw-config] 07AgentIsai 03410ce48 - T12305: Enable password policies for Global Patrollers and Wiki mechanics [08:57:57] miraheze/mw-config - AgentIsai the build passed. [09:01:01] [02mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/410ce489b17d...94c247af0410 [09:01:03] [02mw-config] 07AgentIsai 0394c247a - T12504: Enable Wikibase siteLinkGroups [09:01:53] miraheze/mw-config - AgentIsai the build has errored. [09:23:50] !log [@test151] starting deploy of {'config': True} to test151 [09:23:51] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:25] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [09:51:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:48] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 22s [09:51:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:02:05] PROBLEM - looneypyramids.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'looneypyramids.wiki' expires in 15 day(s) (Fri 11 Oct 2024 09:34:01 AM GMT +0000). [10:02:17] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4801c5b59bb0...d02477df5a70 [10:02:19] [02ssl] 07WikiTideSSLBot 03d02477d - Bot: Update SSL cert for looneypyramids.wiki [10:15:41] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 27.90 ms [10:17:45] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.80 ms [10:24:48] PROBLEM - landofliberos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'landofliberos.com' expires in 15 day(s) (Fri 11 Oct 2024 10:16:43 AM GMT +0000). [10:25:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d02477df5a70...a7848571b2f0 [10:25:01] [02ssl] 07WikiTideSSLBot 03a784857 - Bot: Update SSL cert for landofliberos.com [10:31:04] RECOVERY - looneypyramids.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'looneypyramids.wiki' will expire on Tue 24 Dec 2024 09:03:41 AM GMT +0000. [10:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:54:29] RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Tue 24 Dec 2024 09:26:24 AM GMT +0000. [11:34:26] [02dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e2576f7cd51b...016f48ac59f5 [11:34:27] [02dns] 07paladox 03016f48a - Revert "Add jobrunner.wikitide.org (#619)" [11:42:27] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 136.15 ms [11:46:30] [02dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/016f48ac59f5...ed28f5fc687b [11:46:33] [02dns] 07paladox 03ed28f5f - Revert "Make jobrunner.wikitide.net use test151 temporarily" [11:46:35] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.52 ms [11:54:25] [02dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ed28f5fc687b...45755173febc [11:54:27] [02dns] 07paladox 034575517 - Load balance jobrunner across mwtasks* [11:55:37] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [11:55:40] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/598cbdab011d [11:55:42] [02puppet] 07paladox 03598cbda - changeprop: load balance across mwtasks [11:55:45] [02puppet] 07paladox opened pull request 03#3967: changeprop: load balance across mwtasks - 13https://github.com/miraheze/puppet/pull/3967 [11:55:47] [02puppet] 07paladox closed pull request 03#3967: changeprop: load balance across mwtasks - 13https://github.com/miraheze/puppet/pull/3967 [11:55:49] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23959dcf8591...cebbd5b875e4 [11:55:52] [02puppet] 07paladox 03cebbd5b - changeprop: load balance across mwtasks (#3967) [11:55:55] [02puppet] 07paladox deleted branch 03paladox-patch-1 [11:55:57] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [11:55:59] [02puppet] 07coderabbitai[bot] commented on pull request 03#3967: changeprop: load balance across mwtasks - 13https://github.com/miraheze/puppet/pull/3967#issuecomment-2373874506 [12:09:33] PROBLEM - changeprop151 PowerDNS Recursor on changeprop151 is CRITICAL: No response from DNS ::1 [12:11:33] RECOVERY - changeprop151 PowerDNS Recursor on changeprop151 is OK: DNS OK: 0.032 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:28:51] PROBLEM - changeprop151 Puppet on changeprop151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:29:21] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'webkinzguide.com' expires in 15 day(s) (Fri 11 Oct 2024 12:06:14 PM GMT +0000). [12:29:32] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7848571b2f0...e1303c1bdbde [12:29:33] [02ssl] 07WikiTideSSLBot 03e1303c1 - Bot: Update SSL cert for webkinzguide.com [12:33:21] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cebbd5b875e4...c01ddcf68026 [12:33:22] [02puppet] 07paladox 03c01ddcf - changeprop151: lower workers to 4 [12:34:51] RECOVERY - changeprop151 Puppet on changeprop151 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [12:37:12] !log lower kafka181 cores to 8 [12:38:54] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:39:01] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:39:01] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:06] PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:39:07] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:39:07] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:39:11] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:17] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 5 backends are down. mw151 mw161 mw181 mw174 mw184 [12:39:18] !log lower changeprop151 ram to 12gb [12:39:22] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:39:36] !log lower changeprop151 cores to 8 [12:39:40] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:39:58] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 [12:40:03] PROBLEM - db171 MariaDB Connections on db171 is CRITICAL: Critical connection usage: 96.2%Current connections: 481 [12:40:13] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:40:19] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:40:40] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:40:57] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.587 second response time [12:41:00] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.216 second response time [12:41:05] RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.067 second response time [12:41:06] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:41:06] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.055 second response time [12:41:16] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4009 bytes in 0.073 second response time [12:41:31] PROBLEM - changeprop151 NTP time on changeprop151 is CRITICAL: NTP CRITICAL: Offset 0.707066685 secs [12:42:03] PROBLEM - db171 MariaDB Connections on db171 is WARNING: Warning connection usage: 84.4%Current connections: 422 [12:42:09] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.123 second response time [12:42:54] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [12:42:55] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:42:55] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:57] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:12] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:12] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.277 second response time [12:43:33] RECOVERY - changeprop151 NTP time on changeprop151 is OK: NTP OK: Offset -5.382299423e-05 secs [12:43:37] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:44:52] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.294 second response time [12:45:10] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.253 second response time [12:45:16] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:45:17] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:45:36] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 1.895 second response time [12:45:47] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4009 bytes in 0.060 second response time [12:46:03] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 16%Current connections: 80 [12:46:14] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.053 second response time [12:46:46] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.049 second response time [12:46:52] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.836 second response time [12:46:54] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.133 second response time [12:46:59] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time [12:47:12] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4008 bytes in 0.061 second response time [12:47:12] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3818 bytes in 0.132 second response time [12:47:13] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.057 second response time [12:47:16] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.052 second response time [12:47:17] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:47:58] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:48:54] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.000 second response time on 10.0.15.148 port 7200 [12:53:04] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 27.98 ms [12:55:07] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.87 ms [12:58:47] RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Tue 24 Dec 2024 11:30:56 AM GMT +0000. [13:11:33] PROBLEM - dkpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dkpedia.com' expires in 15 day(s) (Fri 11 Oct 2024 01:04:25 PM GMT +0000). [13:11:45] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1303c1bdbde...35f595c28855 [13:11:47] [02ssl] 07WikiTideSSLBot 0335f595c - Bot: Update SSL cert for dkpedia.com [13:15:26] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'solaswiki.org' expires in 15 day(s) (Fri 11 Oct 2024 12:58:53 PM GMT +0000). [13:15:40] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/35f595c28855...48f0c00e9f2d [13:15:42] [02ssl] 07WikiTideSSLBot 0348f0c00 - Bot: Update SSL cert for solaswiki.org [14:07:57] [02puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c01ddcf68026...65f7554441d3 [14:08:00] [02puppet] 07AgentIsai 0365f7554 - T12130: Add Flourish to CSP [14:08:36] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Tue 24 Dec 2024 12:13:09 PM GMT +0000. [14:12:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:13:17] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Tue 24 Dec 2024 12:17:02 PM GMT +0000. [14:20:14]