[00:32:03] <MirahezeRelay>	 <cosmicalpha, replying to agentisai> oh lol I forgot to take that into account
[01:05:17] <botifico>	 [02.github] 07Universal-Omega created branch 03revert-53-AgentIsai-patch-1 - 13https://github.com/miraheze/.github
[01:05:18] <botifico>	 [02.github] 07Universal-Omega pushed 031 commit to 03revert-53-AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/77b98bb06aa4
[01:05:21] <botifico>	 [02.github] 07Universal-Omega 0377b98bb - Revert "Add keepdependencies key (#53)"
[01:05:24] <botifico>	 [02.github] 07Universal-Omega opened pull request 03#54: Revert "Add keepdependencies key" - 13https://github.com/miraheze/.github/pull/54
[01:05:27] <botifico>	 [02.github] 07Universal-Omega closed pull request 03#54: Revert "Add keepdependencies key" - 13https://github.com/miraheze/.github/pull/54
[01:05:29] <botifico>	 [02.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/e4949264876a...827535c61669
[01:05:31] <botifico>	 [02.github] 07Universal-Omega 03827535c - Revert "Add keepdependencies key (#53)" (#54)
[01:05:34] <botifico>	 [02.github] 07Universal-Omega deleted branch 03revert-53-AgentIsai-patch-1 - 13https://github.com/miraheze/.github
[01:05:37] <botifico>	 [02.github] 07Universal-Omega deleted branch 03revert-53-AgentIsai-patch-1
[01:07:11] <botifico>	 [02ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/a0eb0b23b106...e28b4d5c97da
[01:07:13] <botifico>	 [02ci-scripts] 07Universal-Omega 03e28b4d5 - Check if file exists
[01:08:25] <MirahezeRelay>	 <cosmicalpha, replying to botifico> CC @agentisai all .github/workflows/mediawiki-tests.yaml must be the same so that wasn't a valid solution
[01:08:51] <MirahezeRelay>	 <cosmicalpha> It has to be the same on all extensions
[01:09:05] <MirahezeRelay>	 <cosmicalpha> I think I fixed it
[01:10:43] <MirahezeRelay>	 <cosmicalpha, replying to botifico> with this
[01:14:41] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build passed.
[01:14:52] <MirahezeRelay>	 <cosmicalpha, replying to miraheze-github> There you go @agentisai
[01:15:15] <MirahezeRelay>	 <cosmicalpha> I reran CI and it does work now.
[01:15:18] <MirahezeRelay>	 <cosmicalpha> Finally lol
[01:17:17] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1811622439
[01:17:55] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1811622439
[01:50:47] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[01:59:08] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[02:37:52] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.65 ms
[02:42:00] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[02:53:11] <icinga-miraheze>	 PROBLEM - rosettacode.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rosettacode.org' expires in 15 day(s) (Fri 08 Nov 2024 02:38:28 AM GMT +0000).
[02:54:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/01ef803a22a9...b0c2be52624b
[02:54:25] <botifico>	 [02ssl] 07WikiTideSSLBot 03b0c2be5 - Bot: Update SSL cert for rosettacode.org
[03:21:53] <icinga-miraheze>	 RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Tue 21 Jan 2025 01:55:49 AM GMT +0000.
[03:43:43] <icinga-miraheze>	 PROBLEM - polandballwiki.com - Cloudflare on sslhost is WARNING: WARNING - Certificate 'polandballwiki.com' expires in 30 day(s) (Sat 23 Nov 2024 03:25:15 AM GMT +0000).
[03:53:36] <MacFan4000>	 !log manually delete dumps for nenawikiwiki
[03:53:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:56:47] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[04:05:09] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms
[04:13:26] <icinga-miraheze>	 RECOVERY - polandballwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Tue 21 Jan 2025 02:46:26 AM GMT +0000.
[05:01:03] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[05:07:18] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[05:21:42] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[05:23:43] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[05:35:10] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.69 ms
[05:41:24] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.64 ms
[06:04:14] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[06:06:15] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms
[06:12:32] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[06:16:40] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms
[08:02:10] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[08:06:17] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 30.32 ms
[08:28:52] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 30.81 ms
[08:37:15] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 30.20 ms
[08:39:56] <botifico>	 [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1812192408
[08:39:59] <botifico>	 [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1812206494
[08:40:02] <botifico>	 [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1812197082
[08:45:37] <miraheze-github>	 miraheze/MirahezeMagic - colleirose the build has errored.
[09:11:20] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1812293351
[10:32:30] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:36:38] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.76 ms
[10:49:01] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:52:04] <MirahezeRelay>	 <groupnebula563_0765> I think
[10:52:14] <MirahezeRelay>	 <groupnebula563_0765> there may be an issue with mattermost1
[10:52:25] <MirahezeRelay>	 <groupnebula563_0765> just
[10:52:29] <MirahezeRelay>	 <groupnebula563_0765> just a hunch
[10:53:09] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms
[10:59:24] <MirahezeRelay>	 <rhinosf1, replying to groupnebula563_0765> Kinda
[10:59:29] <MirahezeRelay>	 <rhinosf1> Intermittently over ip?6
[11:01:30] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[11:01:39] <MirahezeRelay>	 <groupnebula563_0765, replying to icinga-miraheze> yeah yeah
[11:01:42] <MirahezeRelay>	 <groupnebula563_0765> we get it
[11:07:45] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.72 ms
[11:18:08] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[11:24:22] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.72 ms
[12:42:18] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 138.95 ms
[12:44:17] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.96 ms
[13:01:33] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[13:07:48] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.83 ms
[13:16:07] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[13:22:21] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.02 ms
[13:28:35] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 138.95 ms
[13:30:34] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.98 ms
[13:42:34] <botifico>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[13:42:36] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d26780b06d40
[13:42:38] <botifico>	 [02puppet] 07paladox 03d26780b - changeprop151: increase concurrency for jobs
[13:42:41] <botifico>	 [02puppet] 07paladox opened pull request 03#4060: changeprop151: increase concurrency for jobs - 13https://github.com/miraheze/puppet/pull/4060
[13:42:44] <botifico>	 [02puppet] 07paladox closed pull request 03#4060: changeprop151: increase concurrency for jobs - 13https://github.com/miraheze/puppet/pull/4060
[13:42:47] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/83729a4aa8f2...cee43330a238
[13:42:50] <botifico>	 [02puppet] 07paladox 03cee4333 - changeprop151: increase concurrency for jobs (#4060)
[13:42:52] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[13:42:55] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1
[13:50:50] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[13:55:41] <botifico>	 [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03showpageviews [+3/-3/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/1ba12aff5b60...407559fa0ea9
[13:55:42] <botifico>	 [02MatomoAnalytics] 07AgentIsai 03407559f - Implement suggestions
[13:55:44] <botifico>	 [02MatomoAnalytics] 07AgentIsai synchronize pull request 03#137: Show page views in action=info - 13https://github.com/miraheze/MatomoAnalytics/pull/137
[13:57:08] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.76 ms
[13:58:41] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build has errored.
[13:59:00] <botifico>	 [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03showpageviews [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/407559fa0ea9...a61d895b1dbd
[13:59:00] <botifico>	 [02MatomoAnalytics] 07AgentIsai 03a61d895 - Fix SQL path
[13:59:01] <botifico>	 [02MatomoAnalytics] 07AgentIsai synchronize pull request 03#137: Show page views in action=info - 13https://github.com/miraheze/MatomoAnalytics/pull/137
[14:03:54] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build passed.
[14:04:40] <botifico>	 [02MatomoAnalytics] 07AgentIsai closed pull request 03#137: Show page views in action=info - 13https://github.com/miraheze/MatomoAnalytics/pull/137
[14:04:41] <botifico>	 [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03master [+4/-1/±12] 13https://github.com/miraheze/MatomoAnalytics/compare/f0e86c1f3777...c7ca6e595d24
[14:04:43] <botifico>	 [02MatomoAnalytics] 07AgentIsai 03c7ca6e5 - Show page views in action=info (#137)
[14:05:37] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MatomoAnalytics'} to all
[14:05:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:06:14] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'MatomoAnalytics'} to all - SUCCESS in 36s
[14:06:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:06:55] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': '1.42'} to all
[14:07:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:09:34] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build passed.
[14:11:56] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03mariadb-mediawiki-passwords [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/f7bfd65e36d5...43486c20e49a
[14:11:59] <botifico>	 [02puppet] 07redbluegreenhat 0343486c2 - Use different passwords on beta and prod for MediaWiki
[14:12:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:12:00] <botifico>	 [02puppet] 07redbluegreenhat synchronize pull request 03#4058: Use different passwords on beta and prod for MediaWiki - 13https://github.com/miraheze/puppet/pull/4058
[14:13:22] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03mariadb-mediawiki-passwords [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/43486c20e49a...3f866e61da72
[14:13:25] <botifico>	 [02puppet] 07redbluegreenhat 033f866e6 - Use different passwords on beta and prod for MediaWiki
[14:13:27] <botifico>	 [02puppet] 07redbluegreenhat synchronize pull request 03#4058: Use different passwords on beta and prod for MediaWiki - 13https://github.com/miraheze/puppet/pull/4058
[14:14:19] <botifico>	 [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/c7ca6e595d24...59e86c2ce4c4
[14:14:22] <botifico>	 [02MatomoAnalytics] 07AgentIsai 0359e86c2 - Use page URLs instead
[14:14:42] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'l10n': True, 'versions': '1.42'} to all - SUCCESS in 467s
[14:14:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:15:46] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03mariadb-mediawiki-passwords [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/3f866e61da72...fc959932641f
[14:15:49] <botifico>	 [02puppet] 07redbluegreenhat 03fc95993 - Use different passwords on beta and prod for MediaWiki
[14:15:50] <botifico>	 [02puppet] 07redbluegreenhat synchronize pull request 03#4058: Use different passwords on beta and prod for MediaWiki - 13https://github.com/miraheze/puppet/pull/4058
[14:19:01] <miraheze-github>	 miraheze/MatomoAnalytics - AgentIsai the build passed.
[14:24:43] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[14:25:26] <botifico>	 [02dns] 07redbluegreenhat closed pull request 03#637: add balloonfightwiki.org zone - 13https://github.com/miraheze/dns/pull/637
[14:25:29] <botifico>	 [02dns] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/2e8fb7f8ba1d...34566da4cd3a
[14:25:30] <botifico>	 [02dns] 07MacFan4000 0334566da - add balloonfightwiki.org zone (#637)
[14:26:25] <miraheze-github>	 miraheze/dns - redbluegreenhat the build passed.
[14:28:51] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.83 ms
[14:32:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:00:39] <MirahezeRelay>	 <pixldev> @agentisai have there been any recent updates to WikiSEO? Someone reported a change in behavior lately but I don’t see anything
[15:02:48] <MirahezeRelay>	 <agentisai> Not to the best of my knowledge
[15:02:51] <MirahezeRelay>	 <agentisai> What’s been reported?
[15:07:39] <MirahezeRelay>	 <pixldev> https://phighting.wiki/Coil/Gallery#Default
[15:14:38] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[15:20:52] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.89 ms
[15:49:26] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[15:53:33] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.76 ms
[15:53:38] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#4058: Use different passwords on beta and prod for MediaWiki - 13https://github.com/miraheze/puppet/pull/4058
[15:53:41] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/cee43330a238...fe398241945c
[15:53:42] <botifico>	 [02puppet] 07redbluegreenhat 03fe39824 - Use different passwords on beta and prod for MediaWiki (#4058)
[15:53:44] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03mariadb-mediawiki-passwords
[15:53:46] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03mariadb-mediawiki-passwords - 13https://github.com/miraheze/puppet
[15:55:55] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4591 bytes in 0.075 second response time
[15:56:07] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4591 bytes in 0.064 second response time
[15:58:27] <icinga-miraheze>	 PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:02:20] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe398241945c...c844510947c3
[16:02:21] <botifico>	 [02puppet] 07redbluegreenhat 03c844510 - fix
[16:02:48] <MirahezeRelay>	 <bluemoon0332> made a silly little typo
[16:04:27] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[16:11:23] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[16:13:24] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms
[16:17:55] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time
[16:18:07] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.142 second response time
[16:18:55] <MirahezeRelay>	 <bluemoon0332> some servers pulled the new credentials due to them starting the puppet run when I changed stuff in private git but before I merged https://github.com/miraheze/puppet/pull/4058
[16:19:48] <MirahezeRelay>	 <bluemoon0332> well, actually it was more like "before puppet pulled the new puppet code"
[16:27:22] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03mirabeta-new-users [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1d1c46157d92
[16:27:23] <botifico>	 [02mw-config] 07redbluegreenhat 031d1c461 - Switch mirabeta to the new db users
[16:27:26] <botifico>	 [02mw-config] 07redbluegreenhat created branch 03mirabeta-new-users - 13https://github.com/miraheze/mw-config
[16:27:53] <botifico>	 [02mw-config] 07redbluegreenhat opened pull request 03#5720: Switch mirabeta to the new db users - 13https://github.com/miraheze/mw-config/pull/5720
[16:28:55] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[16:34:45] <botifico>	 [02mw-config] 07redbluegreenhat closed pull request 03#5720: Switch mirabeta to the new db users - 13https://github.com/miraheze/mw-config/pull/5720
[16:34:48] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03mirabeta-new-users
[16:34:51] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03mirabeta-new-users - 13https://github.com/miraheze/mw-config
[16:37:31] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03flagging [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/14117a0e8775...c0efd7e789cc
[16:37:33] <botifico>	 [02CreateWiki] 07Universal-Omega 03c0efd7e - Add trackChange
[16:37:34] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#606: Add support for request flagging - 13https://github.com/miraheze/CreateWiki/pull/606
[16:39:25] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/452cd9cc6902...566ccc897ac6
[16:39:26] <botifico>	 [02mw-config] 07redbluegreenhat 03566ccc8 - Switch to the new database users
[16:40:22] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[16:40:24] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[16:40:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:40:31] <MirahezeLSBot>	 !log [alex@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw154.wikitide.net
[16:40:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:41:27] <MirahezeRelay>	 <bluemoon0332> talking about bad timing...
[16:41:40] <MirahezeRelay>	 <bluemoon0332> I think puppet's sentient, it knows the worst possible time to run
[16:42:17] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[16:42:18] <MirahezeLSBot>	 !log [alex@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost
[16:42:50] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[16:42:58] <MirahezeLSBot>	 !log [alex@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw161.wikitide.net
[16:43:08] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[16:43:09] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[16:43:44] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[16:43:52] <MirahezeLSBot>	 !log [alex@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw162.wikitide.net
[16:44:08] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all
[16:44:35] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 27s
[16:44:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:44:49] <MirahezeRelay>	 <bluemoon0332> of course, if we refuse to deploy the changes with the new db user due to a failing db connection due to using the wrong user we have a bit of a chicken-and-egg problem
[16:44:59] <MirahezeRelay>	 <rhinosf1> --force
[16:45:05] <MirahezeRelay>	 <bluemoon0332> yeah, that's what I did
[16:45:18] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:45:28] <MirahezeRelay>	 <rhinosf1> That is the correct situation for --force
[16:45:53] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.79 ms
[16:47:44] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813167621
[16:47:47] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813167384
[16:47:50] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813168885
[16:47:51] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813172973
[16:47:52] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813169864
[16:47:54] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813173204
[16:47:54] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms
[16:47:55] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813174057
[16:47:57] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813175618
[16:48:00] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813175773
[16:48:01] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813173539
[16:48:02] <botifico>	 [02MatomoAnalytics] 07Universal-Omega reviewed pull request 03#137 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/137#discussion_r1813176018
[16:49:36] <MirahezeRelay>	 <bluemoon0332> I'll give it a while then drop the old users everywhere
[16:51:10] <MirahezeRelay>	 <bluemoon0332> so what I did wrong is trying to run puppet at the same time
[16:51:39] <MirahezeRelay>	 <bluemoon0332> what I should have done is just merge that PR, change private git, and let the puppet push both at the same time
[16:52:08] <MirahezeRelay>	 <bluemoon0332> instead of the mess I did
[16:52:14] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[16:54:03] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> The issue with --force also is that still attempts to connect to DB so if DB connection completely times out then it still can't be deployed also. We encountered that issue before and have to manually rsync. (different issue but reminded me)
[16:54:15] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms
[16:55:30] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> I will fix that
[17:00:27] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: CRITICAL - Host Unreachable (2604:2dc0:202:300::7c6)
[17:04:17] <MirahezeLSBot>	 !log [alex@puppet181] Dropped the old DB users on all dbs (DROP USER IF EXISTS mediawiki,cargouser,wikiadmin via salt)
[17:04:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:06:41] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.77 ms
[17:29:27] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b0c2be52624b...9b323fc5ac60
[17:29:30] <botifico>	 [02ssl] 07WikiTideSSLBot 039b323fc - Bot: Add SSL cert for balloonfightwiki.org
[17:39:26] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildtextindex.php --wiki=shinwikievangelionfrwiki (START)
[17:39:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:39:39] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildtextindex.php --wiki=shinwikievangelionfrwiki (END - exit=256)
[17:39:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:42:15] <MacFan4000>	 !log (shinwikievangelionfrwiki) ALTER TABLE /*_*/searchindex MODIFY si_title LONGTEXT NOT NULL
[17:42:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:42:25] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildtextindex.php --wiki=shinwikievangelionfrwiki (START)
[17:42:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:42:58] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildtextindex.php --wiki=shinwikievangelionfrwiki (END - exit=0)
[17:43:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:01:15] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:05:19] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.75 ms
[18:09:36] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:11:35] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[18:29:45] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:35:53] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms
[18:53:28] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:55:27] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.77 ms
[19:13:37] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[19:15:36] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.62 ms
[20:29:10] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[20:31:09] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.63 ms
[23:06:45] <botifico>	 [02MatomoAnalytics] 07lihaohong6 opened pull request 03#138: Update localization - 13https://github.com/miraheze/MatomoAnalytics/pull/138
[23:06:52] <botifico>	 [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request 03#138: Update localization - 13https://github.com/miraheze/MatomoAnalytics/pull/138#issuecomment-2433722031
[23:16:32] <botifico>	 [02MatomoAnalytics] 07OAuthority closed pull request 03#138: Update localization - 13https://github.com/miraheze/MatomoAnalytics/pull/138
[23:16:34] <botifico>	 [02MatomoAnalytics] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/59e86c2ce4c4...da8a248ff002
[23:16:37] <botifico>	 [02MatomoAnalytics] 07lihaohong6 03da8a248 - Update localization (#138)
[23:17:39] <botifico>	 [02MatomoAnalytics] 07OAuthority created branch 03revert-138-master - 13https://github.com/miraheze/MatomoAnalytics
[23:17:41] <botifico>	 [02MatomoAnalytics] 07OAuthority pushed 031 commit to 03revert-138-master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/a6ea30551bce
[23:17:42] <botifico>	 [02MatomoAnalytics] 07OAuthority 03a6ea305 - Revert "Update localization (#138)"
[23:17:43] <botifico>	 [02MatomoAnalytics] 07OAuthority opened pull request 03#139: Revert "Update localization" - 13https://github.com/miraheze/MatomoAnalytics/pull/139
[23:17:49] <botifico>	 [02MatomoAnalytics] 07OAuthority closed pull request 03#139: Revert "Update localization" - 13https://github.com/miraheze/MatomoAnalytics/pull/139
[23:17:52] <botifico>	 [02MatomoAnalytics] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/da8a248ff002...f10ace4c5133
[23:17:55] <botifico>	 [02MatomoAnalytics] 07OAuthority 03f10ace4 - Revert "Update localization (#138)" (#139)
[23:17:56] <botifico>	 [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request 03#139: Revert "Update localization" - 13https://github.com/miraheze/MatomoAnalytics/pull/139#issuecomment-2433758029
[23:21:04] <miraheze-github>	 miraheze/MatomoAnalytics - lihaohong6 the build passed.
[23:22:37] <miraheze-github>	 miraheze/MatomoAnalytics - OAuthority the build passed.
[23:22:43] <miraheze-github_>	 miraheze/MatomoAnalytics - OAuthority the build passed.
[23:38:28] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[23:40:26] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[23:44:44] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[23:48:47] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.62 ms