[00:06:59] <botifico>	 [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491
[01:05:26] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.67 ms
[01:07:25] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.72 ms
[01:27:48] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.78 ms
[01:33:51] <botifico>	 [02CreateWiki] 07Universal-Omega closed pull request 03#604: Use constants in CreateWikiNotificationsManager - 13https://github.com/miraheze/CreateWiki/pull/604
[01:33:52] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.62 ms
[01:33:54] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ff4bd47c551a...84ba370983d5
[01:33:55] <botifico>	 [02CreateWiki] 07Universal-Omega 0384ba370 - Use constants in CreateWikiNotificationsManager (#604)
[01:33:56] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03const
[01:33:58] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03const - 13https://github.com/miraheze/CreateWiki
[01:34:39] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5718: Add default for CreateWikiPurposes - 13https://github.com/miraheze/mw-config/pull/5718
[01:34:41] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/3ac7e0d1c08a...4981a509ad81
[01:34:42] <botifico>	 [02mw-config] 07Universal-Omega 034981a50 - Add default for CreateWikiPurposes (#5718)
[01:34:44] <botifico>	 [02mw-config] 07Universal-Omega deleted branch 03purposes-default
[01:34:46] <botifico>	 [02mw-config] 07Universal-Omega deleted branch 03purposes-default - 13https://github.com/miraheze/mw-config
[01:35:35] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[01:44:08] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[01:53:17] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[01:53:18] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[01:53:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:53:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:04] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[02:02:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:31] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 27s
[02:02:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:19:37] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 31.67 ms
[02:21:36] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.67 ms
[02:43:26] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[02:45:25] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.64 ms
[02:55:54] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.84 ms
[02:57:53] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.02 ms
[03:33:54] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[03:40:02] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.83 ms
[03:47:01] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[03:54:12] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.89 ms
[04:00:18] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[04:22:24] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[04:26:27] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.75 ms
[04:40:39] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.70 ms
[04:44:35] <botifico>	 [02CreateWiki] 07Universal-Omega closed pull request 03#603: Allow moredetails to be reopened when edited or commented - 13https://github.com/miraheze/CreateWiki/pull/603
[04:44:38] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/84ba370983d5...1ab0bf11f539
[04:44:40] <botifico>	 [02CreateWiki] 07Universal-Omega 031ab0bf1 - Allow moredetails to be reopened when edited or commented (#603)
[04:44:41] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03moredetails-reopen - 13https://github.com/miraheze/CreateWiki
[04:44:42] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03moredetails-reopen
[04:45:03] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to all
[04:45:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:45:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 26s
[04:45:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:46:59] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[04:48:45] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms
[04:54:12] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[04:56:18] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 139.99 ms
[04:58:18] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.92 ms
[05:12:22] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 32.26 ms
[05:14:21] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.65 ms
[06:09:38] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.73 ms
[06:19:46] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[06:54:17] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 32.15 ms
[06:58:19] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[07:02:35] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.69 ms
[07:04:34] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[07:08:52] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.81 ms
[07:10:51] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms
[07:24:28] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[07:26:27] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.67 ms
[07:44:35] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[07:46:28] <botifico>	 [02puppet] 07redbluegreenhat created branch 03dbcopy-new-key - 13https://github.com/miraheze/puppet
[07:46:31] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03dbcopy-new-key [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fc6364164c91
[07:46:33] <botifico>	 [02puppet] 07redbluegreenhat 03fc63641 - Change SSH key for dbcopy & autodeploy private key to db*
[07:46:34] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms
[07:46:34] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#4049: Change SSH key for dbcopy & autodeploy private key to db* - 13https://github.com/miraheze/puppet/pull/4049
[07:47:08] <botifico>	 [02puppet] 07github-actions[bot] pushed 031 commit to 03dbcopy-new-key [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fc6364164c91...a8f0667e23eb
[07:47:09] <botifico>	 [02puppet] 07github-actions 03a8f0667 - CI: lint puppet code to standards
[07:47:11] <botifico>	 [02puppet] 07github-actions[bot] synchronize pull request 03#4049: Change SSH key for dbcopy & autodeploy private key to db* - 13https://github.com/miraheze/puppet/pull/4049
[07:48:52] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d0e6375e3834...aa313bb1be8b
[07:48:53] <botifico>	 [02puppet] 07redbluegreenhat 03aa313bb - Change SSH key for dbcopy & autodeploy private key to db* (#4049)
[07:48:54] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#4049: Change SSH key for dbcopy & autodeploy private key to db* - 13https://github.com/miraheze/puppet/pull/4049
[07:48:56] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03dbcopy-new-key - 13https://github.com/miraheze/puppet
[07:48:58] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03dbcopy-new-key
[07:54:26] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa313bb1be8b...19d3101d5d22
[07:54:29] <botifico>	 [02puppet] 07redbluegreenhat 0319d3101 - content -> source, fix syntax
[07:56:54] <icinga-miraheze>	 PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:58:54] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[08:33:24] <MirahezeLSBot>	 !log [alex@test151] Upgraded packages linux-image-6.1.0-26-amd64, and linux-image-amd64
[08:33:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:33:46] <MirahezeRelay>	 <bluemoon0332> !log turned off test151
[08:33:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:34:55] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.69 ms
[08:35:04] <MirahezeRelay>	 <bluemoon0332> downtimed test151 in Icinga
[08:41:04] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.10 ms
[08:53:14] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.74 ms
[08:55:13] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.12 ms
[09:44:43] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.67 ms
[09:46:42] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.74 ms
[09:51:02] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[09:57:11] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.64 ms
[10:05:48] <botifico>	 [02dns] 07redbluegreenhat pushed 031 commit to 03db172-private [+0/-0/±1] 13https://github.com/miraheze/dns/commit/2eb5ff351d7d
[10:05:50] <botifico>	 [02dns] 07redbluegreenhat 032eb5ff3 - Add db172-private.wikitide.net
[10:05:51] <botifico>	 [02dns] 07redbluegreenhat created branch 03db172-private - 13https://github.com/miraheze/dns
[10:06:08] <botifico>	 [02dns] 07redbluegreenhat opened pull request 03#635: Add db172-private.wikitide.net - 13https://github.com/miraheze/dns/pull/635
[10:07:18] <miraheze-github>	 miraheze/dns - redbluegreenhat the build passed.
[10:07:28] <botifico>	 [02dns] 07redbluegreenhat closed pull request 03#635: Add db172-private.wikitide.net - 13https://github.com/miraheze/dns/pull/635
[10:07:31] <botifico>	 [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5f6981415858...5275b4ad482f
[10:07:32] <botifico>	 [02dns] 07redbluegreenhat 035275b4a - Add db172-private.wikitide.net (#635)
[10:07:34] <botifico>	 [02dns] 07redbluegreenhat deleted branch 03db172-private
[10:07:35] <botifico>	 [02dns] 07redbluegreenhat deleted branch 03db172-private - 13https://github.com/miraheze/dns
[10:08:49] <miraheze-github>	 miraheze/dns - redbluegreenhat the build passed.
[10:09:23] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.63 ms
[10:11:21] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[10:15:42] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:18:35] <botifico>	 [02mw-config] 07redbluegreenhat created branch 03db172-c5 - 13https://github.com/miraheze/mw-config
[10:18:38] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03db172-c5 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/7eb3b3fc95e9
[10:18:41] <botifico>	 [02mw-config] 07redbluegreenhat 037eb3b3f - Add db172 (c5)
[10:18:47] <botifico>	 [02mw-config] 07redbluegreenhat opened pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719
[10:19:42] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[10:22:19] <botifico>	 [02mw-config] 07Universal-Omega commented on pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719#issuecomment-2423744650
[10:23:02] <botifico>	 [02mw-config] 07Universal-Omega edited a comment on pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719#issuecomment-2423744650
[10:23:50] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.35 ms
[10:32:05] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.62 ms
[10:33:44] <botifico>	 [02mw-config] 07redbluegreenhat commented on pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719#issuecomment-2423749162
[10:36:26] <icinga-miraheze>	 PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100%
[10:38:26] <icinga-miraheze>	 RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 31.71 ms
[10:42:23] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.86 ms
[10:54:36] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:56:35] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms
[11:00:56] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[11:03:19] <botifico>	 [02mw-config] 07Universal-Omega commented on pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719#issuecomment-2423762187
[11:05:04] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.34 ms
[11:22:40] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 139.96 ms
[11:26:44] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.97 ms
[11:30:55] <icinga-miraheze>	 PROBLEM - Host test151 is DOWN: CRITICAL - Host Unreachable (10.0.15.118)
[11:38:13] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.73 ms
[11:48:22] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms
[12:20:09] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.71 ms
[12:22:08] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.72 ms
[12:28:21] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.71 ms
[12:34:26] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.78 ms
[12:38:46] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[12:40:45] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.80 ms
[13:13:21] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[13:17:19] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.75 ms
[13:35:59] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03db172-c5 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/7eb3b3fc95e9...3b3be4e958fa
[13:36:02] <botifico>	 [02mw-config] 07redbluegreenhat 033b3be4e - Add db172
[13:36:05] <botifico>	 [02mw-config] 07redbluegreenhat synchronize pull request 03#5719: Add db172 (c5) - 13https://github.com/miraheze/mw-config/pull/5719
[13:36:14] <botifico>	 [02mw-config] 07redbluegreenhat edited pull request 03#5719: Add db172 - 13https://github.com/miraheze/mw-config/pull/5719
[13:36:32] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.72 ms
[13:36:55] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[13:41:43] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03db172-c5 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/3b3be4e958fa...bddd11d14ea0
[13:41:45] <botifico>	 [02mw-config] 07redbluegreenhat 03bddd11d - Add db172
[13:41:47] <botifico>	 [02mw-config] 07redbluegreenhat synchronize pull request 03#5719: Add db172 - 13https://github.com/miraheze/mw-config/pull/5719
[13:42:41] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms
[13:42:43] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[14:30:25] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.73 ms
[14:31:10] <botifico>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[14:31:12] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6df36637d0f7
[14:31:15] <botifico>	 [02puppet] 07paladox 036df3663 - db: increase buffer pool to 38G
[14:31:17] <botifico>	 [02puppet] 07paladox opened pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:32:24] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 32.12 ms
[14:32:34] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6df36637d0f7...8168724cb482
[14:32:35] <botifico>	 [02puppet] 07paladox 038168724 - Update db151.yaml
[14:32:36] <botifico>	 [02puppet] 07paladox synchronize pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:32:44] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8168724cb482...6c964471c192
[14:32:45] <botifico>	 [02puppet] 07paladox 036c96447 - Update db161.yaml
[14:32:46] <botifico>	 [02puppet] 07paladox synchronize pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:32:53] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6c964471c192...65e3df835e81
[14:32:55] <botifico>	 [02puppet] 07paladox 0365e3df8 - Update db171.yaml
[14:32:57] <botifico>	 [02puppet] 07paladox synchronize pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:33:06] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/65e3df835e81...d2de76417cea
[14:33:08] <botifico>	 [02puppet] 07paladox 03d2de764 - Update db181.yaml
[14:33:10] <botifico>	 [02puppet] 07paladox synchronize pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:36:48] <MirahezeRelay>	 <paladox> !log db: set buffer pool to 38g
[14:36:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:36:59] <botifico>	 [02puppet] 07paladox closed pull request 03#4050: db: increase buffer pool to 38G - 13https://github.com/miraheze/puppet/pull/4050
[14:37:02] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/19d3101d5d22...71614365cf4c
[14:37:05] <botifico>	 [02puppet] 07paladox 037161436 - db: increase buffer pool to 38G (#4050)
[14:37:08] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1
[14:37:11] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[14:39:58] <botifico>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[14:39:59] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/28e2aa349d81
[14:40:02] <botifico>	 [02puppet] 07paladox 0328e2aa3 - mariadb: reduce innodb_io_capacity
[14:40:05] <botifico>	 [02puppet] 07paladox opened pull request 03#4051: mariadb: reduce innodb_io_capacity - 13https://github.com/miraheze/puppet/pull/4051
[14:40:36] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.72 ms
[14:44:21] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71614365cf4c...a3333571cb18
[14:44:23] <botifico>	 [02puppet] 07paladox 03a333357 - mariadb: reduce innodb_io_capacity (#4051)
[14:44:25] <botifico>	 [02puppet] 07paladox closed pull request 03#4051: mariadb: reduce innodb_io_capacity - 13https://github.com/miraheze/puppet/pull/4051
[14:44:28] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[14:44:30] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1
[14:48:43] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.77 ms
[15:08:38] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/101b7cc00674
[15:08:39] <botifico>	 [02puppet] 07paladox 03101b7cc - mariadb: increase max_connections to 1k
[15:08:40] <botifico>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[15:08:42] <botifico>	 [02puppet] 07paladox opened pull request 03#4052: mariadb: increase max_connections to 1k - 13https://github.com/miraheze/puppet/pull/4052
[15:08:52] <botifico>	 [02puppet] 07paladox closed pull request 03#4052: mariadb: increase max_connections to 1k - 13https://github.com/miraheze/puppet/pull/4052
[15:08:53] <botifico>	 [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a3333571cb18...34a0b4e3e691
[15:08:54] <botifico>	 [02puppet] 07paladox 0334a0b4e - mariadb: increase max_connections to 1k (#4052)
[15:08:57] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet
[15:08:59] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-1
[15:19:34] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[15:24:21] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.71 ms
[15:24:51] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 139.90 ms
[15:26:24] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms
[15:26:50] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.96 ms
[15:33:33] <botifico>	 [02mw-config] 07Universal-Omega commented on pull request 03#5719: Add db172 - 13https://github.com/miraheze/mw-config/pull/5719#issuecomment-2423974592
[15:36:04] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 139.96 ms
[15:38:05] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.97 ms
[15:42:17] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 139.93 ms
[15:44:17] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.95 ms
[15:48:27] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[16:07:15] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 31.69 ms
[16:09:12] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.75 ms
[16:17:24] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[16:23:25] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.96 ms
[16:36:38] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.77 ms
[16:40:34] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.67 ms
[17:28:09] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.65 ms
[17:36:14] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms
[17:40:31] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.73 ms
[17:44:32] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms
[18:22:38] <botifico>	 [02mw-config] 07redbluegreenhat closed pull request 03#5719: Add db172 - 13https://github.com/miraheze/mw-config/pull/5719
[18:22:41] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/4981a509ad81...f1965041e28e
[18:22:43] <botifico>	 [02mw-config] 07redbluegreenhat 03f196504 - Add db172 (#5719)
[18:22:45] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03db172-c5 - 13https://github.com/miraheze/mw-config
[18:22:46] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03db172-c5
[18:23:14] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[18:23:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:23:38] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[18:23:41] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s
[18:23:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:28:07] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.71 ms
[18:29:56] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[18:29:57] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[18:29:58] <icinga-miraheze>	 RECOVERY - Host test151 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms
[18:30:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:30:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:32:37] <MirahezeRelay>	 <bluemoon0332> !log testglobal: UPDATE cw_wikis SET wiki_dbcluster="c1" WHERE wiki_dbcluster="c2";
[18:32:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:34:08] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.64 ms
[18:40:34] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CreateWiki/maintenance/generateMissingCache.php --wiki=metawikibeta (END - exit=0)
[18:40:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:44:58] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[18:44:59] <MirahezeLSBot>	 !log [alex@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost
[18:45:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:45:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:45:29] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.103 second response time
[18:45:36] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151
[18:45:37] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s
[18:45:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:45:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:49:05] <MirahezeRelay>	 <bluemoon0332> [1/4] ```
[18:49:05] <MirahezeRelay>	 <bluemoon0332> [2/4] > strpos( wfHostname(), 'test')
[18:49:05] <MirahezeRelay>	 <bluemoon0332> [3/4] = 0```
[18:49:06] <MirahezeRelay>	 <bluemoon0332> [4/4] _sigh_
[18:52:23] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f1965041e28e...263767af93fd
[18:52:26] <botifico>	 [02mw-config] 07redbluegreenhat 03263767a - Fix check for test*
[18:52:35] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151
[18:52:36] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s
[18:52:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:52:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:20] <MirahezeRelay>	 <bluemoon0332> `(Cannot access the database: Cannot access the database: Unknown database 'commonswiki' (db172))` ????
[18:53:21] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[18:53:25] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.379 second response time
[18:53:31] <MirahezeRelay>	 <bluemoon0332> why is mirabeta attempting to access a prod database exactly?
[18:54:58] <MirahezeRelay>	 <bluemoon0332> https://github.com/miraheze/mw-config/blob/f1965041e28e5b49b8511e5da8404d81ec2a5ab8/GlobalSettings.php#L497C1-L529C2 👀
[19:00:36] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/263767af93fd...21d050466dab
[19:00:39] <botifico>	 [02mw-config] 07redbluegreenhat 0321d0504 - Always disable Miraheze Commons on mirabeta
[19:00:45] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151
[19:00:46] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s
[19:00:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:01:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:01:33] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build has errored.
[19:02:17] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[19:02:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:30] <MirahezeLSBot>	 !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw172.wikitide.net
[19:02:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:38] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/21d050466dab...d43108e5151d
[19:02:40] <botifico>	 [02mw-config] 07redbluegreenhat 03d43108e - Fix
[19:02:53] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:02:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:59] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True, 'force': True} to test151
[19:03:00] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'pull': 'config', 'config': True, 'force': True} to test151 - SUCCESS in 0s
[19:03:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:03:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:03:15] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s
[19:03:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:03:30] <MirahezeRelay>	 <bluemoon0332> alright, mirabeta is back up
[19:03:34] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[19:05:37] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> I guess there should be a commoner's
[19:05:52] <MirahezeRelay>	 <bluemoon0332> probably
[19:06:00] <MirahezeRelay>	 <rhinosf1> Also well done
[19:06:13] <MirahezeRelay>	 <rhinosf1> Now to change the db password and firewall it off
[19:06:18] <MirahezeRelay>	 <bluemoon0332> yep
[19:06:21] <MirahezeRelay>	 <rhinosf1> We should rotate the prod MediaWiki user too
[19:06:25] <MirahezeRelay>	 <bluemoon0332> the hard part is done at least
[19:06:28] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync]
[19:06:31] <MirahezeRelay>	 <rhinosf1> So it's fully separated
[19:06:57] <MirahezeRelay>	 <rhinosf1> For changing prod db password, create a 2024 user and then change config to with to that account
[19:07:05] <MirahezeRelay>	 <rhinosf1> To avoid downtime
[19:07:11] <MirahezeRelay>	 <rhinosf1> That's how wikimedia do it
[19:34:27] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:33:08] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.67 ms
[20:37:09] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.67 ms
[21:35:35] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.67 ms
[21:39:40] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms
[22:04:04] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.63 ms
[22:08:05] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.60 ms
[23:11:19] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.61 ms
[23:13:16] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms
[23:44:34] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.65 ms
[23:48:35] <icinga-miraheze>	 RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.65 ms