[00:07:31] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.61 ms [00:17:39] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms [00:29:50] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 31.67 ms [00:31:48] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.69 ms [00:39:59] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.61 ms [00:44:00] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.63 ms [01:01:05] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.42] [01:05:16] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/288b93ea63b3...562b659d4794 [01:05:19] [02ssl] 07MacFan4000 03562b659 - add tanuki.iml.menhera.org, sarasa.iml.menhera.org, and dreamscience.iml.menhera.org [01:29:06] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:13:30] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:15:28] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.67 ms [04:04:57] [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03graphs [+0/-0/±3] 13https://github.com/miraheze/MatomoAnalytics/compare/a7442e389e4b...2a6abdb356dc [04:05:00] [02MatomoAnalytics] 07AgentIsai 032a6abdb - Add page views per day [04:05:01] [02MatomoAnalytics] 07AgentIsai synchronize pull request 03#140: Add chart support - 13https://github.com/miraheze/MatomoAnalytics/pull/140 [04:09:04] miraheze/MatomoAnalytics - AgentIsai the build has errored. [04:21:39] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kodiak.wiki' expires in 15 day(s) (Mon 11 Nov 2024 04:12:08 AM GMT +0000). [04:22:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/562b659d4794...f1333be8dce0 [04:22:02] [02ssl] 07WikiTideSSLBot 03f1333be - Bot: Update SSL cert for kodiak.wiki [04:42:26] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:44:25] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.11 ms [04:50:18] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Fri 24 Jan 2025 03:23:25 AM GMT +0000. [05:01:49] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 28.11 ms [05:12:08] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.15 ms [05:16:28] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:20:35] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.11 ms [06:37:28] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:45:41] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.06 ms [07:29:34] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817705609 [07:29:35] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817706335 [07:29:36] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817706900 [07:29:36] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817706097 [07:29:38] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817706226 [07:29:38] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817707105 [07:29:42] [02MatomoAnalytics] 07BlankEclair reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817707369 [07:39:24] [02CreateWiki] 07BlankEclair reviewed pull request 03#579 commit - 13https://github.com/miraheze/CreateWiki/pull/579#discussion_r1817708529 [07:41:09] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#520 commit - 13https://github.com/miraheze/MirahezeMagic/pull/520#discussion_r1817708977 [07:41:10] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#520 commit - 13https://github.com/miraheze/MirahezeMagic/pull/520#discussion_r1817708931 [07:45:49] [02CreateWiki] 07BlankEclair reviewed pull request 03#606 commit - 13https://github.com/miraheze/CreateWiki/pull/606#discussion_r1817709216 [07:47:39] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#520 commit - 13https://github.com/miraheze/MirahezeMagic/pull/520#discussion_r1817709835 [07:50:27] [02MirahezeMagic] 07BlankEclair reviewed pull request 03#520 commit - 13https://github.com/miraheze/MirahezeMagic/pull/520#discussion_r1817710207 [07:52:39] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#520 commit - 13https://github.com/miraheze/MirahezeMagic/pull/520#discussion_r1817710520 [08:14:59] [02RequestSSL] 07BlankEclair pushed 032 commits to 03T12801 [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/compare/3b542ddb2a86^...d2c62b37171a [08:15:01] [02RequestSSL] 07BlankEclair 033b542dd - Explicitly require the reason in Special:RequestSSL [08:15:04] [02RequestSSL] 07BlankEclair 03d2c62b3 - T12801: Update duplicate request check to check the database name [08:15:06] [02RequestSSL] 07BlankEclair created branch 03T12801 - 13https://github.com/miraheze/RequestSSL [08:17:48] [02RequestSSL] 07BlankEclair opened pull request 03#57: T12801: Update duplicate request check to check the database name (+ one other thing) - 13https://github.com/miraheze/RequestSSL/pull/57 [08:17:54] [02RequestSSL] 07coderabbitai[bot] commented on pull request 03#57: T12801: Update duplicate request check to check the database name (+ one other thing) - 13https://github.com/miraheze/RequestSSL/pull/57#issuecomment-2439428551 [08:27:22] miraheze/RequestSSL - BlankEclair the build passed. [08:41:52] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:50:06] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.18 ms [09:26:10] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:30:14] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.18 ms [10:32:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [10:36:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.10 ms [11:46:13] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:50:17] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.09 ms [12:10:28] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 134.53 ms [12:12:30] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 134.35 ms [12:38:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:42:19] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.18 ms [13:05:08] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 134.53 ms [13:07:11] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 134.36 ms [13:25:28] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:27:27] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.40 ms [13:38:16] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 134.34 ms [13:42:23] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 134.35 ms [13:57:58] [02landing] 07amire80 opened pull request 03#151: Improve spelling and grammar in the "about" message - 13https://github.com/miraheze/landing/pull/151 [13:58:03] [02landing] 07coderabbitai[bot] commented on pull request 03#151: Improve spelling and grammar in the "about" message - 13https://github.com/miraheze/landing/pull/151#issuecomment-2439592367 [13:58:42] miraheze/landing - amire80 the build passed. [14:35:26] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:39:30] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.16 ms [15:03:41] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:09:50] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.55 ms [15:31:47] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:31:50] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e43de25f6000 [15:31:52] [02puppet] 07paladox 03e43de25 - wikitide-backup: remove caching the token for now [15:31:53] [02puppet] 07paladox opened pull request 03#4062: wikitide-backup: remove caching the token for now - 13https://github.com/miraheze/puppet/pull/4062 [15:31:57] [02puppet] 07coderabbitai[bot] commented on pull request 03#4062: wikitide-backup: remove caching the token for now - 13https://github.com/miraheze/puppet/pull/4062#issuecomment-2439622890 [15:32:30] [02puppet] 07paladox closed pull request 03#4062: wikitide-backup: remove caching the token for now - 13https://github.com/miraheze/puppet/pull/4062 [15:32:32] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a166c703b722...b3bcdb973b1c [15:32:34] [02puppet] 07paladox 03b3bcdb9 - wikitide-backup: remove caching the token for now (#4062) [15:32:37] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:32:38] [02puppet] 07paladox deleted branch 03paladox-patch-1 [15:50:06] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/dcb08fe0e3a9 [15:50:07] [02puppet] 07paladox 03dcb08fe - wikitide-backup: add except to pca_connection [15:50:09] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:50:10] [02puppet] 07paladox opened pull request 03#4063: wikitide-backup: add except to pca_connection - 13https://github.com/miraheze/puppet/pull/4063 [15:50:17] [02puppet] 07coderabbitai[bot] commented on pull request 03#4063: wikitide-backup: add except to pca_connection - 13https://github.com/miraheze/puppet/pull/4063#issuecomment-2439627831 [15:50:40] [02puppet] 07paladox closed pull request 03#4063: wikitide-backup: add except to pca_connection - 13https://github.com/miraheze/puppet/pull/4063 [15:50:41] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3bcdb973b1c...b89922596674 [15:50:43] [02puppet] 07paladox 03b899225 - wikitide-backup: add except to pca_connection (#4063) [15:50:45] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [15:50:47] [02puppet] 07paladox deleted branch 03paladox-patch-1 [16:12:49] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:21:00] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.14 ms [16:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:57:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:58:12] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:04:44] PROBLEM - swiftobject161 Disk Space on swiftobject161 is WARNING: DISK WARNING - free space: / 146874MiB (10% inode=86%); [17:10:40] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.12 ms [19:29:56] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:36:05] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.08 ms [20:21:19] [02MatomoAnalytics] 07AgentIsai reviewed pull request 03#140 commit - 13https://github.com/miraheze/MatomoAnalytics/pull/140#discussion_r1817923350 [20:26:12] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c7b26e5f6206...cd33c575fc47 [20:26:13] [02puppet] 07paladox 03cd33c57 - Temp [20:26:16] [02puppet] 07paladox synchronize pull request 03#4040: Temp - 13https://github.com/miraheze/puppet/pull/4040 [20:26:19] [02puppet] 07coderabbitai[bot] commented on pull request 03#4040: Temp - 13https://github.com/miraheze/puppet/pull/4040#issuecomment-2439727121 [20:29:29] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.11, 10.23, 4.58 [20:31:24] [02landing] 07OAuthority closed pull request 03#151: Improve spelling and grammar in the "about" message - 13https://github.com/miraheze/landing/pull/151 [20:31:24] [02landing] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/e5e0231fe5af...656e33145ff4 [20:31:25] [02landing] 07amire80 03656e331 - Improve spelling and grammar in the "about" message (#151) [20:31:29] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.68, 7.23, 4.17 [20:32:21] miraheze/landing - OAuthority the build passed. [20:40:53] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [21:35:43] [02puppet] 07paladox created branch 03revert-4062-paladox-patch-1 - 13https://github.com/miraheze/puppet [21:35:46] [02puppet] 07paladox pushed 031 commit to 03revert-4062-paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8497f94bee02 [21:35:49] [02puppet] 07paladox 038497f94 - Revert "wikitide-backup: remove caching the token for now (#4062)" [21:35:51] [02puppet] 07paladox opened pull request 03#4064: Revert "wikitide-backup: remove caching the token for now" - 13https://github.com/miraheze/puppet/pull/4064 [21:35:52] [02puppet] 07paladox closed pull request 03#4064: Revert "wikitide-backup: remove caching the token for now" - 13https://github.com/miraheze/puppet/pull/4064 [21:35:54] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b89922596674...6b1114fb0e6f [21:35:57] [02puppet] 07paladox 036b1114f - Revert "wikitide-backup: remove caching the token for now" (#4064) [21:35:58] [02puppet] 07coderabbitai[bot] commented on pull request 03#4064: Revert "wikitide-backup: remove caching the token for now" - 13https://github.com/miraheze/puppet/pull/4064#issuecomment-2439743193 [21:35:59] [02puppet] 07paladox deleted branch 03revert-4062-paladox-patch-1 - 13https://github.com/miraheze/puppet [21:36:02] [02puppet] 07paladox deleted branch 03revert-4062-paladox-patch-1 [21:36:04] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [21:36:21] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd33c575fc47...650c2039ee2d [21:36:23] [02puppet] 07paladox 03650c203 - Temp [21:36:25] [02puppet] 07paladox synchronize pull request 03#4040: Temp - 13https://github.com/miraheze/puppet/pull/4040 [21:44:17] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.15 ms [22:02:31] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:02:40] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [22:04:35] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 28.16 ms [23:06:06] [02landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/656e33145ff4...87ac18218798 [23:06:08] [02landing] 07AgentIsai 0387ac182 - Add missing comma [23:06:58] miraheze/landing - AgentIsai the build passed. [23:20:07] [02landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/87ac18218798...08118c48be53 [23:20:08] [02landing] 07AgentIsai 0308118c4 - Check JSON in i18n directory [23:21:56] [02landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/08118c48be53...de71145afa4f [23:21:58] [02landing] 07AgentIsai 03de71145 - Fix array [23:22:41] miraheze/landing - AgentIsai the build passed. [23:37:42] [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03graphs [+1/-0/±5] 13https://github.com/miraheze/MatomoAnalytics/compare/2a6abdb356dc...ee81b9a54187 [23:37:45] [02MatomoAnalytics] 07AgentIsai 03ee81b9a - Update [23:37:47] [02MatomoAnalytics] 07AgentIsai synchronize pull request 03#140: Add chart support - 13https://github.com/miraheze/MatomoAnalytics/pull/140 [23:41:37] miraheze/MatomoAnalytics - AgentIsai the build has errored. [23:56:52] [02MatomoAnalytics] 07AgentIsai pushed 031 commit to 03graphs [+0/-0/±4] 13https://github.com/miraheze/MatomoAnalytics/compare/ee81b9a54187...89cf21de51fe [23:56:55] [02MatomoAnalytics] 07AgentIsai 0389cf21d - Add info header [23:56:58] [02MatomoAnalytics] 07AgentIsai synchronize pull request 03#140: Add chart support - 13https://github.com/miraheze/MatomoAnalytics/pull/140