[00:01:12] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/9ef4bdb046e646eb10318e598341d42b990755a1 [00:01:12] 02statichelp/03main 07WikiTideBot 039ef4bdb Bot: Auto-update Tech namespace pages 2025-06-23 00:01:10 [00:06:08] [02ssl] 07asko1 opened pull request #864: T13892: Redirect handhelds.miraheze.org to handhelds.wiki (07miraheze:03main...07asko1:03T13892) 13https://github.com/miraheze/ssl/pull/864 [00:06:09] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57891MiB (12% inode=99%); [00:06:17] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 58513MiB (12% inode=99%); [00:06:39] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 58002MiB (12% inode=99%); [00:09:13] [02ssl] 07coderabbitai[bot] commented on pull request #864: A new redirect entry was added to the `redirects.yaml` configuration. This entry redirects requests from `handhelds.miraheze.org` to `handhelds.wiki`, using the `miraheze-origin-cert` SSL certificate, Cloudflare as the certificate authority, and […] 13https://github.com/miraheze/ssl/pull/864#issuecomment-2994553738 [00:41:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03dns-packages 13https://github.com/miraheze/puppet/commit/400aceefcc10f764119aab04bfa327686009e91b [00:41:04] 02puppet/03dns-packages 07CosmicAlpha 03400acee Fix [00:46:07] [02puppet] 07Universal-Omega created 03order-packages (+1 new commit) 13https://github.com/miraheze/puppet/commit/c4319efe857a [00:46:07] 02puppet/03order-packages 07CosmicAlpha 03c4319ef Order packages alphabetically [00:46:11] [02puppet] 07Universal-Omega opened pull request #4423: Order packages alphabetically (03main...03order-packages) 13https://github.com/miraheze/puppet/pull/4423 [00:46:17] [02puppet] 07coderabbitai[bot] commented on pull request #4423:

[…] 13https://github.com/miraheze/puppet/pull/4423#issuecomment-2994591149 [00:52:09] [02mw-config] 07songnguxyz opened pull request #5997: cgwiki: Protection level fix (07miraheze:03main...07songnguxyz:03fixcgprotectedfr) 13https://github.com/miraheze/mw-config/pull/5997 [00:52:57] miraheze/mw-config - songnguxyz the build passed. [01:02:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/9f00695938e2adfb6f22d6ce40666cf998edf0b6 [01:03:38] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/45b5737913db70b0bfd3e27722b1533985b56460 [01:03:48] 02puppet/03order-packages 07CosmicAlpha 039f00695 Update [01:03:49] 02puppet/03order-packages 07CosmicAlpha 0345b5737 Update [01:20:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750638020000&orgId=1&to=1750641653353 [01:40:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750639220000&orgId=1&to=1750642820000 [01:42:44] !log MariaDB [mhglobal]> UPDATE mw_permissions SET perm_permissions = JSON_REMOVE( perm_permissions, JSON_UNQUOTE(JSON_SEARCH(perm_permissions, 'one', 'writeapi')) ) WHERE JSON_SEARCH(perm_permissions, 'one', 'writeapi') IS NOT NULL; [01:42:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750640900000&orgId=1&to=1750644533354 [02:19:23] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [02:23:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750641740000&orgId=1&to=1750645340000 [02:42:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750642940000&orgId=1&to=1750646573357 [02:52:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750643540000&orgId=1&to=1750647140000 [03:12:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750644740000&orgId=1&to=1750648373358 [03:15:28] [02ssl] 07Universal-Omega merged 07asko1's pull request #864: T13892: Redirect handhelds.miraheze.org to handhelds.wiki (07miraheze:03main...07asko1:03T13892) 13https://github.com/miraheze/ssl/pull/864 [03:15:31] [02ssl] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/52af5046cf004f9cc4549aba2663e9ab894811ec [03:15:31] 02ssl/03main 07Asko 0352af504 T13892: Redirect handhelds.miraheze.org to handhelds.wiki (#864)… [03:17:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750645040000&orgId=1&to=1750648640000 [04:08:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750648100000&orgId=1&to=1750651733352 [04:13:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750648340000&orgId=1&to=1750651940000 [04:35:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750649720000&orgId=1&to=1750653353352 [04:45:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750649720000&orgId=1&to=1750653920000 [05:10:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750651820000&orgId=1&to=1750655453353 [05:35:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750653200000&orgId=1&to=1750656800000 [07:20:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750659620000&orgId=1&to=1750663253352 [07:30:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750660160000&orgId=1&to=1750663760000 [07:40:04] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23446MiB (10% inode=95%); [07:42:04] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 28203MiB (12% inode=95%); [08:01:24] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [10:04:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750669460000&orgId=1&to=1750673093357 [10:44:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750671800000&orgId=1&to=1750675400000 [12:11:01] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/d6bc7b204f8573dcbe0f63b6fb1a895366609ca1 [12:11:01] 02DiscordNotifications/03main 07translatewiki.net 03d6bc7b2 Localisation updates from https://translatewiki.net. [12:11:01] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/b53822f90c79aae60ff79b864804ceaf8ee806ba [12:11:01] 02CreateWiki/03main 07translatewiki.net 03b53822f Localisation updates from https://translatewiki.net. [12:11:03] [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/982caa13bb92237fe5fd2c877a586dac2ef5ea13 [12:11:05] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/0b5de8996595c0c3eb36709f84db270b47a91b52 [12:11:07] 02ImportDump/03main 07translatewiki.net 03982caa1 Localisation updates from https://translatewiki.net. [12:11:09] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/8874e6b3ffeff779893f797d263258f700c8066f [12:11:10] 02ManageWiki/03main 07translatewiki.net 038874e6b Localisation updates from https://translatewiki.net. [12:11:12] [02RequestSSL] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestSSL/commit/24bb80153fd0b3590d2cfa12ad0500788f1bdab1 [12:11:14] 02MatomoAnalytics/03main 07translatewiki.net 030b5de89 Localisation updates from https://translatewiki.net. [12:11:16] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/c0566802f88452b256e7ed2a11bbbcbe21f960c5 [12:11:18] 02MirahezeMagic/03main 07translatewiki.net 03c056680 Localisation updates from https://translatewiki.net. [12:11:19] 02RequestSSL/03main 07translatewiki.net 0324bb801 Localisation updates from https://translatewiki.net. [12:16:57] miraheze/ManageWiki - translatewiki the build passed. [12:18:59] miraheze/MatomoAnalytics - translatewiki the build passed. [12:23:54] miraheze/MirahezeMagic - translatewiki the build passed. [12:29:50] miraheze/CreateWiki - translatewiki the build passed. [12:32:06] miraheze/RequestSSL - translatewiki the build passed. [12:32:10] miraheze/ImportDump - translatewiki the build passed. [12:33:45] miraheze/DiscordNotifications - translatewiki the build passed. [13:39:45] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49842MiB (10% inode=99%); [13:50:39] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49888MiB (10% inode=99%); [13:56:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750683380000&orgId=1&to=1750687013354 [14:01:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750683560000&orgId=1&to=1750687160000 [14:18:34] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [14:20:33] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -6.84261322e-05 secs [14:29:31] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49928MiB (10% inode=99%); [15:44:02] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/ff57ea919429266986e1dae0ba8f293e70ad1f95 [15:44:03] 02puppet/03order-packages 07CosmicAlpha 03ff57ea9 Update [15:44:49] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/d7be017ff32ac789c0210272392dce7985eaa8fa [15:44:49] 02puppet/03order-packages 07CosmicAlpha 03d7be017 - [15:49:03] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/d9554f5ca9627bfd8b3a3793ef3d5af43f2859da [15:49:03] 02puppet/03order-packages 07CosmicAlpha 03d9554f5 Update [15:51:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/9e1d92dd2ca80e9809ffbcc8dc4dcec86a97cf37 [15:51:41] 02puppet/03order-packages 07CosmicAlpha 039e1d92d - [16:00:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/70081222ae6078a9503a70d34d53b8170ef4742b [16:00:36] 02puppet/03order-packages 07CosmicAlpha 037008122 Update [16:20:22] I need my wiki to get set and ready pretty urgently, can someone handle [[phorge:T13889]]? [16:20:23]

[16:21:14] {{phorge:T13889}} [16:21:17]

[1/2] Open: Import request for 100bangaiwiki [16:21:18]

[2/2] [16:21:33] wiki-bot can do that... interesting [17:01:06] might get more attention in #tech, also worth pinging someone atp [17:08:04] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23360MiB (10% inode=95%); [17:10:04] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 28002MiB (12% inode=95%); [17:13:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750695200000&orgId=1&to=1750698833357 [17:28:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750696040000&orgId=1&to=1750699640000 [17:32:53] [02mw-config] 07RhinosF1 created 03RhinosF1-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/2cf6e799c177 [17:32:53] 02mw-config/03RhinosF1-patch-2 07RhinosF1 032cf6e79 depool c1 for maint [17:32:58] [02mw-config] 07RhinosF1 opened pull request #5998: depool c1 for maint (03main...03RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/5998 [17:33:31] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03RhinosF1-patch-2 13https://github.com/miraheze/mw-config/commit/3dcb93ebeb0a592d3b14bbb60de57a0b81dc2886 [17:33:31] 02mw-config/03RhinosF1-patch-2 07github-actions 033dcb93e CI: lint code to MediaWiki standards… [17:33:59] miraheze/mw-config - RhinosF1 the build passed. [17:54:56] PROBLEM - arceniawiki.doughhearts.com - Cloudflare on sslhost is WARNING: WARNING - Certificate 'arceniawiki.doughhearts.com' expires in 28 day(s) (Tue 22 Jul 2025 05:43:16 PM GMT +0000). [18:21:25] [02mw-config] 07RhinosF1 merged pull request #5998: depool c1 for maint (03main...03RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/5998 [18:21:27] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6cf9920e10fb0ae79921aa080f990a178d1c3868 [18:21:28] 02mw-config/03main 07RhinosF1 036cf9920 depool c1 for maint (#5998)… [18:21:30] [02mw-config] 07RhinosF1 04deleted 03RhinosF1-patch-2 at 033dcb93e 13https://api.github.com/repos/miraheze/mw-config/commit/3dcb93e [18:21:40] !log [rhinos@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [18:21:41] !log [rhinos@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [18:21:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:17] miraheze/mw-config - RhinosF1 the build passed. [18:30:19] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:30:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:25] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:30:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:44] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s [18:30:46] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [18:30:46] !log shutdown all vms on cloud15 [18:30:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:09] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2fe18e73f21b [18:33:09] 02puppet/03paladox-patch-11 07paladox 032fe18e7 Temporarily depool mem151 [18:33:13] [02puppet] 07paladox opened pull request #4424: Temporarily depool mem151 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4424 [18:33:18] [02puppet] 07paladox merged pull request #4424: Temporarily depool mem151 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4424 [18:33:20] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7b22a833148fcdad323c4aef55c9ffae7597ad5a [18:33:22] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 032fe18e7 13https://api.github.com/repos/miraheze/puppet/commit/2fe18e7 [18:33:23] 02puppet/03main 07paladox 037b22a83 Temporarily depool mem151 (#4424) [18:33:25] [02puppet] 07coderabbitai[bot] commented on pull request #4424:

[…] 13https://github.com/miraheze/puppet/pull/4424#issuecomment-2997543444 [19:13:17] guys miraheze no worky [19:13:30] make it worky please [19:14:27] "Bad gateway" [19:14:37] "The web server reported a bad gateway error." [19:20:38] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:21:52] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [19:22:07] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4038 bytes in 5.683 second response time [19:25:06] PROBLEM - wiki.jill-jimmy.com - Cloudflare on sslhost is WARNING: WARNING - Certificate 'wiki.jill-jimmy.com' expires in 28 day(s) (Tue 22 Jul 2025 07:08:02 PM GMT +0000). [19:26:15] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:54:46] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4038 bytes in 9.983 second response time [19:55:13] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4093 bytes in 3.402 second response time [19:55:23] !log starting vms on cloud15 [19:56:01] RECOVERY - Host os151 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [19:56:01] RECOVERY - os151 SSH on os151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0) [19:56:04] RECOVERY - os151 conntrack_table_size on os151 is OK: OK: nf_conntrack is 0 % full [19:56:08] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/fd4d9f71b039d4ce25fdc5998d42f039c6efd8e5 [19:56:08] 02puppet/03main 07paladox 03fd4d9f7 Revert "Temporarily depool mem151 (#4424)"… [19:56:13] RECOVERY - Host mwtask151 is UP: PING OK - Packet loss = 0%, RTA = 76.10 ms [19:56:18] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [19:56:20] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0) [19:56:22] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4038 bytes in 2.809 second response time [19:56:23] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4038 bytes in 0.923 second response time [19:56:38] RECOVERY - Host cloud15 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [19:56:40] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:56:42] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet last ran 1 hour ago [19:57:07] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:57:12] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet last ran 1 hour ago [19:57:24] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750701710000&orgId=1&to=1750708644109[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750701710000&orgId=1&to=1750708580000[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700210000&orgId=1&to=1750708610000[Grafana] FIRING: A [19:57:24] y high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750708644109[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750708644109[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750699910000&orgId=1&to=1750708610000[Grafana] FIRING: The medi [19:57:24] ueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750704980000&orgId=1&to=1750708644109 [19:57:39] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/6d882e133673 [19:57:39] 02mw-config/03paladox-patch-4 07paladox 036d882e1 Repool C1 cluster [19:57:39] RECOVERY - ping on os151 is OK: PING OK - Packet loss = 0%, RTA = 0.32 ms [19:57:42] [02mw-config] 07paladox opened pull request #5999: Repool C1 cluster (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5999 [19:57:52] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 1.95, 0.76, 0.28 [19:57:57] RECOVERY - os151 PowerDNS Recursor on os151 is OK: DNS OK: 0.108 seconds response time. os151.fsslc.wtnet returns 10.0.15.111 [19:58:32] miraheze/mw-config - paladox the build passed. [19:58:40] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:59:02] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.415 second response time [19:59:03] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0dfb18dc76236c4294292dae9c55530f76ac1970 [19:59:03] 02mw-config/03main 07RhinosF1 030dfb18d repool c1 [19:59:07] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:59:16] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:59:44] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [19:59:55] miraheze/mw-config - RhinosF1 the build passed. [19:59:57] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 275 bytes in 0.036 second response time [20:00:04] !log [rhinos@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [20:00:06] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [20:00:07] !log [rhinos@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 2s [20:00:30] [02mw-config] 07paladox closed pull request #5999: Repool C1 cluster (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5999 [20:00:34] [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 036d882e1 13https://api.github.com/repos/miraheze/mw-config/commit/6d882e1 [20:00:45] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [20:00:58] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [20:01:07] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [20:01:11] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (3 critical updates). [20:01:51] RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:02:23] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:24] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750701710000&orgId=1&to=1750708910000[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750705070000&orgId=1&to=1750708670000[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750705010000&orgId=1&to=1750708610000[Grafana] FIRING: An unusually high number of threats [20:02:24] reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750708944111[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750708944111[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750704980000&orgId=1&to=1750708730000 [20:03:19] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/2c9cbf59ce9bff7c5c119dac396414ed1ec137a1 [20:03:19] 02mw-config/03main 07RhinosF1 032c9cbf5 Update Sitenotice.php [20:03:30] !log [rhinos@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [20:03:31] !log [rhinos@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [20:03:32] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:03:55] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [20:04:08] miraheze/mw-config - RhinosF1 the build passed. [20:05:13] RECOVERY - mw151.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [20:06:35] RECOVERY - mw152.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [20:09:55] !log puppet181: sudo upgrade-packages -y [20:13:07] !log [root@puppet181] Upgraded packages on mw152: xserver-common, xserver-xorg-core, and xvfb [20:13:24] !log [root@puppet181] Upgraded packages on mw151: xserver-common, xserver-xorg-core, and xvfb [20:13:36] !log [root@puppet181] Upgraded packages on mw153: xserver-common, xserver-xorg-core, and xvfb [20:13:55] !log [root@puppet181] Upgraded packages on mw161: xserver-common, xserver-xorg-core, and xvfb [20:14:10] !log [root@puppet181] Upgraded packages on mw172: xserver-common, xserver-xorg-core, and xvfb [20:14:24] !log [root@puppet181] Upgraded packages on mw162: xserver-common, xserver-xorg-core, and xvfb [20:14:37] !log [root@puppet181] Upgraded packages on mwtask151: xserver-common, xserver-xorg-core, and xvfb [20:14:39] RECOVERY - mw172 APT on mw172 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:14:41] RECOVERY - mw162 APT on mw162 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:14:51] !log [root@puppet181] Upgraded packages on mw171: xserver-common, xserver-xorg-core, and xvfb [20:14:53] RECOVERY - mw161 APT on mw161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:15:06] !log [root@puppet181] Upgraded packages on mw163: xserver-common, xserver-xorg-core, and xvfb [20:15:11] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:15:21] !log [root@puppet181] Upgraded packages on mw173: xserver-common, xserver-xorg-core, and xvfb [20:15:37] !log [root@puppet181] Upgraded packages on mw201: xserver-common, xserver-xorg-core, and xvfb [20:15:44] RECOVERY - mw163 APT on mw163 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:15:51] !log [root@puppet181] Upgraded packages on mw192: xserver-common, xserver-xorg-core, and xvfb [20:16:02] RECOVERY - mw192 APT on mw192 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:16:07] !log [root@puppet181] Upgraded packages on mw191: xserver-common, xserver-xorg-core, and xvfb [20:16:21] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:16:22] !log [root@puppet181] Upgraded packages on mw193: xserver-common, xserver-xorg-core, and xvfb [20:16:37] !log [root@puppet181] Upgraded packages on mw181: xserver-common, xserver-xorg-core, and xvfb [20:16:52] !log [root@puppet181] Upgraded packages on mw203: xserver-common, xserver-xorg-core, and xvfb [20:17:08] !log [root@puppet181] Upgraded packages on mw182: xserver-common, xserver-xorg-core, and xvfb [20:17:13] RECOVERY - mw173 APT on mw173 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:17:18] RECOVERY - mw201 APT on mw201 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:17:24] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750705310000&orgId=1&to=1750708910000[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750706000000&orgId=1&to=1750709844112[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://gr [20:17:24] tide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750709844112[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750709844112 [20:17:24] !log [root@puppet181] Upgraded packages on mw202: xserver-common, xserver-xorg-core, and xvfb [20:17:27] RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:17:33] RECOVERY - mw181 APT on mw181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:17:40] RECOVERY - mw182 APT on mw182 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:17:41] !log [root@puppet181] Upgraded packages on mw183: xserver-common, xserver-xorg-core, and xvfb [20:17:52] RECOVERY - mw191 APT on mw191 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:18:03] RECOVERY - mw203 APT on mw203 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:18:05] RECOVERY - mw193 APT on mw193 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:18:18] !log [root@puppet181] Upgraded packages on mwtask161: xserver-common, xserver-xorg-core, and xvfb [20:18:24] RECOVERY - mw183 APT on mw183 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:18:37] RECOVERY - mw202 APT on mw202 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:18:42] !log [void@cloud15] Cleared IPMI server event log [20:19:04] !log [root@puppet181] Upgraded packages on mwtask171: xserver-common, xserver-xorg-core, and xvfb [20:19:10] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:19:26] RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK [20:19:27] !log [root@puppet181] Upgraded packages on mwtask181: xserver-common, xserver-xorg-core, and xvfb [20:19:52] !log [root@puppet181] Upgraded packages on test151: xserver-common, xserver-xorg-core, and xvfb [20:19:53] RECOVERY - mw153.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [20:20:24] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:20:45] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [20:21:58] [02mw-config] 07SomeMWDev left a file comment in pull request #5997 038831f7b: This should probably remain `editcgprotected`, otherwise you'll have to rename the right as well 13https://github.com/miraheze/mw-config/pull/5997#discussion_r2162460847 [20:22:03] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:22:21] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:22:24] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750706540000&orgId=1&to=1750710140000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750710144113[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1 [20:22:24] 0144113 [20:22:40] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:23:52] ^ that was me breaking puppet it's fixed running again now. [20:24:21] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:48] all you gotta do to fix everything is do a dance for the sudo gods [20:24:53] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:24:59] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:28:26] from all my experience with tech i can confidently say that typing sudo pacman -S hyprland will solve all problems with everything everywhere [20:28:38] including non-tech related ones [20:31:10] !log renamed master to main in private git on puppet181 [20:31:15] !log created role/common/mediawiki_beta.yaml in private puppet [20:31:47] is it dead? [20:32:00] @rhinosf1 is logbot dead? [20:32:24] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750706840000&orgId=1&to=1750710440000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750710660000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750700460000&orgId=1&to=1750710660000 [20:32:37] no [20:32:42] !log test [20:32:54] it's online just not responding [21:20:09] [02CreateWiki] 07OAuthority left a file comment in pull request #605 03b5b4bcd: Don't forget the spaces for the array elements 😄 13https://github.com/miraheze/CreateWiki/pull/605#discussion_r2162552530 [21:28:53] [1/2] @originalauthority i don't think we need spaces here in this case, at least according to the wmf code conventions [21:28:54] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1386820332245221438/Screenshot_20250623_232635_Firefox.jpg?ex=685b1895&is=6859c715&hm=e3c31a15501ac555017b28e8e1f6aad2e923affb5e5767078cdf31373071bf89& [21:31:45] That gives the impression its only for numbers to me (95% probability I'm wrong). I've always done spaces though [21:31:48] i'm gonna check core [21:31:59] I suspect you're probably right though [21:32:22] oh you can't do codesearch for just `[ "` dump fuckin system [21:33:49] mediawiki core doesn't even follow the conventions smh [21:34:13] try `/[ "` [21:34:53] (wmf please implement https://phabricator.wikimedia.org/T381325) [21:35:11] you doth appear to be correct [21:45:07] tbh not sure why wmf decided to do it this way since it's just inconsistent [21:45:35] i guess you can distinguish between array accessing and array creation faster [21:49:37] PROBLEM - cloud15 IPMI Sensors on cloud15 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [22:06:50] [02puppet] 07Universal-Omega created 03trixie from 03main (+0 new commit) 13https://github.com/miraheze/puppet/compare/trixie [22:10:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03trixie 13https://github.com/miraheze/puppet/commit/526d4ed74847da28d3a65eaa1b586a5bac48bf0f [22:10:46] 02puppet/03trixie 07CosmicAlpha 03526d4ed Add initial support for debian trixie [22:10:58] [02puppet] 07Universal-Omega opened pull request #4425: Add initial support for debian trixie (03main...03trixie) 13https://github.com/miraheze/puppet/pull/4425 [22:11:26] [02puppet] 07coderabbitai[bot] commented on pull request #4425:

[…] 13https://github.com/miraheze/puppet/pull/4425#issuecomment-2998110482 [22:18:22] [02puppet] 07Universal-Omega merged pull request #4425: Add initial support for debian trixie (03main...03trixie) 13https://github.com/miraheze/puppet/pull/4425 [22:18:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2bb000a4bd368f80a5c47d9872cdffeba687e4ea [22:18:23] 02puppet/03main 07CosmicAlpha 032bb000a Add initial support for debian trixie [22:18:25] [02puppet] 07Universal-Omega 04deleted 03trixie at 03526d4ed 13https://api.github.com/repos/miraheze/puppet/commit/526d4ed [22:26:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03order-packages 13https://github.com/miraheze/puppet/commit/a49b41f0885c9ae806b239947aa2c268a5a50396 [22:26:04] 02puppet/03order-packages 07CosmicAlpha 03a49b41f Merge branch 'main' into order-packages [22:27:55] PROBLEM - wiki.tdrweb.top - Cloudflare on sslhost is WARNING: WARNING - Certificate 'wiki.tdrweb.top' expires in 28 day(s) (Tue 22 Jul 2025 10:14:15 PM GMT +0000). [22:30:56] spaces unless it's numbers is my personal convention lol [22:31:38] til that didn't work lol [22:57:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750715840000&orgId=1&to=1750719473351 [23:12:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750715840000&orgId=1&to=1750720340000 [23:44:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750718660000&orgId=1&to=1750722293351