[00:01:03] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/8c7ceff0301a9e023aed24aa48b9c0ccff24bef2 [00:01:03] 02statichelp/03main 07WikiTideBot 038c7ceff Bot: Auto-update Tech namespace pages 2026-06-13 00:01:00 [00:19:45] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86106MiB (19.0% inode=100%); [00:32:28] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.11, 11.60, 8.67 [00:33:47] actually a very helpful reaction lmeow [00:47:17] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44596MiB (9.8% inode=100%); [00:47:21] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 5 backends are down. mw151 mw161 mw153 mw191 mw203 [00:48:07] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:48:10] ohh noo [00:48:18] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw151 mw172 mw181 mw201 [00:48:18] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw181 mw201 [00:48:52] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw161 mw193 [00:49:17] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 87173MiB (19.2% inode=100%); [00:49:22] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [00:50:07] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.067 second response time [00:50:18] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [00:50:18] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [00:50:52] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [01:02:18] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw151 mw181 [01:02:51] @Infrastructure Specialists It's db161 again [01:03:07] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [01:03:27] ugh [01:04:18] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [01:04:22] depooling [01:04:58] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all [01:05:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:18] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [01:05:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:12] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all [01:08:15] !log repool c2 [01:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:30] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 18s [01:08:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:28] [02mediawiki-repos] 07SomeMWDev created 03commentstreams-dont-disable-parser-cache (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/16a4f81849d3 [01:21:28] 02mediawiki-repos/03commentstreams-dont-disable-parser-cache 07SomeRandomDeveloper 0316a4f81 Add a patch for CommentStreams to avoid disabling the parser cache [01:21:39] [02mediawiki-repos] 07SomeMWDev opened pull request #148: Add a patch for CommentStreams to avoid disabling the parser cache (03main...03commentstreams-dont-disable-parser-cache) 13https://github.com/miraheze/mediawiki-repos/pull/148 [01:22:12] [02mediawiki-repos] 07SomeMWDev merged pull request #148: Add a patch for CommentStreams to avoid disabling the parser cache (03main...03commentstreams-dont-disable-parser-cache) 13https://github.com/miraheze/mediawiki-repos/pull/148 [01:22:12] [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/982c2d27e3ed5c469cd43ed610ca5197723ffc3b [01:22:12] 02mediawiki-repos/03main 07SomeRandomDeveloper 03982c2d2 Add a patch for CommentStreams to avoid disabling the parser cache (#148) [01:22:14] [02mediawiki-repos] 07SomeMWDev 04deleted 03commentstreams-dont-disable-parser-cache at 0316a4f81 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/16a4f81 [01:25:09] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'CommentStreams'} to test151 [01:25:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:07] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'CommentStreams'} to test151 [01:26:09] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'CommentStreams'} to test151 - SUCCESS in 1s [01:26:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:54] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.88, 2.41, 11.23 [01:29:32] omg this warning always jumpscares me when it appears while load is going down, because it looks like it's going up again 😭 [01:30:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e1f71b2a8d2b63364dff881344a79d5c1508c8d1 [01:30:18] 02ssl/03main 07WikiTideBot 03e1f71b2 Bot: Auto-update domain lists [01:31:07] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [01:32:54] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.11, 2.92, 9.42 [01:33:53] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to all [01:33:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:27] <.yowin> [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1515178252900106340/image.jpg?ex=6a2e0f36&is=6a2cbdb6&hm=93e708a41085fd5393db313c6d8433e24c22c8602bb1b24ad4ffe1947835fc53& [02:17:27] <.yowin> [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1515178253558616104/image.jpg?ex=6a2e0f36&is=6a2cbdb6&hm=ee058723ae0215f54abfa8fa824f319ef74dc8bbb8fa6c1a98a81f2a55366c0e& [02:17:27] <.yowin> [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1515178254124974202/image.jpg?ex=6a2e0f36&is=6a2cbdb6&hm=47578a70a5c257a0eb87a3a05889eb6cd72ee6cf4781361363695c77e4df47e7& [02:17:27] <.yowin> [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1515178254657392751/image.jpg?ex=6a2e0f36&is=6a2cbdb6&hm=ddb4ce8d849a70bf6470efba85beca0265fe6d92cc574bf7d6511ff32dad41b8& [03:04:01] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [03:17:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.39, 14.32, 11.55 [03:21:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.60, 15.98, 12.79 [03:29:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.45, 15.50, 14.04 [03:30:32] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/87be1fb68b3f3109a7c2c3016b0617019a1ea0bb [03:30:32] 02ssl/03main 07WikiTideBot 0387be1fb Bot: Auto-update domain lists [03:31:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.21, 16.01, 14.39 [03:37:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.01, 15.34, 14.81 [03:45:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.54, 13.86, 14.08 [03:47:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.50, 14.71, 14.39 [03:57:35] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 9.87, 12.04, 13.49 [08:21:47] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshLinks --wiki=fantendowiki (END - exit=256) [08:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:54] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 6.80, 13.32, 7.79 [08:26:54] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.67, 9.68, 7.13 [12:05:09] !log [wwr@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to all [12:05:10] !log [wwr@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to all - SUCCESS in 0s [12:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:40] patch doesn't seem to have been applied properly [12:58:07] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:ReplaceTextEligible --wiki=pinmasterwiki (END - exit=0) [12:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:59:46] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.98, 5.79, 4.07 [13:00:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.00, 6.03, 4.41 [13:01:46] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.55, 5.35, 4.12 [13:02:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.62, 5.56, 4.44 [13:30:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.47, 6.08, 5.03 [13:32:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.15, 6.04, 5.14 [16:22:21] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.66, 14.87, 12.83 [17:08:01] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.32, 15.20, 15.93 [17:31:51] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 9.51, 11.50, 13.32 [18:00:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e8eaefe61ded595498af242b6b62257d9557448c [18:00:19] 02ssl/03main 07WikiTideBot 03e8eaefe Bot: Auto-update domain lists [18:39:01] !log [void@db171] restart sql-backup service [18:39:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:40:01] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [18:57:22] @abaddriverlol btw [18:57:40] still has parser expiry of 0 [19:10:00] [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1515433071606763692/IMG-20260601-WA3041.jpg?ex=6a2efc87&is=6a2dab07&hm=c519983b9f19c5d926f732f8bb05dbba45a60fc4a22f4ff03d520bca04883e76& [19:10:00] [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1515433072143892611/photo_2026-06-09_01-45-07.jpg?ex=6a2efc87&is=6a2dab07&hm=bfd03f17f1da41bd05d48d0e51c31ba8f335c6cd72e6dd33989cd43bade467ec& [19:10:01] [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1515433072957591692/PXL_20260605_184613_854.jpg?ex=6a2efc87&is=6a2dab07&hm=2ebd3f55bce4da30bad58c2c90a41707853fdef29b2b00ee3496bebcdd9907fd& [19:10:01] [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1515433073552916673/PXL_20260605_060133_753.jpg?ex=6a2efc88&is=6a2dab08&hm=269fed84dbfad3d3da8cc9d7cc72a349f92ee2aa16227e59adcb5c9bf8d899b8&