[00:01:22] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/fd3d4668a26e79b16dedd52788deab3ee2f01787 [00:01:23] 02statichelp/03main 07WikiTideBot 03fd3d466 Bot: Auto-update Tech namespace pages 2026-08-25 00:01:19 [00:26:29] !log [petramagna@test151] starting deploy of {'folders': '1.46/extensions/ConfirmEdit'} to test151 [00:26:30] !log [petramagna@test151] finished deploy of {'folders': '1.46/extensions/ConfirmEdit'} to test151 - SUCCESS in 0s [00:26:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:16] [02mediawiki-repos] 07SomeMWDev created 03speedscope-prod (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/cccf4ec08db2 [00:40:16] 02mediawiki-repos/03speedscope-prod 07SomeRandomDeveloper 03cccf4ec Deploy Speedscope on 1.45/prod as well… [00:40:35] [02mediawiki-repos] 07SomeMWDev opened pull request #163: Deploy Speedscope on 1.45/prod as well (03main...03speedscope-prod) 13https://github.com/miraheze/mediawiki-repos/pull/163 [00:41:24] [02mediawiki-repos] 07SomeMWDev merged pull request #163: Deploy Speedscope on 1.45/prod as well (03main...03speedscope-prod) 13https://github.com/miraheze/mediawiki-repos/pull/163 [00:41:24] [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/e10f5b12573d2bcd4d61b4b624aea4598c9a9e69 [00:41:24] 02mediawiki-repos/03main 07SomeRandomDeveloper 03e10f5b1 Deploy Speedscope on 1.45/prod as well (#163)… [00:41:26] [02mediawiki-repos] 07SomeMWDev 04deleted 03speedscope-prod at 03cccf4ec 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/cccf4ec [00:45:33] !log [somerandomdeveloper@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Speedscope'} to all [00:45:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:37] !log [somerandomdeveloper@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Speedscope'} to all - SUCCESS in 63s [00:46:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:49] !log [somerandomdeveloper@mwtask181] starting deploy of {'l10n': True, 'extension_list': True, 'versions': '1.45'} to all [00:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:09] [02mw-config] 07SomeMWDev created 03speedscope-prod (+1 new commit) 13https://github.com/miraheze/mw-config/commit/1cf069365c51 [00:58:09] 02mw-config/03speedscope-prod 07SomeRandomDeveloper 031cf0693 Enable Speedscope extension on prod… [00:58:27] [02mw-config] 07SomeMWDev opened pull request #6508: Enable Speedscope extension on prod (03main...03speedscope-prod) 13https://github.com/miraheze/mw-config/pull/6508 [00:59:26] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03speedscope-prod 13https://github.com/miraheze/mw-config/commit/12be659269bf319f5565897fd0427a5c484e9df8 [00:59:28] 02mw-config/03speedscope-prod 07github-actions 0312be659 CI: lint code to MediaWiki standards… [00:59:56] miraheze/mw-config - SomeMWDev the build passed. [01:01:16] miraheze/mw-config - github-actions[bot] the build passed. [01:01:25] !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'extension_list': True, 'versions': '1.45'} to all - SUCCESS in 516s [01:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:51] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:ReplaceTextEligible --wiki=battlenationswiki (END - exit=33280) [01:31:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:59] Looks like this script runs two SQL queries per deleted page; I decided to cancel the script after ~20mins lol [03:03:04] PROBLEM - mon181 Check unit status of grafana-backup on mon181 is CRITICAL: CRITICAL: Status of the systemd unit grafana-backup [03:03:18] PROBLEM - db181 Check unit status of sql-backup on db181 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [03:22:08] [02puppet] 07Universal-Omega created 03network (+1 new commit) 13https://github.com/miraheze/puppet/commit/4b8aac5d8b21 [03:22:08] 02puppet/03network 07CosmicAlpha 034b8aac5 Use src_sets tather than raw firewall querying [03:23:36] [02puppet] 07Universal-Omega opened pull request #4930: Use src_sets tather than raw firewall querying (03main...03network) 13https://github.com/miraheze/puppet/pull/4930 [03:24:12] [02puppet] 07github-actions[bot] pushed 1 new commit to 03network 13https://github.com/miraheze/puppet/commit/696b8673081bbfbf3cb9d9407b260289494906f0 [03:24:13] 02puppet/03network 07github-actions 03696b867 CI: lint puppet code to standards… [03:28:39] [02puppet] 07Universal-Omega created 03irc-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/3b16ec3fc8cd [03:28:39] 02puppet/03irc-nftables 07CosmicAlpha 033b16ec3 irc: include nftables alongside ferm… [03:28:45] [02puppet] 07Universal-Omega opened pull request #4931: irc: include nftables alongside ferm (03network...03irc-nftables) 13https://github.com/miraheze/puppet/pull/4931 [03:30:09] [02puppet] 07Universal-Omega created 03cache-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/a6dad9b4dd58 [03:30:10] 02puppet/03cache-nftables 07CosmicAlpha 03a6dad9b cache: include nftables alongside ferm [03:30:17] [02puppet] 07Universal-Omega opened pull request #4932: cache: include nftables alongside ferm (03main...03cache-nftables) 13https://github.com/miraheze/puppet/pull/4932 [03:30:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/42a665a8043c410e4d0190086333d067481e1851 [03:30:20] 02ssl/03main 07WikiTideBot 0342a665a Bot: Auto-update domain lists [03:31:07] [02puppet] 07Universal-Omega created 03mediawiki-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/275e65ef555b [03:31:09] 02puppet/03mediawiki-nftables 07CosmicAlpha 03275e65e mediawiki: include nftables alongside ferm… [03:31:31] [02puppet] 07Universal-Omega 04deleted 03mediawiki-nftables at 03275e65e 13https://api.github.com/repos/miraheze/puppet/commit/275e65e [03:32:24] [02puppet] 07Universal-Omega created 03mediawiki-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/98a60e4cc524 [03:32:25] 02puppet/03mediawiki-nftables 07CosmicAlpha 0398a60e4 mediawiki: include nftables alongside ferm… [03:32:29] [02puppet] 07Universal-Omega opened pull request #4933: mediawiki: include nftables alongside ferm (03main...03mediawiki-nftables) 13https://github.com/miraheze/puppet/pull/4933 [03:33:27] [02puppet] 07Universal-Omega closed pull request #4931: irc: include nftables alongside ferm (03network...03irc-nftables) 13https://github.com/miraheze/puppet/pull/4931 [03:33:32] [02puppet] 07Universal-Omega 04deleted 03irc-nftables at 033b16ec3 13https://api.github.com/repos/miraheze/puppet/commit/3b16ec3 [03:34:20] [02puppet] 07Universal-Omega created 03irc-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/b79c3edca92a [03:34:20] 02puppet/03irc-nftables 07CosmicAlpha 03b79c3ed irc: include nftables alongside ferm… [03:34:35] [02puppet] 07Universal-Omega opened pull request #4934: irc: include nftables alongside ferm (03main...03irc-nftables) 13https://github.com/miraheze/puppet/pull/4934 [04:30:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c7073cb7889507e4e4453104b2a461bb7bd1cd05 [04:30:18] 02ssl/03main 07WikiTideBot 03c7073cb Bot: Auto-update domain lists [05:00:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/08b4224f4f911638230a96e98f51bbcf25d53a9f [05:00:21] 02ssl/03main 07WikiTideBot 0308b4224 Bot: Auto-update domain lists [06:37:37] miraheze/mw-config - joakim the build passed. [08:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a7370800ac173fee1c530945afbd43dd1c139451 [08:30:20] 02ssl/03main 07WikiTideBot 03a737080 Bot: Auto-update domain lists [12:02:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [12:30:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:30:22] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b5b938284314d07bb7d730e1e07d95d1f3f3d43a [12:30:22] 02ssl/03main 07WikiTideBot 03b5b9382 Bot: Auto-update domain lists [12:48:41] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 16.01, 10.67, 6.58 [12:50:41] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.53, 10.45, 7.00 [12:52:41] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 7.07, 9.09, 6.92 [13:04:41] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 35.35, 22.64, 12.89 [13:12:41] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 7.50, 11.65, 11.57 [13:16:41] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.11, 8.16, 10.16 [13:56:34] [02puppet] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8795033baf92fc033b4bf9a12f904fdbeb30e756 [13:56:34] 02puppet/03main 07RhinosF1 038795033 Revert "Remove SSH keys for myself"… [13:56:50] i can cause trouble again in 30 minutes time [14:00:28] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4fcec9645d20c6ebe33d033c6a343ce3d7cdd57c [14:00:28] 02ssl/03main 07WikiTideBot 034fcec96 Bot: Auto-update domain lists [14:23:04] [02mw-config] 07SomeMWDev pushed 1 new commit to 03speedscope-prod 13https://github.com/miraheze/mw-config/commit/29f6be85a86ce8624b54f6cf1183a8af130b7e2b [14:23:04] 02mw-config/03speedscope-prod 07SomeRandomDeveloper 0329f6be8 Use use statement [14:25:30] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [14:25:31] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:58] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [14:25:59] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [14:26:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:02] !log [somerandomdeveloper@test151] starting deploy of {'config': True, 'force': True} to test151 [14:26:03] !log [somerandomdeveloper@test151] finished deploy of {'config': True, 'force': True} to test151 - SUCCESS in 0s [14:26:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:25] [02mw-config] 07SomeMWDev created 03fix-error (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0511d060920b [14:27:25] 02mw-config/03fix-error 07SomeRandomDeveloper 030511d06 Revert "Use $wi->isBeta()"… [14:27:38] [02mw-config] 07SomeMWDev opened pull request #6509: Revert "Use $wi->isBeta()" (03main...03fix-error) 13https://github.com/miraheze/mw-config/pull/6509 [14:28:09] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 348 bytes in 0.015 second response time [14:28:23] [02mw-config] 07SomeMWDev merged pull request #6509: Revert "Use $wi->isBeta()" (03main...03fix-error) 13https://github.com/miraheze/mw-config/pull/6509 [14:28:23] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/af97b585c73f518277949b224ed459ce547161eb [14:28:23] 02mw-config/03main 07SomeRandomDeveloper 03af97b58 Revert "Use $wi->isBeta()" (#6509)… [14:28:24] [02mw-config] 07SomeMWDev 04deleted 03fix-error at 030511d06 13https://api.github.com/repos/miraheze/mw-config/commit/0511d06 [14:28:27] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [14:28:33] !log [somerandomdeveloper@test151] starting deploy of {'config': True, 'force': True} to test151 [14:28:34] !log [somerandomdeveloper@test151] finished deploy of {'config': True, 'force': True} to test151 - SUCCESS in 0s [14:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:45] miraheze/mw-config - SomeMWDev the build passed. [14:29:09] @cosmicalpha FYI I reverted because `$wi` is not defined yet when Speedscope is initialized [14:29:10] [GitHub] [miraheze/mw-config] Universal-Omega: Use $wi->isBeta() | 2 changes in 1 file | Authored 1 day, 10 hours ago | Committed 1 day, 10 hours ago [14:29:41] miraheze/mw-config - SomeMWDev the build passed. [14:30:10] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.398 second response time [14:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/25b120f96f373de652ea661cab546847737f6971 [14:30:21] 02ssl/03main 07WikiTideBot 0325b120f Bot: Auto-update domain lists [14:30:27] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 342 bytes in 0.047 second response time [14:31:09] [02mw-config] 07SomeMWDev 04force-pushed 03speedscope-prod from 0329f6be8 to 033603a6e 13https://api.github.com/repos/miraheze/mw-config/commits/speedscope-prod [14:31:10] 02mw-config/03speedscope-prod 07SomeRandomDeveloper 03e1a48dc Enable Speedscope extension on prod… [14:31:12] 02mw-config/03speedscope-prod 07github-actions 032056798 CI: lint code to MediaWiki standards… [14:31:13] 02mw-config/03speedscope-prod 07SomeRandomDeveloper 033603a6e Use use statement [14:32:27] miraheze/mw-config - SomeMWDev the build passed. [14:33:54] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [14:33:55] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:33:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:49:43] [02mw-config] 07SomeMWDev merged pull request #6508: Enable Speedscope extension on prod (03main...03speedscope-prod) 13https://github.com/miraheze/mw-config/pull/6508 [14:49:44] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/d2e7d70c31fd46bba76987da2624ecd43753d30e [14:49:44] 02mw-config/03main 07SomeRandomDeveloper 03d2e7d70 Enable Speedscope extension on prod (#6508)… [14:49:45] [02mw-config] 07SomeMWDev 04deleted 03speedscope-prod at 033603a6e 13https://api.github.com/repos/miraheze/mw-config/commit/3603a6e [14:49:55] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to mwtask181 [14:49:56] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to mwtask181 - SUCCESS in 1s [14:49:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:50:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:50:59] miraheze/mw-config - SomeMWDev the build passed. [14:51:23] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:51:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:51:53] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [14:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f67a353a40ebf42bde0a9d6f06fed97523f4307b [16:00:25] 02ssl/03main 07WikiTideBot 03f67a353 Bot: Auto-update domain lists [16:37:45] Oh whoops lol. Thanks! I did deploy it to beta and it didn't crash though? odd. [16:38:29] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1621ae40e0a19dc370447810eb4b4ec6d786fe06 [16:38:29] 02puppet/03main 07CosmicAlpha 031621ae4 irc: include nftables alongside ferm… [16:38:31] [02puppet] 07Universal-Omega merged pull request #4934: irc: include nftables alongside ferm (03main...03irc-nftables) 13https://github.com/miraheze/puppet/pull/4934 [16:38:33] [02puppet] 07Universal-Omega 04deleted 03irc-nftables at 03b79c3ed 13https://api.github.com/repos/miraheze/puppet/commit/b79c3ed [16:38:58] [02puppet] 07Universal-Omega merged pull request #4932: cache: include nftables alongside ferm (03main...03cache-nftables) 13https://github.com/miraheze/puppet/pull/4932 [16:38:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c7cb5f231a11a111f8d308d27d4f1db85a71528c [16:38:58] 02puppet/03main 07CosmicAlpha 03c7cb5f2 cache: include nftables alongside ferm… [16:39:00] [02puppet] 07Universal-Omega 04deleted 03cache-nftables at 03a6dad9b 13https://api.github.com/repos/miraheze/puppet/commit/a6dad9b [16:39:16] [02puppet] 07Universal-Omega merged pull request #4933: mediawiki: include nftables alongside ferm (03main...03mediawiki-nftables) 13https://github.com/miraheze/puppet/pull/4933 [16:39:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6df75a7ba66d672035e2ec57a7ae789d8105fecd [16:39:17] 02puppet/03main 07CosmicAlpha 036df75a7 mediawiki: include nftables alongside ferm… [16:39:18] [02puppet] 07Universal-Omega 04deleted 03mediawiki-nftables at 0398a60e4 13https://api.github.com/repos/miraheze/puppet/commit/98a60e4 [16:39:34] I don't think you deployed it; I deployed to beta and it went down after my deployment [16:40:28] I locally applied it before I committed it... maybe it didn't apply properly anyway thanks and sorry about that... [16:40:55] No worries, happens [16:42:14] Btw the canary checks didn't fail for my initial deployment, only when I tried to deploy the revert. Ofc this is beta so there's only one server, but this still seems kinda wrong to me because it seemed like it was running the canary checks before syncing the config? [16:48:12] [02puppet] 07Universal-Omega created 03cp161-nftables (+1 new commit) 13https://github.com/miraheze/puppet/commit/5a96f7d5cb0c [16:48:12] 02puppet/03cp161-nftables 07CosmicAlpha 035a96f7d cp161: include nftables alongside ferm… [16:48:18] [02puppet] 07Universal-Omega opened pull request #4935: cp161: include nftables alongside ferm (03main...03cp161-nftables) 13https://github.com/miraheze/puppet/pull/4935 [17:30:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw171 [17:35:33] !log depool mw151, mw181, and mw193 in varnish on cp* [17:35:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:25] !log downtime mw151, mw181, and mw193 in icinga [17:38:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:37] !log reboot mw151, mw181, and mw193 [17:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:48] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 035c9e46e to 031ae5372 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [17:42:49] 02TSPortal/03renovate/prod-deps 07renovate[bot] 031ae5372 Update prod dependencies [17:45:09] !log repool mw151, mw181, and mw193 in varnish on cp* [17:45:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:46:20] [02puppet] 07Universal-Omega merged pull request #4935: cp161: include nftables alongside ferm (03main...03cp161-nftables) 13https://github.com/miraheze/puppet/pull/4935 [17:46:20] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4b191ad1f836d77a66e25738b89397ee6ccec228 [17:46:20] 02puppet/03main 07CosmicAlpha 034b191ad cp161: include nftables alongside ferm… [17:46:22] [02puppet] 07Universal-Omega 04deleted 03cp161-nftables at 035a96f7d 13https://api.github.com/repos/miraheze/puppet/commit/5a96f7d [17:50:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 32 backends are healthy [17:54:03] [02puppet] 07Universal-Omega created 03beta-internal (+1 new commit) 13https://github.com/miraheze/puppet/commit/6c7d60b3843e [17:54:03] 02puppet/03beta-internal 07CosmicAlpha 036c7d60b mediawiki_beta: use internal network [17:54:08] [02puppet] 07Universal-Omega opened pull request #4936: mediawiki_beta: use internal network (03main...03beta-internal) 13https://github.com/miraheze/puppet/pull/4936 [18:32:49] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:42:50] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:55:30] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [18:55:59] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 98 packages available for upgrade (3 critical updates). [18:56:13] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 154 packages available for upgrade (3 critical updates). [18:58:07] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 93 packages available for upgrade (3 critical updates). [18:58:10] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 90 packages available for upgrade (3 critical updates). [18:58:10] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [18:58:23] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [18:58:46] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 100 packages available for upgrade (3 critical updates). [18:59:07] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (3 critical updates). [19:00:09] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:00:15] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:00:16] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 95 packages available for upgrade (3 critical updates). [19:00:24] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:00:55] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 148 packages available for upgrade (3 critical updates). [19:02:03] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 91 packages available for upgrade (3 critical updates). [19:02:16] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:02:17] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:02:20] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 89 packages available for upgrade (3 critical updates). [19:03:25] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 90 packages available for upgrade (3 critical updates). [19:03:34] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:03:47] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 148 packages available for upgrade (3 critical updates). [19:04:05] PROBLEM - eventgate182 APT on eventgate182 is CRITICAL: APT CRITICAL: 94 packages available for upgrade (4 critical updates). [19:04:11] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 100 packages available for upgrade (3 critical updates). [19:04:13] [02puppet] 07Universal-Omega created 03recursive-dns (+1 new commit) 13https://github.com/miraheze/puppet/commit/a7c2afb6585c [19:04:14] 02puppet/03recursive-dns 07CosmicAlpha 03a7c2afb Support DNS recursion [19:04:19] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 100 packages available for upgrade (3 critical updates). [19:04:19] [02puppet] 07Universal-Omega opened pull request #4937: Support DNS recursion (03main...03recursive-dns) 13https://github.com/miraheze/puppet/pull/4937 [19:04:21] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:04:23] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 154 packages available for upgrade (3 critical updates). [19:04:29] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 149 packages available for upgrade (4 critical updates). [19:04:42] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:04:56] [02puppet] 07github-actions[bot] pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/aec6269e5869d24ce5630df98bf9e75c8eb7abde [19:04:56] 02puppet/03recursive-dns 07github-actions 03aec6269 CI: lint puppet code to standards… [19:05:09] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 148 packages available for upgrade (3 critical updates). [19:05:30] PROBLEM - db192 APT on db192 is CRITICAL: APT CRITICAL: 98 packages available for upgrade (3 critical updates). [19:05:33] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (3 critical updates). [19:05:39] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 92 packages available for upgrade (3 critical updates). [19:05:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/5f50c8c846084f8284bd563cb5019ad52b1ad7ba [19:05:53] 02puppet/03recursive-dns 07CosmicAlpha 035f50c8c Update [19:06:11] PROBLEM - swiftobject211 APT on swiftobject211 is CRITICAL: APT CRITICAL: 90 packages available for upgrade (3 critical updates). [19:06:19] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (3 critical updates). [19:06:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/8165a6b8ffff8c52c8785c95138c99d4aa89f034 [19:06:42] 02puppet/03recursive-dns 07CosmicAlpha 038165a6b Update [19:06:43] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 148 packages available for upgrade (3 critical updates). [19:06:46] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:06:52] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:07:35] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/82aa09cd942d22fc0e45d129709234b6fb678a45 [19:07:35] 02puppet/03recursive-dns 07CosmicAlpha 0382aa09c Update [19:07:44] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (3 critical updates). [19:08:23] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [19:08:35] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:08:45] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 96 packages available for upgrade (3 critical updates). [19:08:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/7c46d7544ce38f49783ffa13bdd6e8eed17bae6d [19:08:51] 02puppet/03recursive-dns 07CosmicAlpha 037c46d75 Update [19:09:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/15189b8c364bfec40fa06b7eaa13aeb847ea00d8 [19:09:07] 02puppet/03recursive-dns 07CosmicAlpha 0315189b8 Update [19:09:25] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [19:09:26] PROBLEM - changeprop202 APT on changeprop202 is CRITICAL: APT CRITICAL: 94 packages available for upgrade (4 critical updates). [19:09:41] PROBLEM - mattermost2 APT on mattermost2 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [19:10:36] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:11:16] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:11:18] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:11:28] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:11:40] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:11:42] PROBLEM - rdb191 APT on rdb191 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [19:11:50] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 96 packages available for upgrade (3 critical updates). [19:12:30] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 97 packages available for upgrade (3 critical updates). [19:12:33] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 91 packages available for upgrade (3 critical updates). [19:12:38] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (3 critical updates). [19:12:38] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (3 critical updates). [19:12:39] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 147 packages available for upgrade (3 critical updates). [19:12:48] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 101 packages available for upgrade (3 critical updates). [19:12:59] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 101 packages available for upgrade (3 critical updates). [19:13:09] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (3 critical updates). [19:13:18] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 146 packages available for upgrade (3 critical updates). [19:13:45] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:14:45] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 101 packages available for upgrade (3 critical updates). [19:15:15] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures [19:15:18] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 96 packages available for upgrade (3 critical updates). [19:15:20] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 146 packages available for upgrade (3 critical updates). [19:15:28] PROBLEM - speedscope211 APT on speedscope211 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:15:33] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:16:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/3c174469bf1781b96babc1936ba9df5e6904a075 [19:16:44] 02puppet/03recursive-dns 07CosmicAlpha 033c17446 Update [19:16:49] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (3 critical updates). [19:16:50] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (3 critical updates). [19:17:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/701606617504cd6e092f8808cd79a57857970c1f [19:17:25] 02puppet/03recursive-dns 07CosmicAlpha 037016066 Update [19:17:56] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 89 packages available for upgrade (3 critical updates). [19:18:16] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 90 packages available for upgrade (3 critical updates). [19:18:28] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 154 packages available for upgrade (3 critical updates). [19:19:03] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (3 critical updates). [19:19:17] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:20:03] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 146 packages available for upgrade (3 critical updates). [19:20:11] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (3 critical updates). [19:21:27] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:22:49] PROBLEM - db201 APT on db201 is CRITICAL: APT CRITICAL: 98 packages available for upgrade (3 critical updates). [19:23:01] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:23:08] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/c925e7516541b9e2941559d0b45e7eb5b8c591d8 [19:23:09] 02puppet/03recursive-dns 07CosmicAlpha 03c925e75 Update [19:23:30] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 93 packages available for upgrade (3 critical updates). [19:23:37] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 95 packages available for upgrade (3 critical updates). [19:23:43] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 141 packages available for upgrade (3 critical updates). [19:31:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/883eb8209987ed0a80055a749c303bd4757531fb [19:31:43] 02puppet/03recursive-dns 07CosmicAlpha 03883eb82 Update [19:32:15] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/cdb1d80b8f111e7b451508724ca80ac07872c77c [19:32:15] 02puppet/03recursive-dns 07CosmicAlpha 03cdb1d80 Update [19:32:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/e03559c202991575e66f20ac06d31b802d45c9a4 [19:32:52] 02puppet/03recursive-dns 07CosmicAlpha 03e03559c Update [19:33:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/5defeb95fcfecfb8c787bff0733eeb9812d996b7 [19:33:25] 02puppet/03recursive-dns 07CosmicAlpha 035defeb9 Update [19:44:27] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/477271c8b7a25c0aa0170088d033647453356f1a [19:44:27] 02puppet/03recursive-dns 07CosmicAlpha 03477271c Update [19:45:37] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/753514176ecf480049a746d718fa655924fc6f8b [19:45:38] 02puppet/03recursive-dns 07CosmicAlpha 037535141 Update [19:46:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/5094d9fa6aa354922c8780782beef25a47fbcc5d [19:46:28] 02puppet/03recursive-dns 07CosmicAlpha 035094d9f Update [19:49:29] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [19:49:41] [02puppet] 07SomeMWDev created 03update-speedscope (+1 new commit) 13https://github.com/miraheze/puppet/commit/d73e1f8f0b68 [19:49:41] 02puppet/03update-speedscope 07SomeRandomDeveloper 03d73e1f8 Update Speedscope image [19:50:14] [02puppet] 07SomeMWDev merged pull request #4938: Update Speedscope image (03main...03update-speedscope) 13https://github.com/miraheze/puppet/pull/4938 [19:50:14] [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/54571a13ae4d659016b5e269339b8577834b355c [19:50:15] 02puppet/03main 07SomeRandomDeveloper 0354571a1 Update Speedscope image (#4938) [19:50:17] [02puppet] 07SomeMWDev 04deleted 03update-speedscope at 03d73e1f8 13https://api.github.com/repos/miraheze/puppet/commit/d73e1f8 [19:50:19] [02puppet] 07SomeMWDev opened pull request #4938: Update Speedscope image (03main...03update-speedscope) 13https://github.com/miraheze/puppet/pull/4938 [19:54:52] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [19:58:25] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures [19:59:43] PROBLEM - speedscope211 speedscope.wikitide.net HTTPS on speedscope211 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.011 second response time [19:59:44] PROBLEM - speedscope211 speedscope 3000 check on speedscope211 is CRITICAL: connect to address 127.0.0.1 and port 3000: Connection refused [20:01:07] PROBLEM - speedscope211 HTTPS on speedscope211 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.011 second response time [20:01:29] [02puppet] 07SomeMWDev created 03speedscope-proxy (+1 new commit) 13https://github.com/miraheze/puppet/commit/e15f415f3c9b [20:01:31] 02puppet/03speedscope-proxy 07SomeRandomDeveloper 03e15f415 Set http proxy for prisma in Speedscope container [20:01:43] [02puppet] 07SomeMWDev opened pull request #4939: Set http proxy for prisma in Speedscope container (03main...03speedscope-proxy) 13https://github.com/miraheze/puppet/pull/4939 [20:03:23] [02puppet] 07SomeMWDev pushed 1 new commit to 03speedscope-proxy 13https://github.com/miraheze/puppet/commit/2f808b854fcdc9437e7d0c47effc56006f428aaf [20:03:23] 02puppet/03speedscope-proxy 07SomeRandomDeveloper 032f808b8 indentation [20:03:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/49b35e5993b44d9838a926b11f1d9fd1a2486d34 [20:03:25] 02puppet/03recursive-dns 07CosmicAlpha 0349b35e5 Merge branch 'main' into recursive-dns [20:03:38] [02puppet] 07SomeMWDev merged pull request #4939: Set http proxy for prisma in Speedscope container (03main...03speedscope-proxy) 13https://github.com/miraheze/puppet/pull/4939 [20:03:38] [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7d4333215bdd216068c940a0ce02fedd4d478487 [20:03:38] 02puppet/03main 07SomeRandomDeveloper 037d43332 Set http proxy for prisma in Speedscope container (#4939) [20:03:40] [02puppet] 07SomeMWDev 04deleted 03speedscope-proxy at 032f808b8 13https://api.github.com/repos/miraheze/puppet/commit/2f808b8 [20:04:21] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:04:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/040cd64a8e10074b88605cdf59db0655c07a8781 [20:04:24] 02puppet/03recursive-dns 07CosmicAlpha 03040cd64 Merge branch 'main' into recursive-dns [20:09:44] RECOVERY - speedscope211 speedscope 3000 check on speedscope211 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 3000 [20:10:45] PROBLEM - speedscope211 Puppet on speedscope211 is WARNING: WARNING: Puppet is currently disabled, message: testing, last run 2 minutes ago with 0 failures [20:11:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/692604f15e49f4d147e908bab0f0e8f23e0c2c54 [20:11:40] 02puppet/03recursive-dns 07CosmicAlpha 03692604f Update [20:14:23] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures [20:14:36] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:07] RECOVERY - speedscope211 HTTPS on speedscope211 is OK: HTTP OK: HTTP/2 200 - 354 bytes in 0.013 second response time [20:15:43] RECOVERY - speedscope211 speedscope.wikitide.net HTTPS on speedscope211 is OK: HTTP OK: HTTP/1.1 200 OK - 295 bytes in 0.166 second response time [20:16:32] [02puppet] 07SomeMWDev created 03speedscope-disable-migration (+1 new commit) 13https://github.com/miraheze/puppet/commit/838d4f352f5d [20:16:33] 02puppet/03speedscope-disable-migration 07SomeRandomDeveloper 03838d4f3 speedscope: Don't migrate DB automatically [20:16:40] [02puppet] 07SomeMWDev opened pull request #4940: speedscope: Don't migrate DB automatically (03main...03speedscope-disable-migration) 13https://github.com/miraheze/puppet/pull/4940 [20:17:05] [02puppet] 07SomeMWDev merged pull request #4940: speedscope: Don't migrate DB automatically (03main...03speedscope-disable-migration) 13https://github.com/miraheze/puppet/pull/4940 [20:17:07] [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/30717b2cf154bc1e155f35b56f23b562144d1e59 [20:17:09] [02puppet] 07SomeMWDev 04deleted 03speedscope-disable-migration at 03838d4f3 13https://api.github.com/repos/miraheze/puppet/commit/838d4f3 [20:17:11] 02puppet/03main 07SomeRandomDeveloper 0330717b2 speedscope: Don't migrate DB automatically (#4940) [20:22:56] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [20:24:49] PROBLEM - mem191 APT on mem191 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [20:26:48] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (4 critical updates). [20:33:37] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/d9eebc341bf670aad02cf6f06f3855b34d52bb77 [20:33:37] 02puppet/03recursive-dns 07CosmicAlpha 03d9eebc3 Update [20:35:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/2b63728cdf0eb830ce94a09aa565112ca6242862 [20:35:41] 02puppet/03recursive-dns 07CosmicAlpha 032b63728 Update [20:46:14] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [21:10:59] !log removed public IPv6 address from test151 [21:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:40] [02puppet] 07Universal-Omega retitled PR #4937: "Support DNS recursion" ➜ "Support DNS recursion and use internal network for beta" 13https://github.com/miraheze/puppet/pull/4937 [21:12:39] [02puppet] 07Universal-Omega pushed 1 new commit to 03recursive-dns 13https://github.com/miraheze/puppet/commit/ea4bdfb85163eea79a52e5487d9fbd89d0670093 [21:12:39] 02puppet/03recursive-dns 07CosmicAlpha 03ea4bdfb Update [21:13:41] [02puppet] 07Universal-Omega merged pull request #4937: Support DNS recursion and use internal network for beta (03main...03recursive-dns) 13https://github.com/miraheze/puppet/pull/4937 [21:13:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8c964fc4b0ad0b40e70c535303341a7e286e827c [21:13:44] 02puppet/03main 07CosmicAlpha 038c964fc Support DNS recursion and use internal network for beta… [21:13:46] [02puppet] 07Universal-Omega 04deleted 03recursive-dns at 03ea4bdfb 13https://api.github.com/repos/miraheze/puppet/commit/ea4bdfb [21:17:16] RECOVERY - bots171 APT on bots171 is OK: APT OK: 95 packages available for upgrade (0 critical updates). [21:17:41] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 151 packages available for upgrade (0 critical updates). [21:17:42] RECOVERY - bast161 APT on bast161 is OK: APT OK: 88 packages available for upgrade (0 critical updates). [21:17:47] RECOVERY - db172 APT on db172 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:17:52] RECOVERY - cp161 APT on cp161 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:18:25] RECOVERY - cp171 APT on cp171 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [21:18:29] RECOVERY - cloud21 APT on cloud21 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [21:18:44] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 145 packages available for upgrade (0 critical updates). [21:19:05] RECOVERY - cp191 APT on cp191 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:19:06] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 145 packages available for upgrade (0 critical updates). [21:20:13] RECOVERY - db171 APT on db171 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:20:21] RECOVERY - bast181 APT on bast181 is OK: APT OK: 89 packages available for upgrade (0 critical updates). [21:20:56] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [21:21:16] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 151 packages available for upgrade (0 critical updates). [21:21:24] RECOVERY - db161 APT on db161 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [21:21:28] RECOVERY - db151 APT on db151 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:21:35] RECOVERY - cp201 APT on cp201 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:21:51] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 145 packages available for upgrade (0 critical updates). [21:22:29] RECOVERY - db192 APT on db192 is OK: APT OK: 95 packages available for upgrade (0 critical updates). [21:22:45] RECOVERY - speedscope211 Puppet on speedscope211 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:22:46] RECOVERY - changeprop202 APT on changeprop202 is OK: APT OK: 90 packages available for upgrade (0 critical updates). [21:23:08] RECOVERY - db182 APT on db182 is OK: APT OK: 90 packages available for upgrade (0 critical updates). [21:24:00] RECOVERY - eventgate182 APT on eventgate182 is OK: APT OK: 90 packages available for upgrade (0 critical updates). [21:24:44] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 151 packages available for upgrade (0 critical updates). [21:24:54] RECOVERY - db201 APT on db201 is OK: APT OK: 95 packages available for upgrade (0 critical updates). [21:24:58] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [21:25:00] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [21:25:07] RECOVERY - db181 APT on db181 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:25:51] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 90 packages available for upgrade (0 critical updates). [21:26:05] RECOVERY - mem161 APT on mem161 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [21:26:09] RECOVERY - mattermost2 APT on mattermost2 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:26:25] RECOVERY - mw151 APT on mw151 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:26:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:26:59] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:27:03] [02puppet] 07Universal-Omega closed pull request #4936: mediawiki_beta: use internal network (03main...03beta-internal) 13https://github.com/miraheze/puppet/pull/4936 [21:27:03] [02puppet] 07Universal-Omega commented on pull request #4936: Done in #4937 instead 13https://github.com/miraheze/puppet/pull/4936#issuecomment-5417032582 [21:27:03] RECOVERY - llm191 APT on llm191 is OK: APT OK: 94 packages available for upgrade (0 critical updates). [21:27:05] RECOVERY - mon181 APT on mon181 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [21:27:06] [02puppet] 07Universal-Omega 04deleted 03beta-internal at 036c7d60b 13https://api.github.com/repos/miraheze/puppet/commit/6c7d60b [21:27:09] RECOVERY - mem151 APT on mem151 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [21:28:32] RECOVERY - mw171 APT on mw171 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:28:50] RECOVERY - mw152 APT on mw152 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:28:52] RECOVERY - mem201 APT on mem201 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [21:29:14] RECOVERY - mw163 APT on mw163 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:07] RECOVERY - mw153 APT on mw153 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:14] RECOVERY - mw161 APT on mw161 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:18] RECOVERY - mw172 APT on mw172 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/0b6c23bbe0a295bdb19612db6524722224f1e9b6 [21:30:22] 02ssl/03main 07WikiTideBot 030b6c23b Bot: Auto-update domain lists [21:30:50] !log upgrade Speedscope extension on beta and prod (mwdeploy logs are broken) [21:30:53] RECOVERY - mw173 APT on mw173 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:59] RECOVERY - mw162 APT on mw162 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:30:59] RECOVERY - mw191 APT on mw191 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:31:21] RECOVERY - mw181 APT on mw181 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:32:15] RECOVERY - mw193 APT on mw193 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:32:28] RECOVERY - mw182 APT on mw182 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:33:37] RECOVERY - mw192 APT on mw192 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:33:56] RECOVERY - mw183 APT on mw183 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:34:27] RECOVERY - mw201 APT on mw201 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:34:35] RECOVERY - mem191 APT on mem191 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [21:35:06] RECOVERY - mw202 APT on mw202 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:35:17] RECOVERY - mw203 APT on mw203 is OK: APT OK: 138 packages available for upgrade (0 critical updates). [21:35:32] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [21:35:40] RECOVERY - os161 APT on os161 is OK: APT OK: 87 packages available for upgrade (0 critical updates). [21:36:12] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [21:36:23] !log test [21:36:24] RECOVERY - ns1 APT on ns1 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:36:25] !log [universalomega@puppet181] Upgraded packages on phorge171: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:36:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:44] !log [universalomega@puppet181] Upgraded packages on speedscope211: openssl-provider-legacy, libssl3t64, linux-libc-dev, and openssl [21:36:47] !log restarted irclogserverbot.service and logbot.service on bots171 [21:36:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:57] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 144 packages available for upgrade (0 critical updates). [21:37:04] !log [universalomega@puppet181] Upgraded packages on reports171: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:37:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:17] Thanks @abaddriverlol [21:37:23] !log [universalomega@puppet181] Upgraded packages on prometheus151: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:37:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:37] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 145 packages available for upgrade (0 critical updates). [21:37:42] !log [universalomega@puppet181] Upgraded packages on puppet181: openssl-provider-legacy, libssl3t64, linux-libc-dev, and openssl [21:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:01] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [21:38:03] !log [universalomega@puppet181] Upgraded packages on os162: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:38:05] RECOVERY - os162 APT on os162 is OK: APT OK: 87 packages available for upgrade (0 critical updates). [21:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:24] !log [universalomega@puppet181] Upgraded packages on swiftac171: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:38:26] RECOVERY - speedscope211 APT on speedscope211 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:38:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:38] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [21:38:41] RECOVERY - reports171 APT on reports171 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:38:42] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [21:38:44] !log [universalomega@puppet181] Upgraded packages on os151: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:38:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:52] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 93 packages available for upgrade (0 critical updates). [21:39:06] !log [universalomega@puppet181] Upgraded packages on rdb191: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:39:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:13] RECOVERY - os151 APT on os151 is OK: APT OK: 87 packages available for upgrade (0 critical updates). [21:39:25] !log [universalomega@puppet181] Upgraded packages on swiftobject151: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:39:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:45] !log [universalomega@puppet181] Upgraded packages on ns2: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:39:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:40:13] !log [universalomega@puppet181] Upgraded packages on os202: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:40:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:40:19] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [21:40:20] RECOVERY - os202 APT on os202 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [21:40:33] !log [universalomega@puppet181] Upgraded packages on os191: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:40:46] RECOVERY - rdb191 APT on rdb191 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [21:40:46] RECOVERY - ns2 APT on ns2 is OK: APT OK: 88 packages available for upgrade (0 critical updates). [21:40:54] !log [universalomega@puppet181] Upgraded packages on swiftobject161: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:40:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:14] !log [universalomega@puppet181] Upgraded packages on swiftobject171: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:35] !log [universalomega@puppet181] Upgraded packages on os201: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:41:36] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [21:41:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:47] RECOVERY - os191 APT on os191 is OK: APT OK: 86 packages available for upgrade (0 critical updates). [21:41:55] !log [universalomega@puppet181] Upgraded packages on swiftproxy161: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:05] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [21:42:13] !log [universalomega@puppet181] Upgraded packages on swiftobject211: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:33] !log [universalomega@puppet181] Upgraded packages on swiftproxy171: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:42:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:38] RECOVERY - os201 APT on os201 is OK: APT OK: 86 packages available for upgrade (0 critical updates). [21:42:54] !log [universalomega@puppet181] Upgraded packages on swiftobject191: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:42:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:58] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 93 packages available for upgrade (0 critical updates). [21:43:17] !log [universalomega@puppet181] Upgraded packages on test151: linux-libc-dev, openssl-provider-legacy, libssl-dev, libssl3t64, and openssl [21:43:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:43:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:29] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:43:37] !log [universalomega@puppet181] Upgraded packages on swiftobject181: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:43:40] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [21:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:48] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 93 packages available for upgrade (0 critical updates). [21:43:58] !log [universalomega@puppet181] Upgraded packages on swiftobject201: linux-libc-dev, openssl-provider-legacy, libssl3t64, and openssl [21:44:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:05] RECOVERY - test151 APT on test151 is OK: APT OK: 145 packages available for upgrade (0 critical updates). [21:44:10] RECOVERY - swiftobject211 APT on swiftobject211 is OK: APT OK: 87 packages available for upgrade (0 critical updates). [21:44:15] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [21:44:43] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [22:15:42] [02mw-config] 07SomeMWDev created 03T15905 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/7fc4d9e65b63 [22:15:43] 02mw-config/03T15905 07SomeRandomDeveloper 037fc4d9e Enable Speedscope sampling for 0.2% of prod requests… [22:16:07] [02mw-config] 07SomeMWDev opened pull request #6510: Enable Speedscope sampling for 0.2% of prod requests (03main...03T15905) 13https://github.com/miraheze/mw-config/pull/6510 [22:17:54] [02mw-config] 07SomeMWDev merged pull request #6510: Enable Speedscope sampling for 0.2% of prod requests (03main...03T15905) 13https://github.com/miraheze/mw-config/pull/6510 [22:17:56] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/5b09ccbfdd2872009806c59ec33473225b8447db [22:17:58] [02mw-config] 07SomeMWDev 04deleted 03T15905 at 037fc4d9e 13https://api.github.com/repos/miraheze/mw-config/commit/7fc4d9e [22:18:00] 02mw-config/03main 07SomeRandomDeveloper 035b09ccb Enable Speedscope sampling for 0.2% of prod requests (#6510)… [22:18:08] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [22:18:09] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [22:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:29] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:58] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [22:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:14] miraheze/mw-config - SomeMWDev the build passed. [22:19:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b506ac7b83195fe4343cd43458619752fb1bc08a [22:19:18] 02puppet/03main 07CosmicAlpha 03b506ac7 db172: include nftables alongside ferm… [22:30:33] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/6c6c1e737106 [22:30:33] 02puppet/03Universal-Omega-patch-1 07CosmicAlpha 036c6c1e7 changeprop202: include nftables alongside ferm [22:30:40] [02puppet] 07Universal-Omega opened pull request #4941: changeprop202: include nftables alongside ferm (03main...03Universal-Omega-patch-1) 13https://github.com/miraheze/puppet/pull/4941 [22:31:34] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/855df2fe2b6b [22:31:34] 02puppet/03Universal-Omega-patch-2 07CosmicAlpha 03855df2f eventgate182: include nftables alongside ferm [22:31:41] [02puppet] 07Universal-Omega opened pull request #4942: eventgate182: include nftables alongside ferm (03main...03Universal-Omega-patch-2) 13https://github.com/miraheze/puppet/pull/4942 [22:32:29] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7764f3624b34 [22:32:29] 02puppet/03Universal-Omega-patch-3 07CosmicAlpha 037764f36 graylog161: include nftables alongside ferm [22:32:35] [02puppet] 07Universal-Omega opened pull request #4943: graylog161: include nftables alongside ferm (03main...03Universal-Omega-patch-3) 13https://github.com/miraheze/puppet/pull/4943 [22:33:22] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-4 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4774ac27af7f [22:33:23] 02puppet/03Universal-Omega-patch-4 07CosmicAlpha 034774ac2 kafka182: include nftables alongside ferm [22:33:27] [02puppet] 07Universal-Omega opened pull request #4944: kafka182: include nftables alongside ferm (03main...03Universal-Omega-patch-4) 13https://github.com/miraheze/puppet/pull/4944 [22:34:05] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e557ab4cf220 [22:34:05] 02puppet/03Universal-Omega-patch-5 07CosmicAlpha 03e557ab4 ldap171: include nftables alongside ferm [22:34:10] [02puppet] 07Universal-Omega opened pull request #4945: ldap171: include nftables alongside ferm (03Universal-Omega-patch-4...03Universal-Omega-patch-5) 13https://github.com/miraheze/puppet/pull/4945 [22:34:15] [02puppet] 07Universal-Omega closed pull request #4945: ldap171: include nftables alongside ferm (03Universal-Omega-patch-4...03Universal-Omega-patch-5) 13https://github.com/miraheze/puppet/pull/4945 [22:34:19] [02puppet] 07Universal-Omega 04deleted 03Universal-Omega-patch-5 at 03e557ab4 13https://api.github.com/repos/miraheze/puppet/commit/e557ab4 [22:34:42] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f675eadd3a1c [22:34:44] 02puppet/03Universal-Omega-patch-5 07CosmicAlpha 03f675ead ldap171: include nftables alongside ferm [22:34:48] [02puppet] 07Universal-Omega opened pull request #4946: ldap171: include nftables alongside ferm (03main...03Universal-Omega-patch-5) 13https://github.com/miraheze/puppet/pull/4946 [22:35:18] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/ed93c1a1e0d3 [22:35:18] 02puppet/03Universal-Omega-patch-6 07CosmicAlpha 03ed93c1a matomo151: include nftables alongside ferm [22:35:24] [02puppet] 07Universal-Omega opened pull request #4947: matomo151: include nftables alongside ferm (03main...03Universal-Omega-patch-6) 13https://github.com/miraheze/puppet/pull/4947 [22:36:09] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-7 (+1 new commit) 13https://github.com/miraheze/puppet/commit/de27a5283cc3 [22:36:09] 02puppet/03Universal-Omega-patch-7 07CosmicAlpha 03de27a52 mattermost2: include nftables alongside ferm [22:36:16] [02puppet] 07Universal-Omega opened pull request #4948: mattermost2: include nftables alongside ferm (03main...03Universal-Omega-patch-7) 13https://github.com/miraheze/puppet/pull/4948 [22:36:46] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw153 [22:36:53] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/79adf2cd7600 [22:36:54] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw153 [22:36:54] 02puppet/03Universal-Omega-patch-8 07CosmicAlpha 0379adf2c mem151: include nftables alongside ferm [22:36:58] [02puppet] 07Universal-Omega opened pull request #4949: mem151: include nftables alongside ferm (03main...03Universal-Omega-patch-8) 13https://github.com/miraheze/puppet/pull/4949 [22:37:19] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10005 milliseconds [22:37:32] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-9 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f7d0d505e429 [22:37:32] 02puppet/03Universal-Omega-patch-9 07CosmicAlpha 03f7d0d50 mon181: include nftables alongside ferm [22:37:38] [02puppet] 07Universal-Omega opened pull request #4950: mon181: include nftables alongside ferm (03main...03Universal-Omega-patch-9) 13https://github.com/miraheze/puppet/pull/4950 [22:38:24] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-10 (+1 new commit) 13https://github.com/miraheze/puppet/commit/3a3fb2ac0f7f [22:38:24] 02puppet/03Universal-Omega-patch-10 07CosmicAlpha 033a3fb2a ns1: include nftables alongside ferm [22:38:30] [02puppet] 07Universal-Omega opened pull request #4951: ns1: include nftables alongside ferm (03main...03Universal-Omega-patch-10) 13https://github.com/miraheze/puppet/pull/4951 [22:38:45] PROBLEM - mw153 Current Load on mw153 is CRITICAL: LOAD CRITICAL - total load average: 21.77, 28.93, 17.00 [22:38:46] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 32 backends are healthy [22:38:54] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 32 backends are healthy [22:39:04] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/55aa8c1f35f9 [22:39:05] 02puppet/03Universal-Omega-patch-11 07CosmicAlpha 0355aa8c1 ns2: include nftables alongside ferm [22:39:10] [02puppet] 07Universal-Omega opened pull request #4952: ns2: include nftables alongside ferm (03main...03Universal-Omega-patch-11) 13https://github.com/miraheze/puppet/pull/4952 [22:39:18] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.073 second response time [22:40:04] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-12 (+1 new commit) 13https://github.com/miraheze/puppet/commit/170b48e4cdd8 [22:40:04] 02puppet/03Universal-Omega-patch-12 07CosmicAlpha 03170b48e phorge171: include nftables alongside ferm [22:40:10] [02puppet] 07Universal-Omega opened pull request #4953: phorge171: include nftables alongside ferm (03main...03Universal-Omega-patch-12) 13https://github.com/miraheze/puppet/pull/4953 [22:40:44] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 8.21, 21.43, 15.70 [22:41:03] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-13 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1462ad56b812 [22:41:03] 02puppet/03Universal-Omega-patch-13 07CosmicAlpha 031462ad5 puppet181: include nftables alongside ferm [22:41:08] [02puppet] 07Universal-Omega opened pull request #4954: puppet181: include nftables alongside ferm (03main...03Universal-Omega-patch-13) 13https://github.com/miraheze/puppet/pull/4954 [22:41:40] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-14 (+1 new commit) 13https://github.com/miraheze/puppet/commit/da353ec5cd76 [22:41:42] 02puppet/03Universal-Omega-patch-14 07CosmicAlpha 03da353ec rdb191: include nftables alongside ferm [22:41:46] [02puppet] 07Universal-Omega opened pull request #4955: rdb191: include nftables alongside ferm (03main...03Universal-Omega-patch-14) 13https://github.com/miraheze/puppet/pull/4955 [22:42:17] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-15 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2d210e3a4a33 [22:42:17] 02puppet/03Universal-Omega-patch-15 07CosmicAlpha 032d210e3 reports171: include nftables alongside ferm [22:42:22] [02puppet] 07Universal-Omega opened pull request #4956: reports171: include nftables alongside ferm (03main...03Universal-Omega-patch-15) 13https://github.com/miraheze/puppet/pull/4956 [22:42:43] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 5.61, 16.25, 14.48 [22:43:06] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-16 (+1 new commit) 13https://github.com/miraheze/puppet/commit/07dc4afafbac [22:43:06] 02puppet/03Universal-Omega-patch-16 07CosmicAlpha 0307dc4af swiftac171: include nftables alongside ferm [22:43:10] [02puppet] 07Universal-Omega opened pull request #4957: swiftac171: include nftables alongside ferm (03main...03Universal-Omega-patch-16) 13https://github.com/miraheze/puppet/pull/4957 [22:43:45] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-17 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f7436a5b93e9 [22:43:47] 02puppet/03Universal-Omega-patch-17 07CosmicAlpha 03f7436a5 swiftobject151: include nftables alongside ferm [22:43:51] [02puppet] 07Universal-Omega opened pull request #4958: swiftobject151: include nftables alongside ferm (03main...03Universal-Omega-patch-17) 13https://github.com/miraheze/puppet/pull/4958 [22:44:26] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-18 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c0952df56c13 [22:44:26] 02puppet/03Universal-Omega-patch-18 07CosmicAlpha 03c0952df swiftproxy161: include nftables alongside ferm [22:44:32] [02puppet] 07Universal-Omega opened pull request #4959: swiftproxy161: include nftables alongside ferm (03main...03Universal-Omega-patch-18) 13https://github.com/miraheze/puppet/pull/4959 [22:45:14] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-19 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1074e4b8a29e [22:45:14] 02puppet/03Universal-Omega-patch-19 07CosmicAlpha 031074e4b os151: include nftables alongside ferm [22:45:20] [02puppet] 07Universal-Omega opened pull request #4960: os151: include nftables alongside ferm (03main...03Universal-Omega-patch-19) 13https://github.com/miraheze/puppet/pull/4960 [22:46:10] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-20 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4ceae6dc93dd [22:46:10] 02puppet/03Universal-Omega-patch-20 07CosmicAlpha 034ceae6d os162: include nftables alongside ferm [22:46:15] [02puppet] 07Universal-Omega opened pull request #4961: os162: include nftables alongside ferm (03main...03Universal-Omega-patch-20) 13https://github.com/miraheze/puppet/pull/4961 [22:50:48] [02puppet] 07SomeMWDev created 03speedscope-sqlite3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/923b16dc8725 [22:50:49] 02puppet/03speedscope-sqlite3 07SomeRandomDeveloper 03923b16d speedscope: install sqlite3 for maintenance purposes [22:51:08] [02puppet] 07SomeMWDev closed pull request #4898: mwdeploy: fix KeyError when applying core patches (03main...03T15591) 13https://github.com/miraheze/puppet/pull/4898 [22:51:26] [02puppet] 07Universal-Omega created 03Universal-Omega-patch-21 (+1 new commit) 13https://github.com/miraheze/puppet/commit/001c226d5420 [22:51:26] 02puppet/03Universal-Omega-patch-21 07CosmicAlpha 03001c226 cloud21: include nftables alongside ferm [22:51:32] [02puppet] 07Universal-Omega opened pull request #4962: cloud21: include nftables alongside ferm (03main...03Universal-Omega-patch-21) 13https://github.com/miraheze/puppet/pull/4962 [22:51:38] [02puppet] 07SomeMWDev 04deleted 03T15591 at 03064f315 13https://api.github.com/repos/miraheze/puppet/commit/064f315 [22:51:49] [02puppet] 07SomeMWDev opened pull request #4963: speedscope: install sqlite3 for maintenance purposes (03main...03speedscope-sqlite3) 13https://github.com/miraheze/puppet/pull/4963 [22:52:36] [02puppet] 07SomeMWDev merged pull request #4963: speedscope: install sqlite3 for maintenance purposes (03main...03speedscope-sqlite3) 13https://github.com/miraheze/puppet/pull/4963 [22:52:36] [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/03c91739425233127c4ba62071c65803d4b62e45 [22:52:37] 02puppet/03main 07SomeRandomDeveloper 0303c9173 speedscope: install sqlite3 for maintenance purposes (#4963) [22:52:39] [02puppet] 07SomeMWDev 04deleted 03speedscope-sqlite3 at 03923b16d 13https://api.github.com/repos/miraheze/puppet/commit/923b16d [22:55:45] people, place your bets. is it gonna be cargo, smw, commentstreams or something else [22:56:04] actually magicnocache and randomselection deserve a dishonourable mention [22:56:23] [1/2] ..or MirahezeFunctions, which takes 50-60ms for every single request to any endpoint [22:56:23] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1541944356997701692/image.png?ex=6a8f6f16&is=6a8e1d96&hm=b3bbaf040ff57211c105d45a6b471835b6c5c08be2022a173ae2e0dcb7946b14& [22:56:41] I will work on perf improvements for that next week. [22:57:17] but it will probably take 24 hours before we have a daily aggregation with proper data, since we have so many wikis with such a variety of extensions, I don't think hourly aggregations will be that useful, at least for the parts caused by extensions [22:57:49] It at one point was more than 300ms (according to xhprof) before I reworked large parts of it one day lol. [22:58:03] in three minutes we will hopefully have the first hourly aggregation with 42mins of data [22:58:17] wow [22:58:25] tbf xhprof makes everything like 3x slower [22:58:28] but still [22:59:14] It does but after the fixes MirahezeFunctions disappeared from any of the top 30 or so results in xhprof. Anyway, 60ms is way to much for one thing yeah. gotta be some way to improve it.. [23:00:32] 60ms is more than a cached page view request should ideally take (though not sure if we can realistically get to that point, at least with our current CPUs) [23:02:19] https://speedscope.wikitide.net/view/aggregation/latest/hourly exists now, consisting of 2386 profiles (which means that 0.2% is a good sampling rate if we want about 3600/hr) [23:02:48] MirahezeFunctions::__construct is 18% of wall time [23:03:20] commentstreams is 1.5% of wall time [23:05:11] Cargo looks fine on a global level at least [23:06:18] [02puppet] 07Universal-Omega pushed 1 new commit to 03openai-config 13https://github.com/miraheze/puppet/commit/bc02394039cc89bcd7939efc4d330179ab5da201 [23:06:18] 02puppet/03openai-config 07CosmicAlpha 03bc02394 Merge branch 'main' into openai-config [23:07:55] yeah looks like smw is more of a performance issue than cargo [23:08:14] then again, cargo's issues were more security than performance [23:08:15] [1/2] Measured per-wiki it probably takes a lot longer. Same goes for SMW which benefits from being available on so few wikis. [23:08:15] [2/2] It'll still be good to know which global extension(s) is the most time-consuming though. [23:09:01] Yeah I think we can mostly use these aggregations to reduce the load on the MW servers (and also maybe DB servers) in total, but not performance on individual wikis [23:09:03] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures [23:09:07] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [23:09:16] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures [23:09:20] scribunto seemingly [23:09:23] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures [23:09:24] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures [23:09:31] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures [23:09:38] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [23:09:42] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures [23:09:47] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures [23:09:52] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [23:09:54] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures [23:09:58] although duh [23:10:02] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures [23:10:09] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures [23:10:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03openai-config 13https://github.com/miraheze/puppet/commit/85044a7461bfb59e73c80679e9ffbd6398a64f21 [23:10:13] 02puppet/03openai-config 07CosmicAlpha 0385044a7 Fix [23:10:18] PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 23 minutes ago with 0 failures [23:10:22] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures [23:10:31] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [23:10:44] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures [23:10:46] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [23:10:48] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [23:10:49] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 19 minutes ago with 0 failures [23:10:59] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures [23:11:00] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [23:11:40] DisplayTitle is one that I did not expect [23:12:10] the PageLinkRendererBegin part? [23:12:19] oh yeah it queries a page prop... [23:12:38] and `DisplayTitleService::getDisplayTitle` too but that might be a sub thing [23:12:47] used on 1539 wikis [23:12:49] yeah looks like it is [23:13:04] I wonder if people are just enabling the extension because they want to use core display titles but don't understand the difference [23:13:15] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03responses-api 13https://github.com/miraheze/CreateWiki/commit/7d6d23947681ef6c0b850c323def1cf8b4c71c45 [23:13:15] 02CreateWiki/03responses-api 07CosmicAlpha 037d6d239 Merge branch 'main' into responses-api [23:13:46] if we just removed it from all wikis would anyone notice lmao [23:13:48] this probably applies to a few similar extensions as well, would be cool if there was a way to figure out which wikis are not using it at all, add some sort of notice to the talk pages of bureaucrats, and then disable it after X days [23:13:50] !log [universalomega@test151] starting deploy of {'force_upgrade': True, 'versions': '1.46', 'upgrade_extensions': 'CreateWiki'} to test151 [23:13:52] or that lol [23:13:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:59] !log [universalomega@test151] finished deploy of {'force_upgrade': True, 'versions': '1.46', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 8s [23:14:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:14] [1/3] urlshortener makes sense [23:16:15] [2/3] tabberneue is kinda hard to tell what's actually tabberneue and what's just normal parsing [23:16:15] [3/3] SimpleBlogPage seems outsized for its usage [23:17:55] TabberNeue is there because it needs to recursively parse its content. [23:18:25] Like on https://speedscope.wikitide.net/view/38130b9282badd90ebf882fbe551950c the underlying Lua invocations are slow but they are grouped under TabberNeue. [23:19:05] SpeciallyCursed also seems a bit outsized for its usage [23:19:39] [1/2] Very bad in terms of total time. Negligible in terms of time spent on itself. [23:19:39] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1541950211579060337/image.png?ex=6a8f748a&is=6a8e230a&hm=f104d9ea1156c27d6c3dd858ddf10ef7008ebaf3bc78628fa05e52e850c6593a& [23:20:37] miraheze/CreateWiki - Universal-Omega the build passed. [23:20:42] [1/2] btw not sure if you've seen this but if you want to figure out which functions call another function (e.g. DisplayDisplayTitleService::getDisplayTitleTitle in this case), you can click on the function in the sandwich view [23:20:42] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1541950478357635102/image.png?ex=6a8f74ca&is=6a8e234a&hm=c264d3966fa95defb23e6eb39578d2a671a4a4dcc7fe071dee0c79dda679c219& [23:22:06] I've created two tasks and added them to the "Potential improvement" column at btw, I think it would be good to track all potential improvements we stumble upon this way [23:24:03] which part? [23:24:44] MediaWiki\Extension\SpeciallyCursed\Hooks::onSpecialPage_initList takes a while [23:25:26] Most of that seems to be creating a parser [23:26:22] I wonder if there's any downside to just using the main parser [23:26:58] !log disabled puppet on mw*, will remain disabled for a few hours, testing new AI on beta. Do not re-enable puppet without first reverting local changes on puppet181. [23:27:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:33] @Technology Team just FYI. [23:27:52] well the main parser doesn't allow raw HTML does it? [23:28:51] it should allow it as well through `::startExternalParse()` [23:29:15] but I think the easiest solution could be to just either pass the ParserFactory to the special pages, or some other object which lazily creates a parser [23:29:27] [23:29:47] wtf [23:30:13] why is it spending so much time on gadgets stuff [23:32:52] although I don't think many people actually use this endpoint so idk why speciallycursed is so high [23:32:55] I was thinking. Are we able to bulk submit older wiki requests to beta and compare the verdict with the old AI to ensure there are no regressions? [23:33:37] That would likely cause to much usage, also I don't really think we can... [23:33:55] because AFAICT the hook is run on every request, at least in some cases [23:34:30] have you tried promptfoo, if you can create a dataset from old requests you can then just run an eval to see how often it aligns with previous requests [23:34:49] theoretically it would allow 1:1 comparisons [23:35:03] openai killed the old evals platform and replaced it with promptfoo [23:35:05] which is worse but still something [23:35:24] That's the idea behind the new CreateWiki AI prototype I made a while ago which has unit tests in that format (wiki request + expected response). [23:35:48] you'd probably be better just running the evals on a subset of known good request outcomes [23:35:57] like random 5% or something [23:41:09] I'm not sure how that happens, on profiles at WO the thing that takes the longest is MirahezeFunctions::loadExtensions, there's no setServers or getServers which has a meaningful effect on performance [23:41:38] maybe the difference is that you don't have 26k wikis the code loops through every time [23:42:05] does it loop through each wiki in the list on each request? [23:42:32] [1/2] yeah [23:42:32] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1541955972023582803/image.png?ex=6a8f79e7&is=6a8e2867&hm=2b26e49bf4aa29b4addc1ce0cbf7a680bd5785046e1bba08e092959648439485& [23:42:36] including deleted ones [23:42:53] there are also two iterations per wiki because it loops through self::SUFFIXES for some reason [23:43:20] even though surely the code doesn't need to support beta if it's run on prod, and vice versa [23:44:01] hmm [23:44:18] looping through self::SUFFIXES could be reduced by doing a check for hostname [23:44:34] although I don't know if the whole approach is great [23:44:42] or just checking whether we're on a beta server or a prod server? [23:44:49] yes thats what I mean [23:44:54] ah [23:45:00] if hostname starts with mw then don't do it [23:45:07] or the inverse [23:45:24] or just use one suffix, either wiki or wikibeta depending on the hostname, which should cut half of the iterations [23:46:06] couldn't the whole list of wiki URLs just be cached in memcached rather than rebuilding every time [23:46:20] although that would probably cause some fun cache desyncing [23:46:46] and mhfunctions is invoked before mediawiki which would also make it a pain to cache [23:48:02] yeah..