[00:01:01] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/1e81832458c5f907c9a23f7d5d8d0d16e4435e10 [00:01:01] 02statichelp/03main 07WikiTideBot 031e81832 Bot: Auto-update Tech namespace pages 2026-05-27 00:00:58 [00:01:55] [02mw-config] 07SomeMWDev created 03T15445 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/b7797e5fa8e9 [00:01:55] 02mw-config/03T15445 07SomeRandomDeveloper 03b7797e5 T15445: Disable Maps extension [00:02:30] [02mw-config] 07SomeMWDev opened pull request #6425: T15445: Disable Maps extension (03main...03T15445) 13https://github.com/miraheze/mw-config/pull/6425 [00:03:33] [02mw-config] 07SomeMWDev merged pull request #6425: T15445: Disable Maps extension (03main...03T15445) 13https://github.com/miraheze/mw-config/pull/6425 [00:03:33] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/8f4c552410bf61af881beeb51bbb55fa20cda3e8 [00:03:33] 02mw-config/03main 07SomeRandomDeveloper 038f4c552 T15445: Disable Maps extension (#6425) [00:03:34] miraheze/mw-config - SomeMWDev the build passed. [00:03:34] [02mw-config] 07SomeMWDev 04deleted 03T15445 at 03b7797e5 13https://api.github.com/repos/miraheze/mw-config/commit/b7797e5 [00:03:37] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:01] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [00:04:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:45] miraheze/mw-config - SomeMWDev the build passed. [01:05:56] !log [skye@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/maps.php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:GenerateExtensionDatabaseList metawiki --directory=~/maps (START) [01:05:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:09] !log [skye@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/maps.php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:GenerateExtensionDatabaseList metawiki --directory=~/maps (END - exit=33280) [01:06:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:32] wwwwwhat is cro on [01:07:41] oh my god my keyboard is gonna kill me [01:35:53] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 03e7d231a to 03598eae4 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [01:35:53] 02TSPortal/03renovate/prod-deps 07renovate[bot] 03598eae4 Update prod dependencies [02:47:17] [1/2] !log [skye@mwtask171] sudo -u www-data foreachwikiindblist maps.php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:NotifyWikiUsers --header='Maps extensions has been disabled' --message='The technology team has disabled the Maps extension due to a security vulnerability. Please check the Tech:Noticeboard page on Meta for more details.' --link='m:Tech:Noticeboard#Maps_has_ [02:47:17] [2/2] been_globally_disabled' --link-label='Tech noticeboard' --group=bureaucrat --group=sysop [02:47:51] i typoed "extensions" 😭 [02:48:02] is logbot ok [02:48:23] meow [02:48:30] !log test [02:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:21] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [03:06:42] !log [skye@mwtask171] sudo -u www-data foreachwikiindblist /home/skye/maps/maps.php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:NotifyWikiUsers --header='Maps extensions has been disabled' ... --group=bureaucrat --group=sysop [03:06:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:04] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [04:10:04] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [04:57:16] [02mw-config] 07The-Voidwalker created 03The-Voidwalker-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/c31d6dacc02d [04:57:16] 02mw-config/03The-Voidwalker-patch-1 07The-Voidwalker 03c31d6da add fallback handler for GeoJson content model while maps is disabled [04:57:25] [02mw-config] 07The-Voidwalker opened pull request #6426: add fallback handler for GeoJson content model while maps is disabled (03main...03The-Voidwalker-patch-1) 13https://github.com/miraheze/mw-config/pull/6426 [04:58:30] miraheze/mw-config - The-Voidwalker the build has errored. [05:01:28] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03The-Voidwalker-patch-1 13https://github.com/miraheze/mw-config/commit/57775fed644107f3668a632bfe0793c3dea166cf [05:01:28] 02mw-config/03The-Voidwalker-patch-1 07The-Voidwalker 0357775fe swap to array_key_exists per CI [05:02:32] miraheze/mw-config - The-Voidwalker the build has errored. [05:03:14] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03The-Voidwalker-patch-1 13https://github.com/miraheze/mw-config/commit/caf52ca413d416c81ca95204f04f8dea6cf35d33 [05:03:15] 02mw-config/03The-Voidwalker-patch-1 07The-Voidwalker 03caf52ca >:( [05:04:23] miraheze/mw-config - The-Voidwalker the build passed. [11:06:34] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=fantendowiki --no-updates --username-prefix=fandom:fantendo -- fantendo_pages_full.xml (END - exit=0) [11:06:35] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=fantendowiki (START) [11:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:06:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [12:01:59] [1/2] ## Happy Eid Adha Mubarak! [12:02:00] [2/2] ## Who shall slaughter ze sheep? [12:16:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [12:18:33] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [12:26:41] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/92cf5f5bba67b7f70b6f5af4bde3ecf7768af77d [12:26:41] 02puppet/03main 07paladox 0392cf5f5 Update Mattermost version to 11.7.2 [12:40:44] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [12:51:48] PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl [12:52:04] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [12:53:47] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png] [12:56:36] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [12:56:56] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45] [12:58:24] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 03598eae4 to 032f8b4b4 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [12:58:25] 02TSPortal/03renovate/prod-deps 07renovate[bot] 032f8b4b4 Update prod dependencies [12:59:48] RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl [13:00:04] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [13:17:59] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:20:36] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:22:56] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:57:36] [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-13.1.13 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/6a5bb0d21e79 [16:57:36] 02mw-config/03dependabot/composer/phpunit/phpunit-13.1.13 07dependabot[bot] 036a5bb0d Update phpunit/phpunit requirement from 13.1.12 to 13.1.13… [16:57:38] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6427 (Update phpunit/phpunit requirement from 13.1.12 to 13.1.13) 13https://github.com/miraheze/mw-config/pull/6427 [16:57:38] [02mw-config] 07dependabot[bot] opened pull request #6427: Update phpunit/phpunit requirement from 13.1.12 to 13.1.13 (03main...03dependabot/composer/phpunit/phpunit-13.1.13) 13https://github.com/miraheze/mw-config/pull/6427 [16:57:40] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6427 (Update phpunit/phpunit requirement from 13.1.12 to 13.1.13) 13https://github.com/miraheze/mw-config/pull/6427 [16:58:41] miraheze/mw-config - dependabot[bot] the build passed. [19:10:21] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [19:10:43] !log [void@db171] systemctl reset-failed sql-backup && systemctl restart sql-backup [19:10:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:04] [02TSPortal] 07renovate[bot] created 03renovate/dev-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/6471e5910623 [19:14:04] 02TSPortal/03renovate/dev-deps 07renovate[bot] 036471e59 Update dependency phpunit/phpunit to v13.1.13 [20:31:49] [02mw-config] 07The-Voidwalker created 03patch-fallbackContentHandlers (+1 new commit) 13https://github.com/miraheze/mw-config/commit/e65d897e709e [20:31:49] 02mw-config/03patch-fallbackContentHandlers 07The-Voidwalker 03e65d897 add FallbackContentHandler for content models added by non-enabled extensions… [20:32:02] [02mw-config] 07The-Voidwalker opened pull request #6428: add FallbackContentHandler for content models added by non-enabled extensions (03main...03patch-fallbackContentHandlers) 13https://github.com/miraheze/mw-config/pull/6428 [20:32:56] [02mw-config] 07The-Voidwalker closed pull request #6426: add fallback handler for GeoJson content model while maps is disabled (03main...03The-Voidwalker-patch-1) 13https://github.com/miraheze/mw-config/pull/6426 [20:32:57] [02mw-config] 07The-Voidwalker commented on pull request #6426: Handled better by #6428 13https://github.com/miraheze/mw-config/pull/6426#issuecomment-4558439478 [20:33:11] miraheze/mw-config - The-Voidwalker the build has errored. [20:37:27] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03patch-fallbackContentHandlers 13https://github.com/miraheze/mw-config/commit/f825c6856e693f5ad44752bdc5088fb233bca64a [20:37:29] 02mw-config/03patch-fallbackContentHandlers 07The-Voidwalker 03f825c68 add contentModels to test schema [20:38:41] miraheze/mw-config - The-Voidwalker the build has errored. [20:39:37] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03patch-fallbackContentHandlers 13https://github.com/miraheze/mw-config/commit/815a72d6d56d3702e97d68e45cd83908255526f5 [20:39:37] 02mw-config/03patch-fallbackContentHandlers 07The-Voidwalker 03815a72d typo [20:46:38] !log [void@test151] starting deploy of {'config': True} to test151 [20:46:39] !log [void@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [20:46:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:30] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 138 packages available for upgrade (1 critical updates). [20:56:20] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (2 critical updates). [20:58:10] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (2 critical updates). [21:01:24] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (1 critical updates). [21:04:57] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (1 critical updates). [21:05:10] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 138 packages available for upgrade (1 critical updates). [21:07:38] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (1 critical updates). [21:09:39] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (2 critical updates). [21:11:21] [02mw-config] 07The-Voidwalker un-readied pull request #6428: add FallbackContentHandler for content models added by non-enabled extensions (03main...03patch-fallbackContentHandlers) 13https://github.com/miraheze/mw-config/pull/6428 [21:12:31] [02mw-config] 07The-Voidwalker commented on pull request #6428: Not ready for merge. When testing on beta, disabling and enabling an extension that adds a content model (TemplateStyles), the content model (sanitized-css) _remains unavailable_ for reasons I am finding impossible to explain. 13https://github.com/miraheze/mw-config/pull/6428#issuecomment-4558711079 [21:13:12] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (2 critical updates). [21:13:39] !log [void@test151] starting deploy of {'config': True} to test151 [21:13:40] !log [void@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [21:13:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:19:15] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 138 packages available for upgrade (1 critical updates). [21:20:53] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (1 critical updates). [21:29:00] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03patch-fallbackContentHandlers 13https://github.com/miraheze/mw-config/commit/0793f506c0134f58363583a8c01530fe2a9c7a98 [21:29:00] 02mw-config/03patch-fallbackContentHandlers 07The-Voidwalker 030793f50 fix extension active check… [21:30:03] miraheze/mw-config - The-Voidwalker the build passed. [21:47:07] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0afa1cf5e94c977c395740af47d4d13fe9003565 [21:47:07] 02mw-config/03main 07Skye 030afa1cf Revoke createblogpost (BlogPage) in closed wikis… [21:47:54] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:47:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:17] miraheze/mw-config - pskyechology the build passed. [21:48:17] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [21:48:18] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [21:48:20] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [21:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:38] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshLinks --wiki=fantendowiki (START) [22:03:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:04:18] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshLinks --wiki=fantendowiki (END - exit=33280) [22:04:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:24] [02TSPortal] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/6471e5910623f3571d537f20e6d6e68696697d7e [22:05:24] 02TSPortal/03main 07renovate[bot] 036471e59 Update dependency phpunit/phpunit to v13.1.13 [22:05:25] [02TSPortal] 07renovate[bot] 04deleted 03renovate/dev-deps at 036471e59 13https://api.github.com/repos/miraheze/TSPortal/commit/6471e59 [22:05:58] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 032f8b4b4 to 033340f65 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [22:05:59] 02TSPortal/03renovate/prod-deps 07renovate[bot] 033340f65 Update prod dependencies [22:06:19] !log [void@test151] starting deploy of {'config': True} to test151 [22:06:20] !log [void@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:06:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:41] [02mw-config] 07The-Voidwalker readied pull request #6428: add FallbackContentHandler for content models added by non-enabled extensions (03main...03patch-fallbackContentHandlers) 13https://github.com/miraheze/mw-config/pull/6428 [22:09:53] !log [void@test151] starting deploy of {'config': True} to test151 [22:09:54] !log [void@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:09:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:09:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:46] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03patch-fallbackContentHandlers 13https://github.com/miraheze/mw-config/commit/a2db81d3ab3472318d486615fa82a814c519aa2c [22:35:46] 02mw-config/03patch-fallbackContentHandlers 07The-Voidwalker 03a2db81d Merge branch 'main' into patch-fallbackContentHandlers [22:36:57] miraheze/mw-config - The-Voidwalker the build passed. [22:40:40] [02mw-config] 07The-Voidwalker merged pull request #6428: add FallbackContentHandler for content models added by non-enabled extensions (03main...03patch-fallbackContentHandlers) 13https://github.com/miraheze/mw-config/pull/6428 [22:40:42] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/bf3e8af007071bcc200d47ceeff8e14199c6619a [22:40:44] [02mw-config] 07The-Voidwalker 04deleted 03patch-fallbackContentHandlers at 03a2db81d 13https://api.github.com/repos/miraheze/mw-config/commit/a2db81d [22:40:46] 02mw-config/03main 07The-Voidwalker 03bf3e8af add FallbackContentHandler for content models added by non-enabled extensions (#6428)… [22:41:06] !log [void@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:41:07] [02mw-config] 07The-Voidwalker 04deleted 03patch-fallbackContentHandlers at 03a2db81d 13https://api.github.com/repos/miraheze/mw-config/commit/a2db81d [22:41:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:31] !log [void@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [22:41:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:52] miraheze/mw-config - The-Voidwalker the build passed. [22:57:55] !log [void@puppet181] Upgraded packages on cp161: libvarnishapi3 and varnish [22:57:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:07] !log [void@puppet181] Upgraded packages on cloud21: libunbound8 [22:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:28] !log [void@puppet181] Upgraded packages on cp191: libvarnishapi3 and varnish [22:58:29] RECOVERY - cp161 APT on cp161 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [22:58:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:43] !log [void@puppet181] Upgraded packages on cloud15: libunbound8 [22:58:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:52] RECOVERY - cloud21 APT on cloud21 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [22:58:57] !log [void@puppet181] Upgraded packages on cloud17: libunbound8 [22:59:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:12] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 137 packages available for upgrade (0 critical updates). [22:59:21] !log [void@puppet181] Upgraded packages on cp171: libvarnishapi3 and varnish [22:59:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:35] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [22:59:46] !log [void@puppet181] Upgraded packages on cp201: libvarnishapi3 and varnish [22:59:58] !log [void@puppet181] Upgraded packages on cloud20: libunbound8 [23:00:12] !log [void@puppet181] Upgraded packages on cloud18: libunbound8 [23:00:16] RECOVERY - cp191 APT on cp191 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [23:00:18] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [23:00:24] !log [void@puppet181] Upgraded packages on cloud16: libunbound8 [23:00:35] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 137 packages available for upgrade (0 critical updates). [23:00:36] !log [void@puppet181] Upgraded packages on cloud19: libunbound8 [23:00:40] PROBLEM - cp191 Varnish Backends on cp191 is WARNING: No backends detected. If this is an error, see readme.txt [23:00:41] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 137 packages available for upgrade (0 critical updates). [23:00:45] RECOVERY - cp201 APT on cp201 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [23:00:51] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [23:00:54] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:01:08] RECOVERY - cp171 APT on cp171 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [23:01:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [23:01:22] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [23:01:22] PROBLEM - cp171 Varnish Backends on cp171 is WARNING: No backends detected. If this is an error, see readme.txt [23:01:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [23:01:32] PROBLEM - cp161 Varnish Backends on cp161 is WARNING: No backends detected. If this is an error, see readme.txt [23:01:39] PROBLEM - cp191 health.wikitide.net HTTPS on cp191 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.010 second response time [23:02:00] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [23:02:02] PROBLEM - cp201 health.wikitide.net HTTPS on cp201 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.010 second response time [23:02:40] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [23:02:49] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [23:02:54] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1514 bytes in 0.091 second response time [23:03:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:03:22] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [23:03:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:03:32] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [23:03:38] RECOVERY - cp191 health.wikitide.net HTTPS on cp191 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.009 second response time [23:04:00] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.083 second response time [23:04:01] RECOVERY - cp201 health.wikitide.net HTTPS on cp201 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.016 second response time [23:04:18] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 10% [23:05:52] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 12.60, 7.50, 4.10 [23:06:19] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is CRITICAL: LOAD CRITICAL - total load average: 10.31, 6.58, 3.81 [23:08:00] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats [23:09:48] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:11:48] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.563 second response time [23:14:59] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:07] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 8.898 second response time [23:20:00] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats [23:20:19] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is WARNING: LOAD WARNING - total load average: 5.23, 7.81, 7.24 [23:20:41] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:19] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is CRITICAL: LOAD CRITICAL - total load average: 13.12, 9.65, 7.96 [23:22:39] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.647 second response time [23:24:00] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats [23:30:19] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is WARNING: LOAD WARNING - total load average: 5.49, 7.68, 7.97 [23:30:49] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [23:35:23] !log ban an IP in cloudflare for odd behavior related to static [23:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:00] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats [23:41:52] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 6.07, 6.78, 7.96 [23:42:19] RECOVERY - swiftproxy171 Current Load on swiftproxy171 is OK: LOAD OK - total load average: 3.54, 5.63, 6.78 [23:49:52] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 3.15, 4.98, 6.64