[00:01:00] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/89ddebbfe15eae8911d0fc32030c7136e6014ad7 [00:01:00] 02statichelp/03main 07WikiTideBot 0389ddebb Bot: Auto-update Tech namespace pages 2026-02-25 00:00:58 [00:11:41] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/61e15b3d1c6790234e1b7393850bb8a388984fdf [00:11:41] 02mw-config/03main 07Peter Li 0361e15b3 T14964: Add MultimediaViewer as a default extension (#6312)… [00:11:43] [02mw-config] 07lihaohong6 merged pull request #6312: T14964: Add MultimediaViewer as a default extension (07miraheze:03main...07lihaohong6:03T14964) 13https://github.com/miraheze/mw-config/pull/6312 [00:11:50] !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:11:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:16] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [00:12:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e3d63c29c0fa01de75b0d0656c56883377fad640 [00:12:17] 02puppet/03main 07CosmicAlpha 03e3d63c2 mwdeploy: don't save patch rejects [00:12:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:54] miraheze/mw-config - lihaohong6 the build passed. [00:13:43] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=ss14wiki --setting=wgLocaltimezone --sourcelist=./sourcelist.txt (END - exit=65280) [00:13:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:51] miraheze/puppet - Universal-Omega the build passed. [00:16:03] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawiki --setting=wgLocaltimezone --sourcelist=./sourcelist.txt (END - exit=0) [00:16:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:32] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 82126MiB (18.1% inode=100%); [00:20:19] !log [salt-user@mw161] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:20] !log [salt-user@mw172] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:21] !log [salt-user@mw152] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:22] !log [salt-user@mw171] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:23] !log [salt-user@mw151] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:24] !log [salt-user@mw162] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:25] !log [salt-user@mw153] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:26] !log [salt-user@mw191] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:27] !log [salt-user@mw173] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:28] !log [salt-user@mwtask161] rm /srv/mediawiki/cache/*ss14wiki*.php [00:20:29] !log [salt-user@mw201] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:30] !log [salt-user@mwtask151] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:31] !log [salt-user@mw193] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:32] !log [salt-user@mw163] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:32] ugh whoops. Did not mean to do that. [00:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:33] !log [salt-user@mw181] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:34] !log [salt-user@mw202] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:35] !log [salt-user@mw203] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:36] !log [salt-user@mw192] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:37] !log [salt-user@mwtask171] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:38] !log [salt-user@mw183] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:39] !log [salt-user@mwtask181] rm /srv/mediawiki/cache/*ss14wiki*.php [00:20:40] !log [salt-user@mw182] rm /srv/mediawiki/cache/config-ss14wiki.php /srv/mediawiki/cache/ss14wiki.php [00:20:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:21] I meant to logsalmsg on puppet only not in salt... well... should probably edit SAL to remove that. [00:26:57] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/82b23c7b22af14a65f184d61a157c1ff334b3054 [00:26:57] 02mediawiki-repos/03main 07CosmicAlpha 0382b23c7 Document that non-git repos must use discard [00:39:43] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=cavewiki --sleep=1 '--comment=Importing images from https://speleo.miraheze.org ([[phorge:T14886|T14886]])' -- images/ (END - exit=0) [00:39:44] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=cavewiki --update (START) [00:39:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:45] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=cavewiki --update (END - exit=0) [00:39:46] !log [blankeclair@mwtask161] Finished import for cavewiki (XML: None; Images: images/) (END - exit=0) [00:39:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:02] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=animatorvsanimationwiki (END - exit=0) [00:40:03] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=animatorvsanimationwiki (START) [00:40:04] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=animatorvsanimationwiki (END - exit=0) [00:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:05] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=animatorvsanimationwiki --update (START) [00:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:06] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=animatorvsanimationwiki --update (END - exit=0) [00:40:07] !log [blankeclair@mwtask171] Finished import for animatorvsanimationwiki (XML: animatorvsanimation_pages_full.xml?dl=1&noscript=1&rlkey=2srmcxqq8utphzlhejttan4wn&st=gxhjxdyr; Images: None) (END - exit=0) [00:40:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:46:52] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44513MiB (9.8% inode=100%); [00:56:52] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 82375MiB (18.2% inode=100%); [01:02:39] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44379MiB (9.8% inode=100%); [01:12:39] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 82775MiB (18.2% inode=100%); [01:14:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/06abb2232658ca55ebf9bac5d2875fb7d8024cc8 [01:14:54] 02puppet/03main 07CosmicAlpha 0306abb22 mwdeploy: more clear path logging [01:16:27] miraheze/puppet - Universal-Omega the build passed. [01:20:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/db25ecb427f238e3be864cbaa00952b1c4b34e38 [01:20:51] 02puppet/03main 07CosmicAlpha 03db25ecb Fix permissions for mwdeploy to access patches [01:25:58] !log move patch for T14297 to the private patch repo [01:26:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:02] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [02:50:02] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [03:02:09] PROBLEM - os201 PowerDNS Recursor on os201 is CRITICAL: No response from DNS 127.0.0.1 [03:02:12] PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: No response from DNS 127.0.0.1 [03:03:49] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:04:06] RECOVERY - os201 PowerDNS Recursor on os201 is OK: DNS OK: 0.045 seconds response time. os201.fsslc.wtnet returns 10.0.20.156 [03:04:07] RECOVERY - bast161 PowerDNS Recursor on bast161 is OK: DNS OK: 0.306 seconds response time. bast161.wikitide.net returns 2602:294:0:b13::101,38.46.223.203 [03:05:47] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:31:51] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [03:33:47] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:16] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.55, 11.97, 6.48 [04:26:16] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.96, 8.72, 5.96 [05:01:53] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-special-pages [15:34:13] PROBLEM - graylog161 graylog tls port 12210 ssl cert check on graylog161 is WARNING: WARNING - Certificate 'wikitide.net' expires in 15 day(s) (Fri Mar 13 15:30:27 2026 +0000). [17:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/59161ea3132447b414b0a96f04eed77ca7cb1928 [17:00:26] 02ssl/03main 07WikiTideBot 0359161ea Bot: Auto-update domain lists [17:03:27] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772035370000&orgId=1&to=1772039007592 [17:08:27] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772035370000&orgId=1&to=1772039090000 [17:08:59] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772035640000&orgId=1&to=1772039339008 [17:13:42] [02ManageWiki] 07Universal-Omega created 03type-constants (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/cc061782e9d9 [17:13:42] 02ManageWiki/03type-constants 07CosmicAlpha 03cc06178 Type hint constants [17:13:49] [02ManageWiki] 07Universal-Omega opened pull request #772: Type hint constants (03main...03type-constants) 13https://github.com/miraheze/ManageWiki/pull/772 [17:13:59] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772035640000&orgId=1&to=1772039420000 [17:14:06] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/1414e7156ca86156f213a68fa870e4ff9055b03c [17:14:06] 02ManageWiki/03type-constants 07CosmicAlpha 031414e71 Bump [17:14:29] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/f38105e33eff6622d656829607fad50634c7411d [17:14:30] 02ManageWiki/03type-constants 07CosmicAlpha 03f38105e Bump [17:14:59] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036000000&orgId=1&to=1772039699102 [17:15:31] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/810f1d3b5f0b59f144b1c1f1e0de65effd78970f [17:15:31] 02ManageWiki/03type-constants 07CosmicAlpha 03810f1d3 Type [17:16:35] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/191f9c8fcf1c4f4f1b52ed01814f2e074f607d41 [17:16:35] 02ManageWiki/03type-constants 07CosmicAlpha 03191f9c8 Type [17:16:52] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.57, 7.03, 3.96 [17:17:20] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/a75afb7590f65358603ad91944f6ce9a3aeffce6 [17:17:20] 02ManageWiki/03type-constants 07CosmicAlpha 03a75afb7 Type [17:17:31] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:18:07] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:32] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/27bdd2fde483ce22b2cd4a72b517414ec304fe78 [17:18:32] 02ManageWiki/03type-constants 07CosmicAlpha 0327bdd2f Type [17:18:57] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:05] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/689ed9cb36195952e7108408b30e2f41147a9f1a [17:19:05] 02ManageWiki/03type-constants 07CosmicAlpha 03689ed9c Type [17:19:35] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/75ca2bc066930435e6c6f5e22afa5350cd9a8d7c [17:19:35] 02ManageWiki/03type-constants 07CosmicAlpha 0375ca2bc Type [17:19:53] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/1cb2cfe201df7adc26707d3113132bb6583634b5 [17:19:53] 02ManageWiki/03type-constants 07CosmicAlpha 031cb2cfe Type [17:19:59] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036000000&orgId=1&to=1772039990000 [17:20:27] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/4958a7df3aa4b29346b447a7e0e8375f48a34287 [17:20:27] 02ManageWiki/03type-constants 07CosmicAlpha 034958a7d Type [17:20:53] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:59] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036000000&orgId=1&to=1772040059681 [17:21:02] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:21:02] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [17:21:14] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/f68c0d8832c8b4ee059d60caa365d854e6aa933a [17:21:14] 02ManageWiki/03type-constants 07CosmicAlpha 03f68c0d8 Type [17:21:32] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.201 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:21:42] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/96d7120c4dde26329ab5a3a96501449412206154 [17:21:42] 02ManageWiki/03type-constants 07CosmicAlpha 0396d7120 Type [17:22:06] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/31de001b34a25a888eba1dc72b06346af12cf073 [17:22:06] 02ManageWiki/03type-constants 07CosmicAlpha 0331de001 Type [17:22:29] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/87663e9e2c914515352f17acf3e20ba9f17240f2 [17:22:29] 02ManageWiki/03type-constants 07CosmicAlpha 0387663e9 Type [17:22:51] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [17:23:16] PROBLEM - mw172 SSH on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:32] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw172 [17:23:43] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 8.860 second response time [17:23:49] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/cde91cb9d0d9f1d2448835ea95f35b0792a6adc8 [17:23:49] 02ManageWiki/03type-constants 07CosmicAlpha 03cde91cb Type [17:23:52] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw172 [17:24:21] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03type-constants 13https://github.com/miraheze/ManageWiki/commit/6249ac1c62f8d5a8fc8258e2ce79448ad3d1945d [17:24:21] 02ManageWiki/03type-constants 07CosmicAlpha 036249ac1 Type [17:24:53] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.13, 7.00, 5.80 [17:25:02] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.53, 20.47, 11.79 [17:25:13] RECOVERY - mw172 SSH on mw172 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [17:25:32] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [17:25:39] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.260 second response time [17:25:52] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [17:25:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036570000&orgId=1&to=1772040170000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772040359681[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [17:25:59] 81[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036000000&orgId=1&to=1772040230000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036540000&orgId=1&to=1772040230000 [17:26:51] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 7.59, 6.15 [17:26:57] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.50, 15.16, 10.87 [17:29:54] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:29:57] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:08] miraheze/ManageWiki - Universal-Omega the build passed. [17:31:53] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [17:31:59] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:35:27] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.83, 6.45, 6.71 [17:39:25] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.59, 8.55, 7.48 [17:40:45] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:40:57] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:41:47] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:58] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:42:10] promethetus r u good [17:43:41] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [17:43:42] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 38 minutes ago with 0 failures [17:43:42] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:43:53] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.195 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:44:02] i guess you are [17:45:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772037890000&orgId=1&to=1772041490000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772041559684[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [17:45:59] 84[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772037380000&orgId=1&to=1772041559684[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772037890000&orgId=1&to=1772041559684 [17:46:22] [02ManageWiki] 07Universal-Omega created 03no-null-conf (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/61ed28d8881c [17:46:22] 02ManageWiki/03no-null-conf 07CosmicAlpha 0361ed28d If ManageWikiSiteConfiguration is null, don't try and use it [17:46:29] [02ManageWiki] 07Universal-Omega opened pull request #773: If ManageWikiSiteConfiguration is null, don't try and use it (03main...03no-null-conf) 13https://github.com/miraheze/ManageWiki/pull/773 [17:47:34] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.21, 6.37, 7.46 [17:48:39] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772037890000&orgId=1&to=1772041490000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772041859684[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [17:50:59] 84[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038070000&orgId=1&to=1772041790000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772037890000&orgId=1&to=1772041610000 [17:51:30] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.63, 8.10, 7.86 [17:52:09] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:52:33] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:46] miraheze/ManageWiki - Universal-Omega the build passed. [17:52:53] [02ManageWiki] 07Universal-Omega merged pull request #773: If ManageWikiSiteConfiguration is null, don't try and use it (03main...03no-null-conf) 13https://github.com/miraheze/ManageWiki/pull/773 [17:52:55] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/d2e8126243bc4042016bafd9cca7c3818183f7f5 [17:52:57] [02ManageWiki] 07Universal-Omega 04deleted 03no-null-conf at 0361ed28d 13https://api.github.com/repos/miraheze/ManageWiki/commit/61ed28d [17:52:58] 02ManageWiki/03main 07CosmicAlpha 03d2e8126 If ManageWikiSiteConfiguration is null, don't try and use it [17:53:24] [02ManageWiki] 07Universal-Omega 04force-pushed 03strict_types from 0379835ab to 0370c565d 13https://api.github.com/repos/miraheze/ManageWiki/commits/strict_types [17:53:24] 02ManageWiki/03strict_types 07CosmicAlpha 0370c565d Use strict_types [17:53:47] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:28] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.283 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [17:54:32] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:55:41] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [17:55:51] miraheze/ManageWiki - Universal-Omega the build has errored. [17:55:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038430000&orgId=1&to=1772042150000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772042159685[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [17:55:59] 85[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038070000&orgId=1&to=1772042159685[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038550000&orgId=1&to=1772042159685 [17:59:13] miraheze/ManageWiki - Universal-Omega the build passed. [17:59:26] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.07, 6.14, 7.52 [18:00:26] [02ManageWiki] 07Universal-Omega merged pull request #772: Type hint constants (03main...03type-constants) 13https://github.com/miraheze/ManageWiki/pull/772 [18:00:26] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/ef10f1c45c1142d007065e2219c05d97d6cbf5ad [18:00:27] 02ManageWiki/03main 07CosmicAlpha 03ef10f1c Type hint constants [18:00:28] [02ManageWiki] 07Universal-Omega 04deleted 03type-constants at 036249ac1 13https://api.github.com/repos/miraheze/ManageWiki/commit/6249ac1 [18:00:44] [02ManageWiki] 07Universal-Omega 04force-pushed 03strict_types from 0370c565d to 035d0edbb 13https://api.github.com/repos/miraheze/ManageWiki/commits/strict_types [18:00:44] 02ManageWiki/03strict_types 07CosmicAlpha 035d0edbb Use strict_types [18:00:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038790000&orgId=1&to=1772042390000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772042459686[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [18:01:00] 86[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038670000&orgId=1&to=1772042360000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038550000&orgId=1&to=1772042360000 [18:03:20] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.37, 6.43, 7.20 [18:05:18] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.49, 5.25, 6.68 [18:05:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772039090000&orgId=1&to=1772042690000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772042759687[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [18:05:59] 87[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772038880000&orgId=1&to=1772042759687[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772039090000&orgId=1&to=1772042759687 [18:06:36] miraheze/ManageWiki - Universal-Omega the build passed. [18:07:02] miraheze/ManageWiki - Universal-Omega the build passed. [18:07:23] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:10:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772039270000&orgId=1&to=1772042870000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772043059688[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [18:11:00] 88[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772039180000&orgId=1&to=1772042870000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772039090000&orgId=1&to=1772042870000 [18:15:02] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 12.04, 8.13, 6.91 [18:18:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.11, 7.23, 7.05 [18:20:55] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.20, 7.90, 7.27 [18:24:51] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.84, 6.37, 6.89 [18:26:51] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.14, 5.18, 6.38 [18:30:59] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772044259691[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772044259691[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time p [18:30:59] s://grafana.wikitide.net/d/GtxbP1Xnk?from=1772040410000&orgId=1&to=1772044259691 [18:35:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772040710000&orgId=1&to=1772044310000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772044559692[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [18:35:59] 92[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772040830000&orgId=1&to=1772044550000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772040710000&orgId=1&to=1772044520000 [18:37:52] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 13.34, 9.36, 7.58 [18:38:21] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:39:47] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:39:47] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:56] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:43:25] PROBLEM - prometheus151 ferm_active on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:07] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [18:45:10] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:45:10] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 39 minutes ago with 0 failures [18:45:15] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.195 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:45:21] RECOVERY - prometheus151 ferm_active on prometheus151 is OK: OK ferm input default policy is set [18:45:59] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041250000&orgId=1&to=1772045159694[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772045159694[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from [18:45:59] 0000&orgId=1&to=1772045159694[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772040830000&orgId=1&to=1772045159694 [18:46:58] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [18:50:59] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041610000&orgId=1&to=1772045210000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772045459694[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1 [18:50:59] 94[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041730000&orgId=1&to=1772045450000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041580000&orgId=1&to=1772045270000 [18:51:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:54:01] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:54:03] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:56:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:08] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:22] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [18:58:33] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.316 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116 [18:58:58] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:59:00] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [18:59:02] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [18:59:15] [02puppet] 07Universal-Omega created 03firejail-magick (+1 new commit) 13https://github.com/miraheze/puppet/commit/69fc77189e5a [18:59:15] 02puppet/03firejail-magick 07CosmicAlpha 0369fc771 Rename mediawiki-firejail-convert to match ImageMagick 7 [18:59:22] [02puppet] 07Universal-Omega opened pull request #4792: Rename mediawiki-firejail-convert to match ImageMagick 7 (03main...03firejail-magick) 13https://github.com/miraheze/puppet/pull/4792 [19:00:59] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772042090000&orgId=1&to=1772046059696[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772046059696[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from [19:00:59] 0000&orgId=1&to=1772046059696[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041730000&orgId=1&to=1772046059696 [19:01:56] PROBLEM - prometheus151 Prometheus on prometheus151 is CRITICAL: connect to address 10.0.15.116 and port 9090: Connection refused [19:02:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03firejail-magick 13https://github.com/miraheze/puppet/commit/63ec6f455134e3fdc849073972a03a12f76da357 [19:02:13] 02puppet/03firejail-magick 07CosmicAlpha 0363ec6f4 Update [19:08:08] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.00, 2.63, 7.39 [19:10:05] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.00, 1.78, 6.53 [19:20:59] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772043530000&orgId=1&to=1772047259699[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772042090000&orgId=1&to=1772047259699[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/G [19:20:59] rom=1772036460000&orgId=1&to=1772047259699[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772047259699[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041730000&orgId=1&to=1772047259699 [19:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/45488236d15ed88512938bedf85513124f135572 [19:30:23] 02ssl/03main 07WikiTideBot 034548823 Bot: Auto-update domain lists [19:30:28] [02puppet] 07Universal-Omega created 03prometheus-drop-path (+1 new commit) 13https://github.com/miraheze/puppet/commit/a700805cc4f1 [19:30:28] 02puppet/03prometheus-drop-path 07CosmicAlpha 03a700805 prometheus: drop path metrics in statsd_exporter [19:30:35] [02puppet] 07Universal-Omega opened pull request #4793: prometheus: drop path metrics in statsd_exporter (03main...03prometheus-drop-path) 13https://github.com/miraheze/puppet/pull/4793 [19:37:56] RECOVERY - prometheus151 Prometheus on prometheus151 is OK: TCP OK - 0.000 second response time on 10.0.15.116 port 9090 [19:40:59] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772044730000&orgId=1&to=1772048330000[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772044730000&orgId=1&to=1772048330000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772048459702[Grafana] [19:40:59] MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772048459702[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772041730000&orgId=1&to=1772048390000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772044700000&orgId=1&to=1772048450000 [19:55:40] [02puppet] 07Universal-Omega created 03prometheus-remove-cadvisor (+1 new commit) 13https://github.com/miraheze/puppet/commit/835a21d79406 [19:55:42] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 03835a21d prometheus: remove cadvisor [19:55:48] [02puppet] 07Universal-Omega opened pull request #4794: prometheus: remove cadvisor (03main...03prometheus-remove-cadvisor) 13https://github.com/miraheze/puppet/pull/4794 [19:56:23] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-remove-cadvisor 13https://github.com/miraheze/puppet/commit/a410f8771b4812298eee418f18ab84559fe01910 [19:56:23] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 03a410f87 Remove [19:56:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-remove-cadvisor 13https://github.com/miraheze/puppet/commit/b521df601ad6606ccced3548490188254f49af20 [19:56:43] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 03b521df6 Remove [19:57:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-remove-cadvisor 13https://github.com/miraheze/puppet/commit/1ad4f5e87e0f17527aa18d06211e5f985a7cf61b [19:57:22] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 031ad4f5e Remove [19:57:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-remove-cadvisor 13https://github.com/miraheze/puppet/commit/db1570a2230b8661c73ee8a20ebfe385f1cf72b1 [19:57:40] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 03db1570a Remove [19:57:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-remove-cadvisor 13https://github.com/miraheze/puppet/commit/cccd95bb0671e606d68bc370fa879f9f48c56622 [19:57:58] 02puppet/03prometheus-remove-cadvisor 07CosmicAlpha 03cccd95b Remove [19:59:16] [02puppet] 07Universal-Omega merged pull request #4794: prometheus: remove cadvisor (03main...03prometheus-remove-cadvisor) 13https://github.com/miraheze/puppet/pull/4794 [19:59:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/712740c62bdcbcb2c2e5633713d453925c2b51c9 [19:59:17] 02puppet/03main 07CosmicAlpha 03712740c prometheus: remove cadvisor [19:59:19] [02puppet] 07Universal-Omega 04deleted 03prometheus-remove-cadvisor at 03cccd95b 13https://api.github.com/repos/miraheze/puppet/commit/cccd95b [20:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9be562966fca8458ad415cbe6c7c89640a4c5a5f [20:00:26] 02ssl/03main 07WikiTideBot 039be5629 Bot: Auto-update domain lists [20:12:20] [02puppet] 07Universal-Omega created 03prometheus-drop (+1 new commit) 13https://github.com/miraheze/puppet/commit/86ed144b6fb4 [20:12:20] 02puppet/03prometheus-drop 07CosmicAlpha 0386ed144 prometheus: drop mediawiki_action_api_modules_latency_.* metrics [20:12:28] [02puppet] 07Universal-Omega opened pull request #4795: prometheus: drop mediawiki_action_api_modules_latency_.* metrics (03main...03prometheus-drop) 13https://github.com/miraheze/puppet/pull/4795 [20:15:59] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772050260000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772036460000&orgId=1&to=1772050260000 [20:19:02] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/84b5d65c876eeb6c41dd6fcd5a926c6888e525bb [20:19:02] 02puppet/03main 07CosmicAlpha 0384b5d65 prometheus: drop mediawiki_action_api_modules_latency_.* metrics [20:19:02] [02puppet] 07Universal-Omega merged pull request #4795: prometheus: drop mediawiki_action_api_modules_latency_.* metrics (03main...03prometheus-drop) 13https://github.com/miraheze/puppet/pull/4795 [20:19:04] [02puppet] 07Universal-Omega 04deleted 03prometheus-drop at 0386ed144 13https://api.github.com/repos/miraheze/puppet/commit/86ed144 [20:20:32] [02puppet] 07Universal-Omega pushed 1 new commit to 03prometheus-drop-path 13https://github.com/miraheze/puppet/commit/4e4b8bb21e96f17454f9998a7bfcd68162fe8d66 [20:20:32] 02puppet/03prometheus-drop-path 07CosmicAlpha 034e4b8bb Merge branch 'main' into prometheus-drop-path [20:21:00] [02puppet] 07Universal-Omega commented on pull request #4793: Another option if #4795 doesn't help. 13https://github.com/miraheze/puppet/pull/4793#issuecomment-3961812437 [20:22:28] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:55] [02puppet] 07Universal-Omega merged pull request #4792: Rename mediawiki-firejail-convert to match ImageMagick 7 (03main...03firejail-magick) 13https://github.com/miraheze/puppet/pull/4792 [20:44:57] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/552446155049567cbd1f8b9dbec2802727d5aa00 [20:44:59] [02puppet] 07Universal-Omega 04deleted 03firejail-magick at 0363ec6f4 13https://api.github.com/repos/miraheze/puppet/commit/63ec6f4 [20:45:01] 02puppet/03main 07CosmicAlpha 035524461 Rename mediawiki-firejail-convert to match ImageMagick 7 [20:46:33] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0e77f009e4e5c1e6608346608df3982b6024b3a4 [20:46:33] 02mw-config/03main 07CosmicAlpha 030e77f00 Use mediawiki-firejail-magick for ImageMagick [20:47:47] miraheze/mw-config - Universal-Omega the build passed. [20:53:45] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/mediawiki-firejail-convert] [20:59:41] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [21:00:25] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/efa978425e603b6afc1d6702230ac30bc2efffc7 [21:00:25] 02ssl/03main 07WikiTideBot 03efa9784 Bot: Auto-update domain lists [21:08:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/96f871cbaf937d7c3b1803bfb31b92406d693e57 [21:08:23] 02puppet/03main 07CosmicAlpha 0396f871c cache: lower file size to 345G [21:15:17] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 11.99, 6.55, 3.61 [21:16:40] PROBLEM - swiftproxy171 Current Load on swiftproxy171 is CRITICAL: LOAD CRITICAL - total load average: 15.02, 9.47, 4.93 [21:16:51] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.79, 7.22, 4.17 [21:17:12] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 5.72, 6.79, 4.10 [21:20:15] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:08] @blankeclair ^ probably this is why [21:21:09] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 12.76, 9.13, 5.58 [21:21:16] oh [21:22:15] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.410 second response time [21:22:20] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats [21:22:41] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.10, 2.16, 0.75 [21:22:48] RECOVERY - swiftproxy171 Current Load on swiftproxy171 is OK: LOAD OK - total load average: 6.35, 2.09, 0.74 [21:22:52] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 2.55, 0.89 [21:22:52] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.99, 2.60, 0.92 [21:23:22] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 8.40, 3.19, 1.17 [21:25:12] @cosmicalpha why has there been no communication in regards to swift changes happening (i.e potential downtime/errors)? or did I miss it? [21:25:46] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 5.50, 4.30, 1.90 [21:25:50] This was not intentionally planned. I am looking into it now. [21:26:25] !log reboot swiftproxy * [21:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:03] Also looking into why this has happened two days in a row. [21:27:37] oh damn I was right about surprise downtimes later on from trixie upgrades [21:28:39] Seems nginx is OOMing on both servers continually for a bit and memcached is and swiftproxy171 also. [21:29:37] Highly doubt it's related tbh. If it were it wouldnt start weeks after they were upgraded. [21:31:03] fuck has it already been weeks since mw* upgrades [21:31:45] Well this isnt an mw issue its a swift issue on swiftproxy* [21:32:33] Been more than a month since swift was upgraded now. [21:34:34] I have been fighting servers OOMing all day though. First prometheus and now swift. [21:34:58] [1/3] true but it started two days ago and we upgraded mw* two days ago [21:34:58] [2/3] correlation is not causation but it is an alright uneducated guess [21:34:59] [3/3] anyway, wishing you strength in figuring it out <3 [21:36:48] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats [21:39:37] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.14, 6.69, 4.61 [21:41:36] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 6.47, 6.71, 4.88 [21:48:22] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:48:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:46] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [21:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:27] [02mw-config] 07Universal-Omega commented on pull request #6320: @dependabot rebase 13https://github.com/miraheze/mw-config/pull/6320#issuecomment-3962324442 [21:55:10]