[00:04:22] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 58674MiB (12% inode=99%); [00:07:05] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58792MiB (12% inode=99%); [00:38:09] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [00:38:10] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 92001 --e 94000 (running for 59:25) [00:38:11] mwtask161: not running [00:38:12] mwtask171: not running [00:38:13] mwtask181: not running [01:38:11] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [01:38:12] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 92001 --e 94000 (running for 01:59:27) [01:38:13] mwtask161: not running [01:38:14] mwtask171: not running [01:38:16] mwtask181: not running [01:40:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:45:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:25:47] [02mw-config] 07OAuthority commented on pull request #5894: LGTM. Please can you keep the IPs in the same order as listed on the Cloudflare IP page so that it is easier for us to check the PR in future against the list on Cloudflare's website. I have amended this PR to reflect this :). 13https://github.com/miraheze/mw-config/pull/5894#issuecomment-2780147014 [02:26:55] [02mw-config] 07Aeywoo commented on pull request #5894: It is in format to the Cloudflare IP Files. 13https://github.com/miraheze/mw-config/pull/5894#issuecomment-2780147492 [02:29:01] miraheze/mw-config - OAuthority the build passed. [02:29:20] [02mw-config] 07OAuthority merged 07Aeywoo's pull request #5894: Updating Cloudflare's IPs. (07miraheze:03master...07Aeywoo:03patch-9) 13https://github.com/miraheze/mw-config/pull/5894 [02:29:20] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/2e67a064483df2e3c0f9f67ac6a4397b0aba8506 [02:29:22] 02mw-config/03master 07Aeywoo 032e67a06 Updating Cloudflare's IPs. (#5894)… [02:29:44] [02mw-config] 07OAuthority created 03revert-5894-patch-9 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/ae058e83e192 [02:29:44] 02mw-config/03revert-5894-patch-9 07Luke 03ae058e8 Revert "Updating Cloudflare's IPs. (#5894)"… [02:29:52] [02mw-config] 07OAuthority opened pull request #5897: Revert "Updating Cloudflare's IPs." (03master...03revert-5894-patch-9) 13https://github.com/miraheze/mw-config/pull/5897 [02:30:14] miraheze/mw-config - OAuthority the build passed. [02:30:19] [02mw-config] 07OAuthority merged pull request #5897: Revert "Updating Cloudflare's IPs." (03master...03revert-5894-patch-9) 13https://github.com/miraheze/mw-config/pull/5897 [02:30:21] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/69cbdaea2b616dd32a469d3a9c7a51a1c535848c [02:30:22] 02mw-config/03master 07Luke 0369cbdae Revert "Updating Cloudflare's IPs." (#5897)… [02:30:24] [02mw-config] 07OAuthority 04deleted 03revert-5894-patch-9 at 03ae058e8 13https://github.com/miraheze/mw-config/commit/ae058e8 [02:30:44] miraheze/mw-config - OAuthority the build passed. [02:31:12] miraheze/mw-config - OAuthority the build passed. [02:34:23] [02mw-config] 07OAuthority commented on pull request #5894: I'm slightly confused -- looking again at this, there seems to be no difference between the original and your change, other than you moved the IPs around? 13https://github.com/miraheze/mw-config/pull/5894#issuecomment-2780150895 [02:36:30] [02mw-config] 07OAuthority merged 07Friskygote's pull request #5892: Update Super Animal Royale license badge (07miraheze:03master...07Friskygote:03master) 13https://github.com/miraheze/mw-config/pull/5892 [02:36:31] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/86d3e0c871e5f9558a75109167413db4c8a4808d [02:36:31] 02mw-config/03master 07Frisk 0386d3e0c Update Super Animal Royale license badge (#5892)… [02:37:25] miraheze/mw-config - OAuthority the build passed. [02:38:14] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [02:38:15] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 94001 --e 96000 (running for 49:59) [02:38:16] mwtask161: not running [02:38:17] mwtask171: not running [02:38:18] mwtask181: not running [03:00:50] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 4 seconds old and 2963330 bytes [03:01:03] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 2 seconds old and 3206279 bytes [03:02:56] PROBLEM - db172 Backups SQL on db172 is WARNING: FILE_AGE WARNING: /var/log/db-backups/db-backups/db-backups.log is 1382574 seconds old and 1953 bytes [03:38:16] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [03:38:17] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 94001 --e 96000 (running for 01:50:01) [03:38:18] mwtask161: not running [03:38:19] mwtask171: not running [03:38:20] mwtask181: not running [04:38:18] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [04:38:19] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 96001 (running for 56:49) [04:38:20] mwtask161: not running [04:38:21] mwtask171: not running [04:38:22] mwtask181: not running [05:38:21] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [05:38:22] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ysmwikiwiki 96001 (running for 01:56:51) [05:38:23] mwtask161: not running [05:38:24] mwtask171: not running [05:38:25] mwtask181: not running [06:38:23] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [06:38:24] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 23:31) [06:38:25] mwtask161: not running [06:38:26] mwtask171: not running [06:38:27] mwtask181: not running [07:38:25] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [07:38:26] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 01:23:34) [07:38:27] mwtask161: not running [07:38:28] mwtask171: not running [07:38:29] mwtask181: not running [07:53:50] PROBLEM - you.r-fit.cc - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:38:27] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [08:38:28] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 02:23:36) [08:38:29] mwtask161: not running [08:38:30] mwtask171: not running [08:38:31] mwtask181: not running [09:22:29] RECOVERY - you.r-fit.cc - Cloudflare on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Sat 24 May 2025 05:41:10 PM GMT +0000. [09:38:30] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [09:38:31] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 03:23:38) [09:38:32] mwtask161: not running [09:38:33] mwtask171: not running [09:38:34] mwtask181: not running [10:27:56] [02mw-config] 07RhinosF1 assigned pull request #5895 to 07BlankEclair (Re-enabling Skin:Evelution per T13471) 13https://github.com/miraheze/mw-config/pull/5895 [10:38:32] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [10:38:33] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 04:23:40) [10:38:34] mwtask161: not running [10:38:35] mwtask171: not running [10:38:36] mwtask181: not running [11:22:27] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:22:39] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:24:06] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (2 critical updates). [11:24:44] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:24:53] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:25:53] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [11:25:54] PROBLEM - cp38 APT on cp38 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:25:55] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates). [11:26:21] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:26:58] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:27:12] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:27:23] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:27:24] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:27:41] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:28:42] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:28:56] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:29:35] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [11:29:41] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:29:43] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates). [11:29:48] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:29:57] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [11:30:10] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:30:24] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:31:18] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:31:26] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:31:27] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (2 critical updates). [11:31:47] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:31:51] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:32:15] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:32:17] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:32:18] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:32:21] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (2 critical updates). [11:33:11] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:33:41] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:33:51] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:34:05] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:34:23] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:34:29] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (2 critical updates). [11:34:41] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (2 critical updates). [11:35:59] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:36:14] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (2 critical updates). [11:36:58] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (2 critical updates). [11:37:53] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:38:17] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:38:34] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [11:38:35] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 05:23:43) [11:38:36] mwtask161: not running [11:38:37] mwtask171: not running [11:38:38] mwtask181: not running [11:39:28] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:40:15] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:40:24] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:40:39] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:41:05] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:41:53] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:42:12] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:42:17] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [11:42:22] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:42:28] Oh do one icinga [11:43:03] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (2 critical updates). [11:45:03] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates). [11:45:29] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:46:39] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:46:51] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [11:47:00] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:47:05] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (2 critical updates). [11:47:23] PROBLEM - changeprop201 APT on changeprop201 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [11:48:15] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:48:27] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:48:33] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (2 critical updates). [11:48:57] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:49:08] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:49:17] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (2 critical updates). [11:49:59] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:50:22] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:50:38] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (2 critical updates). [11:56:31] miraheze/mw-config - Aeywoo the build passed. [12:38:36] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [12:38:37] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 06:23:45) [12:38:38] mwtask161: not running [12:38:39] mwtask171: not running [12:38:40] mwtask181: not running [12:43:44] !log depool cp37 from beta [12:43:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:44:01] PROBLEM - mirabeta.org - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [12:46:56] !log pool cp37/38 into both high traffic and low traffic ips slightly to warm things up [12:46:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:27] [02dns] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/e1abf00fda9cbc8e13ed4023addc89dfd687eb78 [13:03:27] 02dns/03master 07RhinosF1 03e1abf00 Delete zones/dariawiki.org [13:03:52] [02ssl] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/eac473e3d99b86c60fc717a40fb8491f38f154b2 [13:03:53] 02ssl/03master 07RhinosF1 03eac473e Update certs.yaml [13:04:18] [02ssl] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/1f3e4c33f9f72a5d67af2907d80f0c94110e4cce [13:04:18] 02ssl/03master 07RhinosF1 031f3e4c3 Update redirects.yaml [13:04:39] [02ssl] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/8943836e5198a3ff9ad05020426f3f47d3ddc37a [13:04:39] 02ssl/03master 07RhinosF1 038943836 Delete certificates/dariawiki.org.crt [13:05:40] miraheze/dns - RhinosF1 the build passed. [13:12:31] RECOVERY - mirabeta.org - Cloudflare on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Sun 25 May 2025 07:12:02 AM GMT +0000. [13:16:49] [02mw-config] 07RhinosF1 merged 07Aeywoo's pull request #5889: Undeploy Skin:Eveskin per T13411 (07miraheze:03master...07Aeywoo:03patch-7) 13https://github.com/miraheze/mw-config/pull/5889 [13:16:49] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/8a21c041d3625101d633f53f2a01347900cda3e2 [13:16:49] 02mw-config/03master 07Aeywoo 038a21c04 Undeploy Skin:Eveskin per T13411 (#5889)… [13:17:03] [02mw-config] 07RhinosF1 merged 07Aeywoo's pull request #5888: Remove SemanticMediaWiki Conflict (07miraheze:03master...07Aeywoo:03patch-6) 13https://github.com/miraheze/mw-config/pull/5888 [13:17:05] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/d8ff91b6097839d479b81f51cee215ebc9665605 [13:17:06] 02mw-config/03master 07Aeywoo 03d8ff91b Remove SemanticMediaWiki Conflict (#5888)… [13:17:42] miraheze/mw-config - RhinosF1 the build passed. [13:17:47] [02mw-config] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/3307ca3af529335e82c87c061d803417ef6e095e [13:17:47] 02mw-config/03master 07Aeywoo 033307ca3 Removing closed wikis from local wiki PHP files (#5893)… [13:17:47] [02mw-config] 07RhinosF1 merged 07Aeywoo's pull request #5893: Removing closed wikis from local wiki PHP files (07miraheze:03master...07Aeywoo:03patch-8) 13https://github.com/miraheze/mw-config/pull/5893 [13:17:55] miraheze/mw-config - RhinosF1 the build passed. [13:18:37] miraheze/mw-config - RhinosF1 the build passed. [13:19:09] [02mediawiki-repos] 07RhinosF1 merged 07Aeywoo's pull request #70: Undeploy Skin:Eveskin per T13411 (07miraheze:03master...07Aeywoo:03patch-1) 13https://github.com/miraheze/mediawiki-repos/pull/70 [13:19:11] [02mediawiki-repos] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/mediawiki-repos/commit/e93317ac9d68fc0e50c68256c861eaec394c1eee [13:19:12] 02mediawiki-repos/03master 07Aeywoo 03e93317a Undeploy Skin:Eveskin per T13411 (#70)… [13:21:58] !log [rhinos@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [13:21:59] !log [rhinos@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [13:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:47] !log [rhinos@test151] starting deploy of {'folders': '1.44/skins', 'ignore_time': True} to test151 [13:23:50] !log [rhinos@test151] finished deploy of {'folders': '1.44/skins', 'ignore_time': True} to test151 - SUCCESS in 2s [13:23:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:08] !log [rhinos@test151] starting deploy of {'folders': '1.43/skins', 'ignore_time': True} to test151 [13:24:10] !log [rhinos@test151] finished deploy of {'folders': '1.43/skins', 'ignore_time': True} to test151 - SUCCESS in 2s [13:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:11] !log [rhinos@test151] starting deploy of {'extension_list': True, 'versions': '1.44'} to test151 [13:27:12] !log [rhinos@test151] finished deploy of {'extension_list': True, 'versions': '1.44'} to test151 - SUCCESS in 0s [13:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:20] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:28:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:44] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [13:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:25] !log [rhinos@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.43'} to all [13:30:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:39] !log [rhinos@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.44'} to test151 [13:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:48] !log [rhinos@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.43'} to all - SUCCESS in 23s [13:30:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:31:56] !log sudo salt-ssh -E 'mw.*' cmd.run 'rm -rf /srv/mediawiki/1.43/skins/Eveskin' [13:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:44] !log [rhinos@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.43'} to all [13:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:32] [02puppet] 07RhinosF1 created 03RhinosF1-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/453fb2cff4ff [13:35:32] 02puppet/03RhinosF1-patch-1 07RhinosF1 03453fb2c beta: use REL1_44 for beta [13:35:58] [02puppet] 07RhinosF1 drafted pull request #4288: [MediaWiki 1.44-alpha will be branched as a beta on 22-04-2025] beta: use REL1_44 for beta (03master...03RhinosF1-patch-1) 13https://github.com/miraheze/puppet/pull/4288 [13:36:06] [02puppet] 07coderabbitai[bot] commented on pull request #4288: --- […] 13https://github.com/miraheze/puppet/pull/4288#issuecomment-2780727996 [13:36:32] [02puppet] 07RhinosF1 commented on pull request #4288: @coderabbitai review 13https://github.com/miraheze/puppet/pull/4288#issuecomment-2780728180 [13:36:37] [02puppet] 07coderabbitai[bot] commented on pull request #4288:

[…] 13https://github.com/miraheze/puppet/pull/4288#issuecomment-2780728212 [13:37:05] !log [rhinos@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.44'} to test151 [13:37:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:41] !log [rhinos@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.43'} to all [13:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:36] !log [rhinos@test151] finished deploy of {'world': True, 'l10n': True, 'versions': '1.44'} to test151 - SUCCESS in 91s [13:38:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:39] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [13:38:40] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 07:23:47) [13:38:41] mwtask161: not running [13:38:42] mwtask171: not running [13:38:43] mwtask181: not running [13:44:27] !log [rhinos@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': '1.43'} to all - SUCCESS in 405s [13:44:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:51] [02mw-config] 07RhinosF1 assigned pull request #5895 to 07OAuthority (Re-enabling Skin:Evelution per T13471) 13https://github.com/miraheze/mw-config/pull/5895 [14:09:52] [02mw-config] 07RhinosF1 unassigned pull request #5895 from 07BlankEclair (Re-enabling Skin:Evelution per T13471) 13https://github.com/miraheze/mw-config/pull/5895 [14:10:15] [02mw-config] 07RhinosF1 closed pull request #5895: Re-enabling Skin:Evelution per T13471 (07miraheze:03master...07Aeywoo:03patch-10) 13https://github.com/miraheze/mw-config/pull/5895 [14:10:15] [02mw-config] 07RhinosF1 commented on pull request #5895: https://issue-tracker.miraheze.org/T13471#270359 13https://github.com/miraheze/mw-config/pull/5895#issuecomment-2780741933 [14:11:15] [02mw-config] 07RhinosF1 assigned pull request #5890 to 07Universal-Omega (Logging: be more verbose for authevents) 13https://github.com/miraheze/mw-config/pull/5890 [14:11:36] [02mw-config] 07RhinosF1 assigned pull request #5887 to 07Aeywoo (Add Extension:LinkTitles Configuration settings and remove simulatorwiki references) 13https://github.com/miraheze/mw-config/pull/5887 [14:11:47] [02mw-config] 07RhinosF1 commented on pull request #5887: @Aeywoo: please resolve conflicts 13https://github.com/miraheze/mw-config/pull/5887#issuecomment-2780742483 [14:12:08] [02mw-config] 07RhinosF1 pushed 1 new commit to 03no-cookie-domain 13https://github.com/miraheze/mw-config/commit/918699462861b2775ac5ba884ff9b7b6ef6c132a [14:12:08] 02mw-config/03no-cookie-domain 07RhinosF1 039186994 Update GlobalSettings.php… [14:12:13] [02mw-config] 07RhinosF1 assigned pull request #5878 to 07Universal-Omega (Don't set $wgCentralAuthCookieDomain) 13https://github.com/miraheze/mw-config/pull/5878 [14:12:25] [02mw-config] 07RhinosF1 assigned pull request #5875 to 07Universal-Omega (Enable chunking for Swift dumps) 13https://github.com/miraheze/mw-config/pull/5875 [14:12:49] [02mw-config] 07RhinosF1 assigned pull request #5827 to 07Universal-Omega (Cleanup and improve performance of MirahezeFunctions) 13https://github.com/miraheze/mw-config/pull/5827 [14:13:03] [02mw-config] 07RhinosF1 assigned pull request #5822 to 07Universal-Omega (Rename and namespace MirahezeFunctions to ConfigurationSetup) 13https://github.com/miraheze/mw-config/pull/5822 [14:13:06] miraheze/mw-config - RhinosF1 the build passed. [14:13:11] [02mw-config] 07RhinosF1 commented on pull request #5822: conflicts 13https://github.com/miraheze/mw-config/pull/5822#issuecomment-2780742989 [14:13:33] [02mw-config] 07RhinosF1 commented on pull request #5687: status? 13https://github.com/miraheze/mw-config/pull/5687#issuecomment-2780743136 [14:13:42] [02mw-config] 07RhinosF1 assigned pull request #5687 to 07Universal-Omega (Try wgSettings) 13https://github.com/miraheze/mw-config/pull/5687 [14:13:58] [02mw-config] 07RhinosF1 commented on pull request #5678: @paladox: status? 13https://github.com/miraheze/mw-config/pull/5678#issuecomment-2780743328 [14:14:01] [02mw-config] 07RhinosF1 assigned pull request #5678 to 07paladox (Support new event/jobrunner for beta) 13https://github.com/miraheze/mw-config/pull/5678 [14:14:17] [02mw-config] 07RhinosF1 assigned pull request #5529 to 07AgentIsai (Allow using orain.org and wiki.surf also) 13https://github.com/miraheze/mw-config/pull/5529 [14:14:30] [02mw-config] 07RhinosF1 commented on pull request #5529: @AgentIsai: can you chase cf on our open tickets? 13https://github.com/miraheze/mw-config/pull/5529#issuecomment-2780743601 [14:15:23] [02puppet] 07RhinosF1 self-assigned pull request #4288 ([MediaWiki 1.44-alpha will be branched as a beta on 22-04-2025] beta: use REL1_44 for beta) 13https://github.com/miraheze/puppet/pull/4288 [14:15:32] [02puppet] 07RhinosF1 self-assigned pull request #4265 (mwdeploy: disable InsecureRequestWarning when using localhost) 13https://github.com/miraheze/puppet/pull/4265 [14:16:03] [02puppet] 07RhinosF1 assigned pull request #4251 to 07BlankEclair (install CampaignEvents per T13325) 13https://github.com/miraheze/puppet/pull/4251 [14:16:24] [02puppet] 07RhinosF1 pushed 1 new commit to 03mhglobal-backup-daily 13https://github.com/miraheze/puppet/commit/40ea609478e382f03d7fc73657983a2542f446d9 [14:16:24] 02puppet/03mhglobal-backup-daily 07RhinosF1 0340ea609 Merge branch 'master' into mhglobal-backup-daily [14:16:32] [02puppet] 07RhinosF1 approved pull request #4239 13https://github.com/miraheze/puppet/pull/4239#pullrequestreview-2744907870 [14:16:39] [02puppet] 07RhinosF1 assigned pull request #4239 to 07Universal-Omega (Backup mhglobal on a daily basis) 13https://github.com/miraheze/puppet/pull/4239 [14:17:04] [02puppet] 07RhinosF1 commented on pull request #4222: status? 13https://github.com/miraheze/puppet/pull/4222#issuecomment-2780745033 [14:17:08] [02puppet] 07RhinosF1 pushed 1 new commit to 03foundation 13https://github.com/miraheze/puppet/commit/1bef2478ff8290b5f1b7b05ed0d0a8796dc042bc [14:17:08] 02puppet/03foundation 07RhinosF1 031bef247 Merge branch 'master' into foundation [14:17:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:17:46] [02puppet] 07RhinosF1 requested changes on pull request #4209: standard across the rest of our stuff is f-strings I think 13https://github.com/miraheze/puppet/pull/4209#pullrequestreview-2744908173 [14:17:59] [02puppet] 07RhinosF1 assigned pull request #4209 to 07paladox (ircecho: Fix print statement) 13https://github.com/miraheze/puppet/pull/4209 [14:18:12] [02puppet] 07RhinosF1 assigned pull request #4222 to 07Universal-Omega (Switch foundation.wikitide.org to wikitidefoundation.org) 13https://github.com/miraheze/puppet/pull/4222 [14:18:25] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49930MiB (10% inode=99%); [14:18:55] [02puppet] 07RhinosF1 approved pull request #4117 13https://github.com/miraheze/puppet/pull/4117#pullrequestreview-2744908350 [14:19:44] [02puppet] 07RhinosF1 assigned pull request #4113 to 07Universal-Omega (Set autocommit to 0) 13https://github.com/miraheze/puppet/pull/4113 [14:20:07] [02puppet] 07RhinosF1 closed pull request #3914: DNM: T12537: Migrate to BIND (03master...03T12537-bind) 13https://github.com/miraheze/puppet/pull/3914 [14:20:34] [02puppet] 07RhinosF1 assigned pull request #4040 to 07paladox (Temp) 13https://github.com/miraheze/puppet/pull/4040 [14:20:42] [02puppet] 07RhinosF1 commented on pull request #4040: still needed? 13https://github.com/miraheze/puppet/pull/4040#issuecomment-2780747178 [14:21:09] [02puppet] 07coderabbitai[bot] commented on pull request #4027: --- […] 13https://github.com/miraheze/puppet/pull/4027#issuecomment-2780747297 [14:21:15] [02puppet] 07RhinosF1 self-assigned pull request #4027 (monitoring: only apply ssl service if the domain doesn't end in .miraheze.org) 13https://github.com/miraheze/puppet/pull/4027 [14:21:49] [02puppet] 07RhinosF1 assigned pull request #4027 to 07paladox (monitoring: only apply ssl service if the domain doesn't end in .miraheze.org) 13https://github.com/miraheze/puppet/pull/4027 [14:23:23] [02puppet] 07RhinosF1 approved pull request #4027: we should only be monitoring things we can actually do something about, there are better ways of monitoring cf domains 13https://github.com/miraheze/puppet/pull/4027#pullrequestreview-2744909466 [14:23:47] [02puppet] 07RhinosF1 assigned pull request #3933 to 07Universal-Omega (Add python script to update static tech docs) 13https://github.com/miraheze/puppet/pull/3933 [14:23:55] [02puppet] 07RhinosF1 unassigned pull request #4027 from 07paladox (monitoring: only apply ssl service if the domain doesn't end in .miraheze.org) 13https://github.com/miraheze/puppet/pull/4027 [14:24:00] [02puppet] 07RhinosF1 merged 07MacFan4000's pull request #4027: monitoring: only apply ssl service if the domain doesn't end in .miraheze.org (07miraheze:03master...07MacFan4000:03patch-53) 13https://github.com/miraheze/puppet/pull/4027 [14:24:00] [02puppet] 07RhinosF1 pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/69da7151a1d8bed01e63ce7caf6762419080a1da [14:24:02] 02puppet/03master 07MacFan4000 0369da715 monitoring: only apply ssl service if the domain doesn't end in .miraheze.org (#4027)… [14:25:10] [02mw-config] 07paladox closed pull request #5678: Support new event/jobrunner for beta (03master...03paladox-patch-1) 13https://github.com/miraheze/mw-config/pull/5678 [14:25:13] [02mw-config] 07paladox 04deleted 03paladox-patch-1 at 03511af7c 13https://github.com/miraheze/mw-config/commit/511af7c [14:30:23] [02mediawiki-repos] 07Aeywoo opened pull request #71: Undeploy Skin:Evelution per T13471#270359 (07miraheze:03master...07Aeywoo:03patch-1) 13https://github.com/miraheze/mediawiki-repos/pull/71 [14:30:30] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #71: --- […] 13https://github.com/miraheze/mediawiki-repos/pull/71#issuecomment-2780759733 [14:32:47] [02mw-config] 07Aeywoo opened pull request #5898: Undeploy Skin:Evelution per T13471#270359 (07miraheze:03master...07Aeywoo:03patch-6) 13https://github.com/miraheze/mw-config/pull/5898 [14:33:35] miraheze/mw-config - Aeywoo the build passed. [14:38:41] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [14:38:42] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 08:23:50) [14:38:43] mwtask161: not running [14:38:44] mwtask171: not running [14:38:45] mwtask181: not running [14:42:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:38:43] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [15:38:44] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 09:23:52) [15:38:45] mwtask161: not running [15:38:46] mwtask171: not running [15:38:47] mwtask181: not running [15:48:39] PROBLEM - volunteerforukraine.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'volunteerforukraine.com' expires in 14 day(s) (Sun 20 Apr 2025 03:19:31 PM GMT +0000). [16:15:26] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.77, 7.18, 5.95 [16:17:21] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.36, 7.52, 6.21 [16:19:15] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.36, 7.15, 6.24 [16:21:10] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.80, 8.09, 6.68 [16:38:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.72, 7.86, 7.65 [16:38:45] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [16:38:46] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 10:23:54) [16:38:48] mwtask161: not running [16:38:49] mwtask171: not running [16:38:50] mwtask181: not running [16:40:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.29, 7.99, 7.72 [16:42:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.74, 7.60, 7.62 [16:46:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.59, 7.93, 7.72 [16:50:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.36, 7.75, 7.71 [16:56:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 8.04, 7.83 [17:02:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.74, 7.95, 7.89 [17:04:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.41, 8.13, 7.96 [17:08:37] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=animalroyalezhwiki --logwiki=metawiki FriskyGoat Frisk (END - exit=256) [17:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.93, 7.81, 7.96 [17:20:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.28, 7.73, 7.88 [17:22:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.39, 7.59, 7.81 [17:28:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.33, 8.13, 7.98 [17:30:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.88, 7.71, 7.85 [17:32:05] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/attachAccount.php --wiki=animalroyalezhwiki --userlist=/home/rhinos/fixrename.txt (END - exit=0) [17:32:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:50] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=animalroyalezhwiki --logwiki=metawiki FriskyGoat Frisk (END - exit=0) [17:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.82, 7.70, 7.76 [17:38:48] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [17:38:49] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 11:23:57) [17:38:50] mwtask161: not running [17:38:51] mwtask171: not running [17:38:52] mwtask181: not running [17:40:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.92, 7.86, 7.82 [17:44:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.71, 8.09, 7.90 [17:46:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.12, 7.68, 7.78 [18:16:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.05, 7.17, 7.01 [18:18:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.69, 7.34, 7.09 [18:18:52] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49888MiB (10% inode=99%); [18:30:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.73, 7.79, 7.35 [18:32:43] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 5.38, 6.93, 7.09 [18:34:43] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 3.22, 5.45, 6.52 [18:38:50] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [18:38:51] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 12:23:59) [18:38:52] mwtask161: not running [18:38:53] mwtask171: not running [18:38:54] mwtask181: not running [19:38:53] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [19:38:54] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=zhindividualballwiki 10001 (running for 13:24:01) [19:38:55] mwtask161: not running [19:38:56] mwtask171: not running [19:38:57] mwtask181: not running [20:03:57] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php reassignEdits --wiki=animalroyalezhwiki 'Frisk (Usurped)' 'Frisk' (END - exit=0) [20:04:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:59] [02puppet] 07paladox created 03haproxy (+1 new commit) 13https://github.com/miraheze/puppet/commit/79a69d555596 [20:15:59] 02puppet/03haproxy 07paladox 0379a69d5 Add new cache/haproxy/varnish roles… [20:16:07] !log depool cp38 from production [20:16:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:10] [02puppet] 07paladox opened pull request #4289: Add new cache/haproxy/varnish roles (03master...03haproxy) 13https://github.com/miraheze/puppet/pull/4289 [20:16:16] [02puppet] 07coderabbitai[bot] commented on pull request #4289: --- […] 13https://github.com/miraheze/puppet/pull/4289#issuecomment-2781079707 [20:18:07] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03description-extra 13https://github.com/miraheze/ManageWiki/commit/acc179d349ddfbecb9d4f13e07beef488332e0e7 [20:18:07] 02ManageWiki/03description-extra 07CosmicAlpha 03acc179d Add option [20:19:58] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03description-extra 13https://github.com/miraheze/ManageWiki/commit/bcf0ef5cee7933056137083bbb5c954174269b8b [20:19:59] 02ManageWiki/03description-extra 07CosmicAlpha 03bcf0ef5 Disable [20:20:44] [02puppet] 07paladox 04force-pushed 03haproxy from 0379a69d5 to 03c1e1fca 13https://github.com/miraheze/puppet/commits/haproxy [20:20:44] 02puppet/03haproxy 07paladox 03c1e1fca Add new cache/haproxy/varnish roles… [20:23:15] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 61867MiB (13% inode=99%); [20:23:56] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [20:23:58] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [20:24:01] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [20:24:01] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [20:24:20] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [20:24:23] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [20:24:44] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [20:24:44] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [20:24:45] [02puppet] 07paladox 04force-pushed 03haproxy from 03c1e1fca to 031a62676 13https://github.com/miraheze/puppet/commits/haproxy [20:24:45] 02puppet/03haproxy 07paladox 031a62676 Add new cache/haproxy/varnish roles… [20:24:50] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [20:24:50] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [20:24:55] PROBLEM - cp36 Puppet on cp36 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [20:24:56] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [20:24:59] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [20:25:01] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [20:25:02] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:25:03] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [20:25:07] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [20:25:11] PROBLEM - cp38 Nginx Backend for mw191 on cp38 is CRITICAL: connect to address localhost and port 8129: Connection refused [20:25:17] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [20:25:18] miraheze/ManageWiki - Universal-Omega the build passed. [20:25:19] PROBLEM - cp38 Nginx Backend for mw163 on cp38 is CRITICAL: connect to address localhost and port 8123: Connection refused [20:25:23] PROBLEM - cp38 Nginx Backend for mw153 on cp38 is CRITICAL: connect to address localhost and port 8121: Connection refused [20:25:24] PROBLEM - cp38 Nginx Backend for mwtask161 on cp38 is CRITICAL: connect to address localhost and port 8163: Connection refused [20:25:25] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:25:27] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [20:25:30] PROBLEM - cp38 Nginx Backend for mw192 on cp38 is CRITICAL: connect to address localhost and port 8130: Connection refused [20:25:32] PROBLEM - cp38 Nginx Backend for mw173 on cp38 is CRITICAL: connect to address localhost and port 8125: Connection refused [20:25:37] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [20:25:42] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [20:25:44] PROBLEM - cp38 Nginx Backend for mw202 on cp38 is CRITICAL: connect to address localhost and port 8133: Connection refused [20:25:46] PROBLEM - cp38 Nginx Backend for puppet181 on cp38 is CRITICAL: connect to address localhost and port 8204: Connection refused [20:25:47] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [20:25:50] PROBLEM - cp38 Puppet on cp38 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:25:50]