[00:01:16] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/ccae0ff3ce6bde0a3cbed4379a6f6516cd80469b [00:01:16] 02statichelp/03main 07WikiTideBot 03ccae0ff Bot: Auto-update Tech namespace pages 2025-09-02 00:01:13 [00:06:26] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57044MiB (12% inode=99%); [00:07:37] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56535MiB (12% inode=99%); [00:08:52] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57293MiB (12% inode=99%); [00:10:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756771857228[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756771857228 [00:16:48] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cf14ced3dd93e67f6876120149150d4d9463f30f [00:16:48] 02puppet/03main 07dependabot[bot] 03cf14ced build(deps): bump actions/checkout from 4 to 5 (#4472)… [00:16:50] [02puppet] 07Universal-Omega merged 07dependabot[bot]'s pull request #4472: build(deps): bump actions/checkout from 4 to 5 (03main...03dependabot/github_actions/actions/checkout-5) 13https://github.com/miraheze/puppet/pull/4472 [00:16:52] [02puppet] 07Universal-Omega 04deleted 03dependabot/github_actions/actions/checkout-5 at 035c58947 13https://api.github.com/repos/miraheze/puppet/commit/5c58947 [00:18:00] miraheze/puppet - Universal-Omega the build has errored. [00:19:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03dns-packages 13https://github.com/miraheze/puppet/commit/2dd37104127af2c2ca330eef25205ccfcc18e6a6 [00:19:53] 02puppet/03dns-packages 07CosmicAlpha 032dd3710 Merge branch 'main' into dns-packages [00:27:58] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.13, 5.46, 4.44 [00:28:10] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.42, 5.36, 4.41 [00:28:14] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [00:36:04] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [02:25:32] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/19e8c256f7919ec8ebc7a85ed3c61741b3409e58 [02:25:32] 02ssl/03main 07WikiTideBot 0319e8c25 Bot: Auto-update domain lists [02:30:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756780257247[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756776440000&orgId=1&to=1756780040000 [02:33:19] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 154.03 ms [02:35:19] PROBLEM - ping6 on mattermost1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.02 ms [02:47:19] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 154.02 ms [02:53:26] PROBLEM - ping6 on mattermost1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.01 ms [03:23:24] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 148.15 ms [03:45:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756784757256[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756780940000&orgId=1&to=1756784757256 [03:55:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756785357257[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756781480000&orgId=1&to=1756785080000 [04:15:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756786557259[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756782860000&orgId=1&to=1756786557259 [04:40:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756788057262[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756784300000&orgId=1&to=1756787900000 [06:00:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756792857274[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756789220000&orgId=1&to=1756792857274 [06:05:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756793157275[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756789520000&orgId=1&to=1756793120000 [06:17:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.34, 20.53, 19.83 [06:23:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.85, 20.23, 20.05 [07:03:04] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23446MiB (10% inode=93%); [07:04:03] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.42, 20.66, 20.02 [07:10:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756797057285[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756793180000&orgId=1&to=1756797057285 [07:35:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756798557289[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756794860000&orgId=1&to=1756798460000 [07:49:04] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 54225MiB (24% inode=93%); [07:59:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.87, 20.08, 20.30 [08:03:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.19, 20.66, 20.48 [08:09:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 21.45, 20.78 [08:11:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.18, 21.24, 20.79 [08:35:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756802157297[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756798280000&orgId=1&to=1756802157297 [08:40:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756802457298[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756798580000&orgId=1&to=1756802180000 [09:13:37] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49834MiB (10% inode=99%); [09:35:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.05, 19.49, 20.25 [09:52:26] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%); [10:08:52] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49947MiB (10% inode=99%); [10:58:47] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Medik'} to test151 [10:58:50] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Medik'} to test151 - SUCCESS in 3s [11:03:37] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/9935a775e799 [11:03:37] 02puppet/03paladox-patch-1 07paladox 039935a77 changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 15 [11:03:40] [02puppet] 07paladox opened pull request #4489: changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job co… (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4489 [11:03:46] [02puppet] 07paladox retitled PR #4489: "changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job co…" ➜ "changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 15" 13https://github.com/miraheze/puppet/pull/4489 [11:03:48] [02puppet] 07coderabbitai[bot] commented on pull request #4489:
[…] 13https://github.com/miraheze/puppet/pull/4489#issuecomment-3244850159 [11:05:45] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/68e329aaa84db2555aa0acf9bb1f3f030abbef03 [11:05:45] 02puppet/03paladox-patch-1 07paladox 0368e329a Update changeprop201.yaml [11:07:47] PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [11:19:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.43, 19.56, 18.33 [11:21:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.12, 19.34, 18.42 [11:22:26] [02puppet] 07paladox created 03revert-4410-paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/55a361b2bebd [11:22:27] 02puppet/03revert-4410-paladox-patch-11 07paladox 0355a361b Revert "changeprop201: Decrease notificationGetStartedJob reenqueue_delay (#4…"… [11:22:38] [02puppet] 07paladox opened pull request #4490: Revert "changeprop201: Decrease notificationGetStartedJob reenqueue_delay" (03main...03revert-4410-paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4490 [11:22:45] [02puppet] 07coderabbitai[bot] commented on pull request #4490:
[…] 13https://github.com/miraheze/puppet/pull/4490#issuecomment-3244911785 [11:22:58] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d552f4e7f17523bf8fc539cc4acc5c9b4e0f7f33 [11:22:58] 02puppet/03main 07paladox 03d552f4e Revert "changeprop201: Decrease notificationGetStartedJob reenqueue_delay" (#4490)… [11:23:00] [02puppet] 07paladox merged pull request #4490: Revert "changeprop201: Decrease notificationGetStartedJob reenqueue_delay" (03main...03revert-4410-paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4490 [11:23:01] [02puppet] 07paladox 04deleted 03revert-4410-paladox-patch-11 at 0355a361b 13https://api.github.com/repos/miraheze/puppet/commit/55a361b [11:26:31] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c59b0bb98e50 [11:26:31] 02puppet/03paladox-patch-6 07paladox 03c59b0bb changeprop201: Fix metrics… [11:26:36] [02puppet] 07paladox opened pull request #4491: changeprop201: Fix metrics (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4491 [11:26:42] [02puppet] 07coderabbitai[bot] commented on pull request #4491:
[…] 13https://github.com/miraheze/puppet/pull/4491#issuecomment-3244922871 [11:26:45] [02puppet] 07paladox merged pull request #4491: changeprop201: Fix metrics (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4491 [11:26:46] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2e7279569108a2df6ca9e37e9a263e29a6ecf615 [11:26:46] 02puppet/03main 07paladox 032e72795 changeprop201: Fix metrics (#4491)… [11:26:48] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03c59b0bb 13https://api.github.com/repos/miraheze/puppet/commit/c59b0bb [11:29:57] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c6f1d2b0af8f [11:29:57] 02puppet/03paladox-patch-6 07paladox 03c6f1d2b changeprop201: Add notificationReEngageJob… [11:30:06] [02puppet] 07paladox opened pull request #4492: changeprop201: Add notificationReEngageJob (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4492 [11:30:12] [02puppet] 07coderabbitai[bot] commented on pull request #4492:
[…] 13https://github.com/miraheze/puppet/pull/4492#issuecomment-3244932803 [11:30:14] [02puppet] 07paladox merged pull request #4492: changeprop201: Add notificationReEngageJob (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4492 [11:30:16] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0b377752b9d60d15a87c019a1d0e4abac4724cf2 [11:30:18] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03c6f1d2b 13https://api.github.com/repos/miraheze/puppet/commit/c6f1d2b [11:30:20] 02puppet/03main 07paladox 030b37775 changeprop201: Add notificationReEngageJob (#4492)… [11:33:31] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/d0ae7f04c67eff871df83f0aab838ef842ce1e58 [11:33:31] 02puppet/03paladox-patch-1 07paladox 03d0ae7f0 Update changeprop201.yaml [11:33:39] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/d9dbd0a9c8671a56611db918a3b1f21da94ae9ef [11:33:39] 02puppet/03paladox-patch-1 07paladox 03d9dbd0a Merge branch 'main' into paladox-patch-1 [12:32:22] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 21.43, 19.08 [12:34:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.81, 20.29, 19.01 [12:36:36] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_skins': 'Medik'} to all [12:36:50] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 18.98, 16.51 [12:37:04] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_skins': 'Medik'} to all - SUCCESS in 29s [12:40:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756816857329 [12:42:50] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.88, 22.67, 19.02 [12:46:50] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 8.03, 16.36, 17.47 [12:49:49] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 17.00, 14.84 [12:57:34] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.07, 20.19, 17.86 [13:01:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 23.29, 19.67 [13:03:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.63, 19.43, 18.73 [13:07:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.81, 21.97, 19.94 [13:10:50] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 20.68, 16.90 [13:12:50] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 20.63, 20.00, 17.08 [13:14:50] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 11.16, 18.13, 16.85 [13:15:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.42, 19.68, 20.10 [13:19:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.41, 22.90, 21.34 [13:25:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 16.75, 20.03, 20.88 [13:29:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 22.20, 21.55 [13:30:51] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.06, 18.89, 16.99 [13:32:51] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.18, 18.95, 17.25 [13:35:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.24, 23.55, 22.46 [13:46:50] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.67, 20.20, 18.70 [13:48:50] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 18.67, 19.79, 18.74 [13:57:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.90, 16.03, 19.77 [13:59:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.20, 5.66, 4.06 [13:59:49] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.34, 5.98, 4.34 [14:01:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 6.46, 4.55 [14:03:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.67, 20.30, 20.46 [14:03:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.67, 5.96, 4.62 [14:03:49] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.32, 5.95, 4.78 [14:05:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 21.96, 21.05 [14:11:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 10.37, 19.48, 20.70 [14:15:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 5.96, 4.71 [14:17:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 22.90, 21.90 [14:17:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.89, 5.61, 4.75 [14:19:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.87, 22.99, 22.04 [14:21:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.42, 22.32 [14:25:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 13.79, 19.60, 21.21 [14:29:19] !log upgrade phroge on phorge171 [14:29:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 22.10, 21.88 [14:45:26] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f496280b74f9ddaa6ed8f7f4a9b21ea56cec75bd [14:45:27] 02puppet/03main 07paladox 03f496280 Revert "mediawiki_task: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5"… [14:48:29] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/openvox-keyring.gpg] [14:50:51] !log restart logbot on bots171 [14:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:52:28] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:57:32] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.63, 20.11, 22.72 [15:01:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.28, 13.96, 19.58 [15:32:50] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.11, 5.83, 4.77 [15:36:45] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.71, 5.36, 4.88 [15:41:57] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:57] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [15:53:39] [02puppet] 07Universal-Omega created 03ircecho-fstrings (+1 new commit) 13https://github.com/miraheze/puppet/commit/f5e02ffd6983 [15:53:39] 02puppet/03ircecho-fstrings 07CosmicAlpha 03f5e02ff ircecho: use f-strings [15:53:45] [02puppet] 07Universal-Omega opened pull request #4493: ircecho: use f-strings (03main...03ircecho-fstrings) 13https://github.com/miraheze/puppet/pull/4493 [15:53:52] [02puppet] 07coderabbitai[bot] commented on pull request #4493:
[…] 13https://github.com/miraheze/puppet/pull/4493#issuecomment-3245915581 [15:55:01] miraheze/puppet - Universal-Omega the build has errored. [16:10:57] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756829210000 [16:31:36] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.55, 5.75, 4.68 [16:41:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.27, 6.12, 5.58 [17:26:27] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 6.88, 5.18 [17:27:49] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.39, 6.62, 5.16 [17:28:25] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.42, 6.68, 5.31 [17:29:49] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.04, 5.74, 5.02 [17:54:40] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.62, 6.32, 4.78 [17:55:48] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.58, 6.56, 5.05 [17:56:35] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 8.17, 6.86, 5.15 [17:57:45] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.23, 5.48 [17:59:43] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.15, 7.20, 5.69 [18:00:25] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.43, 7.36, 5.75 [18:07:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.09, 6.48, 6.07 [18:08:05] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.66, 6.37, 5.99 [18:15:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.25, 6.79, 6.27 [18:17:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.56, 6.71, 6.31 [18:22:30] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.63, 6.89, 6.37 [18:23:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.56, 7.00, 6.52 [18:25:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.36, 5.96, 6.22 [18:26:20] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 2.14, 5.40, 5.97 [19:52:22] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/4ceec874e03e4b6514225b5dd08f56f84e74fd05 [19:52:22] 02puppet/03paladox-patch-1 07paladox 034ceec87 Merge branch 'main' into paladox-patch-1 [19:52:38] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/42a40743699bd78aa2ae60788dc095ce4ba99b57 [19:52:38] 02puppet/03paladox-patch-1 07paladox 0342a4074 Update changeprop201.yaml [19:52:48] [02puppet] 07paladox retitled PR #4489: "changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 15" ➜ "changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 20" 13https://github.com/miraheze/puppet/pull/4489 [19:52:51] [02puppet] 07paladox merged pull request #4489: changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 20 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4489 [19:52:53] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8c9f7735d1b5491bac9eb75f289830d8d0f15b78 [19:52:55] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0342a4074 13https://api.github.com/repos/miraheze/puppet/commit/42a4074 [19:52:57] 02puppet/03main 07paladox 038c9f773 changeprop201: Incease parsoidCachePrewarm and htmlCacheUpdate job concurrency to 20 (#4489) [19:53:47] RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:55:26] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/303db16e6c13440ed1a8a2b8bc95b12644dee425 [19:55:26] 02puppet/03main 07paladox 03303db16 changeprop: increase concurrency to 20 [19:55:58] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c35d0383c360 [19:55:58] 02puppet/03paladox-patch-1 07paladox 03c35d038 changeprop201: decrease low traffic concurrency to 20 [19:56:01] [02puppet] 07paladox opened pull request #4494: changeprop201: decrease low traffic concurrency to 20 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4494 [19:56:07] [02puppet] 07coderabbitai[bot] commented on pull request #4494:
[…] 13https://github.com/miraheze/puppet/pull/4494#issuecomment-3246605791 [19:56:08] [02puppet] 07paladox merged pull request #4494: changeprop201: decrease low traffic concurrency to 20 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4494 [19:56:10] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/03b29d5e0d34ce726f5bd2441635f98d70894a0f [19:56:12] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03c35d038 13https://api.github.com/repos/miraheze/puppet/commit/c35d038 [19:56:13] 02puppet/03main 07paladox 0303b29d5 changeprop201: decrease low traffic concurrency to 20 (#4494) [20:11:49] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.46, 6.60, 4.91 [20:13:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.79, 6.87, 5.13 [20:17:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.05, 7.40, 5.75 [20:19:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.80, 7.49, 5.98 [20:23:49] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.60, 6.24, 5.90 [20:25:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 1.58, 4.95, 5.52 [20:58:40] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 6.38, 4.34 [20:59:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.72, 6.53, 4.58 [21:00:35] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.96, 6.90, 4.77 [21:03:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.17, 7.28, 5.33 [21:05:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.22, 7.05, 5.50 [21:10:12] PROBLEM - os202 Current Load on os202 is CRITICAL: connect to address 10.0.20.167 port 5666: Connection refusedconnect to host 10.0.20.167 port 5666: Connection refused [21:11:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.38, 1.19, 0.41 [21:12:07] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 1.82, 0.71, 0.27 [21:23:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.33, 5.45, 3.21 [21:23:49] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.26, 4.92, 2.33 [21:25:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.27, 5.59, 3.55 [21:25:49] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.56, 4.63, 2.55 [21:29:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.53, 6.78, 4.51 [21:31:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.59, 5.94, 4.51 [21:56:25] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 6.77, 5.54 [21:58:22] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.43, 7.02, 5.79 [22:14:04] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.14, 6.62, 6.63 [22:21:53] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.11, 5.95, 5.92 [22:23:50] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.44, 6.43, 6.10 [22:29:43] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.18, 7.52, 6.71 [22:31:41] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.15, 5.79, 6.17 [22:51:19] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1963MiB (10% inode=90%); [22:53:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.17, 4.95, 3.98 [22:57:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.45, 6.69, 4.91 [22:59:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 8.00, 7.05, 5.26 [23:03:19] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6864MiB (38% inode=90%); [23:06:28] [02mw-config] 07SomeMWDev created 03T14192 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/e2bd00fe404b [23:06:28] 02mw-config/03T14192 07SomeRandomDeveloper 03e2bd00f T14192: Set $smwgNamespacesWithSemanticLinks for vanillaeuropawiki… [23:06:47] [02mw-config] 07SomeMWDev opened pull request #6078: T14192: Set $smwgNamespacesWithSemanticLinks for vanillaeuropawiki (03main...03T14192) 13https://github.com/miraheze/mw-config/pull/6078 [23:07:05] [02mw-config] 07SomeMWDev 04force-pushed 03T14192 from 03e2bd00f to 030825c54 13https://api.github.com/repos/miraheze/mw-config/commits/T14192 [23:07:07] 02mw-config/03T14192 07SomeRandomDeveloper 030825c54 T14192: Set $smwgNamespacesWithSemanticLinks for vanillaeuropawiki… [23:07:38] miraheze/mw-config - SomeMWDev the build passed. [23:07:56] miraheze/mw-config - SomeMWDev the build passed. [23:09:58] [02mw-config] 07SomeMWDev merged pull request #6078: T14192: Set $smwgNamespacesWithSemanticLinks for vanillaeuropawiki (03main...03T14192) 13https://github.com/miraheze/mw-config/pull/6078 [23:09:58] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/03f61b911a78dfb235442d7ca152685d0c106a99 [23:09:58] 02mw-config/03main 07SomeRandomDeveloper 0303f61b9 T14192: Set $smwgNamespacesWithSemanticLinks for vanillaeuropawiki (#6078)… [23:09:59] [02mw-config] 07SomeMWDev 04deleted 03T14192 at 030825c54 13https://api.github.com/repos/miraheze/mw-config/commit/0825c54 [23:10:21] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [23:10:22] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [23:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:36] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:10:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:58] miraheze/mw-config - SomeMWDev the build passed. [23:11:02] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [23:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 7.57, 6.53 [23:13:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 1.72, 5.43, 5.88 [23:20:57] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'ArticleFeedbackv5'} to test151 [23:20:58] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'ArticleFeedbackv5'} to test151 - SUCCESS in 1s [23:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.14, 5.83, 5.61 [23:27:34] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.18, 7.35, 6.38 [23:29:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 5.85, 7.01, 6.39 [23:31:34] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.06, 5.27, 5.83 [23:56:32] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [23:56:33] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:56:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:01] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [23:59:02] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:59:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log