[00:04:40] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 53399MiB (11% inode=99%); [00:04:43] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 55088MiB (12% inode=99%); [00:33:38] PROBLEM - kafka181 NTP time on kafka181 is WARNING: NTP WARNING: Offset 0.1134001613 secs [00:35:37] RECOVERY - kafka181 NTP time on kafka181 is OK: NTP OK: Offset 0.05388307571 secs [00:39:50] PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [00:44:40] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.23, 13.41, 5.94 [00:46:40] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 16.86, 15.08, 7.50 [00:56:40] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 30.44, 21.74, 12.82 [00:58:40] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 8.57, 17.24, 12.33 [01:01:21] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [01:03:21] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 31.01 ms [01:08:25] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.70, 19.06, 12.40 [01:10:25] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 16.50, 19.09, 13.31 [01:12:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:18:25] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 28.84, 22.00, 15.87 [01:20:25] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 14.51, 20.10, 16.00 [01:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:16:40] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 29.37, 15.74, 6.90 [02:20:40] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 8.75, 16.38, 9.53 [02:21:26] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/7503011a9409a0e6d0a86add80651a3fd77cbfd1 [02:21:26] 02dns/03master 07MacFan4000 037503011 remove changeprop151 and rdb161… [02:22:25] miraheze/dns - MacFan4000 the build passed. [02:30:59] interesting, now that I look further into it, rdb161 never actually existed... [02:36:29] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/db-backups/db-backups/db-backups.log is 4 seconds old and 123 bytes [02:44:22] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 29.71, 18.91, 8.75 [02:47:59] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/e120d4fe52fa48deccc6a0727093bdebcda7e6bb [02:47:59] 02puppet/03master 07CosmicAlpha 03e120d4f Remove absented cron [02:48:22] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.10, 17.05, 10.67 [02:49:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/8f1b59b13a2d0b074f3779e0a26450eaad3166b6 [02:49:43] 02puppet/03master 07CosmicAlpha 038f1b59b Remove absented cron [03:01:18] wow icinga is clean apart from 1 SEL entry for cloud20 [03:04:37] I guess we could slightly tweak the inlet temp threshold (assuming thats what cause the most recent alert) [03:04:43] caused* [03:05:31] ok, def is, looks to to be exactly at the threshold right now [03:05:44] cc @cosmicalpha ^ [03:06:23] Its okay for now it's not urgent I'll look at it at some point more. [03:21:15] PROBLEM - wiki.cwkmediagroup.org - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:14:25] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.06, 17.40, 7.94 [04:18:25] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 7.62, 17.11, 10.41 [05:49:52] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [05:51:52] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 30.98 ms [06:59:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.81, 16.91, 8.65 [07:01:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.66, 12.69, 8.10 [07:41:01] PROBLEM - wiki.cogsci.live - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [09:12:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:42:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:46:04] [02mw-config] 07BlankEclair pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/824be6067f0a7a2466ee796d714f662a545b4822 [09:46:04] 02mw-config/03master 07BlankEclair 03824be60 Disable global maintenance sitenotice [09:46:39] !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [09:46:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:46:43] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49945MiB (10% inode=99%); [09:47:00] !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [09:47:00] miraheze/mw-config - BlankEclair the build passed. [09:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:47:53] PROBLEM - en.aryavratpedia.co - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [09:54:23] PROBLEM - hi.aryavratpedia.co - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [09:54:40] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49894MiB (10% inode=99%); [09:57:10] PROBLEM - gu.aryavratpedia.co - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:48:11] [02mw-config] 07enews opened pull request #5849: Custom mh and mw icons for fisch (07miraheze:03master...07enews:03patch-2) 13https://github.com/miraheze/mw-config/pull/5849 [11:50:12] miraheze/mw-config - enews the build has errored. [11:51:34] [02mw-config] 07RhinosF1 left a file comment in pull request #5849 03065dcf0: this is never closed, needs a ] 13https://github.com/miraheze/mw-config/pull/5849#discussion_r1983216982 [12:07:33] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 28.39, 13.85, 5.86 [12:07:37] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw161 mw181 mw191 [12:09:11] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 13 backends are down. mw151 mw161 mw162 mw171 mw172 mw181 mw153 mw163 mw173 mw183 mw191 mw193 mw201 [12:09:23] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 8 backends are down. mw161 mw162 mw181 mw153 mw163 mw191 mw201 mw202 [12:09:49] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/c5e656e6745234d32f7f79f59643778870ad808d [12:09:49] 02MirahezeMagic/03master 07translatewiki.net 03c5e656e Localisation updates from https://translatewiki.net. [12:09:51] [02DataDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/DataDump/commit/cdfca496ccf8e2b1c3bea69b04ee96651ea6c01f [12:09:52] 02DataDump/03master 07translatewiki.net 03cdfca49 Localisation updates from https://translatewiki.net. [12:10:16] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:10:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 44% [12:12:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [12:13:08] huh [12:13:52] [02mw-config] 07enews left a file comment in pull request #5849 03065dcf0: Oops, I'm not familiar with PHP, will edit hold on! 13https://github.com/miraheze/mw-config/pull/5849#discussion_r1983247098 [12:14:20] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.331 second response time [12:14:37] miraheze/DataDump - translatewiki the build passed. [12:14:53] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:14:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 50% [12:15:59] miraheze/mw-config - enews the build passed. [12:16:11] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:16:56] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 30% [12:17:03] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:17:16] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [12:17:32] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:17:35] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:19:32] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.009 second response time [12:19:58] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:18] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.150 second response time [12:20:46] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 8.049 second response time [12:20:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [12:21:09] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 15% [12:21:31] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.513 second response time [12:21:32] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 3.724 second response time [12:21:33] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.208 second response time [12:21:33] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:21:40] miraheze/MirahezeMagic - translatewiki the build passed. [12:21:57] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.172 second response time [12:22:17] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:29] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.515 second response time [12:24:08] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [12:24:11] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.473 second response time [12:26:02] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.139 second response time [12:26:48] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:01] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:30:14] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9511MiB (10% inode=98%); [12:30:20] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:30:49] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:54] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.899 second response time [12:30:56] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 57% [12:31:42] [02mw-config] 07RhinosF1 approved pull request #5849 13https://github.com/miraheze/mw-config/pull/5849#pullrequestreview-2664303777 [12:32:26] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 7.220 second response time [12:32:44] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 1.029 second response time [12:32:49] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.861 second response time [12:32:56] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 12% [12:33:13] BlankEclair: https://github.com/miraheze/mw-config/pull/5849#pullrequestreview-2664303777 [12:33:13] [GitHub] [miraheze/mw-config #5849] open PR by enews, created 45 minutes, 3 seconds ago: Custom mh and mw icons for fisch | Adding custom mh and mw icons for fisch [12:33:23] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [12:33:37] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [12:34:05] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.481 second response time [12:35:00] RhinosF1: potential csp concerns? doubt fisch.miraheze.org can load resources cross-domain to hybridcafe.wiki [12:35:24] BlankEclair: ye good point [12:35:28] you're right [12:35:30] guess i'll comment that there? [12:35:41] that needs to be on static [12:35:53] eh, cafe.miraheze.org would technically work [12:35:58] ye [12:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:36:10] piss off icinga-miraheze please [12:36:47] [02mw-config] 07BlankEclair commented on pull request #5849: I don't think this will work as fisch.miraheze.org (or a custom domain if any) would not be able to load resources cross-origin to hybridcafe.wiki thanks to the CSP. The images used should ideally be uploaded to fischwiki, and then the […] 13https://github.com/miraheze/mw-config/pull/5849#issuecomment-2703729156 [12:37:11] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [12:51:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:53:33] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.20, 8.36, 12.00 [12:57:33] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.42, 3.96, 9.35 [13:21:54] [02mw-config] 07enews commented on pull request #5849: Aw i see. Could we give it a go? I’ll switch it if not. 13https://github.com/miraheze/mw-config/pull/5849#issuecomment-2703840358 [13:44:08] [02mw-config] 07enews commented on pull request #5849: @BlankEclair I've changed it to cafe.miraheze.org. I won't upload it for now directly due to licensing issues. 13https://github.com/miraheze/mw-config/pull/5849#issuecomment-2703893498 [13:44:18] miraheze/mw-config - enews the build passed. [14:11:29] [02mw-config] 07AgentIsai left a file comment in pull request #5849 0336f08b6: Why use img_auth combined with serving directly from the wiki rather than a direct connection to static.wikitide.net? 13https://github.com/miraheze/mw-config/pull/5849#discussion_r1983426408 [15:08:33] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [15:10:22] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.63, 14.54, 6.84 [15:10:59] RECOVERY - Host mattermost1 is UP: PING WARNING - Packet loss = 93%, RTA = 31.15 ms [15:14:22] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 20.30, 17.78, 9.94 [17:34:02] [02puppet] 07Universal-Omega created 03mhglobal-backup-daily (+1 new commit) 13https://github.com/miraheze/puppet/commit/bbbd56e06c43 [17:34:03] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [17:34:03] 02puppet/03mhglobal-backup-daily 07CosmicAlpha 03bbbd56e Backup mhglobal on a daily basis [17:34:07] [02puppet] 07Universal-Omega opened pull request #4239: Backup mhglobal on a daily basis (03master...03mhglobal-backup-daily) 13https://github.com/miraheze/puppet/pull/4239 [17:34:15] [02puppet] 07coderabbitai[bot] commented on pull request #4239: --- […] 13https://github.com/miraheze/puppet/pull/4239#issuecomment-2704512498 [17:36:03] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 31.88 ms [17:42:48] [02puppet] 07Universal-Omega created 03remove-wikitide-backup-legacy (+1 new commit) 13https://github.com/miraheze/puppet/commit/f930054a4e19 [17:42:50] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 03f930054 Remove wikitide-backup-legacy and minor cleanup [17:42:52] [02puppet] 07Universal-Omega opened pull request #4240: Remove wikitide-backup-legacy and minor cleanup (03master...03remove-wikitide-backup-legacy) 13https://github.com/miraheze/puppet/pull/4240 [17:42:58] [02puppet] 07coderabbitai[bot] commented on pull request #4240: --- […] 13https://github.com/miraheze/puppet/pull/4240#issuecomment-2704534222 [17:43:26] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/18ee84bb4f8e66c7657aaa96fb8c69057bcf57c9 [17:43:26] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 0318ee84b Update [17:43:46] miraheze/puppet - Universal-Omega the build has errored. [17:44:18] miraheze/puppet - Universal-Omega the build has errored. [17:46:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/aa36bfe9f80f9215192cd0d2287ddf595bebcd98 [17:46:52] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 03aa36bfe Remove [17:47:48] miraheze/puppet - Universal-Omega the build has errored. [17:48:01] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/0c8a12dfcccf6d395ad600d9f4279bd9f1c337c7 [17:48:01] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 030c8a12d Update [17:48:40] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.01, 14.69, 6.18 [17:49:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/5368a1b788fc4b25f177858b724af07cc6c9a6bf [17:49:00] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 035368a1b Update [17:49:14] miraheze/puppet - Universal-Omega the build has errored. [17:49:27] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/a905e9c350e4b887f060f394179279f78f43ac79 [17:49:27] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 03a905e9c - [17:50:22] miraheze/puppet - Universal-Omega the build has errored. [17:50:58] miraheze/puppet - Universal-Omega the build has errored. [17:51:30] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-wikitide-backup-legacy 13https://github.com/miraheze/puppet/commit/4c2c8fc903364f6c5d6ebef188c6507a0cedec14 [17:51:30] 02puppet/03remove-wikitide-backup-legacy 07CosmicAlpha 034c2c8fc Use stdlib::ensure_packages [17:52:40] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 18.60, 20.11, 10.58 [17:53:29] miraheze/puppet - Universal-Omega the build has errored. [17:59:41] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/f39fb78bc8ccd9df564d72e5c5ccda4aaa985ab3 [17:59:41] 02ssl/03master 07MacFan4000 03f39fb78 remove expired domain [18:12:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/526468e39bd04f3f91e45e80e58c2296a6833bf8 [18:12:42] 02puppet/03master 07CosmicAlpha 03526468e Remove wikitide-backup-legacy and minor cleanup (#4240) [18:12:44] [02puppet] 07Universal-Omega merged pull request #4240: Remove wikitide-backup-legacy and minor cleanup (03master...03remove-wikitide-backup-legacy) 13https://github.com/miraheze/puppet/pull/4240 [18:12:46] [02puppet] 07Universal-Omega 04deleted 03remove-wikitide-backup-legacy at 034c2c8fc 13https://github.com/miraheze/puppet/commit/4c2c8fc [18:13:37] miraheze/puppet - Universal-Omega the build has errored. [18:49:10] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/c78cbb4c277693bbfd96566b824c350d943b4690 [18:49:10] 02dns/03master 07MacFan4000 03c78cbb4 add TXT on wikitide.org for atlassian domain verification [18:50:10] miraheze/dns - MacFan4000 the build passed. [18:58:28] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=chronocrashwiki chronocrashwiki.xml (START) [18:58:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:04] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=chronocrashwiki chronocrashwiki.xml (END - exit=0) [18:59:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:46] using massregex gives me a badgateway error but the massregex still goes thru [20:25:16] ..huh [20:31:26] i sadly didnt ss but i hit execute and it takes a minute to process then gives me the rror [20:31:44] i was changing like 600 pages thou [20:31:54] and on a chromebook so [20:32:11] it worked fine for 1 page thou [20:32:11] Did it fail to do some pages [20:32:23] i dont think so [20:32:24] May be took to long for a response or smt [20:32:38] all the pages i checked went thru [20:33:44] prb took too long to show the results so it timed out [22:11:00] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 15.64, 7.33 [22:14:37] PROBLEM - decimatedrive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'decimatedrive.wiki' expires in 14 day(s) (Fri 21 Mar 2025 09:50:10 PM GMT +0000). [22:14:48] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/0a8f35698b78045aa7939f5d36c5ee6855de7029 [22:14:48] 02ssl/03master 07WikiTideSSLBot 030a8f356 Bot: Update SSL cert for decimatedrive.wiki [22:28:52] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.72, 23.52, 19.37 [22:30:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.80, 19.73, 18.59 [22:57:35] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.66, 17.99, 8.13 [23:01:35] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 9.09, 17.99, 10.80 [23:11:45] [02CreateWiki] 07Universal-Omega created 03maint-cleanup (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/5c554c1f1396 [23:11:45] 02CreateWiki/03maint-cleanup 07CosmicAlpha 035c554c1 Cleanup/modernize maintenance scripts… [23:11:49] [02CreateWiki] 07Universal-Omega opened pull request #695: Cleanup/modernize maintenance scripts (03master...03maint-cleanup) 13https://github.com/miraheze/CreateWiki/pull/695 [23:11:58] [02CreateWiki] 07coderabbitai[bot] commented on pull request #695: --- […] 13https://github.com/miraheze/CreateWiki/pull/695#issuecomment-2705144541 [23:12:30] RECOVERY - decimatedrive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'decimatedrive.wiki' will expire on Wed 04 Jun 2025 09:16:12 PM GMT +0000. [23:17:22] [02CreateWiki] 07codecov[bot] commented on pull request #695: All modified and coverable lines are covered by tests :white_check_mark: […] 13https://github.com/miraheze/CreateWiki/pull/695#issuecomment-2705151501 [23:18:27] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.45, 17.27, 8.82 [23:23:36] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03maint-cleanup 13https://github.com/miraheze/CreateWiki/commit/b8836ff6ca8cafbf112504a6b7a0348e52bd740e [23:23:36] 02CreateWiki/03maint-cleanup 07CosmicAlpha 03b8836ff Update [23:23:53] miraheze/CreateWiki - Universal-Omega the build passed. [23:24:24] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.24, 16.79, 12.14 [23:29:46] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/fe3e5747378e7298b8cce26c35b488bff5561404 [23:29:47] 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03fe3e574 - [23:35:19] miraheze/CreateWiki - Universal-Omega the build passed. [23:40:15] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.44, 15.22, 10.92 [23:41:02] miraheze/CreateWiki - Universal-Omega the build has errored. [23:44:51] [02mw-config] 07RevitIRL opened pull request #5850: Update LocalSettings.php - copyright year 2024 > 2025 (07miraheze:03master...07RevitIRL:03patch-5) 13https://github.com/miraheze/mw-config/pull/5850 [23:45:37] miraheze/mw-config - RevitIRL the build passed. [23:46:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.13, 17.79, 13.83 [23:59:56] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 18.01, 11.18