[00:00:04] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03remove-hooks [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/cb46c892353f...a134607a0084 [00:00:06] [02MirahezeMagic] 07Universal-Omega 03a134607 - Remove [00:00:07] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#542: Remove unused hooks - 13https://github.com/miraheze/MirahezeMagic/pull/542 [00:00:52] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#542: Remove unused hooks - 13https://github.com/miraheze/MirahezeMagic/pull/542 [00:00:55] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03remove-hooks [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/a134607a0084...65a94cfbb338 [00:00:56] [02MirahezeMagic] 07Universal-Omega 0365a94cf - Remove [00:02:21] [02MirahezeMagic] 07Universal-Omega edited pull request 03#540: Remove SpecialMirahezeSurvey - 13https://github.com/miraheze/MirahezeMagic/pull/540 [00:09:51] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:35:22] [02ci-scripts] 07Universal-Omega created branch 03branch-rules - 13https://github.com/miraheze/ci-scripts [00:35:25] [02ci-scripts] 07Universal-Omega pushed 031 commit to 03branch-rules [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/commit/0f6dccb01099 [00:35:28] [02ci-scripts] 07Universal-Omega 030f6dccb - resolved_dependencies: add branch rules for excluding/including [00:35:31] [02ci-scripts] 07Universal-Omega opened pull request 03#2: resolved_dependencies: add branch rules for excluding/including - 13https://github.com/miraheze/ci-scripts/pull/2 [00:35:33] [02ci-scripts] 07coderabbitai[bot] commented on pull request 03#2: resolved_dependencies: add branch rules for excluding/including - 13https://github.com/miraheze/ci-scripts/pull/2#issuecomment-2495722024 [00:38:24] [02ci-scripts] 07Universal-Omega closed pull request 03#2: resolved_dependencies: add branch rules for excluding/including - 13https://github.com/miraheze/ci-scripts/pull/2 [00:38:25] [02ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/ce45f0bc061c...3680364ef049 [00:38:27] [02ci-scripts] 07Universal-Omega 033680364 - resolved_dependencies: add branch rules for excluding/including [00:38:28] [02ci-scripts] 07Universal-Omega deleted branch 03branch-rules [00:38:31] [02ci-scripts] 07Universal-Omega deleted branch 03branch-rules - 13https://github.com/miraheze/ci-scripts [00:40:57] miraheze/MirahezeMagic - Universal-Omega the build has errored. [00:53:15] [02ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/3680364ef049...6c8653641290 [00:53:18] [02ci-scripts] 07Universal-Omega 036c86536 - Try fix [00:56:34] miraheze/MirahezeMagic - Universal-Omega the build has errored. [01:38:21] [02.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/e6fcde89225c...4f919445c953 [01:38:24] [02.github] 07Universal-Omega 034f91944 - Set MEDIAWIKI_VERSION env [01:50:04] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:53:10] [02ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/6c8653641290...f36d971b719b [01:53:12] [02ci-scripts] 07Universal-Omega 03f36d971 - Remove $wgWikimediaJenkinsCI [01:54:05] [02MirahezeMagic] 07Universal-Omega closed pull request 03#542: Remove unused hooks - 13https://github.com/miraheze/MirahezeMagic/pull/542 [01:54:07] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/0453f250e7bc...678a5533b249 [01:54:10] [02MirahezeMagic] 07Universal-Omega 03678a553 - Remove unused hooks (#542) [01:54:12] [02MirahezeMagic] 07Universal-Omega deleted branch 03remove-hooks [01:54:13] [02MirahezeMagic] 07Universal-Omega deleted branch 03remove-hooks - 13https://github.com/miraheze/MirahezeMagic [01:54:53] [02MirahezeMagic] 07Universal-Omega closed pull request 03#540: Remove SpecialMirahezeSurvey - 13https://github.com/miraheze/MirahezeMagic/pull/540 [01:54:54] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±4] 13https://github.com/miraheze/MirahezeMagic/compare/678a5533b249...c23c12a8bf7b [01:54:57] [02MirahezeMagic] 07Universal-Omega 03c23c12a - Remove SpecialMirahezeSurvey (#540) [01:54:58] [02MirahezeMagic] 07Universal-Omega deleted branch 03remove-survey [01:55:00] [02MirahezeMagic] 07Universal-Omega deleted branch 03remove-survey - 13https://github.com/miraheze/MirahezeMagic [01:58:48] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/c23c12a8bf7b...b3c8fd7e779d [01:58:50] [02MirahezeMagic] 07Universal-Omega 03b3c8fd7 - MirahezeIRCRCFeedFormatter: simplify (#541) [01:58:52] [02MirahezeMagic] 07Universal-Omega closed pull request 03#541: MirahezeIRCRCFeedFormatter: simplify - 13https://github.com/miraheze/MirahezeMagic/pull/541 [01:58:55] [02MirahezeMagic] 07Universal-Omega deleted branch 03simplify-rcfeed - 13https://github.com/miraheze/MirahezeMagic [01:58:58] [02MirahezeMagic] 07Universal-Omega deleted branch 03simplify-rcfeed [02:07:29] [02.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/4f919445c953...723cfbe898b5 [02:07:30] [02.github] 07Universal-Omega 03723cfbe - Remove [02:10:14] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:14:38] [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/b3c8fd7e779d...3fc7c7928e9a [02:14:40] [02MirahezeMagic] 07Universal-Omega 033fc7c79 - README: Fix typo [02:16:10] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b4803e59d353...9ab4d8083cb1 [02:16:13] [02mw-config] 07Universal-Omega 039ab4d80 - Remove wgMirahezeSurveyEnabled [02:17:57] miraheze/mw-config - Universal-Omega the build passed. [02:20:30] [02mw-config] 07Universal-Omega closed pull request 03#5738: Test DataDump chunking - 13https://github.com/miraheze/mw-config/pull/5738 [02:20:32] [02mw-config] 07Universal-Omega deleted branch 03test-chunking - 13https://github.com/miraheze/mw-config [02:20:34] [02mw-config] 07Universal-Omega deleted branch 03test-chunking [02:25:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:22:32] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw172 [03:24:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [03:43:07] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 137.64 ms [03:49:10] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.63 ms [03:59:36] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw163 [04:00:32] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw163 [04:01:36] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [04:02:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [09:28:36] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw162 [09:28:58] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw162 [09:30:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [09:30:58] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [09:34:56] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw174 [09:36:32] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw154 [09:38:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [09:38:55] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [11:36:25] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw152 mw161 [11:36:38] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw152 mw161 mw164 [11:46:22] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [11:46:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:01:07] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw182 mw173 [12:01:16] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw182 mw173 [12:01:55] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:56] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:02:02] PROBLEM - mw182 APT on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:16] Errr [12:02:41] That's not good [12:03:53] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time [12:03:54] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.047 second response time [12:03:59] RECOVERY - mw182 APT on mw182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:04:58] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:05:06] It's back [12:05:15] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:07:23] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:07:35] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.47, 12.46, 6.36 [12:08:09] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 31.06, 19.07, 11.15 [12:08:10] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [12:08:14] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:14] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:24] Huh [12:08:25] PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 39.32, 19.22, 9.65 [12:08:37] I think we went down I’m getting 502 on meta [12:08:48] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:08:51] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 10 backends are down. mw152 mw162 mw171 mw181 mw182 mw153 mw154 mw163 mw164 mw173 [12:08:58] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 37.74, 20.16, 9.50 [12:09:02] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:04] PROBLEM - mw173 Current Load on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:14] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 12 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw173 mw174 mw183 [12:09:14] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 33.89, 22.48, 12.08 [12:09:17] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.82, 17.29, 8.99 [12:09:17] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 38.17, 26.05, 13.70 [12:09:27] PROBLEM - mw154 Current Load on mw154 is CRITICAL: LOAD CRITICAL - total load average: 32.56, 18.91, 9.98 [12:09:28] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:10:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 15.25, 16.59, 9.79 [12:10:58] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:17] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:21] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4106 bytes in 5.232 second response time [12:11:29] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4105 bytes in 5.090 second response time [12:11:31] PROBLEM - mw173 SSH on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:32] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.01, 16.28, 9.59 [12:12:08] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 2.262 second response time [12:12:11] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.611 second response time [12:12:14] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:12:35] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:41] Yes we are [12:12:48] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [12:12:48] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.136 second response time [12:13:28] RECOVERY - mw173 SSH on mw173 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:13:35] PROBLEM - mw174 Current Load on mw174 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 21.79, 12.76 [12:13:43] PROBLEM - mw164 SSH on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:46] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:13:58] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 32.02, 25.18, 14.18 [12:14:20] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:32] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.395 second response time [12:14:37] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:14:42] PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 30.78, 23.62, 14.21 [12:14:46] PROBLEM - mw153 Current Load on mw153 is CRITICAL: LOAD CRITICAL - total load average: 28.07, 21.79, 13.43 [12:14:47] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.150 second response time [12:14:49] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:03] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.786 second response time [12:15:15] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.429 second response time [12:15:31] PROBLEM - mw174 Current Load on mw174 is WARNING: LOAD WARNING - total load average: 22.46, 22.33, 14.15 [12:15:38] RECOVERY - mw164 SSH on mw164 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:15:44] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.158 second response time [12:15:47] PROBLEM - mw154 Current Load on mw154 is WARNING: LOAD WARNING - total load average: 20.14, 21.85, 14.63 [12:16:07] PROBLEM - mw164 Current Load on mw164 is CRITICAL: LOAD CRITICAL - total load average: 29.16, 28.28, 17.25 [12:16:18] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.478 second response time [12:16:23] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 2.386 second response time [12:16:29] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [12:16:37] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 40.39, 26.58, 15.33 [12:16:44] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 18.65, 22.12, 14.87 [12:17:20] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.460 second response time [12:17:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.45, 19.44, 13.15 [12:17:28] PROBLEM - mw174 Current Load on mw174 is CRITICAL: LOAD CRITICAL - total load average: 38.52, 26.09, 16.33 [12:17:44] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:19] BlankEclair: I think the databases are sad [12:18:20] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:18:25] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:26] oh no :( [12:18:27] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 56 - Recv failure: Connection reset by peer [12:18:28] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:18:30] I've no idea why [12:18:30] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:18:43] it's okay database, you've had a few long years of service [12:18:45] :neocat_pat: [12:18:46] PROBLEM - mw154 Current Load on mw154 is CRITICAL: LOAD CRITICAL - total load average: 42.09, 27.35, 17.79 [12:18:51] RhinosF1 looks like the MW are under high load, is there any huge jobs running? [12:19:19] somewhat but not really related: https://catgirl.center/notes/a0xpgnklm3x602kl [12:19:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.91, 22.76, 16.18 [12:19:30] I see a huge spike in connections on the db [12:19:32] No idea why [12:19:37] PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 36.28, 27.93, 18.32 [12:20:26] Connection spike could be the MW losing connection and reconnecting, wonder if the high load could be triggering a connection loss to the db causing connection spike [12:20:29] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:20:32] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.058 second response time [12:20:32] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:20:35] RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.057 second response time [12:20:44] No it couldn't [12:20:44] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.180 second response time [12:20:44] PROBLEM - mw154 Current Load on mw154 is WARNING: LOAD WARNING - total load average: 18.82, 23.71, 17.62 [12:20:53] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:21:05] A connection spike is probably more likely something generating lots of queries [12:21:10] !log [alex@db171] enabled slowlog [12:21:13] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 14.23, 19.29, 14.80 [12:21:27] Possibly a slow query causing things to back up [12:21:39] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.330 second response time [12:21:44] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:58] PROBLEM - mw164 SSH on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:58] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:21] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.387 second response time [12:22:34] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 4.664 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:22:45] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:22:48] PROBLEM - mw154 Current Load on mw154 is CRITICAL: LOAD CRITICAL - total load average: 33.60, 26.78, 19.43 [12:22:54] @bluemoon0332 what queries currently running [12:22:59] In processlist [12:23:03] PROBLEM - mw151 SSH on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:33] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 34.89, 25.13, 18.44 [12:23:37] > [24/11/2024 23:23] PROBLEM - mw151 SSH on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:38] PROBLEM - mw164 PowerDNS Recursor on mw164 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:23:39] is this concerning? [12:23:42] PROBLEM - mw164 php-fpm on mw164 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:23:46] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:52] PROBLEM - mw162 SSH on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:58] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:24:14] PROBLEM - mw164 APT on mw164 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:15] PROBLEM - mw164 conntrack_table_size on mw164 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:18] PROBLEM - mw164 Puppet on mw164 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:21] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.411 second response time [12:24:50] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:00] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [12:25:05] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 5.179 second response time [12:25:10] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:12] PROBLEM - mw164 ferm_active on mw164 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:48] RECOVERY - mw162 SSH on mw162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:25:51] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.839 second response time [12:25:53] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:56] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.060 second response time [12:25:59] PROBLEM - mw161 PowerDNS Recursor on mw161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:26:42] BlankEclair: all of it is concerning [12:26:47] RECOVERY - mw164 conntrack_table_size on mw164 is OK: OK: nf_conntrack is 0 % full [12:26:49] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:49] RECOVERY - mw164 APT on mw164 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:26:50] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 7.200 second response time [12:26:52] PROBLEM - mw163 PowerDNS Recursor on mw163 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:26:54] RECOVERY - mw164 Puppet on mw164 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [12:26:54] RhinosF1: yeah, but is this one more so? [12:27:00] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.642 second response time [12:27:00] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.222 second response time [12:27:07] since https going to shit is kinda expected from db goign to shit [12:27:08] RECOVERY - mw164 ferm_active on mw164 is OK: OK ferm input default policy is set [12:27:10] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.953 second response time [12:27:10] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:27:15] PROBLEM - mw153 SSH on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:19] PROBLEM - mw153 PowerDNS Recursor on mw153 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:27:21] BlankEclair: nah, it just means php was crying too much and hogged all the cpu [12:27:25] oh okay [12:27:34] i might disconnect soon(ish) [12:27:50] RECOVERY - mw164 PowerDNS Recursor on mw164 is OK: DNS OK: 0.148 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:27:52] RECOVERY - mw164 php-fpm on mw164 is OK: PROCS OK: 21 processes with command name 'php-fpm8.2' [12:27:55] there's a ton of connections that have been sleeping for like 400 seconds on db171 [12:28:04] RECOVERY - mw164 SSH on mw164 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:28:04] From? [12:28:04] PROBLEM - mw173 SSH on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:09] everyone [12:28:24] @bluemoon0332 set DBClusterMainteance [12:28:31] And see if we can just kill everything [12:28:46] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:47] RECOVERY - mw163 PowerDNS Recursor on mw163 is OK: DNS OK: 0.116 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:28:54] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.629 second response time [12:29:07] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.57, 22.87, 21.37 [12:29:09] It's a circuit breaker but it might work [12:29:17] RECOVERY - mw153 SSH on mw153 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:29:20] RECOVERY - mw153 PowerDNS Recursor on mw153 is OK: DNS OK: 5.538 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:29:31] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:29:35] other than that nothing really remarkable [12:29:47] I'd say circuit break it [12:29:52] It'll bring the rest of the farm off [12:30:00] And hopefully whatever is upset will stop [12:30:01] RECOVERY - mw173 SSH on mw173 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:30:03] RECOVERY - mw161 PowerDNS Recursor on mw161 is OK: DNS OK: 1.677 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:30:07] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [12:30:08] It might just come back [12:30:14] But I've got nothing [12:30:16] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.309 second response time [12:30:41] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.161 second response time [12:30:47] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [12:30:47] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4105 bytes in 0.061 second response time [12:30:48] RECOVERY - mw154 Current Load on mw154 is OK: LOAD OK - total load average: 7.44, 19.35, 19.60 [12:30:48] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 7.47, 21.82, 21.62 [12:30:51] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.052 second response time [12:30:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 9.67, 22.94, 23.10 [12:31:11] RECOVERY - mw151 SSH on mw151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [12:31:15] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.048 second response time [12:31:17] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.053 second response time [12:31:18] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.049 second response time [12:31:18] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 5.95, 20.31, 21.82 [12:31:19] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 51.41, 33.16, 25.32 [12:31:29] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3909 bytes in 0.051 second response time [12:31:39] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 4.81, 19.87, 22.93 [12:31:41] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 8.20, 23.20, 22.59 [12:31:42] RECOVERY - mw174 Current Load on mw174 is OK: LOAD OK - total load average: 4.68, 18.56, 19.65 [12:31:52] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 5.33, 19.20, 22.43 [12:31:57] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 4.77, 21.37, 23.60 [12:31:59] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [12:32:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:32:35] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 4.30, 20.45, 22.15 [12:32:43] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 2.99, 15.48, 19.33 [12:32:47] I created https://issue-tracker.miraheze.org/T12921 [12:33:07] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:33:15] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 8.58, 22.90, 22.52 [12:33:18] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 2.23, 14.13, 19.36 [12:33:25] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 3.03, 17.47, 22.77 [12:33:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 1.91, 13.86, 20.35 [12:33:39] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 2.44, 16.29, 20.14 [12:33:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 3.59, 17.39, 19.43 [12:33:49] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 2.04, 13.57, 20.01 [12:33:55] sleepers have gone away for some reason [12:34:35] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 2.06, 14.22, 19.66 [12:34:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 1.99, 11.15, 18.19 [12:35:15] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 3.28, 16.33, 20.16 [12:35:53] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 2.12, 10.71, 18.68 [12:36:55] Ye we're back [12:37:13] PROBLEM - mw164 Current Load on mw164 is WARNING: LOAD WARNING - total load average: 1.13, 11.68, 23.12 [12:37:24] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 2.30, 9.53, 18.35 [12:39:03] [1/12] there's also a ton of this on the error log [12:39:03] [2/12] ``` [12:39:03] [3/12] 2024-11-24 12:15:23 30753342 [Warning] Aborted connection 30753342 to db: 'customcountryhumanswiki' user: 'mediawiki2024' host: '10.0.16.133' (Got an error reading communication packets) [12:39:04] [4/12] 2024-11-24 12:15:23 30748988 [Warning] Aborted connection 30748988 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.133' (Got an error reading communication packets) [12:39:04] [5/12] 2024-11-24 12:15:24 30738897 [Warning] Aborted connection 30738897 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:04] [6/12] 2024-11-24 12:15:24 30739421 [Warning] Aborted connection 30739421 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:05] [7/12] 2024-11-24 12:15:24 30739663 [Warning] Aborted connection 30739663 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:05] [8/12] 2024-11-24 12:15:24 30739585 [Warning] Aborted connection 30739585 to db: 'loginwiki' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:05] [9/12] 2024-11-24 12:15:24 30739017 [Warning] Aborted connection 30739017 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:06] [10/12] 2024-11-24 12:15:24 30738885 [Warning] Aborted connection 30738885 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:06] [11/12] 2024-11-24 12:15:24 30738887 [Warning] Aborted connection 30738887 to db: 'mhglobal' user: 'mediawiki2024' host: '10.0.16.152' (Got an error reading communication packets) [12:39:06] [12/12] ``` [12:39:21] I'll copy the current log to root's home dir for later [12:41:06] RECOVERY - mw164 Current Load on mw164 is OK: LOAD OK - total load average: 2.09, 6.30, 18.38 [12:41:56] @bluemoon0332 move to Mattermost [13:03:10] @rhinosf1 what flavor of ah shit was it today [13:05:23] Database [13:05:46] Great [13:06:14] @pixldev ye cause we have no idea on the cause [13:06:26] Oh come on [13:06:48] it do be like that lmao [13:07:01] anyway, nini [13:10:53] Night [13:21:40] @pixldev I think we might have a cause but still to find a fix [13:23:16] [02GlobalNewFiles] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/GlobalNewFiles [13:23:19] [02GlobalNewFiles] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/63453ca8fd68 [13:23:21] [02GlobalNewFiles] 07paladox 0363453ca - Use getFullUrl instead of getViewUrl [13:23:22] [02GlobalNewFiles] 07paladox opened pull request 03#69: Use getFullUrl instead of getViewUrl - 13https://github.com/miraheze/GlobalNewFiles/pull/69 [13:23:25] [02GlobalNewFiles] 07coderabbitai[bot] commented on pull request 03#69: Use getFullUrl instead of getViewUrl - 13https://github.com/miraheze/GlobalNewFiles/pull/69#issuecomment-2496000268 [13:23:38] [02GlobalNewFiles] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/63453ca8fd68...2f20fdefb623 [13:23:39] [02GlobalNewFiles] 07paladox 032f20fde - Update GlobalNewFilesMoveJob.php [13:23:41] [02GlobalNewFiles] 07paladox synchronize pull request 03#69: Use getFullUrl instead of getViewUrl - 13https://github.com/miraheze/GlobalNewFiles/pull/69 [13:27:53] [02GlobalNewFiles] 07paladox closed pull request 03#69: Use getFullUrl instead of getViewUrl - 13https://github.com/miraheze/GlobalNewFiles/pull/69 [13:27:54] [02GlobalNewFiles] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/GlobalNewFiles/compare/220374dc9677...93c24f24c853 [13:27:57] [02GlobalNewFiles] 07paladox 0393c24f2 - Merge pull request #69 from miraheze/paladox-patch-2 [13:28:00] [02GlobalNewFiles] 07paladox deleted branch 03paladox-patch-2 [13:28:01] miraheze/GlobalNewFiles - paladox the build passed. [13:28:02] [02GlobalNewFiles] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/GlobalNewFiles [13:28:22] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'GlobalNewFiles'} to all [13:28:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:31:19] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'GlobalNewFiles'} to test151 [13:31:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:09] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'GlobalNewFiles'} to test151 - SUCCESS in 50s [13:32:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:21] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:32:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:39] miraheze/GlobalNewFiles - paladox the build passed. [13:37:41] !log db161: mhglobal > DELETE FROM gnf_files where files_url like '/w/%'; [13:37:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:44] [02mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://github.com/miraheze/mw-config/commit/fb5c6afa6cf2 [14:41:46] [02mw-config] 07paladox 03fb5c6af - Create RunJobs.php [14:41:49] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [14:41:51] [02mw-config] 07paladox opened pull request 03#5753: Create RunJobs.php - 13https://github.com/miraheze/mw-config/pull/5753 [14:42:18]