[00:06:44] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56769MiB (12% inode=99%); [00:07:46] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56804MiB (12% inode=99%); [00:08:07] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57493MiB (12% inode=99%); [00:16:36] !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all [00:16:38] !log [universalomega@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:16:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:50] !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all [00:16:53] !log [universalomega@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:16:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:14] !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'force': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all [00:19:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:05] !log [universalomega@mwtask181] finished deploy of {'extension_list': True, 'force': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 50s [00:20:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:19] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:21:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:44] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [00:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:48] miraheze/mw-config - Universal-Omega the build passed. [01:08:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:08:21] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:08:45] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:08:45] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [01:08:45] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:08:45] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [01:08:46] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [01:08:50] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:08:50] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [01:08:51] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:08:58] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:08:59] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:09:05] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:09:05] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:09:06] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:09:14] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:09:14] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.015 second response time [01:09:16] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:09:19] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:09:20] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:09:26] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:09:26] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 45.99, 22.21, 9.16 [01:09:37] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:09:37] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:09:43] 🔥 [01:09:56] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:10:00] fire! [01:10:00] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:01] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:02] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:10:04] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:04] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [01:10:06] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:09] WHY DOES THIS HAPPEN RIGHT AFTER I START DOING SQL [01:10:10] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:10:11] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:10:12] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:10:14] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:10:17] tf is happen [01:10:20] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:10:21] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:10:21] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:23] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:24] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:10:27] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:29] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:32] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:10:33] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:44] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [01:10:51] UMMMMM [01:10:54] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:11] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.931 second response time [01:12:11] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 3.484 second response time [01:12:12] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 4.185 second response time [01:12:12] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 2.936 second response time [01:12:13] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [01:12:16] UH [01:12:19] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [01:12:19] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.023 second response time [01:12:19] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.022 second response time [01:12:20] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.020 second response time [01:12:21] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.200 second response time [01:12:25] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.213 second response time [01:12:26] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [01:12:31] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.021 second response time [01:12:33] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time [01:12:33] wat [01:12:43] i think he ate the fire [01:12:46] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [01:12:47] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [01:12:50] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 4244 bytes in 0.184 second response time [01:12:55] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.026 second response time [01:12:56] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [01:12:57] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [01:12:59] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [01:13:02] the fuck was happen [01:13:08] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.029 second response time [01:13:14] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.028 second response time [01:13:15] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.025 second response time [01:13:15] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [01:13:19] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 4244 bytes in 0.192 second response time [01:13:19] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [01:13:20] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [01:13:24] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time [01:13:24] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.029 second response time [01:13:31] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.216 second response time [01:13:37] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [01:13:41] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [01:14:01] Might have been db related, load on db161 peaked at 130 [01:14:01] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [01:14:02] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.028 second response time [01:14:02] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.208 second response time [01:14:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [01:14:04] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4034 bytes in 0.063 second response time [01:14:07] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.022 second response time [01:14:10] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [01:14:21] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [01:14:38] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4034 bytes in 0.086 second response time [01:14:38] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4088 bytes in 0.082 second response time [01:14:40] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4034 bytes in 0.072 second response time [01:14:44] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 32% [01:15:33] sorry, i dont speak server but how the heck do you go above 100 [01:16:26] last i checked computers cannot give birth to more computers for more cores or whatever [01:16:39] i regularly have processes that go over 100% cpu [01:16:54] h o w [01:17:01] 100% = 1 core fully used [01:17:06] 200% = two cores fully used [01:17:09] ah [01:17:21] silly [01:18:32] siwwy wittle computie [01:19:15] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Cosmos'} to test151 [01:19:17] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Cosmos'} to test151 - SUCCESS in 1s [01:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:10] my quad core windows vm: https://files.catbox.moe/dvgpa1.png [01:20:52] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Cosmos'} to all [01:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:53] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Cosmos'} to all - SUCCESS in 60s [01:21:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:57] https://files.catbox.moe/axly9r.png [01:23:27] beautiful drawing <3 [01:23:50] thank you <3 [01:27:26] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.57, 4.32, 11.12 [01:29:26] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.52, 3.36, 9.94 [01:32:39] [1/2] It could always be worse [01:32:40] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1397390927647215666/use_da_cpu.png?ex=68818d37&is=68803bb7&hm=75b299c16eea05a044ed216e7ffbc25705f14305bacbba6cf54b054eeae2ccbd& [01:33:14] (had to dig that out of the archives) [01:34:46] WHAT THE FRICK [01:36:41] Here at Miraheze, we like to use the entire CPU [01:36:52] we paid for the entire cpu, we're using the entire cpu [01:37:29] we should mine crypto with our unused thinking power [01:37:38] :3 [01:37:50] we have bank interest at home [01:39:07] we should get enough money to buy a windmill and start earning profit off of that ngl [04:09:50] We also like to use the entire HDD until it fails too [05:29:21] is this a wd green reference [05:29:37] DB141 ref [05:30:08] I think thats the only time we’ve ever retired a server name outside normal naming conventions [05:33:19] damn [08:50:44] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49832MiB (10% inode=99%); [08:55:46] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49864MiB (10% inode=99%); [09:34:39] Am I reading that right, 10% free? [09:35:53] I can’t remember what inode means in icinga check script [09:36:33] hopefully the 99% means free inodes [09:37:09] Guess we’ll find out soon 💀 [09:37:23] oops! no more files! [09:37:49] I mean technically its on CP, so it wouldnt take us completely down [09:38:44] (My guess is likely a large logfile) [09:41:09] The cps flap every day [09:41:15] I suspect it's bigger logs or something [09:41:29] We probably should adjust retention or make the disk like 5GB bigger [09:42:29] They operate at 88% - 92% full every day [09:46:07] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49938MiB (10% inode=99%); [09:49:53] https://issue-tracker.miraheze.org/T14031 [10:10:43] @cosmicalpha ^ [10:10:56] I've seen. [10:11:52] Its because the cache creates a file that is 100% of the alloted cache storage then populates that too. Then it goes down when logs clear up some then comes back again. [10:12:43] I'm not concerned about why but warnings shouldn't happen every day [10:13:16] They aren't concerning at all. Sure we should fix but its not 100% urgent. [10:13:59] anyway its past 4AM I can look and respond more to this after I sleep. [10:14:41] Yup [11:08:11] CA sleeping before 5 AM?? [11:51:28] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 24.05, 11.78, 5.92 [11:53:28] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.83, 8.45, 5.40 [16:43:23] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6d37534e4c79ed21748b996a98a45c05bf288bd6 [16:43:23] 02mw-config/03main 07CosmicAlpha 036d37534 Add virtual-centralnotice [16:43:50] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [16:43:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:16] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [16:44:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:21] miraheze/mw-config - Universal-Omega the build passed. [17:44:59] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php cleanupTitles --wiki=testwiki --dry-run (END - exit=0) [17:45:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:45:42] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php cleanupTitles --wiki=testwiki (END - exit=0) [17:45:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:47:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753289240000&orgId=1&to=1753292873356 [18:02:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753289960000&orgId=1&to=1753293560000 [19:07:35] !log backup first then run update.php for testwiki [19:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:43] !log backup first then run update.php for ldapwikiwiki [19:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:45] PROBLEM - wiki.orcaslicer.com - Cloudflare on sslhost is WARNING: WARNING - Certificate 'wiki.orcaslicer.com' expires in 28 day(s) (Thu 21 Aug 2025 07:14:01 PM GMT +0000). [19:59:54] [1/8] !log testwiki: DELETE FROM flow_revision [19:59:55] [2/8] WHERE rev_type_id = UNHEX('05ce73fd54c0c43178d6ce'); [19:59:55] [3/8] !log testwiki: DELETE FROM flow_tree_revision [19:59:55] [4/8] WHERE tree_rev_id = UNHEX('05ce73fd54c0c43178d6ce'); [19:59:55] [5/8] !log testwiki: DELETE FROM flow_tree_revision [19:59:56] [6/8] WHERE tree_rev_id = UNHEX('05ce74119843de20cfb9d3'); [19:59:56] [7/8] !log testwiki: DELETE FROM flow_tree_revision [19:59:56] [8/8] WHERE tree_rev_id = UNHEX('05ce74d286e3de20cfb9d3'); [20:00:15] hmm guess that doesnt work lol [20:02:31] !log testwiki: DELETE FROM flow_revision WHERE rev_type_id = UNHEX('05ce73fd54c0c43178d6ce'); [20:02:31] !log testwiki: DELETE FROM flow_tree_revision WHERE tree_rev_id = UNHEX('05ce73fd54c0c43178d6ce'); [20:02:31] !log testwiki: DELETE FROM flow_tree_revision WHERE tree_rev_id = UNHEX('05ce74119843de20cfb9d3'); [20:02:31] !log testwiki: DELETE FROM flow_tree_revision WHERE tree_rev_id = UNHEX('05ce74d286e3de20cfb9d3'); [20:02:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:47] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'FontAwesome'} to test151 [20:02:48] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'FontAwesome'} to test151 - SUCCESS in 1s [20:02:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:26] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'FontAwesome'} to all [20:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:52] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'FontAwesome'} to all - SUCCESS in 25s [20:03:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:36] !log sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php Flow:FlowPopulateLinksTables --wiki=testwiki [20:05:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:10] [02DiscordNotifications] 07Universal-Omega approved pull request #105 13https://github.com/miraheze/DiscordNotifications/pull/105#pullrequestreview-3048884156 [20:18:52] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'RatePage'} to test151 [20:18:53] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'RatePage'} to test151 - SUCCESS in 1s [20:18:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:02] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'RatePage'} to all [20:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:27] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'RatePage'} to all - SUCCESS in 24s [20:19:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:34] [02DiscordNotifications] 07SomeMWDev merged pull request #105: Skip empty or blank webhook URLs without logging (03main...03skip-empty-urls) 13https://github.com/miraheze/DiscordNotifications/pull/105 [20:23:36] [02DiscordNotifications] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/ef8f164918e784afdbe9a1f207c15b9533a8ecbc [20:23:38] 02DiscordNotifications/03main 07SomeRandomDeveloper 03ef8f164 Skip empty or blank webhook URLs without logging (#105) [20:23:40] [02DiscordNotifications] 07SomeMWDev 04deleted 03skip-empty-urls at 03d22ba6a 13https://api.github.com/repos/miraheze/DiscordNotifications/commit/d22ba6a [20:23:58] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DiscordNotifications'} to test151 [20:24:00] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DiscordNotifications'} to test151 - SUCCESS in 1s [20:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:37] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DiscordNotifications'} to all [20:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:22] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DiscordNotifications'} to all - SUCCESS in 44s [20:25:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:10] [02mw-config] 07pskyechology opened pull request #6032: Only allow sendemail for global NDA-signed functionaries (07miraheze:03main...07pskyechology:03T14033) 13https://github.com/miraheze/mw-config/pull/6032 [20:37:02] miraheze/mw-config - pskyechology the build passed. [20:41:30] !log [skye@test151] starting deploy of {'config': True} to test151 [20:41:31] !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [20:41:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:20] miraheze/DiscordNotifications - SomeMWDev the build passed. [20:49:03] miraheze/mw-config - pskyechology the build passed. [20:51:20] [02mw-config] 07pskyechology commented on pull request #6032: Changes verified as working on mirabeta. Newly created wikis' `user` groups don't have sendemail. 13https://github.com/miraheze/mw-config/pull/6032#issuecomment-3110137601 [20:51:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753300280000&orgId=1&to=1753303913356 [21:01:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753300880000&orgId=1&to=1753304480000