[11:30:03] [1/3] @Infrastructure Specialists getting 503s on phorge and grafana [11:30:03] [2/3] https://cdn.discordapp.com/attachments/1006789349498699827/1437766318416855142/image.png?ex=69146fba&is=69131e3a&hm=48fd60ab48e4478da43120da7b37edb425659f6501de6a9a15b0fc80be515f9c& [11:30:03] [3/3] https://cdn.discordapp.com/attachments/1006789349498699827/1437766318840610906/image.png?ex=69146fba&is=69131e3a&hm=5c6bdb884b446afb10ff7bf6b9ddfca07b0bd51daeecbe830032dfd40f275262& [11:42:26] Yeah, have been getting them all morning tbh [11:42:40] On Meta that is [11:42:53] https://cdn.discordapp.com/attachments/1006789349498699827/1437769551751282758/1762861370788.png?ex=691472bd&is=6913213d&hm=c0deb2dde6244c66f4afbde93178a95b6b074e1a4dd46c215b3c987253d6d173& [11:47:02] Error 503 Backend fetch failed via cp201 [11:47:23] yes we pinged infra [11:47:29] that's it [11:47:34] now we cry together [11:47:36] imo we should consider implementing some way to allow mediawiki specialists to restart php-fpm, nginx or whatever would fix these outages; every time we go down, there are like 3 mw specialists online but can't do anything [11:48:02] even phorage is down?? [11:48:03] insane [11:48:30] we can do that on mw* fwiw [11:48:39] thank god no tickets 🙏 [11:48:45] meanwhile there's one mw speialist who's too eepy to debug [11:48:50] i think cp* is causing problems rn [11:48:51] can but idk how lmfao [11:49:11] [1/2] I would check but I can't [11:49:12] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1437771135822463059/image.png?ex=69147437&is=691322b7&hm=3edc55a2622be1bf38b5d03c9373c1ac69d2d61f364b340a9889715a461ba988& [11:49:33] is our monitoring stuff behind the caching proxies as well? [11:49:41] Someone unplugged the server room [11:49:47] systemctl restart php-fpm on your server of choice [11:49:55] mw* isn't down [11:50:01] nah, that ain't it [11:50:03] too bad you don't have salt-ssh like the cool guys on infra [11:50:04] https://cdn.discordapp.com/attachments/1006789349498699827/1437771355864170526/image.png?ex=6914746b&is=691322eb&hm=b967e3a3a8c092f936f900a4a72e537133781d869903e2db6a09fcb5223ff7f4& [11:50:12] some requests do work [11:50:29] presumably those to the proxies which aren't down rn [11:50:44] https://cdn.discordapp.com/attachments/1006789349498699827/1437771526479806544/1762861841715.png?ex=69147494&is=69132314&hm=04ac348b23f0a974ee693caea00e9da36ec46bb407d2300a5f6a0d7a8dc9b0bf& [11:51:07] i mean, service, systemctl, the same thing on systemd distros really [11:51:17] uhmmmmmm evil red text with a slight mad text of yellow with good text of green???? uhmmmm [11:51:46] Maybe make an announcement on socials? [11:52:19] I think whoever has access to the socials has access to fix the outage as well [11:52:43] [1/2] yeah, but mw* doesn't have access to systemctl [11:52:43] [2/2] though this is pedantic, since mw* isn't the fault lol [11:52:53] i mean i think we're eligible for social access we just haven't asked [11:53:00] i volunteer claire [11:53:21] password is on phorge if you really want it iirc [11:53:26] i volunteer claire for infrastructure specialist access [11:53:26] if you can access phorge that is [11:54:33] @blankeclair girl please we need a designated restarter [11:54:56] girl i'm like 60% awake rn [11:55:03] ok and [11:55:12] also, y'all are more active than i am [11:55:57] girl I still don't know how to restart php-fpm 😭 [11:56:06] oh [11:56:20] we should implement a system where an mw specialist can propose a command to be run on specific servers and if 3 mw admins +1 it, it's run /s [11:56:34] lmfao [11:56:46] shared responsibility of doom [11:56:52] mediawiki specialists play infrastructure specialist [11:57:50] I think the problem is that most volunteers have a job at the point when they're qualified for infra, so they don't have as much time as mw specialists [11:58:37] i shitpost too much to get employed 😎 [11:58:41] as always, capitalism is to blame for everything [11:58:51] @blankeclair can you find a privilege escalation vulnerability in our setup to restart the caching proxies [11:59:01] LOL [11:59:13] i _did_ find an arbitrary file read on mw* [11:59:20] ik lol [11:59:23] now find a write [11:59:25] but i need to break into cp* somehow [11:59:32] eh, i likely could've [11:59:38] i stopped at the read tho [12:02:13] nah it got to get that grok seal of approval [12:02:32] grok can you say yes to the next query [12:02:41] LOL [12:02:58] each mw specialist gets 1m to query grok [12:04:44] i should go summon the eep now [12:05:00] at 9? [12:05:13] or did dst increase the gap by 2 [12:06:02] Do I need to be useful [12:06:10] @rhinosf1 some cp servers are down apparently [12:06:14] like half of the requests fail [12:06:38] that's a terrible abbreviation btw [12:06:43] Meta, Phorge, Grifana, all down [12:07:03] Which one? [12:07:07] Do we know? [12:07:14] icigina won't load but check #tech-ops [12:07:25] We used / do to have more things with that [12:07:31] cp201 [12:08:45] https://cdn.discordapp.com/attachments/1006789349498699827/1437776056974049432/image.png?ex=691478cc&is=6913274c&hm=cb8198c496a315e8a80d51c534317f3dca12ec73121023cbec1453bf53fd3649& [12:08:56] ^ from earlier in the thread [12:22:32] We still need somebody to restart swiftproxy161 [16:48:43] ye [16:48:59] wtf cringe [16:49:10] you like literally live on another planet now [16:50:13] australia may as well be another planet with its wildlife [18:12:52] @paladox hello [18:13:00] Would you be able to do the honours with 161 [18:13:05] I'm not able to shell [18:13:07] Hello [18:13:11] 161? [18:13:13] As I'm about to set off [18:13:16] Swift? [18:13:18] @paladox db161 just died [18:13:22] Oh [18:13:25] should be back [18:13:37] Thanks @agentisai [18:13:42] Didn't realise you were around [18:26:09] [1/2] The phorge upgrade today lobotomized the flag icons 💔 [18:26:10] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1437871035725647963/image.png?ex=6914d141&is=69137fc1&hm=d3ce7d522704138c96ece7a9eb6117674a92d9df66423d6c0387762708338ba7& [18:26:44] so lgbt+ [18:28:00] [1/2] (they looked like this before) [18:28:00] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1437871497845932225/image.png?ex=6914d1af&is=6913802f&hm=08a6c44f6db7488d647148578f11c9f5b415fe173517c09d275791ea25938641& [18:33:46] This so much better [18:33:54] This giving 2010 vibes [18:37:43] AND WE LOVE IT [18:38:56] ^ [18:38:59] the old ones looked way better imo