[00:54:39] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10275344 (10RLazarus) (For the avoidance of doubt: We'll need //some// form of solution to this problem before turning off the mwmaint hosts, but I'm not workin... [08:25:04] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10275780 (10elukey) My opinion it totally not relevant but I've read the proposal and it seems great, thank... [10:38:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: CPU 2 machine check error detected for rdb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T376961#10276151 (10SLyngshede-WMF) [11:04:35] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Degraded RAID on wikikube-worker2068 - https://phabricator.wikimedia.org/T378255#10276212 (10tappof) I'm adding the serviceops tag to the task. I believe the issue is real for /dev/sda. ` root@wikikube-worker2068:~# udevadm info --query=all --name=/dev/sda | gr... [11:33:17] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10276308 (10MatthewVernon) I suspect I've missed something, so forgive an obvious question, but: how does t... [13:22:04] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10276766 (10Dreamy_Jazz) MediaModeration jobs have gone back to normal processing. We can probably close this unless we want to follow-up on comments made above... [13:34:25] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10LPL Essential (LPL Essential 2024 Jul-Oct): Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10276799 (10MSantos) >>! In T372753#10144965, @akosiaris wrote: >>>! In T372753#10142067, @MSantos wrote:... [13:35:42] 06serviceops, 06Content-Transform-Team-WIP, 10CX-cxserver, 10RESTBase Sunsetting, and 2 others: Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10276810 (10MSantos) a:05santhosh→03None [13:35:45] 06serviceops, 06Content-Transform-Team-WIP, 10CX-cxserver, 10RESTBase Sunsetting, and 2 others: Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10276816 (10MSantos) [13:46:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: CPU 2 machine check error detected for rdb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T376961#10276849 (10Jclark-ctr) Dell has agreed to replace mainboard and cpu. should be this week [14:10:49] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Degraded RAID on wikikube-worker2068 - https://phabricator.wikimedia.org/T378255#10276995 (10Jhancock.wm) thanks for the confirmation! the server was out of warranty so I replaced the bad drive with one from stock. please let us know if that helped clear it up an... [14:47:19] 06serviceops: Monitoring to surface "low-traffic" jobs isolation failure - https://phabricator.wikimedia.org/T378609 (10Scott_French) 03NEW [14:47:36] 06serviceops: Monitoring to surface "low-traffic" jobs isolation failure - https://phabricator.wikimedia.org/T378609#10277238 (10Scott_French) p:05Triage→03Low [14:50:50] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10277255 (10Scott_French) Thanks @Ammarpad for the pointer to where this category came from, and thanks @cscott for providing some context. Indeed, this seems lik... [15:53:17] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10277636 (10kostajh) 05Open→03Resolved a:03kostajh Thanks everyone! [21:23:50] 06serviceops, 10Charts, 10Wikimedia-Extension-setup, 07Epic, 07Wikimedia-extension-review-queue: Epic: Deploy Chart extension in production - https://phabricator.wikimedia.org/T369944#10279226 (10CCiufo-WMF)