[10:12:33] 06serviceops, 10FlaggedRevs, 10WMF-JobQueue: Spike in JobQueue job backlog time (500ms -> 4-8 minutes) - https://phabricator.wikimedia.org/T378385#10287837 (10Ladsgroup) It shouldn't be too hard to give it a dedicated lane with small concurrency [11:14:35] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Degraded RAID on wikikube-worker2068 - https://phabricator.wikimedia.org/T378255#10288061 (10Clement_Goubert) 05Open→03In progress a:03Clement_Goubert [11:28:47] 06serviceops, 10Maps (Kartotherian): Strategy to slowly move Kartotherian's traffic from bare metal to k8s - https://phabricator.wikimedia.org/T378944 (10elukey) 03NEW [11:29:33] hey folks! [11:29:54] I created https://phabricator.wikimedia.org/T378944 to kick of the discussion about how to migrate kartotherian to Wikikube [11:30:16] not urgent, but when/if you have time lemme know what you think about it (and if it is feasible/doable) [11:33:33] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Degraded RAID on wikikube-worker2068 - https://phabricator.wikimedia.org/T378255#10288128 (10Clement_Goubert) Partition table copied to the new disk and added it to the software raid. Rebuild in progress. ` cgoubert@wikikube-worker2068:~$ cat /proc/mdstat Person... [11:40:06] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, and 2 others: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10288126 (10elukey) I created T378944 to kick off the discussion about how to best move Kartotherian to k8s when ready. [12:46:40] FYI, I'll upload the backports of the latest PHP security release to our PHP 7.4 icu67 buster build in a bit [12:46:56] these have been running on mwdebug since last week without any issues [12:47:07] task is https://phabricator.wikimedia.org/T378173 [14:36:29] <_joe_> moritzm: thanks :) [15:08:06] I'll upgrade the remaining non-mwdebug and non-k8s use cases tomorrow (mwmaint, video scalers) [15:15:01] <_joe_> so today the first deploy should in theory pick up the change [16:50:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021#10289692 (10VRiley-WMF) @Clement_Goubert It seems that there are servers already named wikikube-worker1240, wikikube-worker1241, and wikikube-worker12... [16:56:45] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021#10289750 (10Clement_Goubert) Yeah sorry about that, I got confused with the host renaming we've been doing. Thanks for catching it. I'll amend the tas... [17:12:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021#10289838 (10Clement_Goubert) [17:13:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker13[05-12] - https://phabricator.wikimedia.org/T377021#10289850 (10Clement_Goubert) [17:16:13] 06serviceops: wikikube-worker13[05-12] implementation tracking - https://phabricator.wikimedia.org/T377022#10289865 (10Clement_Goubert) [17:16:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker13[05-12] - https://phabricator.wikimedia.org/T377021#10289885 (10Clement_Goubert) [18:01:48] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290123 (10akosiaris) >>! In T374683#10263499, @BPirkle wrote: > The fix for the i... [18:07:15] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10290152 (10Krd) Just happend again: Request from .43.46 via cp3066 cp3066, Varnish XID 230643229 Upstream caches: cp3066 int Error: 429, at Mon, 04 Nov 2024 18:05:37 GMT [18:08:18] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290163 (10akosiaris) Change reverted [18:46:53] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290345 (10daniel) @akosiaris The "after" output you post above would be the output without RESTbase com... [18:52:16] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290356 (10akosiaris) >>! In T374683#10290345, @daniel wrote: > @akosiaris The "after" output you post a... [19:06:07] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290391 (10daniel) > My mistake. I missed that part. I assume that we are OK with... [19:07:52] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10290407 (10akosiaris) Summarizing from a discussion on IRC with @daniel I missed t... [20:42:15] 06serviceops, 06Data-Persistence, 06SRE: DegradedArray email alerts for aqs1013 and aqs1014 are firing since April 18 - https://phabricator.wikimedia.org/T373490#10290736 (10Eevans) 05Open→03Resolved a:03Eevans aqs1013 has been decommissioned (T379026), and aqs1014 fixed; Closing [20:53:41] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10290804 (10Jhancock.wm) [21:17:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10290906 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubestage2003.codfw.wmnet with OS bookworm [21:17:28] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10290907 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubestage2004.codfw.wmnet with OS bookworm [21:23:23] 06serviceops: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10290917 (10Scott_French) 05Open→03Resolved All migration releases have been turned up. The traffic migration itself will be tracked in T377038. [21:58:31] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10290996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubestage2004.codfw.wmnet with OS bookworm completed: - kubestage2004 (... [22:00:37] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10290998 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubestage2003.codfw.wmnet with OS bookworm completed: - kubestage2003 (... [22:04:13] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10291008 (10Jhancock.wm) [22:06:01] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10291009 (10Jhancock.wm) 05Open→03Resolved @Clement_Goubert this pair is ready [22:07:01] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Move video transcoding to use Shellbox - https://phabricator.wikimedia.org/T356241#10291014 (10Scott_French) Summarizing a bit of debugging at the end of last week: After shellbox-video was enabled in commons last week (October... [22:15:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291022 (10Jhancock.wm) [22:35:48] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291072 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host mc-gp2004.codfw.wmnet with OS bookworm [22:35:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291073 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host mc-gp2005.codfw.wmnet with OS bookworm [22:35:53] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host mc-gp2006.codfw.wmnet with OS bookworm [23:17:26] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291133 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host mc-gp2004.codfw.wmnet with OS bookworm completed: - mc-gp2004 (**PASS**)... [23:18:20] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291134 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host mc-gp2005.codfw.wmnet with OS bookworm completed: - mc-gp2005 (**WARN**)... [23:21:06] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291139 (10Jhancock.wm) [23:56:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10291268 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host mc-gp2006.codfw.wmnet with OS bookworm executed with errors: - mc-gp2006 (*...