[00:28:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037287 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1251.eqiad.wmnet with OS bullseye... [00:30:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037303 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bullseye... [00:34:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037310 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1256.eqiad.wmnet with OS bullseye... [00:40:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037322 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1255.eqiad.wmnet with OS bullseye... [00:41:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037323 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1254.eqiad.wmnet with OS bullseye... [00:44:43] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037327 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1258.eqiad.wmnet with OS bullseye... [00:48:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037329 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1253.eqiad.wmnet with OS bullseye... [00:51:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037330 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1259.eqiad.wmnet with OS bullseye... [00:53:16] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037331 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bull... [00:53:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037332 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1252.eqiad.wmnet with OS bullseye... [00:57:25] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037333 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1257.eqiad.wmnet with OS bullseye... [01:07:54] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037348 (10Jclark-ctr) [01:08:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10037349 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bullseye... [08:36:37] 06serviceops, 07Kubernetes: Remove deprecated cloudnative-pg charts from chart-museum - https://phabricator.wikimedia.org/T371667#10037755 (10akosiaris) Putting it under #serviceops. We don't currently have docs on how to remove charts from chartmuseum although the API does exist https://chartmuseum.com/docs/.... [08:37:08] 06serviceops, 07Kubernetes: Remove deprecated cloudnative-pg charts from chart-museum - https://phabricator.wikimedia.org/T371667#10037759 (10brouberol) [09:04:04] 06serviceops, 10Cassandra: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10037809 (10Cparle) Sorry @Eevans this has been sitting in an open tab for > 1 month now ... is it possible to access the data gateway from beta so we could test it that way? [09:43:45] 06serviceops, 07Kubernetes: Remove deprecated cloudnative-pg charts from chart-museum - https://phabricator.wikimedia.org/T371667#10037905 (10brouberol) Actually, change of plans. Long story short, the controller performs a hardcoded `app.kubernetes.io/name: cloudnative-pg` Deployment label filtering, meaning... [10:59:29] 06serviceops, 10MediaWiki-Platform-Team (Radar): Enable extstore to a subset of memcached servers (experiment) - https://phabricator.wikimedia.org/T352885#10038179 (10jijiki) 05In progress→03Stalled Based on findings in T370185, we are unsure if we saw any overall benefit here. I will mark this as stalled... [11:24:23] 06serviceops, 10MW-on-K8s, 10Observability-Metrics: Move mw-accesslog-metrics instance to k8s - https://phabricator.wikimedia.org/T371214#10038284 (10kamila) On further thought, doing it properly, i.e. creating a k8s-aux cluster in codfw and placing it there, seems like a better idea. That's going to take mo... [11:35:26] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, and 2 others: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10038350 (10MSantos) [11:39:40] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 07Video: Create a script for running jobs from stdin - https://phabricator.wikimedia.org/T371688 (10hnowlan) 03NEW [11:44:22] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#10038395 (10hnowlan) [13:38:56] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10038647 (10jijiki) [13:41:15] kamila_: btw I'm generally in favor of an aux cluster in codfw, and could perhaps assist with setup [13:50:04] cdanis: that'd be appreciated, thank you <3 [13:50:43] I don't think it's super high priority, will ping you when I get to it [13:50:51] cool :) [13:51:04] we'd probably also need to expand the aux cluster in eqiad, btw [13:51:42] there was never a lot of CPU available there, it's still on ganeti nodes for ctrl and workers [14:18:27] cdanis: o/ since we are talking about it - I came across https://phabricator.wikimedia.org/T344230 some days ago, do you think we could couple it with the expansion? [14:18:33] (I can help in case) [14:18:49] elukey: I'm definitely game [14:18:50] not sure if we'd need to delete/re-create VMs or move them [14:19:41] > Update the networking configuration in /etc/network/interfaces. This is an error prone process with no review. Make sure you get it right otherwise you are in for some painful debugging [14:19:43] haha [14:22:46] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 07Video: Build and add Mercurius to PHP base image - https://phabricator.wikimedia.org/T371699 (10hnowlan) 03NEW [14:26:14] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 07Video: Add logic to scap to create videoscaling Job each release - https://phabricator.wikimedia.org/T371700 (10hnowlan) 03NEW [14:38:59] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 07Video: Videoscaler/Mercurius deployment - https://phabricator.wikimedia.org/T371701 (10hnowlan) 03NEW [14:52:05] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 07Video: Create a script for running jobs from stdin - https://phabricator.wikimedia.org/T371688#10038886 (10hnowlan) 05Open→03Invalid Better version of this in T369048 [23:21:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039923 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bull... [23:24:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039924 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1251.eqiad.wmnet with OS bull... [23:26:45] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039939 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1252.eqiad.wmnet with OS bull... [23:31:03] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039941 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1253.eqiad.wmnet with OS bull... [23:37:05] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039942 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1250.eqiad.wmnet with OS bullseye... [23:41:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039947 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1251.eqiad.wmnet with OS bullseye... [23:44:39] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039975 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1252.eqiad.wmnet with OS bullseye... [23:46:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1253.eqiad.wmnet with OS bullseye... [23:48:48] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039987 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1257.eqiad.wmnet with OS bull... [23:48:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1254.eqiad.wmnet with OS bull... [23:48:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039992 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1256.eqiad.wmnet with OS bull... [23:49:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10039994 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1255.eqiad.wmnet with OS bull...