[02:04:37] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Update iDRAC on mw2260.codfw.wmnet - https://phabricator.wikimedia.org/T373934#10115689 (10Scott_French) [07:23:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#10116066 (10brouberol) We now have an [alert](https://gerrit.wikimedia.org/r/c/operations/alerts/+/1070... [08:01:26] hey folks [08:01:37] during a deploy this error popped up [08:02:04] 07:48:04 /usr/bin/sudo /usr/local/sbin/mediawiki-image-download 2024-09-04-074735-publish (ran as mwdeploy@wikikube-worker2080.codfw.wmnet) returned [1]: Pulling 'docker-registry.di [08:02:21] afaics the host was depooled after a reimage yesterday 07:48:04 /usr/bin/sudo /usr/local/sbin/mediawiki-image-download 2024-09-04-074735-publish (ran as mwdeploy@wikikube-worker2080.codfw.wmnet) returned [1]: Pulling 'docker-registry.di [08:02:25] ufff sorry [08:02:31] https://sal.toolforge.org/production?p=0&q=wikikube-worker2080&d= [08:02:57] err it was "pooled", my bad [08:09:09] I don't see anything strange on the node, plus the docker config seems ok (even the auth under /var/lib/kubelet) [08:20:37] elukey: taking a look [08:24:09] <3 [08:25:15] elukey: Running it manually works ok [08:25:29] wanna bet it's a transient issue with the docker-registry's nginx? [08:28:02] yeah, 401 on the registry side for some reason [08:28:48] I tried to run the command but failed :( [08:28:52] funny thing is, I just tried it and it worked [08:28:56] elukey: what did you try? [08:29:21] the same linked in the error msg, but probably I should have used sudo -u mwdeploy ? [08:29:26] nope [08:30:41] scap connects as mwdeploy, then runs sudo /usr/local/sbin/mediawiki-image-download $whatever [08:31:19] by the way we're down to 60% of wikikube hosts doing that prefetch https://phabricator.wikimedia.org/T366778 [08:32:00] then bad timing on my side :) [08:32:17] And honestly even then scap should just skip that host and not error out [08:32:33] what do you mean bad timing? [08:33:45] me running the command, I got authorization failed two times [08:34:29] if that error happens, do we need to do anything on the failed to as follow up? [08:35:07] not really, just retry the deployment, or depool the host and ping us [08:35:18] or depool the host and retry the deployment and ping us :P [08:36:00] sre.k8s.pool-depool-node cookbook does what is needed, cordons off the node and pools inactive in conftool [08:36:13] pooled=inactive hosts don't get the scap prefetch [08:36:52] but then again, this error should be a warning at most, it's a performance optimization so that the image is already there when we start trying to start containers with it [08:37:22] (and it's on its way out, as you can see in the task) [08:37:54] if there's really an issue between the host and the registry, containers won't be able to start on it [08:39:57] however, there seems to be a real issue [08:40:08] there's a mw-api-int pod in ImagePullBackOff on that node [08:40:13] so it's not just you x) [08:40:50] also I'm an idiot and was on wikikube-worker2008 [08:46:40] credentials are good [09:02:08] ah HA [09:02:12] smoking gun [09:04:27] basically, in order for docker auth to worker, we have to first deny authorization to a specific endpoint [09:04:48] This is done by building a list of the nodes that need to be denied access to this endpoint [09:05:36] however, yesterday, we've renamed a host, but didn't finish up because it needs an idrac update [09:05:56] that host was declared in puppet, but the DNS change wasn't done [09:06:20] that makes the loop that builds that host list fail, and all hosts after that one can't auth [09:18:15] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 07Wikimedia-production-error: wikikube-worker2080.codfw.wmnet can't auth to registry - https://phabricator.wikimedia.org/T373982 (10Clement_Goubert) 03NEW p:05Triage→03High [09:24:52] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10116431 (10akosiaris) {F57453778} Nothing concerning after all this time, if anything median time was a very sligh... [09:29:53] 06serviceops, 06All-and-every-Wikisource, 10Thumbor, 13Patch-For-Review: Elevated 429 responses from Thumbor on codfw starting 2024-08-14 00:00 UTC - https://phabricator.wikimedia.org/T372470#10116450 (10hnowlan) >>! In T372470#10113572, @Midleading wrote: > Almost no new PDF file thumbnails can be generat... [09:30:31] claime: re: https://gerrit.wikimedia.org/r/1070544 - one thing that I didn't get - why is it 2079 and not 2080? [09:31:35] elukey: it's making the registry fail to build the list [09:31:52] okok, not easy to follow sorry :D [09:32:00] any host reimaged after 2079 got introduced can't auth [09:32:07] but there's only one, 2080, :D [09:32:30] elukey: is the explanation in https://phabricator.wikimedia.org/T373982 clearer? [09:32:37] I've tried to make it clear as possible but... [09:33:28] 06serviceops, 10MW-on-K8s, 10Scap: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10116469 (10akosiaris) [09:33:39] claime: yes yes I failed to read the link between 2079, the puppet errors on the registry and 2080 [09:33:45] thanks :) [09:36:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10116476 (10JMeybohm) [09:39:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10116484 (10JMeybohm) [09:40:38] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10116480 (10JMeybohm) a:03JMeybohm As ML is already running some kubernetes workers on bookworm for a while, we decided to move away from backporting containerd to bullsey... [09:53:49] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, and 2 others: wikikube-worker2080.codfw.wmnet can't auth to registry - https://phabricator.wikimedia.org/T373982#10116533 (10Clement_Goubert) 05In progress→03Resolved Pulling restricted images now works from `wikikube-worker2080`, resolving. [10:09:14] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116620 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2325 to... [10:14:18] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116658 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumb... [10:14:24] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116660 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:24:15] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373916#10116698 (10Clement_Goubert) [10:28:04] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Update iDRAC on mw2260.codfw.wmnet - https://phabricator.wikimedia.org/T373934#10116702 (10Clement_Goubert) 05Open→03Invalid @Papaul We'll be decommissioning this host, sorry :) [10:56:31] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116778 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2316 to w... [11:04:48] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116839 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [11:09:27] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116845 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2318 to w... [11:09:29] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116846 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2319 to w... [11:10:02] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116850 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2317 to w... [11:14:57] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116856 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumberin... [11:19:50] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116870 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by hnowlan@cumin1002 Renumbe... [11:20:04] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10116871 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wi... [12:08:38] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117057 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [12:08:42] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117058 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by hnowlan@cumin1002 Renumbering... [12:28:47] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117134 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes200... [12:28:49] 06serviceops, 06MW-Interfaces-Team, 06Traffic: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#10117125 (10akosiaris) @daniel, We 've already setup the `/api/` part to point to mw-ext-api, which has introduced a minor regression per T373998. We can always rollback this, but I... [12:32:35] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117157 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes201... [12:36:35] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes203... [12:40:53] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117195 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [12:41:23] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117196 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [12:42:05] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117204 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [12:42:44] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117206 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [12:49:27] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117288 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [12:50:35] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117292 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [12:50:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117293 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [13:06:05] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117355 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by hnowlan@cumin1002 Renumbe... [13:06:36] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117356 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wi... [13:29:06] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117373 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [13:29:11] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117374 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering f... [13:35:35] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117401 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2086.codfw.wmne... [13:35:41] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117402 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering for host wikikube-worke... [13:56:28] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117457 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2082.codfw.wm... [14:16:49] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117589 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2087.codfw.wmne... [14:16:51] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117590 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering for host wikikube-worke... [14:17:34] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117592 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by hnowlan@cumin1002 Renumbering for host wikikube-wor... [14:38:21] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117668 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumbering for host wikikub... [14:41:55] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10117677 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2036.cod... [14:57:36] 06serviceops, 06Data Products, 10RESTBase Sunsetting, 07Code-Health-Objective, 13Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213#10117783 (10Miriam) 05Open→03Declined As per a discussion with @fkaaelin, we are closing this task as declined an... [15:10:11] 06serviceops, 10decommission-hardware: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018 (10Scott_French) 03NEW [15:13:41] 06serviceops, 06DC-Ops, 10ops-codfw: kubernetes2035 (renamed to wikikube-worker2087) reporting "Comm Error: Backplane 0" - https://phabricator.wikimedia.org/T374019 (10JMeybohm) 03NEW [15:17:23] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373916#10117950 (10JMeybohm) [15:24:56] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018#10117969 (10ops-monitoring-bot) depool host mw2267.codfw.wmnet by swfrench@cumin2002 with reason: None [15:25:33] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018#10117970 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by swfrench@cumin2002 depool for host mw2267.codfw.wmnet completed: -... [15:29:27] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission mw226[1-2].codfw.wmnet mw22[68-77].codfw.wmnet - https://phabricator.wikimedia.org/T371262#10118008 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:56:22] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118127 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by hnowlan@cumin1002 Renumbering for host wikikube... [15:56:36] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118128 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2083.codf... [16:04:22] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018#10118169 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by swfrench@cumin2002 for hosts: `mw[2260,2267].codfw.wmnet` - mw2260.codfw.wmne... [16:16:25] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118230 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2036.codfw.w... [16:20:48] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118258 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumbering for host wikikube-wo... [16:39:20] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018#10118404 (10Scott_French) a:05Scott_French→03None [16:49:22] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118561 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2083.codfw.wm... [17:10:24] 06serviceops, 07Datacenter-Switchover: sre.switchdc.mediawiki cookbook should take a task-id argument - https://phabricator.wikimedia.org/T330273#10118740 (10Scott_French) a:05Clement_Goubert→03Scott_French [17:10:26] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10118739 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by hnowlan@cumin1002 Renumbering for host wikikube-wor... [17:52:40] 06serviceops, 07Datacenter-Switchover: Pre-switchover cookbook testing - https://phabricator.wikimedia.org/T374047 (10Scott_French) 03NEW [17:53:53] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10119014 (10Trizek-WMF) [17:55:50] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10119020 (10Trizek-WMF) [19:49:29] 06serviceops, 10MW-on-K8s: MW image version for maintenance scripts - https://phabricator.wikimedia.org/T359127#10119330 (10RLazarus)