[01:29:38] 06serviceops, 07Datacenter-Switchover: Audit / update switchover-related cookbooks - https://phabricator.wikimedia.org/T372649#10143033 (10Scott_French) 05Open→03Resolved At this point, it's rather unlikely anything new is going to be discovered by inspection. Any issues shaken out as part of pre-switc... [06:56:49] 06serviceops, 13Patch-For-Review: kafka-main200[6789] and kafka-main2010 implementation tracking - https://phabricator.wikimedia.org/T363210#10143215 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a5b4f45c-79cb-4b77-b41e-ea8a9d2a6286) set by jayme@cumin1002 for 1 day, 0:00:00 on 2 host(s)... [07:30:20] 06serviceops, 10CirrusSearch, 06MediaWiki-Platform-Team, 03Discovery-Search (Current work): PHP web requests running for multiple hours - https://phabricator.wikimedia.org/T374662#10143281 (10Joe) A big difference between php-fpm and HHVM is that PHP doesn't have natively a wall clock based timeout. So for... [07:36:15] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission kafka-main2005.codfw.wmnet - https://phabricator.wikimedia.org/T374688 (10JMeybohm) 03NEW [07:36:28] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission kafka-main2005.codfw.wmnet - https://phabricator.wikimedia.org/T374688#10143297 (10JMeybohm) [07:36:32] 06serviceops, 13Patch-For-Review: kafka-main200[6789] and kafka-main2010 implementation tracking - https://phabricator.wikimedia.org/T363210#10143298 (10JMeybohm) [07:42:10] 06serviceops, 13Patch-For-Review: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10143305 (10elukey) [07:42:19] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored mw-debug k8s deployment - https://phabricator.wikimedia.org/T372604#10143307 (10Joe) >>! In T372604#10142394, @Scott_French wrote: > Alas, among the ways mw-debug is special, it configures an extra NodePort service that skips envoy (i.e., direct to apa... [09:34:54] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: decommission kafka-main2005.codfw.wmnet - https://phabricator.wikimedia.org/T374688#10143508 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: `kafka-main2005.codfw.wmnet` - kafka-main20... [09:37:35] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: decommission kafka-main2005.codfw.wmnet - https://phabricator.wikimedia.org/T374688#10143510 (10JMeybohm) a:05JMeybohm→03None [09:38:13] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10143513 (10akosiaris) >>! In T371130#10141416, @Trizek-WMF wrote: > Switching from 14:00 to 15:00 UTC required extra checks on the [[... [09:44:35] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10143536 (10JMeybohm) [10:03:06] 06serviceops, 13Patch-For-Review: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10143587 (10elukey) All new VMs created! Next steps: * Move thumbor-eqiad to poolcounter1006 * Update the MediaWiki's k8s network policy to allow the new nodes. * Test the new nodes via mw-debu... [11:36:17] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Communications support for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10143886 (10jijiki) [11:39:58] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Communication for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10143888 (10jijiki) a:05jijiki→03None [11:53:51] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143949 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from kubernetes2059 to wikikube-worker2114 completed: - kubernetes2... [12:05:11] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10143960 (10akosiaris) >>! In T371130#10143513, @akosiaris wrote: >>>! In T371130#10141416, @Trizek-WMF wrote: >> Switching from 14:00... [12:06:18] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143967 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from kubernetes2060 to wikikube-worker2115 completed: - kubernetes2... [12:11:47] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143984 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet [12:11:50] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143985 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet [12:12:54] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143987 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye [12:12:56] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye [12:13:52] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10143989 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2301 to wikikube-worker2116 completed: - mw2301 (**PASS**)... [12:25:11] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144045 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2303 to wikikube-worker2118 completed: - mw2303 (**PASS**)... [12:26:46] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144048 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2116.codfw.wmnet [12:27:12] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144049 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2116.codfw.wmnet with OS bullseye [12:30:44] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144070 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2304 to wikikube-worker2119 completed: - mw2304 (**PASS**)... [12:34:48] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Communication for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10144104 (10Bugreporter) We should also mention if you reset your Wikitech password in this transitional period, for LDAP users with known SUL conn... [12:40:12] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144121 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2118.codfw.wmnet [12:41:15] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2305 to wikikube-worker2120 completed: - mw2305 (**PASS**)... [12:41:22] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144127 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2118.codfw.wmnet with OS bullseye [12:45:14] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144155 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2119.codfw.wmnet [12:45:36] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144156 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2119.codfw.wmnet with OS bullseye [12:52:29] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144209 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2120.codfw.wmnet [12:52:32] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144210 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2302 to wikikube-worker2117 completed: - mw2302 (**PASS**)... [12:52:45] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144211 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2120.codfw.wmnet with OS bullseye [12:53:28] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144212 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2117.codfw.wmnet [12:53:49] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144213 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2117.codfw.wmnet completed... [12:55:17] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144221 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2117.codfw.wmnet [12:55:35] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144224 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2117.codfw.wmnet with OS bullseye [13:11:17] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144268 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2118.codfw.wmnet with OS bullseye executed... [13:11:21] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144269 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2118.codfw.wmnet completed... [13:11:59] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144270 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2118.codfw.wmnet [13:12:17] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144271 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2118.codfw.wmnet with OS bullseye [13:42:15] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144305 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye executed with errors: - wikiku... [13:42:17] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144306 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet completed: - wikikube-worker211... [13:52:49] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144327 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet [13:53:24] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144329 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye [13:53:48] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144332 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye executed with errors: - wikiku... [13:53:50] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144333 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet completed: - wikikube-worker211... [13:55:51] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144337 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye executed with errors: - wikiku... [13:55:52] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144338 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed: - wikikube-worker211... [13:58:40] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144342 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet [13:59:11] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144344 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye [14:00:22] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144345 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet [14:00:30] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144346 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed: - wikikube-worker211... [14:01:08] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144348 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet [14:01:24] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144352 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye [14:01:28] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144353 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye executed with errors: - wikiku... [14:01:29] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144354 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet completed: - wikikube-worker211... [14:01:35] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye executed with errors: - wikiku... [14:01:37] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144358 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed: - wikikube-worker211... [14:03:26] 06serviceops, 10CirrusSearch, 06MediaWiki-Platform-Team, 03Discovery-Search (Current work): PHP web requests running for multiple hours - https://phabricator.wikimedia.org/T374662#10144367 (10EBernhardson) >>! In T374662#10143281, @Joe wrote: > I should note though that we do have exceptions to the `201` s... [14:08:39] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes: Degraded RAID on wikikube-worker2092 - https://phabricator.wikimedia.org/T374409#10144395 (10Jhancock.wm) or the delivery gets messed up. will update when I have it in hand. [14:08:55] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144396 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2116.codfw.wmnet with OS bullseye executed with errors: - wikiku... [14:08:57] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144397 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2116.codfw.wmnet completed: - wikikube-worker211... [14:17:58] dcausse OK, we're on the same page then. I was wondering if it was worth the effort to transfer categories on the graph split hosts just to keep alerts from firing (ref T373935 ). If you think it's better to get rid of categories on these hosts, we can do that instead [14:18:31] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144429 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2119.codfw.wmnet with OS bullseye executed with errors: - wikiku... [14:18:32] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144430 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2119.codfw.wmnet completed: - wikikube-worker211... [14:18:39] oops, wrong room [14:30:13] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144487 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2118.codfw.wmnet with OS bullseye completed: - wikikube-worker21... [14:35:34] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes: Degraded RAID on wikikube-worker2092 - https://phabricator.wikimedia.org/T374409#10144500 (10Clement_Goubert) Logistics... Thanks for the update! [14:39:20] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2117.codfw.wmnet with OS bullseye executed with errors: - wikiku... [14:39:24] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144507 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2117.codfw.wmnet completed: - wikikube-worker211... [14:41:49] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144511 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2118.codfw.wmnet completed: - wikikube-worker211... [14:45:04] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144514 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2120.codfw.wmnet with OS bullseye completed: - wikikube-worker21... [15:17:35] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144581 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2120.codfw.wmnet completed... [15:17:36] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144582 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2120.codfw.wmnet completed... [15:27:33] 06serviceops, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144588 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2313 to wikikube-worker2121 completed: - mw2313 (**PASS**)... [15:35:25] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144635 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2314 to wikikube-worker2122 completed: - mw2314 (**PASS**) - ✔️ Downtimed host on... [15:36:12] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored mw-debug k8s deployment - https://phabricator.wikimedia.org/T372604#10144644 (10Scott_French) Thanks, @joe. Great, so if the original use case is still applicable (and, agreed, being able to benchmark "next" is a desirable property), then it's straight... [15:39:39] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144662 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2315 to wikikube-worker2123 completed: - mw2315 (**PASS**) - ✔️ Downtimed host on... [15:50:12] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144677 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2121.codfw.wmnet [15:50:45] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144679 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2121.codfw.wmnet with OS bullseye [15:56:55] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144688 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2122.codfw.wmnet [15:57:24] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144690 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2122.codfw.wmnet with OS bullseye [16:04:59] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144716 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2123.codfw.wmnet [16:05:00] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144717 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2123.codfw.wmnet with OS bullseye [16:17:45] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10144776 (10hnowlan) We have at least partially addressed the healthchecking issues by introducing a second readiness probe on the shell... [16:23:45] 06serviceops, 06Discovery-Search: Use kafka-main-[eqiad|codfw].external-services.svc.cluster.local to discover kafka brokers in kafka client running in k8s - https://phabricator.wikimedia.org/T374729 (10dcausse) 03NEW [16:35:37] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144831 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2121.codfw.wmnet with OS bullseye completed: - wikikube-worker212... [16:46:27] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144853 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2122.codfw.wmnet with OS bullseye completed: - wikikube-worker212... [16:46:55] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144866 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2121.codfw.wmnet completed: - wikikube-worker2121... [16:51:05] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144880 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2123.codfw.wmnet with OS bullseye completed: - wikikube-worker212... [16:52:52] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144886 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2122.codfw.wmnet completed: - wikikube-worker2122... [16:59:14] 06serviceops: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10144932 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2123.codfw.wmnet completed: - wikikube-worker2123... [17:10:11] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10LPL Essential (LPL Essential 2024 Jul-Sep): Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10144965 (10akosiaris) >>! In T372753#10142067, @MSantos wrote: > After further investigation, this is con... [17:11:06] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374733 (10Scott_French) 03NEW [17:21:32] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10145005 (10akosiaris) No perceptible changes in deployment times, I 'll merge the 5% drop today and I think we can... [17:21:54] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10145006 (10akosiaris) p:05Triage→03Medium [17:23:11] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10145010 (10akosiaris) [17:29:43] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374733#10145018 (10akosiaris) [18:31:59] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10145359 (10Scott_French) Thank you for catching that @Trizek-WMF and @akosiaris for updating the docs.