[07:34:04] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128731 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumbering for host kubestage2002.codfw.wmnet [07:37:28] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128732 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm [07:37:49] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128733 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm executed... [07:37:50] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128734 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering for host kubestage2002.codfw.wmnet completed:... [08:17:12] hello folks [08:17:27] as FYI I am starting in a bit the reimage of dragonfly-supernode2001 to Bookworm [08:20:09] 06serviceops, 06SRE: Migrate dragonfly-supernodes to Bookworm - https://phabricator.wikimedia.org/T332011#10128871 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin2002 for host dragonfly-supernode2001.codfw.wmnet with OS bookworm [08:25:20] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128883 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2320.codfw.wmnet [08:25:21] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128884 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2321.codfw.wmnet [08:25:28] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128885 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2322.codfw.wmnet [08:25:45] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128886 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2332.codfw.wmnet [08:27:00] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128888 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2031.codfw.wmnet [08:27:21] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128889 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2034.codfw.wmnet [08:30:01] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128909 (10MoritzMuehlenhoff) Something went wrong with the 2430 rename, it's still showing up in Puppetboard: https://puppetboard.wikimedia.org/node/mw2430... [08:31:19] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128913 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2008.codfw.wmnet [08:31:41] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128914 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2010.codfw.wmnet [08:32:02] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128915 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2018.codfw.wmnet [08:34:14] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128919 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2025.codfw.wmnet [08:34:24] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128925 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2027.codfw.wmnet [08:34:46] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128926 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2028.codfw.wmnet [08:34:49] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to bullseye - https://phabricator.wikimedia.org/T331969#10128924 (10elukey) Tried to file a patch but I realized that we don't have the `helm` package for Bookworm/Bullseye, so the build fails. I am wondering if the current version of chartmuseum r... [08:34:50] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128927 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2029.codfw.wmnet [08:35:16] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128929 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2033.codfw.wmnet [08:35:25] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128930 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2035.codfw.wmnet [08:36:03] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128931 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2055.codfw.wmnet [08:36:06] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128933 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2057.codfw.wmnet [08:36:56] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128936 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2055.codfw.wmnet [08:37:15] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10128944 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: kubernetes2054.codfw.wmnet [08:46:44] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to bullseye - https://phabricator.wikimedia.org/T331969#10129014 (10JMeybohm) helm3 should be fine. It might as well be that the build-dependency is not required as we're doing a full vendor anyways. I don't recall why the it is there, sorry [08:54:36] 06serviceops, 06SRE: Migrate dragonfly-supernodes to Bookworm - https://phabricator.wikimedia.org/T332011#10129047 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin2002 for host dragonfly-supernode2001.codfw.wmnet with OS bookworm completed: - dragonfly-supernode2001 (**PASS**... [08:56:16] 06serviceops, 06SRE: Migrate dragonfly-supernodes to Bookworm - https://phabricator.wikimedia.org/T332011#10129052 (10elukey) First node reimaged! Everything looks good afaics. Next steps: * Wait some MW deployments to make sure that nothing unexpected pops up. * Reimage the eqiad VM. [09:08:22] dragonfly seems good! [09:08:32] will wait a couple of days and I'll do 1001 as well [09:08:41] I left a proposal in https://phabricator.wikimedia.org/T332015#10129086 for the reimage of poolcounter [09:08:51] lemme know if it makes sense [09:09:23] 06serviceops: Migrate poolcounter hosts to bullseye - https://phabricator.wikimedia.org/T332015#10129086 (10elukey) Hey folks, as far as I can get both `poolcounter` (debian upstream) and `poolcounter-prometheus-exporter` (bookworm-wikimedia) are already good to go, so we could attempt a reimage of one of the no... [09:45:00] 06serviceops: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10129170 (10MoritzMuehlenhoff) [10:14:32] 06serviceops: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10129278 (10elukey) Better procedure after chatting with Moritz: * Create a new VM on Bookworm, test that everything is ok etc.. * Swap a "live-buster" poolcounter with the new VM (via MW deployment). * Let it bake f... [10:21:19] inflatador: sorry for the delay - I have absolutely no idea about that diff on the api-gateway SLO, I don't think I've ever edited that file before :/ [10:26:56] Hello, I deployed a new namespace for the wikidata-query-gui to staging-codfw, which is looking good so far. Any objections to continue with the other wikikube clusters? [10:26:56] change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071075 [10:29:19] jelto: go ahead [10:30:27] thx I'll do [10:59:21] 06serviceops: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10129362 (10Clement_Goubert) Sounds good. From what I can see, `poolcounter2004.codfw.wmnet` and `poolcounter1005.eqiad.wmnet` are the least used, depending on whether you plan on doing the update before or after {T37... [11:01:25] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129371 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2293.codfw.wmnet [11:01:35] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129372 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2295.codfw.wmnet [11:01:43] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129373 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2296.codfw.wmnet [11:02:19] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129375 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2312.codfw.wmnet [11:02:27] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129376 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2316.codfw.wmnet [11:02:58] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129391 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2317.codfw.wmnet [11:03:02] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129393 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2318.codfw.wmnet [11:03:08] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129394 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2319.codfw.wmnet [11:03:46] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129395 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2377.codfw.wmnet [11:03:48] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129396 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2378.codfw.wmnet [11:04:47] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129397 (10MoritzMuehlenhoff) mw2379 is also still in puppetboard: https://puppetboard.wikimedia.org/catalog/mw2379.codfw.wmnet [11:05:33] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129398 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2380.codfw.wmnet [11:05:38] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129399 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2381.codfw.wmnet [11:05:41] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129400 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2382.codfw.wmnet [11:05:49] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129401 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2383.codfw.wmnet [11:06:15] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129404 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2384.codfw.wmnet [11:06:20] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129405 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2385.codfw.wmnet [11:06:26] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129406 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2386.codfw.wmnet [11:06:34] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129407 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2387.codfw.wmnet [11:07:03] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129408 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2388.codfw.wmnet [11:07:06] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129409 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2389.codfw.wmnet [11:07:35] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129410 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2402.codfw.wmnet [11:07:41] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129411 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2406.codfw.wmnet [11:07:58] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129412 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2407.codfw.wmnet [11:08:55] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129416 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2420.codfw.wmnet [11:09:09] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129418 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2422.codfw.wmnet [11:09:22] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129419 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2423.codfw.wmnet [11:09:38] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129420 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2430.codfw.wmnet [11:09:47] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129421 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2434.codfw.wmnet [11:09:55] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129422 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2435.codfw.wmnet [11:10:01] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129423 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by jmm: for 1 hosts: mw2379.codfw.wmnet [11:24:10] 06serviceops, 10Thumbor: Majority of thumbor containers on pods occasionally getting into a stuck state - https://phabricator.wikimedia.org/T374350 (10hnowlan) 03NEW [11:38:47] I'll be moving dragonfly-supernode2001 to Puppet 7 in a few [12:39:30] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129651 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw2431 to wikikube-worker2104 completed: - mw2431 (**PAS... [12:43:27] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129663 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by kamila@cumin1002 Renumbering for host wikikube-worker2104.codfw.wmnet [12:43:49] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129664 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker2104.codfw.wmnet with OS bullseye [13:27:25] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.09.06 - 2024.09.27), 07Kubernetes: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10129815 (10bking) a:03bking [13:28:36] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.09.06 - 2024.09.27), 07Kubernetes: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10129817 (10bking) [13:28:38] 06serviceops: Mediawiki mcrouter errors on codfw - https://phabricator.wikimedia.org/T374025#10129818 (10jijiki) a:03jijiki [13:29:08] 06serviceops: Mediawiki mcrouter errors on codfw - https://phabricator.wikimedia.org/T374025#10129827 (10jijiki) [13:51:20] 06serviceops, 10Scap, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#10129946 (10Jdforrester-WMF) I just ran into this – no error, no warning, I... [13:54:41] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10129966 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker2104.codfw.wmnet with OS bullseye co... [14:02:18] 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130008 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by kamila@cumin1002 Renumbering for host wikikube-worker2104.codfw.wmnet com... [14:03:09] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374249#10130013 (10kamila) [14:31:23] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2428 to wikikube-worker2105 completed: - mw2428 (**PASS**) - ✔️ Downtime... [14:36:33] 06serviceops, 06Infrastructure-Foundations: Issues reimaging kubernetes workers due to user conflicts in systemd-timesyncd - https://phabricator.wikimedia.org/T373819#10130188 (10joanna_borun) p:05Triage→03Medium [14:36:44] FYI I've merged the ported mw alerts.git so you might see them on -operations . Left to do is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071193 [14:37:26] godog: yep, centralauth one just fired [14:37:52] I saw that claime ! also resolved mmhh [14:40:18] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130215 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2429 to wikikube-worker2106 completed: - mw2429 (**PASS**) - ✔️ Downtime... [14:45:18] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130257 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumbering for host wikikube-worker2105.codfw.wmnet [14:45:24] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2105.codfw.wmnet with OS bullseye [14:50:01] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130275 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumbering for host wikikube-worker2106.codfw.wmnet [14:50:12] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130276 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2106.codfw.wmnet with OS bullseye [15:03:33] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Comm Error: backplane 0 when reimaging wikikube-worker2095 - https://phabricator.wikimedia.org/T374258#10130345 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm @hnowlan since it was unpingable anyway, I power cycled the server and reseated the cable b... [15:09:05] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130387 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2095.codfw.wmnet with OS bullseye [15:26:35] 06serviceops: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366 (10jijiki) 03NEW [15:27:27] 06serviceops: Mediawiki mcrouter errors on codfw - https://phabricator.wikimedia.org/T374025#10130492 (10jijiki) 05Open→03Resolved We have concluded so far, that the errors are related to T374366, so we are closing this for now. [15:27:46] 06serviceops: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10130498 (10jijiki) [15:33:50] 06serviceops: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10130549 (10jijiki) [15:33:52] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130550 (10jijiki) [15:46:14] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130609 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2095.codfw.wmnet with OS bullseye completed: - wikikube-w... [15:48:18] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission mw2260.codfw.wmnet, mw2267.codfw.wmnet - https://phabricator.wikimedia.org/T374018#10130610 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:02:02] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130710 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2105.codfw.wmnet with OS bullseye completed: - wikikube-... [16:06:46] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10130731 (10JMeybohm) [16:11:05] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374249#10130741 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:11:07] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130746 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumbering for host wikikube-worker2105.codfw.wmnet completed: - wikikube-w... [16:11:16] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130753 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2106.codfw.wmnet with OS bullseye completed: - wikikube-... [16:11:18] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10130750 (10jijiki) [16:14:33] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10130738 (10jijiki) @Ladsgroup and I tested wikitech.wikimedia.org on mwdebug1001 The following tests worked as expected: * Reading and articl... [16:19:12] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130803 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumbering for host wikikube-worker2106.codfw.wmnet completed: - wikikube-w... [16:20:24] 06serviceops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10130807 (10ops-monitoring-bot) Cookbook cookbooks.sre.debmonitor.remove-hosts run by cgoubert: for 2 hosts: mw[2428-2429].codfw.wmnet [16:23:47] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10130812 (10Ladsgroup) Regarding jobs, since we don't have an easy control where it'll be consumed, it will use the default wikitech config whic... [16:38:12] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374380 (10Clement_Goubert) 03NEW [16:56:57] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373591#10131008 (10Jhancock.wm) mw2379 is causing an alert in netbox https://netbox.wikimedia.org/extras/scripts/results/89809/ [16:58:28] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T374249#10131011 (10Jhancock.wm) mw2431 is causing an alert in netbox https://netbox.wikimedia.org/extras/scripts/results/89809/ [17:08:51] 06serviceops, 10MW-on-K8s, 07Datacenter-Switchover, 13Patch-For-Review: Update DC switchover cookbooks to handle maintenance scripts on k8s - https://phabricator.wikimedia.org/T359130#10131055 (10RLazarus) [17:36:48] Hmm, I'm seeing errors from non-wikifunctions.org on mw-wikifunctions; e.g. https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.09.09?id=N9bX15EB8AvXVHu89xna is from foundation.wikimedia.org – is this expected? [17:38:12] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#10131212 (10bking) [17:42:26] 06serviceops, 10Charts, 10Wikimedia-Extension-setup, 07Epic, 07Wikimedia-extension-review-queue: Epic: Deploy Chart extension in production - https://phabricator.wikimedia.org/T369944#10131220 (10CCiufo-WMF) [18:18:07] 06serviceops, 10LPL Essential, 10MinT, 10Community Wishlist (Translations), 10Community-Tech (Island Fox (Sept 9 - 20)): Caching service request for MinT - https://phabricator.wikimedia.org/T370755#10131347 (10MusikAnimal) [21:23:36] James_F: that's probably the standard "appserver" httpbb test suite running against mw-wikifunctions (which includes this check [0]). in fact, the timing on that log entry matches the hourly timer on cumin1002. [21:23:36] I see similar calls to other non-wikifunctions domains in the access log with the httpbb UA, though not this specific one given the sampling involved. [21:23:36] which is to say, working as implemented, if kind of weird (e.g., maybe wikifunctions should have its own test suite?). [21:23:36] [0] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/httpbb/appserver/test_wikimania_wikimedia.yaml#53 [21:43:48] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.09.06 - 2024.09.27), 07Kubernetes, 13Patch-For-Review: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10132095 (10bking) A few observations from pairing session with @RKemper : -...