[07:35:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10147484 (10JMeybohm) [07:35:33] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10147485 (10JMeybohm) [07:45:23] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to bullseye - https://phabricator.wikimedia.org/T331969#10147504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm [07:53:50] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147518 (10MoritzMuehlenhoff) [08:08:50] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147562 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm executed with errors: - chartmuseum2001 (... [08:09:20] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147563 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm [08:14:57] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147582 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm executed with errors: - chartmuseum2001 (... [08:15:18] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147583 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm [08:19:32] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147629 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm executed with errors: - chartmuseum2001 (... [08:25:13] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147634 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm [08:33:20] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147647 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm executed with errors: - chartmuseum2001 (... [08:37:22] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147668 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm [08:39:09] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10147672 (10elukey) Due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035854, the VM's RAM was bumped to 2G. [08:51:10] hello folks [08:51:25] qq - how do I deploy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1072717 ? Via helmfile on all affected namespaces? [08:52:11] elukey: https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#Automatic_deployment [08:52:39] elukey: uh, wrong link. there is a "no image rebuild" section [08:55:55] jayme: ah ack, so IIUC it would be better to use scap in this case - merge and then run scap sync-world with those options [08:57:48] elukey: yep [09:04:50] going to use the MW infra window to deploy (in ~1h) https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240916T1000 [09:20:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10147888 (10JMeybohm) a:03JMeybohm @akosiaris and I had a discussion about this and it seems pretty complex to aim for replacing ferm completely with... [09:24:58] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10147899 (10JMeybohm) This is what I currently see when puppet is fixing a manual change to 00_defs (`/usr/local/sbin/fer... [09:49:20] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10148056 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1002 for host chartmuseum2001.codfw.wmnet with OS bookworm completed: - chartmuseum2001 (**PASS**)... [09:57:42] chartmuseum2001 up and running with bookworm [09:57:48] lemme know if you see anything weird [09:58:49] moving to the network policies for poolcounter [10:05:15] aaand done [10:06:07] next step is to schedule https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1072206 [10:19:51] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10148243 (10elukey) The reimage of 2001 went fine, I just repooled it. Let's wait for a day before moving to 1001 so if anything weird comes up, we'll have a quick way to fix (depool 2001). N... [10:20:35] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10148252 (10elukey) a:05jhathaway→03elukey [10:26:24] 06serviceops, 13Patch-For-Review: Migrate poolcounter hosts to bookworm - https://phabricator.wikimedia.org/T332015#10148318 (10elukey) Thumbor has been migrated to the new poolcounter VMs, and the MW network policies support the new VM's IPs. Next steps: * Test the new poolcounter nodes on mwdebug (in the c... [10:45:06] 06serviceops: Migrate docker registry hosts to bookworm - https://phabricator.wikimedia.org/T332016#10148380 (10elukey) >>! In T332016#9930632, @JMeybohm wrote: >>>! In T332016#9929672, @Clement_Goubert wrote: >> @JMeybohm could you check the `httpbb` tests are still relevant and returning the expected results?... [11:26:46] 06serviceops, 06SRE, 13Patch-For-Review: Migrate chartmuseum to Bookworm - https://phabricator.wikimedia.org/T331969#10148515 (10MoritzMuehlenhoff) >>! In T331969#10148243, @elukey wrote: > The reimage of 2001 went fine, I just repooled it. Let's wait for a day before moving to 1001 so if anything weird come... [13:36:44] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, and 2 others: hewiki: Use backing node service instead of RESTBase on pregeneration changeprop rules - https://phabricator.wikimedia.org/T372749#10148885 (10Jgiannelos) [13:36:47] 06serviceops, 10Page Content Service, 07Code-Health-Objective: hewiki: Route mobile-html to the backing node service instead of RESTBase - https://phabricator.wikimedia.org/T372746#10148888 (10Jgiannelos) [13:36:51] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, and 2 others: hewiki: Use backing node service instead of RESTBase on pregeneration changeprop rules - https://phabricator.wikimedia.org/T372749#10148887 (10Jgiannelos) After a discussion with @Seddon it looks like h... [14:15:37] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10149040 (10joanna_borun) p:05Triage→03High [14:38:01] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Pre-switchover cookbook testing - https://phabricator.wikimedia.org/T374047#10149276 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from codfw to eqiad - finished with... [14:39:36] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Pre-switchover cookbook testing - https://phabricator.wikimedia.org/T374047#10149287 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad -... [16:30:14] hey folks [16:30:31] I've scheduled the poolcounter swap during tomorrow's MW infra window - https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240917T1000 [16:30:37] lemme know if you have anything against it [16:34:42] 👍 [18:31:36] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Pre-switchover cookbook testing - https://phabricator.wikimedia.org/T374047#10150494 (10Scott_French) No new issues discovered during the live test today. There were a couple of documentation tweaks to make as follow-on (e.g., 03-set-db-readonly no l...