[08:28:25] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120363 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumbering for host wikikube-w... [08:28:36] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker2088.codfw.... [09:00:29] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120464 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from mw2434 to wik... [09:01:32] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120472 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from mw2435 to wik... [09:02:20] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120477 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [09:02:33] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120478 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering f... [09:03:08] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120480 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [09:03:18] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120481 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering f... [09:03:48] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120482 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [09:04:09] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120483 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [09:04:55] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120484 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jayme@cumin1002 Renumberi... [09:05:11] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120485 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wiki... [09:12:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120495 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube... [09:19:52] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373916#10120515 (10JMeybohm) [09:19:57] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120516 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering f... [09:23:41] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10120529 (10JMeybohm) [09:23:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10120531 (10JMeybohm) [09:54:38] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120668 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2089.codfw.wmne... [09:59:46] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120679 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering for host wikikube-worke... [10:03:43] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120684 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by hnowlan@cumin1002 Renumbering for host wikikube... [10:03:56] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120685 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2084.codf... [10:47:35] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373916#10120858 (10hnowlan) [10:57:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2084.codfw.wm... [11:01:05] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120886 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by hnowlan@cumin1002 Renumbering for host wikikube-wor... [11:21:22] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120933 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker2090.codfw.wmne... [11:22:12] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120940 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jayme@cumin1002 Renumbering for host wikikube-worke... [11:25:22] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120948 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumbering for host wikikub... [11:26:49] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10120950 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2029.cod... [12:22:07] conf2005 is in this afternoon's network migration so it'll have a network disconnection that should be short, is there anything in particular we should do for it? akosiaris ? [12:33:30] I'll go ahead and depool the k8s nodes [12:38:27] claime: the etcd mirror runs on that node IIRC, and it could page if broken (not sure if we changed it but it was like that in the past) [12:39:01] maybe we could check the status of etcd and zookeeper in codfw, to make sure both ensembles are fine etc.. [12:39:07] (so if we get down a node it will be fine) [12:41:47] oh it's 1630 UTC, I'll depool the k8s nodes just before the staff meeting then [12:41:53] no need to it this early [12:43:05] thanks [12:43:38] etcd looks ok in codfw [12:47:36] ugh, it's the zookeeper leader [12:51:22] other than that zookeeper seems in good health [12:51:36] so it's only the etcd mirror that may pose issue [12:54:46] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121332 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2029.codfw.w... [12:58:42] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121341 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumbering for host wikikube-wo... [13:01:24] yeah [13:01:32] hmm [13:01:37] * akosiaris thinking [13:05:04] claime: probably bite the bullet and restart the mirror if it breaks? [13:05:19] the rest should be mostly fine, we might have to restart pybal's that connect to it if any [13:05:24] pybals* [13:05:43] in the past they had croaked badly when losing connectivity to etcd for long enough [13:07:50] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114 (10Andrew) 03NEW [13:10:21] for the zk leader no problem, it will move to another health node [13:16:36] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10121414 (10Andrew) cc'ing @Dzahn because he's done some wikitech-static maintenance in the past and might be inte... [13:57:09] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10121624 (10jijiki) a:05jijiki→03None [15:00:41] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121908 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw2420 to wi... [15:01:49] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121910 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw2421 to wi... [15:03:33] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121913 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by kamila@cumin1002 Renumber... [15:04:04] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121914 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [15:12:44] topranks: our k8s nodes are depooled [15:12:55] claime: thanks!! [15:17:24] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121992 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by kamila@cumin1002 Renumber... [15:17:45] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10121994 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [16:49:41] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10122317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [16:49:43] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10122318 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by kamila@cumin1002 Renumbering... [16:51:21] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10122337 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [16:53:27] 06serviceops, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10122347 (10daniel) [16:55:24] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10122354 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by kamila@cumin1002 Renumbering... [16:55:41] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: kubernetes2035 (renamed to wikikube-worker2087) reporting "Comm Error: Backplane 0" - https://phabricator.wikimedia.org/T374019#10122351 (10Jhancock.wm) @JMeybohm power cycled it and reseated the connection between the system board and the backplane. looks like i... [17:06:49] Work completed on T373096 if there are any re-pools that need to be done [17:07:16] ty, will repool k8s nodes [17:07:22] etcdmirror seems to have survived [17:07:37] some good news :) [17:32:43] claime: we should chaosmonkey etcdmirror sometime [19:07:49] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10122931 (10CDanis) As it turns out this was discussed at the [[ https://www.mediawiki.org/wiki/Kubernetes_SIG | k8s SIG ]] but the bug didn't get updated. [[ https://www.medi... [20:53:24] 06serviceops, 06MW-Interfaces-Team, 06Traffic: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#10123234 (10BPirkle) > I was wondering if we have enough consensus to proceed in the path of having /api/ be rewritten in MediaWiki to /w/rest.php We do not have a consensus at this... [22:01:31] hnowlan just a heads-up, I updated our (Search) SLO dashboard, which applied an unmerged change to the API gateway draft SLO dashboard. Diff is at https://phabricator.wikimedia.org/P68729 if you'd like to take a look. (pinging you 'cause you were the last to edit the API Gateway wikitech page) [22:50:17] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10123586 (10cmooney) >>! In T344171#10081275, @CDanis wrote: >> It might be quite naive (and does not solve my concern from above) but could we have a subnet delegation or a fo... [23:09:54] 06serviceops, 06Data-Engineering, 06Data-Platform, 06SRE: DegradedArray email alerts for aqs1013 and aqs1014 are firing since April 18 - https://phabricator.wikimedia.org/T373490#10123659 (10andrea.denisse) [23:11:35] 06serviceops, 10MW-on-K8s, 10Shellbox, 10SRE-swift-storage, 13Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322#10123661 (10tstarling) 05In progress→03Open [23:54:38] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10123888 (10cmooney) >>! In T344171#10122931, @CDanis wrote: > In summary Alex is looking at configuring Calico's IPAM to keep a set of coredns pods on static IP addresses, whi...