[07:20:02] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9836573 (10ABran-WMF) [07:20:15] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9836574 (10ABran-WMF) [07:24:38] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9836585 (10ABran-WMF) [10:18:41] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9837047 (10MatthewVernon) @cmooney can you give me a time estimate for when you're going to be doing these, please? I'd like to put notes in my calendar. [10:21:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837068 (10MatthewVernon) From a swift POV, should just be "check cluster is happy afterwards" [10:22:28] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837074 (10ABran-WMF) [10:22:39] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837077 (10ABran-WMF) [10:22:52] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837072 (10MatthewVernon) [10:23:08] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837079 (10ABran-WMF) [10:23:18] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837080 (10ABran-WMF) [10:23:28] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837081 (10MatthewVernon) Again, from a swift POV, this should just be a case of checking the cluster is happy after... [10:23:40] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9837083 (10ABran-WMF) [10:23:50] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837085 (10ABran-WMF) [10:24:00] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837084 (10MatthewVernon) [10:24:16] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837086 (10ABran-WMF) [10:24:34] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837089 (10ABran-WMF) [10:24:45] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837091 (10ABran-WMF) [10:24:59] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837092 (10ABran-WMF) [10:25:09] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837093 (10ABran-WMF) [10:25:27] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837099 (10ABran-WMF) [10:25:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837103 (10MatthewVernon) [swift-wise, just need to check cluster OK afterwards] [10:26:12] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837104 (10MatthewVernon) [10:27:03] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837106 (10MatthewVernon) [10:27:17] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837111 (10MatthewVernon) [swift-wise, cluster health just needs checking afterwards] [10:28:30] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837115 (10MatthewVernon) [10:29:40] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837117 (10MatthewVernon) [from a swift POV, just need to check cluster OK afterwards] [10:30:59] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837121 (10MatthewVernon) [10:31:59] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837137 (10MatthewVernon) ms-fe1014 will need depooling before this work is done (and then repooling afterwards). T... [10:33:11] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837166 (10MatthewVernon) [10:33:24] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837167 (10MatthewVernon) [swift-wise, just need to check cluster OK afterwards] [10:35:05] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837189 (10MatthewVernon) [10:36:28] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837210 (10MatthewVernon) swift-wise, just need to check the cluster's happy afterwards. moss-be1003 is part of the... [10:37:55] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837218 (10MatthewVernon) [10:40:57] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837237 (10MatthewVernon) [10:42:00] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837239 (10MatthewVernon) I'm on annual leave this day, so someone else will have to handle the ms frontend, which n... [10:42:32] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837241 (10MatthewVernon) [10:44:39] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837254 (10MatthewVernon) swift-wise, this should just be a case of checking the cluster is healthy afterwards. I'm... [10:44:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837257 (10MatthewVernon) [10:45:44] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837261 (10MatthewVernon) From the swift POV, this is just checking the cluster is happy afterwards. I'm on annual l... [16:17:09] 10Wikimedia-Apache-configuration, 10MediaWiki-Configuration, 06Security-Team, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9838592 (10mmartorana) a:03mmartorana [16:25:00] sukhe: o/ [16:25:04] hello! [16:25:15] one second, chrome is acting up. i think i need to restart it [16:25:17] I am looking up where you last left; brett was on it but he is out on PTO today [16:25:24] i'll be back in just a minute [16:25:28] np [16:26:11] alright, that seems better [16:26:38] * dduvall takes a look at the checklist [16:26:56] sukhe: i believe we're at step 3 of https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service [16:27:05] note that probes were not silenced, however [16:27:15] looking [16:27:54] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035543/2 [16:28:09] yes, exactly [16:28:30] yeah, it's not paging so it's fine. I will sync up later with olly to understand why we were unable to silence these. but let's take it from here and I will silence any alerts that pop up [16:28:37] i submitted the patch chain according (i think) to the checklist on that wikitech page [16:28:49] ok, sounds good [16:28:51] yep thanks, we will just go through it [16:30:27] one thing that wasn't super clear to me was which changes needed to be deployed _after_ a k8s/helm destroy, but i sort of guessed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035798/1 being the last in the chain and to be merged after k8s undeployment [16:31:12] I am also not very clear on that part but let's see [16:31:27] merging current changes [16:38:54] still running sorry, rolling out changes [16:39:11] np, very exciting [16:39:13] haha [16:39:16] if that counts sure! [16:39:19] :D [16:39:31] i just like the idea of undeploying something for once [16:39:38] that's true [16:54:41] dduvall: secondary done, will move on to the primary LVS shortly [16:54:54] ack [16:58:23] 10Wikimedia-Apache-configuration, 10MediaWiki-Configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9838900 (10sbassett) p:05Triage→03Medium [17:08:21] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9838952 (10Reedy) Are you sure? I think it just hasn't been renamed... From `hieradata/common/mediawiki.yaml`... [17:12:55] sukhe: seems like now might be an ok time to do `helmfile destroy` as well. not sure [17:13:33] dduvall: I think once we merge the last patch 1035798 then yeah. it will be a learning experience for all of us to see how that goes :P [17:13:50] yeah, for sure [17:14:04] maybe i can make a note on wikitech either way [17:14:30] https://wikitech.wikimedia.org/wiki/Kubernetes/Remove_a_service [17:14:40] this is in a way separate from the LVS changes we are making right now [17:15:00] right [17:15:05] I mean it's definitely a part of this change but they are separate [17:15:47] whether our helmfile wrapper interacts with the discovery url is unclear to me, but so long as it doesn't, this seems right [17:16:27] maybe we don't wrap helmfile anymore actually [17:17:18] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9838998 (10Papaul) [17:18:20] dduvall: that wraps it up on the LVS/CDN side at least [17:18:31] \ [17:18:33] do you want me to take care of the https://wikitech.wikimedia.org/wiki/Kubernetes/Remove_a_service or do you have access? [17:18:34] \o/ [17:18:46] I have nveer done it but happy for it to be a first for me [17:18:47] i have access for that. i'll run `helmfile destroy` now [17:19:10] you have already removed some of those bits but stuff like https://github.com/wikimedia/operations-deployment-charts/tree/master/helmfile.d/services/blubberoid remain [17:19:49] right. i'll remove that once i know for sure it's totally removed from our k8s clusters [17:19:52] yep [17:20:06] gl, please feel free to ping me here in case you need something that requires access/anything else [17:20:52] will do, thanks sukhe! [17:24:05] removed from puppetdb as well [17:52:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9839193 (10Papaul) [17:53:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9839201 (10Papaul) [17:58:36] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839218 (10sbassett) wikimediafoundation.org definitely points to Automattic: `lang=shell $ whois 192.0.66.2... [18:32:37] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839366 (10Reedy) The wiki was moved to https://foundation.wikimedia.org as per the ‘server_name’, and is stil... [18:35:10] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839376 (10sbassett) Ok, so wikimediafoundation.org and foundation.wikimedia.org are just kind of obnoxiously... [18:35:30] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839373 (10Reedy) #foundation.wikimedia.org {T188776} [18:36:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102 (10RobH) 03NEW p:05Triage→03Medium [18:36:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102#9839398 (10RobH) [18:36:37] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839400 (10Reedy) If there’s nothing custom in that docroot folder (I haven’t checked as not at a computer atm... [18:37:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102#9839422 (10RobH) [18:38:47] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839431 (10sbassett) Looks like all symlinks to the default dir? https://gerrit.wikimedia.org/r/plugins/gitil... [20:12:11] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: HAProxy should not log information we don't actually need - https://phabricator.wikimedia.org/T365566#9839808 (10Fabfur) 05Open→03Resolved [20:14:36] 10Wikimedia-Apache-configuration, 06Security-Team, 07SecTeam-Processed, 07Security: Remove docroot/wikimediafoundation.org/ folder from mediawiki-config - https://phabricator.wikimedia.org/T366005#9839823 (10Reedy) ` ~/git/operations/mediawiki-config/docroot$ ls -al total 48 drwxrwxr-x 12 reedy reedy 4096... [21:58:26] 06Traffic, 10MW-on-K8s, 06serviceops: XWD: Allow choosing datacentre in k8s-mwdebug - https://phabricator.wikimedia.org/T365478#9840260 (10jijiki) 05In progress→03Resolved