[09:08:53] elukey: FWIW, I've done the first Dell thanos-be in codfw, and it did UEFI HTTP boot off the correct NIC. Presumably by saying this I have now jinxed all the others... [09:17:05] Emperor: hahahah exactly, you have enjoyed your last set of working servers :D [09:17:13] jokes aside, good to know! [11:13:18] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12114149 (10hnowlan) graphite2004 doesn't need depooling. centrallog will have some gaps in logs on-disk but that's unavoidable and low-impact. [12:46:13] netops, ok for this edit on clinic duty documentation? https://wikitech.wikimedia.org/w/index.php?title=SRE%2FClinic_Duty&diff=2435778&oldid=2426422 [13:53:55] 10netops, 06Infrastructure-Foundations, 06Traffic, 06cloud-services-team (FY2025/2026-Q3-Q4): LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12114971 (10ssingh) >>! In T430651#12109242, @cmooney wrote: > @fabfur have you any insight into... [14:40:25] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115161 (10cmooney) [14:55:39] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12115229 (10ayounsi) p:05Triage→03Medium [15:20:20] 10netops, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12115415 (10fnegri) [15:21:31] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12115433 (10fnegri) [15:23:58] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#12115460 (10fnegri) [15:24:16] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12115461 (10fnegri) [15:46:02] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115582 (10CWilliams-WMF) [18:58:23] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12116562 (10BTullis) I have cordoned `dse-k8s-worker2003` ` root@deploy2003:~# kubectl cordon dse-k8s-worker2003.codfw.wmnet node/dse-k8s-worker2003... [19:31:02] how to add a custom SSH host key into https://config-master.wikimedia.org/known_hosts ? like the one for gerrit.wikimedia.org that is in there but for the sshd on port 29418, not the standard sshd [21:24:10] mutante: I'm not sure offhand, modules/ssh/manifests/publish_fingerprints.pp, if that file is used to drive https://config-master.wikimedia.org/known_hosts, I don't see a custom option at present? [21:46:49] jhathaway: thanks. that seems to confirm what I was thinking - they come from puppetdb - but then it's how that custom one got into the db. maybe with the old version mentioned in there or manually somehow [21:47:08] so far traced back from wmf-laptop to config-master to puppetdb [21:47:53] will search in old tickets. it was probably someone asking "can we get gerrit in there" a couple years back [21:53:17] mutante: https://codesearch.wmcloud.org/puppet/?q=%40%40sshkey has traces of both the gerrit and gitlab keys on there [21:55:36] taavi: aah! the ssh_key resource exports it to the db. I think I knew this once and needed the reminder. thanks [21:55:54] @@sshkey even.. let's see [21:57:19] and that's also where the gerrit-lb machines were added:) ok, got it :) [21:58:43] and dancy just uploaded a patch for that. duh. sometimes it's just about waiting a few minutes