[05:34:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11422574 (10Papaul) @ayounsi @cmooney please see below the steps to replace the loopback IPs on cr3/4-ulsfo and mr1-ulsfo If all this looks good, I will setup... [08:13:44] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11422660 (10Diskdance) I can see that potential DoS factor is a valid con... [09:24:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11422756 (10ayounsi) Awesome, thx! The loopbacks are also in Puppet : https://github.com/search?q=repo%3Awikimedia%2Foperations-puppet%20198.35.26.193&type=co... [09:58:51] 10Acme-chief, 06Traffic: Let's Encrypt Decreasing Certificate Lifetimes to 45 Days - https://phabricator.wikimedia.org/T411467 (10Vgutierrez) 03NEW [09:58:59] 10Acme-chief, 06Traffic: Let's Encrypt Decreasing Certificate Lifetimes to 45 Days - https://phabricator.wikimedia.org/T411467#11422849 (10Vgutierrez) p:05Triage→03Medium [10:18:46] 10Acme-chief, 06Traffic: Let's Encrypt Decreasing Certificate Lifetimes to 45 Days - https://phabricator.wikimedia.org/T411467#11422900 (10Vgutierrez) [12:02:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11423243 (10cmooney) >>! In T405499#11411590, @VRiley-WMF wrote: > Hey @cmooney It has been reused for that purpose, however it's still being worked on... [12:05:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: ssw1-d8-eqiad cross-rack links incorrect in Netbox - https://phabricator.wikimedia.org/T411480 (10cmooney) 03NEW p:05Triage→03Medium [12:51:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11423398 (10cmooney) @papaul as @ayounsi mentions you need to change it in puppet where it is also. Principally to change what IPs the hosts doing BGP are goi... [13:45:48] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910#11423686 (10cmooney) 05Open→03Resolved Thanks to the awesome work of @jhathaway this is no longer a requirement. We can use `--no82` with a host in BIOS... [14:46:27] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503 (10daniel) 03NEW [14:47:18] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11423971 (10daniel) [14:47:43] FIRING: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [14:48:52] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11423977 (10daniel) [14:50:40] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11423984 (10daniel) For the record, I asked @taavi about including information about the tool or user in requests coming from WMCS. He said it's not possible because the tools use HTTPS, we c... [15:02:43] RESOLVED: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [15:07:38] hello traffic friends - any concerns if at some point in the next couple of hours I merge / apply [0] to move eqiad PyBals over to conf1008 temporarily? this is a pre-step for migrating etcd on conf1007 to cfssl/pki [1] (we did something similar in codfw). [15:07:38] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1213601 [15:07:38] [1] https://phabricator.wikimedia.org/T352245 [15:18:10] swfrench-wmf: +1ed [15:18:25] ah, thanks vgutierrez! [15:31:40] 10netops, 06Infrastructure-Foundations, 06serviceops, 07Puppet: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508 (10taavi) 03NEW [15:40:54] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11424246 (10taavi) I don't think we currently have any places outside of https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_IP_space that publish our IP space. Would it be helpful if we publi... [16:12:13] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11424434 (10BCornwall) @cmooney Yes, that looks good to me. We can still go for Dec 3 - fe... [16:13:55] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11424463 (10BCornwall) @cmooney Yes, that looks good to me. We can still go for Dec 4 - fe... [16:32:23] vgutierrez: silly question, we can IPIP from a public LB IP to realservers w/ private IPs no problem, right? [16:33:38] sure [16:34:24] all of our cp servers for instance only have private iPs [16:35:00] the VIP is on the loopback interface as usual [16:35:33] but as long as the core router / L3 switch is able to route the packets it should be fine [16:39:49] ack [16:44:07] hello again - just a heads-up that I'll be moving forward with those PyBal config changes shortly :) [16:49:07] vg & sukhe I know you're probably still in a meeting, no rush I'm about to break for lunch, but please ptal at https://phabricator.wikimedia.org/T365259#11424689 [16:58:49] cdanis: thank you [16:58:51] reading [16:59:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11424827 (10Papaul) @ayounsi @cmooney thanks for the feedback. [17:02:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11424870 (10Papaul) @ssingh We are planning on doing the first phase(loopback IP change on core routers and management router) of the ULSFO refresh next week D... [17:04:58] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11424887 (10BCornwall) [17:08:39] * swfrench-wmf is done messing with eqiad PyBals now [17:21:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: ssw1-d8-eqiad cross-rack links incorrect in Netbox - https://phabricator.wikimedia.org/T411480#11425017 (10VRiley-WMF) 05Open→03Resolved Updated cable paths for the new switches in D8 to E1 and F1 [17:29:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11425063 (10VRiley-WMF) 05Open→03In progress removing and updating cables [17:41:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11425144 (10VRiley-WMF) 05In progress→03Resolved a:03VRiley-WMF this is completed [17:54:41] hello again traffic friends - FYI, if all goes according to plan during the upcoming infra window, I plan to merge and apply [0] to move eqiad PyBals back to conf1007. let me know if there are any concerns / conflicts. [17:54:41] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1213603 [17:55:04] swfrench-wmf: thanks for checking as always, no issues [17:56:02] * swfrench-wmf thumbs up [18:06:39] 10Wikimedia-Apache-configuration, 13Patch-For-Review: nb.wikiversity.org redirects to 404 page on BetaWikiversity - https://phabricator.wikimedia.org/T407553#11425346 (10Pppery) 05Open→03Resolved [18:46:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11425543 (10ssingh) >>! In T408892#11424869, @Papaul wrote: > @ssingh We are planning on doing the first phase(loopback IP change on core routers and managemen... [20:00:32] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11425771 (10daniel) >>! In T411503#11424246, @taavi wrote: > I don't think we currently have any places outside of https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_IP_space that publish our... [21:13:24] curious to know more about the NUMA issues sukh.e mentioned yesterday if there's a breadcrumb to follow, like a phab issue [21:23:38] 06Traffic, 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team (Radar): Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552 (10Krinkle) 03NEW [21:29:22] 06Traffic, 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426103 (10taavi) Duplicate of {T401489}? [21:40:14] 06Traffic, 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426155 (10Krinkle) 05Open→03Invalid Thanks. [21:45:38] 06Traffic, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Separate error templates for 5xx (server errors) and 4xx (IP blocks) or parametrize Varnish messaging - https://phabricator.wikimedia.org/T401489#11426182 (10bd808) [21:45:46] 06Traffic, 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426185 (10bd808) →14Duplicate dup:03T401489 [21:47:14] ori: sorry, it seems like the ping got lost in the sea of other pings. I will follow up tomorrow morning when v<>g is online [21:47:22] * sukhe adds a reminder [21:52:49] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11426194 (10Tgr) >>! In T368344#11422660, @Diskdance wrote: > So should w... [23:24:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11426444 (10Papaul) @ssingh yes we have to depool the site, yes 10 AM CT [23:30:17] cccccbukvgbcrdlerhjhgkibbfddkcurgbnrufvvguin [23:32:33] I'll raise you a cccccctrnruvnenrtcbcbidhrvhtitfdrcfcctnfkkie [23:47:50] it's because it's the nano model that disappears fully into the slot.. it's just an inch from the tab key for the little finger. but before it was the nano it stood out the port and physically broke when dropping the laptop