[05:02:23] 10Traffic, 10DNS, 10SRE, 10Software-Licensing: Add LICENSE to operations/dns scripts - https://phabricator.wikimedia.org/T291323 (10Marostegui) p:05Triage→03Medium [05:10:04] 10HTTPS, 10SRE, 10VPS-project-Codesearch, 10serviceops: Codesearch main page redirect uses http instead of https - https://phabricator.wikimedia.org/T290819 (10Marostegui) p:05Triage→03Medium [06:05:48] hi! I want to add 3 new LVSs for shellbox: https://gerrit.wikimedia.org/r/q/topic:shellbox2-lvs - let me know when a good time to coordinate that would be [06:24:57] (VarnishTrafficDrop) firing: 69% GET drop in text@codfw during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [06:29:57] (VarnishTrafficDrop) resolved: 69% GET drop in text@codfw during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [06:43:22] 10HTTPS, 10VPS-project-Codesearch: Codesearch main page redirect uses http instead of https - https://phabricator.wikimedia.org/T290819 (10Legoktm) Seems like we need something like https://stackoverflow.com/questions/14810795/flask-url-for-generating-http-url-instead-of-https/37842465#37842465 in codesearch. [08:42:57] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10Volans) [08:43:01] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10Volans) 05Resolved→03Open @ssingh I understand there was some issue with the DNS setup between Netbox automation and manual records. I'll try to shade some light here: - When an... [08:47:43] 10Traffic, 10MW-on-K8s, 10Performance-Team, 10Release-Engineering-Team, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) I have some alternative ideas. Specifically, right now we have a limited number of different clusters, due to the complexity of c... [08:49:04] 10Traffic, 10MW-on-K8s, 10Performance-Team, 10Release-Engineering-Team, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) I forgot to add: offering the beta feature would be nice, and given it only regards logged-in users, it would not need a split of... [12:40:02] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10BBlack) Thanks for the clarity, makes a lot of sense! We **can** make this work in either direction, I think (manual or automatic for this handful of IPs/hostnames which occupy thes... [13:53:32] 10netops, 10Infrastructure-Foundations, 10serviceops: TCP retransmissions in eqiad and codfw - https://phabricator.wikimedia.org/T291385 (10jijiki) [14:09:06] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10Volans) >>! In T289536#7365249, @BBlack wrote: > Thanks for the clarity, makes a lot of sense! > > We **can** make this work in either direction, I think (manual or automatic for th... [14:09:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: TCP retransmissions in eqiad and codfw - https://phabricator.wikimedia.org/T291385 (10jijiki) [14:13:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: TCP retransmissions in eqiad and codfw - https://phabricator.wikimedia.org/T291385 (10cmooney) Thanks Effie. I think as well as the microbursts / drops you observed at the server-side, on the 1G interfaces, performance is probably impacted by on... [14:28:39] 10Traffic, 10Infrastructure-Foundations, 10SRE: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10MoritzMuehlenhoff) For production: * OpenSSL in Buster and Bullseye is not affected (only ship OpenSSL 1.1) * OpenSSL updates for openssl 1.0.2 in St... [14:38:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: TCP retransmissions in eqiad and codfw - https://phabricator.wikimedia.org/T291385 (10cmooney) Ok so looking at the results from the two hosts in question I'm not sure we can make any definitive conclusions. Following the switchover back to eqia... [15:50:03] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ssingh) >>! In T289536#7365588, @Volans wrote: > That said Netbox is not and will probably never be (from upstream comments) a DNS source of truth. We already have cases not well co... [16:42:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10procurement: Move AMS-IX port to 802.1q tagged and get "private vlan" added - https://phabricator.wikimedia.org/T291407 (10cmooney) [17:54:33] 10Traffic, 10SRE, 10MW-1.35-notes (1.35.0-wmf.40; 2020-07-07), 10Patch-For-Review, and 2 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Legoktm)