[09:27:09] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Matrix, 10Operations: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs - https://phabricator.wikimedia.org/T223835 (10Volans) p:05Triage→03Normal [12:48:49] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Matrix, and 2 others: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs - https://phabricator.wikimedia.org/T223835 (10jbond) @Tgr while reviewing the change created by volans i noticed that currently `wikimedia.modular.im.` dose not exi... [13:36:09] 10Traffic, 10Operations: ATS is currently adding its own server header - https://phabricator.wikimedia.org/T224119 (10Vgutierrez) [13:36:26] 10Traffic, 10Operations: ATS is currently adding its own server header - https://phabricator.wikimedia.org/T224119 (10Vgutierrez) p:05Triage→03Normal [13:41:51] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10BBlack) @HMarcus - The record is live, can you try the validation and let me know how it goes? Note there was already another google site verification token like this at the s... [14:01:35] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache upload nodes in esams - https://phabricator.wikimedia.org/T222937 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts: ` ['cp3046.esams.wmnet'] ` The log can be found i... [14:38:20] 10Traffic, 10DNS, 10Operations: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10Maintenance_bot) [14:53:05] 10netops, 10Operations, 10Wikimedia-Logstash, 10User-herron: Migrate network device syslogs to Kafka logging pipeline - https://phabricator.wikimedia.org/T224128 (10herron) p:05Triage→03Normal [15:21:50] 10Traffic, 10Operations, 10ops-eqiad: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293 (10Maintenance_bot) [15:32:09] 10Acme-chief, 10HTTPS, 10Traffic, 10Operations: acme-chief: Validate that configured certificates can be actually issued - https://phabricator.wikimedia.org/T220518 (10Maintenance_bot) [15:32:27] 10Traffic, 10MediaWiki-extensions-UrlShortener, 10Operations, 10User-Ladsgroup: Make UrlShortener 404s cacheable - https://phabricator.wikimedia.org/T220190 (10Maintenance_bot) [15:34:01] 10Acme-chief: Implement server-side OCSP stapling - https://phabricator.wikimedia.org/T219765 (10Maintenance_bot) [15:46:53] 10Traffic, 10Operations: Renew Digicert Unified in 2019 - https://phabricator.wikimedia.org/T209515 (10Maintenance_bot) [15:49:22] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission old eqiad caches - https://phabricator.wikimedia.org/T208584 (10Maintenance_bot) [15:49:40] 10Traffic, 10Operations, 10Performance-Team (Radar): Refactor public-facing DYNA scheme for primary project hostnames in our DNS - https://phabricator.wikimedia.org/T208263 (10Maintenance_bot) [17:16:45] 10Traffic, 10DNS, 10Operations: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10HMarcus) @BBlack , receiving the following error from Google when attempting to verify: {F29207898} Would you mind clarifying the context of the second token? Are you saying the same domain name... [17:31:47] 10Traffic, 10DNS, 10Operations: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10BBlack) The context of the second token is that all of our canonical wiki domains, including `wikimedia.org`, already have persistent Google Site Verification TXT tokens so that we can manage Google... [17:35:37] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10BBlack) The above is deployed. I'd wait a full 10 minutes from the time of this comment to re-test, in case they've negative-cached the previous lookup, then try again and let... [17:42:42] 10Traffic, 10Operations, 10Performance-Team (Radar): Refactor public-facing DYNA scheme for primary project hostnames in our DNS - https://phabricator.wikimedia.org/T208263 (10BBlack) 05Open→03Resolved a:03BBlack Scheme has been stable for ~1w now and seems to be working out fine. The net reduction in... [17:43:12] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10Dzahn) [17:49:23] Now that I started wireshark I looked into other rabbit hole issues, like why the rec-dns servers have a constant 15% tcp retransmits rate? Seems like the rec-dns server doesn't register the ack of the LVS tcp healthcheck on the first try, and (always?) re-send the syn-ack [17:51:18] 10Traffic, 10Operations: Lower geodns TTLs from 600 (10min) to 300 (5min) - https://phabricator.wikimedia.org/T140365 (10BBlack) So we've reduced query volume by ~32% in T208263 . Since the last significant updates here, we've also deployed newer versions of our authdns software which perform even better, and... [18:13:26] trying to come up with another runbook, this time for "check_strongswan". I certainly see the explanation in code comments what it does etc.. but the "what should you actually do"-part.. i just think "You probably want to tell the traffic team about it" so far [18:13:38] but maybe that's fair [18:14:15] * mutante starts with the part where it just explains.. "# Parses output of 'ipsec statusall':".. etc [18:21:54] ipsec is a rabbit hole too [18:22:16] 1/ troubleshot the issue [18:22:16] 2/ write what you did to troubleshot the issue in that runbook [18:22:17] :) [18:24:58] lol, a meta runbook to write one?:) works! [18:38:39] https://wikitech.wikimedia.org/wiki/Monitoring/strongswan :p [18:41:04] haha yeah [19:05:21] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10HMarcus) {F29208502} Thanks so much for your help Brandon, that did it. I will follow up with the Google team to see if any additional DNS records are needed. Would you prefe... [19:30:04] 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: GSuite Test Domain Verification - https://phabricator.wikimedia.org/T223921 (10BBlack) Either is fine. I assume you won't be able to do anything else with this (e.g. make https://gsuite-test.wikimedia.org/ work) without some followup records added on our... [21:45:47] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Matrix, and 2 others: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs - https://phabricator.wikimedia.org/T223835 (10Tgr) Yeah, it shouldn't be merged before the server is up (which is in a few days if all goes well). The Matrix server...