[00:42:21] 10Traffic, 10Operations: /sec-warning page: please add a helpful XML comment explaining why it's being delivered. - https://phabricator.wikimedia.org/T240794 (10Krenair) I'd prefer we talked about the drawbacks of serving the /sec-warning HTML with different HTTP status codes, than doing anything special in th... [03:24:03] Do we have some standard response to the encryption-on-your-site-is-unnecessary-its-all-public type messages? [03:24:25] 10Traffic, 10Operations: Start warning and deprecation process for all legacy TLS - https://phabricator.wikimedia.org/T238038 (10Krenair) Speaking of stats - though on TLS versions rather than ciphers - do we have numbers on how many connections/requests/users were using TLS 1.0/1.1? [03:27:33] 10Traffic, 10Operations: HTTPS/Browser Recommendations page on Wikitech is outdated - https://phabricator.wikimedia.org/T240813 (10Emufarmers) [03:32:40] 10Traffic, 10Operations, 10Documentation: Update TLS/HTTP documentation on wikitech - https://phabricator.wikimedia.org/T96844 (10AntiCompositeNumber) This task has been open since 2015 and the Wikitech HTTPS documentation is still in need of improvement. While the HTTPS pages on Wikitech have been updated s... [03:55:04] 10Traffic, 10Operations: HTTPS/Browser Recommendations page on Wikitech is outdated - https://phabricator.wikimedia.org/T240813 (10Emufarmers) [03:55:07] 10Traffic, 10Operations, 10Documentation: Update TLS/HTTP documentation on wikitech - https://phabricator.wikimedia.org/T96844 (10Emufarmers) [06:18:45] maybe the phab task or the blogpost (if any) from when we switched to HTTPS [06:36:26] 10netops, 10Operations: Routinator RSYNC errors - https://phabricator.wikimedia.org/T240817 (10ayounsi) 05Open→03Resolved p:05Triage→03Normal [06:40:43] 10netops, 10DC-Ops, 10Operations: Juniper network device audit - all sites - https://phabricator.wikimedia.org/T213843 (10ayounsi) Got an email from a new person on charge of the task. Sent them the updated list of what needs to be updated. [07:53:32] ema, vgutierrez, fyi, there are 2 CP related Icinga alerts that poped up during the weekend, one is "PROCS CRITICAL: 0 processes with UID = 115 (vhtcpd), args 'vhtcpd'" the other "WARNING: traffic_server requires restarting" [07:54:30] XioNoX: morning! [07:54:37] good morning! [07:55:24] XioNoX: yes I've depooled cp1075 and am opening a task for the former issue [08:05:41] 10Traffic, 10Operations: vhtcpd segfaulted and did not get restarted - https://phabricator.wikimedia.org/T240826 (10ema) [08:05:49] 10Traffic, 10Operations: vhtcpd segfaulted and did not get restarted - https://phabricator.wikimedia.org/T240826 (10ema) p:05Triage→03High [08:11:25] 10Traffic, 10Operations: vhtcpd segfaulted and did not get restarted - https://phabricator.wikimedia.org/T240826 (10ema) [09:00:13] 10netops, 10Operations: Network issues reaching phabricator on IPv6 (Comcast/Portland OR) - https://phabricator.wikimedia.org/T240488 (10jcrespo) @ayounsi For context, I suggested him on IRC to file a ticket if problems were continuous to have a look (not necessarily to fix them, if there is nothing we can do... [10:00:39] ema: buongiorno :) [10:00:53] I see a lot of "WARNING: traffic_server requires restarting", sorry if it is known or already wip [10:01:41] elukey: hi! I know :) [10:02:39] <3 [10:13:51] 10Traffic, 10Operations: vhtcpd segfaulted and did not get restarted - https://phabricator.wikimedia.org/T240826 (10MoritzMuehlenhoff) The vhtcpd.service unit is auto-generated from the vthpcd sysvinit script by means of the systemd-sysv-generator, which simply printfs "Restart=No" to the systemd units it gene... [11:27:35] 10netops, 10DC-Ops, 10Operations: Juniper network device audit - all sites - https://phabricator.wikimedia.org/T213843 (10faidon) Thanks @ayounsi! Appreciate the follow up. What exactly did you ask them to do in this last communciation? This has been going on for too long, so I think we need a change of str... [11:29:22] 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS: cleanup network allocations - https://phabricator.wikimedia.org/T240670 (10aborrero) Needs discussion in the next WMCS team meeting: I would like to double check with you all that doing this cleanup is right, and make sure yo... [12:49:45] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10Patch-For-Review: LDF service does not Vary responses by Accept, sending incorrect cached responses to clients - https://phabricator.wikimedia.org/T232006 (10Gehel) Note that we already set a `Vary: Accept` HTTP header at nginx level (https... [13:54:13] 10netops, 10DC-Ops, 10Operations: Juniper network device audit - all sites - https://phabricator.wikimedia.org/T213843 (10ayounsi) Removing the serials: ` Hi Jim, No problem to re-state everything as clear as I can if it's the last one :) Serials that should have their install base set to Equinix Ashburn (... [14:53:02] 10Traffic, 10Operations: Secure shared ticket key rotation for anycast authdns - https://phabricator.wikimedia.org/T240863 (10BBlack) [15:00:46] 10Traffic, 10Operations: Create a system for distributed shared secret material to server tmps - https://phabricator.wikimedia.org/T240866 (10BBlack) [15:51:59] 10Traffic, 10DNS, 10Operations, 10Research: Add wikiworkshop.org to the Foundation's DNS - https://phabricator.wikimedia.org/T240303 (10leila) @jcrespo makes sense. I'll schedule a meeting for after the holidays. :) (btw, I'm not sure when I can resolve this task. Please go ahead and do it if it's consider... [17:09:54] 10Traffic, 10Operations: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) Refactoring the dependencies a little here: Really (2) above's sub-point about shared ticket key rotation won't matter until we're anycasting, so I've made a separate task (+subtask) in T240863 to... [23:34:12] 10Traffic, 10Operations: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) Actually we can't realistically do global monitoring from icinga either, because icinga isn't on Buster and so it doesn't have the right library/tool access to check a TLSv1.3-only service, so we'... [23:37:58] 10Traffic, 10Operations: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) External queries now working (note they all return a codfw IP without edns-client-subnet in play, because codfw is closest to my laptop and PROXYv2 is working for sending the "real" client IP fro... [23:38:49] 10Traffic, 10Operations: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) 05Open→03Resolved a:03BBlack