[01:29:36] 10Traffic, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10Krenair) [01:30:02] 10Traffic, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10Krenair) [09:26:46] 10Traffic, 10Operations, 10Patch-For-Review: ATS: Add the ability to check if origin server responses can be cached and their lifetime to the Lua plugin - https://phabricator.wikimedia.org/T251537 (10ema) >>! In T251537#6132201, @ema wrote: > https://github.com/apache/trafficserver/pull/6767 That's been mer... [11:23:21] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, and 8 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214 (10Aklapper) [11:59:45] 10Acme-chief, 10cloud-services-team (Kanban): tools/toolsbeta: improve acme-chief integration - https://phabricator.wikimedia.org/T252762 (10aborrero) [12:09:13] 10Traffic, 10Cloud-Services, 10Operations, 10cloud-services-team, and 5 others: Deprecate `base::service_unit` in puppet - https://phabricator.wikimedia.org/T194724 (10MoritzMuehlenhoff) [14:44:43] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Remove North Korea from data quality traffic entropy reports - https://phabricator.wikimedia.org/T251546 (10Nuria) 05Open→03Resolved [15:09:13] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) 05Open→03Stalled a:05mmodell→03None I am currently unable to drive this forward as all the change... [16:33:48] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) p:05Triage→03High [16:48:33] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) I disabled the mentioned link on the fpc2 side (so we don't risk fully losing access to fpc1) first. Then on the fpc1 side to check if the alert was caused by this DAC. Unfortunately it loo... [17:00:28] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) `pic-slot 1 port 3 member 1` was a leftover port configured as VC port, but without any cable connected to it. Errors are still happening. [17:14:34] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) Disabled the last link, and the errors are still showing up, so I'm confused on where the issue is coming from. [17:20:37] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) From T218059#5075466 it probably due to the link disabled in T251663 acting up. @Jclark-ctr, please unplug fpc1:1/0 (and remove/store the optics) from both sides, fpc8:1/0 (link should be d... [17:54:38] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10ayounsi) Unplugging that link caused fpc1 to lose connectivity to the remaining of the VC, while it's neither a VCP, nor enabled. > asw2-d-eqiad fpc1 PFEMAN: Shutting down in 5 seconds, PFEMAN Resync... [19:03:29] things I did not expect: we have ASNs that are in eqiad's routing table but not in eqsin's routing table, and vice versa [19:27:08] cdanis: do you have an example? I'm curious [19:27:15] I do! [19:28:18] so I extracted routing tables, foolishly, by running `show route table inet.0 terse` and then grabbing all the numbers from the ASN column on the right, removing non-digit characters (some were wrapped in () or [] or {}) [19:30:08] https://phabricator.wikimedia.org/P11197 [19:30:46] the first column is asns unique to eqiad, the second unique to eqsin [19:31:17] (so for instance 10437 appears in eqiad's table, but not in eqsin's) [19:33:30] interesting, looks like it's not an end network, but a transit [19:33:59] so I didn't just grab the 'terminal' ASNs [19:34:03] so maybe that's my mistake [19:34:11] not sure of the usual nomenclature here [19:36:44] it usually change depending on context [19:36:47] at least for me [19:37:15] here transit/origin seems appropriate [19:37:43] as it's an AS from where a prefix originates [19:38:20] so 10437 can be a small regional transit provider [19:42:26] right [19:43:15] so here's just origins: https://phabricator.wikimedia.org/P11200 [21:11:38] 10Acme-chief, 10cloud-services-team (Kanban): tools/toolsbeta: improve acme-chief integration - https://phabricator.wikimedia.org/T252762 (10Krenair) do we really want to go down the path of setting acme-chief up in toolsbeta before doing the thing we agreed? I feel like this is basically motivated by {T252199... [23:32:16] 10Traffic, 10Operations, 10Performance-Team (Radar): Edge cache response time per server should be monitored - https://phabricator.wikimedia.org/T238086 (10dpifke) The dashboard was broken such that it would not load even load the settings page. It seemed to hang indefinitely; I left it open in the backgrou... [23:37:42] 10Traffic, 10Operations, 10Performance-Team (Radar): Edge cache response time per server should be monitored - https://phabricator.wikimedia.org/T238086 (10dpifke) Regarding the label pollution, I added a regex to $dc which excludes values containing numerals (and thus hostnames). This fixes the drop-downs... [23:55:33] 10Traffic, 10Operations, 10Performance-Team (Radar), 10Sustainability (Incident Prevention): Edge cache response time per server should be monitored - https://phabricator.wikimedia.org/T238086 (10Krinkle)