[07:39:03] 10netops, 10Operations: Zayo link eqiad-codfw down - TTN-0004110251 - https://phabricator.wikimedia.org/T253610 (10ayounsi) p:05Triage→03Medium [11:20:20] 10netops, 10Operations: Zayo link eqiad-codfw (OGYX/120003//ZYO) down - TTN-0004110251 - https://phabricator.wikimedia.org/T253610 (10CDanis) [12:12:23] vgutierrez: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/598717/ is pretty impressive. Nice that ATS exposes that :-) [12:15:09] yeah, that's an ongoing upstream effort... mainly https://github.com/apache/trafficserver/pull/6571 [12:15:19] (and some PRs fixing issues) [14:11:58] 10netops, 10Operations: scrape ripe atlas data for a few anchors at other large networks - https://phabricator.wikimedia.org/T252890 (10CDanis) >>! In T252890#6154888, @ayounsi wrote: > Good idea! What's the limit? While I'm not very concerned about cardinality limits in Prometheus here, I think we'd start to... [15:25:22] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Certificate *.wikipedia.org valid until 2020-06-20 - https://phabricator.wikimedia.org/T251726 (10Dzahn) >>! In T251726#6109023, @Vgutierrez wrote: > IMHO 7 / 3 is not enough for the unified cert even when LE is the issuer considering our anti cloc... [16:48:31] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards. - https://phabricator.wikimedia.org/T253655 (10Krinkle) [16:54:48] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards. - https://phabricator.wikimedia.org/T253655 (10CDanis) Thanks for filing this @Krinkle. Ideally Grafana would al... [16:55:07] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards - https://phabricator.wikimedia.org/T253655 (10Krinkle) [17:11:41] 10netops, 10Operations: intermittent brief data dropouts for esams netflow data - https://phabricator.wikimedia.org/T253128 (10CDanis) There's lots of this in kern.log on netflow3001: `May 26 15:15:03 netflow3001 kernel: [1214948.953863] nfacctd[422]: segfault at 0 ip 00007f54d61ebb17 sp 00007ffdcefb36e8 erro... [17:23:52] 10Traffic, 10netops, 10Operations: Anycast: consistent routers->servers routing - https://phabricator.wikimedia.org/T253666 (10ayounsi) p:05Triage→03Medium [17:24:25] 10Traffic, 10Analytics, 10Operations, 10Privacy Engineering: Publishing project anomaly data for censorship researchers. Evaluate privacy threats - https://phabricator.wikimedia.org/T183990 (10JFishback_WMF) [17:26:20] 10Traffic, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10Andrew) [18:33:00] 10Traffic, 10netops, 10Operations: Anycast: consistent routers->servers routing - https://phabricator.wikimedia.org/T253666 (10BBlack) Even if we could experimentally verify option A, we probably can't trust it across future firmware differences (between sites, or between two routers in a site). Option B vi... [20:30:51] 10Traffic, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), 10Developer Productivity: Whitelist x-wikimedia-debug header field (currently not allowed by Access-Control-Allow-Headers in preflight response) - https://phabricator.wikimedia.org/T252826 (10eprodromou) p:05Triage→03Medium