[07:53:55] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [08:52:16] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [09:37:08] 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [12:57:27] 10Traffic, 10Operations, 10RESTBase, 10Platform Team Workboards (Clinic Duty Team): RESTBase CORS redirect resolve should not hit frontend caches - https://phabricator.wikimedia.org/T259054 (10ema) [13:13:18] 10Traffic, 10Operations, 10RESTBase, 10Platform Team Workboards (Clinic Duty Team): RESTBase CORS redirect resolve should not hit frontend caches - https://phabricator.wikimedia.org/T259054 (10ema) For the record I don't think this currently causes any specific functional issue. I've spotted a few RESTBase... [14:21:33] 10Traffic, 10Operations: varnishmtail silently stops working if varnishncsa crashes - https://phabricator.wikimedia.org/T259020 (10ema) I've opened https://github.com/google/mtail/issues/331 to get an opinion from upstream. [14:48:19] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10Andrew) Is there any reason not to close this? Are there still asset tags or netbox things left to do? [15:37:19] so it looks like we fixed the http2 active client connections leak on ats-tls: https://grafana.wikimedia.org/d/6uhkG6OZk/ats-instance-drilldown?panelId=56&fullscreen&orgId=1&from=now-6h&to=now&var-site=esams%20prometheus%2Fops&var-instance=cp3064&var-layer=tls [15:37:24] \o/ [15:40:43] \o/ [15:41:23] vgutierrez: \m/ [15:44:08] we're just running 56 patches on top of 8.0.8 [15:44:14] /o\ [15:44:37] and I was scared about nginx and varnish when I've joined WMF for the same reason [15:44:39] lol [15:49:54] vgutierrez: nice! [15:56:48] 10Traffic, 10Operations, 10observability, 10serviceops, 10Patch-For-Review: monitoring for mismatched LVS realserver addresses/configurations - https://phabricator.wikimedia.org/T258648 (10CDanis) 05Open→03Declined The structure of the data makes this teeth-pullingly impossibly difficult to do well,... [15:56:51] 10Traffic, 10Operations, 10observability, 10serviceops: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10CDanis) [16:06:32] 10Acme-chief: acme-chief: support for generating a concatenated cert/key file - https://phabricator.wikimedia.org/T255249 (10bd808) It turns out that we really need a .crt.chained.key version of this that includes the LE signed cert, the local key, and the intermediate signing chain. ` $ curl -v https://public.p... [16:06:55] vgutierrez: yay! [16:14:42] 10Traffic, 10Operations, 10observability, 10serviceops, 10Patch-For-Review: monitoring for mismatched LVS realserver addresses/configurations - https://phabricator.wikimedia.org/T258648 (10Joe) Truth is that every host that has multiple pools that use the same backend IP can happily live with only one po... [16:59:58] 10Acme-chief, 10Patch-For-Review: acme-chief: support for generating a concatenated cert/key file - https://phabricator.wikimedia.org/T255249 (10bd808) 05Resolved→03Open [17:03:14] bd808: thanks for submitting the CRs, I'll review them tomorrow EU morning [17:03:43] vgutierrez: ack! I was mostly copy-n-pasting from you prior patch :) [17:18:08] 10Traffic, 10Operations, 10Phabricator, 10serviceops, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) So I think that this is all that remains: * [] cache layer proxy wss://phabricator.wikimedia.org to aphl...