[06:59:48] <wikibugs>	 10netops, 10Operations: ulsfo - codfw Zayo link down - https://phabricator.wikimedia.org/T255393 (10ayounsi) p:05Triage→03Medium
[07:38:07] <wikibugs>	 10netops, 10Operations, 10fundraising-tech-ops, 10WMF-NDA: Deploy pfw policy 1591901800 for T122104 - https://phabricator.wikimedia.org/T255185 (10ayounsi) 05Open→03Resolved a:03ayounsi Done!
[08:29:52] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10Vgutierrez) Checking against eqsin with `curl --resolve noc.wikimedia.org:443:$(dig +short text-lb.eqsin.wikimedia.org) https://noc.wikimedia.org` I do get a 503 and...
[08:36:35] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10Vgutierrez) `curl --http1.1 -H 'Host: noc.wikimedia.org' https://mwmaint.discovery.wmnet` from cp5010 returns a HTTP 200 as expected
[08:58:31] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10Vgutierrez) varnish-fe also shows a 503: `vgutierrez@cp5010:~$ sudo -i varnishlog -n frontend -q "ReqHeader:Host eq noc.wikimedia.org" *   << Request  >> 1066411391 -...
[09:02:57] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10Vgutierrez) Filtering by BeReqHeader we can see how varnish-fe apparently gets a 200 from ats-be and returns a 503 cause the "body cannot be fetched":  `vgutierrez@cp...
[11:28:18] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10ema) It seems that something might be going wrong at the ats-tls<->varnish-fe level. Hitting varnish-fe directly on cp5007 I constantly get a 200 response with body a...
[12:03:49] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10ema) >>! In T255368#6223544, @ema wrote: > It seems that something might be going wrong at the ats-tls<->varnish-fe level. Hitting varnish-fe directly on cp5007 I con...
[12:57:32] <wikibugs>	 10Acme-chief, 10Patch-For-Review: acme-chief: support for generating a concatenated cert/key file - https://phabricator.wikimedia.org/T255249 (10Vgutierrez) 05Open→03Resolved This seems to be working (from my tests on acmechief-test1001): `root@acmechief-test1001:/var/lib/acme-chief/certs/mirrors/new# grep...
[12:58:22] <vgutierrez>	 !log upgrade acme-chief to version 0.26
[12:58:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:24:34] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10ema) Apparently `TE:chunked` is not added only on cache hits, but occasionally on miss/pass too. https://gerrit.wikimedia.org/r/605578 does make sense and it's good w...
[16:12:39] <wikibugs>	 10Traffic, 10Operations: ats-backend throttles connections under heavy load - https://phabricator.wikimedia.org/T254714 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez
[21:05:47] <wikibugs>	 10Traffic, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10bd808) >>! In T161256#6221208, @TheDJ wrote: > For future reference.. I think these types of subdomains still require some m...
[21:27:17] <wikibugs>	 10Traffic, 10Operations, 10Services (watching), 10Sustainability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820 (10Krinkle)
[21:27:30] <wikibugs>	 10Traffic, 10Operations, 10Services (watching), 10Sustainability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820 (10Krinkle)
[21:56:23] <wikibugs>	 10netops, 10Operations, 10fundraising-tech-ops, 10WMF-NDA: Deploy pfw policy 1591901800 for T122104 - https://phabricator.wikimedia.org/T255185 (10Dwisehaupt) Thanks. Runsgood.
[22:00:55] <wikibugs>	 10Traffic, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10bd808) 05Open→03Resolved a:03Andrew I'm going to call this {{done}}. Maps is covered and we can do something similar i...