[08:00:45] 10Traffic, 10Operations: TLS config issue for nginx on Buster - https://phabricator.wikimedia.org/T228730 (10elukey) [08:01:19] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Operations, and 2 others: TLS certificates for Analytics origin servers - https://phabricator.wikimedia.org/T227860 (10elukey) 05Open→03Stalled Stalled due to https://phabricator.wikimedia.org/T228730 [08:29:48] hi, when you get a chance could you let me know what you think of https://gerrit.wikimedia.org/r/c/operations/puppet/+/523891 ? thanks! [08:30:01] relatedly also https://gerrit.wikimedia.org/r/c/operations/puppet/+/523892 but less important/urgent [12:45:45] 10Traffic, 10Operations: TLS config issue for nginx on Buster - https://phabricator.wikimedia.org/T228730 (10BBlack) If we need this to work ASAP, probably the most-expedient thing to do would be to patch our puppetization to exclude the patched features from config on buster only, and use the vendor package.... [13:07:51] 10Traffic, 10Operations: TLS config issue for nginx on Buster - https://phabricator.wikimedia.org/T228730 (10elukey) @BBlack thanks for the info! Not in a real rush, I was working on https://phabricator.wikimedia.org/T227860 to add TLS capabilities to the Analytics UIs, the only delay will be for Traffic :) [13:08:51] 10Traffic, 10Operations: TLS config issue for nginx on Buster - https://phabricator.wikimedia.org/T228730 (10ema) p:05Triage→03Normal [13:37:14] mhh looks like there was a single client and "many" 500s for that https://logstash.wikimedia.org/goto/09193826887ee93034b0aa108d8d9121 [14:01:38] godog: interesting, for all of those ttfb is > 20s [14:01:40] 10netops, 10Operations: AS63541's session down reported by cr1-eqsin - https://phabricator.wikimedia.org/T228617 (10herron) p:05Triage→03Normal [14:09:32] ema: I'll check swift logs too [14:15:33] ema: yeah looks like swift served a bunch of 200s from that client and 499 which is when the client disconnects prematurely [16:54:35] when you get a chance please review https://gerrit.wikimedia.org/r/c/operations/puppet/+/523891 as it is currently blocking me :( [17:16:24] godog: +1'd, seems sane! [17:16:47] (I didn't check compiler for anything dumb though, but it seems ok to remove these alerts) [17:17:01] awesome, thanks bblack ! will run pcc and merge tomorrow [17:17:23] yeah seems ok to remove to me too, there's a bunch of tuning to the availability ones too coming up [17:17:45] gotta run, ttyl! [17:18:18] cya [19:31:24] bblack: Hey, starting tomorrow, every page view will call this Special page that can be cached in VCL and client, It would be great if you take a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/525142 [19:31:48] I didn't know varnish strips out the cache-control headers until now [19:31:51] sorry [19:32:12] (If we don't get it by tomorrow, it might explode) [19:32:52] that's rather short notice! [19:34:01] yes, I'm extremely sorry, if you don't have time to take a look, I might revert the patch and backport it [19:34:28] the problem was that it localhost it was sending the cache-control headers so we thought it's good to go [19:35:16] https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/LinkedData/EntityDataRequestHandler.php#L562 [19:36:07] Things like "https://wikidata.org/wiki/Special:EntityData/Q467998.json?revision=1138463" should be cached [19:37:22] we're in the middle of an incident right now [19:37:27] but I'll look back at this later [19:38:23] Thanks [19:47:19] I'm sorry again for this [21:00:43] Amir1: I'm almost more-concerned in general that every pageview will fetch wikidata directly now heh [21:02:07] ... and I'm not sure we should be letting the client cache that (why? vs current policy in general) [21:04:01] I'm using https://www.wikidata.org/wiki/Special:EntityData/Q58050939.json?revision=806140466 as a test URL [21:04:11] it does seem to be Varnish-cached, but yes, not client-cached [21:05:16] have you talked to analytics about this, etc? I don't think it's normal for us to make actual content (as opposed to javascript or metadata or something) client-cacheable, at least not historically. [21:05:23] it seems surprising to me anyways [21:06:09] in any case, it's varnish-cacheable, so I doubt anything will explode [21:10:32] bblack: This is about every page view of the wikidata, so nothing will change if someone checks English Wikipedia [21:10:56] it will just add another call to every page view of wikidata [21:11:34] regarding not being able to cache, it's good that it's Varnish-cached so it doesn't explode drastically [21:12:29] the data is basically similar to getting wikitext of a page at any certain revision, since they are immutable (you will get a new revision) they should be completely cache-able. [21:13:03] If it's not exploding because it's varnish, I can make a ticket and discuss that later