[05:09:59] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Development services), and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10mmodell) [06:34:10] 10Traffic, 10Operations, 10Wikidata, 10serviceops, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10Dzahn) >>! In T99531#5414154, @BBlack wrote: > track down various revert patches first before we close it up (revert the DNS repo stuff and w... [07:40:08] 10Traffic, 10Operations, 10docker-pkg, 10serviceops: Getting registry metadata from a public client fails on our registry - https://phabricator.wikimedia.org/T220085 (10ema) It seems that CL is returned properly now: ` $ curl -v --http1.1 https://docker-registry.wikimedia.org/v2/python3/manifests/latest 2... [08:12:49] ema: regarding piwik caching headers, I still see the "pass" header despite https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534034/ being merged, is there anything else we need to do? [08:13:29] nuria: good morning! [08:13:49] it's strange, the VCL changes should have triggered auto-reloads [08:14:13] let me look into that [08:19:56] nuria: some time ago we've folded cache_misc into cache_text by introducing a concept called "separate VCL". piwik and others fall in the separate VCL category, and we do have a bug with it, as it turns out! [08:20:22] when separate vcl changes, we don't trigger a vcl reload [08:28:10] nuria: in about 30 minutes the piwik problem should be fixed [09:14:01] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10ema) >>! In T230772#5461321, @Nuria wrote: > I can see the cache-control: max-age=604800 , I think @ema needs to change something on his end so varnish /ATS... [10:21:48] 10Traffic, 10Operations, 10Patch-For-Review: ATS-tls isn't enforcing the same list of curves as nginx during TLS handshake - https://phabricator.wikimedia.org/T231859 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez Solved, now ATS has the same behaviour as nginx: ` vgutierrez@cp5001:~$ openssl s_client... [11:23:03] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10MoritzMuehlenhoff) >>! In T231811#5462395, @ayounsi wrote: > Hosts in the `cloud-hosts1-b-eqiad` vlan are behind the `labs-in4` firewall filter (applied on traff... [11:28:15] nuria: please paste the full x-cache header [11:28:46] nuria: err, sorry! I got confused by the backscroll :) [12:22:43] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) I think things look good now: [nuriaruiz@nurieta][~/tips]$ curl -v https://piwik.wikimedia.org/piwik.js > piwik * Trying 91.198.174.192... % Tota... [13:17:33] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Jclark-ctr) @Cmjohnson removed 2nd dac cable yes it plugged into d7 xe-7/0/2 [13:44:52] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) While the caching header is correctly served, when the request is in the context of the foundation website, Varnish is doing a pass: {F30222372, siz... [13:50:38] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) Inddeed! I see server-timing: cache;desc="pass" on reg chrome window as well, thanks @Gilles for catching that [13:53:36] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10ema) >>! In T230772#5464917, @Gilles wrote: > I'm guessing it might be coming from the cookies? Which the Chrome developer tools weren't showing. We've had t... [13:55:39] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) I think the issue is that misc_recv_pass is applied to every site. You want it to apply to wikis (where people can log in), but not on non-wiki websi... [13:56:38] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) Or I guess, to be conservative, add Wikimedia wiki login cookies to the filter only if you're in the context of a non-wiki site. [13:57:11] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) This should probably be its own task, though, it's not specific to piwik.js [14:04:30] 10Traffic, 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) Right this is an issue for all resources served for https://stats.wikimedia.org/v2 and, I am gusessing, other non wiki domains. Can we consider blac... [14:24:56] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10Andrew) Moving the iron exception to cloudnet1003 works for me -- presuming we mean adding it to 'wmcs::openstack::eqiad1::net'. One thing I'm not clear on is h... [14:28:49] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10MoritzMuehlenhoff) >>! In T231811#5465056, @Andrew wrote: > Moving the iron exception to cloudnet1003 works for me -- presuming we mean adding it to 'wmcs::opens... [14:41:18] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10Andrew) >>! In T231811#5465070, @MoritzMuehlenhoff wrote: > > On the system/ferm level there's fleet-wide Ferm rule which grants SSH access from Cumin masters.... [14:43:40] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10MoritzMuehlenhoff) >>! In T231811#5465102, @Andrew wrote: >>>! In T231811#5465070, @MoritzMuehlenhoff wrote: >> >> On the system/ferm level there's fleet-wide F... [15:34:27] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10Andrew) install_console works just fine from cumin1001. So, no need for a special case here, we can just go ahead and decom iron. Thanks all! [15:35:41] 10netops, 10Operations: Check router ACLs for early install SSH access from puppet masters/cumin hosts - https://phabricator.wikimedia.org/T231811 (10MoritzMuehlenhoff) 05Open→03Declined Great, thanks. Closing the task, will proceed with iron decom.