[05:35:24] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2274270 (10bearND) [09:05:33] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 3 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#2275051 (10Lydia_Pintscher) [13:28:33] 10Traffic, 10Varnish, 06Operations, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2275960 (10BBlack) [13:46:29] 10Traffic, 10Varnish, 06Operations, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2275985 (10BBlack) So, as it turns out, this is a general varnishd bug in our specific varnishd build. For purposes of this bug, our varnishd code is essen... [13:46:54] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Convert misc cluster to Varnish 4 - https://phabricator.wikimedia.org/T131501#2275988 (10BBlack) [13:46:59] 10Traffic, 06Discovery, 06Operations, 10Wikidata, 10Wikidata-Query-Service: Wikidata Query Service REST endpoint returns truncated results - https://phabricator.wikimedia.org/T133490#2233245 (10BBlack) [13:47:48] 10Traffic, 06Discovery, 06Operations, 10Wikidata, 10Wikidata-Query-Service: Wikidata Query Service REST endpoint returns truncated results - https://phabricator.wikimedia.org/T133490#2233245 (10BBlack) We now have some understanding of the mechanism of this bug ( T133866#2275985 ). It should go away in... [13:54:19] 10Traffic, 10Varnish, 06Operations, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2276000 (10Anomie) [13:55:10] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2276001 (10faidon) For what it's worth, when trying to switch our mirror on Launchpad to https: > The URI scheme "https" is not allowed. Only URIs with the f... [14:14:07] 10Traffic, 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Upgrade analytics-eqiad Kafka cluster to Kafka 0.9 (or 0.10?) - https://phabricator.wikimedia.org/T121562#2276061 (10Ottomata) [14:32:59] 10Traffic, 06Discovery, 06Operations, 10Wikidata, and 2 others: Wikidata Query Service REST endpoint returns truncated results - https://phabricator.wikimedia.org/T133490#2276138 (10BBlack) 05Open>03Resolved a:03BBlack This works now. There's a significant pause at the start of the transfer from the... [14:33:43] 10Traffic, 10Varnish, 06Operations, 10Wikidata, 13Patch-For-Review: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2276144 (10BBlack) 05Open>03Resolved My test cases on cache_text work now, should be resolved! [14:45:28] 10Traffic, 10Varnish, 06Operations, 10Wikidata, 13Patch-For-Review: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2276263 (10MZMcBride) This looks fixed to me. Thank you, @BBlack! [15:36:15] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276577 (10BBlack) Patch updated to include matching `redirect=0` (and some comments made clearer, and the deliver-time code was re-arranged to be slightly more... [15:57:06] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276670 (10mobrovac) >>! In T134464#2270692, @GWicke wrote: > 301s (returned for title normalization) are always safe to cache (and we do send the corresponding... [16:06:21] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276728 (10GWicke) @mobrovac, looking back at your earlier comment I now realize that I misunderstood your concern. The example fix you gave there actually point... [16:17:07] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276765 (10BBlack) Well, we chose to strip it (or not) before sending the request to RB, before we've seen the response. So we can't logically make the call to... [16:19:45] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276773 (10mobrovac) >>! In T134464#2276765, @BBlack wrote: > Well, we chose to strip it (or not) before sending the request to RB, before we've seen the respons... [16:21:34] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276780 (10BBlack) So, just to be sure I understand: the problem here is that when both title **and** wikitext redirects apply to a single request, with ?redirec... [16:26:55] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276812 (10BBlack) Sorry, that comment was written before I read @mobrovac's. So the issue is just ensuring ?redirect=false is preserved in the 301's Location? [16:28:36] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276816 (10mobrovac) >>! In T134464#2276780, @BBlack wrote: > So, just to be sure I understand: the problem here is that when both title **and** wikitext redirec... [16:28:53] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2276817 (10mobrovac) >>! In T134464#2276812, @BBlack wrote: > Sorry, that comment was written before I read @mobrovac's. So the issue is just ensuring ?redirect... [16:46:33] 10Traffic, 10Varnish, 06Operations, 10Wikidata, 13Patch-For-Review: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2276929 (10Trung.anh.dinh) Thank you for the fix @BBlack :) [17:09:58] 10Traffic, 10Varnish, 06Operations, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2277093 (10TerraCodes) [18:50:13] 10Traffic, 10Varnish, 06Operations, 06Performance-Team: Understand and improve streaming behaviour from Varnish - https://phabricator.wikimedia.org/T126015#2277474 (10BBlack) 05Open>03Resolved a:03BBlack Closing this for now: we tried enabling streaming in the text cluster for pass-traffic, and all i... [18:51:36] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: commons.wikimedia.org home page has 404s loaded from JS (RL?) - https://phabricator.wikimedia.org/T134368#2277480 (10BBlack) Ping. Still seeing these 404s on https://commons.wikimedia.org/ . ... [18:53:48] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277493 (10BBlack) I saw the deprecation email about RESTBase itself. Are we clear to pull the DNS for r... [18:55:18] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2277496 (10BBlack) Ping - Reading, Mobile, someone should know what's causing this, right? [18:58:04] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2216638 (10GWicke) @bblack, if there is no specific reason to pull the DNS soon, I would propose to keep... [18:59:35] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277517 (10BBlack) Ok [19:00:41] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277537 (10BBlack) And actually, on the issue of the `.eqiad.` hostnames: they've been broken for a while... [19:03:19] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2277546 (10Jdlrobson) 05Open>03Invalid This is due to MobileContext::isBlacklistedPage and this: https://github.com/wikimedia/operations-me... [19:03:37] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277548 (10BBlack) >>! In T133001#2277513, @GWicke wrote: > @bblack, if there is no specific reason to pu... [19:10:15] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277566 (10GWicke) @bblack, restbase.wikimedia.org was never published or in any way supposed to work. Al... [19:13:13] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 3 others: Support following MediaWiki redirects when retrieving HTML revisions - https://phabricator.wikimedia.org/T118548#2277584 (10GWicke) [19:13:19] 10Traffic, 06Operations, 10Parsoid, 10RESTBase, and 4 others: Make RB ?redirect=false cache-efficient - https://phabricator.wikimedia.org/T134464#2277581 (10GWicke) 05Open>03Resolved a:03GWicke Thank you, @bblack! [19:20:15] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: commons.wikimedia.org home page has 404s loaded from JS (RL?) - https://phabricator.wikimedia.org/T134368#2277600 (10Krinkle) I can't reproduce this. Wikimedia app servers haven't produced `/static/1.27.0-$version/` urls since February 26 ([config diff](ht... [19:20:52] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277601 (10BBlack) Ok got it, thanks! [19:22:29] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277605 (10Krinkle) [19:26:00] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277614 (10Krinkle) While we waited the usual 30 days before removing the symlinks for old branches, they were still cached client side (which has no e... [19:32:53] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277677 (10BBlack) Ok. I can see that they're gone in a fresh browser install or incognito view. But I've definitely closed the whole browser app sin... [19:42:53] 10Traffic, 10DNS, 06Operations: Replace test hostnames in datecenter-specific subdomains with dashed names - https://phabricator.wikimedia.org/T134807#2277734 (10BBlack) [19:43:04] 10Traffic, 10DNS, 06Operations: Replace test hostnames in datecenter-specific subdomains with dashed names - https://phabricator.wikimedia.org/T134807#2277747 (10BBlack) p:05Triage>03Low [20:16:06] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277829 (10Krinkle) >>! In T134368#2277677, @BBlack wrote: > Ok. I can see that they're gone in a fresh browser install or incognito view. But I've d... [20:16:19] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2216638 (10Krinkle) > restbase 600 IN DYNA geoip!text-addrs Despite it not being documented,... [20:18:37] 10Traffic, 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Upgrade analytics-eqiad Kafka cluster to Kafka 0.9 - https://phabricator.wikimedia.org/T121562#2277851 (10Ottomata) [20:19:22] 10Traffic, 10Citoid, 10ContentTranslation-cxserver, 06Operations, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2277852 (10BBlack) If something important breaks, we can put that name back in. But in that case, we sho... [20:19:47] 10Traffic, 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Upgrade analytics-eqiad Kafka cluster to Kafka 0.9 - https://phabricator.wikimedia.org/T121562#1881753 (10Ottomata) Upgraded the analytics Kafka cluster in deployment-prep today. Along the way I had to create an extra... [20:21:10] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277855 (10BBlack) So, how do we get rid of those stored broken URIs? It seems like a lot of clients will be stuck with them for a very long time... [20:42:59] 10netops, 06Operations, 13Patch-For-Review: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330#1721704 (10hashar) As part of T134808 on CI and beta cluster I have ran: ``` salt -v '*' cmd.run 'dpkg --purge libganglia1 ganglia-monitor; rm -fR /etc/ganglia' ``` Some... [20:49:11] 10Traffic, 10MediaWiki-ResourceLoader, 06Operations: Image urls in CSS remain cached with old $wgResourceBasePath - https://phabricator.wikimedia.org/T134368#2277976 (10Krinkle) The proper solution is to roll out pure module-content based versioning to file modules (which covers jquery-ui). More specifically... [23:05:49] 10Traffic, 10domains, 10DNS, 06Operations, and 2 others: point wikilovesmonument.org ns to wmf - https://phabricator.wikimedia.org/T118468#1801065 (10Platonides) wikilovesmonument**s**.org is a bit special for being the one for the international team, but note that for many domains (eg. wikilovesmonument.C... [23:06:21] 10Traffic, 10domains, 10DNS, 06Operations, and 2 others: point wikilovesmonuments.org ns to wmf - https://phabricator.wikimedia.org/T118468#2278838 (10Platonides) [23:12:26] 10Traffic, 10DNS, 10Mail, 06Operations, and 2 others: phabricator.wikimedia.org has no SPF record - https://phabricator.wikimedia.org/T116806#1758902 (10Platonides) @scfc in that case, your server should not reject based on SPF for accounts that you forwared there. There are many ways to treat SPF, the mos...