[06:30:10] 10Traffic, 10DNS, 10Operations, 10User-DannyS712: DNS_PROBE_FINISHED_NXDOMAIN for mobile version of internal.wikimedia.org - https://phabricator.wikimedia.org/T264565 (10DannyS712) [06:33:16] 10Traffic, 10DNS, 10Operations, 10User-DannyS712: DNS_PROBE_FINISHED_NXDOMAIN for mobile version of internal.wikimedia.org - https://phabricator.wikimedia.org/T264565 (10DannyS712) The same occurs for the mobile view links at https://collab.wikimedia.org/wiki/Main_Page and https://board.wikimedia.org/wiki/... [07:56:39] 10Traffic, 10Operations, 10Performance-Team: Elevated latency starting 2020-09-28 - https://phabricator.wikimedia.org/T264398 (10Gilles) @BBlack we don't see elevated latency that lasts for days like that on train rollouts and rollbacks. Train rollbacks are a frequent event. We're now at 6 days of consistent... [08:09:26] 10Traffic, 10Operations, 10Performance-Team: Elevated latency starting 2020-09-28 - https://phabricator.wikimedia.org/T264398 (10Gilles) As for per-host on esams, it is very clear on every host. Even clearer if you zoom our and switch to a 1day rolling average: {F32373885} {F32373886} {F32373887} {F32373... [09:31:42] 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10ema) p:05Triage→03Medium [09:31:51] 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10ema) I've broadened the search to the past 2 months, and there are a total of 10 matching log entries, all of which are from #parsoid. All are related to ed... [10:53:01] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Elevated latency starting 2020-09-28 - https://phabricator.wikimedia.org/T264398 (10ema) Varnish downgraded on cp3052. I've made a new dashboard comparing response time on cp3052 (v5) vs cp3054 (v6): https://grafana.wikimedia.org/d/EiAVq3FGz... [11:54:37] 10Traffic, 10Operations, 10Patch-For-Review: Wikidough: Upgrade to dnsdist 1.5.0 - https://phabricator.wikimedia.org/T263789 (10ssingh) [12:00:23] 10Traffic, 10DNS, 10Operations, 10User-DannyS712: DNS_PROBE_FINISHED_NXDOMAIN for mobile version of internal.wikimedia.org - https://phabricator.wikimedia.org/T264565 (10Peachey88) [12:00:37] 10Traffic, 10DNS, 10Operations, 10Mobile, 10Patch-For-Review: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882 (10Peachey88) [12:00:56] 10Traffic, 10DNS, 10Operations: DNS_PROBE_FINISHED_NXDOMAIN for mobile version of internal.wikimedia.org - https://phabricator.wikimedia.org/T264565 (10DannyS712) [13:40:34] 10Wikimedia-Apache-configuration: Wikimedia's servers don't correctly rewrite short URLs when the URL ends in a semicolon - https://phabricator.wikimedia.org/T264614 (10Reedy) This might be a dupe... It feels dejavu >Readers cannot enter ";" in the search box to find the article on the semicolon That's potenti... [14:16:48] 10Wikimedia-Apache-configuration: Wikimedia's servers don't correctly rewrite short URLs when the URL ends in a semicolon - https://phabricator.wikimedia.org/T264614 (10Aklapper) {T238285}? [14:37:34] 10Wikimedia-Apache-configuration: Wikimedia's servers don't correctly rewrite short URLs when the URL ends in a semicolon - https://phabricator.wikimedia.org/T264614 (10Reedy) >>! In T264614#6517587, @Aklapper wrote: > {T238285}? Looks like it, yeah. Bar the search issue [14:38:04] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Reedy) [14:38:05] 10Wikimedia-Apache-configuration: Wikimedia's servers don't correctly rewrite short URLs when the URL ends in a semicolon - https://phabricator.wikimedia.org/T264614 (10Reedy) [16:23:31] 10Traffic, 10Analytics, 10Operations: ~1 request/minute to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021 (10fdans) Just pinging @Ottomata for when he's back from vacation. [16:26:30] 10Traffic, 10Analytics-Radar, 10Operations, 10Wikimedia-General-or-Unknown: Cookie “WMF-Last-Access-Global” has been rejected for invalid domain. - https://phabricator.wikimedia.org/T261803 (10fdans) [16:28:07] 10Traffic, 10netops, 10Analytics, 10Operations: Turnilo: per-second rates for wmf_netflow bytes + packets - https://phabricator.wikimedia.org/T263290 (10fdans) [16:28:11] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10fdans) [16:34:31] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10BBlack) With the dupe merger, maybe we owe a status update here: We're pretty sure this is a bug in Apache Tra... [16:41:56] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10fdans) a:03klausman [16:56:29] 10Traffic, 10Operations, 10Platform Team Initiatives (API Gateway), 10Story: Client Developer has a cookie-free API call - https://phabricator.wikimedia.org/T258748 (10eprodromou) [17:21:38] 10Traffic, 10Operations, 10Patch-For-Review: Wikidough: Upgrade to dnsdist 1.5.0 - https://phabricator.wikimedia.org/T263789 (10ssingh) Another important change in 1.5.0 is https://github.com/PowerDNS/pdns/pull/7138 [dnsdist/rec: Drop remaining capabilities after startup]. For our dnsdist instance, this is h... [18:38:19] 10Traffic, 10Operations, 10Performance-Team (Radar): Elevated latency starting 2020-09-28 - https://phabricator.wikimedia.org/T264398 (10Gilles) [22:43:39] quoting a question from an old but open ticket: 'VarnishStatus (added by @mmodell in 2015) running on deployment-cache-upload06.deployment-prep.eqiad.wmflabs as the last Diamond collector, is that still in use? (Or even useful given that we now use the ATS sandwich?)' [22:52:33] presumably that's referring entirely to beta cluster? [22:52:44] I have no idea what either the traffic stack or the metrics setup looks there tbh [22:58:46] cdanis: it's the last Diamond collector (globally, it seems) but also the only instance in beta using that role. so yes. there is also a dedicated 'traffic' project that uses the role::cache::upload. [22:59:03] there is a long history of removing all the diamond stuff in the past [22:59:03] that sounds awfully like cruft [22:59:25] yea, it does. this is like the last sanity check :) [22:59:48] because i did see some users still logging in on that machine in general [23:00:26] maybe i should just stop the service but not delete stuff for today [23:01:34] i will do that and log it in cloud SAL and on that old ticket [23:02:40] well, puppet still cares about it and restarts it.. people probably did not want to delete it because something was using it :) [23:05:31] comes from a class role::beta::availability_collector ... let me just suggest do delete that in gerrit .. shrug