[03:27:14] 10Traffic, 10Operations, 10ops-codfw: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Vgutierrez) @papaul you can proceed at will with lvs2009 and lvs2010 because they are not handling production traffic at the moment [04:17:40] 10Traffic, 10Operations: ATS lua script reload doesn't work as expected - https://phabricator.wikimedia.org/T233274 (10Vgutierrez) nice catch, I have to backport the SSL EC cache PR... so I guess I'll include this one as well [04:24:38] 10netops, 10Operations: mr1-eqsin.oob IPv6 connectivity flapping - https://phabricator.wikimedia.org/T227967 (10Marostegui) 05Resolved→03Open This has been flapping overnight (times in UTC+2): ` [02:46:49] <+icinga-wm> PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [03:45:4... [04:41:31] 10Traffic, 10Operations: ATS lua script reload doesn't work as expected - https://phabricator.wikimedia.org/T233274 (10Vgutierrez) hmmm it looks like the commit is on 8.x but the feature is disabled by default, see https://github.com/apache/trafficserver/pull/3940/commits/cc7b7dd7540fb33d93aa969defb416461c0202e7 [06:06:23] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) Taking a quick dump of the `$_SERVER` variable shows the content of `REQUEST_URI` are - as expected - une... [06:17:15] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) No, MediaWiki [prefers `REQUEST_URI`](https://github.com/wikimedia/mediawiki/blob/a03feb8ab05c29ecdf... [06:21:58] 10netops, 10Operations: mr1-eqsin.oob IPv6 connectivity flapping - https://phabricator.wikimedia.org/T227967 (10elukey) The ripe atlas ipv6 alert is in CRITICAL state as well again. [06:22:46] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) If I set `AllowEncodedSlashes` to `NoDecode` it works as intended, but: * It must be set at the vhost le... [06:25:39] 10netops, 10Operations: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin - https://phabricator.wikimedia.org/T228015 (10elukey) 05Resolved→03Open This seems re-happening (from my home again): ` [..] 4. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.... [06:37:09] 10netops, 10Operations: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin - https://phabricator.wikimedia.org/T228015 (10elukey) Contacted Hurricane Electric, at this point we could add the HE path to AVOID-PATH? [06:46:44] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) After further examination, given MediaWiki indeed uses `REQUEST_URI`, and given that is unencoded in all... [07:05:02] 10netops, 10Operations: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin - https://phabricator.wikimedia.org/T228015 (10elukey) 05Open→03Resolved HE fixed the issue on their end, all good now. [07:05:26] 10netops, 10Operations: mr1-eqsin.oob IPv6 connectivity flapping - https://phabricator.wikimedia.org/T227967 (10elukey) 05Open→03Resolved Related to a HE issue, see T228015 [07:20:16] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10Joe) 05Open→03Invalid Just confirmed the problem is at the varnish layer, not at the mediawiki one. I modi... [07:53:26] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Anycast recdns - https://phabricator.wikimedia.org/T186550 (10ayounsi) >>! In T186550#5383508, @elukey wrote: > Couple of notes about the anycast-healthchecker: > 2) `python3-docopt` seems to be required by the healthchecker's nagios monitor, and it wa... [07:53:33] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Anycast recdns - https://phabricator.wikimedia.org/T186550 (10ayounsi) 05Open→03Resolved [07:53:40] 10Traffic, 10netops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10ayounsi) [09:06:50] 10Traffic, 10MediaWiki-REST-API, 10Operations, 10Parsoid-PHP, 10CPT Initiatives (Core REST API in PHP): Varnish/ATS should not decode URIs for /w/rest.php - https://phabricator.wikimedia.org/T235478 (10mobrovac) p:05Triage→03High [09:24:35] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) Thank you @Joe for taking the time to investigate. I created {T235478} to follow up with the traffic... [10:40:10] 10Traffic, 10CommRel-Specialists-Support, 10Core Platform Team, 10Editing-team, and 12 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Elitre) [11:40:00] https://ripe79.ripe.net/archives/video/178/ might be interesting for traffic [12:48:23] that one is full of useful findings too: https://ripe79.ripe.net/archives/video/184/ [13:17:51] 10Traffic, 10DNS, 10Operations, 10Toolforge, 10cloud-services-team (Kanban): Update authoratiative nameservers for the toolforge.org domain to point to Designate - https://phabricator.wikimedia.org/T235303 (10Andrew) @robh should I email Doneva myself and cc you for approval or do you want to reach out y... [13:34:17] 10Traffic, 10Operations, 10Phabricator: Access Forbidden to Phabricator at WikiArabia 2019 (Morocco) - https://phabricator.wikimedia.org/T234598 (10Reedy) @SandraF_WMF Do you happen to remember if you got that IP address before connecting to your VPN? As it seems to belong to an Indian hosting company - http... [13:38:46] 10Traffic, 10Operations, 10Phabricator: Access Forbidden to Phabricator at WikiArabia 2019 (Morocco) - https://phabricator.wikimedia.org/T234598 (10Reedy) p:05High→03Normal Event was two weeks ago.... [14:55:04] 10Traffic, 10MediaWiki-REST-API, 10Operations, 10Parsoid-PHP, 10CPT Initiatives (Core REST API in PHP): Varnish/ATS should not decode URIs for /w/rest.php - https://phabricator.wikimedia.org/T235478 (10mobrovac) [15:12:38] 10Traffic, 10DNS, 10Operations, 10Toolforge, 10cloud-services-team (Kanban): Update authoratiative nameservers for the toolforge.org domain to point to Designate - https://phabricator.wikimedia.org/T235303 (10RobH) >>! In T235303#5575089, @Andrew wrote: > @robh should I email Doneva myself and cc you for... [17:04:48] 10Traffic, 10Operations, 10ops-codfw: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) @Vgutierrez firmware upgrade complete on both servers [17:21:18] 10netops, 10DC-Ops, 10Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541 (10Papaul) @wiki_willy tis task has been open since 2015 and it is not moving and nothing has been decided on this. I mentioned that the AP that we used to have onsite is no longer working. 1- if w... [17:26:05] 10netops, 10DC-Ops, 10Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541 (10wiki_willy) Hi @Papaul - if there aren't any objections from anyone, I think we can just resolve this. You have your primary connection via MIFI and a backup option via CyrusOne. And since it... [17:30:42] 10netops, 10DC-Ops, 10Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541 (10Papaul) 05Stalled→03Resolved [17:52:31] 10Traffic, 10Operations: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10ema) >>! In T234256#5573825, @Andrew wrote: >> Error while evaluating a Function Call, Could not find data item profile::trafficse... [17:55:40] 10Traffic, 10MediaWiki-REST-API, 10Operations, 10Parsoid-PHP, and 2 others: Varnish/ATS should not decode URIs for /w/rest.php - https://phabricator.wikimedia.org/T235478 (10WDoranWMF) [17:59:38] 10Traffic, 10Operations: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10Andrew) @ema, the missing key in my paste is a different key from the one you mentioned in your comment. You were talking about... [18:08:48] 10Traffic, 10Operations: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10ema) >>! In T234256#5576473, @Andrew wrote: > @ema, the missing key in my paste is a different key from the one you mentioned in y... [18:38:43] 10Traffic, 10Operations, 10Patch-For-Review: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10ema) 05Open→03Resolved a:03ema Puppet works again on both instances after adding `tls::parent_rules`,... [19:01:07] 10Traffic, 10MediaWiki-REST-API, 10Operations, 10Parsoid-PHP, and 2 others: Varnish/ATS should not decode URIs for /w/rest.php - https://phabricator.wikimedia.org/T235478 (10ema) In ATS-land, URI path normalization can be configured by passing the right parameters to the script `normalize-path.lua`. See [[... [20:18:47] 10netops, 10DC-Ops, 10Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541 (10Dzahn) 05Resolved→03Declined [21:05:46] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission radon - https://phabricator.wikimedia.org/T202040 (10Jclark-ctr) a:05Cmjohnson→03Jclark-ctr