[09:47:57] 10Traffic, 10DNS, 10Operations, 10Toolforge, 10cloud-services-team (Kanban): Update authoratiative nameservers for the toolforge.org domain to point to Designate - https://phabricator.wikimedia.org/T235303 (10aborrero) p:05Triage→03High a:03Andrew I believe the last person doing this kind of moveme... [10:28:24] 10Traffic, 10Operations, 10Patch-For-Review: ATS-tls nodes on the text cluster have a slightly higher rate of failed fetches on varnish-fe - https://phabricator.wikimedia.org/T234887 (10Vgutierrez) After *a lot* of debugging, I've found something that could explain this behaviour. Testing locally with HTTP/1... [10:42:19] 10Traffic, 10Operations, 10Patch-For-Review: ATS-tls nodes on the text cluster have a slightly higher rate of failed fetches on varnish-fe - https://phabricator.wikimedia.org/T234887 (10Vgutierrez) Reported to upstream in https://github.com/apache/trafficserver/issues/6018 [14:34:43] 10Traffic, 10Operations: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10jijiki) p:05Triage→03Normal [14:45:46] 10Traffic, 10Operations, 10Phabricator: Access Forbidden to Phabricator at WikiArabia 2019 (Morocco) - https://phabricator.wikimedia.org/T234598 (10jijiki) p:05Triage→03High [14:46:08] dear traffic, I triaged T234598 to high, feel free to lower it to normal, up to you [14:46:09] T234598: Access Forbidden to Phabricator at WikiArabia 2019 (Morocco) - https://phabricator.wikimedia.org/T234598 [15:11:03] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10serviceops, 10CPT Initiatives (Core REST API in PHP): MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:11:32] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) Tagging #techcom for visibility. [15:16:54] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:18:15] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:19:54] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:23:55] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) According to Apache2 documentation, setting [`AllowEncodedSlashes NoDecode`](https://httpd.apache.or... [17:54:05] 10Wikimedia-Apache-configuration, 10MediaWiki-REST-API, 10TechCom, 10serviceops, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [18:08:54] 10Traffic, 10Operations, 10Patch-For-Review: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10ema) Thanks for filing this task @Andrew! I did add `profile::trafficserver::tls::inbound_tls_settings` to `... [18:10:13] 10Traffic, 10Operations, 10serviceops, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10ema) 05Open→03Stalled [19:43:35] 10Traffic, 10Operations: ATS lua script reload doesn't work as expected - https://phabricator.wikimedia.org/T233274 (10ema) The ability to reload global lua scripts has been added in 9.x only: https://github.com/apache/trafficserver/commit/e6147753cd65c3edd32b365e09b4d65edcffdd01 This explains why reloading a... [20:52:03] 10Traffic, 10Operations, 10ops-codfw: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Papaul) @Vgutierrez thanks for the update. The plan was for us to decommission one lvs in rack A2 and B2 so I can use the existing cables to setup lvs2007 and lvs2008. Both lvs2007 and 2008 ar... [23:55:21] 10Traffic, 10Operations: Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs - https://phabricator.wikimedia.org/T234256 (10Andrew) > Error while evaluating a Function Call, Could not find data item profile::trafficserver::tls::parent_rules If you don't...