[09:13:20] ema: vgutierrez: hello! I will need to change the host for doc.wikimedia.org, we have prepared a brand new one. Daniel wrote a puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480536/ [09:13:40] next is to get it deployed on our cache infra, but I don't know how to proceed. Should it be scheduled ahead of time? [09:25:43] hashar: bonjour monsieur! I'll review and merge today, no need for particular coordination [09:39:10] ema: thank you! [09:50:36] 10Traffic, 10Operations, 10Patch-For-Review: ATS production-ready as a backend cache layer - https://phabricator.wikimedia.org/T207048 (10ema) 05Open→03Resolved We've added TLS support for maps and fixed the SAN list on swift to ensure proper TLS connections with upload origin servers. This is thus done. [09:58:04] _joe_: out of curiosity, what's the purpose of the X-Powered-By header in checkcommands.cfg.erb? [09:58:25] <_joe_> ema: it testifies if the page was rendered with php7 or hhvm [09:58:48] <_joe_> a page rendered with hhvm will have X-Powered-By: HHVM [09:59:05] <_joe_> we will have X-Powered-By: php7.2.... [09:59:10] <_joe_> for pages rendered with php 7 [10:03:30] _joe_: oh ok. We should maybe update the description of T206339 then, as we're not actually using that header to split the cache? [10:03:30] T206339: Separate Traffic layer caches for PHP7/HHVM - https://phabricator.wikimedia.org/T206339 [10:05:34] (we're using X-Seven, added as a request header if the browser sends the right cookie) [10:12:04] <_joe_> that's a request header right? [10:12:13] <_joe_> not a response header [10:12:20] <_joe_> X-Powered-By is a response header [10:14:31] yes, X-Seven is a request header [10:47:59] ema: thanks for the merge. doc.wikimedia.org works as expected :) [10:48:36] np! [11:02:16] 10Traffic, 10Operations, 10Patch-For-Review: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10ema) 05Open→03Resolved a:03ema The patch by @Vgutierrez fixed this bug. Closing. [11:10:18] 10Traffic, 10Operations: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263 (10ema) [11:10:26] 10Traffic, 10Operations: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263 (10ema) p:05Triage→03Normal [11:26:06] _joe_: what I meant is: the task description does not seem accurate. It should say that we split the cache using X-Seven, and not X-Powered-By? [11:26:40] <_joe_> we do, because we do it with Vary [11:26:52] <_joe_> which can only work with request headers [11:30:08] 10Traffic, 10Operations, 10Patch-For-Review: Separate Traffic layer caches for PHP7/HHVM - https://phabricator.wikimedia.org/T206339 (10ema) [11:30:21] _joe_: ok, description updated. Thanks :) [14:12:42] 10Traffic, 10Operations, 10Patch-For-Review: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Ricordisamoa) I still get dropped connections (not 414) with much longer URLs (total header size from 9442 bytes onward). Is that expected? [15:23:11] 10Traffic, 10Operations, 10Patch-For-Review: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Anomie) 05Resolved→03Open Confirmed. The URLs described earlier work now, and there's even a range of URLs that give a 414 from Apache with HTTP/2 now, but I still get a dro... [15:26:06] 10Traffic, 10Operations, 10monitoring: prometheus-based graph significantly slower than statsd equivalent - https://phabricator.wikimedia.org/T212312 (10CDanis) Is this basically the same as T190992? Anyway I'm making all the 'slow prometheus query' tasks sub-tasks of the prometheus 2.x upgrade T187987 as t... [15:26:42] 10Traffic, 10Operations, 10monitoring: prometheus-based graph significantly slower than statsd equivalent - https://phabricator.wikimedia.org/T212312 (10CDanis) [15:26:44] 10Traffic, 10Operations, 10monitoring: prometheus: slow dashboards due to suboptimal query_range performance - https://phabricator.wikimedia.org/T190992 (10CDanis) [15:48:30] 10netops, 10Operations, 10fundraising-tech-ops: Refresh Minfraud IP list - https://phabricator.wikimedia.org/T213100 (10cwdent) 05Open→03Resolved a:03cwdent @ayounsi thanks for the help, this looks good [16:07:08] 10Certcentral: Avoid inter-hosts puppet dependencies on certificate deployment - https://phabricator.wikimedia.org/T213301 (10Vgutierrez) [16:07:17] 10Certcentral: Avoid inter-hosts puppet dependencies on certificate deployment - https://phabricator.wikimedia.org/T213301 (10Vgutierrez) p:05Triage→03Normal [16:35:08] 10netops, 10Operations, 10monitoring, 10Patch-For-Review: Icinga check for VRRP - https://phabricator.wikimedia.org/T150264 (10ayounsi) a:05ayounsi→03faidon CR tested and updated, ready for reviews. [19:48:18] 10Traffic, 10ExternalGuidance, 10Operations: Deliver mobile-based version for automatic translations - https://phabricator.wikimedia.org/T212197 (10dr0ptp4kt) Hi @bblack , any suggestion here? We determined on a separate thread that I'll be the one writing the patch. CC @Arrbee @ggellerman [23:20:51] 10netops, 10Operations: Add eqsin routing special cases to jnt - https://phabricator.wikimedia.org/T211930 (10ayounsi) a:05ayounsi→03faidon Thanks for the feedback! If the back and forth in diffs via a task gets old, we can find a different solution. 1. 2. 3. Indeed using the ASXXX_in and _out as well as... [23:56:11] 10netops, 10Operations, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) The following need to be pushed to routers to stop adding a higher local_pref to routes learned via peering (IXP+private). `lang=diff [edit policy-options policy-...