[06:27:56] <ema>	 elukey: see Xio.NoX comment above for an explanation of yesterday's network issues in case you haven't yet :)
[06:35:46] <elukey>	 ema: o/
[06:35:55] <elukey>	 do you mean the link flapping?
[06:36:15] <elukey>	 I'll ask him to explain how to get that from the logs, it is not really clear to me :D
[06:36:23] <ema>	 yes, + the comment about router upgrade and https://phabricator.wikimedia.org/T196432
[06:42:18] <elukey>	 ahhhhvery nice!
[06:42:29] <elukey>	 thanks!
[06:59:37] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10VisualEditor: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10ema) >>! In T213214#5018951, @The_RedBurn wrote: > I did noti...
[07:02:32] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10VisualEditor: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10The_RedBurn) >>! In T213214#5020003, @ema wrote: >>>! In T213...
[07:03:14] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10VisualEditor, 10User-Ryasmeen: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10ema) 05Open→03Resolved a:03ema >>! I...
[08:30:55] <wikibugs>	 10Traffic, 10ExternalGuidance, 10Operations, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Deliver mobile-based version for automatic translations - https://phabricator.wikimedia.org/T212197 (10KartikMistry) Update: We are deploying ExternalGuidance to all Wikis (but mainly visible for...
[08:48:48] <wikibugs>	 10Traffic, 10Operations, 10Patch-For-Review: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893 (10ema) 05Open→03Resolved a:03ema Closing this as cp1077 is fine and back in service. For the general Varnish s...
[09:07:43] <wikibugs>	 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Addshore) >>  In the case that lead to this ticket, it was a remote client at Orange issuing a very high rate of these uncachea...
[09:07:53] <ema>	 the relationship between number of cached objects and hitrate is quite interesting
[09:08:20] <ema>	 I'm staring at `dstat --varnishstat --varnish-hit 10` on cp3030, where a few minutes ago I've restarted varnish-fe 
[09:09:12] <ema>	 hitrate is ~80% with a little over 1M objects, while on full caches (eg: cp3033) we've got ~87% hitrate and > 10M objects
[09:12:35] <ema>	 who knows if someone investigated this relationship in the literature and came up with simple admission policies based on the number of objects?
[09:12:44] <ema>	 where's Danielsberger when you need him
[09:37:34] <wikibugs>	 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Addshore) The only other thing I was going to add (forgot before i hit submit on the last post)   Within the cluster varnish ca...
[10:25:42] <mutante>	 now looking for Icinga URLs for monitoring checks "LVS HTTP IPv4" / IPv6 etc. i guess just  https://wikitech.wikimedia.org/wiki/LVS   is the best there is because it needs to be a generic one for each service using that defined type  lvs::monitor_service_http_https ?
[10:35:42] <ema>	 mutante: yeah, maybe https://wikitech.wikimedia.org/wiki/LVS#Diagnosing_problems ? We'd need to add some more content to make it useful, but it seems like a good choice for reference in case of fire
[10:36:36] <mutante>	 ema: ah, yeah, that anchor seems good. using that )
[10:37:10] <mutante>	 lvs::monitor_services has more specific ones for each service.. i can find other ones there
[11:29:06] <ema>	 FTR: cp3030 now reached 86% hitrate with about 6M objects
[13:29:30] <Krinkle>	 The debug profile index at https://performance.wikimedia.org/xhgui/ is currently cached for quite a while (3 hours for my PoP, and still counting). As far as I can tell, the backend does not serve Cache-Control. What is the default behaviour in that case? I was expecting maybe something like 5 minutes.
[14:03:14] <wikibugs>	 10netops, 10Operations: eqiad 10G ports needs - https://phabricator.wikimedia.org/T190364 (10jcrespo) I have better information of Databases backups/provisioning service:  in the end, only 2 hosts per datacenter T216137 T216138 over the next 3-5 years  A summary of the logical architechture can be seen at: htt...
[14:17:46] <ema>	 Krinkle: default_ttl is 86400s (24h) per caching layer
[14:18:13] <ema>	 Krinkle: if you're unlucky, thus, you can get Age up to ~ default_ttl*n_layers
[14:38:38] <Krinkle>	 ema: Hm.. right. I've noticed the same on integration.wikimedia.org as well, with regards to code coverage reports. There, I find it sometimes even after 2 days, but it's always up to date on refresh after that. So I guess that's a case of grace doing a background refresh while serving stale on more time.
[14:38:48] <Krinkle>	 doc.wikimedia.org, *
[15:07:02] <bblack>	 if there's no cache headers, the default_ttl is going to just multiply across layers, so yeah you could see 2-4 days.
[15:07:07] <bblack>	 (depending on your edge)
[15:07:54] <bblack>	 for low-traffic/infra apps with no cache policy emissions, we can also just set them to never cache pretty easily in the puppet metadata
[15:50:31] <wikibugs>	 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: Internal DNS resolver responds with NXDOMAIN for localhost AAAA - https://phabricator.wikimedia.org/T125170 (10jbond) FYI this got fixed in pdns version 4.1.0 (its actully in 4.1.0-alpha1) via https://github.com/PowerDNS/pdns/pull/5223
[18:12:21] <Krinkle>	 bblack: Thanks. I'll try upstream ( https://github.com/perftools/xhgui/issues/261 ), and if that doesn't work out, I'll file a task for a puppet change.
[18:13:44] <ema>	 Krinkle: https://wikitech.wikimedia.org/wiki/Varnish#TTL hopefully this helps clarify things :)
[18:16:08] <Krinkle>	 ema: thanks, I knew most of that, but it's good to see it all in once place. I'll bookmark this.
[18:16:27] <Krinkle>	 ema: one question though about ttl * 4. Do we not carry the Age value forward as the value propagates?
[18:18:32] <Krinkle>	 for the load.php startup module, for example, we set a ttl of 5 minutes. We tolerate that to be upto 10 min practice due to overlap between client and server. But I'd really not want to see it be possible that a client gets sent a copy from Varnish more than 5 minutes old, unless it was (in)directly known to be valid still according to MediaWiki.
[18:19:19] <Krinkle>	 I'd also tolerate the first request after 5min to be stale until a regeneration req has completed (during grace).
[18:19:41] <ema>	 Krinkle: it does propagate, yes
[18:19:50] <Krinkle>	 Does our VCL currently offer a way to ensure that, and if so, what do I need to do to get that?
[18:25:37] <ema>	 Krinkle: we can probably look at Age and act accordingly, but I'm not really sure. I'll find out tomorrow! 
[18:26:44] <ema>	 but no, our VCL currently does not allow that
[18:34:52] <wikibugs>	 10Traffic, 10Operations, 10Patch-For-Review: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263 (10ema) Preliminary testing went well: the `cp-ats` cluster in codfw served all frontend misses from cp2002 for about one hour between 17:18 and 18...
[18:51:25] <Krinkle>	 ema: OK. I wrote a little script that I'll let run for a few hours (polling every 10 seconds) to see what current behaviour is over time. observing http-age (varnish), http-expires (MW) and backend-timing/t (the real age, from MW).
[18:53:32] <wikibugs>	 10Traffic, 10netops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10jbond) Some comments for consideration  >We set up those (now global across our network!) service IPs behind LVS in all of our sites. is auth dns already behi...
[18:54:21] <Krinkle>	 https://gist.github.com/Krinkle/4593a5d76a474927e52b5b9fa19585a1#file-02-output-tsv
[18:54:59] <Krinkle>	 already seeing some interesting results.
[21:14:04] <Krinkle>	 ema: If an object is renewed via INM/304 from varnish backend to MW, does the object get modified or just renewed as is? Specifically, do any of the HTTP headers on the 304 response get mixed into the old object?
[21:16:17] <Krinkle>	 I have about a dozen other questions, I think maybe it would help if we create a flow chart that shows what can happen from start to end with an object. 
[21:17:08] <Krinkle>	 Could do a short meeting later this or next week to Q&A, then I can document it for myself.
[22:30:24] <wikibugs>	 10Traffic, 10netops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10ayounsi) >>! In T98006#5022090, @jbond wrote: Thanks for your comments! > is auth dns already behind LVS.  AuthDNS is not behind LVS and currently have static...