[03:51:25] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2781793 (10GWicke) @anomie, I see- "page 1" being the default is indeed one way where there could be fragmentation. It might indeed make sense to make an... [09:02:14] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2781961 (10Tgr) The reason the current thumb URL format does not allow larger-than-original (and in most cases same-as-original) sizes is to avoid cache f... [09:30:57] 10Traffic, 06Analytics-Kanban, 06Operations: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2781980 (10elukey) We had a discussion on #wikimedia-traffic about this and the Analytics team completely agrees with what Brandon sai... [13:20:58] ok, routing cache_text esams to codfw first, then starting the esams upgrade to v4 [13:26:37] possibly using the right yaml file this time... [13:27:43] I just got here a bit ago [13:27:58] and saw the revert, but, maybe I'm too asleep. the original looked correct? [13:28:17] bblack: hi! Yeah I'm still confused by the US elections. It was indeed correct. [13:30:36] (woke up at 5am to follow). Anyways, coffee and then I'll try again :) [13:48:41] 10netops, 10DBA, 06Labs, 10Labs-Infrastructure, and 3 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2782298 (10mark) [13:49:33] 10netops, 06Operations, 10hardware-requests, 10ops-eqiad, 13Patch-For-Review: Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium - https://phabricator.wikimedia.org/T149829#2782302 (10mark) [13:54:40] ok, fixed and puppet-salted (sorry for the mess) [14:02:01] 10netops, 06Operations, 10hardware-requests, 10ops-eqiad, 13Patch-For-Review: Move labsdb1008 to production, rename it back to db1095, use it as a temporary sanitarium - https://phabricator.wikimedia.org/T149829#2764445 (10mark) labsdb1008 is in rack C3 right now. We can simply move it to vlan private1-c... [14:29:11] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782384 (10Gilles) >>! In T66214#2781560, @Anomie wrote: > Say we didn't already have a parameter to select the page of a PDF. Then we add the parameter.... [14:31:15] so many layers to the curve-logging problem [14:32:22] 1. OpenSSL doesn't provide an official server-side API to ask "what ECDHE curve did we actually use for this fresh connection?" (it does offer a function that gives you client<->server shared curve, which is the same selection criteria, though...) [14:32:47] 2. So you use the function above but, it doesn't work for resumed sessions, only for new ones. Why? [14:33:25] 3. Well, on some level because when you resume a session, you're not re-keying, so you don't actually *use* ECDHE in that case [14:33:56] 4. But! the SSL_SESSION structure has members which save that information, so you should be able to get it from those. and in fact, that function we're using does use those, so wtf? [14:34:52] 5. Oh, because the SSL_SESSION structure is just the temporary in-memory representation. When it's serialized to real-world session storage and back, there's a separate ASN.1 representation of that structure for serialization, which doesn't bother encoding the part about ECDHE curves (why would it, since we don't need them for resumption)? [14:35:44] 6. so at this point you rewind and say, hey, we can just log the curve choice for *new* sessions and not-log it for resumed ones. that makes sense on some level, doesn't it? [14:36:15] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782399 (10Gilles) >>! In T66214#2781793, @GWicke wrote: > @anomie, I see- "page 1" being the default is indeed one way where there could be fragmentation... [14:37:32] 7. except ... we're logging all of this per-Request, not per-session. All of our stats are per-request. It's actually the better way to do it, because otherwise if we only logged them on the first request of each session (not that nginx gives us any easy way to do that anyways), we'd see stats heavily skewed in favor of UAs that make many connections for few requests, and not in favor of UAs th [14:37:38] at make one connection and pipe many requests in (in other words, ancient-browsers would unfairly dominate modern HTTP/2 ones) [14:38:25] the onyl "right" answer to that is to log them per-request, because a given pageview or whatever has the same count of requests regardless of HTTP/1 connection parallelism, HTTP/2 non-parallelism, or session resumption. [14:39:15] if we only logged curve choices from all the requests in a new session, but not all the requests in a resumed one, it would seriously skew the results. Ditto (if nginx even had a way) if we only logged it for the first request of a new connection but not the first request of a resumed one. [14:40:44] 8. ok, so rewind again. clearly we should just add the curve data to the serialized ssl session format, so we can log it correctly on every request, resumed-session or not. [14:41:11] 9. But, apparently the simple act of adding a few new fields to the ASN.1 representation breaks a large swath of the existing testsuite in OpenSSL :P [14:41:49] 10. Maybe I can leave that alone, and add the chosen curve to nginx's ssl session structure, which contains said ASN.1, since we only need to use it from nginx? [14:42:13] and now I'm sitting there looking at how invasive such a patch ends up being for various arcane reasons [14:42:45] how hard should it be to get an accurate fucking number for "this rough percent of our clients are using X25519 key exchange" :P [14:45:14] * bblack rant mode off [14:48:38] bblack: I know this is my standard solution to all problems in the world including climate change, but: systemtap? [14:48:56] :) [14:49:36] even systemtap wouldn't know without code changes. when a session is resumed, the client doesn't re-send its list of acceptable curves, and openssl doesn't have that info in the serialized session storage it's resuming from, either. [14:49:52] (can systemtap do election polling next time?) [14:50:35] but assuming systemtap did work in this case: are you sayign systemtap is an acceptable stats-logging source we can run full-time sending to ganglia or whatever? [14:50:54] s/ganglia/graphite/ [14:50:59] s/graphite/promethus/ [14:51:01] whatever :) [14:51:16] bblack: nope, I was thinking of something like the h2-vs-spdy tests we ran in the past [14:51:36] ah yeah, I want this in long-term stats though [14:51:51] there will be future curve changes too, and we want history to compare and trends to watch, etc [14:53:28] well, I say the client doesn't re-send the curve list. I don't think it does on resume. In any case OpenSSL basically ignores it if it does. But maybe I should check for it. [14:54:06] if that's there on the wire, it might offer a path towards a functioning but less-hacky patch [14:56:16] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782449 (10Anomie) >>! In T66214#2781793, @GWicke wrote: > @anomie, I see- "page 1" being the default is indeed one way where there could be fragmentation... [14:58:22] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782454 (10GWicke) > Currently the Varnish task seem to only focus on parameter reordering, when this is already an early sign that as the URL scheme evol... [15:11:06] !log upgrading cp3033 (text-esams) to varnish 4 -- T131503 [15:11:06] Not expecting to hear !log here [15:11:06] T131503: Convert text cluster to Varnish 4 - https://phabricator.wikimedia.org/T131503 [15:11:12] yeah [15:23:34] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782563 (10Anomie) >>! In T66214#2782384, @Gilles wrote: > In regards to the client cache fragmentation, I think most clients are consistent with themselv... [15:28:24] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782611 (10Anomie) >>! In T66214#2782454, @GWicke wrote: > The bigger issue I see with query strings is that (contrary to what I thought earlier) all clie... [16:00:55] 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2782663 (10hoo) @BBlack Given {T109458} is not implemented, there is no caching for these pages right now. If you consider this a r... [16:26:12] text esams upgraded, overall hitrate looks good, frontends re-filling [16:34:23] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782727 (10Gilles) >>! In T66214#2782449, @Anomie wrote: > I think redirects are probably //essential//. You say that the performance impact of redirects... [17:18:22] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782833 (10GWicke) >>>! In T66214#2782454, @GWicke wrote: >> This is because any thumbnail can have any number of author-supplied parameters already > >... [17:24:00] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782850 (10Gilles) Provided by editors in the context of wikitext, yes. I thought you meant the media's author was somehow able to set default parameters.... [17:28:25] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782860 (10GWicke) @gilles: Having such author-supplied parameters in a thumb means that changing the size is not just a matter of adding a *single* query... [17:28:37] 10Traffic, 10netops, 06Operations: Re-setup lvs1007-lvs1012, replace lvs1001-lvs1006 - https://phabricator.wikimedia.org/T150256#2782861 (10BBlack) [17:34:16] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782869 (10Gilles) In both cases you need to parse a string and insert the value in the right place. I hope we're not going to design an API a certain way... [17:34:30] 10netops, 06Operations: HTCP purges flood across CODFW - https://phabricator.wikimedia.org/T133387#2782871 (10akosiaris) Some more information about this. After quite a bit of debugging I 've gathered the following facts * The issue is present across all CODFW rows as well as asw2-d-eqiad * The issue only man... [17:36:03] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782875 (10Gilles) You can look at any two formats and make up a use case that would be easier in one than the other. I don't think that should be the sta... [17:43:02] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782890 (10GWicke) The point is that a custom ordered-querystring serialization function will be needed, while in the path-based API regular path manipula... [17:49:48] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782915 (10Gilles) The pros and cons of the current URI scheme should be included in the same matrix. Because you'll find that almost all of the upsides y... [18:02:39] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782978 (10GWicke) [18:07:34] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2782999 (10Gilles) I'm sorry but the pros and cons you've entered are completely biased and focus on very subjective properties. [18:09:43] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783005 (10Gilles) "Easy to select size and thumb type." How is it not easy in the second example? What constitutes "mis-use"? [18:17:03] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783026 (10Gilles) I would once again recommend that you write code that would parse, generate and cover a few known use cases for all considered schemes,... [18:48:13] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783095 (10Anomie) > and (generally) avoid specifying default values explicitly. (Exception might be the page parameter.) Maybe you didn't get it after a... [19:11:24] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783168 (10GWicke) > Maybe you didn't get it after all. There's nothing special about the "page" parameter, you could have the same problem with "t=0s" an... [19:18:04] heh curve logging now works per-request. the nginx change is ugly though, and unlikely to be the kind of thing that gets upstreamed. it should really be fixed at the openssl level. [19:23:53] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783210 (10GWicke) [20:53:20] 10Traffic, 06Operations, 10media-storage: Unexplained increase in thumbnail 500s - https://phabricator.wikimedia.org/T147648#2699497 (10JoeWalsh) `Wikipedia/942 CFNetwork/808.0.2 Darwin/16.0.0` is a user agent passed by the iOS app's WebView (which displays the article HTML). In some cases, the iOS app alter... [21:04:23] 10Traffic, 06Operations, 10media-storage: Unexplained increase in thumbnail 500s - https://phabricator.wikimedia.org/T147648#2783750 (10JoeWalsh) I made a change to prevent the iOS app from requesting zero width thumbnails. Should go out with version 5.3.0 [21:24:49] 10Traffic, 06Operations, 10media-storage: Unexplained increase in thumbnail 500s - https://phabricator.wikimedia.org/T147648#2783834 (10BBlack) @JoeWalsh - sounds great! can we get a link to the change? [21:26:16] 10netops, 06Operations: Low IPv6 bandwth from Free.fr (AS12322) > Zayo > eqiad - https://phabricator.wikimedia.org/T150374#2783846 (10hashar) [21:27:32] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2783861 (10GWicke) [22:47:59] 10netops, 10EventBus, 06Operations, 10ops-codfw: kafka2003 switch port configuration - https://phabricator.wikimedia.org/T150380#2784090 (10Papaul) [23:42:48] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2784282 (10GWicke) IRC meeting notes discussing this RFC: E355 Next steps: 1) look into which users rely on the current thumb format 2) investigate effo... [23:49:47] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2784337 (10GWicke)