[02:28:15] 10Traffic, 6operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2017669 (10BBlack) I was thinking about all of this last night, and we can probably mitigate some of the low-TTL-cap operational concerns with some modifications to the parameters and usage of grace mode.... [03:43:48] 7HTTPS, 10Huggle: Huggle 2 fails on HTTP used when HTTPS expected - https://phabricator.wikimedia.org/T126357#2017744 (10Jkbw) Hi, could someone be so kind to put a compiled version including this hack to http://tools.wmflabs.org/huggle/files/huggle%202.1.26-a.exe ? Thanks in advance. [07:00:47] 7HTTPS, 10Huggle: Huggle 2 fails on HTTP used when HTTPS expected - https://phabricator.wikimedia.org/T126357#2017882 (10DVdm) Yes, I will do later today. Have to remove some extra debugging code first. I also have added a bit of code to strip the page header, sidebar and footer out of the diff window--see [[... [08:56:24] 10Traffic, 6operations: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2017991 (10elukey) a:3elukey [08:57:04] ema: took ownership of https://phabricator.wikimedia.org/T124278, will talk with Andrew about next steps, I'll start it next week after the jessie migration of mcxxxx [09:12:30] elukey: fantastic! [09:26:05] let me know if you have specific deadlines in mind :) [13:17:37] 10Traffic, 6operations: Upgrade LVS servers to a 4.3+ kernel - https://phabricator.wikimedia.org/T119515#2018468 (10faidon) 5Open>3Resolved a:3faidon I just finished upgading the rest to 4.4.0 as well. Considering this task resolved. [13:42:09] 10Traffic, 6Performance-Team, 6operations, 5Patch-For-Review: Disable SPDY on cache_text for a week - https://phabricator.wikimedia.org/T125979#2018503 (10BBlack) So, looking at @Krinkle's metrics from above ( https://grafana.wikimedia.org/dashboard/db/performance-metrics ) from the change to now, we're se... [15:46:43] bblack: how are is the keep alive change looking so far? [15:51:13] elukey: not tested live yet, works fine on cp1008 [15:51:20] have other issues to sort out first :) [15:52:06] one of which is that (apparently without real user impact? because nothing obvious happened in graphs) the openssl-1.0.2f bugfix update started spamming these to nginx error log: [15:52:10] 2016/02/11 15:50:52 [crit] 3133#3133: *390957745 SSL_shutdown() failed (SSL: error:140C5042:SSL routines:ssl_undefined_function:called a function you should not call) while SSL handshaking, client: 75.174.71.179, server: 0.0.0.0:443 [15:53:40] o_O [15:54:10] from a related bugreport that's actually about libressl, this seems likely: [15:54:13] It looks like the alert is [15:54:15] due to LibreSSL dropped the support for export ciphers, while [15:54:18] nginx calls SSL_CTX_set_tmp_rsa_callback() to be able to support [15:54:21] them if configured to do so [15:54:22] I bet 1.0.2f changelog dropped export too [15:55:29] well changelog doesn't mention that explicitly, but still [15:56:28] something similar... [15:57:23] ok here we go: [15:57:23] http://openssl.6102.n7.nabble.com/Strange-problem-with-1-0-2f-SSL-shutdown-in-multithreaded-server-td62990.html [15:58:53] * elukey reads [16:13:14] https://github.com/nginx/nginx/search?utf8=%E2%9C%93&q=SSL_CTX_set_tmp_rsa_callback - mention only libressl [16:14:00] 10Traffic, 6operations: openssl-1.0.2f introduced minor bug with nginx - https://phabricator.wikimedia.org/T126616#2018818 (10BBlack) 3NEW a:3MoritzMuehlenhoff [16:14:38] the error reported in the commit was different though [16:15:35] elukey: see task linked above, I updated with more investigation [16:15:53] yep I was reading [16:39:19] 10Traffic, 6operations, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#2018958 (10BBlack) I'm live-testing this (at a setting of `1`) on an eqiad text machine (cp1065) right now. The connection/thread stuff I'm watching is the output of: ```... [16:41:50] 10Traffic, 6operations, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#2018972 (10BBlack) The above (if it holds) basically confirms earlier thinking: that this is the "right" thing to do and helps, but we've got upload-cluster-specific issues... [16:48:02] 10Traffic, 6operations, 5Patch-For-Review: HTTP/1.1 keepalive for local nginx->varnish conns - https://phabricator.wikimedia.org/T107749#2019001 (10BBlack) Should also note: the initial test of this was with keepalive's idle conns parameter set to `4`, whereas the new tests are at `1`. That alone may improv... [17:25:07] re: the keepalive stuff ( T107749 ) and the SPDY stuff ( T125979 ), there's a conspiracy theory brewing in the back of my head that there's a small chance they're inter-related due to a complex chain of things... [17:26:11] that effectively even if Chrome wanted to fetch the CSS before the body finished (or at least, earlier than it does now), that due to SPDY both requests are going into the same nginx worker process, which must clear transactions serially to varnish-fe when it's doing serial 1x non-keepalive to varnish-fe.... [17:26:42] whereas with the keepalive patch applied, it's at least possible for nginx to fetch it while its body-fetch to varnish-fe is still running in another connection. [17:27:30] nginx and/or chrome may have other limitations that prevent parallelism there regardless, but if they don't that could be what causes the serialization... [18:16:59] 10Traffic, 10Analytics-EventLogging, 6operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#2019446 (10Milimetric) [18:17:03] 10Traffic, 10Analytics, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 6operations: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#2019444 (10Milimetric) 5Open>3declined We decided to not split and merge events, this would be... [20:02:11] 7HTTPS, 10OTRS, 6operations, 5Patch-For-Review: ssl certificate replacement: ticket.wikimedia.org (expires 2016-02-16) - https://phabricator.wikimedia.org/T122320#2019918 (10Dzahn) deleted ticket.wikimedia.org.key in private repo pinged Robh, he is revoking the cert also deleted "new.ticket.wikimedia.org... [20:31:44] 7Varnish, 10RESTBase, 6Services, 6operations, 3Mobile-Content-Service: Enable caching for the Mobile Content Service's RESTBase public endpoints - https://phabricator.wikimedia.org/T113591#2020012 (10GWicke) A basic PR is now available at https://github.com/wikimedia/restbase/pull/511. This does not set... [23:05:22] 10Traffic, 6Performance-Team, 6operations: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2020906 (10ori) The Chrome team [[ http://blog.chromium.org/2016/02/transitioning-from-spdy-to-http2.html | announced ]] that they will drop support for SPDY (and NPN) in three months, on May 15th.