[08:07:38] 10Traffic, 10Continuous-Integration-Infrastructure, 06Operations, 10Packaging: piuparts fail with WARN: Broken symlinks: /etc/systemd/system... - https://phabricator.wikimedia.org/T141454#2501246 (10hashar) syslog-ng fixed the warning by adding `rm` and `rmdir` in the package `postrm` step: http://anonscm.... [09:50:30] 10Traffic, 06Operations: Age header reset to 0 after 24 hours on varnish frontends - https://phabricator.wikimedia.org/T141373#2501317 (10ema) >>! In T141373#2498969, @ema wrote: > Next steps: > > - port the test case to v4 and see how varnish 4 behaves https://phabricator.wikimedia.org/P3592 is a functional... [10:50:31] 10Wikimedia-Apache-configuration, 06Operations, 07HHVM, 07Wikimedia-log-errors: Fix Apache proxy_fcgi error "Invalid argument: AH01075: Error dispatching request to" (Causing HTTP 503) - https://phabricator.wikimedia.org/T73487#2501353 (10Joe) a:03Joe [12:19:04] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2501557 (10mark) >>! In T120425#2444215, @Nemo_bis wrote: > Yes AFAICT, from a quick check. What makes you think it wouldn't be? Ops,... [12:23:13] 10netops, 06Operations: Turn up new eqiad-esams wave (Level3) - https://phabricator.wikimedia.org/T136717#2501583 (10mark) [12:47:22] ema: to get a good repro on what we see in prod, though, probably the origin server in VTC needs to send Last-Modified as well [12:47:51] without a Last-Modified from the origin, there would be no date to try If-Modified-Since against, thus avoid 304-related issues [12:48:27] (not that the VTC "server" probably does 304 correctly without setting up a manual sequenced response, but you'll be able to see if either varnishd even asks for If-Modified-Since... [12:48:31] ) [13:02:39] so from bluecoat docs back in 2015 [13:02:40] #conf t -> ssl -> proxy dhe-ciphers disable [13:02:41] With 6.5.5.1 and above, DHE for SSL proxy is now disabled by default (can still be enabled, if needed). [13:03:22] their rationale in their knowledge base is something like "Hey DHE is more secure, but it costs way more CPU cycles to do forward secrecy, so we'll disable it by default because our appliance doesn't have that much CPU" [13:03:56] of course this is all when they were finally enabling DHE while the rest of the world was moving to ECDHE. Now even newer versions of their software do support modern ciphers. [13:04:19] there's just a decently-sized installed base that has these versions whose best ciphers (DHE) are disabled-by-default [13:04:45] but owners of those proxies can upgrade, or they can configure DHE enabled [13:05:03] (well, unless it causes their little appliance to saturate it's CPU and die) [13:08:25] also, +1 to Google for causing them pain. All 4 of their most-recent "Featured Articles" in https://bto.bluecoat.com/knowledgebase are about google breaking on bluecoat in May heh [13:08:47] they all read like "chrome + google servers did some new ssl stuff we can't handle, we'll try to fix it in the next release!" [13:09:38] oh 3/4, the last one is about their strict http protocol validation being a failure [13:12:40] openssl release ecdhe+ecdsa support in 2010, and TLSv1.2 in 2012. yet in 2015 bluecoats releases are saying things like "we implemented this new more-secure DHE thing, but it's disabled by default" [13:13:04] that's ridiculous for a company that claims to be good at anything security or SSL related [13:14:57] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2501715 (10Nemo_bis) >>! In T120425#2501557, @mark wrote: > @Nemo_bis: see Faidon's reply on Dec 7; if you provide us with your IP add... [13:58:21] 10Wikimedia-Apache-configuration, 06Operations, 07HHVM, 07Wikimedia-log-errors: Fix Apache proxy_fcgi error "Invalid argument: AH01075: Error dispatching request to" (Causing HTTP 503) - https://phabricator.wikimedia.org/T73487#2501796 (10Joe) I just uploaded the latest version of the apache package to rep... [14:04:58] \o/ \o/ ---^ [14:18:23] _joe_: that's our jessie apache package that's built against openssl-1.0.2 right? [14:18:45] _joe_: or do you mean trusty? honestly I don't even remember if we did a trusty apache + 1.0.2 [14:18:52] (I think not) [14:19:07] moritzm: ^ ? [14:20:28] <_joe_> akosiaris: jessie [14:20:35] <_joe_> err bblack [14:22:57] ok [14:23:19] I know moritzm made an apache jessie package for us before, now I don't remember if it was just upstream version bump (backport?) or openssl-1.0.2 or both [14:23:33] I'd have to dig to remember, I just want to make sure we don't regress it, we we're stock before either [14:26:02] yay found the ticket: https://phabricator.wikimedia.org/T133217 [14:27:20] so yeah the main thing is just that the new package uses openssl-1.0.2, which I think it would by default if it's built for jessie by us [14:27:46] (as in uses it as the compile-time dependency for feature support, not just links it at runtime) [14:28:18] upgrading one of our one-off sites would tell us for sure, since they'd lose working config and/or DHE support if not [14:28:37] like lists [14:28:52] trying :) [14:30:19] _joe_: 2.4.10-10+deb8u4+wmf3 is the new one, right? [14:30:30] <_joe_> yes [14:30:46] <_joe_> bblack: it should be linked to 1.0.2 allright [14:31:05] <_joe_> bblack: I can check [14:31:39] well it has to link 1.0.2, I think that's all we have [14:31:47] but it has to compile against 1.0.2 headers, too [14:31:58] (which I think is also what we'd have in any chroot build on our jessie) [14:32:26] in any case, functional testing confirms it did [14:32:45] ssllabs check on lists.wikimedia.org post-upgrade still shows DHE ciphersuites with our custom 2K key [14:32:47] <_joe_> bblack: mine too, take into account that we install openssl 1.0.2 by default [14:32:55] <_joe_> so does our build system [14:33:10] yeah [14:33:13] <_joe_> bblack: I might have found a friend who still owns an xbox 360 btw [14:33:45] <_joe_> but when I said "upgrade to the last service pack" he didn't look so amused [14:33:45] cool, it'd be nice to close the ticket. the check is just "make sure xbox 360 has latest updates, try to hit wikipedia from its browser" [14:34:01] we know un-updated ones failed last year [14:34:08] we just don't know if one of the udpates since fixed it [14:38:02] our custom apache2 package is built against 1.0.2, so it needs to be built for jessie-wikimedia [14:39:24] the old package had a small patch to force libssl >= 1.0.2, other than that no changes [14:45:16] ok [14:45:32] I did that forcing in our nginx at one point too I think, but I guess since 1.0.2 is our default, anything we build here gets it anyways [14:45:37] (even as headers for building) [14:45:55] (well, anything we build here specifically for jessie-wikimedia) [14:49:30] yeah, this was only to guard against accidentally building with DISTREL=jessie instead of DISTREL=jessie-wikimedia [16:31:17] 10Traffic, 06Operations: Support TLS chacha20-poly1305 AEAD ciphers - https://phabricator.wikimedia.org/T131908#2502239 (10BBlack) The facts of the situation: 1. We considered long ago in another ticket using cloudflare's OpenSSL patch (an older one) to implement the old Draft versions of ChaCha20-Poly1305 ci... [16:31:56] ^ long-winded dump of everything about the chacha20-poly1305 situation, arguing for using Cloudflare's (newer) patch (again). [16:32:18] moritzm: your input appreciated if you'd like to make a counter-case against it, as always :) [16:39:47] bblack: sure, I'll follow up on the task. not sure whether I'll get to it today, though (and I'm out until next Wednesday) [16:41:01] moritzm: np, it's not a rush thing :) [20:06:02] 10Traffic, 06Operations: SSL cert for policy.wm.org expiring Aug 27 - https://phabricator.wikimedia.org/T141564#2503063 (10Dzahn) [20:07:08] 10Traffic, 06Operations: SSL cert for policy.wm.org expiring Aug 27 - https://phabricator.wikimedia.org/T141564#2503063 (10Dzahn) a:05RobH>03None [20:07:45] 10Traffic, 06Operations: SSL certificate for policy.wikimedia.org - https://phabricator.wikimedia.org/T110197#1571514 (10Dzahn) This cert is going to expire next month. now. Created a subtask for that. [20:09:25] 10Traffic, 06Operations: SSL certificate for policy.wikimedia.org - https://phabricator.wikimedia.org/T110197#2503097 (10Dzahn) [20:09:28] 10Traffic, 06Operations: SSL cert for policy.wm.org expiring Aug 27 - https://phabricator.wikimedia.org/T141564#2503093 (10Dzahn) 05Open>03Invalid oops, duplicate of T140263 [20:09:58] 10Traffic, 06Operations: SSL cert for policy.wm.org expiring Aug 27 - https://phabricator.wikimedia.org/T141564#2503099 (10Dzahn) [20:10:01] 10Traffic, 06Operations: SSL certificate for policy.wikimedia.org - https://phabricator.wikimedia.org/T110197#1571514 (10Dzahn) [22:17:50] 07HTTPS, 10Traffic, 10Incident-20150820-OCSP, 06Operations, 05Wikimedia-Incident: Make OCSP Stapling support more generic and robust - https://phabricator.wikimedia.org/T93927#2503668 (10greg)