[02:02:37] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-IRC-RC-Server, and 2 others: Remove the "HTTPS to HTTP" url filter in the IRC feed - https://phabricator.wikimedia.org/T122933#1916986 (10Dzahn) 19:04 < mutante> that irc link thing there. it has a comment "can be merged anytime. no impact" 19:04 < mutante> and... [02:04:54] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-IRC-RC-Server, and 2 others: Remove the "HTTPS to HTTP" url filter in the IRC feed - https://phabricator.wikimedia.org/T122933#2258586 (10Dzahn) 05Open>03Resolved a:03Dzahn claiming resolved, but _please_ confirm [02:27:06] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-IRC-RC-Server, and 2 others: Remove the "HTTPS to HTTP" url filter in the IRC feed - https://phabricator.wikimedia.org/T122933#2258641 (10Dzahn) @Johan the change happened now, late but still on May 2nd as announced [08:19:00] 10Traffic, 06Operations: Evaluate Apache Traffic Server - https://phabricator.wikimedia.org/T96853#1227616 (10MoritzMuehlenhoff) Is that still planned now that we're moving to varnish 4? [10:09:42] 07Varnish, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2259398 (10Lydia_Pintscher) [11:07:22] 10Traffic, 06Operations: Evaluate Apache Traffic Server - https://phabricator.wikimedia.org/T96853#2259602 (10BBlack) Yes. We decided the varnish 4 migration was the next necessary practical step regardless, but I think at least evaluating other alternatives (which are more open-source-friendly, privacy-frien... [11:08:42] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 3 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#2259603 (10Lydia_Pintscher) So when I look at http://graphite.wikimedia.org/render/?width=586&height=308&_salt... [11:19:50] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 3 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#2259612 (10BBlack) I really don't think it's specifically Wikidata-related either at this point. Wikidata mig... [11:26:37] 07Varnish, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2259636 (10BBlack) Do you know if some normal traffic is affected, such that we'd know a start date for a recent change in behavior? Or is it suspected that it was always this way? I'... [12:27:50] 10Traffic, 10MobileFrontend, 06Operations, 07Regression, 07user-notice: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2072828 (10hashar) https://gerrit.wikimedia.org/r... [12:35:32] 07Varnish, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2259823 (10hoo) >>! In T133866#2259636, @BBlack wrote: > Do you know if some normal traffic is affected, such that we'd know a start date for a recent change in behavior? Or is it susp... [15:08:37] 07Varnish, 10Wikidata: Varnish seems to sometimes mangle uncompressed API results - https://phabricator.wikimedia.org/T133866#2260314 (10BBlack) Just jotting down the things I know so far from investigating this morning. I still don't have a good answer yet. Based on just the test URL, debugging it extensive... [16:10:22] bblack: openssl 1.0.2h is working fine in my tests on multatuli, can I install it on cp1008 for some more tests (before uploading to carbon)? [16:10:35] moritzm: please do [16:10:42] ok [16:21:43] bblack: pinkunicorn was the one going against cp1008, right? seems fine, I'll upload to carbon [16:22:54] yes, pinkunicorn.wikimedia.org in e.g. ssllabs [16:25:25] ok, thanks [16:30:55] uploaded to carbon [16:31:50] thanks, will install + restart nginx on caches shortly [16:38:33] ok, thanks! I'll test drive the Ubuntu update for a bit in labs and start upgrading the other systems tomorrow morning [16:58:23] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2260824 (10JAllemandou) [17:29:04] 07HTTPS, 10Traffic, 10OTRS, 06Operations, 13Patch-For-Review: ssl certificate replacement: ticket.wikimedia.org (expires 2016-02-16) - https://phabricator.wikimedia.org/T122320#2260936 (10RobH) [17:29:06] 07HTTPS, 10Traffic, 06Operations: ssl certificate replacement: tendril.wikimedia.org (expires 2016-02-15) - https://phabricator.wikimedia.org/T122319#2260939 (10RobH) [17:29:09] 07HTTPS, 10Traffic, 06Operations: acquire SSL certificate for w.wiki - https://phabricator.wikimedia.org/T91612#2260945 (10RobH) [17:29:12] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Set up "w.wiki" domain for usage with UrlShortener - https://phabricator.wikimedia.org/T108649#2260944 (10RobH) [17:29:38] 07HTTPS, 10Traffic, 06Operations: SSL cert needed for benefactorevents.wikimedia.org - https://phabricator.wikimedia.org/T115028#2260969 (10RobH) [17:49:27] 10Traffic, 06Mobile-Apps, 06Operations: alias /apple-app-site-association and /.well-known/apple-app-site-association - https://phabricator.wikimedia.org/T130647#2261033 (10fgiunchedi) p:05Low>03High [17:50:58] 10Traffic, 06Mobile-Apps, 06Operations: alias /apple-app-site-association and /.well-known/apple-app-site-association - https://phabricator.wikimedia.org/T130647#2141611 (10fgiunchedi) this is back and doubled our rate of 404s starting 16:27 UTC [17:54:51] 10Traffic, 06DC-Ops, 06Operations, 10ops-codfw: lvs2002 Embedded Flash/SD-CARD iLO errors - https://phabricator.wikimedia.org/T126321#2261045 (10BBlack) We'll want to coordinate the downtime on this to minimize impact, by shutting down pybal softly on lvs2002 before shutting down the OS. [22:22:11] 10Traffic, 06Operations: confctl: give regexen more freedom - https://phabricator.wikimedia.org/T134323#2262017 (10BBlack) [22:23:30] 10Traffic, 06Operations: confctl select needs a -y flag? - https://phabricator.wikimedia.org/T134324#2262031 (10BBlack)