[01:47:46] it impacts performance too, since HTTP/2 requires TLS 1.2 [01:48:13] and as you would expect there is a drop in HTTP/2 stats coinciding with the rise of clients using TLS 1.1 [04:07:15] yeah there are all kinds of correlations to that event in different places [06:16:34] <_joe_> morning [06:34:01] mourning [08:04:39] 10Wikimedia-Apache-configuration, 06Operations: Apache mod_status metrics only available in ganglia - https://phabricator.wikimedia.org/T141424#2498147 (10elukey) [08:04:47] 10Wikimedia-Apache-configuration, 06Operations: Apache mod_status metrics only available in ganglia - https://phabricator.wikimedia.org/T141424#2498160 (10elukey) p:05Triage>03Low [09:17:53] hashar: the .deb autobuilding thing sounds really cool! [09:18:27] with lintian too [09:18:47] ema: yeah kind of [09:18:56] the piuparts step is broken though [09:19:02] it runs in a debootstrap .tar.gz [09:19:16] which does not recognizes the -wikimedia flavors [09:19:50] so if you get a package build for jessie-wikimedia (with jessie-backports , our random thirdparty packages and random back ports), piuparts might entirely fail [09:19:57] but yeah lintian is nice :) [09:20:18] piuparts for varnish4 reports that after purge a few systemd related service files are left behind [09:20:29] some symlinks iirc. That might be a bug in dh_systemd [09:21:02] if you have some .deb packages in Gerrit, you can comment 'recheck' on the latest merged change or an open change and that will trigger the Jenkins job and report back to Gerrit [09:22:18] really? [09:22:25] nice :) [09:24:10] nice! [09:25:02] I don't know how to feel about the new Gerrit UI [09:28:48] hashar: debian-glue :) [09:29:25] the Gerrit UI it takes a bit of time to adjust. In previous Gerrit version (2.8) it was available as an option and switched to it months ago [09:29:30] that helped with the transition [09:29:37] after a couple weeks you will have adjusted :} [09:29:41] I am not missing the old one [09:29:55] and debian glue is http://jenkins-debian-glue.org/ :D [09:30:15] written by a DD and he even pushed it to the Debian project after I have asked for it \O/ [09:30:35] yeah I also don't miss the old UI [09:31:05] somehow this one is prettier to the eye, but the UX seems fundamentally unchanged [09:31:19] pick a random git repo. Run generate-git-snapshot && build-and-provide-package && lintian-junit-report [09:31:21] done :} [09:31:34] that reduce the whole crazy toolchain to just three commands :D [09:32:30] in my bash I have a "gerrit" function: ssh -p 29418 hashar@$host "gerrit $ARGS" [09:32:40] and an a "+2" alias [09:32:56] that magically Code-Review +2 on the change,patch passed as argument [09:33:40] I am going on vacations end of week so can't really baby sit the debian-glue issue [09:33:48] but if tasks get filled I will review them when I am back :} [09:34:15] https://integration.wikimedia.org/ci/job/debian-glue/258/tapTestReport/piuparts.tap-44/ [09:34:19] not ok 44 WARN: Broken symlinks: /etc/systemd/system/multi-user.target.wants/diamond.service -> /lib/systemd/system/diamond.service [09:34:43] was for python-diamond . Looks like dh_systemd miss some rm in its purge [09:35:16] hashar: yeah I just triggered the diamond build [09:40:12] going to trigger it on the 70 changes that had not it run yet :D [09:40:12] gerrit query --current-patch-set 'is:open NOT label:verified=2,jenkins-bot project:^operations/debs/.*'|egrep '(ref|project):' [09:40:16] magic [09:41:16] mmh I don't understand the broken symlinks part. Purged diamond and python-diamond on cp1008, the multi-user target symlink is gone [09:41:30] perhaps it's a false positive [09:44:28] yeah :( [09:47:06] ema: zuul is busy https://integration.wikimedia.org/zuul/ :D [09:48:13] hashar: how about an ops session on CI stuff? [09:48:50] integration.w.o looks great but I have no idea what I'm looking at really [09:49:34] I mean, the docs on wikitech are very clear [09:49:41] but still :) [09:54:55] which doc ? [09:55:28] https://integration.wikimedia.org/zuul/ shows changes being processed by our "CI workflow engine" ™ named Zuul [09:56:00] it basically listen for events from Gerrit (eg: new patchset created #1 on change 12345 against repo puppet.git) [09:56:14] then enqueue that events, process it to figure out which jobs to run in Jenkins [09:56:29] and the status page represent its internal state. Namely changes and jobs being run for them [09:56:56] presenting CI options for ops is definitely a good idea. I thought about debian-glue and rspec for puppet [09:57:01] both needs to be polished a bit more though [10:00:56] hashar: https://www.mediawiki.org/wiki/Continuous_integration and friends [10:03:10] ah yeah there is a bunch of docs :} [14:55:40] 10netops, 06Operations, 10ops-eqiad: Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones - https://phabricator.wikimedia.org/T140765#2498949 (10Cmjohnson) @faidon yes, I did not swap cr1's fan tray but will swap out w/a new one. @mark no, we did not get new air filters [14:57:29] 10netops, 06Operations, 10ops-eqiad: Replace cr1/2-eqiad PSUs/fantrays with high-capacity ones - https://phabricator.wikimedia.org/T140765#2498953 (10mark) Probably worth buying some new ones now, and perhaps some new REs... [14:58:35] so looking at our other crappy ciphers in the non-forward-secret list... we know why des-cbc3-sha is there (IE8-on-XP). The aes128 options that use sha256 and/or gcm have tiny stats and are relatively-irrelevant. and then there's AES128-SHA, which has a decent population, but we don't know of any popular things that should use it... [14:58:54] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2498955 (10Nuria) >For experiments on "readers", we need to think carefully about how to minimize this problem and talk to other organizations > that do AB testing without a use... [14:58:55] (well when I say "decent population", I mean on about the same order as IE8-XP, well under 1%) [14:59:37] looking at some live logging, as it turns out we do see small amounts of traffic from expected ancient devices, e.g. ancient feature phones and set-top boxes [15:00:08] but an appreciable amount of it all comes from zscaler's cloud service, and from hosts in others' networks that are also clearly zscaler appliances, or the very similar ProxySG appliance [15:00:50] one particularly awesome example is 179.21.223.91.in-addr.arpa domain name pointer zscalerbe.nc3a.nato.int [15:01:01] apparently NATO uses a borked zscaler appliance, too :P [15:01:36] their cloud service and appliance appears to (by default? or some option they turn on for "performance"?) downgrade outbound connection security to AES128-SHA after proxying it [15:01:49] the clients behind it are often modern, e.g. Chrome/51 [15:03:24] those zscalers also only use TLSv1.0, too [15:03:46] 10Traffic, 06Operations: Age header reset to 0 after 24 hours on varnish frontends - https://phabricator.wikimedia.org/T141373#2498969 (10ema) https://phabricator.wikimedia.org/P3585 is a VTC test case with two varnishes targeting varnish 3 and default VCL except for a 5s TTL cap on the varnish frontend. The t... [15:05:55] bblack: isn't ProxySG the Bluecoat appliance that wraps around Squid to act as an enterprise proxy? [15:06:34] each probably hide a lot of users (an enterprise) and I would not be surprised if they had lot of old versions out in the field [15:06:53] in any big org, it is typically a mess / long process to upgrade a system :/ [15:09:25] I have basically zero sympathy for them, though. [15:09:50] if you're going to proxy and filter https from corporate clients, you have a responsibility to keep that system up to date [15:10:29] it's creating a SPOF for the org where breaking this one ancient crappy proxy device is breaking security for all the users in the org, and if it's not being maintained it's probably hackable, and it's definitely downgrading security. [15:11:06] the whole privacy issue aside, I'm sure there's affected people using office computers to legitimately access the outside world on their lunch break and don't realize the implications of the downgrade [15:11:22] after all, they're running Chrome, it should be pretty secure to go access your bank website with from the office, right? :P [15:13:42] sorry got disconnected [15:13:49] semi-related to all the above, apparently RHEL/CentOS 7 just shipper a Firefox ESR release that broke AEAD heh [15:13:54] https://bugzilla.redhat.com/show_bug.cgi?id=1343202 [15:14:17] something to do with redhat's NSS library. so it can't do AEAD/TLSv1.2 stuff properly, and thus also breaks HTTP/2. to be fixed in a future update, apparently. [15:14:49] I have seen case of the enterprise having its own cert that is installed on all workstation. Browser then establish a connection on the enterprise proxy that terminate the connection and resign it with the enterprise cert (mitm). So they can do blacklist/whitelist of traffic, virus check etc [15:16:41] yeah, that's what all of these are (zscaler, bluecoat, proxysg, etc) [15:16:56] I donno what you missed above, but to repeat MHO: [15:17:01] 15:09 < bblack> I have basically zero sympathy for them, though. [15:17:01] 15:09 < bblack> if you're going to proxy and filter https from corporate clients, you have a responsibility to keep that system up to date [15:17:04] 15:10 < bblack> it's creating a SPOF for the org where breaking this one ancient crappy proxy device is breaking security for all the users in the org, and if it's not being maintained it's probably hackable, and it's definitely downgrading security. [15:17:08] 15:11 < bblack> the whole privacy issue aside, I'm sure there's affected people using office computers to legitimately access the outside world on their lunch break and don't realize the implications of the downgrade [15:17:12] 15:11 < bblack> after all, they're running Chrome, it should be pretty secure to go access your bank website with from the office, right? :P [15:17:27] thx! [15:17:35] yeah I agree with you :-} [15:17:46] we can't really support all the legacy/outdated systems out there [15:18:11] or that ends up with an egg and chicken problem. While we still support them there is no incentive to upgrade [15:18:20] if the proxy was doing modern crypto outbound, and it's well-maintained, then you're down to just the privacy argument. and honestly, they own the client machines, they probably have the right to invade privacy there, although there's ??? about private activities done on break time with work computer. [15:18:56] law and common law would know :-} [15:19:11] typically in big corp you sign a code of conduct chart related to internet access [15:19:23] yeah [15:19:26] stating that the company can and will look at everything you do at work [15:19:35] as permitted by law [15:19:46] but do people understand that means this appliance is decrypting transactions with their bank on their lunch break? [15:20:05] typically the law would prevent the company to look at content that is obviously personal. Such as an email with subject [Private] [15:20:07] or online banking [15:20:11] that would need a warrant [15:20:21] at least at home, that security relies on a clean uncompromised client device/browser, and the bank's own security work. [15:20:28] yeah [15:20:35] but now it also relies on "this proxy machine the IT guys supposedly maintain and secure" [15:20:40] then at the central proxy you can establish exception to not mitm some sites (eg banks) [15:20:56] even if they don't intentionally look due to privacy, they're adding new security risks that the user+bank aren't aware of. [15:21:29] yeah but "some sites" - I doubt anyone has an exhaustive list of every bank website. there's probably hundreds of local banks just here in Houston that nobody has ever heard of. [15:21:29] most folks have no idea that their company sysadmin can read all their emails :/ [15:21:37] and non-bank things that are equally important [15:21:43] yup agreed [15:21:58] assume good faith :-} [15:22:24] assume anything maintained by a typical corporate IT department will get hacked [15:22:47] to be fair, assume most end-user devices will get hacked [15:23:05] but the corporate proxy represents a bigger target to get more users in one go [15:27:11] when I look at random traffic logs for any of the AES128 non-forward-secret crappy variants. over and over I keep finding references to these proxies. either they belong to cloud proxy services, or reverse dns has an obvious proxy name in it. [15:27:38] and that's not even accounting for all the ones I can't be sure of because whois/revdns just gives some random company name, but it's clear what's going on because you see Chrome/51 behind the crappy cipher [15:28:10] the legitimate ancient clients like feature phones and set-top boxes are definitely a small minority of that traffic [15:29:29] the legit ones are UAs like: [15:29:31] NokiaX2-02/2.0 (11.84) Profile/MIDP-2.1 Configuration/CLDC-1.1 [15:29:38] Mozilla/5.0 (PLAYSTATION 3 4.80) AppleWebKit/531.22.8 (KHTML, like Gecko) [15:29:48] SAMSUNG-GT-S5260/S5260XXKG2 SHP/VPP/R5 Dolfin/2.0 NexPlayer/3.0 SMM-MMS/1.2.0 profile/MIDP-2.1 configuration/CLDC-1.1 OPN-B [15:30:14] Opera/9.80 (S60; SymbOS; Opera Mobi/SYB-1204232258; U; ru) Presto/2.10.254 Version/12.00 [15:30:28] ah so at least you have a way to map ciphers to user-agent! [15:31:14] only in manual logs, not in big aggregated logs/graphs we have history on [15:32:33] I don't know if I could justify killing the legit clients at this point in time anyways. there's not many of them in the global view, but in some countries those old feature phones might have more sizeable userbase still. [15:34:02] PS3 and such (set-top game consoles) probably aren't worth caring about. if you are in the economic class that buys game consoles, you can afford a phone with software written in the past decade :P [15:41:00] that Nokia X2 is an interesting case. it was actually released in 2014 (and shortly after, the whole produce line canceled). It's based on Android 4.3, but has custom nokia stuff on top, which severely downgrades TLS security from what basic Android 4.3 already offered :/ [15:45:17] anyways, all of that's about justifying keeping AES128-SHA. The other two (AES128-SHA256, AES128-GCM-SHA256), I've yet to log a request anywhere that wasn't pretty obviously a downgrading proxy (because they're all coming from UAs with IE10, IE11, recent Chrome, or the UA string actually names the proxy) [15:54:08] all that aside, we still have this strange TLSv1.1 anomaly [15:54:39] which is that, starting about a week ago, clients negotiating TLSv1.1 and a certain middling cipher choice jumped up dramatically and have stayed [15:55:06] and the culprits all seem to have very similar UA strings: it's a specific rev of Chrome 41 on various windows versions [15:55:09] 84 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36 [15:55:17] 41 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36 [15:55:20] 95 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36 [15:55:37] in esams at least, a lot of them are russian, too [15:55:54] Chrome/41 definitely supports better crypto than what these UAs end up negotiating with us [15:57:59] I think it's some kind of bug, though, and the stats about russia could be a red herring to do with os/browser popularity there and general high population there [16:00:27] bblack: catching up. Sorry forgot to notify I was having an audio :( [16:01:33] I'm mostly just talking out loud randomly, no need to pay attention [16:01:54] Chrome/41, notably, was the first release to support HTTP/2 at all [16:02:12] well your random thoughts are way clever than even my most elaborated speech [16:02:47] and I do see some (minority) of Chrome/41 on windows negotiating strong tlsv1.2 + AEAD like it should. but most don't. the few that do, also don't get HTTP/2, and don't get a resumed session. [16:03:03] all the ones doing crappy TLS are resuming a session (and still no HTTP/2) [16:03:41] arent all those logs injected in the Analytics cluster ? [16:03:57] I mean would it be feasible to have nginx inject a custom header mentioning the TLS version [16:03:59] well not all [16:04:07] some of the crappy ones aren't resuming sessions [16:04:15] then craft dashboard / reports from hadoop? [16:04:16] but none of the good ones I've seen yet are resuming a session [16:04:41] hashar: we do send TLS version and cipher (not to analytics, but to graphite/grafana) [16:05:20] we don't log it with everything else in X-Analytics for analytics cluster, though, which is why we can't ask them to correlate it up with UAs, regions, etc [16:05:58] could it be added? [16:06:04] yes, lots of things could :) [16:06:06] but maybe that is too much work [16:06:31] it's not a ton of work, but if we added everything we could ever possibly want to see there... there's only so much data we want to push on every single request [16:07:05] well that could be logged just for an hour or so in the interest of having a better understanding [16:07:07] generally this doesn't rise to the level that we need analytics. we know what browsers do, and we can sample, etc... [16:07:19] but this week, sure, I wish it was there :) [16:07:19] ok ok :-} [16:07:35] another idea I had for the proxy that have outdated cyphers/tls [16:07:45] and my understanding of analytics cluster data is we can't just turn on random things for an hour, we have to coordinate with them on creating new data fields in hadoop, etc [16:08:15] a nasty thing we could do is setup a Nginx that no more have the outdated cypher then use geodns to point the outdated proxy to that nginx. They would then notice they have an issue :) [16:08:44] or we could block crappy ciphers when we positively ID a modern browser UA string with it [16:09:17] anyways, all of this will get better with user notification I think, which is something that's been sitting on the back burner for far too long [16:09:18] regardless of the OS ? [16:09:46] the general idea that we hook up our cipher data headers to CentralNotice JS somehow and do a site popup about "hey your browser is ancient and insecure, please upgrade..." [16:09:53] ohhh [16:10:05] we had that discussions years ago about IE6.0 [16:10:12] I think that would do a lot of good on this front [16:10:28] and in the case of corp proxies, maybe some users complain to IT and get the proxy box updated or something [16:11:12] then after the banner been running for a few months, stop supporting the old cyphers [16:11:22] sounds neat [16:11:25] well we have some middle-steps we can take there, too [16:11:54] like, we could block (at the MediaWiki level) authenticated user logins with escalated wiki-privileges if they're using a crap cipher == bad browser/proxy. [16:12:03] and then upgrade that eventually to all authenticated wiki logins [16:12:12] and then consider dropping the cipher which then affects all access [16:13:17] step one kind of assumes that CN JS works on all these old devices of course, heh [16:13:27] it probably won't even run on a lot of ancient feature phone browsers and such [16:13:51] but it would run on modern browsers stuck behind lame proxies, and I bet it would run on IE8/XP too (which is the primary crappy real browser we care about) [16:16:53] the really confounding thing about this Chrome/41 issue is how it suddenly started happening a week ago. I can understand there being some notable base of un-upgraded Chrome in the world, but they didn't all just get installed a week ago. [16:17:14] I wonder if there's something special in it that's date-sensitive [16:18:09] it was the first http/2 -enabled release. maybe they put in a date killswitch to disable http/2 a year later in case the first releases had bad bugs that affected http/2 interop? and somehow that ends up screwing cipher selection too because of buggy interactions about http/2 requiring tlsv1.2 + AEAD? [16:18:31] no idea, just random conspiracy theory [16:29:17] bblack: worth asking the CIA^H^H^HChromium folks? [16:29:24] :) [16:29:54] is that chromium 41 regardless of the OS ? [16:29:59] right now I'm syncing down the chromium git repo. I figure there might be answers in there if I look back around the time of Chrome/41. But it's much easier to dig on a local clone than in google search. [16:30:14] all the logs I see are on windows [16:30:30] I think other platforms don't commonly have chrome/41 installed. I don't know why it's lingeringly-popular on windows. [16:31:08] the week-ago date thing could be all sorts of crazy things [16:31:48] e.g. maybe chrome/41 has always been broken this way with us, but a week ago we pushed some small change to site JS that triggers a completely-unrelated bug in Chrome/41 and makes it spam requests at our servers, thus artificially inflating their request volume and statistics impact. [16:31:57] I could invent crazy stories like that all day heh [16:32:45] :) [16:32:47] ahhhhh that is hard [16:33:45] on unrelated notes windows 10 went from 12 to 14% over the last two weeks [16:33:51] and is probably at 16% traffic share now [16:33:56] based on https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os :D [16:34:09] then why would someone use chromium 41 on windows 10.. ;} [16:34:11] win10 has generally been positive for our TLS stats, even if many consider it a negative for users on the whole. [16:34:25] because all previous versions of windows and/or IE have weaker TLS [16:35:02] $ git clone https://chromium.googlesource.com/chromium/src [16:35:03] Cloning into 'src'... [16:35:04] remote: Sending approximately 5.69 GiB ... [16:35:16] ^ I don't think I've ever seen such a huge git repo [16:35:35] must have a bunch of static assets throughout history, icons for the UI and such? [16:36:05] and a few copy of android VM for testing ? :D [16:36:09] who kows [16:36:14] *knows, too [16:36:15] https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-browser/browser-family-timeseries shows a bump of Chromium traffic! [16:36:39] so maybe your crazy scenario (chromium surge of traffic trigger the raise of bad ciphers) has some standing ground [16:37:05] regardless. I have kept you busy way too long already [16:37:34] I've kept myself on this way too long already heh [16:40:54] another random theory about the date: maybe Windows pushed some security update to a crypto library a week ago, which triggered a bug for Chrome/41-on-Windows installs [16:41:09] it seems to affect all windows versions back to ~Vista, though. [16:42:41] there was a Win10 update ~ 1 week before our stats started changing, which contains this in the short public log: [16:42:55] Improved reliability of Windows Media Player, Internet Explorer 11, Windows Explorer, Miracast, and Windows kernel. [16:43:01] Fixed additional issues in .NET, Windows Kernel, Windows Update, authentication, revised daylight saving time, support for PDF files, Bluetooth, Microsoft Edge, Internet Explorer 11, networking, and Wi-Fi connectivity. [16:43:05] Security updates for Microsoft Edge, Internet Explorer 11, Kernel Mode Drivers, Windows Kernel, .NET Framework, Windows Secure Kernel Mode, and Microsoft Print Spooler. [16:45:00] July 21 (a couple days after our stats started changing slowly), Windows 8 had an update that included: [16:45:03] Addressed issue in Microsoft Secure Channel (SChannel) that sometimes causes Transport Layer Security (TLS) 1.2 connections to fail depending on whether the root certificate is configured as part of the certificate chain for server authentication. [16:45:18] (which could be a small backport of something included in the Win10 update for all I know) [16:45:47] same update also has: [16:45:49] Improved support in Microsoft Cryptographic Application Programming Interface (CryptoAPI) to help identify websites that use Secure Hash Algorithm 1 (SHA-1). [16:46:32] Win7 had those same two changelogs as the only thing in its update on July 21 [16:46:47] looks likely / good candidate! [16:47:04] and it's there again in Windows Server 2012, but in an update on Jul 19 [16:47:32] and I'd be willing to bet Chrome (at least back at 41, if not still today) uses MS's SChannel/CryptoAPI stuff on windows [16:50:03] anyway time for me to disappear! [16:50:18] bblack: thx for the above story / investigation. I have learned a lot :} [16:56:25] bye! [16:57:46] 10Traffic, 10Continuous-Integration-Infrastructure, 06Operations, 10Packaging: piuparts fail with WARN: Broken symlinks: /etc/systemd/system... - https://phabricator.wikimedia.org/T141454#2499359 (10hashar) [16:57:56] 10Traffic, 10Continuous-Integration-Infrastructure, 06Operations, 10Packaging: piuparts fail with WARN: Broken symlinks: /etc/systemd/system... - https://phabricator.wikimedia.org/T141454#2499372 (10hashar) p:05Triage>03Low [17:05:33] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2499403 (10Milimetric) >>! In T135762#2497291, @BBlack wrote: >>>! In T135762#2497082, @ellery wrote: >> As far as I can tell, the proposed method also violates the more importa... [17:07:57] could also be buggy SHA-1 cutoff [17:08:44] somewhere around now-ish, various entities (including MS) are supposedly increasingly turning on switches to not trust SHA-1 certs. we don't fall afoul of that, but maybe schannel+chrome/41 has a bug that considers the root cert when it shouldn't and this affects everything somehow [17:08:51] it's a long shot [17:39:17] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2499531 (10mark) p:05High>03Normal [18:30:28] 10netops, 06Operations, 10fundraising-tech-ops, 10ops-eqiad: put pfw1- ge-2/0/11 in the 'fundraising' vlan for new host frqueue1001 - https://phabricator.wikimedia.org/T140991#2499657 (10Jgreen) 05Open>03Resolved a:03Jgreen [18:32:14] 10netops, 06Operations, 10fundraising-tech-ops, 10ops-eqiad: put pfw1- ge-2/0/11 in the 'fundraising' vlan for new host frqueue1001 - https://phabricator.wikimedia.org/T140991#2483556 (10Jgreen) [19:06:02] 07HTTPS, 10Traffic, 06Operations, 07Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2499805 (10BBlack) [19:06:04] 10Traffic, 06Operations: Clean up DNS/redirects for TLS - https://phabricator.wikimedia.org/T102824#2499802 (10BBlack) 05Open>03Resolved a:03BBlack The title wording is unclear, which is perhaps why this ticket lingered open. However, from the subtasks it's clear this was about exception cases within ou... [19:13:23] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: store.wikimedia.org HTTPS issues - https://phabricator.wikimedia.org/T128559#2499855 (10BBlack) [19:13:48] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: store.wikimedia.org HTTPS issues - https://phabricator.wikimedia.org/T128559#2078914 (10BBlack) Fixed up title and description to match what we need out of it today, rather than old audit data and/or old targets. [19:15:55] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2499882 (10BBlack) To re-iterate current status: the main thing missing here in the present is that the STS header is insufficient. It should be `strict-transport-security:max-age=3153... [19:16:56] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2499902 (10BBlack) [19:16:58] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2499901 (10BBlack) [19:18:07] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2201391 (10BBlack) Sub-tasks updated to be in sync with latest audit data. Primary issues here... [19:40:36] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Firefox: Secure connection failed when attempting to send POST request using HTTP/2 (if connection has been idle for a certain time) - https://phabricator.wikimedia.org/T134869#2499986 (10BBlack) 05Open>03Resolved a:03BBlack I think either this has... [19:44:45] 07HTTPS, 10Traffic, 06Operations, 07Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2500003 (10BBlack) [20:11:35] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2500088 (10Nemo_bis) Thanks mark for clarifying the subject. [20:45:36] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#2500130 (10BBlack) Anyone have one to test on? [20:47:09] 07HTTPS, 10Traffic, 10ArchCom-RfC, 10MediaWiki-Configuration, and 2 others: RFC: MediaWiki HTTPS policy - https://phabricator.wikimedia.org/T75953#2500131 (10BBlack) [20:49:44] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#1444194 (10Paladox) I have an xbox 360, I will be near one on the weekend then I will have to search for where I put it. :). [20:50:42] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-General-or-Unknown, 07JavaScript: Use Upgrade Insecure Requests on Wikimedia - https://phabricator.wikimedia.org/T101002#1326308 (10BBlack) >>! In T101002#1326438, @Krinkle wrote: > This header currently results in Chrome changing all HTTP requests opened fro... [20:52:28] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-General-or-Unknown, 07JavaScript: Use Upgrade Insecure Requests on Wikimedia - https://phabricator.wikimedia.org/T101002#2500147 (10BBlack) Also, we may want to merge tasks one direction or the other with T36670. Personally, I'm a fan of setting this for all... [21:00:45] 10Traffic, 06Operations, 10Wikimedia-Planet: mixed-content issues on planet.wikimedia.org - https://phabricator.wikimedia.org/T141480#2500163 (10Dzahn) [21:02:28] 10Traffic, 06Operations, 10Wikimedia-Planet: mixed-content issues on planet.wikimedia.org - https://phabricator.wikimedia.org/T141480#2500184 (10Dzahn) [21:32:42] 07HTTPS, 10Traffic, 06Operations, 07Tracking: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2500330 (10BBlack) [22:08:46] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#2500368 (10Paladox) Testing using the developer tools on internet explorer I set the user agent string to xbox 360 internet explorer and Wik... [22:12:03] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#1444194 (10Platonides) @Paladox did you test it with an actual xbox 360? Or did you just spoof the UA on a Windows system? [22:12:50] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#2500385 (10Paladox) >>! In T105455#2500369, @Platonides wrote: > @Paladox did you test it with an actual xbox 360? Or did you just spoof the... [22:15:33] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Internet-Explorer: Xbox 360 Internet Explorer unable to view Wikipedia - https://phabricator.wikimedia.org/T105455#2500408 (10BBlack) This kind of testing needs the real thing unfortunately. The UA string doesn't have much impact here.