[04:41:59] 07HTTPS, 10Traffic, 06Operations: HTTPS error on status.wikimedia.org (watchmouse certificate mismatch) - https://phabricator.wikimedia.org/T131017#2202093 (10Pokefan95) p:05Triage>03Normal [05:09:04] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202098 (10Dzahn) [05:11:22] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202114 (10Pokefan95) p:05Triage>03Normal [05:12:06] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2202115 (10Pokefan95) p:05Triage>03Normal [05:14:08] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202116 (10Dzahn) looking at the config i already see: 13 Header always set Strict-Transport-Security "max-age=604800" isn't it already enabled? [05:20:37] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202117 (10Dzahn) 05Open>03Invalid already resolved/invalid it's enabled and *.planet. uses use standard cache cluster termination, it's misc-web, besides having a separate wildcard cert,... [05:22:26] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202119 (10Dzahn) @Pokefan do me a favor and update https://wikitech.wikimedia.org/wiki/HTTPS/domains ? can't login on wikitech due to lack of second factor [05:29:06] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202121 (10Pokefan95) @Dhann: Doing... [05:29:23] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202123 (10Dzahn) [05:29:25] 07HTTPS, 10Traffic, 06Operations: Enable HSTS on Wikimedia sites - https://phabricator.wikimedia.org/T40516#2202122 (10Dzahn) [05:31:00] 07HTTPS, 10Traffic, 06Operations: Enable HSTS on Wikimedia sites - https://phabricator.wikimedia.org/T40516#1146945 (10Dzahn) [05:31:02] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202098 (10Dzahn) 05Invalid>03Resolved @Pokefan95 thank you , then there was actually something to resolve, heh [05:35:13] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202129 (10Dzahn) the change that enabled this was https://gerrit.wikimedia.org/r/#/c/253758/ on 2015-11-18 [05:36:41] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202130 (10Pokefan95) @Dhann: For now, I just changed it from "No" to "Yes". What is the duration of the HSTS? [05:39:10] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202131 (10Dzahn) it's max-age=31536000 (from https://www.ssllabs.com/ssltest/analyze.html?d=es.planet.wikimedia.org&s=208.80.153.248) so that means [[ https://duckduckgo.com/?q=31536000+seco... [05:39:30] 07HTTPS, 10Traffic, 06Operations: enable HSTS on *.planet.wikimedia.org - https://phabricator.wikimedia.org/T132543#2202132 (10Pokefan95) Ah, ok, thanks [06:46:54] 07HTTPS, 10Traffic, 06Operations: Preload HSTS for select hostnames within wikimedia.org - https://phabricator.wikimedia.org/T111967#2202184 (10BBlack) Yeah, I'm in the process of enumerating those.... [07:26:22] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2202245 (10BBlack) ========= //Audit Data// ========= Methodology: ----------------- The starting point is our raw D... [07:34:44] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2202250 (10BBlack) While we should fix all of these issues in the long term (they should all be 301->https on the same... [07:43:16] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2202254 (10BBlack) As for the rest of the work, IMHO we should re-purpose the wiki tracking page at https://wikitech.w... [08:53:31] 07HTTPS, 10Traffic, 06Operations: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2202415 (10Chmarkine) >>! In T132521#2202254, @BBlack wrote: > As for the rest of the work, IMHO we should re-purpose... [09:40:34] 10Traffic, 07Varnish, 06Operations, 13Patch-For-Review: Add icinga monitoring for varnish statistics daemons - https://phabricator.wikimedia.org/T131760#2202515 (10ema) 05Open>03Resolved [09:44:47] 07HTTPS, 10Traffic, 06Operations, 07Graphite: HTTPS redirects for graphite.wikimedia.org - https://phabricator.wikimedia.org/T132461#2202522 (10fgiunchedi) the most critical I can think of is `check_graphite` which already supports https (not sure about following redirects) ``` $ /usr/lib/nagios/plugins/c... [10:17:15] 10Traffic, 07Varnish, 10Analytics, 06Operations, 13Patch-For-Review: varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd - https://phabricator.wikimedia.org/T132430#2202604 (10ema) One more: Feb 11 09:17:47 cp4010 varnishstatsd[2820]: Traceback (most recent call la... [10:27:34] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: varnishkafka logrotate cronspam - https://phabricator.wikimedia.org/T129344#2102820 (10elukey) Installed on maps hosts by @ema, we will rollout the new version everywhere along wiht the Varnish 4 upgrade. [10:42:06] 10Traffic, 07Varnish, 10Analytics, 06Operations, 13Patch-For-Review: varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd - https://phabricator.wikimedia.org/T132430#2202664 (10ema) And this one: Mar 18 12:41:35 cp4010 varnishstatsd[10396]: Traceback (most recent ca... [12:30:47] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments-General: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2202895 (10SindyM3) @Akoopal Thank you! I will contact server admin. [13:57:21] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2203163 (10BBlack) Notable: there's an ongoing report of 1.9.14 causing an HTTP/2 proto error in Chrome. We may need to be wary and stick with .13 or wait for .15: http://mailman.ngin... [14:37:24] 07HTTPS, 10Traffic, 06Operations, 10Pybal, 13Patch-For-Review: HTTPS redirects for config-master.wikimedia.org - https://phabricator.wikimedia.org/T132459#2203303 (10BBlack) Re: rolematcher - the only real host I could trace it to in puppetization was fluorine. However, post-merge the update did not get... [14:42:37] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2203332 (10BBlack) [14:42:39] 07HTTPS, 10Traffic, 06Operations, 10Pybal, 13Patch-For-Review: HTTPS redirects for config-master.wikimedia.org - https://phabricator.wikimedia.org/T132459#2203330 (10BBlack) 05Open>03Resolved a:03BBlack [14:47:16] 07HTTPS, 10Traffic, 06Operations: HTTPS redirects for transparency.wikimedia.org - https://phabricator.wikimedia.org/T132464#2203360 (10BBlack) I don't see any mixed content in simple checks, and it seems to not use proto-absolute URLs in general. Since this site is clearly for human consumption, I'll proba... [14:48:00] btw, not sure if you saw the mails to noc@ [14:48:23] eqiad-esams packet loss increased and I've opened a ticket with GTT's NOC [14:48:50] it seems okayish now, so no reason to drain esams IMO [14:49:00] 07HTTPS, 10Traffic, 10Analytics-Cluster, 06Operations: HTTPS redirects for stats.wikimedia.org - https://phabricator.wikimedia.org/T132465#2203363 (10BBlack) I don't see any mixed content in simple checks (and we checked/fixed that in a much earlier ticket: (T93702). Since this site is clearly for human c... [14:49:09] but FYI, in case this gets worse and/or you start seeing 500s from esams varnishes [14:51:27] 07HTTPS, 10Traffic, 06Operations, 06Release-Engineering-Team, 05Gitblit-Deprecate: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2203375 (10BBlack) As gitblit is on the chopping block for deprecation anyways, my inclination is to go ahead and enable HTTPS for this soon... [14:53:01] 07HTTPS, 10Traffic, 10Gitblit, 06Operations, 06Release-Engineering-Team: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2203384 (10greg) [14:55:14] paravoid: thanks! [14:55:29] 07HTTPS, 10Traffic, 10Datasets-General-or-Unknown, 06Operations, 13Patch-For-Review: HTTPS redirects for datasets.wikimedia.org - https://phabricator.wikimedia.org/T132463#2203387 (10Ottomata) datasets.wikimedia.or is hosted on stat1001, not a dataset100x host. I think HTTPS only is fine, but maybe @mil... [14:55:32] paravoid: ok :) [15:11:40] 07HTTPS, 10Traffic, 10Analytics-Cluster, 06Operations: HTTPS redirects for stats.wikimedia.org - https://phabricator.wikimedia.org/T132465#2203508 (10Ottomata) +1 should be fine to do. [15:11:52] 07HTTPS, 10Traffic, 06Operations, 10Parsoid, 13Patch-For-Review: HTTPS redirects for parsoid-tests.wikimedia.org - https://phabricator.wikimedia.org/T132462#2203509 (10BBlack) Brief discussion on mediawiki-parsoid IRC channel seems to indicate this is low risk, so going for it. [15:17:59] 07HTTPS, 10Traffic, 10Datasets-General-or-Unknown, 06Operations, 13Patch-For-Review: HTTPS redirects for datasets.wikimedia.org - https://phabricator.wikimedia.org/T132463#2203529 (10Halfak) +1 for HTTPS only being OK. [15:18:43] 07HTTPS, 10Traffic, 06Operations: HTTPS redirects for transparency.wikimedia.org - https://phabricator.wikimedia.org/T132464#2203530 (10Chmarkine) Redirect to https should be fine, since we enabled HSTS for transparency.wikimedia.org in May 2015.[1] But was there any reason that the redirect was dropped? [1... [15:28:16] 07HTTPS, 10Traffic, 06Operations: HTTPS redirects for transparency.wikimedia.org - https://phabricator.wikimedia.org/T132464#2203583 (10BBlack) @Chmarkine: not sure - those changes are still in puppet, and I've confirmed the backend server for it today (bromine) still has that config deployed as well. But i... [15:32:09] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2203620 (10BBlack) [15:32:10] 07HTTPS, 10Traffic, 06Operations, 10Parsoid, 13Patch-For-Review: HTTPS redirects for parsoid-tests.wikimedia.org - https://phabricator.wikimedia.org/T132462#2203618 (10BBlack) 05Open>03Resolved a:03BBlack [15:35:30] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2203640 (10BBlack) [15:35:43] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#1402411 (10BBlack) [15:35:45] 07HTTPS, 10Traffic, 10Analytics-Cluster, 06Operations, 13Patch-For-Review: HTTPS redirects for stats.wikimedia.org - https://phabricator.wikimedia.org/T132465#2203641 (10BBlack) 05Open>03Resolved a:03BBlack [15:44:01] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2203658 (10BBlack) [15:44:03] 07HTTPS, 10Traffic, 10Datasets-General-or-Unknown, 06Operations, 13Patch-For-Review: HTTPS redirects for datasets.wikimedia.org - https://phabricator.wikimedia.org/T132463#2203656 (10BBlack) 05Open>03Resolved a:03BBlack [15:49:56] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2203682 (10Nuria) [15:50:51] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2197243 (10Nuria) [16:05:39] 10netops, 10Analytics-Cluster, 06Operations, 10hardware-requests: setup/deploy server analytics1003/WMF4541 - https://phabricator.wikimedia.org/T130840#2203753 (10Ottomata) [16:07:27] 10Traffic, 10Analytics, 10DNS, 06Operations: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2197243 (10BBlack) If we're doing this in production, the frontend should probably be through cache_misc. I'm not sure what the backend looks like at all role/software-wise... [16:14:48] 07HTTPS, 10Traffic, 06Operations, 07Graphite, 13Patch-For-Review: HTTPS redirects for graphite.wikimedia.org - https://phabricator.wikimedia.org/T132461#2203793 (10BBlack) With the check_graphite stuff switched to HTTPS, so far neon doesn't seem to be suffering from any significant increase in overall CP... [16:31:59] 10Wikimedia-Apache-configuration: Emit app server hostname in Server response header - https://phabricator.wikimedia.org/T132599#2203862 (10ori) [16:38:56] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 4 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2203890 (10Jdlrobson) @Atsirlin can you make an update the pagebanner template to force a cache flush for the pages that use the... [16:43:02] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 4 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2203893 (10Wrh2) >>! In T121135#2203890, @Jdlrobson wrote: > @Atsirlin can you make an update the pagebanner template to force a... [16:44:41] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2203911 (10BBlack) [16:44:43] 07HTTPS, 10Traffic, 06Operations, 07Graphite, 13Patch-For-Review: HTTPS redirects for graphite.wikimedia.org - https://phabricator.wikimedia.org/T132461#2203909 (10BBlack) 05Open>03Resolved a:03BBlack [17:34:07] 07HTTPS, 10Traffic, 10Gitblit, 06Operations, 06Release-Engineering-Team: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204120 (10BBlack) In lieu of anything more-concrete to go on, I've been monitoring the varnishlog live request flow for git.wikimedia.org this mornin... [17:36:29] 10Wikimedia-Apache-configuration, 13Patch-For-Review: Emit app server hostname in Server response header - https://phabricator.wikimedia.org/T132599#2204122 (10ori) 05Open>03Resolved [17:52:58] 07HTTPS, 10Traffic, 10Gitblit, 06Operations, and 2 others: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204187 (10BBlack) 05Open>03Resolved a:03BBlack Resolving for now, although I suspect this is the most likely of the bunch to trigger some kind of legitimate co... [17:53:00] 07HTTPS, 10Traffic, 06Operations: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2204190 (10BBlack) [18:11:44] 07Varnish, 10MobileFrontend, 13Patch-For-Review, 03Reading-Web-Sprint-70-Lady-and-the-Trumps, 05WMF-deploy-2016-04-26_(1.27.0-wmf.22): Stop default redirecting Samsung Smart TVs to mobile web - https://phabricator.wikimedia.org/T127021#2204277 (10Jdlrobson) @dr0ptp4kt can you sign off? I tried forging a... [18:13:09] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 3 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2204281 (10Jdlrobson) 05Open>03stalled p:05High>03Normal [18:13:28] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 3 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#1870420 (10Jdlrobson) Seems fixed, but we'll check in our next sprint after 2 weeks have passed. [18:14:56] 07HTTPS, 10Traffic, 10Gitblit, 06Operations, and 2 others: HTTPS redirects for git.wikimedia.org - https://phabricator.wikimedia.org/T132460#2204324 (10BBlack) For the record: the oddball `Mozilla/8.0 (Windows 2008 SP32 + 3patch)` seems to be making it through the HTTPS redirect just fine in the logs. [18:21:36] 07HTTPS, 10Traffic, 06Operations: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681#2204346 (10BBlack) [18:49:26] 07HTTPS, 10Traffic, 06Operations: Enable HSTS on Wikimedia sites - https://phabricator.wikimedia.org/T40516#2204462 (10BBlack) [18:49:28] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: let all services on misc-web enforce http->https redirects - https://phabricator.wikimedia.org/T103919#2204458 (10BBlack) 05Open>03Resolved a:03BBlack [20:45:30] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2204871 (10matmarex) [20:45:54] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2204887 (10matmarex) [20:45:56] 10Wikimedia-Apache-configuration: https://test.wikipedia.org/wiki/Bug%3F?action=history doesn't show the history page, unlike https://test.wikipedia.org/w/index.php?title=Bug%3F&action=history - https://phabricator.wikimedia.org/T123276#2204886 (10matmarex) [20:46:03] 10Wikimedia-Apache-configuration: Redirect with a question mark '?' in the title treats everything following it as URL query part when updating the URL - https://phabricator.wikimedia.org/T128380#2072301 (10matmarex) [20:46:05] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2204871 (10matmarex) [21:10:22] 07Varnish, 10Beta-Cluster-Infrastructure, 10Monitoring: Monitor Varnish caches on beta cluster have two varnishd process running - https://phabricator.wikimedia.org/T75944#2204983 (10Krenair) Some reporter script running on puppet exec/cron sending data to graphite and shinken monitoring graphite? [21:34:19] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2205049 (10matmarex) Some examples courtesy of @ori, for each of the bugs mentioned above. (Something went... [21:51:29] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2205129 (10matmarex) And for comparison, similar requests from my local testing wiki, with Apache and a ve... [22:48:57] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2205393 (10BBlack) With the examples, could you be more-specific about what's broken in them? [23:27:14] 10Traffic, 06Operations, 06Zero: Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two - https://phabricator.wikimedia.org/T124482#2205468 (10BBlack) The Zero picture is clearer now from some email threads with @DFoy and @dr0ptp4kt . We're clear for this change on the Zero front already,... [23:47:45] 07Varnish, 10Wikimedia-Apache-configuration, 06Operations: Data passed to HHVM ($_SERVER variables) is a mixed bag of already-decoded and non-decoded nonsense - https://phabricator.wikimedia.org/T132629#2205523 (10matmarex) For T123276: * P2895 https://commons.wikimedia.org/w/api.php?oxrigin=https%3A%2F%2Fw... [23:58:31] 07Varnish, 10MediaWiki-Vagrant: MediaWiki-Vagrant varnish role fails to provision - https://phabricator.wikimedia.org/T132337#2205570 (10bd808) [23:59:47] 07Varnish, 10MediaWiki-Vagrant: MediaWiki-Vagrant varnish role fails to provision - https://phabricator.wikimedia.org/T132337#2194856 (10bd808) This problem may have been fixed by a recently merged patch from @Gilles: https://gerrit.wikimedia.org/r/#/c/282911/