[00:56:35] 10Traffic, 10DNS, 10Mail: Disavow emails from wikipedia.com - https://phabricator.wikimedia.org/T184230#3876973 (10Platonides) [01:35:36] 10Traffic, 10DNS, 10Mail, 10Operations: Disavow emails from wikipedia.com - https://phabricator.wikimedia.org/T184230#3877038 (10Krenair) [02:55:06] 10Traffic, 10Cloud-VPS, 10DNS, 10Beta-Cluster-reproducible: Create some mechanism for instances in projects to modify the project Designate records - https://phabricator.wikimedia.org/T184245#3877216 (10Krenair) [02:56:38] 10Traffic, 10Cloud-VPS, 10DNS, 10Operations, 10Beta-Cluster-reproducible: Create some mechanism for instances in projects to modify the project Designate records - https://phabricator.wikimedia.org/T184245#3877228 (10Krenair) a:05Krenair>03None (alternatively we could just not use designate and inste... [05:01:04] so how long is https://doc.wikimedia.org/cover/ being cached mor? [05:01:06] for* [05:01:20] mutante tried to figure out and wasn't sure if it was a day or an hour [05:01:45] it's all static html/css/js with a little PHP so I'd like it to be more like an hour [05:01:57] and was wondering how to figure that out, and then if its possible to change it for that subdomain [06:23:09] <_joe_> legoktm: so if it has no explicit cache headers [06:23:32] <_joe_> and it's a 200 OK, then the general response is "up to 4 days" [06:23:51] <_joe_> because every caching layer can keep the object for up to a day [06:25:21] uh, that's really long [06:25:35] _joe_: so how can we lower it? [06:26:02] <_joe_> legoktm: set cache headers in the backend I guess? [06:26:21] I think it's just apache... [06:26:32] <_joe_> so it's static files? [06:26:36] yes [06:26:39] <_joe_> served from where? [06:26:42] well, nearly all static [06:26:45] contint1001 [06:26:53] there are a few pages that are small php scripts [06:26:57] <_joe_> so what needs to be not cached? [06:27:04] <_joe_> specifically, I mean [06:28:04] everything under https://doc.wikimedia.org/ I think - right now jenkins is regenerating and the pages are still showing older versions in some cases [06:28:04] <_joe_> uhm, are you sure that's served by contint1001? I see it goes through cache::text [06:28:16] yes [06:28:41] it's /srv/org/wikimedia/doc [06:28:46] <_joe_> legoktm: so set a cache header to a low value, via apache [06:29:19] https://github.com/wikimedia/puppet/blob/production/modules/contint/templates/apache/doc.wikimedia.org.erb in there? [06:29:34] <_joe_> yeah sorry I wrote "docs" [06:29:45] <_joe_> I woke up like 15 minutes ago, and still no coffee [06:30:06] <_joe_> anyways, it's served from cache::misc as I expected [06:31:14] <_joe_> I would suggest creating rules in the apache config for lowering the cache retention, but I'll let ema answer when he's around - actually, you'd be better off writing a ticket. [06:31:31] will do :) [06:42:24] 10Traffic, 10Continuous-Integration-Infrastructure, 10Operations: Lower varnish caching length on doc.wikimedia.org - https://phabricator.wikimedia.org/T184255#3877424 (10Legoktm) [06:42:26] _joe_: ^ [07:31:58] 10Traffic, 10DNS, 10Mail, 10Operations: Disavow emails from wikipedia.com - https://phabricator.wikimedia.org/T184230#3876973 (10Peachey88) I have a funny feeling that fundraising may be using @wikipedia.com aliases in emails. [07:49:14] 10Traffic, 10DNS, 10Mail, 10Operations: Disavow emails from wikipedia.com - https://phabricator.wikimedia.org/T184230#3876973 (10grin) Whoever uses it should be covered by the SPF anyway, that's the point. wikimedia.org. 597 IN TXT "v=spf1 ip4:91.198.174.0/24 ip4:208.80.152.0/22 ip6:... [13:25:48] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10User-fgiunchedi: Add Prometheus client support for varnish/statsd metrics daemons - https://phabricator.wikimedia.org/T177199#3878124 (10fgiunchedi) [13:25:50] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10User-fgiunchedi: Limit http methods reported by varnishmtail - https://phabricator.wikimedia.org/T183926#3878122 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi [13:46:01] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10User-fgiunchedi: Add Prometheus client support for varnish/statsd metrics daemons - https://phabricator.wikimedia.org/T177199#3878207 (10fgiunchedi) Status update: * varnishstats has been replaced with varnishmtail-backend to get a breakdown of stat... [16:37:10] Hello, I'm currently facing an issue and I can't access Wikitech [16:37:24] I get a privacy error trying to access this: https://wikitech.wikimedia.org/ [16:37:58] It's the first time I'm facing this issue [16:38:35] Attackers might be trying to steal your information from wikitech.wikimedia.org (for example, passwords, messages, or credit cards). Learn more [16:38:38] NET::ERR_CERT_AUTHORITY_INVALID [16:39:39] bd808 directed me here maybe I can get some help, anyone please? [17:13:28] Just noticed I can't also access Gerrit :( [19:42:54] 10Traffic, 10Operations, 10ops-eqiad: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#3878953 (10RobH) p:05Triage>03Normal [19:43:49] 10Traffic, 10Operations, 10ops-eqiad: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#3878970 (10RobH) a:03BBlack Assigning this to @bblack to advise on racking proposal & confirm where these should go. Please provide feedback and assign to @Cmjohnson for followup. [19:49:04] d3r1ck, that site uses an LE cert [19:49:41] check that you have either DST Root CA X3 or Let's Encrypt Authority X3 in your store [19:51:08] Krenair: Not sure I understand you :) [19:51:22] But I've been accessing the site, only today evening I can't get to it anymore [19:51:37] I think the problem is on your end [19:51:54] Krenair: Of course, I trust WM servers :D [19:52:07] our PM chat sounded like his ISP was trying to MITM his TLS connections [19:52:07] Krenair: Maybe you could walk me through please, if you have time? [19:52:22] bd808: Which is a very valid point! [19:52:25] wow [19:52:33] ok [19:52:37] We're currently having some useless internet issues in this part of our country :( [19:52:41] d3r1ck, what about wikipedia.org? [19:53:02] I can access WP with no issue [19:53:08] Just Gerrit and Wikitech [19:53:44] ok, what about https://helloworld.letsencrypt.org/ ? [19:53:49] 10Traffic, 10Operations, 10ops-eqiad: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#3878995 (10RobH) Please note that the three business day wait for objections to be noted on the task will end on Tuesday, 2018-01-09. Barring objections, this can be merged by ops clinic duty on t... [19:58:18] Hmm, we've got some other LE sites too, but Wikitech & Gerrit are probably valid enough testcases [20:13:58] Krenair: I can't access that too [20:17:04] d3r1ck: Any chance a different browser works? That'd narrow it to your ISP vs. local machine [20:18:09] The various intermediary tests listed on https://letsencrypt.org/certificates/ might also be telling [20:18:16] no_justification: Doesn't work on other browsers too [20:18:35] Mmk, definitely sounds like an ISP issue then if you can't hit *any* LE site from *any* browser [20:18:45] But from my experiences, I've had such issues when SSL cert are not configured properly [20:18:53] Expecially intermediate certificates [20:19:13] no_justification: Okay! I'll try again tomorrow [20:19:22] I'm pretty sure all of these we're trying are configured correctly, or we'd have wider reports of it failing. [20:19:47] Also: worth importing their self-signed root? Could possibly work if your ISP is futzing with the root cert available in your browser. [20:19:50] no_justification: Okay! Thanks very much for the help. Cc Krenair [20:20:14] (My personal website also uses LE, if you want another datapoint: https://anyonecanedit.org) [20:20:45] no_justification: I can't access that too :( [20:20:53] Your website you just sent [20:21:15] Yeah, so it's definitely a root cert and/or ISP issue, not a particular-site-is-misconfigured issue. [20:21:30] I find it unlikely WMF, LE itself as well as I are all 3 misconfigured. [20:22:06] no_justification: You are correct! [20:22:28] Highest probability is ISP related, infact 90% [20:22:41] no_justification: Thanks again, I'll check this tomorrow and give feedback [20:23:20] Mmk. Best of luck! [20:26:01] :) [21:22:50] 10Traffic, 10Cloud-VPS, 10DNS, 10Operations, 10Beta-Cluster-reproducible: Create some mechanism for instances in projects to modify the project Designate records - https://phabricator.wikimedia.org/T184245#3877216 (10bd808) Related: * https://github.com/hanazuki/acmesmith-designate * {T173469} [22:51:07] 10Traffic, 10DNS, 10Operations, 10Beta-Cluster-reproducible, 10Upstream: Ferm/DNS library weirdness on deployment-mediawiki boxes - https://phabricator.wikimedia.org/T153468#3879374 (10Krenair) Gave up waiting for that (it's been almost a year), sent a message anyway and it's been held for moderation. [23:03:31] 10Traffic, 10DNS, 10Operations, 10Beta-Cluster-reproducible, 10Upstream: Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances - https://phabricator.wikimedia.org/T153468#3879417 (10Krenair) [23:07:44] 10Domains, 10Traffic, 10Operations, 10Research, 10Patch-For-Review: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3879421 (10Dzahn) [23:22:02] 10Traffic, 10DNS, 10Operations, 10Beta-Cluster-reproducible, 10Upstream: Ferm/DNS library weirdness causing puppet errors on 12 deployment-prep instances - https://phabricator.wikimedia.org/T153468#3879428 (10Krenair) [23:31:02] 10Traffic, 10DNS, 10Operations, 10Beta-Cluster-reproducible, 10Upstream: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances - https://phabricator.wikimedia.org/T153468#3879457 (10Krenair)