[00:29:00] 10Traffic, 10DNS, 10Operations, 10Research: Add wikiworkshop.org to the Foundation's DNS - https://phabricator.wikimedia.org/T240303 (10leila) >>! In T240303#5734123, @Krinkle wrote: > This question isn't directly related but might help indirectly clear some confusion: > > Who will pay for the domain name... [01:42:28] 10Traffic, 10Operations: Browser Connection Security warning page apparently produces invalid XML - https://phabricator.wikimedia.org/T240497 (10Reedy) [01:46:13] 10Traffic, 10Operations: Browser Connection Security warning page apparently produces invalid XML - https://phabricator.wikimedia.org/T240497 (10Reedy) It might be invalid XML, but https://validator.w3.org/ has no problem with it {F31473167} And https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img f... [05:20:02] 10Traffic, 10Operations: Browser Connection Security warning page apparently produces invalid XML - https://phabricator.wikimedia.org/T240497 (10DavidBrooks) Oh, goodness, my thinking is muddled today. Blame a cold. This is an API query (GET /w/api.php?action=query...). It expects well-formed XML with some le... [09:02:06] 10Traffic, 10DNS, 10Operations, 10Research: Add wikiworkshop.org to the Foundation's DNS - https://phabricator.wikimedia.org/T240303 (10jcrespo) @BBlack Are Leila's answers covering all your questions or do you need additional information to propose a way to move forward? [09:12:25] 10Traffic, 10Operations: Start warning and deprecation process for all legacy TLS - https://phabricator.wikimedia.org/T238038 (10TheDJ) BTW. We no longer have the cipher stats grafana board ? Too bad, that one was hella interesting. [09:30:49] 10Traffic, 10Operations: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031 (10Joe) a:05Joe→03Volans [10:29:46] 10netops, 10Operations, 10ops-eqiad: Circuit down between cr1-eqiad and cr1-codfw - https://phabricator.wikimedia.org/T240545 (10elukey) p:05Triage→03High [12:55:02] 10Traffic, 10Operations: Browser Connection Security warning page apparently produces invalid XML - https://phabricator.wikimedia.org/T240497 (10Reedy) It kinda is and it isn't. You've been served a static HTML error page that isn't served by the MW API, it's coming from the caches infront of MediaWiki. It doe... [12:55:05] 10Traffic, 10Operations: Browser Connection Security warning page apparently produces invalid XML - https://phabricator.wikimedia.org/T240497 (10Reedy) It kinda is and it isn't. You've been served a static HTML error page that isn't served by the MW API, it's coming from the caches infront of MediaWiki. It doe... [12:58:55] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10Aklapper) [12:58:58] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10Aklapper) [13:12:55] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10Reedy) Noting this is an output of {T238038} [13:14:13] 10Traffic, 10Operations: Start warning and deprecation process for all legacy TLS - https://phabricator.wikimedia.org/T238038 (10Reedy) >>! In T238038#5727398, @TheDJ wrote: > Question. https://wikitech.wikimedia.org/wiki/HTTPS/Browser_Recommendations > > Windows 7: I know it CAN support TLS 1.2, but I can't... [13:29:46] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10BBlack) The way it works is that if the connection isn't using TLSv1.2, the user is served a 302 redirect to `/sec-warning` o... [13:34:27] 10Traffic, 10Operations: Start warning and deprecation process for all legacy TLS - https://phabricator.wikimedia.org/T238038 (10BBlack) >>! In T238038#5734955, @TheDJ wrote: > BTW. We no longer have the cipher stats grafana board ? Too bad, that one was hella interesting. The old cipher stats graphs (the ori... [13:39:07] 10netops, 10Operations, 10ops-eqiad: Circuit down between cr1-eqiad and cr1-codfw - https://phabricator.wikimedia.org/T240545 (10Jclark-ctr) a:03Jclark-ctr [13:42:10] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10Aklapper) Thanks for the detailed explanation. Does that mean the external parser should check the header to realize the mime... [13:46:24] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10BBlack) I'm not even sure what the task is asking for, but yeah in general we're not going to make the sec-warning mechanism... [13:56:05] o/ ema [13:56:11] can we merge these today? [13:56:17] https://gerrit.wikimedia.org/r/c/operations/dns/+/556411 [13:56:17] https://gerrit.wikimedia.org/r/c/operations/puppet/+/556413 [13:56:45] i can do them if you are ok with that (and there isn't any beyond authdns-update & puppet merge stuff that needs to be done) [13:59:02] ottomata: hi! Have you tested that the endpoints work, including those behind TLS? [13:59:09] something like: [13:59:15] curl -v https://intake-logging.wikimedia.org:43192/whatever/uri/should/work --resolve intake-logging.wikimedia.org:43192:$ip_address [14:00:04] $ip_address being the DNS discovery IP of the service [14:01:10] AH crap i need to add that to the SAN now. forgot about that. [14:01:13] aside from that it works [14:01:15] will do that asapo [14:08:10] hm _joe_ we should probably make cergen generate the unencrypted key along with all the other file formats too, eh? [14:08:19] that would skip a step in your tls k8s instructions? [14:21:08] <_joe_> ottomata: yes [14:21:11] <_joe_> please [14:21:15] k will do [14:21:33] 10Traffic, 10Operations, 10serviceops: Appservers behind TLS should support chunked Transfer-Encoding - https://phabricator.wikimedia.org/T240576 (10ema) [14:22:09] _joe_: i noticed we are using secret() in the hiear stuff now, nice! [14:22:30] ok! ema all done. [14:22:55] that works now for all 3 k8s clusters [14:33:31] <_joe_> ottomata: btw I'm working on a different way to use common templates in helm charts [14:35:59] oh ya? [14:46:15] 10Traffic, 10Operations, 10serviceops: Appservers behind TLS should support chunked Transfer-Encoding - https://phabricator.wikimedia.org/T240576 (10Joe) While it should be easy to swap nginx for envoy, we need to also convert `profile::services_proxy` to use envoy at the same time. It should not be impossi... [15:01:25] ottomata: I get 'connection refused' on the non-TLS port (eventgate_logging_external) - http://eventgate-logging-external.svc.eqiad.wmnet:33192 [15:01:38] checking [15:02:22] hm. [15:02:27] it works on kubestage1001.... [15:02:37] oh [15:02:51] oh, we only have lvs for the https port... [15:02:58] right? yeah . [15:03:49] ah.. i see th problem. [15:03:54] varnish can't do https, right? [15:04:15] ema: ? [15:05:22] <_joe_> no. [15:05:29] ah yes we agreed that varnish-be doesn't really matter as it's going away soon anyways [15:05:56] so there's no need to add a LVS service for plain-http eventgate_logging_external [15:06:15] <_joe_> yeah please no :) [15:06:53] ottomata: +1 [15:06:59] ok great, so remove it from text.yaml? [15:07:52] wait, ok, does text_ats.yaml reference the backend def in text.yaml? [15:08:03] ema: ? do I need to lave the directory def there, with port 43192? [15:08:15] but remove the fe routing? [15:11:37] director* [15:16:25] ottomata: you can merge the patch as-is [15:16:48] oh, ok. [15:17:16] merging! [15:18:39] ema: this is just a puppet run on cache (text) hosts then? [15:21:55] hm, puppet run on cache text node doesn't seem to change anything... [15:30:59] 10Traffic, 10Operations: API Querying for XML/JSON, you might get the Browser Connection Security warning HTML page (which is invalid XML) - https://phabricator.wikimedia.org/T240497 (10DavidBrooks) Thanks, BBlack, for the obvious thoughtful care that went into this. And, in my case, it had the desired end-res... [15:38:14] ;ema what am I missing? [15:38:23] ema: * [16:08:47] 10netops, 10Operations, 10ops-eqiad: Circuit down between cr1-eqiad and cr1-codfw - https://phabricator.wikimedia.org/T240545 (10Jclark-ctr) Replaced failed Fiber [16:08:58] 10netops, 10Operations, 10ops-eqiad: Circuit down between cr1-eqiad and cr1-codfw - https://phabricator.wikimedia.org/T240545 (10Jclark-ctr) 05Open→03Resolved [17:36:01] 10Traffic, 10Operations, 10serviceops: Use Envoy instead of nginx for TLS termination on Appservers - https://phabricator.wikimedia.org/T240576 (10ema) [17:39:33] 10Traffic, 10Operations, 10serviceops: Use Envoy instead of nginx for TLS termination on Appservers - https://phabricator.wikimedia.org/T240576 (10ema) This is a severe case of PEBKAC: `curl` uses HTTP/2 by default, that's why the response has no TE:chunked. Forcing curl to use HTTP/1.1 we can see that inde... [17:39:38] 10Traffic, 10Operations, 10serviceops: Use Envoy instead of nginx for TLS termination on Appservers - https://phabricator.wikimedia.org/T240576 (10ema) p:05Triage→03Normal [17:43:46] ottomata: https://intake-logging.wikimedia.org is being routed appropriately it seems (see X-Cache) [17:54:12] AH! [17:54:15] it just took a bit?! [17:54:15] cool [17:54:28] I'm off see you tomorrow! o/ [17:54:36] ok thank yoUUUU! [17:57:22] hm not always routed though [18:02:24] 10netops, 10Operations: Add cloudmetrics1002 to network devices ACL - https://phabricator.wikimedia.org/T240456 (10jcrespo) Hi, @Phamhi How urgent is this? Our netop is on vacations and will return next week. If it cannot wait, I can try to find someone to help you. [18:07:15] 10netops, 10Operations: Add cloudmetrics1002 to network devices ACL - https://phabricator.wikimedia.org/T240456 (10Phamhi) Hi @jcrespo , if he or she comes back early next week then it should be fine. [18:09:30] 10netops, 10Operations: Add cloudmetrics1002 to network devices ACL - https://phabricator.wikimedia.org/T240456 (10jcrespo) a:03ayounsi [18:09:38] 10netops, 10Operations: Add cloudmetrics1002 to network devices ACL - https://phabricator.wikimedia.org/T240456 (10jcrespo) p:05Triage→03High [18:18:18] 10Traffic, 10DNS, 10Operations, 10Research: Add wikiworkshop.org to the Foundation's DNS - https://phabricator.wikimedia.org/T240303 (10jcrespo) a:05leila→03BBlack [18:30:20] 10netops, 10Operations: Facebook BGP peering links down in ulsfo - https://phabricator.wikimedia.org/T239896 (10jcrespo) Your reasoning seems ok to me, but we should CC @ayounsi of changes. [20:43:08] 10Traffic, 10Operations: Fix acme-chief DNS validation correctly - https://phabricator.wikimedia.org/T240614 (10BBlack) p:05Triage→03High [21:23:25] https://phabricator.wikimedia.org/P9867 [21:23:38] ^ first authdns-over-tls on a prod box (only reachable from inside itself for now) [21:24:21] sukhe: ^ too [21:25:08] nice! [21:27:59] 10Traffic, 10Operations, 10Patch-For-Review: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) P9867 <- First internal test query on a prod dns box :) [21:29:41] wooo [21:57:55] :) [23:10:42] 10Traffic, 10Operations: Implement DNS-over-TLS for AuthDNS - https://phabricator.wikimedia.org/T239994 (10BBlack) This is now mostly-working, with heira flag controlling test deployment (currently only on dns4002, which doesn't have any public authserver IPs routed into it at this time). Reminders on the nex... [23:15:07] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission old eqiad caches - https://phabricator.wikimedia.org/T208584 (10Jclark-ctr) a:05Cmjohnson→03Jclark-ctr [23:16:03] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission lvs1007-1012 - https://phabricator.wikimedia.org/T208586 (10Jclark-ctr) a:05Cmjohnson→03Jclark-ctr