[06:37:35] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Smalyshev) > After talking with Stas this apparently makes updating within the updater harder etc as it might result in more wr... [07:51:59] 10Traffic, 10Gerrit, 10Operations, 10Phabricator, 10Security-Team: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Jdrewniak) [08:33:46] XioNoX: "We have completed our initial analysis of your issue and have found we have a loss of signal from customer in Ashburn." and then they opened a case - Telia Carrier Case Reference 00959377 [11:57:10] Krenair: this looks sane to you? https://gerrit.wikimedia.org/r/c/operations/software/acme-chief/+/496756/ [11:59:55] vgutierrez, it doesn't appear to match with the description on https://gerrit.wikimedia.org/r/#/c/operations/software/acme-chief/+/496754/ ? [12:01:54] nice catch, the issue is in the commit message only [12:02:00] the actual path is /var/lib/acme-chief/certs [12:02:48] ok [12:07:43] It should be a painless update, but I'll deploy it on Monday [12:09:54] next cert renewal should be librenms on Monday, so we will see it live soon [13:21:45] vgutierrez: do you think LDAP server certs can switch to certcentral ? [13:21:56] it seems we have some semi-puppetized setup there [13:22:08] which requires manual cert generation for new hosts [13:23:54] hmm probably [13:25:25] vgutierrez: can you tell by looking at the existing cert if it's a candidate for moving to LE? [13:26:37] sure.. paste the openssl x509 -text output somewhere :) [13:26:54] andrewbogott: is the new host already reachable via SSH and has the role? [13:27:07] or was that cloud vps testing [13:27:15] https://www.irccloud.com/pastebin/1CL7qfxw/ [13:27:19] vgutierrez: ^ [13:27:29] mutante: it's ldap-eqiad-replica01.wikimedia.org [13:27:43] and not doing anything important at the moment, so feel free to break it more :) [13:28:16] ok, checking if it gets the right Hiera values [13:28:50] channel 0: open failed: administratively prohibited: open failed [13:29:12] yeah.. that can be issued by LE [13:29:19] did you connect via puppetmaster with install-console or so? [13:29:24] good news :) [13:29:50] as long as the ldap server doesn't suffer with a reload every two months... [13:30:33] mutante: I can ssh to it using iron as the proxy. I'd expect it to just work for you [13:30:54] vgutierrez: it reloads way more often than that now :( That's the issue I'm trying to work around with lvs [13:31:12] iron, wow, havent used that in a while. it's experimental [13:31:17] tries [13:31:35] mutante: I'd really expect any bastion to work though, nothing special about that host and iron [13:31:42] it's just what I copy/pasted [13:32:33] yeah, works for me with bast1002 too. [13:32:51] oh, note that it's a public IP though: ssh ldap-eqiad-replica01.wikimedia.org [13:34:17] yea, ack, i get denied from labs-eqiad-replica01.wikimedia.org while i can ssh to icinga2001.wikimedia.org for example [13:34:32] does "id dzahn" say i exist there? [13:35:05] yeah, looks fine [13:35:09] https://www.irccloud.com/pastebin/yiJ195qb/ [13:35:15] .. eh.. i am on it now [13:35:22] * andrewbogott shrugs [13:35:31] i copy/pasted your line above and works [13:35:55] i see my typo now [13:36:17] labs-eqiad-replica01 vs ldap-eqiad-replica01 duh :) [13:40:16] yeah the error when you attempt to connect to a wrong hostname doesn't help at all [13:40:34] andrewbogott: well.. slapd seems to be happy with the current config, does not crash with a cert error [13:41:01] did you get the error when trying to use it with ldaplist ? [13:41:16] I haven't tried :) I was assuming it would be rejected because of being self-signed [13:41:17] I'll try now [13:42:38] hm, that worked but I need to try ssl [13:43:58] it fails over ssl but with an unhelpful error [13:44:31] * andrewbogott tries some other things [13:44:54] config file testing succeeded [13:45:15] ran "slaptest" [13:46:24] can you get ldapsearch to work? [13:46:45] (I'm still messing with the args… I'm used to using ldapvi which definitely doesn't work over ssl) [13:46:54] "Most clients now have a -Z flag which enables sending the StartTLS extended operation to the server. This extended operation initiates TLS negotiation. To use ldaps://, one must use -H ldaps://. " [13:48:09] andrewbogott: yes, ldapsearch works :) [13:48:18] with TLS? [13:48:30] I have an ldapvi command that works with ldaps on the old ldap server but not the new one [13:48:37] [ldap-eqiad-replica01:/etc/ldap] $ ldapsearch -x mail="andrew*" [13:48:44] not sure yet.. looking [13:48:47] ah, ok, so not ssl [13:49:10] if i add the -Z mentioned above [13:49:18] ldap_start_tls: Connect error (-11) [13:49:19] additional info: (unknown error code) [13:49:19] ldap_result: Can't contact LDAP server (-1) [13:49:41] are we doing StartTLS or ldaps:/ though [13:51:04] either would use the cert, right? [13:51:46] ldapsearch -Z -x mail="andrew*" -H ldaps://ldap-eqiad-replica01.wikimedia.org [13:51:49] ldap_start_tls: Can't contact LDAP server (-1) [13:51:53] this would be -H for ldaps:// [13:52:38] and it fails with one or the other as well [13:52:58] do you have a patch you can point me to that converts from a static cert to LE? Maybe I can just do that across all ldap servers [13:53:03] (unless you feel like doing it :) ) [13:55:30] it's LDAP on port 636 , not StartTLS [13:55:40] andrewbogott: i think one issue for now is ... iptables missing [13:55:53] port 636 is what it's listening on.. but no hole for that [13:56:04] so that's why "cant contact server" before any cert issue [13:56:19] probably puppet didn't get as far as setting up ferm [13:57:16] oh wait, protocol is listed by name and not number, looking more [13:57:19] wait, I see 'ldaps' open to pretty much everyone in iptables [13:57:20] yeah [13:59:22] (I'm pretty impressed that the ldap db seems to be properly populated though. That's one thing that puppet handled like magic) [14:02:44] ok, at least i found some more reason why it fails [14:02:48] TLS: peer cert untrusted or revoked (0x42) [14:03:00] TLS: can't connect: (unknown error code). [14:04:34] http://www.openldap.org/lists/openldap-technical/201312/msg00167.html "And I get this legendary error “TLS: peer cert untrusted or revoked (0x42)” for which number of recommendation have been made online, to set TLS_REQCERT & TLS_CACERT in the /etc/ldap.conf" "Although this did not work for me. " "While openssl and gnutls command can successfully connect and validate the certificate, [14:04:40] ldapsearch and getent miserably fails. " [14:05:11] "Never mind, got it to work" "By setting TLS_REQCERT to allow" [14:05:16] well, lol at the very last part :p [14:05:35] andrewbogott: maybe it's better to just focus on making the switch to LE then [14:05:48] rather than trying to fix this one [14:05:51] Yeah, that'll have to be done with the existing certs expire anyway [14:06:26] btw, the switch to get more debug info was "-d1" [14:06:37] as in [ldap-eqiad-replica01:/etc/ldap] $ ldapsearch -H ldaps://ldap-eqiad-replica01.wikimedia.org -x mail="andrew*" -d1 [14:06:57] and -H is required as opposed to -h and the port should be default [14:06:57] ah, there it is :) [14:07:08] and -p is incompatible with -H [14:08:41] also confirmed that from serpens can speak to ldap-eqiad-replica01 as well.. firewall is fine, it's just the cert [14:10:22] re: the puppet example change i am not sure. guess we should just make a ticket for converting these [14:12:41] T218398 [14:12:41] T218398: Update openldap profile to use LE - https://phabricator.wikimedia.org/T218398 [14:12:58] :) [14:13:04] If you don't know how to do that offhand, I'll dive in in a bit [14:14:01] 10Traffic, 10Cloud-VPS, 10Operations, 10LDAP, 10cloud-services-team (Kanban): Update openldap profile to use LE - https://phabricator.wikimedia.org/T218398 (10Dzahn) [14:14:28] i don't, it's certcentral now [14:14:42] i used to before that [14:15:27] well.. acme-chief nowadays [14:15:29] :) [14:15:30] vgutierrez: can you point me to some puppet code that demonstrates the latest cert approach? (I see lots of puppet patches from you switching from LE to certcentral but also some patches from you ripping out certcentral stuff so, not sure what is the current best practice) [14:15:34] hah, I knew it [14:15:49] 10Traffic, 10Cloud-VPS, 10Operations, 10LDAP, 10cloud-services-team (Kanban): Update openldap profile to use LE - https://phabricator.wikimedia.org/T218398 (10Dzahn) per brief IRC chat with @vgutierrez it should be possible to migrate these to use Certcentral (https://wikitech.wikimedia.org/wiki/Certcent... [14:15:51] so we are in the middle of yet another migration [14:15:52] vgutierrez: heh, ok !:) [14:16:20] but right now you can take librenms as an example [14:16:28] ok, I'll look [14:16:29] thank you [14:16:48] mainly you need 1 commit to let acme chief servers know that you want a certificate [14:17:08] and another one to make your servers get the certificate [14:18:58] andrewbogott: https://github.com/wikimedia/puppet/blob/production/hieradata/role/common/acme_chief.yaml [14:19:20] a change to that file will make acme-chief to get the certificate issued [14:19:29] great, I'll start with that... [14:19:42] 10Traffic, 10Cloud-VPS, 10Operations, 10LDAP, 10cloud-services-team (Kanban): Update openldap profile to use LE - https://phabricator.wikimedia.org/T218398 (10Dzahn) command for testing connection over ldaps with more debug info why it fails: [ldap-eqiad-replica01:/etc/ldap] $ `ldapsearch -H ldaps://lda... [14:21:52] vgutierrez: does 'authorized_regexes:' mean I can have one cert work for multiple endpoint names? [14:22:36] like, ldap-labs.codfw.wikimedia.org, ldap-labs.eqiad.wikimedia.org, ldap-ro.eqiad.wikimedia.org, ldap-ro.codfw.wikimedia.org all in one cert? Or do I need to specify four different ones there? [14:24:42] hmm you need to set that as multiple SNI [14:25:04] the auth_regexes will be used to allow the servers to fetch that certificate [14:25:09] ok [14:27:06] vgutierrez: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496785/ ? [14:30:32] looks good, but could you follow the alphabetical order? [14:30:42] ldap should come before librenms [14:30:53] ah, true [14:32:05] thx <3 [14:32:14] * vgutierrez OCD relieved [14:32:26] hm… do I still need to install a key someplace as well/ [14:32:30] ? [14:36:15] after that's merged and puppet runs in acmechief1001 that should get the certificate issued [14:36:55] after that you can safely add one acme_chief::cert {'ldap'} in your ldap puppetization fetching the cert [14:37:26] ok. And will the cert itself be literally named 'ldap.crt'? [14:37:28] you'll want to use the puppet_svc parameter to set the service to be notified upon certificate changes [14:38:10] you'll get two, /etc/acmecerts/ldap.rsa-2048.crt and ldap.ec-prime256v1.crt [14:38:25] keys will be the same but ended in .key [14:38:34] ok [14:40:47] which should I direct my service to use, prime256v1 or rsa-2048? [14:41:34] and am I still using /etc/ssl/certs/ca-certificates.crt as the ca? [14:44:02] hmmmm to keep using the same kind of cert you should go for the rsa-2048 one [14:44:34] yeah I think so [14:44:41] if you need the chain [14:44:58] you can use .chained.crt to get the cert + the chain [14:45:07] or .chain to get only the chain [14:45:48] ok, next patch… https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496790/ [14:46:04] * andrewbogott tries that in the compiler to see if hiera is right [14:47:41] so [14:47:54] the directory is /etc/acmecerts [14:48:05] and the filename ldap.rsa-2048.crt [14:48:15] ah, ok [14:48:22] what about the key? [14:48:40] same path [14:48:51] and ldap.rsa-2048.key [14:49:00] you can split that in two if you wish [14:49:09] the key is also in /etc/acmecerts? [14:49:25] so get the acme_chief:: cert resource first and then switch openldap to use it [14:49:30] yes, same path [14:52:02] ok… the compiler likes https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496790/ ok. Look right to you? [14:52:18] (I assume that if I do it in one patch, the worst case is that I'll just have to do two puppet runs in a row) [14:53:07] no [14:53:47] they key will be in /etc/acmecerts as well :) [14:54:21] oh, sorry, as you said... [14:54:28] no problem [14:55:00] ok, fixed [14:55:03] this is all my fault.. I haven't put the time to write a wikitech page with the whole process [14:55:08] so thx for your patience [14:55:32] I'm just happy to have located someone who knows how this works :) [14:56:24] looks good, but before merging it trigger a puppet run in acmechief1001 [14:56:50] I'd do it myself but I'm having lunch and I only have my smartphone with me [14:57:24] done [14:58:44] lovely [14:59:44] ok, going to merge this on the secondary first to see what breaks (that's serpens.wikimedia.org) [15:00:09] if everything goes as expected.. you can run as root openssl x509 -text -noout -in /var/lib/acme-chief/live_certs/ldap.rsa-2048.crt [15:00:18] and you'll see your cert :) [15:00:35] hey, so i did not follow the whole thing, but just a quick note.. gerrit uses that LDAP server [15:00:39] afaict [15:01:05] so it might be a dependency cycle if something breaks and to merge the fix [15:02:25] yeah.. splitting it in two would be wise :) [15:02:43] mutante: in theory since we have two servers it should be fine... [15:03:03] (also, I already merged :/ ) [15:03:05] true, ack [15:03:53] oh, my regexp was wrong because I didn't put the actual hostnames in there. One more patch... [15:04:01] :) [15:04:05] reading backlog how it works, cool. tempted to copy/paste the releveant log lines to wikitech [15:08:30] whoah, acmechief1001 is buster? [15:08:33] early adoption! [15:10:12] vgutierrez: ok, I got my certs in /etc/acmecerts [15:10:24] but /var/lib/acme-chief/live_certs/ldap.rsa-2048.crt doesn't exist [15:10:26] and slapd hates me [15:11:01] sorry.. the /var/lib path is valid on acmechief 1001 [15:11:21] can you paste the openssl x509 output for the cert? [15:11:45] oh, I see [15:11:47] "main: TLS init def ctx failed: -1" [15:11:51] ^ that's what slapd says [15:11:56] uh? [15:12:01] that's not helpful at all :) [15:12:30] https://www.irccloud.com/pastebin/Lf0HltPP/ [15:13:36] slapd.conf has: [15:13:38] https://www.irccloud.com/pastebin/00rDnMwj/ [15:15:28] ugh, did I kill acme-chief with a typo? [15:15:48] looks like that [15:16:55] I don't see the mistake yet [15:17:02] (and it made the cert, so that's weird…) [15:17:04] checking... [15:18:06] can you paste the journalctl -u acme-chief output from acmechief 1001? [15:18:15] that should let us know what's going on [15:20:20] there's like 1000+ lines, looking for the good bits [15:20:26] Mar 15 15:18:46 acmechief1001 acme-chief-backend[23651]: raise ACMEError('Unable to push CSR') from order_error [15:20:39] hmmmm [15:20:50] https://www.irccloud.com/pastebin/4wCFmJ9v/ [15:21:13] it's ldap-labtest [15:22:28] I'm fixing one thing which is probably not the real issue... [15:22:42] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496802/ [15:23:29] (and applying) [15:23:38] yeah [15:23:44] I've noticed that [15:23:58] but it shouldn't trigger the issue we are seeing [15:24:06] agreed [15:24:12] please acknowledge the icinga alert [15:24:18] I did, I think? [15:24:27] and I'll handle it as soon as I can go home [15:24:33] thx [15:24:33] maybe it flapped [15:24:49] <+icinga-wm> RECOVERY - Ensure acme-chief-backend is running [15:24:53] something fixed it [15:25:36] I'll check it anyways [15:25:45] yes, it's not really fixed, its flapping [15:25:56] systemctl status acme-chief still no good [15:26:04] yup that's what I suspected [15:26:38] want me to just rip out that section for now? Or is it ok to leave flapping? [15:26:56] indeed.. leave it out [15:27:10] LE will be happier with us :) [15:27:58] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496804/ [15:28:19] so now the question is… why is slapd angry [15:28:37] https://www.irccloud.com/pastebin/VSGWrSBp/ [15:28:38] besides the typo in the commit msg [15:28:40] it looks good [15:29:23] I absolutely cannot spell 'chief' correctly [15:29:28] every time I type it my fingers transpose [15:29:32] even if I'm concentrating [15:29:35] stupid fingers [15:33:23] hmm slapd is running as root? at least when it's trying to open the tls certificate key? [15:33:40] it was an experiment [15:33:46] oh, well, wait... [15:33:48] * andrewbogott checks again [15:33:58] otherwise it can not read the certificate key [15:34:25] so yeah, I think that's the issue. If it runs as root it works, if not then not [15:34:32] so do I just need to change the permissions on the cert? Or... [15:34:47] (I'm running it as root in the meantime to keep icinga quieter) [15:35:02] if that's the case you can use the key_group parameter of acme_chief::cert to set the proper group [15:35:33] on a 'correct' system it runs as openldap [15:35:37] ok, cool [15:37:34] vgutierrez: is this what you mean? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496807/ [15:37:56] indeed :) [15:40:43] ok! Puppet/slapd are happy now... [15:40:46] let's see if it actually works [15:41:45] yep, all looks good [15:41:54] awesome [15:42:14] sorry about the mess with the second ldap cert [15:42:26] I'll check it as soon as possible [15:42:41] It's not urgent, it's for a test box [15:42:43] I'm curious though [15:43:10] I'm going to give this a few more minutes and then will apply on the eqiad ldap server [15:43:20] which will probably go better :) [15:48:51] vgutierrez, mutante, the new certs are now active on both ldap servers and things look good. Thank you! [15:50:06] thanks :) [15:50:28] I'll use this conversation as a template to write the acme-chief doc [15:50:56] I'll ask for your feedback cause I believe that you are the fiesy one on setting up a cert with acme chief on your own [15:50:58] :) [15:51:28] It's quite simple, apart from the mysterious rejection of the ldap-labtest cert [15:51:54] andrewbogott: it's a trusty host, it's openssl might be too old [15:52:08] labtestservices2001 I mean [15:52:20] moritzm: the problem was the generation of the cert — we didn't get as far as trying to apply it I think [15:52:40] I mean, you might /also/ be right, but that wouldn't have broken acme-chief :) [15:53:27] given https://phabricator.wikimedia.org/T218022 I wouldn't bother further about it :-) [15:53:40] LOL [15:53:53] moritzm: true but we'll need an ldap server someplace [16:15:44] mutante: any news from Telia? is it possible to have them CC noc@? [16:20:05] 10Traffic, 10Operations, 10Patch-For-Review: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263 (10ema) RAM cache usage seems to be growing non-stop. I have left `proxy.config.cache.ram_cache.size` to the default value of `-1`, which according... [16:30:42] XioNoX: they called the ticket resolved. forwarded just now [16:33:54] 10netops, 10Operations: eqiad - eqord Telia link down - IC-314533 - https://phabricator.wikimedia.org/T218307 (10Dzahn) Telia claims the issue was resolved. our monitoring can't confirm that. ` Telia Carrier Case Reference 00959377 Service ID IC-314533 Service Type DWDM - 10 Gigabit - Ashburn - Chicago Ca... [16:34:27] interesting, the link is still down [16:35:34] yeah, light is making it one way but not the other, I'll follow up with them [16:38:52] ok, ack ! [16:41:29] 10netops, 10Operations: eqiad - eqord Telia link down - IC-314533 - https://phabricator.wikimedia.org/T218307 (10ayounsi) a:03ayounsi We're getting one way light, followed up with Telia. Ashburn side: ` Physical interface: xe-4/2/0 Laser bias current : 43.070 mA Laser output... [16:41:46] thx again! [17:43:09] andrewbogott: so... I've been checking acme-chief logs [17:44:11] yeah? [17:44:37] it looks like you configured first ldap-labtest with labtestservices2001.wikimedia.org as CN [17:44:49] and then you added ldap-labtest.eqiad.wikimedia.org to the list of SNIs [17:44:53] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Campsite, and 2 others: nuke_limit often reached on esams varnish frontends - https://phabricator.wikimedia.org/T216006 (10ema) The issue should be fixed. @VladimirAlexiev, @Addshore: can you confirm? [17:46:14] for some reason issuing certificates for labtestservices2001.wikimedia.org,ldap-labtest.eqiad.wikimedia.org was rate limited [17:46:49] huh, so if we tried again it might just work? [17:46:55] hmmm nope [17:47:03] the rate limit is quite strict [17:47:31] from https://letsencrypt.org/docs/rate-limits/ --> We also have a Duplicate Certificate limit of 5 certificates per week. [17:47:55] checking the LE order: https://acme-v02.api.letsencrypt.org/acme/order/45156774/356338607 [17:48:14] the error says... "Error finalizing order :: too many certificates already issued for exact set of domains: labtestservices2001.wikimedia.org,ldap-labtest.eqiad.wikimedia.org [17:48:32] I'm wondering if that certificate has been requested before with an older puppetization or something similar [17:48:45] I wouldn't think so but I guess it's possible [17:49:56] anyway, can do it for just labtestservices2001 [17:50:02] I think that's good enough for now [17:50:50] hmmm actually on the file system I see that ldap-labtest.ec-prime256v1.crt is valid for the two SNIs that you've requested [17:52:16] It seems most likely that it rate-limited itself just now, in response to some kind of frantic attempt [17:52:23] https://www.irccloud.com/pastebin/vAT1TO68/ [17:54:24] labtestservices2001 is still unhappy though [17:54:46] what do you mean? [17:55:19] which? [17:56:01] right now after you removed the cert config from acme_chief.yaml, no client can fetch it [17:56:21] andrewbogott: btw, check puppet run on labtestservices2001 [17:56:35] oh, you know [17:56:48] sorry, i was debugging icinga stuff and saw it.. unrelatedly [17:59:22] I'm wondering if the glitch comes from the fact that the CN is not included in the SNI list [17:59:33] for ldap-labtest I mean [18:00:20] should we try https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/496841/ ? [18:02:08] let me know when that patch is merged [18:02:35] I'm already logged in acmechief1001, so I'll monitor that side [18:07:30] cool... acme-chief has successfully issued that cert [18:09:06] https://www.irccloud.com/pastebin/jaDJpymz/ [18:09:59] andrewbogott: right.. it's a bug in acme-chief [18:10:08] this is what happened [18:10:59] you configured ldap-labtest with CN: ldap-labtest.eqiad.wikimedia.org and SNI: labtestservices2001.wikimedia.org [18:11:12] acme-chief successfully issued the certificate once [18:11:49] but LE actually issues a certificate with CN: ldap-labtest.eqiad.wikimedia.org and SNIs: ldap-labtest.eqiad.wikimedia.org + labtestservices2001.wikimedia.org [18:12:14] so acme-chief thinks that the current config doesn't match the cert on disk and moves to re-issuing the certificate [18:12:45] but for LE it's the same cert, so acme-chief ends reaching LE rate limits on a stupid infinite loop [18:13:04] that would do it :( [18:13:26] sadly acme-chief hammered LE badly for ldap-labtest.eqiad.wikimedia.org+labtestservices2001.wikimedia.org [18:13:41] so we need to wait a week to issue a certificate for that exact list of SNIs [18:13:51] that's fine, we probably won't need to ever do it [18:14:02] I'm filling a phab. task now to work on the issue next week [18:14:05] things are pretty much working now, and everything will be renamed soon regardless [18:14:20] ack, let me know if you have some TLS related issue :) [18:23:37] 10Acme-chief: CN + SNI list on config file doesn't match issued certificate on some scenarios - https://phabricator.wikimedia.org/T218418 (10Vgutierrez) p:05Triage→03Normal [18:23:56] that's the task... I'll solve the issue on Monday :) [18:28:52] vgutierrez: do you have a minute to answer some questions about lvs, or are you on your way out the door? [18:28:56] I don't want you to miss dinner :) [18:29:52] TBH I don't feel comfy messing with LVS config on eqiad on a Friday at 18:30 UTC :) [18:30:36] ok [18:31:11] err.... [18:31:26] https://www.irccloud.com/pastebin/oNPeI5ls/ [18:31:27] yeah, I merged it but it doesn't do much so I'll probably just revert [18:31:27] WTF? [18:31:40] hm, well ok I'll definitely revert [18:31:41] dammit [18:35:04] reverted [18:35:41] please.. do not +2 a lvs change on a Friday afternoon :) [18:37:16] ok :) Sorry, I was clearly watching the wrong hosts as that applied [18:39:15] hm, that 'sort' error doesn't really tell me much about where the mistake is :( [18:44:34] you're using &ip_block48 instead of *ip_block48 [18:44:58] so that's why that triggers the null dereference calling sort on serviceip [18:45:20] also.. it should be ip_block048 instead of ip_block48 [18:47:01] check the comments in the CR :) [18:52:34] 10Acme-chief: CN + SNI list on config file doesn't match issued certificate on some scenarios - https://phabricator.wikimedia.org/T218418 (10Krenair) we should probably reject configs that don't include CN in SAN [18:53:16] 10Acme-chief: CN + SNI list on config file doesn't match issued certificate on some scenarios - https://phabricator.wikimedia.org/T218418 (10Vgutierrez) that's the easiest/saner way of solving this issue :) [19:46:03] 10netops, 10Operations, 10monitoring, 10Patch-For-Review: Juniper monitoring - https://phabricator.wikimedia.org/T83992 (10ayounsi) a:03ayounsi