[06:29:09] I'll reboot phabricator, gerrit and gitlab primary in a few [07:32:00] Hey folks! [07:32:28] I am about to disable puppet on multiple hosts (kafka*, cp*, etc..) since I'll need to renew the kafka intermediate ca [07:34:58] arnaudb: o/ am I messing up your morning if I disable the cp* hosts? I noticed that you were moving gitlab behind the CDN [07:35:11] it is done ! [07:35:18] feel free to go ahead [07:35:27] ook.. same for slyngs [07:35:36] I may have picked up the wrong morning [07:36:19] Can you avoid cp7003-7009? [07:36:25] They are currently being rebooted [07:37:00] cp7003 is done... I can stop the cookbook if you need? [07:37:23] nono I can do it in the afternoon [07:37:35] right after lunch, so you can finish your maintenance [07:37:36] no rush [07:37:50] Okay, I'll let you know when we're done [12:06:52] elukey: Rebooting is done for now [12:08:24] Let me know when it's safe to continue [12:24:47] slyngs: ack thanks! Is it a good moment to disable puppet on al cp hosts for some time? [12:25:16] Yes, should be fine. I'll just notify traffic [12:27:59] _joe_, topranks - for your extreme joy I am about to renew the Kafka intermediate CA, this will trigger cert refreshes on all cp hosts, kafka etc. It should just proceed without any manual restart of work to do, but I'll do batch puppet runs to be sure. [12:28:37] <_joe_> elukey: ack i am entering a meeting tunnel that will finish around 7 pm [12:28:44] elukey: ok thanks [12:42:38] maintenance started, both public and private certs have been merged [12:42:43] now running puppet on pki1002 [12:49:32] and also on other pki nodes, all good [12:49:35] proceeding with kafka testd [12:49:38] *test [12:50:21] and it paged pki, sigh, checking sorry [12:52:31] elukey: hey yeah it paged no probs [12:52:40] let me know if I can help [12:58:56] so this is probably a TIL, I ran puppet on the pki nodes with the assumption that cfssl-multirootca (the daemon that serves certs requests etc..) would restart working quickly [12:59:10] it turns out afaics that there is a window of time in which it apparently has to catch up? [12:59:57] still shows some unavail, so this is the first maintenance after the LVS endpoint and there may be some gotchas [13:05:29] wow ok now I get it [13:05:37] "Body did not match regular expression" target=https://[10.2.1.96]:443/metrics regexp="signer=\"discovery2026\"" [13:05:59] we required this since even before LVS, but the service was non-paging [13:06:13] so when you restart the cfssl daemons the prometheus counters reset [13:06:24] until some request for the signers (aka intermediates) happen [13:10:35] basically https://gerrit.wikimedia.org/r/1357279 if anybody has time [13:14:57] elukey: I +1 'd it. Makes sense to me. [13:15:29] also kafka brokers even on 3.7 don't pick up new keystores as I assumed [13:15:32] * elukey sighs [13:17:53] so yeah I need to roll restart all kafka clusters at this point [13:18:03] elukey: Could you ping me when puppet is re-enabled on kafka-jumbo, please. I am in a rolling reboot cookbook, waiting for a clear puppet run. [13:19:00] elukey: Do we need a config change to enable hot-reloading of the keystore? [13:20:01] btullis: yeah that is another possibility, but we never really used it. I was convinced that kafka had some inotify mechanism to hot reload keystores, but apparently not, and I am very sad [13:20:39] slyngs: as FYI I ran puppet on cp7001, haproxykafka picked up the new cert [13:20:51] with the new intermediate I mean [13:21:32] btullis: the kafka-configs command needs a broker id etc.. I think it is safer to just roll restart to be honest, wdyt? [13:22:25] I only have experience with old versions of Kafka, and they needed a restart to pick up new certs [13:22:54] yeah I've read the new 3.7 specs wrongly [13:22:55] elukey: Yes, sounds good to me. I was roll-rebooting anyway, as part of the fleeet-wide kernel update. We can look at hot-reloading keystores in the future. [13:24:06] btullis: if you are ok I am going to run puppet on kafka test, roll restart and make sure nothing explodes [13:24:13] then I'll give you the green light [13:24:25] Super! Thanks. [13:26:01] thanks for the review btw! [13:29:30] ok proceeding on two fronts: [13:29:47] 1) kafka test restarts to pick up new certs and make sure all is ok [13:29:56] 2) cp7xxx rollout (haproxy client certs) [13:30:04] elukey: I'm wondering on the status of the page for pki [13:30:08] (catching up after a meeting) [13:30:22] I think you adjusted the alert? perhaps it won't auto-resolve now so I should do that in victor-ops? [13:30:28] topranks: it needs https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357279 to be propagated on prometheus hosts [13:30:43] I didn't force puppet but it should happen soon-ish [13:30:59] I didn't touch the alert though [13:35:30] topranks: recovery :) [13:36:52] elukey: awesome thank you <3 [13:40:22] is something up with bast1004? I can't seem to reach it but can reach the other bastions fine [13:41:42] that certainly seems to be something wrong with my ISP, interesting [13:43:09] WFM [13:43:46] and it's back, I had 100% packet loss at my ISP it seems. sorry for the noise. [13:43:54] ah ok ^^ [13:47:19] puppet re-enabled on all cp* hosts after checking that magru worked as expected [13:48:23] btullis: kafka-test completed and afaics it looks ok [13:48:49] how do you want to proceed? [13:52:02] I am going to proceed with the logging clusters (Tiziano gave me the green light) [14:01:58] logging-codfw is in progress [14:06:51] is grafana slow for everyone or just me? [14:08:01] seems fine to me [14:08:11] No way I can search for a dashboard :( [14:09:43] https://grafana.wikimedia.org/dashboards?starred=true&query=mysql does this work for you Lucas_WMDE ? [14:10:27] no, doesn’t load any results [14:10:34] same for me [14:10:37] I see a 403 in the network panel (You'll need additional permissions to perform this action. Permissions needed: teams:read) [14:11:22] https://grafana-rw.wikimedia.org/dashboards?starred=true&query=mysql (i.e. logged in) shows “no results found for your query” instead o_O [14:11:32] did someone change grafana permissions? [14:11:47] :-/ [14:13:20] Lucas_WMDE: yeah, now that I logged in I can see things [14:16:40] I think the use of starred=true is the problem with that query [14:23:09] effie: 12:03 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896 we are checking if that can be related to: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?from=now-6h&to=now&timezone=utc&var-site=eqiad&var-group=core&var-shard=$__all&var-role=$__all&viewPanel=panel-16 [14:23:30] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [14:23:33] I am going to start the roll restart of kafka main codfw [14:23:50] marostegui: you have to be a little bit more specific [14:24:32] effie: please join #wikimedia-data-persistence [14:24:32] elukey: please ping me when you are done, I would like to depool eqiad [14:24:36] marostegui: what's up with all the s2 writes? [14:24:50] cdanis: we are also discussing that, but it is unrelated [14:24:53] ack [14:32:53] effie: in theory you can do it anyway if there is a bad situation going on, the certs in other clusters have been picked up just fine [14:33:10] I'll also need to restart kafka-main-eqiad afterwards [14:33:16] and kafka-jumbo [14:34:04] alright then [14:46:37] started kafka-main-eqiad [15:10:23] elukey: Thanks ever so much. My cookbook is still running, so it should carry on as soon as puppet is re-enabled on kafka-jumbo1010. [15:10:51] I will give this one host another restart of kafka after the resto of thosts have finished rebooting. [15:11:06] btullis: ook running puppet now on all kafka jumbos [15:11:30] all the other clusters are done, so I am reasonably sure it the new certs will be picked up fine [15:12:07] btullis: done! [15:12:26] Triffic! Thanks. [15:17:44] topranks, _joe_ all clusters done except jumbo that is in progress, also all haproxykafka restarted etc.. [15:18:06] I am going to step afk in a few for an errand but in case of problems I have phone+laptop with me [15:18:37] <_joe_> elukey: thanks [16:32:23] <_joe_> elukey, btullis what's the situation with kafka-jumbo? it looks like the certs are in critical state rn [16:33:55] _joe_: kafka-jumbo is still mid-rolling-reboot. Currently up to kafka-jumbo1016 have been rebooted. kafaka-jumbo1010 was rebooted before the keystore was replaced, so I will restart the service once more at the end. [16:34:18] <_joe_> any idea why alert-manager is reporting expiring certs? [16:35:20] only on 1010 and 1017-1018 [16:35:30] If I'm reading it correctly. [16:35:52] <_joe_> yeah sorry it was more a few mins ago :) [16:35:55] It's because the service didn't get a rolling restart, independently of the reboot. [16:36:07] Cool. Thanks for checking. [17:44:55] weird that the certs were in critical state, but afaics everything is good