[06:51:37] Amir1: any chance db1274 is no up because of the reboots and it never came back up? [06:52:37] 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance [06:52:39] yep that one [06:53:03] mmm that log says db1174 but the host down is db1274 [06:53:18] Right: 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 (T441633)', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json [06:53:22] I will try to bring it back [06:56:57] Host back [09:06:10] thanks. The logs didn't show anything, I think it's a bug in the newer versions of auto schema [09:20:19] Amir1: are you using the rolling restart scripts? [09:22:32] is there an standard one now? [09:24:56] https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts these [09:25:28] https://wikitech.wikimedia.org/wiki/MariaDB/Upgrading_a_section [09:26:26] ... but ideally they should become a cookbook [09:51:09] claime: yes for x3 and x4 [09:51:33] in general you shouldn't be accessing anything that is not on 3306 as 33XX are multinstance hosts and those are for backups [09:53:57] marostegui: ah cool that also helps my attempt to add mw mariadb to k8s external services [09:54:38] so we only need to access 3306 on these server's ips and not any of the other ports [10:05:00] marostegui: will your future k8s tools need access to these though? [10:05:37] and I suppose in general, need access to every type of db host? [10:06:02] claime: meeting [10:06:26] ack not urgent [10:43:26] claime: I am not sure about the future, probably yeah [10:45:31] marostegui: ok i'll talk with cezmunsta about what's needed for ampinus and go from there then [10:46:18] claime: <3 [11:47:31] claime: the app code it mostly needs m1; it has a connection setting for db1215, but that seems as though it has been superseded by Zarcillo (the related data shown is rather outdated) [11:51:07] cezmunsta: ack so the app only needs access to m1 3306, does it need to access zarcillo? Currently we have various "hacks" to give access to db, for instance for mw it has egress to 10.0.0.0/8 port 3306, other places it's limited to 10.64.0.0/12 and 10.192.0.0/12 [11:52:05] do you see a reason to clamp down harder on that and use external_services with an actual list of server ips (meaning it would need an external_service redeploy when that list changes), or do we consider that Good EnoughTM [11:57:37] It may be wise to confine it, is that much more effort for you in terms of preparation? [11:59:09] As for Zarcillo, I would need to double-check the code - what I have seen so far is not of use though [15:42:33] cezmunsta: I'm trying to see if I justify the effort right now to get mariadb core hosts in external services, or if I just add a netpol in line with what we are already doing, or if you have a stable list of hosts for a netpol basically [15:51:01] claime: ack - well, this is just a dashboard app, which looks at dbbackups in m1, so if you want a list of IPs to cover that then "db[2160,2232].codfw.wmnet,db[1164,1213,1217].eqiad.wmnet" .. although only a single address is present in the settings [15:51:34] cezmunsta: ack