[06:27:33] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es2045:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:30:16] <marostegui>	 will downtime ^but expected
[06:30:27] <marostegui>	 Needs cloning which I will do on monday, because I don't want to leave es5 with no replicas for a weekend
[09:17:20] <moritzm>	 federico3: can you please have a look at https://phabricator.wikimedia.org/T427900#12004423 ? this blocks the setup of the new Cumin node on trixie
[09:19:01] <federico3>	 @moritzm I just requested enabling the trusted runners in gitlab... this might take a while
[09:22:51] <federico3>	 @moritzm if you know of a quicker way to get it done I'm all eyes
[09:23:40] <moritzm>	 ack, thanks. if the releng folks look into it and it's there sometime next week, that's great
[09:24:05] <moritzm>	 we still the changes to add the DB grants anyway
[09:24:14] <moritzm>	 I'll make patches for that later
[10:50:39] <marostegui>	 moritzm: the grants deployment are being worked out by cezmunsta 
[10:53:20] <moritzm>	 excellent, thanks!
[11:53:33] <cezmunsta>	 marostegui: remind me why we have grants defined on hosts  in the templated SQL file and what uses that file
[11:58:39] <marostegui>	 marostegui: just for tracking, nothing really uses it
[11:58:54] <marostegui>	 Hahaha cezmunsta ^
[12:04:02] <cezmunsta>	 Hmm, we store plaintext passwords for admin roles in an SQL file that isn't user except for tracking?  
[12:04:09] <cezmunsta>	 s/user/used
[12:05:48] <federico3>	 cezmunsta: the passwords are replaced with a template/formatter 
[12:09:31] <cezmunsta>	 federico3: IDENTIFIED BY '<%= @cumin_pass %>' puts a plaintext password on the host
[12:10:36] <cezmunsta>	 or isn't that what you were referring to?
[12:15:26] <federico3>	 I mean: the used for tracking is the one in the puppet repo and has no plaintext password in it 
[12:16:10] <cezmunsta>	 I am referring to /etc/mysql/production-grants-shard.sql
[12:16:23] <federico3>	 then for the deployment I've been using hashes in https://gitlab.wikimedia.org/ladsgroup/db-password-rotation/-/blob/main/user_grant_handler.py#L14
[12:17:02] <cezmunsta>	 Those hashes _can_ be broken iirc :) 
[12:17:22] <federico3>	 yes but they are only stored inside the databases, no files written on disk
[12:19:19] <federico3>	 also IIRC mariadb introduced safer hashes but still not very strong (but it's not an issue as long as they are in the DB)
[12:24:21] <jynus>	 that requires chaning authentication method
[12:24:27] <jynus>	 which we haven't
[12:24:46] <jynus>	 have a nice weekend!
[12:33:05] <cezmunsta>	 Cover your eyes: sudo cumin 'P{C:profile::mariadb::grants::production}' 'grep -Fc "IDENTIFIED BY" /etc/mysql/production-grants-shard.sql' 
[15:30:16] <federico3>	 I was able to extract silences from the alertmanager API and ... Matcher(name='instance', value='^(db\\-test1001|db\\-test1002|db\\-test1003|db\\-test2001|db\\-test2002)(\\..+)?(:[0-9]+)?$', is_regex=True, is_equal=True)