[07:20:11] <jynus>	 I'm going to start deploying new grants for dbprov[12]007, doesn't touch core hosts, but it does production for sections without dedicated backup sources
[07:59:25] <jynus>	 could I ask for a +1 here? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1191585
[08:05:55] <Emperor>	 👀
[08:07:11] <jynus>	 be careful, highly impacting to production change
[09:16:03] <jynus>	 I created T405711 with technically doesn't block me, but makes me very slow
[09:16:04] <stashbot>	 T405711: cumin2002 and cumin1003 doesn't have grants to be able to administrate all databases that require backups - https://phabricator.wikimedia.org/T405711
[09:16:06] <jynus>	 for the dbas
[09:18:53] <jynus>	 s/with/which/
[09:44:47] <jynus>	 Also, I saw that https://zarcillo.wikimedia.org/ui/sections was showing incorrect sections for misc hosts
[09:45:02] <jynus>	 it show that db1207 is the master, but it is not
[09:45:59] <jynus>	 it is ok if it is WIP, but I'd prefer to remove that if it is not yet ready to prevent accidents
[10:03:23] <jynus>	 https://i.imgflip.com/a79wbl.jpg
[10:10:46] <jynus>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1191455
[10:11:34] <jynus>	 I just need a sanity/syntax check
[10:11:47] <jynus>	 specially for site.pp
[10:16:44] <Emperor>	 👀
[10:44:51] <jynus>	 ❤
[10:59:05] <jynus>	 https://phabricator.wikimedia.org/T403166#11218833
[17:13:40] <jynus>	 I hate to be so right "it will be unlikely for both dcs to fail"
[17:14:04] <jynus>	 but one failed completely, despite being the hw, not anything related to puppet or automation
[17:28:54] <federico3>	 we have a predictive alert for db2231, looking
[17:30:13] <federico3>	 indeed https://grafana-rw.wikimedia.org/d/419f8741-4177-49bc-939d-6ee002ac9b70/mariadb-free-disk-space-predictions?orgId=1&from=now-2d&to=now&timezone=utc
[17:32:18] <federico3>	 it's a replica in x1 filling / due to /var/log/syslog being spammed by:
[17:32:20] <federico3>	 2025-09-26T00:00:41.329959+00:00 db2231 mysqld[1534807]: 2025-09-26  0:00:41 209632498 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
[17:33:29] <jynus>	 that probably wasn't properly upgraded
[17:34:34] <jynus>	 if it can be depooled, do it, run upgrade and restart the db server
[17:38:14] <federico3>	 it's Friday evening tho, just to be on the safe side I compressed the old syslog.1 file freeing 7.2 GB
[17:38:50] <jynus>	 if it is root it shouln't affect /srv
[17:39:16] <jynus>	 just file the a ticket, even if it is a blank one with a host name
[17:39:26] <federico3>	 yes, the issue is logs filling / (where /var/log/syslog* live)
[17:39:48] <jynus>	 so we don't forget on monday
[17:40:26] <jynus>	 have a nice weekend
[17:41:27] <federico3>	 you too
[19:23:35] <federico3>	 db2239 looks unhappy https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=%24__all&var-server=db2239&var-port=13313&from=2025-09-26T16%3A14%3A50.985Z&to=2025-09-26T19%3A22%3A18.019Z&timezone=utc
[19:24:35] <federico3>	 MariaDB Replica SQL: s3 WARNING slave_sql_state Slave_SQL_Running: No         did it not page?
[21:27:34] <Amir1>	 I think it's a backup source taking back up which during it stops replication