[09:41:42] <marostegui>	 I am removing lots of old grants from misc hosts, they are ips that aren't resolving anymore, but in case you see something strange there, please ping me
[10:01:24] <marostegui>	 Switching es5 eqiad master
[10:17:04] <jynus>	 I will be doing a change of the dump user starting this week to reduce the grants and point to only the new dbprovs
[11:04:01] <marostegui>	 Can I get a sanity check on this change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1111589 ?
[11:06:34] <jynus>	 checking
[11:08:30] <marostegui>	 thanks!
[11:08:42] <jynus>	 looks good
[11:09:16] <marostegui>	 thank you, deplying!
[11:09:21] <marostegui>	 deploying
[11:13:00] <jynus>	 do you happen to know if wmf-mariadb106 is the default config for bullseye on puppet or do I need to set the package?
[11:13:05] <jynus>	 sotty
[11:13:08] <jynus>	 sorry
[11:13:10] <marostegui>	 it is
[11:13:11] <jynus>	 I meant bookworm
[11:13:15] <marostegui>	 Yeah, it is
[11:13:19] <jynus>	 thanks
[11:14:58] <jynus>	 oh, I see, I wasn't aware of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1100068
[11:15:15] <jynus>	 so I was confused why it looked like it was setup but only partially
[12:04:19] <marostegui>	 jynus: It was done only partially 🤷
[12:54:34] <marostegui>	 Amir1: https://phabricator.wikimedia.org/T383689 is this for me?
[12:55:05] <Amir1>	 the codfw is for you 🥺 This one is more of "whoever finds the time"
[12:55:57] <marostegui>	 Yeah, the codfw I will do it next tuesday
[12:56:11] <marostegui>	 I can do this one now if you like, I have some time now
[12:56:22] <Amir1>	 that'd be amazing
[12:56:24] <Amir1>	 thanks <3
[13:08:15] <marostegui>	 Amir1: Done, let me reboot,upgrade and rebuild tables before I can give the old master back to you?
[13:08:38] <Amir1>	 sure!
[13:08:42] <Amir1>	 thanks for the reboot
[13:25:58] <marostegui>	 Amir1: db1163 rebooted and currently rebuilding tables - I will get back to you when it is ready for you
[13:26:11] <Amir1>	 thanks <3 it should take a couple of hours
[13:37:34] <marostegui>	 pagelinks is 100G so I think it will take a bit more than 2h :)
[13:37:42] <marostegui>	 anyway - will ping you
[13:47:00] <jynus>	 I got this notes after mysql_upgrade: "note     : Auto_increment will be checked on each open until CHECK TABLE FOR UPGRADE is executed"
[13:48:17] <jynus>	 is this related to a fix for T375652 ?
[13:48:27] <stashbot>	 T375652: Wikimedia\Rdbms\DBQueryError: Error 1062: Duplicate entry '1' for key 'PRIMARY' Function: MediaWiki\CheckUser\Services\CheckUserLogService::addLogEntry - https://phabricator.wikimedia.org/T375652
[13:49:38] <marostegui>	 That is part of the new mariadb_upgrade script 
[13:50:41] <jynus>	 oh, so this is a check that is done on mysql_upgrade, if I understood it well?
[13:51:09] <jynus>	 as I get the same output if I do "CHECK TABLE actor FOR UPGRADE"
[13:52:15] <marostegui>	 Yeah, it is the same
[13:53:12] <jynus>	 this is weird, because I would expect it to go away after I do the command they suggest
[13:53:19] <jynus>	 (the note)
[13:53:43] <jynus>	 "Finally the OPTIMIZE TABLE made the problem go away."
[13:54:06] <jynus>	 I see on internet, which means the message is a bit missleading- it suggest rebuilding the table, not just checking it
[13:54:35] <marostegui>	 We are rebuilding them
[13:55:17] <jynus>	 yeah, I am "complaining" about the message being unclear, not our action
[13:56:32] <jynus>	 I want to make sure I understand everything and I do it correctly before I do it on more dbs
[13:57:23] <jynus>	 this is a nice resource: https://fromdual.com/mariadb-and-mysql-upgrade-problems
[13:58:59] <jynus>	 indeed I confirmed it goes away after rebuilding
[16:03:04] <marostegui>	 Amir1: Is your work on pc5 stopped? I am going to do some work there tomorrow to keep reorganizing things
[16:04:23] <Emperor>	 [home wifi AP replaced, without toooo many sharks]
[18:08:28] <Amir1>	 marostegui: I've stopped the script for now, it can wait until the changes are done
[18:09:35] <marostegui>	 Amir1: thanks 
[18:09:46] <marostegui>	 Amir1: db1163 is all yours, it's pooled
[18:09:52] <Amir1>	 awesome
[18:16:26] <jinxer-wm>	 FIRING: [13x] SystemdUnitFailed: apt-daily-upgrade.service on ms-be2075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[18:17:05] <Emperor>	 yeah, that system is badly broken T382707 and T383530
[18:17:06] <stashbot>	 T382707: Frequent disk resets on ms-be2075 - https://phabricator.wikimedia.org/T382707
[18:17:06] <stashbot>	 T383530: Degraded RAID due to failed sdy on ms-be2075 - https://phabricator.wikimedia.org/T383530
[18:17:25] <Emperor>	 current catch-22 is it's too broken to run apt, and Dell want an sosreport...
[22:16:26] <jinxer-wm>	 FIRING: [13x] SystemdUnitFailed: apt-daily-upgrade.service on ms-be2075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed