[08:13:58] <arturo>	 hey DBA folks
[08:14:26] <arturo>	 a follow-up from yesterday: the secondary toolsdb (clouddb1002) seems to be running 10.1.41-1
[08:14:33] <arturo>	 shall I downgrade that one as well?
[08:33:50] <jynus>	 yes
[08:34:01] <jynus>	 brook was supposed to disable unatended upgrades
[08:34:16] <jynus>	 did you enable puppet?
[08:35:04] <jynus>	 mysql is running 10.1.38
[08:35:09] <jynus>	 please remove the package
[08:35:27] <jynus>	 and install 10.1.38
[08:35:52] <jynus>	 and remove unatended upgrades, etc
[08:37:23] <jynus>	 10.1.38 or 39 would work well
[08:49:31] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10MarcoAurelio)
[08:49:59] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10MarcoAurelio) Note: domain name changed from `kawikimedia` to `gewikimedia`.
[09:57:18] <wikibugs>	 10DBA, 10Data-Services, 10Operations: Prepare and check storage layer for ge.wikimedia.org - https://phabricator.wikimedia.org/T236404 (10jcrespo) Thanks, being public we DBAs are not a blocker, and we just need a heads up when it is finally deployed to the database.
[10:01:09] <hashar>	 jynus: thank you very much for the quick triage
[10:13:06] <jynus>	 triage? I did nothing
[10:13:15] <jynus>	 just pointed the existing issue
[10:15:17] <jynus>	 hashar: one trick it to look at T30599 for active deadlock cases
[10:15:18] <stashbot>	 T30599: Deadlock tracking bug (tracking) - https://phabricator.wikimedia.org/T30599
[10:30:25] <arturo>	 jynus: instead of disabling unattended-upgrades what I did was to apt pin the version we want to run
[10:32:58] <jynus>	 yeah, that works
[10:33:33] <jynus>	 although the new package is still installed
[10:33:49] <arturo>	 yes, in clouddb1002
[10:34:01] <arturo>	 where can I find the old package? is no longer in the repo it seems
[10:34:13] <jynus>	 I store older versions on my home on install1002
[10:34:27] <jynus>	 check if you can see them
[10:34:33] <arturo>	 looking
[10:35:14] <arturo>	 got it!
[10:35:18] <jynus>	 cool
[10:35:31] <arturo>	 wmf-mariadb101_10.1.39-1_amd64.deb
[10:35:48] <jynus>	 yeah, that is probably the best option
[10:36:00] <jynus>	 what a mess mariadb with 10.1 and 10.3
[10:36:04] <jynus>	 lots of crashes
[10:36:15] <jynus>	 in the latest minor versions
[10:36:16] <arturo>	 :-S
[10:36:58] <jynus>	 may I ask to survey if people would be too annoyed on cloud with an upgrade to mysql 8?
[10:37:18] <jynus>	 not saying it will happen, but it may be more stable (question mark)
[10:37:30] <arturo>	 no idea honestly
[10:37:40] <arturo>	 my knowledge of DB stuff is limited
[10:37:43] <jynus>	 that is why I said to survey, even informally
[10:37:54] <jynus>	 doesn't have to be done by you specifically
[10:38:07] <jynus>	 something to transmit to the team as a potential alternative
[10:38:36] <jynus>	 "this is happening, what would be the blockers for that, if things get worse?"
[10:38:38] <arturo>	 yeah sure, feel free to send an email to cloud-admin@l.w.o or phab tasks
[10:38:45] <jynus>	 ok
[10:39:12] <arturo>	 downgrading wmf-mariadb101 from 10.1.41-1 to 10.1.39-1 already done in clouddb1002
[10:39:31] <jynus>	 cool, it should install properly for the most part
[10:39:45] <jynus>	 and keep mysql up
[10:40:04] <arturo>	 yeah, the install went just fine
[10:40:06] <arturo>	 https://www.irccloud.com/pastebin/3hYWXt04/
[10:40:09] <jynus>	 we may need an upgrade
[10:40:16] <jynus>	 from 10.1.38 to 39
[10:40:25] <jynus>	 can you prepare alerting, I can do it in a second
[10:40:44] <jynus>	 need to bring down the server for some seconds
[10:41:15] <arturo>	 I know clouddb1002 is the secondary, but do you think is safe to do this on friday?
[10:41:33] <jynus>	 yes, it is not safe to leave it with mismatched version
[10:41:38] <arturo>	 ok
[10:41:45] <jynus>	 I wouldn't propose it if not
[10:42:25] <arturo>	 let me downtime a bunch of things
[10:42:36] <jynus>	 yes, that is why I asked
[10:43:42] <arturo>	 let see if we manage to avoid paging folks in the US
[10:43:51] <jynus>	 when you are done, ping me and log for me upgrading mariadb on clouddb1002 from 10.1.38 to 10.1.39
[10:44:02] <arturo>	 ack
[10:44:16] <jynus>	 (waiting for you ping)
[10:45:54] <arturo>	 I can't find anything else to icinga downtime other than toolschecker, so the downtime is done jynus 
[10:45:58] <arturo>	 please go ahead
[10:46:23] <jynus>	 ok, doing, make sure you log
[10:46:48] <arturo>	 done
[10:48:56] <jynus>	 ok, done, Seconds_Behind_Master: 0
[10:49:22] <jynus>	 https://grafana-labs.wikimedia.org/d/000000273/tools-mariadb?orgId=1&from=now-24h&to=now&var-dc=Tools%20Prometheus&var-server=clouddb1002.clouddb-services.eqiad.wmflabs&var-port=9104
[10:50:06] <arturo>	 ok! thanks!
[10:50:11] <jynus>	 you pinned that version, right?
[10:51:04] <arturo>	 yes
[10:51:05] <arturo>	 https://www.irccloud.com/pastebin/aDNS3cvZ/
[10:51:15] <arturo>	 apt ignores the new version
[10:51:18] <jynus>	 ok
[10:51:59] <arturo>	 see modules/profile/manifests/wmcs/services/toolsdb_apt_pinning.pp
[10:52:04] <arturo>	 for details
[10:52:36] <jynus>	 I consider the T236420 incident closed, we can discuss next steps on  T236399
[10:52:37] <stashbot>	 T236420: ToolsDB unstable following unplanned software upgrade - https://phabricator.wikimedia.org/T236420
[10:52:38] <stashbot>	 T236399: Upgrade mariadb on toolsdb servers to 10.1.42 as soon as it is available - https://phabricator.wikimedia.org/T236399
[10:53:02] <arturo>	 fair enough
[17:14:59] <wikibugs>	 10DBA, 10CPT Initiatives (Core REST API in PHP), 10Core Platform Team Workboards (Green): Compose query for minor edit count - https://phabricator.wikimedia.org/T235572 (10WDoranWMF) @jcrespo @Marostegui Would it be possible to get input DBA review on the above query and whether it's acceptable?
[20:01:12] <wikibugs>	 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 4 others: Allow setting the watchlist table to read-only on a per-wiki basis - https://phabricator.wikimedia.org/T160062 (10ifried)
[20:01:42] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, and 5 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067 (10ifried)