[05:47:02] <wikibugs>	 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) Absolutely! Thanks
[05:55:06] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) s3 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1124 [] db1123 [] db1112 [] db1095 [] db107...
[06:26:03] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui)
[07:50:12] <wikibugs>	 10DBA, 10Analytics, 10Patch-For-Review: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1107.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimag...
[08:49:42] <wikibugs>	 10DBA, 10Analytics: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1107.eqiad.wmnet'] `  and were **ALL** successful.
[09:36:47] <marostegui>	 jynus: going to stop replication on db1139:3311 to use transfer.py type=xtrabackup
[09:37:43] <jynus>	 ?
[09:37:52] <jynus>	 copying to where?
[09:37:59] <marostegui>	 to db1107
[09:38:12] <jynus>	 cannot you use the db provisioning servers?
[09:38:35] <marostegui>	 Sure, I just wanted to use xtrabackup as it's been a while
[09:38:41] <marostegui>	 But yeah, I can use dbprov too
[09:39:00] <jynus>	 so no problem with that, but the db provisioning ones were created to, well, provision dbs
[09:39:27] <marostegui>	 I forgot dbprov also have a tar.gz that will require xtrabackup prepare :)
[09:39:32] <marostegui>	 Will use dbprov!
[09:39:40] <jynus>	 no, they are pre-prepared
[09:39:49] <marostegui>	 ah, so just copy + start?
[09:40:03] <jynus>	 you just have to run transfer.py --type=decompress
[09:40:25] <marostegui>	 Cool, following https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Copy_data_from_a_backup_source_to_a_host then!
[09:40:51] <jynus>	 no
[09:40:56] <jynus>	 that is to generate a backup
[09:41:03] <jynus>	 (and you wrote that!)
[09:41:08] <marostegui>	 Provision a precompressed and prepared snapshot
[09:41:14] <jynus>	 that is the one
[09:41:35] <jynus>	 https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Provision_a_precompressed_and_prepared_snapshot
[09:41:36] <marostegui>	 ah sorry, I sent the link linking to the other one, hehe
[09:41:37] <marostegui>	 yeah
[09:41:37] <marostegui>	 that
[09:41:55] <jynus>	 there is also https://wikitech.wikimedia.org/wiki/Transfer.py for the full documentation
[09:42:44] <marostegui>	 I didn't remember I wrote that XD
[09:43:06] <jynus>	 just to be clear, you can do it in any way you want
[09:43:15] <jynus>	 just I am amusing you are not following your own guides
[09:43:19] <marostegui>	 hahaha
[09:43:20] <jynus>	 *amused
[09:43:25] <marostegui>	 Because I didn't even remember I wrote it!
[09:44:25] <jynus>	 the advantages being way faster and no need to stop/affect mysql servers
[09:44:41] <marostegui>	 yeah, definitely
[09:45:00] <marostegui>	 It was a while since I operated with the provisioning servers and backup transfers, so this is a good excercise :)
[09:45:07] <jynus>	 it is indeed cumbersome, but you know we don't have yet fully automated provisioning
[09:46:26] <jynus>	 see: https://wikitech.wikimedia.org/w/index.php?title=MariaDB/Backups&diff=next&oldid=1825423
[09:46:45] <marostegui>	 wow, may
[09:46:47] <jynus>	 and: https://wikitech.wikimedia.org/w/index.php?title=MariaDB/Backups&diff=next&oldid=1825646
[09:46:49] <marostegui>	 time flies
[09:48:43] <jynus>	 if you want to test xtrabackup, that is cool too- we have test-s1 hosts which I use a lot for that
[09:49:03] <jynus>	 e.g. I tested mydumper to recover test-s1 on eqiad
[09:49:14] <marostegui>	 ah nice, how long did it take?
[09:51:22] <jynus>	 I am not ignoring you, I am checking because I don't remember
[09:51:39] <marostegui>	 hahaha don't worry!
[09:53:12] <jynus>	 18 hours, compressed
[09:53:25] <marostegui>	 so 18 to load s1 entirely?
[09:53:34] <jynus>	 compressed
[09:53:47] <marostegui>	 that's pretty cool
[09:53:50] <jynus>	 into percona, which probably has lower perf due to not being optimized
[09:54:11] <jynus>	 e.g. if you want to test, testing myloader is also a good way
[09:54:38] <jynus>	 e.g. maybe disabling a lot of reliability options and disabling conpression
[09:55:27] <marostegui>	 yeah, I can test that too on 10.3 
[09:57:08] <jynus>	 for example, there is no automation to change engine/engine options before myloader
[09:57:23] <jynus>	 (e.g. to disable compression for a faster load)
[09:57:38] <jynus>	 that would be a cool side project if you wanted :-D
[09:58:35] <marostegui>	 are those kinda wishlist documented somewhere?
[09:58:59] <jynus>	 there are some, not sure if that is there
[09:59:03] <jynus>	 let me search the ticket
[09:59:24] <jynus>	 T138562
[09:59:25] <stashbot>	 T138562: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562
[10:00:00] <jynus>	 ^I will add it there
[10:00:22] <jynus>	 I wanted to work on database backup generation if I finished the general backup setup
[10:00:31] <jynus>	 but I don't think that is going to happen before january
[10:03:08] <wikibugs>	 10DBA, 10Epic: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562 (10jcrespo)
[10:03:18] <jynus>	 ^added
[10:04:28] <marostegui>	 thanks .)
[10:04:41] <marostegui>	 January is almost there, specially with all the public holidays we'll have
[10:04:57] <marostegui>	 I was checking the calendar yesterday and most of our december weeks will be 4 days or less
[10:05:02] <wikibugs>	 10DBA: Handle object metadata backups and compare it with stored database object inventory - https://phabricator.wikimedia.org/T205628 (10jcrespo)
[10:05:04] <wikibugs>	 10DBA, 10observability, 10Epic: Improve database alerting (tracking) - https://phabricator.wikimedia.org/T172492 (10jcrespo)
[10:10:53] <wikibugs>	 10DBA, 10Performance-Team, 10conftool: #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10Marostegui) Any rough ETA on when externalLoads will be able to be handled by `dbctl`?
[10:32:17] <wikibugs>	 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10ArielGlenn) Let's see what happens once this is in production.
[14:25:29] <wikibugs>	 10DBA, 10MediaWiki-General, 10Operations: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10Marostegui) db1107 is now running the latest 10.3.20 from MariaDB replicating from s1 master and db1114 (which r...
[14:26:23] <wikibugs>	 10DBA, 10Analytics: Repurpose db1107 as a generic database - https://phabricator.wikimedia.org/T238113 (10Marostegui) 05Open→03Resolved a:03Marostegui db1107 has been reimaged into buster and placed on test-s1 with MariaDB 10.3.20 with replicating from enwiki master and being an intermediate master for d...
[15:00:12] <wikibugs>	 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui)
[15:39:09] <marostegui>	 jynus: it seems that you did compile 10.4 :-) I just realised after compiling one myself, I noticed there was a control file there for 10.4
[15:39:10] <marostegui>	 Nice
[15:39:11] <marostegui>	 :)
[15:39:35] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/operations/software/+/550644/
[15:39:36] <marostegui>	 :)
[15:40:08] <jynus>	 is it on the repo?
[15:40:14] <marostegui>	 yep
[15:40:15] <jynus>	 I think I gathered some notes
[15:40:24] <marostegui>	 root@db1107:~# apt-cache show wmf-mariadb104
[15:40:25] <marostegui>	 Package: wmf-mariadb104
[15:40:25] <marostegui>	 Version: 10.4.10-1
[15:41:38] <jynus>	 actually, I don't have notes
[15:41:46] <jynus>	 other than do the same than with 10.3 on buster
[15:42:14] <marostegui>	 yeah, I didn't notice there was 10.4 there, and I was basically doing the same than with 10.3 to see what would fail, but so far so good
[15:42:21] <marostegui>	 the compilation and the rest are looking good for now
[22:07:02] <wikibugs>	 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiki - https://phabricator.wikimedia.org/T235743 (10Phamhi) 05Open→03Resolved a:03Phamhi The views step is done. Tested successfully on toolsforge.