[08:15:34] jynus: I gather we are at a rather ok spot now with backup1001, right? Is there anything I am blocking you on or anything I can help with (and I have missed it)? [08:18:11] I may have questions here and there [08:18:38] but the bulk of the migration is done, I just have to do all the finishing steps and backup2001, etc. [08:19:06] archive, etc, but I have no blockers [08:19:27] also setup db-only array [08:19:33] lots of finishing things [08:22:40] (I am just busy with db maintenance while manuel is out, so it is going slower than I expected) [08:23:30] gather some starts, possibly increase retention period [08:23:53] lots of immediate followups, you will probably agree with, akosiaris? [08:24:39] (I just want to have your blessing so you can focus on your own goals) [08:30:33] * akosiaris blesses jynus [08:31:26] s/starts/stats/ [10:16:13] 10DBA, 10Schema-change: Remove globalblocks tables from wikis - https://phabricator.wikimedia.org/T230055 (10jcrespo) 05Resolved→03Open The table seems to exist on napwikisource. I wonder if the table is created on install, or was just a one time mistake? ` root@db1123[ptwikimedia]> select * FROM informati... [10:16:17] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10jcrespo) [11:14:11] jynus: o/ - qs about the new analytics bacula job for an-master1002.. how long does it take to show up in bconsole? Just wanted to triple check that I didn't make mistakes etc.. [11:14:43] define show up? [11:14:53] there is already one for an-master1002 but it is for another thing (the namenode's fs image backup) [11:15:14] I am checking in bconsole show client=etc.. [11:15:18] one sec I am in the middle of something and I check [11:15:19] on backup1001 [11:15:30] even later on, no hurry [11:16:20] nevermind, I can see it with shob job=an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup [11:16:25] :) [11:16:25] No backups: 1 (an-master1002), Fresh: 92 jobs [11:16:32] see alert on warning [11:16:41] it is a warning because the first backup has not yet run [11:16:53] you can force a manual run with the command run [11:17:24] I will have more time in 20 minutes or so [11:17:27] nono will wait, I was just triple checking that I didn't make mistakes [11:17:30] thanks :) [11:18:27] check Bacula#Monitoring on wikitech on more commands for debugging I just created [11:24:26] elukey: I've checked and it is scheduled for tomorrow, but I can run it now [11:25:34] an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup is running [11:26:21] it goes relatively fast [11:26:35] thanks! [11:27:45] when finished you can run on shell command line: check_bacula.py an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup [11:28:03] and it will give you some basic stats [11:32:19] elukey: 159834 Full 781 19.16 G OK 06-Nov-19 11:28 an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup [11:32:44] check_bacula.py an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup [11:32:46] 2019-11-06 11:25:19: type: F, status: T, bytes: 19,166,539,616 [11:33:33] you can try to recover if you want, now there bacula is idle [11:33:44] *that [11:38:18] wow nice! [11:39:59] everybody should get familiar with the restore process, because if you have to restore, the last thing you want to do is read and understand it under pressure [11:41:12] I already did one via bconsole following wikitech for Archiva and it took a bit, you are right [16:04:40] 10DBA, 10Schema-change: Remove globalblocks tables from wikis - https://phabricator.wikimedia.org/T230055 (10Reedy) >>! In T230055#5639656, @jcrespo wrote: > The table seems to exist on napwikisource. I wonder if the table is created on install, or was just a one time mistake? > ` > root@db1123[ptwikimedia]> s... [16:05:14] jynus: as a followup to that confusing labtestwikitech thing last week, James wrote a patch but suggests I get a DBA to sign off: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/547596/ [16:05:29] (But no need to interrupt what you're doing now) [16:05:31] looking [16:05:46] thank you! [16:05:53] s-wikitech? [16:06:04] And I guess also https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/547597/ [16:06:21] I think there was concern that you might dislike the naming scheme :) [16:06:46] it's ugly [16:07:01] I suggest you wait for next week's manuel ok [16:07:08] ok [16:07:24] if you prefer different names in the meantime, I'm happy to change [16:07:27] I am ok with the labstestwiki split [16:07:52] please note that will require changes on conftool [16:08:26] can you add your notes to the patch? I'm not likely to be the primary deployer when it comes time to roll out. [16:08:28] or I think you may have overriden that [16:08:40] yeah, the test wiki doesn't use etcd [16:08:45] for load balancing [16:11:58] thank you jynus [17:31:21] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10jbond) @Eevans moritz mentioned there maybe some cassandra consideration to take into account and you could enlighten me as to what they are :) [18:05:42] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) [18:06:27] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) [18:09:46] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10CDanis) [18:10:05] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10CDanis) [18:12:15] [18:12:15] except BlockingIOError as error: [18:12:15] [18:12:15] [18:12:21] sorry :)