[05:14:49] <wikibugs>	 10DBA, 10Patch-For-Review: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) s6 finished compression on labsdb1012, so this host is ready to get all the other tables compressed like labsdb1009 is having at the moment.
[05:19:15] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[07:57:15] <marostegui>	 I am transferring /srv/backups/snapshots/latest/snapshot.s5.2019-05-13--23-09-00.tar.gz from dbprov2001.codfw.wmnet
[09:31:28] <jynus>	 so
[09:32:09] <marostegui>	 o/
[09:33:11] <marostegui>	 So I saw https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510453/
[09:35:44] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[09:38:44] <jynus>	 what do you think of https://gerrit.wikimedia.org/r/c/operations/puppet/+/510453 ?
[09:39:10] <jynus>	 also see https://gerrit.wikimedia.org/r/c/operations/puppet/+/510700
[09:39:39] <marostegui>	 I am checking the last one yeah
[09:39:42] <marostegui>	 THe first one looks good
[09:39:51] <marostegui>	 I think that is kinda what we agreed on
[09:39:54] <marostegui>	 (the schedule)
[09:40:32] <jynus>	 we may even make it pass one week, it depends
[09:40:45] <jynus>	 I am now creating some monitoring
[09:40:59] <jynus>	 I won't have time to fix more stuff, but I think monitoring is nice
[09:41:07] <marostegui>	 sure
[09:41:24] <jynus>	 I will test it, but you can revert if something is wrong
[09:41:25] <marostegui>	 can you also review the cheatsheet and add whatever you think is necessary? ie: re-launch snapshots if they fail
[09:41:36] <jynus>	 where?
[09:41:55] <marostegui>	 on the wikitech page
[09:42:02] <marostegui>	 the small section I created
[09:42:12] <marostegui>	 https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Backups_quick_cheatsheet
[09:42:25] <jynus>	 ok, I can add that
[09:42:49] <marostegui>	 thanks, that'll be helpful
[09:43:09] <jynus>	 so the most common issue I saw
[09:43:15] <jynus>	 is the transfer failing
[09:43:44] <jynus>	 which at the moment requires manual rm of the data on ongoing
[09:43:52] <jynus>	 what I do is go to zarcillo
[09:43:54] <jynus>	 and run
[09:44:22] <jynus>	 mysql.py -h db1115 zarcillo
[09:44:32] <jynus>	 select * FROM backups where start_date > now() - INTERVAL 1 day and type = 'snapshot' order by section;
[09:44:50] <marostegui>	 rm the data from the appropiated dbprov host, no?
[09:44:56] <jynus>	 yes
[09:44:57] <marostegui>	 the failed snapshot I mean
[09:45:01] <jynus>	 it will be on the metadata
[09:45:06] <jynus>	 maybe kill some process
[09:45:11] <jynus>	 it will depend on how it failed
[09:45:25] <jynus>	 once I saw the prepare stuck
[09:45:37] <jynus>	 but mostly, no process ongoing, just transfer failed
[09:45:56] <jynus>	 I don't normally retried when we had daily backups
[09:46:17] <marostegui>	 so you rm the data and then re-launch the snapshotting process?
[09:46:18] <jynus>	 for example if you run now "select * FROM backups where start_date > now() - INTERVAL 3 day and type = 'snapshot' order by section;"
[09:46:39] <jynus>	 you can see one s4 backup on codfw failed
[09:46:54] <jynus>	 (it will be more clear when monitoring is in place)
[09:47:11] <marostegui>	 let me check
[09:47:54] <marostegui>	 ah yep, it is not there
[09:48:22] <jynus>	 both monitoring, retries and prepare based on absolute path will make this unnecessary
[09:48:32] <jynus>	 this is just a FTY for now
[09:48:38] <jynus>	 *FYI
[09:48:52] <marostegui>	 ok
[09:49:01] <jynus>	 but I don't want to redo the backup system just before I go
[09:49:23] <marostegui>	 ofc!
[09:49:37] <marostegui>	 so, giving that s4 codfw failed, how would I proceed?
[09:49:38] <jynus>	 I thought I had more time, but yesterday I got distracted
[09:50:00] <jynus>	 normally, I wouldn't care (except the deletion/kill process)
[09:50:29] <jynus>	 as if there is an ongoing file/dir, backup stops
[09:50:37] <jynus>	 it will be run again
[09:50:47] <jynus>	 otherwise, e.g. many fail, or you want to do it
[09:50:55] <marostegui>	 ah, so there is a retry already?
[09:51:00] <jynus>	 no
[09:51:06] <jynus>	 just, it used to happen every day
[09:51:10] <marostegui>	 ah hehe :)
[09:51:13] <jynus>	 and latest is never deleted
[09:51:18] <marostegui>	 right
[09:51:21] <jynus>	 unless it is successful
[09:51:36] <jynus>	 e.g. if you go to 
[09:52:24] <jynus>	 dbprov2001:/srv/backups/snapshots/latest
[09:52:49] <jynus>	 you will see snapshot.s4.2019-05-12--22-40-35.tar.gz which is a day later than the others
[09:52:58] <jynus>	 but as long as it is there, no big deal
[09:53:19] <marostegui>	 ah I see
[09:53:28] <jynus>	 only archive gets purged
[09:53:37] <marostegui>	 so now that there will no snapshots being generated everyday, what's your recommendation?
[09:53:40] <jynus>	 so mostly cleanup (due to a bug)
[09:53:43] <marostegui>	 generate it manually?
[09:53:47] <jynus>	 yeah, that is a more interesting way
[09:53:54] <jynus>	 I mean, first try to understand why
[09:53:58] <marostegui>	 sure
[09:54:06] <jynus>	 but let's say you know (e.g. host had to be put offline)
[09:54:23] <jynus>	 runing what the cron does is enough, that is
[09:55:11] <jynus>	 this: /usr/bin/python3 /usr/local/bin/remote_backup_mariadb.py
[09:55:28] <jynus>	 that will do a backup of everthing on /etc/mysql/backup.cnf
[09:55:29] <marostegui>	 will that run for all the snapshots?
[09:55:32] <marostegui>	 ah right
[09:55:37] <marostegui>	 so same behaviour as we had with dumps
[09:55:38] <jynus>	 if you just want one, you can edit it
[09:55:42] <marostegui>	 so I should generate a fake config
[09:55:43] <marostegui>	 right
[09:55:43] <jynus>	 the config file
[09:55:46] <marostegui>	 good
[09:55:48] <jynus>	 there is no --config option
[09:55:55] <marostegui>	 ok
[09:56:09] <jynus>	 run it, then let puppet revert the file
[09:56:33] <jynus>	 if you do it like that, logging will be to stout
[09:56:41] <marostegui>	 (please all this process, to the cheatsheet, as it is easier to read than irc logs when you are busy in the middle of the week alone! - doesn't have to be super formatted, just copy paste all these commands)
[09:56:50] <jynus>	 normally logging on cumin goes to journalctl
[09:57:02] <jynus>	 and the local mariadb_backup of dbprovs
[09:57:07] <marostegui>	 ah cool
[09:57:13] <jynus>	 to /var/log/mariadb-backups
[09:57:28] <jynus>	 so you can see if the transfer failed
[09:57:30] <jynus>	 or the prepare
[09:57:33] <jynus>	 or the compression
[09:57:35] <jynus>	 etc.
[09:58:01] <marostegui>	 ah good
[09:58:03] <marostegui>	 that's key!
[09:58:05] <jynus>	 I don't have any undertstanding of something that fails constantly
[09:58:25] <jynus>	 but with 2 failures I see transfer can be sometimes fragile?
[09:58:32] <jynus>	 mariabackup?
[09:58:55] <jynus>	 need more failures to find a pattern
[09:59:01] <marostegui>	 yeah 
[09:59:04] <marostegui>	 hard to say with not many
[09:59:08] <marostegui>	 it could be pretty much anything
[09:59:32] <jynus>	 I think only the ones that failed where the ones that did not stop replication, so could be it?
[09:59:54] <marostegui>	 could be
[10:00:04] <jynus>	 xtrabackup log sadly has to be supressed, because otherwise it overloads the pipe
[10:00:06] <marostegui>	 you added the stop slave to all of them? just to compare?
[10:00:16] <jynus>	 no, but feel free to experiment
[10:00:44] <jynus>	 that is a backups-cumin1001.cnf.erb et al
[10:00:50] <jynus>	 that is easy to manipulate
[10:00:56] <jynus>	 you can also add misc
[10:01:08] <jynus>	 they should work, but haven't added them yet
[13:57:07] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui)
[13:57:47] <jynus>	 issues with s7 response time
[13:58:07] <jynus>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?panelId=11&fullscreen&orgId=1&from=1558011480920&to=1558015080920&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All
[13:58:14] <marostegui>	 is there train on going?
[13:58:28] <marostegui>	 oh wow
[13:59:25] <marostegui>	 T222772
[13:59:25] <stashbot>	 T222772: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772
[13:59:32] <marostegui>	 https://logstash.wikimedia.org/goto/7af37f2092d90c485dc957cafca14ac7
[13:59:33] <marostegui>	 that
[14:01:09] <marostegui>	 it matches the spike of 500s on -operations
[14:01:11] <marostegui>	 on all the DCs
[14:03:03] <jynus>	 query and connection latency from localhost looks fine
[14:03:46] <marostegui>	 I am checking individual hosts on s1 and they look ok so far
[14:05:00] <marostegui>	 it is now recovered
[14:06:07] <jynus>	 I am trying to see if it was only one host, and which ones
[14:07:32] <jynus>	 at the very least db1079 involved
[14:07:51] <marostegui>	 s1 ones were ok
[14:07:53] <jynus>	 it hit 3000 connections
[14:08:09] <marostegui>	 that's s7 no?
[14:08:14] <jynus>	 yes
[14:08:33] <marostegui>	 s1 I didn't see any spike on any hosts
[14:08:39] <marostegui>	 maybe s7 overloaded the rest (centralauth?)
[14:08:58] <jynus>	 threads running was 0 before on db1086
[14:26:29] <cmjohnson1>	 for anyone affected by the pdu swap in b5 I am ready to remove power on 1 side 
[14:43:04] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[15:36:38] <jynus>	 https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=db1115
[15:37:04] <marostegui>	 sweet
[15:37:58] <jynus>	 probably in the future we can create a custom dashboard on zarcillo and just leave there a single alert
[15:42:45] <cmjohnson1>	 marostegui: disk on db1133 has been replaced
[15:43:09] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[15:43:20] <marostegui>	 cmjohnson1: lovely! can you destroy+create the raid there?
[15:43:36] <jynus>	 cmjohnson1: while I have you here, could you check power on dbproxy1006? 
[15:43:55] <jynus>	 it is up, but it seems one of the power supplies may be disconnected or toasted
[15:44:33] <jynus>	 I can put it down if needed
[15:45:07] <jynus>	 (since 25 minutes ago)
[15:45:24] <cmjohnson1>	 jynus fixed...sorry the new pdus require a little more force 
[15:45:30] <jynus>	 it is ok
[15:45:37] <jynus>	 that is why we have alerting :-D
[15:47:32] <jynus>	 thanks, I see it up
[15:50:37] <marostegui>	 jynus: ping me once you've started mysql on db1131 please
[15:50:43] <marostegui>	 no rush, just let me know when done
[15:51:34] <jynus>	 we may wait until tomorrow
[15:51:43] <jynus>	 see dcops
[15:54:47] <marostegui>	 I am not there, but that is ok with me
[16:04:27] <marostegui>	 cmjohnson1: do you want me to check if the raid has rebuilt fine and ping you if we need you to destroy+create?
[16:04:42] <cmjohnson1>	 i am rebuilding it now
[16:04:58] <marostegui>	 nice!
[16:06:04] <cmjohnson1>	 marostegui ...there is a bigger issue now....now all but 1 disk is reporting bad including the newly replaced disk. this will required more work and I am in the middle of other things. I don't have spare cycles at the moment 
[16:06:43] <marostegui>	 cmjohnson1: don't worry, we can keep working with the other hosts, but sounds like we need a new controller from dell?
[22:23:35] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Papaul)
[22:24:33] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Papaul) p:05Triage→03Normal