[08:13:55] <marostegui>	 jynus: I have installed 10.1.33-1 on db1120 and I am having some issues while starting systemctl start mariadb@s2 not sure if it is related to the package and the new systemd unit, just mentioning
[08:13:59] <marostegui>	 I am checking it
[08:14:34] <jynus>	 did you go over my checklist?
[08:15:12] <marostegui>	 the one in our etherpad?
[08:15:46] <marostegui>	 I went thru that one, and I am checking why puppet didn't create /run/mysqld
[08:16:08] <jynus>	 it is created on reboot
[08:16:24] <marostegui>	 Server reboot?
[08:16:28] <jynus>	 it cannot be created on package start because of the bug
[08:16:30] <jynus>	 yes
[08:16:32] <marostegui>	 riiight
[08:16:33] <marostegui>	 ok
[08:16:41] <marostegui>	 let's see
[08:16:43] <jynus>	 it cannot be fixed until buster
[08:16:50] <jynus>	 or it will create the issues I reported
[08:21:36] <jynus>	 I guess I could add, additionally "ExecStart=/usr/bin/systemd-tmpfiles --create"
[08:25:42] <marostegui>	 worked fine on reboot indeed
[08:27:01] <jynus>	 I will add ExecStartPre=/bin/systemd-tmpfiles --create
[08:27:12] <jynus>	 until there is a better option available
[08:27:15] <marostegui>	 Cool!
[08:29:20] <jynus>	 that wouldn't work either
[08:31:54] <jynus>	 I can create some workarounds
[08:32:09] <jynus>	 but I think it is best to stick to the current method- reboot
[08:32:17] <jynus>	 until a newer systemd
[08:32:18] <marostegui>	 It is not a really big deal I think, it is just the first time it bite me, so I was a bit like: eh uh :)
[08:32:28] <marostegui>	 I would say do not spend time on it 
[08:32:38] <jynus>	 note I documented it on the checklist
[08:32:57] <jynus>	 and it should happen on a reimaged- where a reboot happens after puppet ran
[08:33:05] <marostegui>	 Yeah, I wasn't aware of the reboot part, I just added it
[08:33:06] <jynus>	 *not happen on a reimage
[12:48:09] <marostegui>	 Going to deploy a schema change on s3 with replication on codfw
[12:50:03] <jynus>	 noted
[12:50:14] <jynus>	 thanks for the comment!
[12:50:25] <jynus>	 it is helpful
[12:50:36] <marostegui>	 :)
[13:58:56] <jynus>	 db1105@s2 started
[13:59:03] <jynus>	 s1 says it is corrupted
[13:59:36] <marostegui>	 :__(
[13:59:55] <marostegui>	 so gtid isn't great haha
[14:00:29] <jynus>	 well, it was a powerdown (sigint) followed by a sigkill
[14:01:42] <marostegui>	 I was kidding :)
[14:02:12] <marostegui>	 maybe we can recover from backups and then do all the partitioning manually? :(
[14:02:20] <marostegui>	 or mydumper
[14:39:41] <marostegui>	 I am going to enable notifications on db1064, they are disable. Probably a left over from when moving it to x1
[14:40:18] <jynus>	 +1
[15:00:44] <jynus>	 running "recover_section.py --host db1105.eqiad.wmnet --port=3311 --password=$pass /srv/backups/latest/dump.s1.2018-05-15--17-00-02" on dbstore1001
[15:01:09] <marostegui>	 nice :)
[15:01:16] <marostegui>	 we will have to do the partioning afterwards, no?
[15:01:23] <jynus>	 yep
[15:01:28] <jynus>	 but that should be ok
[15:01:53] <jynus>	 I could edit the table definition in advance
[15:01:56] <jynus>	 but the other time
[15:02:07] <jynus>	 it takes so much time, that I prefer to catch up replication
[15:02:13] <jynus>	 and then partition/compress
[15:02:23] <jynus>	 in this case the compression will happen by default
[15:02:26] <marostegui>	 yeah, agreed
[15:02:55] <jynus>	 if I was in a hurry I could copy from codfw
[15:03:02] <jynus>	 but we are ok for now
[15:03:46] <marostegui>	 yeah, we are fine
[15:04:03] <jynus>	 on the bright side, we may have a working transfer.py :-)
[15:04:34] <jynus>	 it took 3 days to import to dbstore1001
[15:04:45] <jynus>	 it probably will take just 12 hours on db1105
[15:04:53] <jynus>	 it could have been much worse!
[15:05:50] <marostegui>	 and we get to test the backups again!
[15:05:51] <marostegui>	 :)
[15:06:07] <jynus>	 I have been recovering from them for a while
[15:06:16] <jynus>	 when building new hosts
[15:06:38] <jynus>	 db2067 story doesn't end
[15:07:22] <marostegui>	 the new disk is still recoverying
[15:07:22] <marostegui>	 :)
[15:09:32] <wikibugs>	 10DBA: Failover s2 primary master - https://phabricator.wikimedia.org/T194870#4212030 (10Marostegui)
[15:10:24] <jynus>	 db1106 should be finnaly up again
[15:11:16] <marostegui>	 \o/
[15:28:02] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1064 - https://phabricator.wikimedia.org/T194885#4212056 (10Marostegui) 05Open>03Resolved a:03Marostegui This is now fixed, I am going to fail the other disk and a new task will be created  ```  Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id:...
[16:16:31] <jynus>	 I am also running a quick compare.py on s2 to check db1115 really went up without corruption