[13:12:40] <jynus>	 hoo, do you run something on snapshot1006?
[13:12:46] <hoo>	 no
[13:12:52] <jynus>	 "you" as in "wikidata-specific"
[13:12:53] <hoo>	 Just caught that
[13:12:55] <jynus>	 backups
[13:13:06] <hoo>	 no, Wikidata stuff is on snapshot1003 still
[13:13:13] <jynus>	 because I obverved a strange behaviour there
[13:13:30] <jynus>	 and wanted to ask you or ariel before filing a bug
[13:13:39] <jynus>	 mmmm, strange
[13:13:49] <hoo>	 What did you observe? Was it Wikidata related?
[13:13:57] <jynus>	 yes, kinda
[13:13:58] <hoo>	 Could be that Ariel runs the xml dumps for wikidata from there
[13:14:20] <jynus>	 the issue is that during failover, dumps were not changed
[13:14:26] <jynus>	 (which is ok)
[13:14:51] <jynus>	 but we observed some backups jobs (in particular, wikidata ones)
[13:15:02] <jynus>	 from that host running on non-dump hosts
[13:15:14] <jynus>	 *non-dump mysql hosts
[13:15:45] <hoo>	 that's weird. It's known (and wanted) that the Wikidata json/ ttl dumpers use regular slaves
[13:16:01] <hoo>	 as they have query patters very similar to what we do during web requests/ in the API
[13:16:09] <hoo>	 They use the same code infact
[13:16:19] <hoo>	 but that's on snapshot1003
[13:16:23] <hoo>	 (running right now)
[13:18:50] <jynus>	 with regular, you mean non-backup specific?
[13:19:20] <jynus>	 anyway, if there is nothing wikidata-specific, I will ask ariel
[13:19:23] <jynus>	 thank you!
[13:19:50] <jynus>	 maybe there is a bug on some script, quering the wrong host
[13:20:50] <hoo_>	 <hoo> that's weird. It's known (and wanted) that the Wikidata json/ ttl dumpers use regular slaves
[13:20:54] <hoo_>	 <hoo> as they have query patters very similar to what we do during web requests/ in the API
[13:20:55] <hoo_>	 <hoo> They use the same code infact
[13:20:58] <hoo_>	 <hoo> but that's on snapshot1003
[13:20:59] <hoo_>	 <hoo> (running right now)
[13:21:02] <hoo_>	 Weird... I lost IPv6 connectivity
[13:21:59] <jynus>	 (I read you, my answer is on the log: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-databases/20160425.txt
[13:22:55] <hoo_>	 Ah nice... forgot that this channel is logged
[13:23:31] <jynus>	 (now you cannot say aything wrong about me when I am not around :-P)
[13:24:17] <hoo>	 :D
[18:07:46] <jynus>	 db1052 is now on jessie
[18:08:02] <jynus>	 db2047 pending reimage
[18:08:08] <jynus>	 things seem stable
[18:08:13] <jynus>	 labs lag going down
[18:35:37] <jynus>	 es2019 same issue
[19:14:34] <volans>	 jynus: if you're still around by any chance...
[19:31:24] <volans>	 I was looking at the space issue on db1047 and found some strange things...
[20:13:30] <volans>	 see T133588 for updates
[20:13:30] <stashbot>	 T133588: Gerrit 285208 broke eventlogging_sync.sh - https://phabricator.wikimedia.org/T133588
[20:59:11] <jynus>	 paravoid^