[09:35:56] <jynus>	 it seems that db1048 lagged tonight- probably will do it more weeks, as they do dumps/OLAP on the slave
[09:36:35] <jynus>	 they should either downtime the lag or we should increase the lag tolerance
[09:41:57] <volans>	 agree, can you remind me the usage for the "m" and "x" shards?
[09:42:30] <jynus>	 s == shard
[09:42:41] <jynus>	 m == misc
[09:43:08] <jynus>	 es == external storage
[09:43:46] <volans>	 x == ?
[09:44:21] <jynus>	 x = Extension Store
[09:44:30] <volans>	 thx :)
[09:44:31] <jynus>	 I had to look that up for the official repo
[09:44:36] <jynus>	 *name
[09:44:51] <jynus>	 names are not important
[09:45:06] <jynus>	 I vagely defined them on the slides I shared with you
[09:45:16] <jynus>	 but s == core relational data
[09:45:50] <jynus>	 m == non mediawiki services, with different degrees of impact (but requiring HA)
[09:46:02] <jynus>	 e.g. wordpress, puppet, gerrit, openstak
[09:46:14] <jynus>	 think, "internal services"
[09:46:18] <volans>	 ok
[09:46:24] <volans>	 es I know :)
[09:46:46] <jynus>	 x was, at the time, mostly created for storage needs
[09:47:06] <jynus>	 now it is a division based mostly on privacy and access patterns
[09:51:07] <jynus>	 es > s > x in terms of impact
[09:51:32] <jynus>	 with the note that es almost never fails
[09:51:48] <jynus>	 becase it is a pure key-value store with primary key access
[09:52:04] <jynus>	 while s is a purely relational system with complex queries
[15:56:35] <volans>	 I'm working with paupaul on es2012
[15:57:27] <jynus>	 great!
[15:58:02] <volans>	 there is a chance that the stripe can be changed without rebuild
[15:59:02] * volans disabled notification on icinga for es2012
[15:59:05] <jynus>	 great! that was my biggest concern
[15:59:15] <jynus>	 :-D
[16:39:55] <volans>	 no luck, we need to reimage as expected
[19:44:27] <volans>	 so thanks to papaul all servers are reimaged, I'll do a pass to check them after dinner
[19:44:54] <jynus>	 thats very fast!
[19:45:04] <jynus>	 congrats!
[19:45:20] <volans>	 j.ynus do you want to test XFS formatting too? I'd say that given current timeframe and no current errors/bad performances we might skip
[19:45:23] <volans>	 thoughts?
[19:45:41] <jynus>	 not sure why not ping me, I am aswering you now
[19:46:06] <jynus>	 aka == I am here
[19:46:29] * volans is lost... 
[19:46:32] <volans>	 what do you mean?
[19:46:40] <jynus>	 j.ynus
[19:46:50] <jynus>	 or was just a typo?
[19:47:05] <volans>	 was not urgent given that I was leaving for dinner :)
[19:47:15] <jynus>	 ok
[19:47:20] <jynus>	 in any case
[19:47:30] <jynus>	 skip it
[19:47:45] <volans>	 +1
[19:47:56] <jynus>	 your idea, I am just supporting it
[19:48:32] <jynus>	 see you next week?
[19:48:38] <volans>	 another failed disk on db1021... was not the same you changed 2 disks?
[19:48:54] <jynus>	 it is probably them being rebuilt
[19:49:02] <jynus>	 I asked chris to change a pair
[19:49:10] <volans>	 ok, make sense
[19:50:07] <jynus>	 I discover the other day that icinga says the same error on failed disk and on rebuilding
[19:50:27] <volans>	 not cool
[19:50:53] <jynus>	 send a patch!
[19:51:07] <volans>	 open a ticket :)