[04:50:31] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Physically move es1017 from D to C row - https://phabricator.wikimedia.org/T197072 (10Marostegui) Anything left after repooling the host?
[04:54:14] <wikibugs>	 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10jcrespo)
[04:54:18] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Physically move es1017 from D to C row - https://phabricator.wikimedia.org/T197072 (10jcrespo) 05Open>03Resolved a:03Cmjohnson I don't think so.
[09:10:28] <jynus>	 marostegui: I got a spike of errors on s6, probably replication
[09:10:45] <jynus>	 or could be just loadbalancer as usual
[09:10:47] <marostegui>	 in codfw
[09:10:49] <marostegui>	 I guess?
[09:10:53] <jynus>	 nope
[09:11:11] <marostegui>	 That is strange, I haven't done anything that could cause lag there
[09:11:15] <marostegui>	 all hosts?
[09:11:20] <jynus>	 8:52, eqiad
[09:11:37] <jynus>	 db1085 I think
[09:12:13] <marostegui>	 Interesting…that was the last host I did, not sure why it would have spiked
[09:12:16] <marostegui>	 I will check
[09:12:28] <jynus>	 something related to ROW format, maybe?
[09:12:44] <jynus>	 did as in, at that time, or yesterday?
[09:13:16] <marostegui>	 No, today, can't say it was around that time yeah
[09:13:27] <marostegui>	 It is not even on graphs, so it must have been something fast no?
[09:13:32] <marostegui>	 Could be related to ROW
[09:13:35] <marostegui>	 Somehow
[09:13:48] <jynus>	 https://logstash.wikimedia.org/goto/ca6ac1c091fb6c426202766d7faf1157
[09:13:54] <jynus>	 it seems for a single second
[09:14:05] <jynus>	 but who knows with the LB in the state it is- not reliable
[09:14:30] <jynus>	 also server db1093 is not replicating?
[09:14:40] <jynus>	 at the same time
[09:15:06] <marostegui>	 That is weird, because I am touching one at the time
[09:15:20] <marostegui>	 and db1093 was done long before db1085
[09:15:21] <jynus>	 I am not too worried, I just ping when I see something weird just in case
[09:16:06] <marostegui>	 yeah yeah
[09:16:10] <marostegui>	 thanks for doing so
[09:16:24] <marostegui>	 I am monitoring it actually, but 1-2 seconds can easily slip indeed
[09:19:27] <jynus>	 I also check more throwly
[09:19:30] <jynus>	 when I deploy
[09:19:33] <jynus>	 that is why
[09:23:04] <jynus>	 I wonder if I could build a better kibana dashboard with all errors without the known issues but only for eqiad
[10:39:56] <wikibugs>	 10DBA: Check recentchanges table and query errors on wikis other than commonswiki and ruwiki - https://phabricator.wikimedia.org/T178290 (10Marostegui)
[12:20:03] <wikibugs>	 10DBA, 10Schema-change: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 (10Marostegui) a:03Marostegui
[12:20:26] <wikibugs>	 10DBA, 10Schema-change: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 (10Marostegui) a:03Marostegui
[12:45:51] <jynus>	 only 3 servers left with jessie on codfw
[12:45:58] <jynus>	 (not counting parsercaches)
[12:52:08] <marostegui>	 \o/
[12:52:29] <marostegui>	 how many do we have in eqiad (not counting masters)?
[12:55:35] <jynus>	 let me see
[12:56:28] <jynus>	 I think 7
[12:57:49] <jynus>	 7 masters eqiad + 7 replicas eqiad + 3 servers codfw + 6 parsercaches
[12:58:00] <Hauskatze>	 23
[12:58:42] <volans>	 marostegui: do you consider standalone ones as masters?
[12:59:00] <jynus>	 4 es hosts
[12:59:06] <jynus>	 1 dbstore1002
[12:59:23] <jynus>	 and 5 labs related
[12:59:28] <jynus>	 we don't have much say on
[13:01:00] <marostegui>	 Nice, not many then!
[13:01:26] <jynus>	 I may finish the codfw ones this week
[13:01:35] * marostegui hugs jynus
[13:25:42] <jynus>	 I have pending repooling