[18:14:57] <cdanis>	 when did we start putting nonzero weights on section masters?
[18:57:42] <marostegui>	 cdanis: since pretty much forever 
[18:58:03] <Amir1>	 cdanis: we didn't but something decided to start reading from them. T416171 
[18:58:04] <stashbot>	 T416171: s2 primary master getting reads? - https://phabricator.wikimedia.org/T416171
[18:58:18] <cdanis>	 Amir1: I was looking at that
[18:58:22] <cdanis>	 there are a LOT of selects to masters nowadays
[18:58:27] <cdanis>	 thousands of qps on big shards
[18:58:28] <Amir1>	 marostegui: I think Chris is asking for "nonzero" weight
[18:58:41] <Amir1>	 something is really wrong
[18:58:53] <cdanis>	 weirdly it seems to start at the codfw switchover
[18:59:03] <cdanis>	 (that or prometheus is lying to me because i broke it)
[18:59:59] <marostegui>	 cdanis: ah sorry I read zero 
[19:00:40] <marostegui>	 cdanis: Yeah, the above phabricator ticket is sort of tracking it, it's very worrying 
[19:02:10] <cdanis>	 I have posted a terrifying graph
[19:02:19] <cdanis>	 many titans died to bring us this information
[19:02:46] <marostegui>	 cdanis: i can't see the graph, did you upload it correctly?
[19:03:01] <cdanis>	 argh
[19:03:03] <cdanis>	 fixed
[19:04:42] <marostegui>	 cdanis: yeah that's pretty terrible
[19:05:53] <cdanis>	 weirdly it doesn't look like that in eqiad pre-switchover, but, I don't know if that's real or not
[19:07:33] <Amir1>	 it could be a patch getting deployed at the same time
[19:07:52] <Amir1>	 follow me for more obvious facts
[19:09:25] <marostegui>	 Amir1: do any of those queries I captured yesterday and pasted there ring any bell?
[19:10:17] <Amir1>	 on the surface all look "okay" since there are many code paths that do a select before write
[19:10:57] <Amir1>	 e.g. updating links tables, when it diffs between the new parsed set and what is the db
[19:11:05] <Amir1>	 to insert/delete the diff
[19:12:08] <Amir1>	 but the ratio of 20 to 1 of read to write queries doesn't make any sense
[19:52:32] <cdanis>	 Amir1: idk, it's pretty much down to the hour
[19:57:22] <Amir1>	 oh interesting 
[20:00:12] <Amir1>	 we had some funny stuff happening when we switched over because mediawiki was doing fun stuff (e.g. swift and thumbnails purge) but let me think about it
[20:42:56] <cdanis>	 I posted some new graphs to the s2 bug
[21:47:32] <Amir1>	 thanks!
[21:47:35] <Amir1>	 cccccbvebrkvcetcernhjtddrkjnururegcldiejjebk
[22:16:14] <cdanis>	 > weirdly it doesn't look like that in eqiad pre-switchover, but, I don't know if that's real or not
[22:16:33] <cdanis>	 correction it was actually probably worse in eqiad pre-switchover  https://phabricator.wikimedia.org/F71711809