[08:07:11] <jynus>	 dtpwiki.pagelinks fixed
[08:14:06] <arnaudb>	 jynus: please log it in the sheet!
[08:14:09] <arnaudb>	 let me find the url
[08:14:29] <arnaudb>	 privately sent
[08:16:00] <jynus>	 note I wasn't fixing the index corruption
[08:16:07] <jynus>	 this is a separate issue
[08:16:07] <arnaudb>	 ah x)
[08:16:18] <jynus>	 this was a data consistency issue
[08:16:27] <arnaudb>	 ack, thanks and sorry for the confusion
[08:16:36] <jynus>	 no prob, that's a nice log you have there
[08:16:59] <arnaudb>	 somehow longer than I'd whish it to be 
[08:17:44] <jynus>	 but I will keep it in minde in case it is similar to the data issue (I don't think it is related, but thanks for the ping)
[08:18:02] <arnaudb>	 yeah I don't think we had the opportunity to talk about this since you got back
[08:44:09] <dhinus>	 o/ can I get a +1 from a DBA on this patch? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1073430
[08:46:12] <jynus>	 mmm, did you get asked to display global blocks publicly?
[08:47:41] <jynus>	 I'd say what you should get is an ok from security not dp
[08:48:15] <jynus>	 I see, it useed to be fully public
[08:48:31] <jynus>	 then looks ok
[08:50:23] <dhinus>	 thanks! do you mind leaving a +1 on the patch? until we define a clearer process, I'd like to have +1s from all involved teams (wmcs, data-persistence, data-engineering) before merging
[08:50:28] <jynus>	 I did
[08:50:36] <jynus>	 before you asked
[08:51:08] <dhinus>	 thanks I should have waited 30 sec more :)
[08:51:09] <jynus>	 I am just more concerned when things are made more public than the opposite
[08:51:27] <dhinus>	 yeah likewise, this seems an "easy" change
[10:49:38] <jynus>	 I saw another discrepancy on the viwiki user table
[10:50:01] <jynus>	 it is non-canonical data (edit count), but still it is not great to have
[12:09:08] <jynus>	 so other than the change_tag table, I fill now confortable with the data comparison between eqiad and codfw
[12:09:13] <jynus>	 *feel
[12:09:29] <jynus>	 https://phabricator.wikimedia.org/T375507
[12:10:22] <arnaudb>	 jynus: I don't have access to the pastes fyi
[12:10:29] <arnaudb>	 P69414 for instance
[12:10:45] <jynus>	 I made them visible for SREs only, are you part of that phab group?
[12:11:06] <arnaudb>	 https://phabricator.wikimedia.org/p/ABran-WMF/ I am not apparently
[12:11:17] <jynus>	 let me add you
[12:11:29] <arnaudb>	 thanks!
[12:11:47] <jynus>	 try now
[12:12:03] <arnaudb>	 looks ok!
[12:12:05] <jynus>	 I plan to make the ticket public eventually
[12:12:14] <arnaudb>	 til {PasteID} does this 👀
[12:12:20] <jynus>	 but not before all the inconsistencies have been checked
[12:12:24] <arnaudb>	 sure
[12:12:40] <jynus>	 no time for the ones left before the switchover
[12:12:50] <arnaudb>	 looks a bit tight indeed
[12:12:51] <jynus>	 but I am happy with the correctly the other ones that were detected
[12:13:06] <arnaudb>	 kudos for the work done anyway, this is less to do in the future!
[12:13:06] <jynus>	 (page, log, user, archive)
[12:13:19] <arnaudb>	 and less risk for replication
[12:13:24] <jynus>	 some of them dated from 2005!
[12:13:51] <arnaudb>	 low risk for replication apparently x)
[12:14:02] <jynus>	 I'd prefer 0 risk
[12:14:08] <arnaudb>	 ^ tis
[12:14:10] <arnaudb>	 this*
[12:14:50] <jynus>	 due to Murphy's law
[13:59:15] <jynus>	 apologies by the way to not make clear that for the check I meant just the core/mw hosts, misc goes on a different side
[14:00:27] <jynus>	 the only think I saw missing was pc1017 pc2017 which I am guessing they are the new hosts for pc5 ?
[14:01:59] <jynus>	 in any case, not a big issue
[14:07:24] <swfrench-wmf>	 hi folks, quick question: I see [0] has not been merged yet. my understanding is that is a lingering follow-up from the pc5 turnup, correct?
[14:07:24] <swfrench-wmf>	 [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1075052
[14:07:36] <swfrench-wmf>	 i.e. it does not block any action we might take as part of the switchover
[14:07:52] <volans>	 see -sre ;)
[14:07:54] <arnaudb>	 swfrench-wmf: this has been discussed on sre
[14:07:59] <arnaudb>	 ah volans you were faster
[14:08:12] <arnaudb>	 swfrench-wmf: anyway, this looks like no trouble
[14:08:15] <jynus>	 it can proceed, no blocker
[14:08:27] <swfrench-wmf>	 ah, oops - crossed wires :)
[14:08:34] <swfrench-wmf>	 thank you both in either case!
[14:08:36] <jynus>	 otherwise I wouldn't have voted +1
[14:08:43] <jynus>	 but good to check
[14:10:13] <jynus>	 for you ease of mind, pcs are active-active, so we would learn faster if it was a blocker :-D
[14:26:36] <jynus>	 volans: what you asked about happened- one of my fixes broke the ROW based sanitarium replica (the idle one, no impact on the active one)
[14:26:51] <jynus>	 but I knew it was safe for regualar production
[14:28:25] <volans>	 that's the important part
[14:29:35] <volans>	 and sorry if I jinxed it
[14:29:48] <jynus>	 nah, good point
[14:29:56] <jynus>	 you were right!
[15:19:18] <swfrench-wmf>	 just wanted to check in and thank you all for the effort involved in getting us ready for the switchover - both the execution of it and the new automation :)
[15:21:30] <arnaudb>	 <3
[15:21:42] <arnaudb>	 thanks swfrench-wmf and thanks for the flawless exec!
[15:22:03] <arnaudb>	 (and preparation ofc)
[15:36:32] <jynus>	 we should tune tomorrow the weights after actual traffic
[15:36:57] <jynus>	 I am thinking of reducing db2175,db2207 and db2189 load a bit
[15:37:09] <jynus>	 as they are in the 12K QPS
[15:37:18] <jynus>	 eg from 500 to 400
[15:38:54] <jynus>	 and a similar thing for s4