[05:30:21] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui) s3 codfw: [] dbstore2002 [] db2094 [] db2074 [] db2057 [] db2050 [] db2043 [] db2036
[05:30:38] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui)
[07:36:04] <marostegui>	 I would like to start disconnecting codfw -> eqiad replication, so basically; stop slave; reset slave all; on all the eqiad masters
[07:41:00] <jynus>	 there is more to it than that-s3 replication on s5
[07:41:20] <marostegui>	 yeah, I was going to leave s3 for the last one
[07:42:02] <marostegui>	 I was planning to do that with with a second pair of eyes (you)
[07:42:40] <marostegui>	 so, s3 and s5 for the last ones
[07:42:47] <marostegui>	 s1,s2,s4,s6,s7,s8, es2,es3,x1
[07:42:50] <marostegui>	 was my idea
[07:51:11] <jynus>	 ok, I will keep fixing wb_terms
[07:51:22] <jynus>	 on labsdbs
[07:51:24] <marostegui>	 <3
[07:51:37] <marostegui>	 So I will proceed with disconnecting replication and once everything is done, start enabling gtid
[07:51:57] <marostegui>	 I will bother you to double check my commands for s3 and s5
[07:58:00] <jynus>	 start enabling gtid?
[07:58:15] <marostegui>	 on codfw
[07:58:19] <jynus>	 maybe we can touch codfw at a later time
[07:58:23] <marostegui>	 sure
[07:59:14] <jynus>	 I prefer to handle the deletion and import first
[07:59:25] <jynus>	 of the migrated wikis
[07:59:28] <marostegui>	 sure
[07:59:30] <marostegui>	 I can do that
[07:59:41] <jynus>	 note I wasn't asking you do it
[07:59:49] <marostegui>	 No, I had it on my plans :)
[07:59:50] <jynus>	 just mentioning it should be done earlier
[08:00:08] <marostegui>	 I get nervous by not having s5 codfw fully equal to s5 eqiad :)
[08:00:27] <jynus>	 and the gtid thing waiting for the root cause analysis of the replication problems
[08:00:37] <marostegui>	 yep
[08:00:43] <jynus>	 which alsmo means no alters on codfw
[08:01:13] <marostegui>	 yeah, I wasn't going to do any alters on codfw with replication
[08:01:19] <marostegui>	 (if gtid was enabled)
[08:14:25] <marostegui>	 jynus: db1075: stop slave reset slave all; 
[08:17:31] <marostegui>	 db1075 is s3, btw
[08:19:28] <jynus>	 sure
[08:21:32] <marostegui>	 s5, db1070: stop all slaves; reset slave all; reset slave 's3' all;
[08:21:55] <jynus>	 ok
[08:22:12] <jynus>	 I am going to start fixing wb_terms on db1087 with replication running
[08:22:19] <marostegui>	 great
[08:53:18] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui)
[08:53:49] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10Marostegui) 05stalled>03Resolved This is all done
[09:05:01] <jynus>	 remove the filters on db2052
[09:11:15] <marostegui>	 yep, it is noted :)
[09:11:32] <marostegui>	 https://phabricator.wikimedia.org/T184805#4654953
[09:18:27] <wikibugs>	 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10Marostegui) >>! In T184805#4654953, @Marostegui wrote: > This was done successfully and new wikis are now live on eqiad. > What is pending now is: > - Run...
[09:46:57] <wikibugs>	 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) a:05jcrespo>03Marostegui
[11:35:42] <jynus>	 I finished the labsdb import, doing a last compare.py check on wb_terms (will take a while)
[11:38:46] <wikibugs>	 10DBA, 10Lexicographical data, 10Wikidata, 10Wikidata-Campsite, and 6 others: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared") - https://phabricator.wikimedia.org/T206743 (10jcrespo) After many fixes during the weekend, wb_terms also fixed on lab...
[13:00:11] <marostegui>	 jynus: Great!!! :)
[13:00:22] <marostegui>	 send me the list of tables when you have a chance
[13:19:31] <jynus>	 I already did, I think
[13:19:47] <marostegui>	 Ah yes
[13:19:52] <marostegui>	 I remember now
[13:19:52] <jynus>	 apparently I sent it to me?
[13:19:57] <marostegui>	 Like 1 week ago?
[13:20:22] <jynus>	 I resent it
[13:20:27] <marostegui>	 thank you
[13:59:33] <jynus>	 the compare.py finished on wb_terms too
[14:33:56] <marostegui>	 So far I have found differences on abuse_filter_log for db1092 (host recloned) and db1087 (host fixed manually)
[14:34:08] <marostegui>	 I am digging further while other tables are being checked
[14:34:47] <jynus>	 abuse_filter_log is filtered
[14:34:58] <marostegui>	 but db1087 shouldn't have filters, right?
[14:35:38] <marostegui>	 There are rows that exist on db1092 but not on db1087
[14:35:43] <marostegui>	 ie: select * from abuse_filter_log where afl_id=5042135
[14:35:48] <jynus>	 that is strange
[14:35:59] <marostegui>	 same with the master (db1071)
[14:35:59] <jynus>	 because it showed no differences before
[14:37:22] <marostegui>	 so for that example, that row is missing on db1071, db1087 and labs (the rest of hosts, including codfw) have it
[14:38:05] <marostegui>	 there are just 66 rows missing 
[14:38:10] <marostegui>	 all between: 
[14:38:16] <marostegui>	 WHERE afl_id BETWEEN 5040001 AND 5050000
[14:39:03] <jynus>	 please note all those down and I will fix them
[14:39:09] <marostegui>	 sure :)
[14:39:26] <marostegui>	 I will report back once I have finished all the checks, or before I finish for the day (whatever comes first!)
[14:40:28] <marostegui>	 do you prefer the task or the email to report whatever differences I find?
[14:47:34] <jynus>	 anything work, I guess email is more handy
[14:47:49] <marostegui>	 sounds good
[15:05:05] <jynus>	 have a nice day, see you tomorrow!
[15:05:55] <marostegui>	 enjoy jynus!
[15:42:11] <wikibugs>	 10DBA, 10Lexicographical data, 10Wikidata, 10Wikidata-Campsite, and 6 others: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared") - https://phabricator.wikimedia.org/T206743 (10Marostegui) The following tables have been re-checked on db1092 (reclone...
[16:43:19] <wikibugs>	 10DBA, 10JADE, 10Operations, 10Epic, and 2 others: [Epic] Extension:JADE scalability concerns - https://phabricator.wikimedia.org/T196547 (10awight)
[16:43:24] <wikibugs>	 10DBA, 10JADE, 10Operations, 10MW-1.32-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), and 4 others: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10awight) 05Open>03Resolved
[16:46:42] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @daniel @Krinkle @Catrope @Marostegui We're ready for another round of TechCom and DBA review, at...
[17:06:41] <wikibugs>	 10DBA, 10Lexicographical data, 10Wikidata, 10Wikidata-Campsite, and 6 others: S8 replication issues leading to rows missing during eqiad -> codfw switch (Was: "A few lexemes disappeared") - https://phabricator.wikimedia.org/T206743 (10Addshore) >>! In T206743#4685334, @jcrespo wrote: > @Addshore did you se...
[17:09:56] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) moving to TechCom inbox for review
[17:44:42] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Anti-Harassment, 10Patch-For-Review: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 (10dbarratt) >>! In T204006#4684840, @Marostegui wrote: > This is all done  THANK YOU!