[05:51:12] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3529979 (10Marostegui) [07:46:06] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530047 (10Marostegui) [07:46:19] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3235290 (10Marostegui) [09:51:22] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3530232 (10jcrespo) @tstarling can you clarify the comment "These bv_*edits tables are intermediate data (edit counts) generated by SecurePoll CLI scripts.... [12:57:27] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530530 (10Marostegui) [14:10:39] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530600 (10Marostegui) db1015 has some differences on the following tables with db1078: ``` abwiki.querycache_info azbwiki.archive abwiki.querycache_info cswikiversity.archive enwikinews.thread_history enwikisource.filearchive enwikiv... [14:11:34] 10DBA, 10Patch-For-Review: duplicate key problems - https://phabricator.wikimedia.org/T151029#3530614 (10Marostegui) Fixed issues on db2077 and db2047 so far (still compressing on db2047): arwiki: linter, tag_summary, wbc_entity_usage hewiki: change_tag eswiki: linter [14:12:31] I just noticed I wasn't on this channel since yesterday when my client killed it [14:12:59] Actually no, since a few hours agi [14:13:00] weird [14:28:51] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530654 (10jcrespo) How confident are about the other tables- did you check all others, only some of them? All dbs, only ones on dblists? Can you tell me more about that? [14:37:05] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530703 (10Marostegui) I have diffed all the tables across all the databases and the ones I mentioned above were the ones with differences across the board. From those tables that have differences, so far I have manually checked the d... [14:43:23] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530731 (10jcrespo) This is interesting, because I see on abwiki.querycache_info: ``` 5c5 < (test"Ancientpages","20170816050008"), --- > ("Ancientpages","20170816050008"), ``` Such change does not exist on the live databases: ``` n... [14:45:58] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530736 (10Marostegui) >>! In T164488#3530731, @jcrespo wrote: > This is interesting, because I see on abwiki.querycache_info: > > ``` > 5c5 > < (test"Ancientpages","20170816050008"), > --- >> ("Ancientpages","20170816050008"), > ```... [14:54:45] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3530779 (10jcrespo) > Sorry, that was an "easter egg" I left, to make sure the diff would actually sent some output :) Thanks, that is good news- :-) it means mydumper works and we do not have a backdoor in our code breaking backups.... [15:39:17] 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3531015 (10Marostegui) And this happened again just now: https://grafana.wikimedia.org/dashboard/db/wikidata-edits?refresh=1m&panelId=1&fullscreen&orgId=1&from=1502980683784&to=1502984283784 And spi... [15:46:00] =/ [15:50:09] 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3531065 (10jcrespo) p:05Normal>03Unbreak! The execution of: ``` www-data 20284 0.0 0.1 328216 60744 ? S 15:30 0:00 php5 /srv/mediawiki-staging/multiversion/MWScript.php extensions/W... [15:53:34] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3531075 (10jcrespo) p:05High>03Unbreak! [16:04:40] 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3531118 (10jcrespo) I think this is just `UPDATE /* Title::invalidateCache */` based on the binlogs, not the above script. [16:06:31] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3531120 (10jcrespo) Setting as unbreak now because this is preventing collaborators from editing ar... [16:08:29] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Wikidata, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3531127 (10jcrespo) [16:12:28] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3531155 (10Bawolff) [16:13:10] 10DBA, 10Operations, 10Wikidata: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3531160 (10jcrespo) which means most probably a direct cause of T164173, getting worse? [17:07:30] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3531257 (10Marostegui) >>! In T164488#3530779, @jcrespo wrote: >> Sorry, that was an "easter egg" I left, to make sure the diff would actually sent some output :) > > Thanks, that is good news- :-) it means mydumper works and we do n... [20:01:20] 10DBA, 10Operations, 10Wikidata, 10Wikidata.org: Wikidata.org currently very slow - https://phabricator.wikimedia.org/T173269#3532017 (10Bugreporter)