[05:53:11] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3257597 (10Marostegui) 05Open>03Resolved All hosts are done: ``` root@neodymium:/home/marosteg... [06:01:23] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3257603 (10Marostegui) db2064 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui/T162611`; do echo $i; mysql -hdb2064.codfw.wmnet --skip-ssl $i -e "show create table... [06:03:10] 10DBA: dbstore2001 takes 3 hours to start MySQL after a crash - https://phabricator.wikimedia.org/T165033#3257605 (10Marostegui) Stop process - 6 minutes: ``` 170512 5:55:05 [Note] /opt/wmf-mariadb10/bin/mysqld: Normal shutdown 170512 6:01:23 [Note] /opt/wmf-mariadb10/bin/mysqld: Shutdown complete 1705... [09:04:37] 10DBA, 06Labs, 10wikitech.wikimedia.org, 07Schema-change: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3250009 (10Marostegui) I have renamed the tables for now on silver and labtestweb2001. So they now look like: ``` root@silver:~# mysql --skip-ssl labswiki... [09:39:20] 10DBA: dbstore2001 takes 3 hours to start MySQL after a crash - https://phabricator.wikimedia.org/T165033#3257985 (10Marostegui) Start process: ``` 170512 06:03:14 mysqld_safe Starting mysqld daemon with databases from /srv/sqldata 170512 6:03:15 [Note] /opt/wmf-mariadb10/bin/mysqld (mysqld 10.0.28-MariaDB) sta... [09:41:19] 10DBA: dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) - https://phabricator.wikimedia.org/T165033#3257986 (10Marostegui) [09:42:34] 10DBA: dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) - https://phabricator.wikimedia.org/T165033#3254845 (10Marostegui) 05Open>03Resolved a:03Marostegui I am going to resolve this ticket now as the investigation is over and we can do at this point I... [09:43:13] 10DBA, 07Epic: Meta ticket: The future of multi source replication slaves vs multi instance ones. - https://phabricator.wikimedia.org/T159423#3257996 (10Marostegui) [09:43:15] 10DBA: dbstore2001 takes 3 hours to start MySQL (was: dbstore2001 takes 3 hours to start MySQL after a crash) - https://phabricator.wikimedia.org/T165033#3257995 (10Marostegui) [09:43:56] 10DBA, 07Epic: Meta ticket: The future of multi source replication slaves vs multi instance ones. - https://phabricator.wikimedia.org/T159423#3067139 (10Marostegui) Probably the issue we see on dbstore2001 (T165033) could be mitigated by having having multi-instance rather than multi-source. [12:01:41] 10DBA, 06Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3258345 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:49:14] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3258455 (10aude) i checked test wikidata (testwikidatawiki) and it needs the change also. the tabl... [13:26:14] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3258545 (10Marostegui) >>! In T162539#3258455, @aude wrote: > i checked test wikidata (testwikidat... [13:31:13] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3258550 (10aude) @Marostegui for small schema changes (e.g. adding a table, or for small wikis lik... [13:34:13] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3258558 (10Marostegui) >>! In T162539#3258550, @aude wrote: > @Marostegui for small schema changes... [14:02:37] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3258614 (10Marostegui) ukwiki is done and only one difference found on dbstore1002: ``` Differences on dbstore1002 TABLE CHUNK CNT_DIFF CRC_DIFF CHUNK_INDEX LOWER_BOUNDARY UPPER_BOUNDARY ukwiki.geo_tags 1 0 1 PRIMARY 37 13702292 ``` [14:03:05] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3258615 (10Marostegui) [16:21:39] 10DBA, 06Commons, 10MediaWiki-Change-tagging, 10MediaWiki-Recent-changes, and 2 others: Allow filtering based on tag on Special:NewFIles - https://phabricator.wikimedia.org/T124214#3258984 (10matmarex) [19:59:43] 10DBA, 06Operations, 10Traffic: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3259609 (10Dzahn) a:03Dzahn