[08:31:12] 10DBA, 10Community-Tech, 10cloud-services-team, 10Security: Create core ip_changes view for replicas - https://phabricator.wikimedia.org/T173891#3645512 (10Marostegui) I can see the table on the replicas, so what is pending is #cloud-services-team to create the view [09:17:08] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of CodeCat → Rua: supervision needed - https://phabricator.wikimedia.org/T176985#3645614 (10alanajjar) Thanks @Legoktm , @MarcoAurelio and @Marostegui [09:20:54] 10Blocked-on-schema-change, 10DBA, 10Commons, 10Community-Liaisons (Oct-Dec 2017): Help communicate read-only time for Commons for schema change required by adding 3D filetype - https://phabricator.wikimedia.org/T176883#3645634 (10Qgil) [10:19:09] jynus: hey, https://gerrit.wikimedia.org/r/381421 [10:20:43] Amir1, cool, will merge [10:21:04] Thanks [11:09:21] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3645874 (10Marostegui) [11:10:57] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3564203 (10Marostegui) [11:11:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3645890 (10Marostegui) The optimize finished on s4 master and it is currently being done on all the slaves. [13:21:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3646278 (10Marostegui) [15:29:56] 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3516428 (10bd808) I performed the other steps from https://wikitech.wikimedia.org/wiki/Add_a_wiki#Cloud_Services : * `/usr/local/sbin/maintain-meta_p --datab... [15:37:49] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3646738 (10Marostegui) [15:41:14] 10DBA, 10MW-1.30-release-notes, 10Community-Tech-Sprint, 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is painfully slow - https://phabricator.wikimedia.org/T175962#3646746 (10MusikAnimal) >>! In T17... [15:43:00] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3646751 (10hoo) Not yet done: ``` mysql:wikiadmin@db1070 [wikidatawiki]> SELECT COUNT(*) FROM wb_terms WHERE term_full_entity_id IS NU... [16:07:10] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3646959 (10bd808) [16:37:34] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647040 (10Marostegui) p:05Triage>03Normal So these are the two query plans: ``` mysql:root@localhost [enwiki]> select @@hostname; +------------+ | @@h... [16:40:00] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647048 (10Marostegui) I just realised that index is NOT on tables.sql, so it must have been added on the old labs server for some reason, so that is why i... [16:44:43] 10DBA, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3647059 (10Ladsgroup) For that see {T176852} The maintenance script can't clean them (probably will jump over them) [16:48:28] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3646922 (10bd808) Explain via https://quarry.wmflabs.org/query/21882 |id|select_type|table|type|possible_keys|key|key_len|ref|rows|Extra| | -- | -- | -- |... [16:52:03] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647075 (10bd808) >>! In T177096#3647048, @Marostegui wrote: > I just realised that index is NOT on tables.sql, so it must have been added on the old labs... [16:53:53] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647077 (10MusikAnimal) >>! In T177096#3647075, @bd808 wrote: > I think we should have all prod indices, but that it is also reasonable to have additional... [16:56:58] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3646922 (10jcrespo) > but that it is also reasonable to have additional indices that make non-MediaWiki queries more performant. I don't disagree, we can... [17:03:14] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647122 (10Marostegui) >>! In T177096#3647096, @jcrespo wrote: >> but that it is also reasonable to have additional indices that make non-MediaWiki queries... [17:44:59] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647248 (10Marostegui) So, thinking about it. Should we: 1) Create the index to unblock @MusikAnimal 2) Think a way of documenting, puppetizing or someth... [18:19:45] 10DBA, 10Data-Services: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3647333 (10bd808) I'd be ok with stalling the index fix for a few days if we can get something properly designed and built relatively quickly. I would be g... [19:38:00] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3647505 (10jcrespo) [19:38:02] 10DBA, 10Patch-For-Review: Implement cron-based mydumper backups on the dbstore role - https://phabricator.wikimedia.org/T169516#3647502 (10jcrespo) 05Open>03Resolved Basic script works and users setup on dbstore2001:s5- pending lot of followup both in hardware and scripting. [19:43:49] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3647509 (10jcrespo) Part of the work (the skeleton) has been done T169516, but that is far frome perfect, with more FIXMES than good code. That should be now fixed and coverted... [19:45:31] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3647513 (10jcrespo) Clarification, the CC is not intended as a "you should do that", but as an apology for the current state, and that it will be done better soon. Thanks to ev... [22:45:23] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3647923 (10bd808) [22:47:59] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3647943 (10bd808)