[05:36:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3698679 (10Marostegui) [05:37:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3686364 (10Marostegui) [06:02:11] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3698688 (10Marostegui) [06:10:03] 10DBA, 10Patch-For-Review: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3698695 (10Marostegui) db1044 and db1072 have been checksummed and we can now move db1095 (Sanitarium) under db1072 whenever we like, so we can decom db1044. [06:32:57] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3698712 (10Marostegui) >>! In T175970#3696906, @jcrespo wrote: > CC @Marostegui Maybe we can setup an unpuppetized copy of x1 from dbstore2002 on dbstore1002? Yeah, dbstore1002 now has plenty of space (1.9T avai... [06:34:02] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3698713 (10jcrespo) Should we do it on the same instance, is a mysqloader fast enough for x1, do you think? [06:34:14] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3698714 (10jcrespo) a:03jcrespo [06:36:28] 10DBA, 10Operations: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3698715 (10Marostegui) >>! In T175970#3698713, @jcrespo wrote: > Should we do it on the same instance, is a mysqloader fast enough for x1, do you think? I would do it on the same instance, otherwise we will need... [08:30:56] I am converting ip_changes table on dbstore1001 to tokudb- it was a 20GB innodb table already [08:43:01] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566137 (10Marostegui) Hi, I have tested this alter table on db2084 (10.1), basically tested... [08:45:46] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3698864 (10Marostegui) Does this need to be backuped or can it just be dropped? Ideally I will rename it on a producti... [08:48:27] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3698865 (10hoo) >>! In T177601#3698864, @Marostegui wrote: > Does this need to be backuped or can it just be dropped?... [08:48:41] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3698866 (10Ladsgroup) A backup would be great, all crazy things can happen. One thing that would be great to know is t... [08:49:55] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3698867 (10Ladsgroup) @hoo, you are right but re-generating it might take some time. [08:54:14] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3698881 (10Marostegui) >>! In T177601#3698866, @Ladsgroup wrote: > A backup would be great, all crazy things can happe... [09:03:40] jynus: marostegui I haven't told labs people to run the maintain-views script, got lost in jobs, without the table, it'll break? [09:04:55] nothing will break [09:05:01] Amir1: it will complain when running mysql_upgrade or if people try to select from it of course, but as I am not planning to drop it today or anything, I would file a ticket with them, please CC me [09:07:13] sure [09:07:18] thanks [09:12:28] https://gerrit.wikimedia.org/r/385334 [09:13:05] checking [09:13:24] why do you think it can break dbstore1001? [09:13:39] well, I should have phrased it as [09:13:54] it will most likely break replication on x1 on dbstore1001 restart/crash [09:14:02] but why? [09:14:25] I think replication parameters are persisted on the replication info? [09:14:28] not sure right now [09:15:03] I am importing the tables to dbstore1002, not to dbstore1001- both are currently not replicationg %wik% tables [09:15:53] ah, i know what you mean [09:16:01] this is more of a [09:16:09] not sure if it will break on dbstore1001 just because you've added the filters [09:16:11] I know this is wrong, can you let me do it anyway? [09:16:20] it is cheaper [09:16:27] as you also said, x1 is broken on dbstore1001 anyways [09:16:31] I am making dbstore1002 good [09:16:45] I want you permission to "break" dbstore1001, although not imediately [09:16:49] haha [09:16:54] I +1ed [09:16:58] cool [09:17:23] not sure it will break [09:17:25] I have deployed the filters to dbstore1002 already [09:17:25] but we will see [09:17:35] will merge now [09:17:37] 10DBA, 10Wikidata, 10cloud-services-team, 10Wikidata-Sprint: Drop replication of wb_entity_per_page in labs - https://phabricator.wikimedia.org/T178661#3698944 (10Ladsgroup) [09:18:22] BTW, I am marking cognate and wikishared despite the %wik% [09:18:52] to remind us of those 3 standalone databases, in adition to wiki-named dbs [09:20:04] import to dbstore1002 is still ongoing, we'll see how much time it takes [09:24:02] 10DBA, 10Wikidata, 10cloud-services-team, 10Wikidata-Sprint: Drop wb_entity_per_page views in labs - https://phabricator.wikimedia.org/T178661#3698996 (10jcrespo) [09:29:02] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3699019 (10jcrespo) To investigate if we need to convert some of these away from uncompressed InnoDB on dbstore1001: ``` +--------------------------------... [09:33:53] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3699022 (10jcrespo) On screen at dbstore1001 CC @marostegui : root@dbstore1001[cebwiki]> ALTER TABLE templatelinks ENGINE=TokuDB; probably commonswiki.r... [09:37:41] 10DBA, 10Operations, 10Patch-For-Review: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3699057 (10jcrespo) I am reimporting x1 on dbstore1002 (including the echo tables), hopefully, that will work and will not break anything else :-/ [10:01:06] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Chinese-Sites, 10Patch-For-Review: Enable description usage tracking on further wikis - https://phabricator.wikimedia.org/T178515#3699094 (10hoo) Deployment scheduled for Monday (Oct 23) 13:00–14:00 UTC. Note: We still don't have a single usage... [10:31:44] 10DBA, 10Operations, 10Patch-For-Review: Lost access to x1-analytics-slave - https://phabricator.wikimedia.org/T175970#3699146 (10jcrespo) 05Open>03Resolved @Etonkovidova, @Catrope sorry for the delay. We how this access to echo tables on dbstore1002 will be relatively reliable. I added grants to all x1... [10:33:18] 10DBA, 10Analytics: Access to x1 broken on stat1006 - https://phabricator.wikimedia.org/T178237#3699150 (10jcrespo) 05duplicate>03Resolved a:03jcrespo This was resolved on T175970. [10:33:48] 10DBA, 10Collaboration-Team-Triage, 10Contributors-Analysis: Replicate Echo tables to analytics-store - https://phabricator.wikimedia.org/T115275#3699155 (10jcrespo) a:03jcrespo This was resolved on T175970. [10:33:58] 10DBA, 10Collaboration-Team-Triage, 10Contributors-Analysis: Replicate Echo tables to analytics-store - https://phabricator.wikimedia.org/T115275#3699159 (10jcrespo) 05stalled>03Resolved [12:02:25] labsdb1010 is back to standby? [12:03:42] yes [12:03:49] ok! [12:03:50] or it was 2 days ago [12:03:54] sure sure [12:04:00] just confirming :) [12:04:09] want to meet on IRC later? [12:04:15] yeah [12:04:19] I do not need it, just offering [12:04:49] i think i am aware of all the stuff you have though [12:05:01] anything that you think might need urgent attention? [12:05:33] not really [12:08:46] any recentchanges shards that needs optimizing? [12:08:54] (apart from s6 and s4 whcih are almost done) [12:09:59] not yet [12:10:10] I left the ticket open I want to purge othre wikis [12:10:18] search purge on phab [12:10:29] but I didn't start, as there was some disagreements on details [12:10:51] svwiki and others may having some perfomance issues, but not as bad as ruwiki/commons [12:11:00] https://phabricator.wikimedia.org/T178290 [12:11:02] right I see [12:11:25] that is up to you, you can ignore it for now unless it gets worse [12:11:41] running right now root@dbstore1001[cebwiki]> ALTER TABLE templatelinks ENGINE=TokuDB; [12:11:44] on dbstore1001 [12:11:48] yeah, saw that with the CC :) [12:11:53] it is at 54.5% [12:11:59] it will be done by tomorrow i assume [12:12:04] I would like to give some more case to dbstore1001 [12:12:23] but again, up to you, depending on how it is behaving [12:12:36] I did already ip_changes [12:14:02] I will keep an eye on: https://phabricator.wikimedia.org/T162789#3699019 [12:53:00] 10DBA, 10Patch-For-Review: Migrate some s4 hosts to file per table - https://phabricator.wikimedia.org/T161088#3699532 (10Marostegui) [13:25:11] hello people! Would it be good to work on https://gerrit.wikimedia.org/r/#/c/385173/ next week? [13:25:25] (analytics dbs puppet refactoring) [13:41:28] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3699611 (10Anomie) >>! In T174569#3698860, @Marostegui wrote: > The ones setting the default t... [13:54:51] elukey: jynus is off on vacation starting next week I believe, so perhaps it's not the best time :) [13:55:36] marostegui is authoritative of course, just trying to cut him some slack :) [13:55:39] actually we are interested on this [13:55:46] it is to decom db1046/47 [13:56:09] now, how fast it can be is another thing... [14:18:55] paravoid: oh yes I haven't targeted Jaime because of the time off, I can work with Manuel to avoid setting things on fire [14:23:38] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3699673 (10jcrespo) @Anomie, for context, he is doing it on a depooled server, that is the mai... [14:57:56] 10DBA, 10Wikidata, 10cloud-services-team, 10Wikidata-Sprint: Drop wb_entity_per_page views in Wiki Replicas - https://phabricator.wikimedia.org/T178661#3699770 (10bd808) [14:58:19] 10DBA, 10Data-Services, 10Wikidata, 10Wikidata-Sprint, 10cloud-services-team (Kanban): Drop wb_entity_per_page views in Wiki Replicas - https://phabricator.wikimedia.org/T178661#3698944 (10bd808) [17:06:19] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3700066 (10Marostegui) And also altering revision on enwiki and image on commons could be almo... [19:39:52] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10User-Daniel, 10Wikidata-Sprint: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3700572 (10matej_suchanek) [23:58:53] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10Documentation, 10TechCom-RfC (ArchCom-Approved): Bump MediaWiki's minimum supported MySQL Version to 5.5.8 - https://phabricator.wikimedia.org/T161232#3700925 (10Legoktm) 05Open>03Resolved a:03Reedy I've updated https://www.mediawiki.org/w/i...