[05:41:26] elukey: I had to start the import again, as looks like there were some issues with the mysql locks, I have increased them and restarted the import with less threads to avoid too much concurrency [05:41:43] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3714388 (10Marostegui) [05:48:13] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Identify tools hosting databases on labsdb100[13] and notify maintainers - https://phabricator.wikimedia.org/T175096#3714415 (10Luke081515) >>! In T175096#3714074, @Quiddity wrote: >>>! In T175096#3711831, @Marostegui wrote: >>>>! In T175096#3711025, @L... [05:55:41] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3714454 (10Marostegui) I have optimize ruwiki.recentchanges on dbstore1001. [05:58:54] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Drop the "wb_terms.wb_terms_language" index - https://phabricator.wikimedia.org/T179106#3714460 (10Marostegui) p:05Triage>03Normal If this requires to depool servers (I don't know yet), I was talking to @Ladsgroup that it might be even worth t... [06:30:02] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: Deploy dropping wb_entity_per_page table - https://phabricator.wikimedia.org/T177601#3714634 (10Marostegui) There have been no errors so I will be dropping this table on Monday (because I do not want to... [07:20:52] marostegui: :( [07:21:17] that's life :) [08:05:08] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3714754 (10Marostegui) Not sure I am getting your last point right, @madhuvishy, taking what  @bd808 wrote on the original t... [08:55:51] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3714894 (10Marostegui) itwiki.pagelinks has been converted to TokuDB. [09:04:25] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3714910 (10Marostegui) db2086:s5 is set, pending finishing the compression on db2040 (s7) to transfer it compressed. [09:07:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3714914 (10Marostegui) Status: - Finished optimizing all the servers in s1,s2,s4,s5,s6 and s7 (pending master) - Dropping now the indexes on s3 - prob... [11:04:05] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3715077 (10Marostegui) The index has been removed from all s3 hosts. The only pending one is the master (db1075) which I will do on Monday. Also I hav... [11:04:20] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3715078 (10Marostegui) [11:46:45] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3715167 (10Marostegui) I will attempt to deploy: https://gerrit.wikimedia.org/r/#/c/386810/ on Monday and see what happens (only codfw) [11:51:56] marostegui: Any chance to get useful insight here https://phabricator.wikimedia.org/T179156#3715170 ? [11:52:54] yeah [11:53:41] i will take a look late [11:53:42] r [11:53:44] ok [11:53:56] Will update the comment… it's actually not about the replicas [11:56:36] marostegui: Updated the comment… no idea if this is just a symptom of the 503s or related to the cause [11:56:38] or both… [11:57:23] I don't think the hosts have any issues, that is probably the consecuence of the 503, but i would need to take a look later [11:57:38] *had any issues [12:03:51] Yeah, thought so :/ Shoot… [12:10:00] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3715248 (10Marostegui) >>! In T162789#3699022, @jcrespo wrote: > On screen at dbstore1001 CC @marostegui : > > root@dbstore1001[cebwiki]> ALTER TABLE t... [14:07:28] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:07:58] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716009 (10Halfak) [14:08:24] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:08:34] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:09:57] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) See discussion of "datasets_p" and what we'd like to do with datasets like this one in T173511. My goal in fili... [14:11:20] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716024 (10Halfak) [14:12:27] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:16:09] 10DBA, 10cloud-services-team, 10Scoring-platform-team (Current): Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) @diego, this is the task for loading that article quality dataset I showed you through #Quarry a few days ago.... [14:30:41] ** (myloader:29235): CRITICAL **: Error restoring log.Edit_11448630 from file log.Edit_11448630.00002.sql.gz: Lock wait timeout exceeded; try restarting transaction [14:30:44] :( [14:30:45] marostegui: --^ [14:33:23] it didn't get a lock in time, there were other transactions locking the same rows is my first wild guess ;) [15:19:06] :( [15:19:50] we can do that table later [15:19:54] if that is the one that always fails [15:20:06] we have two that have failed so far [15:21:20] let's leave it finish [15:21:33] and we can import manually the failed tables if needed on monday [15:21:37] i pressume those are big big ones? [15:26:26] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3716241 (10Marostegui) And commonswiki.templatelinks also optimized on dbstore1001 [15:31:17] Rows: 1147492907 [15:31:20] Yeah, pretty big XD [15:31:41] And the other one… Rows: 350497133 [15:31:58] let's do those two later (and if other fails) too, later [15:32:30] mydumper split into different files when the number of rows (because i said so) is bigger than X [15:32:40] so maybe it is taking ages to process all the files at the same time [15:32:55] we can try just to load the big tables that fail just using one file [15:32:58] on monday I guess [15:33:50] i can actually leave the mydumper running now for those two tables extracting from db1047, it will take some time anyways so it will be done by monday, so we can import it [15:34:44] super thanks! [15:35:00] once event sync script runs, it will be fine, right? [15:35:07] what I mean is [15:35:30] we took a dump from db1047 yesterday, and we will dump those two different tables today, is that fine to import anyways? [15:36:39] yes yes [15:36:43] cool! [15:36:51] mydumper running for those two tables, without splitting them [15:36:54] it would be a problem to import a whole big table with custom replication only [15:36:57] becuase it would take ages [15:37:04] but some days is super fine [15:37:27] if this doesn't even work, we can always go back to the root: mysqldump [15:37:30] XD [15:39:05] :D [16:49:16] have a good weekend! [17:10:10] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716479 (10madhuvishy) @Marostegui Yeah that sounds right to me! Cool if I run that across the wiki replicas? [17:14:22] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716482 (10Marostegui) Maybe only labsdb1010 (sby host) and check that it does what is required and needed. [17:26:35] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716537 (10madhuvishy) @Marostegui Sounds good, thanks [17:38:52] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716553 (10madhuvishy) Cool, I've run ``` GRANT SELECT, SHOW VIEW ON `%wik%`.* TO 'labsdbadmin'@'10.64.37.19'; GRANT SELECT,... [17:43:50] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716562 (10bd808) ``` $ ssh labstore1004.eqiad.wmnet $ mysql -h labsdb1010.eqiad.wmnet -u labsdbadmin -p Enter password: (lab... [18:07:19] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716626 (10madhuvishy) I've now rolled this out to labsdb10[01|03|09|10|11]. @Marostegui Is there a file/config/logs somewher... [18:17:05] 10DBA, 10Operations, 10Ops-Access-Requests, 10cloud-services-team (Kanban): Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3716639 (10Marostegui) I don't think we are tracking the GRANTS anywhere in puppet, but I cannot check now (I am with my phon... [18:20:26] elukey: the dump from db1047 finished for those two tables, I will start loading them now [18:28:09] <3 [18:28:31] thanks a lot for this work marostegui [18:32:37] 10DBA: Add an index on rc_this_oldid - https://phabricator.wikimedia.org/T179199#3716689 (10Anomie) [19:11:19] MediaViewer_10867062_15423246 failed again, we can try to split it into more files, so there will be more files but less locking between them [19:11:33] I guess I will try that, and if that doesn't work, i will try the mysqldump as always [19:13:15] Ah, it failed because there is another process inserting on it [19:13:32] Let's wait for all the processes to finish and then re-import those two alone and that's it