[06:28:05] 10DBA: replication broken on db1124 - https://phabricator.wikimedia.org/T209521 (10Marostegui) This broke again yesterday: ` Dec 06 18:10:48 db1124 mysqld[3110]: 2018-12-06 18:10:48 140545623860992 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pa... [06:39:11] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) Thanks for the heads up @Cmjohnson! [06:45:34] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Marostegui) >>! In T210693#4801008, @Milimetric wrote: > Thanks very much @Anomie, I understand my misunderstand... [06:59:48] 10DBA, 10Analytics, 10Data-Services, 10User-Banyek, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) @nuria do you think we should close this as it is decided we'll go for a host with the same specs and config than the rest of cl... [07:02:29] 10DBA, 10Analytics, 10Data-Services, 10User-Banyek, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) >>! In T210749#4805215, @Marostegui wrote: > @nuria do you think we should close this as it is decided we'll go for a host with the... [07:30:37] 10DBA, 10Wikidata: Make a copy of the current wb_terms table on the MCR testing DB servers - https://phabricator.wikimedia.org/T211338 (10Marostegui) a:03Marostegui As we discussed, this is a one time thing we happen to be able to do now, but this cannot be guaranteed in the future. I will restore wb_terms f... [07:39:09] 10DBA: replication broken on db1124:3318 on wikidata.pagelinks - https://phabricator.wikimedia.org/T209521 (10Marostegui) [08:14:57] 10DBA, 10Operations, 10User-Banyek: Issues with purgeUnusedProjects.php cron job on mwmaint1002 (Fri Oct 26) - https://phabricator.wikimedia.org/T208231 (10Marostegui) >>! In T208231#4803855, @Banyek wrote: > i'd like to add the owner of the script as a subscriber, but I don't know how to find who is it gi... [08:26:26] 10DBA, 10Community-Tech, 10MediaWiki-extensions-PageAssessments, 10Operations, and 2 others: Issues with purgeUnusedProjects.php cron job on mwmaint1002 (Fri Oct 26) - https://phabricator.wikimedia.org/T208231 (10Reedy) [08:39:26] 10DBA, 10Data-Services, 10User-Urbanecm: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10Banyek) [09:36:15] ok, now I am able to connect to the labsdb1005 host from labsdb1004 [09:36:30] I'll run the script [09:36:46] ok [09:37:12] after it finished I remove the `s51230__linkwatcher.linkwatcher_linklog` from the replication_wild_ignore_table variable and restart replication [09:37:21] I suggest you stop mysql [09:37:23] and then start [09:37:26] so it gets the proper config [09:37:38] to avoid messing up with the filters manually (which is very error prone) [09:37:41] good idea! [09:38:17] then: after the import is done, I'll restart mysql service. Check replication, etc. [09:38:53] yep [09:49:17] 10DBA: labsdb1004 replication broken for linkwatcher_linklog table - https://phabricator.wikimedia.org/T211210 (10Banyek) The table is too huge to easily find the inconsistencies (a `SELECT COUNT(*) ...` command took as much time I stopped it) so I decided to reimport everything. The script is started, running o... [09:52:12] 10DBA: labsdb1004 replication broken for linkwatcher_linklog table - https://phabricator.wikimedia.org/T211210 (10Marostegui) You could've used compare.py, it might have taken several hours though. [10:06:07] 10DBA: labsdb1004 replication broken for linkwatcher_linklog table - https://phabricator.wikimedia.org/T211210 (10Banyek) >>! In T211210#4805462, @Marostegui wrote: > You could've used compare.py, it might have taken several hours though. so far I just used it to see if there are diffs, not for finding those [10:07:59] banyek: Errr, why is replication not stopped on labsdb1004? [10:08:15] banyek: If replication isn't stopped what you are doing is useless [10:08:19] because the script will [10:08:40] first it just warms up the table on labsdb1005 [10:08:58] Ah it is not yet dumping? [10:09:24] it does 2 dumps [10:09:27] I know :) [10:09:33] I thought it already passed the first one [10:09:33] first it dunmps it into /dev/null [10:09:46] no, as I calculated it will take like 5 hours [10:09:56] (~20Mb/s and 400G table) [10:09:58] I think it is not worth it [10:10:02] I mean, the warm up [10:10:08] Kill it and comment that part I would say [10:10:42] yea, that makes sense [10:12:44] I removed the warmup phase, and now I running it again [10:13:27] https://phabricator.wikimedia.org/P7894 [10:14:12] great [10:44:48] 10DBA: labsdb1004 replication broken for linkwatcher_linklog table - https://phabricator.wikimedia.org/T211210 (10Marostegui) >>! In T211210#4805471, @Banyek wrote: >>>! In T211210#4805462, @Marostegui wrote: >> You could've used compare.py, it might have taken several hours though. > > so far I just used it to... [12:31:43] 10DBA, 10Operations, 10StructuredDiscussions, 10Growth-Team (Current Sprint), and 2 others: Setup separate logical External Store for Flow in production - https://phabricator.wikimedia.org/T107610 (10Banyek) Sorry for the late answer @Catrope, The bullets you provided seems good to me, I'd say we could ta... [14:31:54] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) >>! In T210693#4805204, @Marostegui wrote: > Whilst we still discuss if this JOIN "feature" can do w... [14:43:56] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Marostegui) Thanks for the detailed analysis. My proposal was more in the line to check that the JOIN is always... [15:13:23] We should probably change the reimport script to use mydumper+myloader [15:16:32] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) To duplicate something like check_private_data in Hadoop, I'd guess a day to write it and a couple o... [15:22:30] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Marostegui) >>! In T210693#4806084, @Milimetric wrote: > To duplicate something like check_private_data in Hadoo... [15:34:33] good idea [15:34:51] I was also thinking to import directly instead of dump/import [15:34:58] but now at least I like to have a progress bar [15:50:56] 10DBA, 10Analytics, 10Data-Services, 10User-Banyek, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Nuria) >@Nuria do you think we should close this as it is decided we'll go for a host with the same specs and config than the rest of clouds... [17:18:27] I asked this in -operations, mirroring it here just in case. I know historically Etherpad's database has not been backed up, but has that since changed? [17:48:57] 10DBA, 10Data-Services, 10User-Urbanecm: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10GTirloni) Views and indexes created. ` MariaDB [meta_p]> SELECT count(*) FROM meta_p.wiki WHERE dbname = 'punjabiwikimedia'; +----------+ | count(*) | +----------+ |... [17:50:36] 10DBA, 10Data-Services, 10User-Urbanecm: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10GTirloni) 05Open>03Resolved [18:33:22] Question answered in another channel, never mind. [21:31:25] 10DBA, 10Data-Services: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10Urbanecm) a:05Urbanecm>03GTirloni Reassigning, as I didn't complete this task - it was assigned to me just to indicate it is blocked on me. Feel free to assign to somebody wrong if I...