[04:13:54] db1071 depool is normal upgrade (db1070 and 71 are remaining precise+10.0.13 slaves) [05:34:07] https://phabricator.wikimedia.org/T106470 [13:54:46] jynus: "My house isn't on fire. It's just currently experiencing an uneven heat distribution." ;-) [13:55:01] That's how I read your database corruption versus database integrity comment. [13:56:03] well :-) [13:56:44] corruption on InnoDB means the db stops immediatelly [13:57:28] acoorring to most labs users, they prefer having wrong data but it being available than downtime [13:58:26] so things at a bit different than how production is handled, where a single drift detected means the server is depooled (because we have other 6) [13:58:47] Is 1069 depooled? [13:58:57] I hadn't realized a production slave was also having issues. [13:59:18] 1069 is not used in production, it just sanitizes data from the master [13:59:30] Hmmm. [13:59:58] so, do not worry, i know about that, and will try to fix it [14:00:10] the how and when is more difficult to say [14:00:46] it is not "we will try", it is "we will fix it" [14:01:09] :-) [14:01:31] maybe we will resync db1069, then slowly apply the changes to labsdb hosts [14:01:59] (although that will create lag for all enwiki) [16:13:28] there is a higher than usual write activity on s4 [19:31:03] Error 'Duplicate entry 'equivalent-fa' for key 'site_ids_type'' on query. Default database: 'hewiki'. Query: 'INSERT /* DBSiteStore::saveSites */ INTO `site_identifiers` (si_site,si_type,si_key) VALUES * on dbstore200[12] [19:34:02] restarting replication does not work, skipping event (it is a duplicate key) does [19:40:55] manually executing UPDATE site_identifiers SET si_site='203' WHERE si_type='equivalent' and si_key='fa'; to sync it with the right value [19:45:15] I will let you identify the root cause, if you want [19:45:21] it is too late for me