[05:11:28] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4223943 (10Marostegui) Thanks for the clarification! At some point, it would be... [05:17:46] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4223946 (10Marostegui) Deletion progress: [x] codfw eqiad: [] dbstore1002 [] l... [06:20:09] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4224032 (10Marostegui) This is how the table looks like after removing all the i... [06:30:53] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4224042 (10Marostegui) @Ladsgroup I am not sure we should keep on with this task... [06:44:15] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4224051 (10Marostegui) That query takes 0.00 on db1104 (which has the indexes) a... [07:13:14] for some reason, db2064 is alerting? [07:13:33] uh? [07:13:54] Ah [07:14:08] Maybe because puppet never run on it as it is off? weird [07:14:22] I will disable notifications [07:17:47] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4224083 (10Marostegui) [07:18:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4224084 (10Marostegui) [07:18:06] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4224085 (10Marostegui) [08:10:59] 10Blocked-on-schema-change, 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10User-Addshore: Change DEFAULT 0 for rev_text_id on production DBs - https://phabricator.wikimedia.org/T190148#4224148 (10Marostegui) [08:11:17] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Ministry-Of-Magic: Schema change for rc_namespace_title_timestamp index - https://phabricator.wikimedia.org/T191519#4224149 (10Marostegui) [08:11:38] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4224150 (10Marostegui) [09:26:14] marostegui: hey, I just got into the office [09:26:24] o/ [09:26:44] let me check [09:27:02] :) [09:31:28] marostegui: can you keep wb_terms_search_key for now until I figure out where it's using term_search_key field and fix it? [09:31:46] yeah, I am reading all of those, just leaving tmp1 aside [09:33:43] no place in the code should be reading from it [09:34:11] yeah, I didn't see anything with tmp [09:34:24] but with the other one, that one query at least gets a huge regression [09:37:37] that should not happen, we should not be reading from that field specially since we are replacing it with empty string [09:39:17] I can try an analyze table on codfw (as the indexes have not been re-added yet there) and see if the query plan changes [09:39:20] let me try that [09:39:54] The analyze will take a few hours of course, as that table is almost 1T :) [09:40:32] if you optimize it it'll be around half [09:40:36] :D [09:40:59] \o/ [10:32:32] marostegui: I found out why it's happening and I made a config patch to fix it, it'll get deployed in morning SWAT (6 hous from now) [10:33:11] cool [10:33:23] I will keep the alter running on db1092 to get the indexes added back [10:33:36] Drop them is fast, so we can test it when your patch gets deployed or tomorrow [10:34:01] I don't want to kill the alter and go thru all the rollback process [17:12:26] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4226142 (10jcrespo) db1117:m5 and db1065 are now loading m2 from a logical dump. The only thing missing aside from that is the copy of m3 and m5 to db2078, and the required switchovers of service. [17:24:45] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4226164 (10Ladsgroup) Okay, with the patch deployed, I think we are good to go h... [17:28:34] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), 10Patch-For-Review: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4226169 (10Marostegui) Great, I will repool db1092 tomorrow and check how it goes [17:31:57] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4226170 (10Marostegui) [17:33:30] 10DBA, 10Operations, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792#4226176 (10Marostegui) [17:33:33] 10DBA, 10Patch-For-Review: Productionize 8 eqiad hosts - https://phabricator.wikimedia.org/T192979#4167457 (10Marostegui) 05Open>03Resolved a:03jcrespo Let's close this task. There are 4 hosts that will be freed up once we have moved to the definitive HW for the new sanitarium hosts (T194780) Let's track...