[04:55:26] 10DBA, 10Operations, 10ops-codfw: db2061 disk with predictive failure - https://phabricator.wikimedia.org/T200059 (10Marostegui) [04:55:39] 10DBA, 10Operations, 10ops-codfw: db2061 disk with predictive failure - https://phabricator.wikimedia.org/T200059 (10Marostegui) p:05Triage>03Normal [05:21:30] 10DBA: db1067 /srv usage is at 82% - https://phabricator.wikimedia.org/T200039 (10Marostegui) 05Open>03Resolved a:03Marostegui The stuff in there was more than a year ago old and it didn't have even mysql privileges, it was clearly a leftover from a copy from some s2 host that was probably reimaged when db... [05:46:20] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10Marostegui) I will take db1120 (row C) instead of db1116 as db1116 is in row A (like db1069 - x1 master). The slave db1064 is in row D, so that way we have all the 3 hosts in different rows. [07:29:28] 10DBA: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:29:41] 10DBA: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) p:05Triage>03High [07:29:54] 10DBA: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) @Ladsgroup is it possible to disable this feature until we better understand what's going on? [07:34:43] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10jcrespo) [07:58:39] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) As per my IRC chat with Amir, I am checking consistency on wikidata. He will elaborate a bit more once he gets to the office on what the p... [08:00:31] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) [08:01:37] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) [08:02:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) [08:58:16] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Ladsgroup) I started the script to back populate change_tag_def from all.dblist and it was doing it on alphabetical order (until I stopped it five minu... [09:05:24] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) Timeline: frwiki.change_tag broke ONLY on codfw sanitarium, which is weird as they have the same data (codfw populated eqiad ones actually... [09:11:18] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Ladsgroup) Regarding bgwiki, I'm pretty sure it's because of the script revision '6114880' in bgwiki is made in 2014 (and we don't add new tags for old... [09:20:12] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) For db2095: This is what was being UPDATED and failed: ``` Last_SQL_Error: Could not execute Update_rows_v1 event on table... [09:39:12] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Ladsgroup) This is extremely weird: ``` wikiadmin@10.64.48.34(bgwiki)> select * from change_tag where ct_log_id = 3964232; +--------+----------+-------... [09:54:26] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) Maybe that is a leftover from all the work done on T154485? It wouldn't be strange as we checksummed thousands of tables so stuff could hav... [09:56:42] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) As this table is really small for bgwiki, I am going to work on it to fix the inconsistencies and will do the same with frwiki (as it is sm... [10:01:08] warning that I will be doing strange things with db1095 and db1102 (alerts are disabled) [10:01:14] :) [10:17:05] BTW, transferring db1095 to db1102 took 59 minutes once I disabled checksums and encryption [10:17:19] nice [10:33:11] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) So this is what we have in the backups for that particular entry (this is from dbstore2002 as that is the backup sour... [10:55:41] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [10:59:05] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) I think I know the history behind those inconsistencies on s2 and s6. Those two sections were done when we were relay... [10:59:53] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [11:00:33] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) a:03Marostegui [11:27:23] 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10jcrespo) Some progress reporting: ``` $ python3 Python 3.4.2 (default, Oct 8 2014, 10:45:20) [GCC 4.9.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import WMFMariaDB >... [11:43:27] 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10jcrespo) 0.7 seconds on performing a switchover with no writes ongoing (easy case): ``` root@neodymium:~/wmfmariadbpy/wmfmariadbpy$ time ./switchover.py db1095 db1102 Starting preflight checks... Setting up or... [11:43:58] marostegui: https://phabricator.wikimedia.org/T199224#4440829 :-) [11:44:11] :O [11:44:16] * marostegui hugs jynus [11:45:31] I will now generate fake write traffic to test it under more realistic conditions [11:45:43] yeah! but it looks super promising!! [13:46:47] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10jcrespo) >>! In T200061#4440613, @Marostegui wrote: > Maybe that is a leftover from all the work done on T154485? It wouldn't be... [14:55:16] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) When we ran pt-table-checksum we excluded tables without PK like change_tag at the time [18:17:28] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1055 - https://phabricator.wikimedia.org/T194118 (10RobH) [18:22:43] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1055 - https://phabricator.wikimedia.org/T194118 (10RobH) a:05RobH>03Cmjohnson [18:24:42] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1056 - https://phabricator.wikimedia.org/T193736 (10RobH) [18:33:56] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1056 - https://phabricator.wikimedia.org/T193736 (10RobH) [18:39:25] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1056 - https://phabricator.wikimedia.org/T193736 (10RobH) a:03Cmjohnson [18:49:25] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1060 - https://phabricator.wikimedia.org/T193732 (10RobH) [18:51:07] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1060 - https://phabricator.wikimedia.org/T193732 (10RobH) [18:56:59] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1060 - https://phabricator.wikimedia.org/T193732 (10RobH) a:05RobH>03Cmjohnson