[06:16:24] 10DBA: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10Marostegui) [06:18:10] 10DBA: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10Marostegui) p:05Triage→03Normal [06:37:54] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui) a:03Marostegui As this has been a very active table, I will start by renaming and will leave it for a long time. Even if there are no writes, we _really_ need to make sur... [06:42:33] 10DBA, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban): [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) Looks like it is indeed not in use: ` root@db1073:/srv/sqldata/nodepooldb... [06:42:46] 10DBA, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban): [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) a:03Marostegui [06:45:18] 10DBA, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban): [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) User removed ` root@db1073.eqiad.wmnet[(none)]> drop user if exists 'node... [06:55:15] 10DBA, 10Patch-For-Review: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2057.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201812190655_marostegui_16179... [06:59:34] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) GTID enabled on s1 db2048 and it looks good: ` Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19 6:58:26 140190713382656 [Note] Slave SQL thread exiting, repli... [07:01:07] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) GTID enabled on s8 db2045 and it looks good: ` Dec 19 07:00:06 db2045 mysqld[3411]: 2018-12-19 7:00:06 140342229124864 [Note] Slave SQL thread exiting, repli... [07:01:12] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) [07:13:31] 10DBA, 10Operations, 10ops-codfw: Upgrade db2057 firmware - https://phabricator.wikimedia.org/T212277 (10Marostegui) p:05Triage→03Normal [07:20:17] 10DBA, 10Patch-For-Review: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2057.codfw.wmnet'] ` and were **ALL** successful. [07:38:40] 10DBA, 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review, 10Release-Engineering-Team (Kanban): [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) 05Open→03Resolved Database dropped: ` root@db1073.eqiad.wmnet[(none)... [08:07:32] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) Confirmed empty on all wikis. [08:07:55] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:27:46] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:27:56] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) a:03Marostegui [08:40:21] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:43:33] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:46:04] marostegui: hey, just one thing regarding tsg_summary. Please rename it after wmf.9 is deployed everywhere (tomorrow evening) [08:46:32] For valid_tag, it doesn't matter [08:47:23] Amir1: I think I won't touch tag_summary till we are back from holidays, I don't want to rename it and leave it like that in a period that we won't have many eyes on it [08:47:27] Does that make sense? [08:47:55] Yeah sure [08:48:24] I hope it improves the performance [08:48:27] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:49:46] I am glad we are getting rid of it :) [08:58:05] marostegui: in the morning I'd like to proceed with T211544 (Backup the tables on the master, and after drop them there with replication enabled) and T85757 do you have any objections? [08:58:06] T211544: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 [08:58:06] T85757: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 [08:58:09] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [08:58:20] banyek: sounds good [08:58:28] 👍 [08:59:24] I hope today I can put up the recatored multiinstance puppet module (for dbstores1003-5) but we'll wait on Jaime anyways with that, so I won't rush it [08:59:45] yeah, put it up so we can discuss [09:20:34] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) I've created backups from the actual tables before the drop: `root@db1066:~/backup_T21544# for table in $(mysql -BN -e "show tables like 'flagged%'" ptwiki --s... [09:39:42] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) `root@db1122.eqiad.wmnet[ptwiki]> SET SESSION sql_log_bin=0; Query OK, 0 rows affected (0.03 sec) root@db1122.eqiad.wmnet[ptwiki]> show tables like 'T211544%';... [09:53:36] 10DBA, 10Patch-For-Review: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10Peachey88) [10:11:36] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) `root@db1066.eqiad.wmnet[ptwiki]> show tables like 'flagged%'; +-----------------------------+ | Tables_in_ptwiki (flagged%) | +-----------------------------+ |... [10:12:05] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) 05Open→03Resolved [10:51:43] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [11:14:01] 10DBA, 10Patch-For-Review: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10Marostegui) 05Open→03Resolved db2057 has been reimaged and recloned. [11:15:11] 10DBA, 10Operations, 10ops-codfw: Upgrade db2057 firmware - https://phabricator.wikimedia.org/T212277 (10Marostegui) @papaul server is powered off, so you can proceed whenever you can. Once you are done, power it on and we will start MySQL and repool it Thanks! [11:15:35] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [11:35:35] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [11:51:45] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [12:10:54] I grab some food [13:17:08] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [13:31:39] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [13:31:49] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) 05Open→03Resolved I have dropped the table everywhere [13:31:54] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [13:37:28] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) I have renamed this table on db1089 (s1) ` root@db1089.eqiad.wmnet[enwiki]> set session sql_log_bin=0; rename table valid_tag to T212254_valid_tag; Query OK, 0 rows affected... [13:37:52] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) a:03Marostegui [13:38:00] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [13:38:28] Amir1: I have renamed valid_tag on a host on enwiki, just to make sure nothing keeps reading from it [13:39:00] thanks! [13:58:28] 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui) [14:01:03] Amir1: what's the story behind valid_tag? just curiosity [14:01:13] I mean, what was it for and why it only has 11 rows on enwiki? [14:02:25] it used be to determine tags that admins define on wiki (vs. tags that software defines) now it resides in a column in change_tag_def. No one knew it existed, we found it while normalizing change_tag table [14:02:56] like no mediawiki developer (it was added ten years ago) [14:03:15] yeah - I was checking the rows on enwiki [14:03:22] and made no much sense to me [14:04:13] and you are just seeing the database design, the codebase for change tags makes me cringe [14:04:23] :D [14:04:49] the whole design of that part of mediawiki is horrible on so many levels [14:05:47] hahaha [14:07:14] so it was used at some point or not even? [14:27:00] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Marostegui) >>! In T200297#4829689, @awight wrote: > Here are some example queries to help with reviewing the DDL. @Marostegui,... [14:28:14] marostegui: the valid_tag? It's being used but super rarely [14:28:26] It used to being used, now it's in change_tag_def [14:30:43] Super rarely…indeed 11 rows in 10 years! [15:02:07] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Marostegui) >>! In T200297#4829689, @awight wrote: > This is a recentchanges query which filters on the same field, so only show... [15:06:48] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [15:07:10] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [15:07:54] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Marostegui) [15:24:52] I'll go soon for the kindergarten, but I'll be online after [15:25:23] ok [15:25:28] I started to refactor the multiinstance profile, I think the patch is large, but the code simplified [15:26:03] I'll do the analytics_dbstore with the 'old' way, but it won't be hard to refactor into the 'new' one if that gets accepted [15:26:26] add me as a reviewer so I can see it :) [15:26:54] I added you, but maybe it's WIP and there was no notification? https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480750/ [15:27:28] Ah weird, I didn't get the mail [15:27:42] the dbstore2002 is weird and the read_only variable which behaves odd [15:28:06] interesting, it also doesn't show up under: incoming reviews [15:28:30] maybe it is because of it is work in progress indeed [15:28:41] what is weird about dbstore2002? [15:29:12] if you check the catalog compiler it shows a lots of changes on that host [15:29:32] on the others I've seen almost none [15:29:50] but probably I just messed up the hieradata file [15:30:38] mmmm weird [15:30:46] maybe it is because you are going from a profile to a role? [15:30:53] I haven't checked the whole patch [15:30:55] just guessing [15:31:04] by checking [15:31:05] - config => role/mariadb/mysqld_config/dbstore_multiinstance.my.cnf.erb [15:31:08] + config => profile/mariadb/mysqld_config/dbstore_multiinstance.my.cnf.erb [15:31:16] I don't know, I just moved the config only once [15:31:41] all the others had the config template under 'profile' and this one had it under 'role' [15:31:47] yeah, but you are doing modules/role and removing module/profile [15:32:08] I'll take an another look on this, with rested eyes :) [15:32:38] but the point is: I merged all the multiinstance profiles into one, but kept the separate roles [15:32:41] it doesn't happen to dbstore2001 [15:32:43] weeeeird [15:33:33] yeah [15:33:54] Anyways, I'll go now for Bori, because She's wainting me, and take an another look later [15:34:00] byee [15:34:05] bye!! [15:44:19] 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Bstorm) [16:06:53] 10DBA, 10Operations, 10ops-codfw: Upgrade db2057 firmware - https://phabricator.wikimedia.org/T212277 (10Papaul) a:05Papaul→03Marostegui Firmware upgrade complete [16:08:27] 10DBA, 10Operations, 10ops-codfw: Upgrade db2057 firmware - https://phabricator.wikimedia.org/T212277 (10Marostegui) Thank you - I will take it from here! [16:43:38] the disk on db1072 is swapped..please check on it [16:44:43] checking [16:47:56] cmjohnson1: megacli (and nagios) says the disk state is degraded [16:49:31] not rebuilding? [16:49:39] not [16:50:15] now it is rebuilding [16:50:58] maybe I was too fast, and the fw doesn't seen the disk? [16:51:02] okay....i just checked it [16:51:15] let's see what happens [16:51:23] 🤞 [17:45:40] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4793282, @Marostegui wrote: > What's the expected growth for that table? Once Jade is fully accepted by... [17:58:56] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T212185 (10Cmjohnson) 05Open→03Resolved The disk is back RECOVERY - MegaRAID on db1072 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy cmjohnson@db1072:~$ sudo megacli -PDList -aALL |grep "Firmware... [18:07:59] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T212185 (10Marostegui) Thank you!! [18:51:38] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4834319, @Marostegui wrote: > Other than a possible misbehaviour of the optimizer, they look ok to me. W... [20:08:29] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10MusikAnimal) If you need a use case, https://xtools.wmflabs.org/autoedits in particular used to be //much// fast... [21:32:07] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) [21:33:50] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) Do you happen to know how big a batch is? [21:34:56] marostegui: ^^ what do you mean? :) [21:35:14] The request is to delete 300+ pages [21:48:29] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) >>! In T212346#4835619, @Marostegui wrote: > Do you happen to know how big a batch is? I'm not sure I understand the question. The batch of pages to delete is actually listed above and consist of 350... [23:05:28] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Nuria) @MusikAnimal note that on the proposed scheme these views are not real time though, they are recreated mo...