[05:58:40] Reedy: Don't be sorry! The more tables to drop, the better! That is actually why I added you on the ticket, to make sure nothing was left behind. So thank you! [06:06:03] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui) After those last merges, is this good to be closed? @Papaul @robh? Thanks! [06:09:23] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) Thanks @reedy! ` root@cumin1001:~# for i in flaggedtemplates flaggedpage_config flaggedimages flaggedpage_p... [06:10:29] 10DBA, 10Operations, 10Performance-Team: Increase parsercache keys TTL from 22 days back to 30 days - https://phabricator.wikimedia.org/T210992 (10Marostegui) Thanks @aaron - I will get a patch out after the code freeze [06:13:41] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul) @Marostegui no need to close the task. It can be assign to @RobH so he can keep track [06:14:15] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui) a:05Papaul>03RobH [06:35:59] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [06:37:11] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:21:02] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [08:21:24] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s7 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1116 [] db1101 [] db1098 [] db1094 [] db10... [08:21:26] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:21:28] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s7 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1116 [] db1101 [] db109... [08:27:46] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) The materialized view generation completed. The total size of the materialized views are ~150 G all tog... [08:38:47] what's up with db1063? [08:52:44] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Zoranzoki21) >>! In T211544#4814942, @Banyek wrote: > ptwikipedia lives in the s2 section, the following hosts needs to be done: > > **s2** > [] db2035 **codfw master*... [08:53:19] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Zoranzoki21) 05stalled>03Open Oh, I forgot to change status to open. [09:04:18] it finished syncing [09:04:29] so raid back to optimal? [09:05:54] as I understand the output of the tool, it's optimal ' State: Optimal' but we still have the predictive failure in slot 7 [09:06:21] cool, let's close the ticket then when you have time, so Chris knows it is all good [09:06:31] 👍 [09:06:33] Yeah, the predictive failure we can forget about it until it finally fails [09:07:43] btw it's fascinating how better readable the output of the `md` devices through `/proc` than any of the hw raid devices [09:08:10] what do you mean? [09:08:46] yesterday when Christ swapped the disk I've seen that, and checked it, and the tool said it's rebuilding [09:09:23] and I remembered that if you use soft raid and cat the `/proc/mdstat` file it not just gives a state report, but it also shows the progress [09:09:31] yeah [09:09:39] but soft raid is terrible XDD [09:09:47] Indeed [09:09:49] I rather have a more complex syntax and a HW controller [09:09:50] XD [09:13:33] I stop replication on db1082 before apply the schema change, fix the triggers, and resume replication [09:17:15] marostegui: yay [09:17:38] addshore: keep me posted on the results! [09:17:47] will do, probably wont have anything until the new year [09:17:53] sure, no worries [09:28:49] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T211537 (10Banyek) 05Open>03Resolved The sync finished, thank you @Cmjohnson ` Virtual Drive: 0 (Target Id: 0) RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0 State: Optimal Number Of Drives per... [09:35:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:16:33] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [13:30:28] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) I'll do first renaming the tables on db1122, and if nothing breaks this week, I'll do the drops [14:46:01] 10DBA, 10Patch-For-Review, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) Tables were renamed on db1122 for proof: `root@cumin2001:~# mysql.py -hdb1122 ptwiki -e "show tables like '%flagged%';" +----------------... [18:22:25] 10DBA: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Banyek) [18:23:27] 10DBA: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10zeljkofilipin) [18:24:32] 10DBA, 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10zeljkofilipin) p:05Triage>03Unbreak! I've added it to train blockers as suggested by @Joe in #wikimedia-operations and raised priority to UBN (since all train blockers have that priority). [18:26:10] 10DBA, 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Anomie) All in all, I don't think there's really a reason to block the train for this any more than we should have been blocking every train since 2016 for T140302, which is the same issue except... [18:28:25] 10DBA, 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10greg) [18:28:50] 10DBA, 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10greg) p:05Unbreak!>03High Per @anomie. [18:29:44] 10DBA, 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10zeljkofilipin) Thanks @Anomie! [19:03:26] 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Anomie) [19:19:13] 10DBA, 10Core Platform Team Kanban (Waiting for Review), 10Patch-For-Review: A huge spike on read rows for commonswiki - https://phabricator.wikimedia.org/T211804 (10Marostegui) Thanks @anomie! Let's get it merged, yes, please! [19:25:18] marostegui: the day has come https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/479266 [19:49:18] Yaaaaaaaay [19:49:22] <3 [19:49:39] Amir1: can you take a look at https://gerrit.wikimedia.org/r/#/c/479257/ ? [21:19:40] Sure [21:26:52] Thank you :)