[06:09:37] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T212185 (10Marostegui) p:05Triage→03High a:03Cmjohnson @Cmjohnson I have triagged this as high priority as this is m3 primary master (phabricator master) [06:13:57] 10DBA, 10Recommendation-API, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Marostegui) 30 running connections? That is a lot. Does it do connection pooling? [08:24:25] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) GTID enabled on es2 master (es2016) and all looks good ` Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18 8:23:36 140618110539520 [Note] Slave SQL thread exit... [08:26:51] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) GTID enabled on es3 master (es2017) and positions look good ` Dec 18 08:26:09 es2017 mysqld[3886]: 2018-12-18 8:26:09 139599671289600 [Note] Slave SQL thread... [09:22:43] 10DBA: Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights - https://phabricator.wikimedia.org/T211973 (10Marostegui) GTID enabled on db2040 (s7 master) ` Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18 8:44:07 140468267357952 [Note] Slave SQL thread exiting, replication sto... [09:23:45] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [09:23:55] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [10:14:20] 10DBA, 10Recommendation-API, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10mobrovac) >>! In T212154#4830288, @Marostegui wrote: > 30 running connections? That is a lot. > Doe... [13:46:50] marostegui: hey, is it okay if I drop 'valid_tag' table from prod? It's not being written/read for months now and it's being dropped any way [14:35:55] Is there many rows on it? [14:36:33] Amir1: if you're going to do it manually, IIRC you want to do it with DROP TABLE IF EXISTS incase it doesn't exist on some hosts as it can fuck up replication [14:37:21] Reedy: three rows in enwiki [14:37:42] it's empty in most wikis [14:52:51] Amir1: Fine by me, but it is better if we do it [14:53:06] Amir1: Specially on s3, which can cause lag if done on the master without pause between drops [14:53:13] marostegui: sure thing! [14:53:18] Amir1: If you want to place a ticket, I can take care of that [14:53:48] hmm, okay [14:53:48] Make sure to add it to: https://phabricator.wikimedia.org/T54921 [14:54:03] noted [14:54:16] Amir1: is that urgent for some reason? [14:54:26] nah [14:55:43] We normally even rename it for a few days before dropping it, just in case [14:56:45] yeah good idea, this thing would be easy though [15:55:48] 10DBA, 10Recommendation-API, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Pchelolo) >>! In T212154#4830612, @mobrovac wrote: >>>! In T212154#4830288, @Marostegui wrote: >> 3... [16:17:57] 10DBA, 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10hashar) p:05Triage→03Normal [21:06:18] 10Blocked-on-schema-change, 10DBA: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Ladsgroup) p:05Triage→03Normal [21:08:53] 10Blocked-on-schema-change, 10DBA: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Ladsgroup) p:05Triage→03Normal [21:09:11] 10Blocked-on-schema-change, 10DBA: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Ladsgroup) [21:23:19] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Zoranzoki21)