[00:07:54] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Thank you for the electrifying IRC meeting today! I owe a few actionables to the group, includin... [00:13:00] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team (Current), 10User-Joe: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10awight) p:05Triage>03Normal [00:17:02] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Part of completing these schemas will be to look at what MCR can do for the schema. It already s... [04:46:11] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [04:46:25] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [04:46:38] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [05:35:46] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [05:47:19] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [05:48:00] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [05:52:37] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [06:39:28] 10DBA, 10Patch-For-Review: Switchover es2 master (es1011) to es1015 - https://phabricator.wikimedia.org/T202364 (10Marostegui) 05Open>03Resolved All the clean up tasks are now done I have enabled semi-sync on es2: ``` root@es1015.eqiad.wmnet[(none)]> SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients';... [06:45:56] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [07:06:58] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [07:22:21] 10DBA, 10MediaWiki-Logging: Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore - https://phabricator.wikimedia.org/T164382 (10jcrespo) 05Open>03stalled [07:28:11] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [07:28:58] should https://phabricator.wikimedia.org/T189101#4031549 be done before and during maintenance? [07:30:03] I think we have to do the first part before [07:30:07] Like [07:30:08] yes [07:30:09] Alter codfw [07:30:18] then failover [07:30:22] and then reimport tables on eqiad [07:30:22] that is why I am asking now [07:30:26] yeah [07:30:36] it is only 2 wikis [07:30:44] so not a huge deal? [07:30:55] I am concerned about the PK, though [07:30:56] Yeah, we have to check just in case, but looks like 2 wikis [07:31:20] shouldn't it keep the unique restriction? [07:31:41] or be dropped? [07:33:23] We could ask yeah [07:36:08] 10DBA, 10MediaWiki-Logging: Evaluate the need for FORCE INDEX (ls_field_val) [now IGNORE INDEX (ls_log_id)], delete the index hint if not needed anymore - https://phabricator.wikimedia.org/T164382 (10aaron) I don't recall. It's been long enough that it's worth testing how queries run without it. [07:38:50] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, 10Schema-change: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101 (10jcrespo) @Lucas_Werkmeister_WMDE What is the plan with constraint_guid?... [08:09:03] I want to restart dbstore2001 and dbstore2002, is that ok with you? [08:09:14] fine by me! [08:26:33] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) This is how many minutes it takes for several backups to finish (it may be missleading, as because concurrent backups, it can vary a lot): ``` root@db1115.eqiad.wmnet[zarcillo]>... [08:29:16] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10Marostegui) That is very useful! I know this is too early to discuss it but maybe, a good alert can be based on minutes, like if a backups takes more than 24h something might be going on. [08:36:05] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) [08:36:51] 10DBA, 10MediaWiki-Database, 10Patch-For-Review: Drop blob_tracking and blob_orphans everywhere - https://phabricator.wikimedia.org/T59186 (10Marostegui) 05Open>03Resolved After almost 5 years...tables gone! :) [08:36:54] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [08:45:48] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) Fixed differences on `echo_notification` on the following wikis: arwikisource arwikibooks bgwiki The check for that table still on-going [09:08:20] 10DBA, 10Cloud-Services: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [09:14:28] 10DBA, 10Cloud-Services: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) For the record, these are the local databases in cloudcontrol1003.wikimedia.org before this task: ``` mysql> show databases; +--------------------+ | Database | +------------... [09:18:26] 10DBA, 10Cloud-Services: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) Some questions to try to get a picture of what are the implications: How big are those databases? How many concurrent connections they have? Are those throttled? (reminder: T188589)... [09:30:45] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) >>! In T202549#4526012, @Marostegui wrote: > Some questions to try to get a picture of what are the implications: > > How big are those databases? > How many conc... [09:49:00] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) >>! In T202549#4526124, @aborrero wrote: > > The DBs are the `nova_eqiad1` and `nova_api_eqiad1` ones which have been bootstrapped but have no actual usage righ... [09:56:40] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) >>! In T202549#4526186, @Marostegui wrote: >>>! In T202549#4526124, @aborrero wrote: >> >> The DBs are the `nova_eqiad1` and `nova_api_eqiad1` ones which have bee... [10:04:59] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) >>! In T202549#4526214, @aborrero wrote: >> > > Well, there is some data, but just basic. A couple of services registered and a couple of test VMs, but nothing... [10:19:41] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [10:20:01] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10jcrespo) Backup configuration for m5 should be reviewed too. [10:27:50] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) >>! In T202549#4526260, @jcrespo wrote: > Backup configuration for m5 should be reviewed too. What do you mean? Are we doing backups of this server, right? Do yo... [10:31:18] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) >>! In T202549#4526283, @aborrero wrote: >>>! In T202549#4526260, @jcrespo wrote: >> Backup configuration for m5 should be reviewed too. > > What do you mean? A... [10:37:11] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) Forget that last part of dumps-misc that is not used anymore :-) [10:42:30] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) >>! In T202549#4526217, @Marostegui wrote: > > It would be good if you could work out the math to see how many connections will be allowed in total from these ser... [10:43:25] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) >>! In T202549#4526309, @aborrero wrote: >>>! In T202549#4526217, @Marostegui wrote: >> >> It would be good if you could work out the math to see how many conne... [10:54:59] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [10:55:09] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [10:55:26] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [11:02:56] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) s2 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1122 [] db1105 [] db1103 [] db1095 [] db1090 [] db1076 [] db1074 [] db1066 [11:02:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) s2 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1122 [] db1105 [] db1103 []... [11:03:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) s2 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1122 [] db1105 [] db1103 [] db1095 [] db1090 []... [11:06:41] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) >>! In T202549#4526312, @Marostegui wrote: > > Is that throttled already? Apparently. @Bstorm applied same config to both deployments (main, eqiad1). [11:07:48] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) Yes, but there is stuff that has not been properly throttled yet: T188589#4523613 [11:36:07] 10DBA, 10Operations, 10Epic: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [11:58:46] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [11:59:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [11:59:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [12:14:08] 10DBA, 10Patch-For-Review: Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)) - https://phabricator.wikimedia.org/T184888 (10Marostegui) @Papaul do you think we can use the BBU from db2064 (T195228) on this host? I am starting to worry that the new x1 host (T199501) will not arrive on tim... [12:32:50] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) This has broken views from dewiki (as @Krenair noticed this morning). That table is a full view, so when a column drops, the view is broken (and I confirmed the table is the sam... [12:41:25] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) >>! In T114117#4526521, @Bstorm wrote: > This has broken views from dewiki (as an IRC user noticed this morning). That table is a full view, so when a column drops, the view... [12:47:06] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) That'd be great. I somehow worry that if I change the view, I'll forget about it later, even though it is a more straightforward thing. [12:47:22] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) Re-running that table on all replicas then. [12:48:34] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) ok I will ping you every time. I can also narrow it to the wikis affected, so you only have to run it for the given wikis. Once s3 is reached (around 800 wikis) you can do a... [12:53:26] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) A full re-run on just one table is fine when commonswiki isn't locked. //rolls dice to see--yup, it is// I have to also do a full re-run on the table to change to a custom view... [12:54:38] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) >>! In T114117#4526576, @Bstorm wrote: > A full re-run on just one table is fine when commonswiki isn't locked. //rolls dice to see--yup, it is// > > I have to also do a fu... [15:39:25] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10Bstorm) Looking closer at the nova connections right now, they are all for two older servers. That said, we have 8 workers and one master... [16:13:30] 10DBA, 10Cloud-Services, 10Operations, 10Patch-For-Review: m5-master overloaded by idle connections to the nova database - https://phabricator.wikimedia.org/T188589 (10aborrero) >>! In T188589#4527126, @Bstorm wrote: > Looking closer at the nova connections right now, they are all for two older servers. T... [16:33:54] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [19:19:25] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Bstorm) There are definitely more workers on there, and larger pools than are on the labcontrol1001/labnet1001 hosts right now. Since this environment is queried by most th... [19:32:23] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Bstorm) If we feel good merging that, and the local db connections seem appropriate after it is in place (they really shouldn't go to 100 right now), it should be less destr... [22:34:03] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) The meeting minutes are pretty accurate, but I wanted to collect a few more points for my own ref...