[00:36:38] 10DBA, 10Data-Services: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3730774 (10bd808) [01:04:04] 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3731402 (10bd808) [07:01:24] 10DBA, 10Data-Services: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3730774 (10Marostegui) To be able to enable this we'd need to also set `innodb_file_format` to Barracuda (which is fine) but this would also require the tables to be DYNAMIC or COMPRESSE... [07:11:23] 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3731402 (10Marostegui) I am going to put this on hold until the 14th when Jaime is back, because I cannot remember if this GRANT was excluded for some specific reason when we set up the... [07:56:54] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3731696 (10Marostegui) I am going to do s4 codfw slave by slave (as db2019 is already done), t... [08:02:11] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3731706 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db1103.eqiad.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017110... [08:02:44] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3731707 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1103.eqiad.wmnet'] ``` Of which those **FAILED**: ``` ['db1103.eqiad.wmnet'] ``` [08:02:54] ah, because of the downtime [08:08:14] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3731712 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db1103.eqiad.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017110... [10:08:38] marostegui: o/ [10:08:39] mysql:research@db1108.eqiad.wmnet [(none)]> use log [10:08:40] ERROR 1044 (42000): Access denied for user 'research'@'%' to database 'log' [10:08:43] :( [10:09:13] checking [10:11:54] fixed [10:11:59] let me check all the grants [10:12:02] to see what else is missing [10:16:04] elukey: db1108:/root/grants.diff [10:16:06] check that file [10:16:11] you want me to apply those too? [10:17:30] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3732181 (10Marostegui) db2085.s5(s8) is done. It is replicating s3 and s5. It will replicate s8 in the future (as specified in the original task description) but as s8 will be a split of wikidata from s5... [10:17:46] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3732182 (10Marostegui) [10:20:03] marostegui: so those are between db1047 and db1108? [10:20:15] yep [10:21:35] checking [10:25:09] so marostegui I'd grant only the bare minimum for the log databse [10:25:34] I can see a lot of grants for databases [10:25:59] check the log.grants file [10:26:04] on the same host and directory [10:51:18] marostegui: +2 [10:51:55] ok i will apply those [10:56:54] elukey: applied [11:05:20] thanks! [11:07:34] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3732403 (10Marostegui) db1103.s4 is now set up. I am compressing InnoDB there. [11:54:59] ok starting eventlogging cleaner on db1108 to see how well it performs with deletes and updates :) [11:56:13] great! [12:26:07] 10DBA, 10Wikidata: Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114904#3732816 (10daniel) @Multichill wb_items_per_site is *always* items. So to get the full item ID, just use concat('Q', ips_item_id). That's not very nice, but viable I th... [12:26:16] 10DBA, 10Wikidata: Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114904#3732817 (10daniel) p:05High>03Normal [12:46:54] 10DBA, 10Cloud-VPS, 10Data-Services, 10Patch-For-Review: Add page_props.pp_value index to Wiki Replicas - https://phabricator.wikimedia.org/T140609#3732840 (10Marostegui) >>! In T140609#3732832, @gerritbot wrote: > Change 386973 merged by Marostegui: > [operations/puppet@production] wikireplicas: Add index... [13:26:08] elukey: how's the eventlogging cleaner going? [13:27:29] marostegui: I am doing 100k batches with 2s of sleep [13:27:54] https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=db1108&refresh=1m&orgId=1 [13:28:26] and the server isn't even noticing it :) [13:28:31] yeah :D [13:28:39] awesome! [13:29:21] it definitely is! this should unblock the eventlogging sanitization efforts finally! [13:29:37] bye bye db1047 and db1046! [13:30:28] SSDs ftw [13:31:24] the migration of db1046 should not be too painful, no? [13:34:41] in theory we should stop eventlogging_sync on the replicas, stop eventlogging on eventlog1001, and they we'd be free to do the maintenance [13:34:55] cool, and without the massive table, it should be "fast" [13:34:56] it needs to be announced since new data will not be available for a bit [13:34:58] like 24h or so [13:34:59] yeah [13:35:13] I cleaned up all the remaining big ones already on hdfs [13:35:26] <3 [13:35:27] so it should be relatively easy (at least compared to db1108) [13:36:03] actually I am seeing that db1046 is 700G [13:36:07] so we can just scp all the data [13:36:22] so it should take around 2h? [13:36:35] stop mysql on db1046, scp to db1107, and start in db1107 [13:36:39] not like with db1108 [13:37:54] it would be even better :) [13:38:07] yeah, i think we should do that [13:38:10] scp + mysql_upgrade [13:38:12] and that should be it [13:38:25] it should take around 2h (or less) [14:01:21] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3733114 (10Marostegui) [16:15:34] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3733538 (10Marostegui) db2037 done: ``` root@neodymium:/home/marostegui/T174569# ./check.sh db... [16:50:37] 10DBA: Decommission database hosts < db2030 (tracking) - https://phabricator.wikimedia.org/T176243#3733621 (10RobH) [16:50:39] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2010 - https://phabricator.wikimedia.org/T175228#3733622 (10RobH) [16:50:42] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3733623 (10RobH) [16:50:44] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3733619 (10RobH) 05Open>03Resolved Whoever went ahead and started the steps marked 'non interruptable' and skipped the switch port disable, please do not do... [17:14:51] 10DBA, 10Data-Services: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3733671 (10russblau) To answer your question: We cannot ensure that titles in a user database are unique unless we can set a UNIQUE KEY on the entire column; two titles might be identica... [17:18:37] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3733680 (10RobH) [17:19:36] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2010 and move m1 codfw to db2078 - https://phabricator.wikimedia.org/T175685#3600184 (10RobH) [18:46:27] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3734020 (10jrbs) I'm sorry. I have no real idea how to reach devs for deployment here. This wiki would be really useful to have right n... [19:18:01] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3734131 (10Reedy) a:03Reedy >>! In T174370#3734020, @jrbs wrote: > I'm sorry. I have no real idea how to reach devs for deployment he... [19:21:39] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3734134 (10jrbs) >>! In T174370#3734131, @Reedy wrote: >>>! In T174370#3734020, @jrbs wrote: >> I'm sorry. I have no real idea how to r... [19:25:06] 10DBA, 10Cloud-VPS, 10Data-Services, 10Patch-For-Review: Add page_props.pp_value index to Wiki Replicas - https://phabricator.wikimedia.org/T140609#3734154 (10bd808) >>! In T140609#3732840, @Marostegui wrote: > I have reverted this change after merging it, because I didn't notice the column is a blob. To b... [20:26:19] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3734326 (10Marostegui) db2044 done: ``` root@neodymium:/home/marostegui/T174569# ./check.sh db... [22:30:19] 10DBA, 10Data-Services: Consider granting `CREATE TEMPORARY TABLES` to labsdbuser - https://phabricator.wikimedia.org/T179628#3734627 (10bd808) [22:30:22] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3734626 (10bd808)