[05:16:56] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: db1105 rebooted itself - https://phabricator.wikimedia.org/T235877 (10Marostegui) The data comparison finished correctly (still no HW) logs. I am going to give this host some weight to help out db1099:3311 so it doesn't get super cold. @Cmjohnson let us know whi... [05:21:57] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [06:10:29] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `dbproxy1006.eqiad.wmnet` - dbproxy1006.eqiad.wmnet (**PASS**) - Downtimed host on Icinga - Downtimed management interfa... [06:23:57] 10DBA: Drop labtestwikitech from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [06:24:26] 10DBA: Drop labtestwikitech from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) p:05Triage→03Normal [06:28:20] 10DBA: Drop labtestwikitech from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) These are the active grants on the DB: ` GRANT ALL PRIVILEGES ON `labtestwiki`.* TO 'wikiadmin'@'10.192.32.22' GRANT ALL PRIVILEGES ON `labtestwiki`.* TO 'wikiadmin'@'10.192.32.24' GRANT ALL PRIVILEGES ON `labtestwiki`... [06:28:42] 10DBA: Drop labtestwikitech from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [06:48:59] 10DBA: Drop labtestwikitech database from m5 - https://phabricator.wikimedia.org/T236010 (10Marostegui) [06:49:58] 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Marostegui) [06:50:13] 10DBA, 10Epic: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Marostegui) [07:04:01] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [07:07:41] 10DBA, 10Core Platform Team, 10MW-1.34-notes (1.34.0-wmf.24; 2019-09-24), 10Performance Issue, 10mariadb-optimizer-bug: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) >>! In T223151#5590287, @Stashbot wrote: > {nav icon=file,... [08:29:49] 10DBA, 10Core Platform Team, 10MW-1.34-notes (1.34.0-wmf.24; 2019-09-24), 10Performance Issue, 10mariadb-optimizer-bug: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) So far on `enwiki` no slow query reported only on db1089. T... [09:25:25] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [09:25:35] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [10:16:36] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10jcrespo) a:05jcrespo→03None [13:24:47] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [19:23:01] marostegui: Sorry for the late notice on this (missed the schedule around it), but will you be around on Thursday if one of us needs help with clouddb1001 needing a failover if it does? The cloudvirt it is on is getting PDU refresh. Our plan is to just briefly stop mariadb and keep going with things, but just in case we need a failover and it doesn't go as documented/planned... [19:23:47] I don't anticipate us needing any help, but I was just wondering if you or jynus will be around for questions or desperate cries for help. [19:50:25] 10Blocked-on-schema-change, 10DBA, 10Analytics, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Nuria) [19:51:38] 10Blocked-on-schema-change, 10DBA, 10Analytics, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Nuria) Pinging analytics temporarily so we know these changes are happening, it shoudl not affect scooping of these tables cause i thin... [20:05:30] bstorm_: i should be around yeah. I do recommend stopping mysql as you mentioned (we are doing so with the production DBs that are also affected my the pdu maintenances) [20:06:50] Thanks!