[05:55:21] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-11-29) rack/setup/install db214[234] - https://phabricator.wikimedia.org/T267041 (10Marostegui) Thank you Papaul. * Memory looks good * CPUs look good * Disk space looks good * RAID level looks good * pvs looks good (we need to add the last 1TB the... [05:56:24] 10DBA: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) [05:56:40] 10DBA: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) p:05Triage→03Medium [05:58:42] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Marostegui) >>! In T268312#6665177, @Bstorm wrote: > So I'm glad I noticed that warning. A few things have come out of it: > 1. I've fi... [06:06:22] 10DBA, 10Patch-For-Review: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10Marostegui) a:03Marostegui [06:21:06] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10Marostegui) @hnowlan the database and users are created. To connect you have to do use `m2-master.eqiad.wmnet`. The users are: ` +------------------------------------------------------------------------------------------------... [06:22:43] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [06:27:30] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10Marostegui) Database added to Misc section on wikitech: https://wikitech.wikimedia.org/w/index.php?title=MariaDB%2Fmisc&type=revision&diff=1890317&oldid=1890045 [07:10:36] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [09:22:40] there was some weirdness on db1086 20 minutes ago [09:23:21] some jobqueue user rename or something [09:32:15] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10jcrespo) >>! In T268505#6665806, @Marostegui wrote: > @jcrespo could you configure this database to be backed up as part of m2? Backup grants configured- I will check next week all backups run as expected. [09:34:33] I am going to uninvite individual people from SRE meeting, hoping that does what I think (invite you still via alias) [09:36:14] I think it worked, and you will avoid duplicate notifications, but check if Persistence meeting has disappeared from your calendars [09:40:13] cal entry still there for me [09:40:38] cool [10:01:44] 10DBA, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Slow load times for Special:Homepage on cswiki - https://phabricator.wikimedia.org/T267216 (10kostajh) a:03kostajh [11:06:43] 10DBA, 10wikitech.wikimedia.org: wikitech database has almost all of its varbinary fields wrong - https://phabricator.wikimedia.org/T269348 (10Ladsgroup) [12:04:31] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Jclark-ctr) downloaded and sent log again to hp. moved dimm again per hp request.. Error continues DIMM 7 to slot 9 and DIMM 9 to slot 7 DIMM 8 to sl... [12:38:52] hello hello [12:39:07] I'd need to move our dear db1108 to another rack [12:39:10] What is up? [12:39:16] hello sobanski :) [12:39:21] elukey: you need help? [12:39:25] Is that part of the 10g moves? [12:40:02] marostegui: just wanted to double check what to do - stop slaves; stop mariadb, shutdown (also pre-step - verify that no backup is in progress) [12:40:06] sobanski: exactly yes [12:41:09] elukey: I'd add downtime and all that should be enough. What I usually do is ask for the new IP beforehand, change it before shutdown so it comes back with the right IP already [12:41:48] elukey: I don't know if that host has slaves, but if they are configured to replicate with hostname (which I assume they are) nothing should be done on the slaves, they'll reconnect automatically [12:42:24] marostegui: so the IP will be the same, we'll move the rack within the row, that part should be ok.. the only thing that pulls from it is dbprov hosts for backups [12:42:54] elukey: ah no IP change! then it should be super smooth. Normally we umount /srv once mysql is stopped [12:43:04] to make sure nothing is really writing [12:43:06] all right will do :) [12:43:11] and apt full-upgrade too [12:43:13] if you want [12:43:18] good point too yes [12:43:37] never checked for in progress backups but I'll read wikitech [12:46:53] elukey: if you run the update, a new version of mariadb will be installed, from 10.4.13 to 10.4.15, so once mysql is up, remember to run mysql_upgrade -S /socket/location/ [12:49:34] marostegui: ah ok then I might avoid it for this move, just to limit the moving parts [12:50:15] elukey: It shouldn't be a big deal, but totally up to you, we can do it at any other time if you feel more comfortable [12:50:25] yep yep thanks :) [12:50:29] yw [13:19:13] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [13:20:00] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [14:19:22] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10WDoranWMF) @jcrespo @Marostegui I am speechless - it's possible I'm crying a little. Thanks for all your help. As a note @hnowlan is off till Monday but @gmodena is about I'll ask him to review and give any responses needed fr... [14:35:39] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10gmodena) @jcrespo @Marostegui you guys rock! Thanks a lot for helping out with this,. I could not test auth myself, but I'll prepare the create statements for the schemas and will sync with @hnowlan on monday. [15:13:41] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) [15:15:50] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) Yep, marking them done. Also I think I worked out the kinks in the maintain-dbusers process yesterday, so I should be able to g... [18:24:28] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) [18:31:24] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) [19:03:39] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) [19:57:00] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` db1151.eqiad.wmnet ` The log can be found in `/var/log/wmf-a... [20:19:55] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1151.eqiad.wmnet'] ` Of which those **FAILED**: ` ['db1151.eqiad.wmnet'] ` [20:22:39] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) 20:19:50 | db1151.eqiad.wmnet | Unable to run wmf-auto-reimage-host: could not convert string to float: "Warning: Permanently added the ECDSA host key for IP... [20:26:20] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` ['db1152.eqiad.wmnet', 'db1153.eqiad.wmnet', 'db1154.eqiad.w... [21:07:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` ['db1151.eqiad.wmnet', 'db1152.eqiad.wmnet', 'db1153.eqiad.w... [21:36:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1155.eqiad.wmnet', 'db1151.eqiad.wmnet', 'db1153.eqiad.wmnet', 'db1152.eqiad.wmnet', 'db1154.eqiad.wmnet... [21:46:22] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) So all the existing replicas will also now have the Toolforge user accounts. When we set up clouddb1020, we just need to run t... [22:01:06] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) [22:01:26] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) [22:53:06] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10RobH) >>! In T267043#6651222, @LSobanski wrote: > @Cmjohnson Would it be possible to plan for racking 5 instead of 3 of the new hosts in one go? It would help us p...