[04:35:32] DBA, Operations, ops-codfw: BIOS upgrade on certain codfw machines - https://phabricator.wikimedia.org/T139714#2440561 (jcrespo) [04:36:18] DBA, Operations, ops-codfw: pc2006 down - https://phabricator.wikimedia.org/T139283#2440574 (jcrespo) Open>Resolved Tracked on T139714 [04:39:12] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2440590 (jcrespo) Open>Resolved I consider this as resolved, let's track BIOS and followup on T139714. [06:32:34] DBA, Operations, Patch-For-Review: All s4 (commons) slaves failed replication - https://phabricator.wikimedia.org/T139346#2440643 (jcrespo) Open>Resolved Incident documented on https://wikitech.wikimedia.org/wiki/Incident_documentation/20160705-commons-replication [11:38:02] DBA, Operations, ops-codfw: BIOS upgrade on certain codfw machines - https://phabricator.wikimedia.org/T139714#2441108 (MoritzMuehlenhoff) p:Triage>Normal [15:12:31] jynus: If I wanted a snapshot of reviewdb to test the upgrade path, what would my options be? Could we copy it over (on the slave of m1) to a second like reviewdb_test, or do like a mysqldump from slave so I can test on a secondary machine? [15:13:19] are we talking db or software upgrade? [15:13:30] Software upgrade, will involve schema changes [15:14:20] we setup a db on produduction with a copy of it, then a second install pointing to it [15:15:02] normally for other ops they do it themselves, I can do it for you if you file a ticket [15:15:10] I can file that right now [15:17:09] then, on actual upgrade I do a backup to be able to go back in case of issues [15:17:45] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441652 (demon) [15:18:11] jynus: That was my thought too. Thanks :) [15:18:53] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441666 (demon) [15:19:17] assuming this was available now, how much would the test take? [15:19:32] How much? Disk space? [15:19:37] time [15:19:49] I have some host available but they should be decommed soon [15:19:57] I could have it done by the end of today [15:20:16] ok, so little time [15:20:29] I won't be able to get it today [15:20:43] because I am already doing overtime to get m3 ready [15:21:00] but it should not take much time [15:21:12] at some point, next week [15:21:18] Could you get me a mysqldump today? I could install mysql on lead locally long enough to test. [15:22:32] ok, but then m3 will be delayed until next week [15:25:59] I know. If there's anything I can do to make things easier or help please ask. I'll raise the priority on the task [15:27:20] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441690 (demon) p:Normal>Unbreak! Because of {T139669} [15:29:39] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441652 (jcrespo) a:jcrespo [16:12:43] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441652 (Krenair) Actually, reviewdb lives on m2, I believe since 2012: ```alex@alex-laptop:~$ ssh gerrit gerrit gsql Welcome... [16:15:26] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441932 (jcrespo) Yes, I didn't even read the description, I am copying from db. I had already dropped old version that used t... [16:17:29] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441935 (demon) [16:40:20] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2441986 (jcrespo) Open>Resolved ``` root@db1042:~$ my reviewdb -e "SHOW TABLES" +-----------------------------+ | Tabl... [16:58:16] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2442059 (jcrespo) I corrected some, but there is still some drift between db1043 and db1048. We cannot progress until that is fixed. [17:01:55] DBA, Operations, Phabricator, Patch-For-Review: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2442065 (mmodell) @jcrespo is there anything I can do to help? [17:05:07] DBA, Gerrit, Operations, Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2442077 (demon) `db1042`, got it!