[08:44:58] labsdb1007 is close to finish [08:45:08] \o/ [08:45:21] it has imported all nodes and almost all ways [10:05:10] 10DBA, 06Operations, 13Patch-For-Review: Install and reimage dbstore1001 as jessie - https://phabricator.wikimedia.org/T153768#3090617 (10Marostegui) The most recent backups for dbstore1001 are still only from Feb, and not March: ``` +--------+-------+----------+-------------------+---------------------+----... [11:35:12] look how spiky things look: https://grafana-admin.wikimedia.org/dashboard/db/mysql-aggregated [11:35:35] wow [11:36:06] • 02:33 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.15) (duration: 12m 17s) [11:37:09] • 03:09 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.15) (duration: 14m 35s) [11:37:24] it has happenned before, maybe it is just a cron job [11:37:40] but keep it in mind in case it does get resolved [11:39:21] sure [11:39:53] can be https://phabricator.wikimedia.org/T130128 closed? [11:41:36] Yep! [11:41:43] I am about to close the x1 import ticket one too :) [11:41:51] so feel free to close that one [11:42:12] I am about to have lunch now [11:43:33] yeah me too [11:43:46] going to have lunch with a friend who i am trying to convince to apply for the ops position! [11:43:52] so wish me luck :p [11:44:45] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#3090723 (10Marostegui) 05Open>03Resolved a:03Marostegui We believe this is solved! [12:09:52] the relations may take another whole day at this speed [13:32:27] there is echo tables for example on enwiki, but not on test2wiki [13:32:35] on dbstore2001 [13:45:06] 10DBA, 13Patch-For-Review: Import x1 on dbstore2001 - https://phabricator.wikimedia.org/T159707#3075661 (10jcrespo) The replication broke because it is missing some tables? [14:48:56] test2wiki is in use?! [14:48:58] damn [14:49:24] my mysqldump excluded "test" [14:49:27] so it matched test2 [14:49:28] :( [14:49:28] did you delete? [14:49:30] no [14:49:31] no [14:49:34] we have testwiki [14:49:38] and test2wiki [14:49:42] i just ignored it on db2033 [14:49:42] those are production wikis [14:49:43] :_( [14:49:49] so i didn't dump it [14:49:50] what??? [14:49:54] why? [14:50:02] i thought it was not in use [14:50:15] so i didn't dump it [14:50:36] I think we discussed the ones to ignore- you even made a list [14:51:44] wait [14:51:59] with ignored them, you mean you didn't dump them? [14:52:09] or that you literally added ignore to db2033? [14:53:48] ok, you just ignored them on the import [14:53:54] then not a huge issue [14:54:05] but you know what is next, right? [14:54:49] yep [14:54:50] also I think there is a wikidatatestwiki [14:55:15] yep, going to review the regex [14:55:18] they are on the dblists [14:55:53] can you send me the regex? you should probably do %wik%, wikishared, flowdb and heartbeat [14:56:13] if x1 is a problem [14:56:20] which I agree [14:56:33] we can use a separate instance [14:56:38] I am reviewing it, i included testwiki [14:56:41] maybe that is easier [14:56:42] so it matched test2wiki [14:56:56] then something went wrong [14:57:14] maybe it didn't import some tables or something? [14:57:15] no, it means it was excluded [14:57:20] no no, test2wiki was excluded [14:57:32] ah [14:57:36] so just a mistake [14:57:39] yes yes [14:57:46] like ingore %test% instead of text [14:57:48] *test [14:57:53] yes :) [14:58:00] ok, no problem [14:58:04] I am plannig to work till late today, so I will leave it running anyways [14:58:05] I am only suffering for you [14:58:16] and not really, just two commands :) [14:58:16] because it is the 3rd time you import? [14:58:23] yes :_( [14:58:36] at least all the checks are done [14:58:43] in the sense, I am not dropping tables and all that [14:58:49] which is the pain with x1 and the other shards in the same host [14:58:52] and that worried me [14:59:20] also I understood at first [14:59:26] you have ignored them on db2033 [14:59:40] which worried me because that is a full production box [14:59:50] just passive [14:59:58] ah no no [15:00:05] db2033 was only used to get the dump :) [15:00:20] we can break dbstores as many times you want [15:00:50] alternatively, we can start trying out multi-instance there [15:01:07] this is another case that is simplified [15:01:42] I leave it to you [15:01:50] but do not work until late for this [15:01:57] this has 0 priority [15:02:47] haha no, no worries :) [15:07:27] BTW [15:07:37] why do you stop the slave to take a dump? [15:07:47] i normally prefer to do it [15:07:53] i could do master-data bla bla [15:08:04] but I normally prefer to do it that way, just manias I guess [15:08:07] ok [15:08:11] As in this case it doesn't really matter [15:08:19] I mean, as it can be stopped easily [15:10:17] 10DBA, 13Patch-For-Review: Rampant differences in indexes and PK on s6 (frwiki, jawiki, ruwiki) for revision table - https://phabricator.wikimedia.org/T159414#3091125 (10Marostegui) db1022: ``` root@neodymium:/home/marostegui/git/software# for i in frwiki jawiki ruwiki; do echo $i;mysql --skip-ssl -hdb1022 $i... [16:43:12] 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3091277 (10Marostegui) It took a long run to finish plwiki, but here are the results. differences in db1047 and quite lots of them in dbstore1002 too. [17:41:35] 10DBA, 13Patch-For-Review: Import x1 on dbstore2001 - https://phabricator.wikimedia.org/T159707#3091425 (10Marostegui) >>! In T159707#3090932, @jcrespo wrote: > The replication broke because it is missing some tables? As we spoke, that was due to a bad regex where it matched test2wiki and excluded it from the... [19:41:47] 10DBA, 13Patch-For-Review: Import x1 on dbstore2001 - https://phabricator.wikimedia.org/T159707#3091895 (10Marostegui) x1 is now replicating on dbstore2001. We will see if it breaks again (hopefully not!) - I will leave it replicating during the weekend.