[12:10:48] anyone online an admin or bureaucrat on beta cluster? [12:19:44] anyone online that can add a user to a new group on beta cluster? [12:24:50] multichill: will you respond to hasher’s email? [12:25:04] do you think we need to create a gwtoolset-admin group? [12:25:15] dan-nl: No, that's just overhead [12:25:29] One group of admins is more than enough ;-) [12:25:49] it looks like all we need is a bureaucrat and make requests via them [12:26:18] dan-nl: Nope, a bureaucrat can't assign the group either, already checked that [12:26:33] dan-nl: Please join #wikimedia-glam [14:33:13] * Coren emerges and reads scrollback. [14:43:46] Betacommand: It must have been broken pretty badly; it was apparently completely wiped and restored. [14:44:20] Restoring the grants and views should take me ~30m. [14:44:51] Coren: went to check replag and got a ugly error sceen [14:47:40] Wiped and restored? Wow. [15:03:21] anomie: Apparently. Whoever did this didn't leave me a note and I see nothing in the admin logs. That said, the DB itself is okay and synced up; and replication is running. Since I see no counterindication, I'll restore the views and users now. [15:06:39] Hmmm. I see some debugging things running on the host. [15:10:40] I saw Coren is back in da hood ;) [15:11:42] Coren: Can you tell what happened to s6.labsdb? It says ACCESS DENIED to users since friday 13th! [15:13:03] hedonil: Apparently it was completely wiped and restored, and he is working on restoring the DB now. See http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20131214.txt [15:13:44] anomie: did read this. was beta also on s6? [15:17:13] Eeesh. I see leftovers logfiles and a screen session. It looks like this is still being actively watched/debugged. [15:30:13] Views created; doing grants now. [15:33:32] Can you tell who's debugging it? I'm not asking you to reveal that information to us, I'm just wondering whether you have someone to bug about it since they didn't leave you a note or an admin log entry. [15:35:15] anomie: Not directly, but my great detective superpowers tells me that would be our DBA. :-) [15:35:29] s6 should be back in the happy fun joy zone now. [15:38:00] Betacommand: Should be full of happies now. [15:41:37] Coren: yep http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag isnt broken any more [15:42:18] Coren: Oh my gosh! db u3710__stats is missing from s6 without a trace :-( [15:42:52] hedonil: That is an unfortunate side effect of a wipe-and-restore. The new setup in eqiad will have user database backups. [15:43:19] hedonil: But also, you shouldn't be using your own account for any actual work -- use a service account (aka 'tool') only. [15:43:58] hedonil: To be more precise, uXXXX databases will never be backed up. [15:44:11] Coren: any chance to get this back??? [15:46:40] * hedonil is on the verge of falling in *deep* despair [15:49:10] hedonil: Sorry, no. You'll have to restore from your own backups or regenerate the data. I'm pretty sure we've never made any secret that we don't do backups of user databases, though I see now that this isn't stated anywhere explicitly. :-( The good news is that Erik has okayed spending the resources do to this in the new labs (since we do have the hardware) [15:49:42] hedonil: Lemme check something. [15:49:51] Coren: so backups of user databases (on tools-db?) will come in the coming months? [15:50:22] * hedonil is crying right now :'( [15:50:49] apper: Yes; the new storage box in eqiad has enough room for us to do it, and that was the only reason why we didn't before (not enough storage) [15:52:08] Coren: perfect [15:53:30] hedonil: You're in luck, it seems. Your DB was so huge it was offloaded to a different disk; and I /might/ be able to restore it if you give me a few. [15:54:01] Coren: Oh my gosh! YES, please. [15:55:26] Caveat: I've never actually attempted to manually weave leftover tablespace onto a restored DB before; this may not even be possible. [15:55:36] But it /should/ work. [15:56:40] hedonil: Go and create your database so that the grants are okay, then log off and make sure *nothing* attempts to access it for a while. [15:57:14] Coren: Will do so [15:57:45] Coren: Can this db have a different name? name of tool? [15:58:02] Nope. It has to be _exactly_ the same as it was. [15:58:10] Coren: k. [16:02:07] Coren: u3710__stats created, all grants restored, all processes offline [16:17:40] noob question of the day: I want to clone an existing repo on GitHub into a tool directory [16:18:17] I need to upload my private key, I assume: is this advisable? [16:20:23] (upload and add) [16:23:24] Ah, or I could just not clone an ssh version. That works. [16:24:33] jarry1250: You could also use agent forwarding, I believe [16:50:42] jarry1250: or clone over https and use password auth [18:12:44] !log restarting grrrit-wm to fix its nickname [18:12:44] restarting is not a valid project. [18:12:52] !log labs restarting grrrit-wm to fix its nickname [18:12:52] labs is not a valid project. [18:12:55] FFS [18:13:00] !log tools restarting grrrit-wm to fix its nickname [18:13:02] Logged the message, Master [18:13:09] I'm just going to go get more coffee now [18:22:32] marktraceur: coffee. http://www.flickr.com/photos/110698835@N04/11355109685/ [18:27:16] hedonil: I'm much more like http://static1.wikia.nocookie.net/__cb20120510041442/legouniverse/images/1/12/Hou_dar_u.jpg after coffee [18:27:26] Which is partly why it's my desktop. [18:27:51] marktraceur: scaring... [18:32:20] marktraceur: I hope you don't work in the complaint dep with this coffee mood ;-) http://www.flickr.com/photos/110698835@N04/11227294924/ [18:35:58] but for all those who love & need coffee (including me): http://www.flickr.com/photos/110698835@N04/11229135414/ [18:37:09] Hah [18:37:35] hedonil: I work in the CR department in that mood for sure [18:37:40] Bug triaging, also good [21:56:33] hedonil: To be trying? [21:56:52] Coren: ok. [22:05:55] Coren: Right now I see tables but show table status is giving me a blank [22:06:47] * Coren ponders. [22:07:20] Yeah, I note they're kinda "there-but-not-quite" [22:11:58] Coren: also database doesn't show up in information_schema.tables [22:12:17] Yeah, I believe that is the root issue. trying to see how to fix. [23:18:15] Coren: If you don't have any better idea I would suggest to recreate the whole databse structure again, so that all tables show up properly in schema - then retry to import the data.