[03:11:45] Is there any way to access labsdb from outside Tools Labs? That would be helpful during development. [03:18:23] Never mind, found it: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Configuring_MySQL_Workbench :) [08:21:07] @token-remind [10:39:21] is it possible to get the start time of a finished job? [10:47:06] (03CR) 10Faidon Liambotis: [V: 032] add an SSH keypair for authdns [labs/private] - 10https://gerrit.wikimedia.org/r/93883 (owner: 10Faidon Liambotis) [11:25:41] lbenedix: already found an answer to your question? [11:29:02] lbenedix: try qacct -o -j see also: http://linux.die.net/man/1/qacct [11:38:52] !log deployment-prep upgrading packages on deployment-parsoid2 [11:38:58] Logged the message, Master [11:41:58] hedonil: my solution was looking for the creation date of the .err and .out files [11:46:35] lbenedix: the qacct -o .. -j .. gives you an output of your jobs with this jobname including start/end time and exit status (success/failed) [12:09:28] !log deployment-prep apt-get dist-upgrade on deployment-eventlogging [12:09:34] Logged the message, Master [16:22:48] (03CR) 10Hashar: "Damian, are you still around ? :-]" [labs/nagios-builder] - 10https://gerrit.wikimedia.org/r/80963 (owner: 10Hashar) [16:33:19] Damianz_: can you take a look at https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard#User:ClueBot_NG ? it's pasting the TFA into warnings [16:38:09] (03CR) 10DamianZaremba: [C: 032 V: 032] "Yep - missed this. Looks good." [labs/nagios-builder] - 10https://gerrit.wikimedia.org/r/80963 (owner: 10Hashar) [16:53:22] (03CR) 10Hashar: "Thank you :-]" [labs/nagios-builder] - 10https://gerrit.wikimedia.org/r/80963 (owner: 10Hashar) [17:34:00] Damianz_: ping [17:48:12] Coren: ping [17:48:15] er [17:48:20] any tool labs root [17:48:27] YuviPanda|away, petan [17:48:36] ugh: What's up? [17:49:03] pm [17:58:42] Coren: Thanks - changed now. Derpy doing it fast and hacky due to being borked moment. [17:58:51] Also number of notifications for a bot when logging into the gui is lolz [17:58:55] Damianz_: ohhai [17:58:58] i just send you an email [17:59:11] Damianz_: also, do oyu know what the TFA issue is? [18:00:31] ugh: TFA? on talk page? [18:00:56] Damianz_: https://en.wikipedia.org/w/index.php?title=User_talk:37.239.2.230&diff=prev&oldid=580477525 [18:01:16] I feel not too bad - andrewbogott left the production nova ldap password in labs once :D #trolldistraction [18:01:26] heh [18:01:31] so can i unblock it now? [18:01:41] Yeah should be good [18:02:11] I'll add sanity checking the right permissions and refusing to start if wrong to the cleanup todo list. I need to push up all the code hacks I put in for tools and work on cleaning them up tbh. [18:02:21] unblocked [18:03:09] Coren: you around? [18:03:18] Thanks [18:03:35] Betacommand: ¿Que pasa? [18:03:49] Coren: any progress on https://bugzilla.wikimedia.org/show_bug.cgi?id=54074 [18:04:00] ugh: The '(No difference)' note? Don't really see what's wrong apart from that [18:04:35] Damianz_: er, did you look at the warning? it added the TFA in it [18:04:45] "The Blackwater fire was caused by a lightning...." [18:05:11] oh [18:05:12] hmm [18:05:13] Damianz_: it's also doing stuff like https://en.wikipedia.org/w/index.php?title=User_talk:184.183.156.190&diff=prev&oldid=580479280 [18:05:20] so i wonder if its not able to get the page text properly [18:06:00] brb [18:06:07] I think I've seen this before - when people do stuff with images and it's to do with the title getting stuff. Should be fixable pretty easily I think. [18:06:08] Betacommand: Frankly, that was simply put aside in my "tons of small mostly cosmetic fixes to handle when free time allows"; albeit the existence of free time implies vast quantities of optimism. :-) I don't think I can fix his in LDAP without risking other breakage, but I can work around it in the global .profile. [18:06:17] Got to go to dinner in like 20min though - so might go on the todo list [18:06:57] Betacommand: Lemme see if I can do a quick fix. [18:09:19] Damianz_: ok, but it needs to get fixed soon [18:25:42] Coren: thanks works [18:26:05] Now 70% of my line isnt the pwd [18:27:35] Damianz_: any update on cluebot reboot? [18:28:02] Betacommand: Should be up and running - still has talkpage bug as ugh said, will look at. Got to go to dinner so bbl [18:28:15] (It's also reverting stuff atm, so I'd say working) [18:31:31] 193gb of logs - Coren can the grid rotate its logs/if I remove then will it get a new fh without me restarting the process (so I can write my own rotation) or you're gonna get tb of logs fast :( [18:33:32] Damianz_: Annoyingly enough, gridengine has no support for log switching. Personally, I recommend that you don't rely on its logging of stderr and stdout and do your own logging (which you can then rotate as you will). The default .out and .err are only useful for single-run jobs and fatals. [18:34:42] That sucks, though really the logging currently sucks [18:34:50] * Damianz_ adds to todo list... really going to dinner now [18:36:17] (although you could do so trickery in your script by doing a fsync(), copy, ftruncate() of stderr/stdout if you really wanted to) [18:39:28] That never ends that well [19:58:34] andrewbogott: ping? [19:58:42] YuviPanda: what's up? [19:58:44] andrewbogott: any idea about the certs, and why they don't seem to take? [19:59:20] /etc/ssl/private/star.wmflabs.org.key is empty. Not sure why yet... [19:59:37] andrewbogott: hmm, perhaps it needs to be specifically installed in some private repo? [19:59:40] or... something? [19:59:53] probably. But… you made it work once before, right? [19:59:57] andrewbogott: Ryan_Lane did something last time, IIRC [20:00:03] oh, ok. [20:00:12] andrewbogott: something other than just applying the puppet classes, IIRC [20:02:31] The role class has 'privatekey => false' [20:02:46] I see that in the code, if that is set to 'true' then it will install the key from labs private, which is non-empty. [20:03:06] YuviPanda: any idea if that's what we need? [20:09:52] a volunteer who tried to sign up on wikitech told me he gets. "There was either an authentication database error or you are not allowed to update your external account.". I know we've had these before, but where do i send him again? [20:10:44] andrewbogott: YuviPanda the .key comes from private repo, the cert from public repo [20:10:56] mutante: Most often that is because the user is trying to create a shell name with invalid characters. [20:11:00] For instance, uppercase letters [20:11:13] they are both installed by install_certificate [20:11:31] you also get the key from it [20:11:44] as long as the file names are correct [20:11:52] mutante: OK. It's clear that if you specify privatekey => false that it installs an empty key. I'm just wondering if there was some intentional/security reason why yuvi wrote it that way. [20:12:20] andrewbogott: let me ask him which name exactly he is trying to use, good point [20:13:06] gotcha, didn't see the "privatekey => false" part right away [20:13:15] didnt have a reason to use that [20:13:21] andrewbogott: I think Ryan_Lane did that bit [20:13:28] don't fully remember [20:17:53] YuviPanda: Hm, the key on proxy-dammit is different from the key in the labs private repo. [20:18:01] So… I guess I will copy that same key over. [20:18:02] my guess is the key has been copied manually [20:18:04] on that old instance [20:18:17] or it's in private/labs [20:18:27] ah [20:22:42] YuviPanda: OK, nginx seems happy now. [20:22:51] I guess I should add a checkbox to the gui to specify http vs. https? [20:23:03] andrewbogott: to the backend? [20:23:18] YuviPanda: I mean, like, on this page https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProxy&action=create&project=mwreview [20:23:23] andrewbogott: I'm okay with just making everything http in the cluster [20:23:23] replace the 'port' field with a checkbox [20:23:46] Oh... [20:23:49] moment, marktraceur is dragging me away [20:23:51] You're right, I'm confused. [20:23:52] i'll brb [20:23:54] ok [20:33:47] andrewbogott: back [20:33:50] andrewbogott: I don't think we can/should have a https checkbox [20:35:13] YuviPanda: You don't think we should proxy https requests at all? [20:35:24] I may not understand what Ryan was asking about... [20:35:41] andrewbogott: no, Ryan was talking about the proxy not listening on 443, which we've now fixed [20:36:21] So what will happen if someone accesses a proxied host via https? [20:39:38] hm, apparently the answer to that is 'it works just fine' [20:41:56] andrewbogott: indeed! [20:42:00] andrewbogott: it acts as an SSL terminator [20:42:14] ok then :) [20:43:51] andrewbogott: :D [23:35:43] Why I am not create Hosted tools on http://tools.wmflabs.org ?