[13:18:27] !ping [13:18:28] pong [13:18:37] lol wut [13:24:15] !ping [13:24:15] pong [13:24:20] \o/ [13:24:54] It connects nicely to IRC, why... [13:39:21] * Damianz pokes wm-bot [13:39:35] How are you still here when tampta went to sleep o.9 [13:40:08] magic happens at Labs zzz... [13:40:24] MAGIC [13:40:43] magic not happening for my bots but there again they'd probably kill themselves in a ball of oom waiting for the api to timeout [13:41:09] actually bots should be up by now... [13:41:20] at least mine is up [13:41:26] Possibly [13:41:31] Can't get to bastion to check [13:42:01] Possibly the irc relay has just died and the bot is up but I can't get to en.wiki to check contribs either :( [15:03:20] Change on 12mediawiki a page Developer access was modified, changed by EpochFail link https://www.mediawiki.org/w/index.php?diff=569529 edit summary: [15:14:33] Change on 12mediawiki a page Talk:Developer access was modified, changed by Kozuch link https://www.mediawiki.org/w/index.php?diff=569533 edit summary: /* There was either an authentication database error or you are not allowed to update your external account. */ new section [17:51:16] Change on 12mediawiki a page Talk:Developer access was modified, changed by ^demon link https://www.mediawiki.org/w/index.php?diff=569591 edit summary: /* There was either an authentication database error or you are not allowed to update your external account. */ re: [18:02:18] petan|wk: done [18:02:28] Ryan_Lane: if I change my cn in LDAP and my username in Gerrit's db, will everything work you think? [18:02:29] no [18:02:30] oh [18:02:30] maybe [18:02:30] if you can figure out how to do so in gerrit [18:02:30] you need to rename your user in mediawiki too [18:02:33] before logging in [18:02:41] otherwise we'll have to merge the accounts, and that's annoying [18:02:52] and how do I do that? [18:03:03] I can do so, just let me know when you want to try it [18:03:20] ^demon: how can I change my userid in Gerrit? :) [18:03:37] hahaha [18:03:38] <^demon> In a meeting, I'll explain in a bit. [18:03:44] <^demon> Short answer: not easily. [18:04:10] yay sql queries [18:10:51] Ryan_Lane: Probably the wrong person to ask, but do you think nagios-wm alerts for labs kit could be thrown in here as well as ops? Most people sit in both, but prod is rather spammy sometimes and labs stuff gets missed by people. [18:18:00] Damianz: oh the production alerts for labs related infrastructure? [18:18:10] that would be difficult [18:18:25] I'd need to update ircecho to post based on regex [18:18:36] I've wanted to do that in the past, but it isn't easy [19:02:26] Yeah (sorry, went home) [19:03:16] regex could get interesting fast, could you not add a service handler for anything under labs to write out to a different file? [19:03:23] Though I imagine with puppet that could be hard [19:10:49] * Damianz wonders what chad tastes like today [19:13:23] petan|wk: Did you get your bot back up? [19:13:45] andrewbogott: I restarted it earlier (wm-bot) [19:13:57] ok, great. [19:35:34] andrewbogott: Testwiki broken :/ Think it might have been a mysql upgrade? [19:36:05] Jarry1250: Yes, probably. That's the next thing on my list to debug. [19:36:19] I can probably suggest a by-hand fix in the meantime if you need it up right away. [19:36:41] If it's not too much trouble? [19:36:58] Lemme see if I can remember how I did this... [19:38:24] Jarry1250: First you need to add the setting mysql_datadir to your project here: https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [19:39:15] Is that via "Add group"? [19:40:02] 'add variable' [19:40:49] Er, stupid question, but where is that link? I get translatesvg [Toggle, Add group] [19:42:11] Hm, I guess you have to add a group to add a variable. [19:42:15] I think the group name is arbitrary. [19:43:17] Cool. [19:43:25] Once you have that variable available, set it to '/var/lib/mysql' [19:43:37] Via the manage instance page. [19:43:58] And, actually, looking at that instance… I think that's maybe a correct long-term fix. So you can just leave that set for good. [19:44:01] (Presuming it fixes things :) ) [19:48:01] andrewbogott: Hmm, set it to '/var/lib/mysql', no change to error message; is a reboot needed, perhaps? [19:48:15] You'll need to force a puppet run... [19:48:21] And then probably stop and restart mysql. [19:48:36] Which is just 'sudo service mysql stop' and then 'sudo service mysql start' [19:48:38] Sure. [19:49:29] 'Sure' meaning 'I already did those things and it didn't help'? [19:51:01] Sure meaning, "Right, I'll try to do those things" :) [19:51:40] Aha! Success! [19:52:06] Thanks andrewbogott, would have been completely lost trying to fix that myself, not going to lie :) [19:52:39] Jarry1250: There was a bug in the code that positioned your db files. That bug is now fixed, but your instance depends on the wrong behavior. [19:52:52] It shouldn't matter as long as you don't put too much into your database. [19:53:13] Should be fine then, great :) [19:53:39] I think that 'too much' i ~10G. So, plenty of wiggle room unless you're importing stuff en mass. [19:53:57] s/mass/masse/ [19:59:32] ssh: connect to host ee-prototype.wmflabs.org port 22: Connection timed out [20:00:39] Ryan_Lane: ^ [20:01:33] kaldari: which instance is this? [20:02:30] not sure [20:06:56] 0000013d [20:06:59] gimme a sec [20:07:04] thanks! [20:07:17] I can ping it internally [20:07:47] oh [20:07:52] don't ssh directly to it [20:08:00] you need to ssh to it through the bastion [20:08:12] ee-prototype.pmtpa.wmflabs [20:08:38] I was [20:08:54] I'll try that domainname [20:09:29] that works [20:17:28] Change on 12mediawiki a page Developer access was modified, changed by Shilad link https://www.mediawiki.org/w/index.php?diff=569637 edit summary: [20:17:55] Change on 12mediawiki a page Developer access was modified, changed by Shilad link https://www.mediawiki.org/w/index.php?diff=569638 edit summary: /* User: */ [20:18:14] Change on 12mediawiki a page Developer access was modified, changed by Shilad link https://www.mediawiki.org/w/index.php?diff=569639 edit summary: /* User:Shilad */ [20:21:57] <^demon> Ryan_Lane: Mind pressing approve on https://gerrit.wikimedia.org/r/#/c/17888/ ? One line CSS fix for gerrit. [20:22:47] ^demon: merged and puppet is running [20:22:53] <^demon> Awesome, thanks. [20:54:40] Ryan_Lane: Is labs DNS a bit broken? Lately I can't access new instances by nickname, only by IP and instance id. [20:54:49] Case in point, mwrevtest aka i-000003a0 [20:54:56] that's a problem with jobs from labsconsole [20:55:11] for some reason the dns isn't updating properly [20:55:27] It's not holding me up, just making sure you know about it. [20:55:30] most likely it's because of the 500 error thrown from nova [20:55:45] when that happens I fix the dns manually [20:55:52] I should write a maintenance script for this [20:56:02] Um… this isn't a consequence of you using /my/ dns code is it? [20:56:09] nope [20:56:18] I assume I would've heard something if you'd upgraded to essex :) [20:56:22] I haven't switched to that just yet [21:29:33] Change on 12mediawiki a page Talk:Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=569653 edit summary: /* There was either an authentication database error or you are not allowed to update your external account. */ re [21:38:50] how can I install php on an instance? I did configure->webserver::php, but nothing happens [21:45:20] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=569660 edit summary: all are done