[02:22:49] ori-l: ping [03:01:19] New patchset: Ryan Lane; "Put in new variable for initial_password for opendj" [labs/private] (master) - https://gerrit.wikimedia.org/r/19701 [03:22:45] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/19701 [04:12:15] jeremyb: hey [16:41:22] Change on 12mediawiki a page Developer access was modified, changed by Hariharshankar link https://www.mediawiki.org/w/index.php?diff=572426 edit summary: [16:59:46] paravoid: I'm returning to the issue from last week about configuring mysql via puppet. [16:59:56] hi [17:00:01] hi! [17:00:17] Have you thought about that any more and do you have any suggestions? [17:00:34] I haven't [17:00:39] My inclination is just to rearrange the dependencies so that the .conf file is set up before mysql is installed. Does that trouble you? [17:00:40] I can think of it now :) [17:00:49] so, what's the goal again? [17:01:10] The immediate, tiny goal is to change the default directory that mysql uses for databases. [17:01:22] to? [17:01:31] The problem is that by the time puppet gets around to creating that conf file, mysql has already configured a different directory. [17:01:43] To /mnt or /srv, someplace outside of the VM's local storage. [17:02:16] This is because installing mysql causes it to auomatically start and do its first-run procedures. [17:02:29] Ideally we would a) install mysql b) change the config c) start mysql [17:02:35] but a and c happen all at once. [17:02:57] so wait [17:03:03] * andrewbogott waits [17:03:06] why can't you put my.cnf there before installing mysql? [17:03:57] * andrewbogott nods vigorously [17:04:01] That is what I am proposing to do. [17:04:06] ah, okay [17:04:09] yeah, sure, that'd work [17:04:14] I'm only hesitating because Ryan briefly opined that that was 'incorrect' [17:04:18] Or, words to that effect. [17:04:33] also, you could have file { "my.cnf": requires => Package, notify => Service } [17:04:48] And, actually, he offered up that opinion before I'd explained the problem. So maybe everyone agrees that that's the right approach. [17:04:56] that would install the package (mysql would be started because of postinst), put my.cnf there and restart mysql [17:05:16] Notifying doesn't seem to work… behavior on first run is different from subsequent startup behavior. [17:05:23] how come? [17:05:41] Well, I've tried doing this by hand... [17:05:55] I have mysql running with the dbs in /var/lib/mysql [17:06:05] I change the conf, then 'service stop' and 'service stop' [17:06:32] At that point, mysql can't come up. Because it's looking in the new dir (/mnt/mysql/) for files that aren't there. [17:06:34] And it just gives up. [17:07:06] It would be possible to have the "my.cnf" stage parse the existing my.cnf, figure out what the current directory is, and move its contents... [17:07:16] that works! But seems complex to automate. [17:07:28] no it's not, we have augeas :) [17:07:36] well, move its contents, no [17:07:49] that'd suck for a number of reasons too [17:08:07] Unfortunately, none of these solutions really allow for the directory to be moved on an existing instance. [17:08:12] Which, ideally, it would. [17:08:23] you could dpkg-reconfigure mysql-server... [17:08:25] Well, wait, my last solution (parse, move, chance config) would allow for that. [17:08:43] Oh, OK, I don't think I considered that option. Lemme try that. [17:08:51] You think that will force the 'first run' behavior? [17:09:03] yeah, that's the idea [17:09:12] I'm not sure if this will work though [17:09:18] just thought of that :) [17:09:26] I'll try it. [17:09:46] additionally [17:09:54] Hm… not sure how to do that in puppet in the correct order. But I can at least see if it's possible by hand. [17:09:59] I've mentioned in the past how I hate that we have to move everything away to special directories [17:10:19] Yeah, well, we could relink /var/lib/mysql to another volume. [17:10:44] But isn't it somewhat normal to explicitly move db files elsewhere? [17:11:05] that is, yes [17:11:31] of all the solutions, placing my.cnf before the package seems the most elegant to me [17:11:48] that doesn't mean I find the others unacceptable though [17:11:50] <^demon> maven-debian-helper is a piece of crap. [17:11:52] <^demon> that is all. [17:11:54] I'll let you choose :-) [17:11:59] ^demon: reportbug :) [17:12:10] <^demon> It's a feature :P [17:12:33] well, wait. Shouldn't a 'correct' puppet manifest not depend on the current state of the system to work properly? [17:12:46] The 'place my.cnf first' solution definitely does depend on the state. [17:12:54] Whereas, if reconfigure works, then it doesn't. [17:12:55] I think. [17:13:20] what do you mean depends on the state? [17:13:23] what's the state? [17:13:25] <^demon> paravoid: It may also be that I have almost zero clue what I'm doing with maven. but anyway :) [17:13:33] ^demon: what's the problem? [17:13:45] I have almost no clue about maven but I have a clue about Debian, so... [17:14:14] <^demon> Well I'd like to have maven spit out the .deb when I'm done compiling, to save me an extra step of building the package. [17:14:21] paravoid: Hm, lemme see if I can articulate this better. [17:14:35] <^demon> mvn-debian only runs in offline mode though, so dependencies aren't satisfied. [17:15:13] ^demon: I have absolutely no idea what you're talking about :) [17:15:19] <^demon> Yeah it's maven stuff :p [17:15:40] paravoid: An example of a 'state' would be if the system happened to already have mysql installed and running, pre-puppet. At that point, our manifests would simply fail. [17:16:03] Granted puppet can't wrestle /any/ system into proper shape. [17:16:05] <^demon> "So if your project has already all its dependencies packaged for Debian, and with the correct Maven metadata, then it's a piece of cake!" <- Hah, that's probably it [17:16:32] But, the idea that the first puppet run does magic things which subsequent puppet runs can't change or undo or even redo… that troubles me. [17:17:00] Does that make sense? I think of puppet as /maintaining/ the state of a server, rather than simply creating that state. [17:17:14] that is correct [17:17:16] The place-my.cnf-first solution works for creating but not for maintaining. [17:17:35] but what would you like puppet to do if the system has mysql already installed and running? [17:17:40] move everything to /srv? [17:17:46] <^demon> OrenBo: Ping [17:18:10] that's a big assumption [17:18:14] I think that if I change the datadir setting for a working server, it should start using that datadir. And leave the former datadir (and data) alone. [17:18:40] reconfigure might be able to do that. [17:19:07] Whereas if we use the 'my.cnf first' solution, then changing $datadir will simply leave mysql unable to come up. [17:19:09] It'll just break. [17:19:31] oh now I see [17:19:43] yes, you're right about that [17:20:15] Basically we need to find every place where there's a notify=>service and make it reconfigure instead. [17:20:26] presuming reconfigure does what we want. [17:20:31] And doesn't clobber existing data :/ [17:20:43] I'll experiment. [17:22:07] well, you could also ship a little helper that bootstraps a new directory :) [17:22:15] yeah. [17:22:30] btw, are you staying in SF after the all-hands to go to the puppet conference? [17:23:38] I'm not sure if you need more ideas [17:23:42] but I have one for you... [17:24:24] ok [17:26:47] Change on 12mediawiki a page Developer access was modified, changed by Theaitetos link https://www.mediawiki.org/w/index.php?diff=572446 edit summary: [17:27:12] exec { 'foo': command => '/bin/ln -s /srv/mysql /var/lib/mysql', creates => '/var/lib/mysql', before => Package['mysql-server'] } [17:30:19] I'm trying to decide if that also has the problem of only working the first time... [17:30:23] I think it does. [17:33:16] kind of [17:33:28] if you happened to have mysql installed before, it won't touch your config [17:33:49] that assumes that my.cnf keeps /var/lib/mysql as the path btw [17:35:39] paravoid: But, if you change datadir subsequent to the first puppet run… everything breaks, right? [17:41:52] is there any issues with local storage right now? [17:42:23] Shouldn't be [17:42:52] Damianz: well I get mobile-testing:/data$ ls -al [17:42:52] ls: cannot access project: No such file or directory [17:42:54] total 4 [17:42:55] drwxr-xr-x 3 root root 0 Aug 14 19:44 . [17:42:56] drwxr-xr-x 26 root root 4096 Aug 2 01:49 .. [17:42:57] d????????? ? ? ? ? ? project [17:43:13] cd project [17:43:15] ? [17:43:17] It's automount [17:43:46] Damianz: it can't cd to it [17:43:56] Dunnot then [17:44:01] cd project [17:44:02] -bash: cd: project: No such file or directory [17:44:07] * Damianz looks at paravoid [17:52:45] ^demon: pong [17:54:11] having issues with maven ? [17:56:14] * OrenBo needs to reboot [17:56:33] <^demon> Yeahhh, I was trying to use mvn-debian (just a wrapper for maven with a couple of extra options set). [17:56:52] <^demon> But it forces offline and won't build, even though the plugins I need are in my ~/.m2/repository [18:00:39] <^demon> There's got to be an easier way to do this. [19:25:57] csteipp: Ryan_Lane: can i have permission to stab someone? please?! [19:26:44] * Damianz hands jeremyb a crowbar [19:27:01] http://lists.wikimedia.org/pipermail/wikitech-l/2012-August/062339.html [19:27:05] why thank you Damianz [19:28:29] jeremyb: why stab? [19:28:56] I agree that we should become a provider [19:29:06] I don't necessarily think we should use the openid extension for it, though [19:29:47] * Ryan_Lane coughs *phpsimplesaml* [19:29:49] Ryan_Lane: chris and others have covered this extensively on lists and on wiki. i don't think we need *another* to be a provider or not debate [19:30:37] Ryan_Lane: I mailed Andreas (simplesamlphp author, whom I've personally know) btw to let him know of the wikipedia library project [19:30:45] sweet [19:30:48] paravoid: thanks [19:30:49] and to ask him about ideas for funding it, maybe they have some money to spend :) [19:30:58] haha [19:31:01] that would be awesome [19:31:03] or an intern [19:31:09] or something [19:31:16] we'll see, it's a bit of a long shot [19:32:22] ori-l: i just wanted to know what you thought about updating. evan made a change for me and it's in master i think [19:32:28] yeah. worth a try, though [19:32:32] (priestly not rosen) [19:32:46] I think I should write a community call for help on OpenStackManager after the upgrade [19:32:59] thinking of community help :) [19:33:17] meh, that'd be nice before you rewrote half of the thing [19:33:20] there's a ton of somewhat easy changes that can be made, and it would be nice to get the community to help :) [19:33:27] that's why I said after the upgrade [19:33:40] the gsoc student was supposed to add rest support [19:33:45] that didn't happen at all [19:33:45] yeah, I'm saying that it would have been nice to get that work for free :P [19:33:53] he flaked out [19:34:01] there was a bit of drama over that [19:34:39] oh? [19:34:54] well, gsoc students get paid [19:35:00] part of it upfront [19:35:26] it's bad for us if too many students get a failing grade [19:35:56] aha [19:36:06] well the upfront payment is very low though (iirc) [19:36:21] Is there an upfront payment now? When I mentored two years ago, it was different [19:36:47] There was a mid-term eval and if you passed that, you got half the money, then a final eval at the end and if you passed that, you got the other half [19:37:16] Accepted students in good standing with their mentoring organization will receive a 500 USD stipend shortly after coding begins on May 21, 2012. [19:37:24] Ha? [19:37:26] That's new [19:37:27] paravoid: ETOOMANYANDREAS. barth, tille, mayer, kolbe [19:37:35] and then $2250 for the mid-term and $2250 for the final term [19:37:36] 2011 too [19:37:50] I could swear 2010 was different [19:37:54] 2010 too :) [19:38:05] $2500 mid-term, $2500 at the end, and $500 for the mentoring org I think [19:38:16] 2009 too [19:38:19] Or was that 2009, one of those [19:40:45] Hah, you're right [19:40:51] There were initial payments in 2009 t oo [19:41:16] technically, Ryan's right [19:41:36] I just backed it with citations [19:41:43] Yeah [19:41:45] heh [19:41:47] I stand corrected [19:42:00] I was a mentor in 2010 but I had completely forgotten about the initial payment [19:51:34] Change on 12mediawiki a page Developer access was modified, changed by Dreilly1982 link https://www.mediawiki.org/w/index.php?diff=572500 edit summary: [19:56:00] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=572502 edit summary: archived some reqs [20:01:22] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=572503 edit summary: all done [21:12:12] paravoid, Ryan_Lane, is there a way to make an exec sleep for a few seconds before running? I'm plagued by 'mysql respawning too fast, stopped' [21:12:51] /bin/sleep 2; [21:13:33] :( [21:15:07] I see several places where people are proposing to add a "sleep =>' feature but I guess it hasn't made it in. [21:19:53] heh [21:19:56] yeah. it's sad [21:47:31] Ryan_Lane and/or paravoid: https://gerrit.wikimedia.org/r/#/c/19940/1/manifests/mysql.pp [21:47:51] I'm pretty confident about that, except for the apparent app-armor race that I had to work around. [21:47:57] Is that a known/familiar problem? [21:48:12] yep [21:48:14] I hate apparmor [21:48:29] with selinux you could just change the file/directory contexts [21:48:43] fucking apparmor and its monolithic config [21:49:34] OK, so it's normal to have to wait a bit after reconfiguring apparmor before you really get permission? [21:50:16] I wonder if we could just put an automatic sleep in the base class... [21:57:33] Ryan_Lane: Any idea why it would work the second time but not the first? I just tried doubling the sleep and it still didn't work on the first pass. [22:10:06] Damn, did we run out of cores again, already? [22:13:00] Not out of cores, I guess. Just very slow startups for some reason. [23:43:38] Ryan_Lane: Is puppet now broken on purpose? Or, at least, known to be broken? [23:43:50] where is it broken? [23:44:15] A couple of my instances give me "Could not retrieve catalog from remote server: Connection refused" [23:44:22] damn [23:44:30] seems puppet ran when it was broken on some instance [23:44:47] in /etc/puppet/puppet.conf, change the server from "puppet" to "virt0.wikimedia.org" [23:44:55] minus the quotes [23:45:27] This instance (well, both actually) are self-hosted. [23:45:32] Which makes me think that… hm. [23:45:39] ldap? [23:45:47] should be fine [23:46:03] Unknown MySQL server host 'undef' [23:46:04] crap [23:47:25] d'you think if I start a fresh instance that it'll work? I don't have anything of value on this one, just made it 20 minutes ago. [23:48:03] no [23:48:05] it won' [23:48:07] won't [23:48:16] my change broke the puppet config on virt0 [23:48:29] In that case, perhaps I will break for dinner :) [23:55:27] andrewbogott: ok. fixed. [23:56:07] Hm… I'm getting the same behavior. But this is still on a self-hosted system. [23:56:34] which instance? [23:57:00] is your puppetmaster running? [23:57:06] /etc/init.d/puppetmaster restart [23:58:28] 'mwrevtest5' [23:59:14] brand new instance? [23:59:33] nope, not this one. [23:59:43] Well, brand new as of lunchtime. [23:59:48] * Ryan_Lane nods