[00:00:05] akoopal: s1 through s7, that is correct (though those are, strictly speaking, clusters /of/ databases, each generally holding multiple project databases) [00:00:57] Coren: ok, but if I create my (well, erwins') userdatabase on all, that should be enough for now [00:01:26] akoopal: What is the database used for? [00:01:26] untill of course you need to split up more [00:01:46] Coren: a cache of categories [00:02:08] Ah, and those require joins with the project databases then? [00:02:12] yes [00:02:16] * Coren nods. [00:02:23] Then seven is what you need. [00:02:28] this is for the related changes tool [00:02:50] and by the looks of it others as well [00:05:45] but I made a good step for now, now bedtime [00:06:05] Cyberpower678: magnus gave me access, so related changes seems to work :-) [00:07:08] akoopal, ok. :-) [00:19:55] hey labs folks, terrrydactyl is working on Wikimetrics and I'd like to get her access to tools [00:21:27] oops, disregard above, just remembered https://tools.wmflabs.org/ [04:27:05] Coren: it seems my tools tool can't receive mails :p I assume it's because mails to tools.tools@tools.wmflabs.org are routed to tools.maintainers@tools.wmflabs.org, but tools.maintainers@tools.wmflabs.org is considered as an address of maintainers tool. [05:48:36] I installed postgresql on my instance, and want to sudo -u postgres -i [05:48:46] Sorry, user nicolas-raoul is not allowed to execute '/bin/bash' as postgres on map.eqiad.wmflabs. [05:58:13] How can I work around this problem? [05:58:35] hi Nicolas [05:58:40] Nicolas: can you sudo as anything at all? [05:59:05] Hello YuviPanda :-) Nice to meet you, thanks for your work always! [05:59:15] Nicolas: :D [06:00:05] sudo worked perfectly when I ran sudo apt-get install postgresql-9.1-postgis [06:01:03] Nicolas: hmm, can you add me to the project on wikitech? I think thig might be a sudo policy issue [06:02:36] sudo -u root whoami <- works directly, no password prompt [06:03:02] sudo -u postgres whoami <- asks for password, fails with my wikitech password [06:04:22] At https://wikitech.wikimedia.org/w/index.php?title=Special:NovaRole&action=addmember&projectname=oxygenguide&rolename=projectadmin I can only add Andrew Bogott, I don't know how to add you... [06:04:41] Nicolas: hmm, moment [06:05:25] sorry! [06:05:29] I was mistaken [06:05:55] "Successfully added YuviPanda to oxygenguide. " [06:05:58] woot [06:06:48] Nicolas: can you also add me as an admin? [06:07:09] Done [06:08:16] Nicolas: can you close your current ssh session, log back in, and try again? [06:08:36] I just tweaked sudo policy [06:09:29] Sorry, user nicolas-raoul is not allowed to execute '/usr/bin/whoami' as postgres on map.eqiad.wmflabs. [06:10:00] in a brand new ssh session (even logged out of bastion) [06:11:50] Nicolas: ok, try this [06:11:55] sudo su [06:12:00] su postgres [06:12:03] bash [06:12:09] and then attempt to execute things? [06:13:34] "sudo su" asks me for a password and fails. Sorry for being a newbie but is it the same password I use to log into wikitech.wikimedia.org? [06:13:35] Nicolas: ^ [06:14:30] Nicolas: no, it shouldn't be asking you for a password at all [06:14:40] Nicolas: mind if I ssh in and take a look? [06:14:57] Sure! of course please :-) [06:15:03] ok :) sshing now [06:16:41] Nicolas: try the same sequence now? [06:18:28] Great that worked! :-) [06:18:48] Nicolas: woot :) [06:19:34] What was the trick? I will probably delete this instance and re-create another one in the future, can I perform the trick myself to avoid disturbing you every time? [06:20:00] Nicolas: so I couldn't get sudo -u to work at all [06:20:17] Nicolas: but this trick - sudo su && su && bash should work everywhere [06:20:19] Nicolas: no tricks needed [06:20:32] Nicolas: I was just trying to mess with sudo policy to see if I could get sudo -u working but no look [06:21:03] "sudo su" failed the first time I tried, though [06:21:48] Nicolas: yeah, that is because I had been messing with sudo policy. If you had tried 'sudo apt-get install ' that would have failed too :) [06:21:59] I see! [06:22:30] Nicolas: so if you create a new instance, this trick should still work [06:23:13] I will try to find a place on the wiki where to add the "sudo -u" workaround. Thanks a lot, see you! :-) [06:23:29] Nicolas: yw! :) [06:34:25] I added the tip at the bottom of this page: https://wikitech.wikimedia.org/wiki/Help:Sudo_Policies [06:35:05] Nicolas: woot! Thank you :) [06:37:39] Nicolas: you can also remove me from the project now if you want [06:59:51] I installed postgresql (via apt-get install) but after I start it with "sudo /etc/init.d/postgresql start", no process appears, zero result for "ps -e | grep postgres", and zero file in /var/log/postgresql/ ... any idea what's going wrong? On an out-of-the-box instance [07:53:46] !log deployment-prep installed mariadb via puppet on deployment-db1. no data yet [07:53:48] Logged the message, Master [08:20:26] hi. how do I create my own repo for the gsoc project on gerrit. [09:02:33] hi all! [09:02:50] where can I see the apache (and php) error logs for tool labs? [09:02:59] i'm tryign to debug http://tools.wmflabs.org/potd-feed/commons/potd-800x600.rss [09:08:33] this is a 503... I don't get it, I must be missing something obvious :P [09:08:37] this is a static rss file... [09:09:22] oh, and the crontab is gone! yay. wtf? [09:14:23] guh [09:14:34] can't figure out how to log into the old toollabs instance either [10:08:01] DanielK_WMDE: you can't log there because these are deleted [10:08:07] there are no old instances [10:09:27] petan: ok. so... where do i find my old contab entries then? [10:09:50] you need to ask Coren :/ he did the migration [10:10:16] petan: fwiw shouldn't Labs be getting IPv6 now it's moving/moved to eqiad? [10:10:25] no idea [10:26:06] DanielK_WMDE: ...DATA.crontab sollten sein [10:29:35] Steinsplitter: wo? [10:30:16] DanielK_WMDE: im home ordner deines tools [10:31:08] Steinsplitter: da liegen genau drei dinge: Mail, potd, und replica.my.cnf [10:31:09] /data/project/deinproj/...DATA.crontab lagen sie bei mir... [10:31:11] DATA gibt's nicht [10:31:15] o_O [10:33:47] Steinsplitter: andere frage: wo kann ich denn das apache error log sehen? [10:36:37] DanielK_WMDE: keine ahnung :/. wenn dein tool zwangsmigriert wurde musst du die migration finalisieren und den webservice neu starten. [10:37:54] i must have missed the memo [10:38:00] is this stuff documented somewhere? [10:38:20] how do i restart the webservice (don't all tools share the web service?) [10:38:59] i mean, this is just a cron job and static files. i didn't expect any trouble migrating this. [10:39:06] but apparently, tehre are some hoops to jump through [10:39:36] --> https://wikitech.wikimedia.org/wiki/Tool_Labs/Migration_to_eqiad [10:39:52] thanks [10:40:21] finish-migration [10:40:21] restart webservice [10:40:21] kann gut sein dass die crons erst bei der finalisierung rüberkopiert werden. np [10:41:47] yea, crontab is restored by finish-migration [10:42:51] Steinsplitter: ok, funzt. danke für deine Hilfe! [12:47:08] Jasper_Deng_away: We had issues migrating to Neutron (which was a requirement) and were against a thight deadline. Rather than risk breaking stuff, we delayed neutron migration. IPv6 is still on the roadmap, but we'll let the migration dust settle first. [12:57:00] hashar: deployment-db1 is loading a dump from deployment-sql now. the dump itself will be internally consistent, but out of date by a couple hours. does that matter to beta? [12:57:08] springle: hello! :-] [12:57:27] we can afford some data loss [12:57:30] if it does i'll figure something else out. but this ould be easiest :) [12:57:37] ok [12:57:59] springle: there is also a wikidata database on the sql02 server [12:58:09] sql02 is the master for the wikidata and the wikivoyages dbs [12:58:24] alternative is to make deployment-db1 a slave, let it catch up, then switch master and traffic [12:59:11] save yourself the hassle of switching master. I am fine dropping out some data [12:59:41] no reason sql02 data cannot go onto db1? [12:59:53] no reason, you can dump the few db there and import them on db1 [12:59:55] it's a bigger vm [12:59:56] ok [13:00:03] what about es1 [13:00:09] then we will have only one master [13:00:33] es1 ? [13:00:37] deployment-es1 [13:00:45] is that an external storage thing? [13:00:45] ohhh [13:00:52] that is ElasticSearch [13:01:05] manybubbles will take care of migrating the data / regenerating the search index [13:01:09] ah ok. nice and ambiguously named like production external storage :D [13:01:17] yeah ES is a confusing acronym :D [13:01:37] I don't think we use external storage on beta [13:01:47] ok easy [13:02:36] the wikivoyages DB we can probably drop them [13:04:46] springle: Re dbstore100[12] and "10 delayed slaves replicating all shards" = slaves replicating one shard each or, well, all shards? I. e., would this setup be usable for cross-wiki joins? [13:04:51] *efficient [13:05:51] scfc_de: yes, recombining all shards in one place. one box will be available for cross-wiki queries [13:05:56] won't be that fast [13:07:12] hashar: yeah, I'll just rebuild the index when the database is ready [13:07:19] or, if it is ready now, I'll go do it now [13:07:31] springle: I am dropping the de,en wikivoyage databases from sql02. They are full imports of the old wikivouages, that is way too big and we don't have any use for them. [13:07:36] I think we named it deployment-elasticXXXX in the new beta [13:07:40] hashar: ok [13:07:46] manybubbles: +1 on name change :] [13:07:53] manybubbles: nice :) [13:07:55] bd808|BUFFER: did it [13:08:08] he is a good man [13:09:41] springle: Available from Labs/Tools?! [13:10:17] scfc_de: er havn't discussed that :) but theoretically it could be [13:10:40] the sanitarium would need an extra box [13:14:04] !log deployment-prep Dropping enwikivoyage and dewikivoyage databases from sql02. Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ [13:14:07] Logged the message, Master [13:14:43] hashar: so happy to dump sql02 now? [13:15:20] springle: yeah dropping enwikivoyage and dewikivoyage there. [13:15:42] so there is information_schema / mysql and wikidatawiki [13:15:49] I guess we only care about the wikidatawiki one [13:17:27] manybubbles: on beta I have dropped wikivoyage from $wgCirrusSearchInterwikiSources ( https://gerrit.wikimedia.org/r/#/c/121047/1/wmf-config/CirrusSearch-labs.php ) [13:17:37] manybubbles: the databases no more exists. [13:18:07] springle: it is great you managed to write a mariadb puppet class from scratch :] [13:20:42] hashar: I have no problem with this [13:20:54] so do the other databases exist? [13:21:41] manybubbles: haven't checked, was merely cleaning out wikivoyage :-] [13:21:46] cool [13:21:51] manybubbles: sean is populating them on the db1 server \O/ [13:22:00] hashar: it's a little simplistic yet. am a puppet noob [13:22:14] but it serves for this [13:22:27] springle: it is better than nothing and it is a good occasion to practice your puppet [13:22:44] :) [13:22:58] I started puppet by copy pasting manifests [13:23:13] then trying to please folks reviewing my changes [13:23:26] sounds about right [13:24:02] my first change ever to puppet is a copy paste : https://gerrit.wikimedia.org/r/#/c/401/1/manifests/site.pp [13:31:15] and configuring db1 :-] [13:31:15] https://gerrit.wikimedia.org/r/#/c/121051/ [13:32:52] wikidata is surprisingly small: 300MB .. while innodb ibdata is 70G. were those other database very large? [13:32:58] on sql02 [13:33:21] we imported on beta all the wikivoyage projects [13:33:28] before they migrated under wikimedia umbrella [13:33:41] so if innodb ibdata is never garbage collected, that would explain the big size [13:33:58] it will probably end up smaller if you export/import [13:34:05] yep, innodb global tablespace never shrinks [13:34:07] * hashar has no clue how innodb works [13:34:26] new vms will use innodb_file_per_table for that reason, amongst others [13:34:46] can you reclaim the space somehow? [13:35:07] yes [13:38:00] springle: have you created a database user for mediawiki? [13:38:11] doing the grants now [13:38:12] springle: should be filled on eqiad in the file /data/project/apache/common-local/wmf-config/PrivateSettings.php [13:38:17] great [13:38:23] it will be the same as the old one [13:38:28] we have user 'mw', you can use whatever else, maybe wikiuser [13:38:42] which do you prefer? [13:38:45] the PrivateSettings.php file is local to the filesystem so update at will [13:38:49] whatever works for you [13:38:55] keeping the same user/pass is fine to me [13:39:09] old one mw. fewer changes to break today :) [13:40:32] there is also an 'oren' user on -sql [13:40:54] can drop it [13:41:05] probably some left over from when we started beta [13:41:09] oren is a volunteer [13:52:29] !log deployment-prep dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull [13:52:32] Logged the message, Master [13:53:12] springle: I managed to access a wiki page!!!!!!!!!!!!!!!!!!!!! [13:53:56] :) [13:54:05] so happy [13:54:20] still loading [13:54:27] up to simplewiki [13:54:38] should be in alpha order [13:54:41] I even managed to login [13:54:58] simplewiki is quite big I think [13:55:49] !log deployment-prep stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) [13:55:51] Logged the message, Master [14:00:30] manybubbles: mind reviewing a tiny change for beta which change CommonSettings.php please? I am paranoid : https://gerrit.wikimedia.org/r/121056 [14:00:39] manybubbles: merely vary the udp2log destination in labs [14:02:53] done [14:02:55] hashar: ^ [14:02:57] thx [14:03:42] in my Special:NovaResources why does my lab instance Puppet status show 'failed' ? [14:04:07] tonythomas: puppet fails there? Connect on your instance and run: sudo puppetd -tv [14:04:17] tonythomas: that will apply puppet and show you a bunch of errors [14:04:28] hashar: actually, I cant connect to .. [14:04:30] tonythomas: or one can look at /var/log/puppet.log [14:05:08] hashar: I can log in to the instance remotely via ssh right ? [14:05:10] !log deployment-prep udp2log functional on eqiad beta cluster \O/ [14:05:13] Logged the message, Master [14:05:29] tonythomas: na you need to ssh to one of the bastion hosts first [14:05:49] oh! I will do that [14:05:51] tonythomas: example configuration https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [14:06:04] tonythomas: that would configure your ssh client so you can do: ssh instance.eqiad.wmflabs [14:06:27] hashar: thanks. let me try that again [14:06:27] ssh client will then figure out the bastion to use, connect to that bastion and connect from the bastion to your instance [14:06:33] ok [14:16:59] !log deployment-prep fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649 21.5M Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} [14:17:01] Logged the message, Master [14:22:15] manybubbles: and here is the elastic search basic conf for beta in eqiad [14:22:17] manybubbles: https://gerrit.wikimedia.org/r/121063 [14:30:39] hashar: for some reasons ssh -A @bastion.wmflabs.org [14:30:42] fails for me [14:30:53] that username will be my gerrit shell username ? [14:33:26] tonythomas: your shell username [14:34:00] tonythomas: most probably tonythomas01 [14:34:20] hashar: of course, that's my gerrit one, and wikitech as I remember, [14:34:24] but still it fails [14:34:30] I have added the ssh keys [14:34:31] yet [14:34:59] it shows the Public Key issue [14:36:56] tonythomas: double check your key in wikitech interface? [14:37:00] it is apparently there [14:37:18] hashar: ssh tonythomas01@gerrit.wikimedia.org -p 29418 works fine [14:37:28] I will try adding the key once again [14:37:46] I think Gerrit uses a different system [14:37:55] aka the key is added to your user preference in Gerrit web interface [14:38:11] yeah! The same key is added here too I think [14:38:49] I gave xclip -sel clip < ~/.ssh/id_rsa.pub , pasted it there, and on submit shows the error - probably due to existing same key [14:38:59] hashar: sorry ! for some reasons I have to quitt [14:39:02] will brb [14:44:05] hashar: Logstash seems to be working in eqiad beta -- https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/default [14:45:23] !log deployment-prep created proxy https://logstash-beta.wmflabs.org for logstash instance [14:45:25] Logged the message, Master [14:49:22] hashar, anything I can help with today? [15:05:40] aude, can you catch me up on what's happening with the 'maps' project? [15:07:29] hashar, anything I can help with today? [15:09:23] andrewbogott: we need map warper (ask chippy) [15:09:32] andrewbogott: sean has created the eqiad database \O/ :-] [15:09:35] wma, check with daniel schwen [15:09:42] aude: I don't know who chippy is [15:09:48] tim waters [15:09:49] here [15:09:52] :) [15:09:54] Ah! [15:10:02] chippy, what's up with 'map warper'? [15:10:04] I do need to add ticket to bugzilla, sorry [15:10:27] rest of the instances are not needed and we can start fresh [15:10:38] chippy, please do it right now. the deadline for this passed weeks ago… [15:10:42] maps-warper instance under the maps project. [15:11:01] aude: What is daniel's irc handle? [15:11:06] Or, can you pm me his email? [15:11:12] one sec [15:11:21] he's here sometimes, usually when he has question or problem [15:12:25] andrewbogott: I got to figure out how to have puppet to create the syslog-ng destination directory. [15:13:05] hashar: Yeah, it's dumb that puppet doesn't do recursive directory creation. Probably just shell out 'mkdir -p' [15:13:14] andrewbogott: good idea [15:13:28] hashar: I bet if you grep you'll find that happening other places in the code [15:14:35] andrewbogott: i just deleted wikidata instances we maintain and don't care about [15:14:46] aude: great, thanks. [15:14:52] there is "wikidata-jenkins" which can be left until the end and deleted [15:15:01] or probably deleted now [15:15:27] there are a few others we don't maintain (e.g. they belonged to students) [15:15:44] if you want to "auto" migrate those and if they survive, ok (if not ok) [15:16:04] hashar: done :) now its connceted [15:16:05] aude: Would you mind making me a bug about those so I don't lose track? [15:16:11] tonythomas: congrats! [15:16:16] ok [15:16:17] I'm happy to migrate them, just don't want to miss anything. [15:16:19] thanks [15:16:30] one belongs to magnus [15:16:37] i'm sure that is migrated [15:16:46] not 100% but 99% sure [15:17:42] hashar: thanks ! now doing sudo puppetd -tv [15:17:55] suppose i could try to ask the students [15:21:00] andrewbogott, aude have posted to bugzilla: https://bugzilla.wikimedia.org/show_bug.cgi?id=63115 [15:22:12] andrewbogott: I came up with mkdir -p , also added ensure => directory to the /home/wikipedia/syslog file {} statement. https://gerrit.wikimedia.org/r/#/c/119256/4..5/manifests/misc/logging.pp [15:22:23] andrewbogott: https://bugzilla.wikimedia.org/show_bug.cgi?id=63116 [15:22:26] chippy: I'll copy over your instance right now. [15:22:33] thank you :) [15:23:02] chippy: There are a few ways to handle shared storage. If you don't require it to be totally up-to-the-minute fresh then you have recent backups in /data/project/glustercopy and /home/glustercopy. [15:23:35] chippy: the pmtpa instance will be shut down as part of the copy [15:24:43] I get this error on runnig sudo puppetd -tv http://pastebin.com/2pwanYzV [15:24:50] any thoughts ? [15:25:37] andrewbogott, thanks. Will it be started after that? Or (as im assuming) there isn't any need to after that [15:25:55] chippy: best to leave the old pmtpa instance to die. [15:25:59] I'll start up the eqiad one. [15:26:41] tonythomas: probably your project doesn't have shared homedirs turned on. That's a bit of a bug on our end... [15:26:53] but if you click on the 'configure' link on the 'manage projects' page there are checkboxes… you should check 'em both. [15:38:04] andrewbogott: I enabled both. and the Service user homedir pattern needs to be changed to something like /home/tonythomas01/ ? [15:38:40] tonythomas: um… what project is this? Are you really using service users? [15:42:19] chippy: Have a look at maps-warper.eqiad.wmflabs -- it should be all set [15:42:28] except you'll probably need to rearrange some of the shared storage bits. [15:42:58] andrewbogott: its a project on impementing VERP feature for mediawiki [15:43:19] you can see the request here https://wikitech.wikimedia.org/wiki/New_Project_Request/mediawiki-VERP [15:43:59] tonythomas: Why would you want to put non-human users in /home? [15:44:20] In labs, those canonically go to /data/project [15:44:59] andrewbogott: That reminds me, we need to alter the wikitech interface to (a) stop creating the old-style LDAP entries and (b) change references to local- to $project. [15:45:17] * andrewbogott makes note [15:45:36] Coren: you mean to change Service user homedir pattern to :/data/project ? [15:45:51] hashar: dump reload has completed. have to zzz now :) let me know if anything appears amiss [15:46:25] springle: thank you very much! [15:46:43] tonythomas: Oh, right, it's /home/%p%u by default. My bad. But yeah, most put it in /data/project [15:46:49] !log deployment-prep deployment-db1 data loaded [15:46:51] Logged the message, Master [15:47:01] hashar: yw [15:47:08] tonythomas: /home/ also works. [15:47:19] It just looks odd to me. :-) [15:47:27] let me try with /data/project [15:47:38] That shouldn't make a functional difference. [15:48:32] ok. [15:49:17] Oh, hey, I think I found a cheat around my NFS race. [15:49:44] But /man/ it's ugly. [15:51:05] Coren: I think I am having some connection error when I gave /data/project [15:51:18] ... "connection error"? [15:51:39] That setting affects only service users/groups. [15:53:34] Coren: Now, when I login, it shows the error Unable to create and initialize directory '/home/tonythomas01'. [15:53:36] and quits [15:54:24] Wait, you're not supposed to try logging in with service users. [15:55:49] Ah, nevermind, your shell account name and wikitech names differ and that confused me. :-) [15:56:01] Coren: yeah :) [15:56:05] they differ [15:56:19] dont know why I made like that, but tonythomas01 is my shell account name [15:56:23] That last error is a known symptom of an instance not having its home properly rw because of the very race condition I had just discussed. :-) [15:56:38] I can fix it manually. What's your instance name? [15:57:19] the instance id is : i-000002ad.eqiad.wmflabs [15:58:01] and Jeff_Green says he too is having trouble logging in [15:58:11] he is away at the moment though [15:59:27] fyi, you don't have to refer to the id. It's also 'box2.eqiad.wmflabs' [15:59:58] Coren: ok [16:01:18] Coren: now its ok [16:01:23] Thanks [16:03:20] tonythomas: No it's not. Please log off. [16:03:21] :-) [16:03:30] Coren: done [16:04:23] Won't be long. [16:04:42] k. [16:06:59] tonythomas: /now/ it's fixed properly. :-) [16:07:17] Coren: earlier, there was nothing inside /home/ [16:07:17] :) [16:07:20] let me see [16:10:29] Coren: sad to bug you again , but now sudo puppetd -tv shows this warning warning: /Stage[main]/Role::Labs::Instance/Mount[/data/project]: Skipping because of failed dependencies [16:10:46] there are no errors and the catalog run finishes [16:10:48] tonythomas: I know, I just noticed. I will have to reboot your instance [16:10:58] To give you /data/project [16:11:01] logged out [16:11:03] ok :) [16:11:20] chippy, still here? [16:13:42] tonythomas: All done. [16:16:21] aude: after those three wikidata-dev instances… are we still waiting on addshore to do more work in that project? [16:17:42] wikidata jenkins can be die when tampa dies [16:17:55] we don't mind keeping it around until then, just in case we want to look up something [16:19:56] aude: Ok, can I move it to the 'finished' section? [16:20:00] yes [16:20:04] cool. [16:20:17] and after chippy signs off on his maps instance, that one too? [16:20:27] yes [16:20:55] ok, thanks [16:22:08] andrewbogott: So, I found a way to prevent the mounts being mounted readonly if the export didn't go through yet -- but at the cost of the mount /failing/ entirely. There are two approaches: let it fail, meaning you don't get the shared filesystems at all until next puppet run tries it again or make puppet wait for the filesystem to become available. I don't particularily like either option. [16:22:45] I think failing until the next puppet run is ok -- certainly better than what happens now. [16:23:04] well, really either sound ok to me. [16:23:09] *sounds [16:23:40] much better, at least :) [16:23:40] The "real" solution is to have instance creation actively make the exports rather than wait for a daemon to run at interval. [16:24:27] Or, actively cause the exports to be created. Same difference. [16:24:34] Yeah. There are a lot of hacky, easy ways to do that… doing it a proper way with an OpenStack API will be a fair bit of trouble. [16:24:58] How often does the daemon run now? Making puppet block on creation /and/ shortening the refresh time seem like a pretty good option. [16:25:16] hm, having a lot of trouble with subj/verb agreement today [16:26:20] andrewbogott: Hmm. 2 min IIRC [16:26:35] * Coren checks. [16:27:12] The problem has never been the actual interval so much as mountd caching what it got the first time 'round for a while after the last unmount [16:28:10] Right, but if you implement either of the above fixes then the interval starts to matter. [16:28:35] Indeed. [16:28:52] Oh, wait, I can't do that. [16:29:18] If I wait for the filesystem to become available rather than fail, then projects without shared home / shared project will hang puppet. [16:29:30]