[00:20:14] how do I schedule a crontab? Should I do it from my account? IE local-voxelbot doesn't have permission to set a crontab [00:20:33] er, you should be able to do it as your tool [00:20:53] crontabs/local-voxelbot/: fdopen: Permission denied [00:21:05] My tool can'... what sdamashek said [00:21:07] *sigh* [00:21:18] fwilson: I'm discussing that with legoktm :P [00:21:22] :) [00:21:33] is that when you try crontab -e? [00:21:45] yes [00:21:51] weird [00:21:56] file a bug :) [00:22:09] on tools-login [00:22:29] tools-login is correct? [00:23:48] yeah [00:43:22] legoktm: I noticed local-voxelbot isn't in the crontab group [00:43:29] ouch. [00:43:32] should that be automatic? [00:43:38] probably [00:48:20] bug fileds [00:48:22] filed* [00:51:02] Tools don't need to be in crontab group. [00:51:26] oh they don't? [00:51:33] how should I schedule crontabs then? [00:51:39] from my account? [00:52:10] sdamashek: Do you "become voxelbot", and then "crontab -e"? [00:52:20] scfc_de: exactly [00:52:51] Hmmm. What does "crontab -l" say? [00:53:02] local-voxelbot@tools-login:~$ crontab -l [00:53:02] crontabs/local-voxelbot/: fopen: Permission denied [00:53:21] cause it's not in the crontab group [00:54:55] No, that's not the cause. local-typoscan isn't in the crontab group either, and "crontab -l" works perfectly. "crontab" is setuid, so the calling user doesn't need to access /var/spool/cron/crontab. [00:55:23] any ideas then? [00:57:09] I don't think normal users can misconfigure their tool accounts in a way to shut them out from cron, and the admins petan and Coren are away, so you'll probably have to wait till tomorrow. [00:57:36] okay [01:00:18] Ryan_Lane, are you currently grinding through instances with puppetValues.php? I'm happy to do the cleanup but don't know how to generate the list of 'old salt' instances. [01:00:32] I did, yeah [01:00:42] still a bunch of instances not reporting with salt, though [01:00:47] not totally sure why [01:01:15] sdamashek: "man crontab" says: "There is one file for each user's crontab under the /var/spool/cron/crontabs directory. Users are not allowed to edit the files under that directory directly to ensure that only users allowed by the system to run periodic tasks can add them, and only syntactically correct crontabs will be written there. This is enforced by having the directory writable only by the crontab group and configuring crontab [01:01:15] command with the setgid bid set for that specific group." [01:01:32] sdamashek: So the trailing "/" in your error message looks very odd. [01:01:49] andrewbogott: I think I got all the ones I can [01:02:23] 30 more -- big improvement! [01:02:40] yeah [01:02:50] Still leaves a lot of mysterious lost instances though. [01:02:52] and at least 5 instances are ubuntu 11.10 [01:02:56] sdamashek: If you do "env | fgrep voxelbot" as the tool account, are there any "/"s there? [01:03:17] still, though, that's like 60 not reporting [01:06:27] Is it possible to make a lucid instance behave? Or are lucid instances permenantly out of the game? [01:10:38] vacation9@tools-login:~$ env | fgrep voxelbot [01:10:39] OLDPWD=/data/project/voxelbot/VoxelBot [01:11:06] sorry irc froze [01:11:31] sdamashek: erm, thats not as your local-user [01:11:42] he said as the tool account [01:11:56] which means $ become voxelbot; first [01:12:15] oh [01:12:23] local-voxelbot@tools-login:~$ env | fgrep voxelbot [01:12:23] USER=local-voxelbot [01:12:23] USERNAME=local-voxelbot [01:12:23] MAIL=/var/mail/local-voxelbot [01:12:24] PWD=/data/project/voxelbot/ [01:12:26] HOME=/data/project/voxelbot/ [01:12:28] LOGNAME=local-voxel [01:12:37] sorry for the spam [01:16:48] Hmmm. I don't understand that, but Coren or petan will have to debug that. [01:18:02] andrewbogott: they should work. is something broken with one? [01:18:21] Nope, just wondering if I should be exluding lucid boxes from the instances that need fixing [01:19:02] precise and lucid should work [01:19:07] only 11.10 are broken [01:19:16] andrewbogott: I have a list of downed hosts on virt0 [01:19:29] hosts-down.txt in /root [01:19:54] this instance is hosed: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000000e2 [01:20:02] drdee: ^^ [01:20:06] that's in analytics [01:20:11] I think it's been dead for ages [01:20:32] in fact, I think it was one of the 30 that was corrupted ages ago and I just never deleted it [01:20:35] Ah, I just generated that same list. Matches. [01:20:53] I'll start at the end :) [01:20:59] heh [01:21:00] cool [01:21:01] thanks [01:21:12] yep. that one is long dead [01:21:14] I'm deleting it [01:21:40] hm, of course there's the problem of instances that predate my root key :( [01:24:44] andrewbogott: mark the ones you get permission denied on in the file [01:24:55] hm. [01:24:58] maybe copy the file [01:25:25] Yeah, I have a copy [01:25:39] When puppet says 'err: Could not request certificate: getaddrinfo: Name or service not known' that makes me think that the instance never came fully up to begin with [01:25:49] there's nothing to do about that, is there? [01:26:20] Oh, or it's possible I just forgot to sudo :( [01:28:27] andrewbogott: some instances may be missing /etc/puppet/puppet.conf [01:29:15] I ran a ddsh to kill puppetmaster::self on instances the class was removed and ran a command to fix puppet.conf [01:29:37] but some instances hadn't run puppet in so long that they didn't have /etc/puppet/puppet.conf.d [01:29:45] will a clean puppet run generally bring salt up, or is there another step to get it going? [01:32:01] clean puppet run should [01:32:06] look in /etc/salt/minion [01:32:20] Ah, good, this one just started working. [01:32:22] if master_finger ends in dd it should be ok [01:35:58] OK… this one is saying "Exiting; no certificate found and waitforcert is disabled" and it has a puppet.conf dated today [01:37:13] andrewbogott: is it empty? [01:37:33] if so, copy a good one from bastion-restricted and change the instance-id in it [01:37:41] puppet.conf? No, not empty. Looks right. [01:37:46] oh [01:37:46] hm [01:37:52] which instance? [01:38:14] bots-gs [01:38:22] which should maybe just be deleted anyway [01:40:08] ah. I know why [01:40:19] look at its hostname in puppet.conf [01:40:30] there's some package that fucks with the hostname [01:41:02] hm. no petan around.... [01:41:40] I'll just make a note to ask him tomorrow… maybe we can delete it. [01:42:16] * Ryan_Lane nods [01:46:29] I'm going to try to resize this tiny instance [01:46:39] hopefully I don't break it [01:46:53] of course, I have serious doubts its being used. / has been at 100% for ages [01:53:28] bleh. I already can't wait to upgrade to grizzly [01:54:19] damn it [01:54:24] killed that tiny instance [01:58:21] ImageNotFound: Image 28 could not be found. [01:58:23] bleh [01:58:33] annoying [01:58:53] thanks diablo -> essex upgrade for that one [01:59:53] are you getting lots of 'no route to host'? [02:00:30] yep [02:00:36] if you get that it likely needs to be rebooted [02:00:42] it probably OOMd [02:08:09] i-0000018f.pmtpa.wmflabs is dead and is 11.10 [02:09:47] I've emailed the owner [02:12:58] i-000001e5.pmtpa.wmflabs was in some weird state in nova [02:14:00] it wouldn't let me reboot it via the nova command, so I changed its state via the database [02:14:02] then rebooted it [02:16:30] same with i-00000207 [02:17:37] hm… the home-migrate project can die, right? That was you and me moving things around... [02:17:55] yep [02:18:11] anything in the project anyway [02:18:15] deleting the project itself is hard [02:18:22] it may leave lingering resources [02:20:25] hm… speaking of which, this instance seems to belong to a no-longer-existing project. -00000500 [02:20:58] ewwww [02:21:07] which project is it supposed to belong to? [02:21:27] ah. I see [02:21:31] that's problematic [02:21:38] I'll recreate the project [02:22:07] and I'll delete the instance [02:22:17] done [02:22:35] There's another one... [02:22:42] 4fa [02:22:54] same project. [02:24:07] deleted [02:24:32] are you down to 2dd yet? [02:25:33] 2dd? [02:25:37] I don't see it in my list [02:25:42] I'm at 207 [02:26:01] it's possible I deleted it? [02:26:36] not sure… it rejects my key [02:26:57] the list I'm working on is hosts-down-andrew-progress.txt [02:27:05] yeah. I deleted it [02:27:12] it should disappear in a bit [02:27:44] ah. we only have 211 and 21c left [02:28:19] heh. except for the no-root key ones [02:28:24] I'll start on those [02:28:44] yeah, that's most of 'em [02:28:54] I'm working on 21c now [02:29:01] I get no host for i-00000282.pmtpa.wmflabs [02:29:03] must be deleted [02:29:18] yep [02:34:18] New patchset: Tim Landscheidt; "Test that Perl scripts are compilable before packaging." [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/68330 [02:35:31] oops, this one is 11.10, no version of salt available. [02:36:26] yep [02:36:34] I've been emailing folks with 11.10 instances [02:36:41] which one are you on? [02:36:54] i-0000061d.pmtpa.wmflabs doesn't let me in either [02:37:12] rebooting it [02:39:50] i-0000051a isn't in the default security group [02:39:56] there's no way it's ever been used. deleting [02:41:05] rebooted 211 and it seems to be ooming again [02:41:26] heh [02:41:36] I'll email about 211 and 21c [02:41:58] it's ooming as a medium? ouch [02:42:16] wtf is going on on that instance? [02:44:18] ok. taking a break for food [02:44:31] Yeah, I'm going to go in a minute too. I'll catch up in the morning. [02:48:56] we're up to 336 out of 380. I feel like that number on wikitech must be off by a few [03:37:34] New patchset: Tim Landscheidt; "Fix variable declaration and enable warnings in job." [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/68331 [04:08:33] wikidatawiki_p replag is increasing :( [04:35:38] why database dump do I want to try and get a complete mirror of wikipedia working? [04:35:45] which* [06:23:54] YuviPanda: ping [06:23:58] ponnng [07:46:03] Change on 12mediawiki a page Wikimedia Labs/Migration of Toolserver tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=710639 edit summary: [+240] /* What do I have to think of when finishing migration? */ after-migration process (draft) [08:05:41] Change on 12mediawiki a page Wikimedia Labs/Migration of Toolserver tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=710643 edit summary: [+300] /* What do I have to think of when finishing migration? */ switching the url [09:13:12] petan: around? [09:13:21] hi [09:13:29] hi [09:13:42] so what's the position on having long running processes on tools-labs? [09:13:44] specifically... deamons? [09:14:14] what exactly you mean [09:14:58] do you mean system daemons (like some service which is part of tool labs infrastructure) or some tool? [09:15:46] petan: some tool [09:16:16] that is something what should be easily possible I guess [09:16:20] !log integration added Yuvipanda to the project with su rights. [09:16:21] Logged the message, Master [09:16:26] you just run it as continuous task [09:16:37] petan: ah. [09:18:12] petan: specific thing in this instance is to listen to gerrit's stream constantly and do things [09:22:58] ok [09:29:46] !log integration integration-apache1 updated the proxy rules in /etc/apache2/sites-enabled to use the integration-jenkins2.pmtpa.wmflabs hostname instead of an IP. [09:29:48] Logged the message, Master [09:32:45] YuviPanda: depending on what you want to do, emails might be more reliable :-) [09:32:59] but anyway, jstart is what you're looking for [09:33:17] yeah, I decided I don't want to maintain something of that sort [09:33:24] will just integrate into our production zuul [09:36:26] !log integration upgraded Gerrit on integration-jenkins2 to 2.6-rc0-144-gb1dadd2 which comes from apt.wm.o [09:36:28] Logged the message, Master [09:42:34] where can I see the error log for my php page [09:42:59] on toolserver [09:52:08] Oren_Bochman: For genuine PHP errors, there's php_error.log in the tool's directory. But server errors aren't viewable by users yet. [09:53:30] thamks [10:08:14] at last I figured out the bugs [12:17:30] !log tools petrb: killing process 31186 31185 69 Jun11 pts/32 1-13:14:41 /usr/bin/perl ./bin/catpagelinks.pl ./enwiki/target/main_pages_sort_by_ids.lst ./enwiki/target/pagelinks_main_sort_by_ids.lst because it seems to be a bot running on login server eating too many resources [12:17:32] Logged the message, Master [12:19:53] theo|cloud, carl-cbm [12:20:01] you are maintainers of local-enwp10? [12:20:12] it is running too many cpu expensive jobs on tools-login [12:21:15] !log tools petrb: killing process 31190 sort -T./enwiki/target of user local-enwp10 for same reason as previous one [12:21:16] Logged the message, Master [12:22:19] !log tools petrb: killing process 31187 sort -T./enwiki/target -t of user local-enwp10 for same reason as previous one [12:22:20] Logged the message, Master [12:22:48] petan: Load on tools-login is less than 2 (and has been for fifteen minutes). [12:23:35] that box has 1 cpu, load higher than 1 isn't good for interactive machine [12:23:51] these all processes were eating 99% of cpu all time [12:24:46] and, according to rules, the bots and similar tasks /should/ run on grid, or at least on execution nodes, not on login server... it slows the box down and causes various troubles [12:25:34] petan: I just wanted to say that there's no urgency. It would have been totally sufficient to ask them to clean that up. [12:25:57] I did sent them e-mail like 2 hours ago and changed priority to 19 [12:26:05] no response... [12:26:46] Some people have a life (or sleep) apart from Wikipedia, and does changing the priority affect the load? [12:27:31] no it doesn't affect the load, but it makes the processes cause less troubles as they have lower priority than interactive processes which are supposed to run there [12:28:57] this isn't the only bot running on -login, I wouldn't have killed it if it wasn't eating as much resources as it could... [12:30:28] tools-login didn't feel laggy to me. [12:31:53] idk... Coren told me to kill all similar processes on -login (and I am actually not doing it, except for really evil ones) if you disagree with that, why you don't ask Coren to change the rules? he is the boss... I am just a monkey [12:35:17] What rules? http://tools.wmflabs.org/?Rules is still empty. But if you're more comfortable, I'll discuss it with your keeper :-). [12:43:24] !log tools petrb: tools-webserver-01 is running quite expensive python job (currently eating almost 1gb of ram) it may need to be fixed or moved to separate webserver, adding swap to prevent machine die OOM [12:43:25] Logged the message, Master [12:46:39] Didn't Coren set a memory limit on the webservers of, hmmm, 512 MByte or something? [12:46:53] I don't know :/ these things aren't documented [12:47:02] but if he did, he did it wrong [12:47:18] because this is a single process with more than 1000mb of resident ram [12:48:33] Did he only change that for PHP? Hmmm. [12:48:53] idk... anyway that python job just died / ended [13:20:44] a930913: you are maintaining bracketbot, can you please move the bot from tools-login to grid? [13:29:01] whats the naming convention for the different language wikis? [13:29:26] enwiki points to what replica? dewiki points to what replica? [13:30:31] FutureTense: database names. XXwiki = XX.wikipedia.org, XXwikibooks = XX.wikibooks.org, etc [13:31:02] for the replicas? [13:32:28] well, its dbname_p for the public views, if that's what you mean. [13:33:02] yes [13:38:33] When trying to connect to dewiki_p im getting an error: 1049, "Unknown database 'dewiki_p'" [13:38:52] from my application, not the shell [13:39:51] kind of stumped on this [13:41:31] FutureTense try sql dewiki -v [13:41:48] fromt he shell? [13:42:00] yes [13:42:13] it works to me [13:42:19] it connect me to server Connecting to dewiki.labsdb [13:42:24] database dewiki_p [13:42:30] yes, it works for me as well [13:42:42] however my application won't connecto to dewiki_p [13:42:43] I think you connect to wrong host [13:42:54] ah [13:43:01] whats the host for enwiki_p [13:43:18] thats it [13:43:28] thx petan [13:43:31] yw [13:44:36] so all of the databases are xxwiki_p and all of the hsots are xxwiki.labsdb [13:44:37] ? [13:47:38] they should be [13:47:55] if you aren't sure you can always use the sql script with -v or --verbose option [13:48:00] it will tell you what it was resolved to [14:19:50] dewiki.labsdb [14:20:04] works for me :> [14:20:29] oh /me slaps self as he continues reading the above [14:44:46] FutureTense, to be more specific. All the hosts are connected to .labsdb where the database is called _p. [14:45:20] enwiktionary.labsdb host the enwiktionary database. [14:45:33] *enwikitionary_p database [14:46:07] wikidatawiki.labsdb hosts the wikidatawiki_p database etc... [14:46:45] From shell, interface, terminal, bash, whatever... sql is the shortcut command to connect to any database. [14:47:03] FutureTense, hope that helps [14:47:19] got it.. [15:05:18] petan or anyone: i heard there is git support on tool labs. is there documentation about it somewhere? [15:06:03] we want to move render tools to a git repository on labs. [15:06:07] petan: Ah yes, I was testing it and forgot to move it over :p [15:09:38] basically we just want to create a git repository on tool labs which we can push into. what's needed for that? scfc_de, addshore, ... anyone? [15:10:45] JohannesK_WMDE: There's no Git server, if you mean that. But you can create git repositories in the file system, of course. [15:11:10] scfc_de: oh. [15:13:10] JohannesK_WMDE: The bigger iron of course is a Gerrit repo. You have to ping demon^ for that. [15:13:27] JohannesK_WMDE: Eh, ^demon. [15:13:32] scfc_de: is there an svn server? [15:13:36] No. [15:13:41] uh [15:13:46] ok... [15:13:55] Why do you need an VCS server on Tool Labs? [15:14:29] well we need some version control. we could just put our stuff on github of course [15:14:46] put stuff in git on gerrit [15:16:00] JohannesK_WMDE: For something close to Wikimedia like RENDER, Gerrit is probably a good idea. [15:16:39] <^demon> scfc_de: I'm more than happy to make new git repos in gerrit for folks. I have a page on mw.org to keep track of them. https://www.mediawiki.org/wiki/Git/New_repositories [15:16:51] <^demon> There's no backlog, so I'll most likely knock it out right away. [15:18:53] ^demon, scfc_de: it looks to me like gerrit is mostly for mediawiki extensions? we just want to have a repo where several users can push to. [15:19:46] I don't think Gerrit allows direct pushing, you're supposed to go through review first... [15:20:04] the majority of repos there are mediawiki extensions but you can still use it for other stuff [15:20:17] uh. okay. [15:20:29] <^demon> Krenair: Depends on the repo. It's mainly extensions we forbid the direct pushing. [15:20:35] <^demon> Of course pushing for review is always encouraged. [15:21:14] <^demon> I'm thinking though...it might be nice to have a "miscellany" repo or somesuch. [15:21:18] <^demon> Allows direct pushing. [15:21:28] <^demon> Just for people to dump one-off things. [15:23:41] Could just make a git repo somewhere on tool labs and use that as a server [15:28:03] Krenair: Yes, if you "abuse" a tool account, you could share a repository quite easily. But with Gerrit, and SourceForge, and GitHub, and ... around -- they usually have made setting up repositories so simple, that it's not worth to bother. [15:28:32] don't have to use a tool account even, could be anywhere on labs [15:28:33] yes [15:28:36] 'git init' [15:28:38] very difficult [15:29:45] But permissions and umasks can be a pain in the ass. [15:30:22] (Though -- obviously -- I have a number of Git repos on Tool Labs -- every "git clone" creates one.) [15:32:14] for me the biggest problem with sharing repositories was always other people leaving uncommitted changes [15:33:25] You have to create bare repos. [15:34:11] yeah, which should be fine if you just want to use it as a 'server' [15:34:51] It's also the only way to push something to a remote repo if the branch you want to update is checked out there. [15:35:18] so, could we just create a git repo on the tool account, and push to it from remote using our user accounts? [15:35:24] using ssh [15:35:28] But the nice thing is that you can use Git hooks for example to create a mini-Jenkins. [15:35:33] JohannesK_WMDE: Yes, that should work. [15:36:00] (But Gerrit (or GitHub) will be more convenient and less error-prone.) [15:45:35] for now, we will try the bare repo way, i think. it is easiest, and we are on a schedule :-] [15:45:49] we can still push to a different remote repo later... [16:10:38] So… replag >.> [16:13:27] legotm: yeah, i read about the 12-hour replag on tools. is that true? [16:14:10] yup [16:14:15] at least on wikidata [16:14:20] wiki_p [16:17:56] strange [16:19:08] if it's that bad now, i don't want to imagine the replag half a year from now when more tools have moved :D [16:19:45] mhm :( [16:22:05] scfc_de, who is/was working on replication? [16:22:53] asher/Coren i think [16:28:10] legoktm where you see replag [16:28:22] wikidatawiki_p [16:28:31] MariaDB [wikidatawiki_p]> SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(MAX(rc_timestamp)) FROM recentchanges; [16:28:35] ah [16:28:36] 109661.000000 [16:29:02] but maybe there is no RC from that time :P [16:29:05] wikidata is small wiki [16:29:08] :D [16:29:16] you're joking right...? [16:29:21] sort of [16:29:41] wikidata has edits every second so that's not the issue. [16:30:02] MariaDB [dewiki_p]> SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(MAX(rc_timestamp)) FROM recentchanges; [16:30:09] 109712.000000 [16:30:25] a remarkably similar number... ;) [16:30:33] so i guess its s5? [16:31:05] i dunno. on enwiki_p it's 0.000000 [16:31:24] https://noc.wikimedia.org/conf/highlight.php?file=s5.dblist is just dewiki and wikidatawiki [16:32:08] then it seems to be a problem with s5 [16:39:20] legoktm, JohannesK_WMDE: are you guys sure that UNIX_TIMESTAMP can actually understand the format used by rc_timestamp? [16:39:39] DanielK_WMDE: yes, thats the command thats been used on the TS for years [16:39:48] and, don't we have a stored porocedure to solve exactly that problem? [16:40:06] er? [16:40:08] ah, that's the custom conversion function? interesting name :P [16:40:20] idk, i just copied it from https://wiki.toolserver.org/view/Replag [16:41:14] according to http://dev.mysql.com/doc/refman/5.5/en/date-and-time-literals.html the answer should be "yes" [16:41:26] as it allows timestamps to be relaxed, not having delimiters [16:41:49] legoktm: ok, you are right, it does understand it [16:42:03] hey #labs - I need to modify my my.cnf but puppet wants to put it back. this is on a single node mediawiki instance. Is there a good way to make the change stick? [16:51:37] manybubbles disable puppet :> [16:53:16] !puppet [16:53:16] learn: http://docs.puppetlabs.com/learning/ troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html [16:53:21] !puppet del [16:53:21] Successfully removed puppet [16:53:31] !puppet is http://scary.hostei.com/ascarywaterorg-/img/scary+puppet1.jpg learn: http://docs.puppetlabs.com/learning/ troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html [16:53:31] Key was added [17:03:05] I know labs usually has amazing technology but why is the Wikidata edit counter reporting "Caution: Replication lag is high, changes newer than 1 days, 7 hours may not be shown." [17:04:12] CP678|iPad: we know. s5 is replagged [17:07:59] Is there a page like https://noc.wikimedia.org/dbtree/ for labs? [17:09:36] legoktm: s5? [17:09:49] all the wikis on s5. aka dewiki and wikidatawiki [17:10:04] Ok. [17:10:19] That made me think of toolserver. [17:10:58] https://noc.wikimedia.org/conf/s5.dblist [17:20:22] petan, how would you feel about deleting bots-gs? I was trying to get puppet working on it last night but it's kind of a mess. [17:20:43] Coren|Away: seems the replag for s5 is incredibly high [17:21:04] andrewbogott I dont think we need puppet there [17:21:17] How does a labs server let that happen? [17:21:33] petan, in theory we need puppet everywhere, to push out updates and such… at the moment its salt is broken. [17:21:34] andrewbogott I would prefer to delete it later if we really need to because that involves installation of grid master servr [17:21:50] petan: OK… mind if I just patch up salt by hand in the meantime? [17:21:56] Ryan_Lane: It clearly is (laggy), but I've no idea why. I expect Asher coudl tell us. [17:21:58] no problem [17:22:13] petan: everything needs puppet [17:22:24] some package is causing the instance to change its hostname [17:22:29] that needs to get fixed [17:22:37] if that happens, the instance can stay [17:22:39] Ryan_Lane did you see how scary is it? [17:22:40] http://scary.hostei.com/ascarywaterorg-/img/scary+puppet1.jpg [17:23:04] I dont know if I want that creepy thing in front of me :/ [17:23:07]