[00:10:57] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:15:47] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.18, 0.13, 0.17 [00:59:57] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3615 MB (20% inode=75%): [01:23:27] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [01:31:27] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 64 MB (4% inode=57%): [01:48:39] 06/22/2012 - 01:48:39 - Updating keys for krenair at /export/keys/krenair [01:52:36] 06/22/2012 - 01:52:36 - Updating keys for krenair at /export/keys/krenair [02:25:07] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache30 i-000002d3 output: Puppet has not run in last 20 hours [02:26:07] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-bits i-00000264 output: Puppet has not run in last 20 hours [02:29:07] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [02:31:07] PROBLEM Puppet freshness is now: CRITICAL on deployment-bastion i-000002bd output: Puppet has not run in last 20 hours [02:39:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-feed i-00000118 output: Puppet has not run in last 20 hours [02:40:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-jobrunner05 i-0000028c output: Puppet has not run in last 20 hours [02:44:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-imagescaler01 i-0000025a output: Puppet has not run in last 20 hours [02:45:06] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [03:08:59] RECOVERY Free ram is now: OK on ganglia-test2 i-00000250 output: OK: 87% free memory [03:43:27] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 13% free memory [03:48:27] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 206 processes [03:53:27] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 198 processes [03:56:57] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [04:03:34] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:06:27] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [04:08:27] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [04:08:37] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:11:57] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:13:27] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:13:27] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [04:21:38] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:21:58] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:26:29] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 8% free memory [04:26:39] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:28:19] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 3% free memory [04:31:29] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:33:19] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:57:40] 06/22/2012 - 04:57:40 - Creating a home directory for slaporte at /export/keys/slaporte [04:58:38] 06/22/2012 - 04:58:38 - Updating keys for slaporte at /export/keys/slaporte [05:43:09] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [05:57:00] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:58:10] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 20% free memory [06:00:10] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache31 i-000002d4 output: Puppet has not run in last 20 hours [06:16:10] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [06:27:00] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.10, 0.17, 0.21 [06:28:20] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 203 processes [06:35:00] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [06:43:20] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 199 processes [08:15:15] hello [08:16:27] paravoid: morning :-] Would you have time to discuss my /etc/wikimedia-realm change https://gerrit.wikimedia.org/r/#/c/12377/ ? :-) [08:48:09] finally finished syncing the conf between beta and prod :-] [08:55:26] hashar: sorry, maybe later ... [08:55:27] deep in shit. [08:55:37] Ryan_Lane: can we have IP? [08:55:41] for feed [08:55:47] paravoid: :o [08:55:51] feed? [08:55:55] Ryan_Lane: irc server [08:56:05] Ryan_Lane: we talked about it yesterday I want to set it up now [08:56:17] in which project again> [08:56:21] deployment-prep [08:56:34] RC feed [08:56:43] you said we can't use production one [08:57:01] Ryan_Lane: oh, here? hi :) [08:57:19] howdy [08:57:24] paravoid: in europe we have kind of morning [08:57:25] :D [08:57:27] paravoid: good luck, whatever you can be working on. [08:58:10] petan: ok, added [08:58:13] cool [08:58:35] petan: make sure it's read-only [08:58:53] except for the bot that's adding feeds [08:59:25] petan: eh? [08:59:43] Ryan_Lane: fail [08:59:51] Ryan_Lane: sure [08:59:53] petan: what? [09:00:00] Ryan_Lane: Failed to allocate new public... [09:00:04] oh? [09:00:07] I wonder if we're out [09:00:10] Ryan_Lane: I'm down to 2 d-i bugs, going for the third [09:00:24] Ryan_Lane: there is one ip we might release I think, let me check [09:00:26] 2 down, 1 to go I mean [09:00:44] hm. no. there's a bunch of IPs left [09:00:44] Ryan_Lane: also, I managed to get a VGA console from the Ciscos [09:00:55] paravoid: heh [09:00:58] hashar: what is mobile.beta.wmflabs.org << it has IP but doesn't work [09:01:44] that involved installing foxyproxy, messing with javaws, sun/openjdk, configuring socks there, finding an unfixed-in-Debian bug in the way it does SSL [09:01:54] disabling SSL in the Ciscos [09:02:15] then finding a bug on their oob where it didn't let me login, figuring out how to logout the web sessions from its ssh [09:02:40] failing, doing that again (it takes like 30sec to logout a session...), clearing firefox cache, logging in again [09:03:32] oh, disabling flashblock since that thing needs flash to login [09:03:48] @search logs [09:03:48] Results (Found 1): hyperon, [09:03:57] lol [09:04:01] petan: Ryan preallocated us 3 or 4 public IP address so we can eventually setup HTTPS and mobile frontends [09:04:22] ah. yep. out of IPs [09:04:38] there's a bunch that are allocated, but aren't used [09:04:42] Ryan_Lane: I guess you can grab some of the IP you allocated to beta [09:04:51] allocated to beta? [09:04:53] hashar: wait [09:04:53] :D [09:04:57] we have some we need [09:05:02] Version: Invalid [09:05:08] fcking Cisco [09:05:10] I don't think we are ever close to deploy the HTTPS frontend or mobile gateway [09:05:14] oh wait [09:05:21] I didn't use the full subnet I don't think [09:05:28] no [09:05:29] I did [09:05:30] now I just log the management, how nice. [09:05:59] paravoid: I'd say it's your own fault for trying to use their stupid bloated web interfaces ;) [09:06:09] hehe [09:06:09] paravoid: looks like those cisco boxes gives you hard work. Should get Juniper systems instead :-] [09:06:12] paravoid: you see different things from the vga console? [09:07:04] well, seeing as that we're out of IPs, it's probably a good time to start looking at making a reverse proxy service [09:07:09] Ryan_Lane: at one point I managed to finish an install with good partioning [09:07:25] what happened? [09:07:25] the machine booted, however there's no console in ttyS0 by default [09:07:34] yeah, it's S1 [09:07:37] and it didn't reply to pings [09:08:09] oh. wait. no it's S0 [09:08:29] yes, but there's no console just after the install [09:08:33] gettys [09:08:42] you can see kernel output but there are no gettys setup [09:08:45] ah. right. puppet does that [09:08:47] so you can't login [09:08:54] !log deployment-prep Deleting hostname mobile.beta.wmflabs.org and releasing 208.80.153.244 [09:08:55] then I tried to login via kvm [09:08:55] Logged the message, Master [09:09:04] no ping? [09:09:08] that doesn't make sense [09:09:10] and then realized that we set the root password to "x" crypted [09:09:21] yes. you must use the ssh key [09:09:27] no network. [09:09:41] and that's why I said "1 bug to go" [09:09:44] heh [09:09:53] is it getting bad network settings from the installer? [09:10:29] nope, I spawned a shell from the installer and everything seemed correct [09:10:37] oh [09:10:38] both in /e/n/i and udev persisten-net [09:10:38] I know [09:10:45] you do? [09:10:51] that subnet isn't allowed to ssh [09:10:58] it doesn't even ping [09:10:58] to and from production [09:11:01] oh [09:11:03] hm [09:11:07] that's also filtered, likely [09:11:15] why? [09:11:19] is it on a different subnet? [09:11:34] because it hosts untrusted vms [09:11:44] is this one virt6? [09:11:56] virt8 [09:12:19] aaand I just lost its management [09:12:23] it doesn't work either via SSH or HTTP [09:12:32] I can't ping to virt4 from sockpuppet either [09:12:33] you can't login [09:12:50] virt7 pings [09:12:54] oh well, I'll try my new partioning changes on virt7 now [09:13:00] it does!??!! [09:13:07] you need to use the new_install key [09:13:12] Ryan_Lane: can you insert dns irc.beta.wmflabs.org to that new IP [09:13:12] I know about that [09:13:16] I just pinged it from virt5 [09:13:16] Ryan_Lane: it doesn't let me [09:13:36] petan: which new IP? [09:13:39] argh dammit [09:13:41] and why wouldn't it let you? [09:13:44] a filter all along [09:13:46] dammit [09:13:47] use the "beta" domain [09:13:50] Ryan_Lane: 208.80.153.244 [09:14:05] petan: don't use the wmflabs domain, use the beta one [09:14:13] then just add "irc" [09:14:31] paravoid: heh [09:14:44] can we just reply "icmp prohibited" instead of dropping traffic please? [09:14:50] this is highly unintuitive [09:14:57] I'm fine with that [09:15:01] I didn't set up those rules ;) [09:22:13] Ryan_Lane: it's official, we lost virt8 mgmt [09:22:28] yay. [09:22:29] you bricked it? :) [09:24:44] Ryan_Lane: FYI, there's cake at WMDE: http://instagr.am/p/MK7EETQQU_/ [09:24:59] oooo [09:25:04] I want a piece [09:25:08] I'm coming in soon [09:25:37] AHHH [09:25:55] Jens_WMDE: let me take a flight from NTE to BER!! I want a piece too [09:26:10] Ryan_Lane: it's from all the cursing. [09:26:17] BER... as if... [09:26:17] paravoid: heh [09:26:32] Jens_WMDE: or TXL :-D [09:26:43] :'( [09:27:10] there is no direct flight though -:-( Only to Dusseldorf [09:27:13] *sigh* [09:27:40] that is what prevented me from spending a week of vacation in Berlin with my family. Was going to be too long for my little daughter [09:35:12] hashar: why we run udp2log on -feed [09:35:58] we should move it to syslog [09:42:08] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 75 MB (5% inode=40%): [09:42:40] Ryan_Lane: is there a version of ratbox somewhere [09:42:47] you use on prod [09:42:57] well, kind of [09:43:01] we have patches in svn [09:43:02] because there is some hack [09:43:07] yes I saw them [09:43:09] and I just patched the lucid version [09:43:16] ok, so is there some .deb [09:43:21] not really [09:43:24] you'd need to build the deb [09:43:31] ok, where are the files [09:43:35] https://gerrit.wikimedia.org/r/#/c/12425/ [09:43:46] apt-get source ircd-ratbox [09:43:51] then replace the debian directory with that [09:45:09] * petan does it [09:49:03] petan: we have no syslog server yet [09:49:13] petan: I have no idea what -feed was meant to be originally :-] [09:49:57] hashar: we have syslog server but it's empty [09:50:01] Ryan_Lane [09:50:06] where is that folder I should replace [09:50:14] https://gerrit.wikimedia.org/r/#/c/12425/ [09:50:16] I know [09:50:18] but on fs [09:50:21] ... [09:50:28] apt-get source ircd-ratbox [09:50:33] petan: we have bug somewhere. The reason is that the syslog class uses syslog-ng which conflict with rsyslog which is installed by some base:: default class [09:50:34] I mean the directory I should replace with that in git [09:50:40] .... [09:50:50] Ryan_Lane: I did install it using apt [09:50:56] aaaah [09:50:56] you've never built a package before? [09:50:58] nvm [09:50:58] :D [09:51:03] now I get it [10:01:44] paravoid: I have opened a bug about debianizing the parsekit PHP extension. Should I assign it to you? https://bugzilla.wikimedia.org/show_bug.cgi?id=37076 [10:02:02] paravoid: that is a low priority bug by the way since we can just install the extension from PECL [10:04:14] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [10:05:24] petan: we have a /var/fs script that does some recursive permission changes on /mnt/upload and /mnt/export . Do you remember from where it is launched? I could not find the crontab :( https://bugzilla.wikimedia.org/show_bug.cgi?id=37116 [10:05:48] petan: it ran on deployment-nfs-memc.pmtpa.wmflabs [10:06:14] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [10:06:47] petan: nm, found it :-] [10:06:54] PROBLEM HTTP is now: CRITICAL on wikistats-history-01 i-000002e2 output: Connection refused [10:07:38] !log deployment-prep {{bug|37116}} removed deployment-nfs-memc cronjob /var/fs which did some nasty recursing file changes. Has been disabled since May 25th anyway. [10:07:39] Logged the message, Master [10:52:51] ohh [10:52:53] 2012-06-22 10:47:47 1Si1P1-0005L3-Gx mchenry.wikimedia.org [2620:0:860:2:219:b9ff:fedd:c027] Network is unreachable [10:52:58] looks like exim prefer IPv6 [10:53:05] which does not work for labs :-] [11:17:53] !log deployment-prep Created /etc/wikimedia-realm file containing 'labs' on -dbdump, -apache30, -apache31 and -jobrunner05. Related puppet change is https://gerrit.wikimedia.org/r/#/c/12377/ [11:17:54] Logged the message, Master [11:20:32] $wgArticlePath = '/wiki/$1'; [11:21:09] thank you! [11:21:16] yw [11:22:08] petan: did you get the package built? [11:22:17] petan: if not, I have a .deb I can send to you [11:24:13] Ryan_Lane: yes it's building now [11:24:17] I was afk [11:24:19] ah ok [11:27:46] PHP is pissing me off [11:27:54] PROBLEM Current Users is now: WARNING on bastion-restricted1 i-0000019b output: USERS WARNING - 6 users currently logged in [11:27:58] no way to strip newline chars form a string :( [11:30:40] got it [11:30:45] trim() does not pass by reference :-] [11:37:15] PROBLEM Disk Space is now: CRITICAL on deployment-feed i-00000118 output: DISK CRITICAL - free space: / 38 MB (2% inode=39%): [11:37:44] petan: did you change something on -feed ? [11:38:09] Change on 12mediawiki a page OAuth was modified, changed by Guillom link https://www.mediawiki.org/w/index.php?diff=553347 edit summary: link [11:39:26] Change on 12mediawiki a page OAuth was modified, changed by Guillom link https://www.mediawiki.org/w/index.php?diff=553350 edit summary: group [11:39:26] hashar: why [11:39:33] nagios report / to be full [11:39:45] gotta play with du -x / --max-depth=1 [11:41:12] hashar: yes I am building some stuff there [11:41:29] building? [11:42:07] that is a very small machine [11:42:15] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 75 MB (5% inode=39%): [11:42:18] I think it is just mean to receive UDP packets and write them to /home/wikipedi [11:42:19] a [11:43:24] petan: ahh ircd-ratbox [11:43:34] definitely do that on a new instance [11:43:39] with a larger disk :-] [11:44:05] I am off for lunch [11:44:34] hashar: why, I will clean after [11:44:35] Or just do it in project storage? [11:44:48] Damianz: how could I install dev tools to project storage [11:44:53] apt install that to /usr/bin [11:45:11] apt doesn't have a --install to this path option? rpm does :D [11:46:36] I was lazy to investigate that [11:58:35] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:35] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [12:09:47] ehm, would it make any sense that puppet classes cannot be found anymore that used to be in test branch, after the recent merge? [12:10:28] Ryan_Lane: Error 400 on SERVER: Could not find class misc::mariadb::client ? removed on purpose or other? [12:12:53] mutante: maybe lost in merge [12:13:00] I'm thinking we should use mysql anyway [12:13:15] I don't want people storing their passwords on the filesystem [12:13:20] and that'll happen if we enable ldap auth [12:13:25] hmm, ok [12:13:40] hundfred: see above, can you try using a mysql puppet class and applying that instead? [12:21:49] the wrong way to test: mount a gluster volume ontop of the share that's being exported [12:21:59] Ryan_Lane: I created the server it's running but your patch doesn't work [12:22:05] Ryan_Lane: it's possible to create channels [12:22:19] try irc.beta.wmflabs.org [12:22:23] petan: for any user? [12:22:26] Ryan_Lane: yes [12:22:36] Ryan_Lane: that patch is wrong I think [12:22:45] can they talk in the channels? [12:22:50] I don't know [12:22:57] we need 2 users to check it :D [12:23:04] or I would need to connect twice [12:23:04] set [12:23:06] sec [12:23:52] It didn't say I can't talk [12:24:05] what's the port? [12:24:07] 6667? [12:24:15] mhm [12:24:22] and which channel to join to test? [12:24:30] test [12:24:44] Ryan_Lane: it doesn't work :/ [12:24:56] I mean it works [12:24:57] that is bad [12:24:58] :D [12:25:50] Ryan_Lane: sigh, looks like they are all gone indeed, also on existing instance. Could not find class misc::mariadb::repository, will have to switch to mysql to unbreak puppet and crons [12:25:57] hundfred: ^ [12:26:07] sorry :( [12:26:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache30 i-000002d3 output: Puppet has not run in last 20 hours [12:27:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-bits i-00000264 output: Puppet has not run in last 20 hours [12:27:25] well, no passwords on systems / LDAP is good :), i just liked not having the oracle message [12:27:50] and it shouldnt be a big deal / drop-in replacement [12:28:42] petan: I'll try to sanitize the config [12:28:47] but it will likely take me a while [12:29:00] mutante: I'll come back to you after you've eaten your db files [12:29:00] and I can't promise it'll be today [12:29:17] Ryan_Lane: You mean it's not puppetized with variables :D [12:29:22] * Damianz thinks of squid [12:29:23] merely for licensing, not actually performance or anything [12:29:34] the entire ircd config is in the private repo [12:29:34] Ryan_Lane: that only changed my cloak [12:29:45] petan!~petan@anonymous.user [12:29:49] but I still can speak [12:30:02] or I think so [12:30:15] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [12:31:03] I won't puppetize it [12:31:03] I think that problem is with patch anyway [12:31:10] I'll just make a cleaned version I can share with you [12:31:14] it should work [12:31:22] ok [12:31:38] I tried to recompile it by hand but it failed [12:31:47] because it couldn't link it [12:31:55] however when I created .deb packate it worked [12:32:00] * package [12:32:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-bastion i-000002bd output: Puppet has not run in last 20 hours [12:32:27] hashar: we still need to fix that [12:32:30] puppet on beta [12:32:59] hundfred: i am checking this out mysql now https://labsconsole.wikimedia.org/wiki/Help:Using_debs_in_labs , its still being worked on [12:33:09] hundfred: s/mysql/myself [12:33:17] :D [12:33:21] that's a cool typo [12:33:33] hundfred: not for db, for the wikistats package itself :p [12:34:57] beware of cool typos: everything is logged! [12:35:02] heh [12:35:15] It will end up in bz :P [12:35:28] petan: what's the hostname you're putting this on? [12:35:40] mutante: www.bash.org/?5300 [12:35:50] damn :p [12:35:51] actually, what's the IP? [12:35:53] petan: ? [12:36:05] not the public one, the private one [12:38:14] Ryan_Lane: inet 10.4.0.83/24 [12:38:51] petan: http://www.bash.org/?34315 [12:38:55] and the hostname/ [12:39:10] petan: ? [12:40:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-feed i-00000118 output: Puppet has not run in last 20 hours [12:40:24] nevermind I got it [12:40:57] petan: use ircd.conf.labs in /root on the instance [12:41:03] ok [12:41:04] you need to set some passwords in some places [12:41:08] right [12:41:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-jobrunner05 i-0000028c output: Puppet has not run in last 20 hours [12:41:30] oh, and change the sid [12:41:53] nevermind I did it [12:42:44] hmm, getting reports from hundfred about problems with ssl cert on puppet run... returns: change from notrun to 0 failed: /bin/ln -s /etc/ssl/certs/wmf-labs.pem ... [12:42:56] Ryan_Lane: what encryption I should use [12:43:01] for pw [12:43:01] Ryan_Lane: http://pastebin.com/ar7NKi0Y [12:44:08] hundfred: you need to configure the instance to add the certs [12:44:08] hundfred: but you still have a "Finished catalog run" to puppet finishes, and these are warnings , not breaking it [12:44:12] yay [12:44:15] Ryan_Lane: works! [12:44:18] great [12:44:30] see, config :) [12:44:49] so... [12:44:53] the script will use the rc user [12:44:55] hundfred: these are warnings about a dependency to execute creating a symlink [12:45:02] and the rc user will be able to create channels and tak [12:45:04] *talk [12:45:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-imagescaler01 i-0000025a output: Puppet has not run in last 20 hours [12:45:37] hundfred: i don't think you have to worry about it right now , you can still apply other puppet classes [12:45:40] this is good, we got to test my new package too :) [12:46:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [12:48:43] Ryan_Lane: now we need bot [12:48:44] :D [12:48:56] I already pushed it into the public repo [12:49:19] I'd say use the script without puppet for now [12:49:32] at some point I'll templatize all of the IRC stuff [12:49:42] then we can use the puppet classes [12:50:10] ok [12:50:13] thumbs up for IRC stuff, like! [12:50:17] where is that repo :) [12:50:20]