[00:26:45] <cajoel>	 Quick Q: When I add or remove a puppet class from the puppet group list for an instance, how long should it take for this to be in effect on the puppet master?
[00:32:17] <jeremyb>	 cajoel: should be as soon as you run puppet again.
[00:32:30] <jeremyb>	 cajoel: you can traverse ldap yourself to triple check it's gone or not :)
[00:32:34] <jeremyb>	 unless ldap's broke
[00:32:42] <jeremyb>	 which happens once in a while
[00:33:04] <cajoel>	 jeremyb: would you mind double checking the Class settings for the group?
[00:33:10] <cajoel>	 maybe I should create a new group?
[00:33:24] <jeremyb>	 errr, i've not done much with the ENC
[00:33:39] <jeremyb>	 cajoel: did you check ldap yourself?
[00:33:56] <cajoel>	 I did not, didn't even realize ldap was holding these relationships..  :)
[00:34:14] <cajoel>	 wouldn't know where to start (which ldap server? which creeds?)
[00:34:24] <scfc_de>	 For the curious: ENC?
[00:34:36] <jeremyb>	 external node classifier
[00:34:54] <jeremyb>	 in this case ldap is the store but OSM is the ENC
[00:35:25] <jeremyb>	 cajoel: try e.g. ldaplist on bastion1
[00:35:43] <scfc_de>	 Ah, okay.  BTW, I never had trouble with the Configure page being different from what Puppet sees.
[00:36:01] <cajoel>	 jkrauska@bastion1:~$ ldaplist
[00:36:02] <cajoel>	 The search returned an error.
[00:36:21] <jeremyb>	 cajoel: well you need to give params :)
[00:36:56] <cajoel>	 If there's one thing I hate more than ldap, it's command line ldap..   :)
[00:37:18] <jeremyb>	 hah
[00:37:24] <scfc_de>	 cajoel: Which instance are you working on?
[00:37:30] <cajoel>	 flow1
[00:37:47] <cajoel>	 trying a brute force change of the group
[00:38:12] <cajoel>	 gist: geoip is failing for me, and I'd like to remove it..  I thought I did, but it's still coming across on puppet runs...
[00:39:11] <scfc_de>	 cajoel: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000096d still lists "geoip".  Where did you disable it?
[00:40:35] <cajoel>	 https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup
[00:41:17] <scfc_de>	 That only customizes the UI.
[00:41:53] <jeremyb>	 erm... /me stabs ldap
[00:42:33] <scfc_de>	 cajoel: So you need to re-add the class to NovaPuppetGroup, then Configure, disable, then remove the class from NovaPuppetGroup (if you want).
[00:43:17] <cajoel>	 yikes
[00:43:28] <jeremyb>	 hah
[00:44:25] <jeremyb>	 cajoel: errr, there's a mediawiki feature called flow...
[00:44:36] <cajoel>	 scfc_de: what's scary is that I followed your instructions..  :)
[00:45:06] <jeremyb>	 also, are you doing anything with AS #s?
[00:45:10] <cajoel>	 (meaning understood)
[00:45:33] <cajoel>	 jeremyb: BGP AS #s?
[00:45:53] <jeremyb>	 ya
[00:46:04] <cajoel>	 um, yes..  that's the intent of the project
[00:46:18] <cajoel>	 tallying bytes / AS
[00:46:36] <cajoel>	 from flow stats on routers
[00:46:47] <jeremyb>	 right, cool
[00:46:58] <andrewbogott>	 cajoel: Jumping in late… the 'NovaPuppetGroup' page allows you to configure the set of puppet classes available in a project.
[00:47:06] <andrewbogott>	 … which is a different question from 'which classes are on this instance.'
[00:47:06] <cajoel>	 andrewbogott: I grok now
[00:47:11] <andrewbogott>	 Ah, ok.
[00:47:27] <jeremyb>	 i've skipped grokking :)
[00:47:52] <cajoel>	 what's not fun is that you can configure a set, delete it, and it's still applied on your instance..  :)
[00:48:40] <cajoel>	 well, and it's not fun that I can't seem to use the geoip class on labs.
[00:49:23] <jeremyb>	 maybe it's a city vs. lite thing
[00:49:30] <jeremyb>	 and volatile repo
[00:49:35] <cajoel>	 appears to be something with the volatile repo..
[00:49:36] <andrewbogott>	 cajoel: Indeed… theoretically that 'NovaPuppetGroup' page should be phased out and the set automatically generated… but that's pending the Great Puppet Refactor.
[00:49:54] <cajoel>	 Error 400 on SERVER: Not authorized to call find on /file_metadata/volatile/GeoIP 
[00:50:00] <jeremyb>	 right
[00:50:08] <jeremyb>	 so figure out how to make it use lite :)
[00:51:02] <cajoel>	 class geoip::data::lite
[00:51:11] <cajoel>	 can I get a $1?
[00:51:30] <jeremyb>	 no bitcoin for u
[00:52:17] <cajoel>	 maybe we should drop an environment check and force lite for labs?
[00:58:00] <dschwen>	 Hi all, any way to access the tool-labs replica dbs from a different labs project?
[00:59:52] <scfc_de>	 dschwen: Yes.  But you need a) to copy /etc/hosts and iptables -t nat and b) accounts on LabsDB -- I'm not sure if the latter are automatically created for service groups in all Labs projects.
[01:00:32] <dschwen>	 oh, I missed the nat part
[01:00:50] <scfc_de>	 (You can do without hosts and iptables, but than you have to connect to labsdb0815:4711.)
[01:01:03] <scfc_de>	 Files are in /data/project/.system on Tools.
[01:03:47] <cajoel>	 ferm seems to be failing in labs too
[01:03:53] <cajoel>	 anyone know if ferm was tested in labs?
[01:04:11] <cajoel>	 I realize it's kinda redundant with security groups... but..
[01:04:50] <dschwen>	 an ssh tunnel would do the trick as well for me, but for some reason I can only go from tools-login to my project and not the other way.
[01:05:04] <dschwen>	 I assume there is some ssh agent magic happening
[01:05:28] <dschwen>	 with my project set up to access a global authorized_keys file
[01:05:34] <scfc_de>	 cajoel: failing = ?
[01:06:08] <cajoel>	 root@flow1:/etc/ferm# service ferm start
[01:06:09] <cajoel>	  * Starting Firewall ferm                                                                                                                                                                                                                                                                    Error in /etc/ferm/conf.d/10_nrpe_5666 line 8:
[01:06:10] <cajoel>	         chain INPUT
[01:06:11] <cajoel>	         {
[01:06:13] <cajoel>	             proto tcp dport 5666
[01:06:14] <cajoel>	             {
[01:06:15] <cajoel>	                 saddr $ INTERNAL <--
[01:06:16] <cajoel>	 no such variable: $INTERNAL
[01:07:36] <cajoel>	 Q: If there a class that was recently added to the production puppet master, how long until it hits labs?  Do they sync?
[01:07:58] <cajoel>	 +is
[01:08:21] <jeremyb>	 sounds like a question for abartov
[01:08:22] <jeremyb>	 gah
[01:08:26] <jeremyb>	 andrewbogott:
[01:08:37] * jeremyb  has been pinging asaf a lot today :)

[01:08:49] <andrewbogott>	 cajoel: Labs uses the same branch as production for puppet.
[01:08:59] <andrewbogott>	 So as soon as it's been merged in gerrit it should be available for labs.
[01:09:13] <cajoel>	 andrewbogott: cannot find module pmacct ...  which I added today.
[01:09:28] <cajoel>	 will double-check my work
[01:09:38] <andrewbogott>	 are you using self-hosted puppet?
[01:09:46] <jeremyb>	 andrewbogott: but how often is it synced from gerrit?
[01:10:10] <cajoel>	 I am using selfhoster.
[01:10:13] <cajoel>	 selfhosted
[01:10:32] <andrewbogott>	 Ah, well, in that case you aren't getting your puppet classes from upstream but rather from the local checkout on your instance...
[01:10:35] <andrewbogott>	 that's what 'self-hosted' means :)
[01:10:49] <andrewbogott>	 So you'll need to update your local checkout.  One moment and I will find you a link
[01:11:32] <andrewbogott>	 cajoel, item one here:  https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ
[01:11:53] <cajoel>	 crap, just unchecked role::puppet::self , and then hosed the SSL 
[01:12:25] <andrewbogott>	 "CAUTION: There is currently no easy way to rollback from role::puppet::self to the central puppetmaster. It's not hard to manually force puppet to run against the central puppetmaster, but there is no programmatic way of removing packages, git clones etc. "
[01:12:25] <cajoel>	 I can dump the self hosted now.
[01:12:35] <cajoel>	 CAUTION indeed..
[01:12:38] <andrewbogott>	 If you don't want to use self-hosted, best to start a fresh instance.
[01:12:42] <andrewbogott>	 If you can stand it
[01:12:46] <cajoel>	 I can stand it
[01:12:49] <cajoel>	 but not at 5pm
[01:12:56] <andrewbogott>	 :)
[01:13:05] <cajoel>	 thanks for the help.
[01:13:05] <scfc_de>	 cajoel: Re ferm, what's the content of /etc/ferm/conf.d/00_*
[01:13:22] <cajoel>	 # Autogenerated by puppet. DO NOT EDIT BY HAND!
[01:13:22] <jeremyb>	 it's 5pm??! weird coast
[01:13:22] <cajoel>	 #
[01:13:23] <cajoel>	 # 10_nrpe_5666:
[01:13:25] <andrewbogott>	 cajoel: Start your new instance now, then it'll be all ready and happy when you come in tomorrow
[01:13:25] <cajoel>	 domain (ip ip6) {
[01:13:27] <cajoel>	 	table filter {
[01:13:28] <cajoel>	 		chain INPUT {
[01:13:30] <cajoel>	 			proto tcp dport 5666 { saddr $INTERNAL ACCEPT; }
[01:13:31] <cajoel>	 		}
[01:13:32] <cajoel>	 	}
[01:13:32] <cajoel>	 }
[01:14:14] <cajoel>	 it's pretty fast to load, etc..   I have one package I'm building by hand until faidon makes me a deb
[01:14:19] <cajoel>	 so it's not completely automated yet...
[01:14:22] <cajoel>	 tomorrow
[01:14:28] <cajoel>	 thanks folks
[01:14:39] <scfc_de>	 cajoel: No, I mean 00_*.  That's where INTERNAL should be defined.
[01:14:54] <cajoel>	 there are no 00 files
[01:15:01] <cajoel>	 so it's maybe an ordering issue with ferm
[01:15:09] <cajoel>	 it downloaded the 10 file
[01:15:18] <cajoel>	 and then tried to restart ferm (notify) 
[01:15:20] <cajoel>	 and barfed
[01:16:36] <scfc_de>	 cajoel: You're right.  It's defined in base::firewall only, and that's a complicated topic :-).
[01:18:26] <cajoel>	 scfc_de: so I need to add base::firewall at a minimum?
[01:18:40] <cajoel>	 and ferm should probably make it a require ??
[01:20:24] <scfc_de>	 I have never used it in practice, but it looks that way; better ask paravoid though, I think he wrote the module.
[01:29:53] <andrewbogott>	 cajoel: Are you using labs to design a system that will eventually be outside of labs?
[01:30:05] <andrewbogott>	 Because for long-term labs projects you don't need to worry much about firewalls, labs instances are firewalled by default.
[01:39:46] <scfc_de>	 andrewbogott: I think cajoel is gone, but today's/yesterday's discussion on -operations was that he wants to set up NAT and use ferm for that.
[06:55:04] <Ryan_Lane>	 andrewbogott, Coren: virt11.pmtpa.wmnet has 100% full on /var/lib/nova/instances
[06:55:29] <Ryan_Lane>	 instances are failing there
[06:56:38] <andrewbogott>	 Hm, I take it the scheduler isn't smart enough to not send new instances there :(
[06:56:45] <Ryan_Lane>	 it is
[06:56:58] <Ryan_Lane>	 it was at 98% previously
[06:57:03] <Ryan_Lane>	 but instances are just using space
[06:57:10] <Ryan_Lane>	 this one, for instance: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000621
[06:57:29] <Ryan_Lane>	 Coren: this pretty directly affect you, based on that instance ;)
[06:57:34] <Ryan_Lane>	 *affects
[06:57:47] <Ryan_Lane>	 that is eating 170GB of disk space on that host
[06:57:54] <andrewbogott>	 oh, so instance storage is overprovisioned you mean
[06:58:38] <Ryan_Lane>	 yeah, we'd underprovision everything else otherwise
[06:58:49] <Ryan_Lane>	 this is eating 50G: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000040d
[06:59:39] <Ryan_Lane>	 300G eaten by _base
[07:00:04] <andrewbogott>	 normally I would pester whoever created ve-roundtrip, but… it's you :)
[07:00:43] <Ryan_Lane>	 it's not :)
[07:01:08] <andrewbogott>	 "instancecreator_username=Ryan Lane"
[07:01:11] <Ryan_Lane>	 yeah
[07:01:19] <Ryan_Lane>	 I don't see how that's accurate :D
[07:01:29] <yuvipanda>	 yeah, has no underscore
[07:01:35] <yuvipanda>	 clearly  a plant
[07:02:19] <Ryan_Lane>	 I'm going to cold-migrate that instance to virt10
[07:02:39] <andrewbogott>	 yuvipanda, you don't know who's the admin of that box do you?
[07:02:50] <andrewbogott>	 Ryan_Lane: works for me
[07:03:06] <yuvipanda>	 no, sadly, I don't. But I think gwicke_away would, if he isn't the admin himself
[07:03:34] <Ryan_Lane>	 it's gwicke_away
[07:03:44] <Ryan_Lane>	 the tools db needs to be moved badly
[07:03:48] <Ryan_Lane>	 to real hardware
[07:04:07] <Ryan_Lane>	 and we need to fix the issue with ephemeral base images bring raw rather than being qcow2
[07:04:29] <Ryan_Lane>	 and we need to fix it upstream. I put a hack into the puppet repo to replace the code that does it
[07:04:50] <Ryan_Lane>	 but it doesn't seem to be working properly
[07:04:56] <andrewbogott>	 Yeah, I thought you already fixed new images so they started out minimal...
[07:04:57] <andrewbogott>	 oh :(
[07:05:35] <Ryan_Lane>	 a bunch of hosts are about to hit 100%, btw
[07:06:00] <andrewbogott>	 Yeah -- I'm not sure what to do about that other than nag people to clean things up...
[07:06:18] <andrewbogott>	 Juggling only helps if there's space to juggle to
[07:06:23] <Ryan_Lane>	 you can cold migrate instances from the full ones to the less full ones
[07:06:27] <Ryan_Lane>	 virt12 is at 30%
[07:06:31] <Ryan_Lane>	 virt10 is at 70
[07:06:42] <Ryan_Lane>	 virt7 is at 75
[07:07:12] <Ryan_Lane>	 it's obvious we need more hardware at this point :)
[07:07:25] <Ryan_Lane>	 I was hoping we'd just move to eqiad and increase the number of hosts there
[07:07:35] <Ryan_Lane>	 and then ship the tampa hosts to eqiad
[07:07:54] <andrewbogott>	 yeah, me too.  there are a lot more hosts in eqiad.
[07:08:00] <andrewbogott>	 Now it's a race...
[07:08:28] <Ryan_Lane>	 this is what I just ran, btw: root@virt0:~# ./cold-migrate ve-roundtrip2 virt10 visualeditor
[07:08:46] <Ryan_Lane>	 after it finishes I'm going to manually delete that instance's directory from virt11
[07:09:15] <Ryan_Lane>	 past freeing up a little space to keep things running, I don't have time to do any more juggling tonight
[07:09:59] <Ryan_Lane>	 I was writing code and my instance just stopped responding :(
[07:09:59] <Ryan_Lane>	 heh
[07:11:03] <andrewbogott>	 Is bastion1 on virt11?
[07:11:30] <Ryan_Lane>	 dunno
[07:11:43] <Ryan_Lane>	 if you can't ssh to it, then likely :)
[07:12:06] <andrewbogott>	 It's working now, just was unhappy earlier.
[07:12:47] <andrewbogott>	 Hm, nope, it's on 10
[07:24:05] <Ryan_Lane>	 andrewbogott: ok, virt11 has some freed up space
[07:24:21] <Ryan_Lane>	 I think it's necessary to reboot all virt11 instances
[07:28:36] <andrewbogott>	 Is there a salt wildcard to match virt11?
[07:28:47] <andrewbogott>	 Or should I just do 'em by hand?
[07:29:11] <Ryan_Lane>	 you need to reboot them via openstack
[07:29:16] <andrewbogott>	 'k
[07:29:30] <Ryan_Lane>	 which unfortunately isn't super easy
[07:30:23] <Ryan_Lane>	 one thing I do is get a list of instances from the database
[07:30:46] <Ryan_Lane>	 well, both instance and project
[07:31:05] <andrewbogott>	 'nova list' can't do that?
[07:31:12] <Ryan_Lane>	 nope. nova list is per tenant
[07:33:41] <Ryan_Lane>	 ok. I sent outage reports to ops list and wikitech-l
[07:33:58] <Ryan_Lane>	 I gotta run
[07:34:44] <andrewbogott>	 197 instances?  That doesn't seem right...
[07:34:51] <andrewbogott>	 Oh, I bet I'm getting deleted ones
[07:37:01] <andrewbogott>	 Ah, 47, much better.
[07:47:46] <andrewbogott>	 Ryan_Lane: um… 'nova --project bastion reboot bastion2' isn't working on virt11… would you expect it to?
[08:13:31] <andrewbogott>	 !log bastion rebooted bastion2 due to virt11 issues
[08:13:33] <labs-morebots>	 Logged the message, dummy
[08:16:04] <andrewbogott>	 !log search rebooted solr-wlm2 due to virt11 storage failure
[08:16:05] <labs-morebots>	 Logged the message, dummy
[08:16:50] <andrewbogott>	 !log orgcharts rebooted orgchart
[08:16:51] <labs-morebots>	 Logged the message, dummy
[08:18:29] <andrewbogott>	 !log dns rebooted new-ns0
[08:18:30] <labs-morebots>	 Logged the message, dummy
[08:18:34] <andrewbogott>	 !log nagios rebooted nagios-dev
[08:18:36] <labs-morebots>	 Logged the message, dummy
[08:20:14] <andrewbogott>	 !log nginx rebooted nginx-devunt
[08:20:15] <labs-morebots>	 Logged the message, dummy
[08:21:19] <Ryan_Lane>	 you have to run it on virt0
[08:22:24] <andrewbogott>	 Yep, I'm well into the tedious part now :)
[08:23:00] <andrewbogott>	 !log account-creation-assistance rebooted accounts-puppetmaster
[08:23:01] <labs-morebots>	 Logged the message, dummy
[08:26:34] <andrewbogott>	 !log opensim rebooted opensim-grid1
[08:26:35] <labs-morebots>	 Logged the message, dummy
[08:32:25] <andrewbogott>	 !log tools rebooted tools-db
[08:32:27] <labs-morebots>	 Logged the message, dummy
[08:42:26] <andrewbogott>	 !log metavidwiki rebooted metavidwiki
[08:42:28] <labs-morebots>	 Logged the message, dummy
[08:43:34] <andrewbogott>	 !log mobile rebooted mobile-varnish
[08:43:35] <labs-morebots>	 Logged the message, dummy
[08:48:02] <andrewbogott>	 !log math rebooted latexml-test
[08:48:04] <labs-morebots>	 Logged the message, dummy
[08:48:57] <andrewbogott>	 !log deployment-prep rebooted deployment-cache-text1
[08:48:58] <labs-morebots>	 Logged the message, dummy
[08:52:27] <andrewbogott>	 !log snuggle rebooted snuggle-large
[08:52:29] <labs-morebots>	 Logged the message, dummy
[08:53:27] <andrewbogott>	 !log sartoris rebooted sartoris-target4
[08:53:28] <labs-morebots>	 Logged the message, dummy
[08:54:53] <andrewbogott>	 !log tools rebooted tools-exec-09
[08:54:54] <labs-morebots>	 Logged the message, dummy
[08:57:39] <andrewbogott>	 !log snuggle rebooted snuggle-redis
[08:57:41] <labs-morebots>	 Logged the message, dummy
[08:57:46] <andrewbogott>	 !log scrumbugz rebooted scrumbugz
[08:57:48] <labs-morebots>	 Logged the message, dummy
[08:58:51] <andrewbogott>	 sheesh
[09:31:55] <hashar>	 :D
[12:41:35] <zhuyifei1999>	 @notify Coren
[12:41:35] <wm-bot>	 This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.)
[12:42:16] <mutante>	 labs: a/s/l?
[12:42:22] <mutante>	 eh, i mean, how's the migration status:)
[13:21:21] <andrewbogott>	 mutante: migration delayed by mhoover having been too sick to work last week.
[13:21:25] <andrewbogott>	 But some bits are coming together.
[13:22:50] <mutante>	 andrewbogott: cool! thx. i was asking because there is a change by Gage pending review
[13:22:55] <mutante>	 that is about monitoring virt0
[13:22:55] <hashar>	 good morning andrew
[13:22:59] <mutante>	 and adding NRPE etc
[13:23:13] <mutante>	 and i though, how much sense does it make on virt0, vs the replacement right now
[13:23:16] <mutante>	 thought
[13:24:06] <mutante>	 but on the other hand, he reacted to outage due to disk space issue
[13:24:58] <mutante>	 andrewbogott: i mean, of course not cool that mhoover is too sick to work :?
[13:27:11] <scfc_de>	 !log tools tools-login: rm -f /var/log/exim4/paniclog (OOM)
[13:27:12] <labs-morebots>	 Logged the message, Master
[13:28:25] <mutante>	 andrewbogott: https://gerrit.wikimedia.org/r/#/c/107424/
[13:35:37] <andrewbogott>	 mutante, what does installing nrpe on virt0 do?
[13:35:48] <andrewbogott>	 Does it monitor virt0, or does it turn virt0 into a monitoring host?
[13:36:49] <scfc_de>	 !log tools After reboot of tools-exec-09, all continuous jobs were successfully restarted ("Rr"); task jobs (1974113, 2188472) failed ("19  : before writing exit_status")
[13:36:50] <labs-morebots>	 Logged the message, Master
[13:38:26] <hashar>	 andrewbogott: just get the nagios nrpe daemon installed there
[13:38:32] <hashar>	 I am not sure whether that includes ferm rules though
[13:38:37] <mutante>	 andrewbogott: install /etc/init.d/nagios-nrpe-server
[13:39:03] <mutante>	 andrewbogott: a service that listens and accepts commands from icinga to then execute checks locally
[13:39:14] <mutante>	 like check_disk , check_dpkg etc
[13:39:19] <hashar>	 there is a ferm::rule statement, but apparently that does not install ferm itself nor enable the ferm service
[13:39:26] <mutante>	 that can't be checked from remote, though they are in base module monitoring
[13:39:32] <mutante>	 so expected on everything pretty much
[13:39:46] <mutante>	 or there is no disk space monitoring
[13:40:00] <mutante>	 but also what hashar said, if the IP is public it needs ferm
[13:40:17] <mutante>	 though it's not as dangerous as executing random commands
[13:40:30] <mutante>	 we just hardcode the check commands and don't pass args over the net
[13:40:42] <mutante>	 so what you could do is make a check look OK when it's not ..
[13:40:43] <hashar>	 mutante: well that would be nice
[13:40:57] <hashar>	 virt0 most probably have Augeas managed iptables rules
[13:41:04] <hashar>	 so one would have to whitelist nrpe port 
[13:41:19] <mutante>	 hashar: it should be that way, that's why we dont make generic NRPE commands
[13:41:28] <mutante>	 but have a separate for each process check etc
[13:42:01] <mutante>	 there is also http://blog.medin.name/blog/2012/12/02/securing-nrpe-with-certificate-based-authentication/
[13:42:54] <hashar>	 yeah might be overkill for us though
[13:43:03] <hashar>	 since the checks are over a "private" network
[13:43:57] <andrewbogott>	 ok, I'm a bit lost, but… sounds like that patch isn't ready to merge… would one of you add comments in gerrit?
[13:44:25] * hashar  ducks

[13:44:52] <mutante>	 heh, i already pasted some
[13:45:01] <mutante>	 copy/paste from IRC earlier
[13:45:36] <mutante>	 hashar: dont_blame_nrpe=0
[13:45:38] <mutante>	 :)
[13:45:44] <mutante>	 it's a real option
[13:45:50] <mutante>	 # Values: 0=do not allow arguments, 1=allow command arguments
[13:45:54] <mutante>	 # *** ENABLING THIS OPTION IS A SECURITY RISK! *** 
[13:45:59] <scfc_de>	 I'm looking into using ferm for NAT on Tools instances; what's the best way to change that default policy?
[13:46:49] <mutante>	 andrewbogott: don't worry then, just for the new server
[13:47:06] <mutante>	 andrewbogott: only drawback is not having disk_space monitoring on virt0
[13:47:14] <mutante>	 but it didnt exist before 
[13:47:44] <mutante>	 or it makes gage dive into ferm, which is also good
[14:53:06] <DanielK_WMDE_>	 i'm confused about jsub
[14:53:13] <DanielK_WMDE_>	 i'm submitting a job once per hour
[14:53:28] <DanielK_WMDE_>	 but the last time it was executed as at 3:17am (UTC).
[14:53:40] <scfc_de>	 DanielK_WMDE_: What does qstat say?
[14:53:50] <DanielK_WMDE_>	 nothing
[14:54:28] <DanielK_WMDE_>	 it's in local-potd-feed's crontab. can you see that?
[14:54:53] <scfc_de>	 Moment.
[14:55:44] <DanielK_WMDE_>	 if i submit the job manually, it seems to work
[14:55:47] * DanielK_WMDE_  tries again

[14:56:47] <DanielK_WMDE_>	 yea, works nicely, within less than a minute
[14:58:18] <scfc_de>	 /var/lib/gridengine/default/common/accounting shows three jobs having run yesterday afternoon and one this morning (3:17Z).
[14:58:44] <scfc_de>	 The crontab only is run at 3:17?!
[14:58:55] <scfc_de>	 "17      3       *       *       *    ..."
[14:59:16] <scfc_de>	 For hourly, that should be "17 * * * * ..."
[15:04:30] <DanielK_WMDE_>	 scfc_de: uh... o_O 
[15:04:51] <scfc_de`>	 :-)
[15:04:52] <DanielK_WMDE_>	 scfc_de: who put that 3 there?! I swear it wasn't me ;)
[15:05:04] <DanielK_WMDE_>	 scfc_de`: sorry for bothering you, i'll now go and hide in shame...
[15:05:22] <scfc_de`>	 I've looked at the crontab first and I didn't notice it either :-).
[15:48:29] <DanielK_WMDE_>	 hm, the toolserver migration guide sais i should put the rewrite rules into the .htaccess in my home. really, in my home, not my ~/public_html?
[15:49:04] <DanielK_WMDE_>	 (i guess this is a question for the toolserver channel...)
[21:14:16] <manybubbles>	 !log deployment-prep finished updating to elasticsearch 0.90.10
[21:14:17] <labs-morebots>	 Logged the message, Master
[21:14:24] <drdee>	 question for you all: how can i detect that a user of a labs instance is using shared storage for $HOME
[21:23:18] <scfc_de>	 drdee: Never tried, but in a shell, either parse "df $HOME", or traverse up $HOME, check if "mountpoint -q" succeeds and then look that up via mount?
[21:23:37] <drdee>	 ty, let me try that
[21:25:25] <drdee>	 scfc_de: output is:
[21:25:27] <drdee>	 Filesystem                                 1K-blocks    Used Available Use% Mounted on
[21:25:28] <drdee>	 projectstorage.pmtpa.wmnet:/analytics-home  52428800 6831616  45597184  14% /home
[21:25:36] <drdee>	 so that means it's shared?
[21:26:33] <scfc_de>	 I *think* all devices with ":" should be remote; how rock solid does your test need to be?
[21:26:53] <drdee>	 100% :)
[21:28:44] <scfc_de>	 http://unix.stackexchange.com/questions/72223/check-if-folder-is-a-mounted-remote-filesystem uses "df" as well, but only three upvotes.
[22:37:21] <dschwen>	 So, I'm trying to puppetize the instances in my labs project and have no idea what I'm doing.
[22:37:22] <dschwen>	 I
[22:37:51] <dschwen>	 I've found quite a bit of docs on how production uses puppet, but no puppet for dummies on labs
[22:42:08] <dschwen>	 I've heard about "selfhosting", but cannot find help on it
[22:45:33] <dschwen>	 oh, found it https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster
[22:58:31] <Coren>	 drdee: A device name with a ":" in it is guaranteed to be remote, afaik, but the absence of a ":" doesn't guarantee the converse (because bind mounts, etc)
[23:13:03] <drdee>	 Coren: ty
[23:56:19] <zhuyifei1999>	 Coren: ping