[00:04:28] Hello! I find myself unable to log in today (either to tools-login.wmflabs.org, where I'm used to logging in, or bastion.wmflabs.org, which appears to be its replacement?) [00:07:22] I'm hoping to update the data for my appliication today, so any assistance with the new login procedures would be appreciated. [00:08:31] (Ah, I assume the "eqiad outage underway" message is the explanation for now? Will try again later, then.) [00:12:27] OK, I think the outage is over. [00:12:32] chippy: try now, if you're still here? [00:12:55] trying [00:13:11] andrewbogott, unable to ssh into bastion :( [00:13:24] really? Which one? [00:13:37] bastion.wmflabs.org [00:14:06] says Permission denied (publickey). [00:14:12] chippy: now? [00:14:36] andrewbogott, in. for both. Thanks [00:14:45] great. [00:14:50] andrewbogott, also able to sudo okay [00:14:58] great! [00:15:18] i can't ssh into wikidata-test instance [00:15:19] woo! [00:15:24] can ssh into wdjenkins [00:15:32] (wikidata-dev) [00:18:53] aude: did wikidata-test work for you previously? [00:19:22] It's a migrated self-hosted puppet instance right? [00:21:18] …aude? [00:24:57] it worked [00:25:05] worked an hour or so ago [00:25:37] self hosted, yes [00:26:21] ok, looking [00:26:26] Migrated from pmtpa, or fresh in eqiad? [00:26:45] fresh in eqiad [00:28:01] did puppet run, before? [00:28:31] yes [00:28:39] all puppetized [00:28:44] 'k [00:29:20] mind if I reboot it? [00:29:26] the master died [00:29:27] go ahead [00:29:28] let me see why [00:30:04] I don't understand why this puppet failure would break logins [00:30:21] it shouldn't [00:30:26] permission denied [00:30:31] ok, I started the master [00:30:35] puppet should recover now [00:30:37] let's see auth [00:30:48] ok [00:31:22] try again [00:31:28] or... wait [00:31:47] oh for crying out loud [00:34:28] aude: should be fixed now [00:35:35] paravoid: works for me. Do we need to do whatever you did on other self-hosted boxes? [00:35:39] paravoid: it's good [00:35:40] thanks! [00:35:45] no [00:35:49] andrewbogott: no, bad sed [00:36:02] ok. [00:36:07] thanks for fixing [00:39:26] chippy, aude, does this close out the 'maps' project? Or are you still salvaging data from pmtpa? [00:40:01] andrewbogott, okay, files got and almost all copied across. Services appear to be running. double checking [00:40:29] whatever chippy says [00:40:49] :) [00:41:09] Krinkle: do you still want to do that file transfer tonight? Or are you on to other things/thwarted by the outage? [00:41:25] yeah the only thing im seeing is a very small permission thing but that's something i will fix [00:41:41] so I think I've copied all the data [00:41:45] double checking now [00:44:46] andrewbogott, yes I believe I'm all set now [00:44:58] great -- thank you! [00:45:56] hm, seven left [00:46:23] I can fix things on my end, one thing that may need looking at is getting the maps-warper.instance-proxy.wmflabs.org running. But I would imagine this magic would work when the migrations complete? [00:46:56] instance-proxy is getting phased out, but you should be able to set up a dedicated hostname using the 'manage web proxies' link in the sidebar [00:47:11] okay thank you :) [00:50:11] andrewbogott, done it. That is awesome. Thanks [00:51:12] np [00:54:19] Krinkle: I'll try to keep a window open… beep me if you're still around. [01:03:38] andrewbogott: Yeah, got pulled away. Gonna have to do this tomorrow.. Cutting it close. [01:03:55] tomorrow's ok, just let me know when you're ready. [01:05:01] andrewbogott: Can you give a prospect of the overal situation in tampa? Is it of the magnitude that like the instances host masters are going to be shutdown or would be there be a chance (without changing labs plans) to have it running there a little longer. [01:06:36] Well, I want to do a soft shutdown /before/ we start pulling apart the servers so there's a few days for stragglers to notice that things are getting shut down. [01:06:56] But you may get an extra day of slack since I can't shut down anything until beta is finished. [03:18:43] ok, let's build a trusty image! :D [03:19:01] andrewbogott: not sure if you're around, but I'll describe what I'm doing as I do it [03:19:35] I'm going to start off by adding the default trusty image to glance, so that I can use a trusty image to build the trusty image [03:19:59] this will use the cloud-init script that gets injected to bootstrap the system so that I can log in [03:26:21] ugh. mediawiki on wikitech is so old [03:26:38] we really need to get rid of SMW. I don't know how we're going to upgrade it now that composer is required [03:27:32] I don't think you can even include it without composer now [03:27:46] what a fucking idiotic move on their part [03:29:39] oh, a trusty image is already there [03:29:43] and an instance it booted with it [03:30:47] ugh. right. puppet 3 [05:17:01] Coren: are you still around ... https://bugzilla.wikimedia.org/show_bug.cgi?id=63152 seen this one ? [06:29:14] andrewbogott_afk, on toro.eqiad.wmflabs, I am getting an error when puppet tries to mount /data/project: [06:29:15] https://dpaste.de/4g8v [06:29:24] I will continue debugging this tomorrow, but I wanted to let you know. [06:29:59] Reproducible by running sudo puppetd -tv [07:51:49] !ping [07:51:49] !pong [08:42:08] YuviPanda are you aware of bug https://bugzilla.wikimedia.org/show_bug.cgi?id=63152 [08:42:18] * YuviPanda peeks [08:42:31] it is preventing many services from functioning [08:42:33] andrewbogott_afk: Coren: I;m unable to log into the existing cvn instances in tampa (cvn-app2.pmtpa.wmflabs). Everything was fine a few days ago. [08:42:37] Now it suddenly aborts connections and claims to be unable to create /home/krinkle (why would it need to create that? I've logged into that instance 100s of times) [08:43:18] Krinkle het kan zijn dat je last heb van diezelfde bug [08:43:21] GerardM-: I don't think I can do anything :( needs andrewbogott_afk or Coren [08:43:42] are they both in the same time zone ? [08:43:57] GerardM-: I think so :) [08:43:58] err [08:43:59] :( [10:48:13] !ping [10:48:14] !pong [10:48:16] ok [10:54:21] !log deployment-prep Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad [10:54:24] Logged the message, Master [11:10:58] !log deployment-prep deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad [11:11:01] Logged the message, Master [11:17:38] YuviPanda: Just because the bot replied doesn't mean you're actually here [11:17:41] * Reedy glares at YuviPanda [11:18:40] ok. so, looking at moving the "awb" project from toolserver to labs [11:18:52] couple of web php scripts with a mysql backend [11:18:59] used for svn snapshot/tarballs [11:19:10] Should this be a new project, or just use tools? [11:19:46] Then just create a service group? [11:29:34] !log deployment-prep MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console [11:29:37] Logged the message, Master [11:39:01] out for lunch [12:54:22] !log deployment-pref Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts [12:54:22] deployment-pref is not a valid project. [12:54:30] !log deployment-prep Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts [12:54:33] Logged the message, Master [12:55:12] !log deployment-prep mediawiki l10n cache being rebuild!!! [12:55:15] Logged the message, Master [13:26:08] Krinkle|detached: Might be gluster being ill. Again. [13:26:35] Krinkle|detached: Why do you need to connect to the tempa instances anyways? All the files from there have been copied over to eqiad. [13:43:07] manybubbles: hi did you get the ElasticSearch in eqiad beta warmed up? [13:43:18] hashar: should be ready! [13:43:53] manybubbles: you might be able to try out by tweaking your DNS hosts entry to point to eqiad [13:44:06] I have been using http://paste.openstack.org/show/74443/ [13:44:30] ah [13:53:12] now I got to debug out why ferm iptables rules are not applied [13:53:14] goooood times [13:54:35] Coren: helllooo GlusterFS /home is broken on deployment-prep project in pmtpa :-/ [13:54:44] still got to access the instances there :/ [13:54:58] Bleh. [13:55:06] I got a permission denied [13:55:11] which is usually because home is dead [13:57:34] hashar, Krinkle|detached, superm401, there was an ldap outage yesterday which caused gluster to lose its marbles. I'll catch up... [13:58:01] !log deployment-prep rebased puppetmaster git repository, reapplied ottomata live hacks. [13:58:04] Logged the message, Master [13:58:08] andrewbogott: thanks :-] [13:58:13] andrewbogott: I'm failing the gluster volume start deployment-prep-home force [13:58:33] If fails instantly with "operation failed" [13:59:05] ooo, will unapply those hashar, thanks [13:59:26] hashar, I thought deployment-prep used nfs? [13:59:30] Is it some of each? [13:59:43] Gluster has clearly decided to be a pain until it dies. [13:59:44] ottomata: I rebased the repo and deleted your local 'otto' branch [13:59:52] Hey, wait, andrewbogott is right. That's NFS! [13:59:58] ottomata: if you need hack on the puppet master, just propose a change in Gerrit and cherry pick it :-] [14:00:20] andrewbogott: Coren: oh men sorry. Yeah deployment-prep uses NFS for /home . Sorry :-( [14:00:31] D'oh. :-) [14:01:44] hashar: What instance is giving you issues? [14:01:52] naw, i'm done hashar, thanks, its really hard to do that when I have know idea what the problems are [14:02:09] i was using that to debug, and meant to reset my changes when I was done [14:02:15] forgot (of course) anyway, fixed now [14:03:46] ottomata: thanks . I did some further cleanup . Looks good now. [14:04:00] Coren: deployment-bastion.pmtpa.wmflabs and deployment-apache32.pmtpa.wmflabs [14:04:17] Coren: but most probably any instance on pmtpa :( [14:04:26] hashar: seen my comment in -operations ? [14:04:44] matanya: replying there [14:04:51] thanks [14:06:13] hashar: Nothing wrong with /home there. Lemme see what's up in the logs. [14:06:42] Aha. The ssh keys volume is dead. I.e.: nothing in pmtpa could work atm. [14:07:37] and I thought ssh relied on pam_ldap or something to grab the user public key [14:17:34] hashar: Have you tested https://gerrit.wikimedia.org/r/#/c/119256/ on labs? Shall I merge it? [14:17:52] ah [14:18:08] the relevant diff is https://gerrit.wikimedia.org/r/#/c/119256/4..5/manifests/misc/logging.pp [14:18:19] looking on the puppet master [14:18:43] It looks right, I just want to verify that you've tried it beforehand [14:18:57] I have tested it [14:18:59] testing now [14:21:07] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter after at /etc/puppet/manifests/misc/logging.pp:49 on node i-0000010b.eqiad.wmflabs [14:21:07] :-( [14:21:33] easy fix luckily [14:26:13] puppet doesn't like the 'after => ' parameter in the file {} statement grr [14:26:58] err: Failed to apply catalog: 'mkdir -p /data/project/syslog' is not qualified and no path was specified. Please qualify the command or specify a path. [14:27:51] hashar: /bin/mkdir [14:28:21] andrewbogott: was the downtime of wikitech an april's fool joke ? [14:28:33] matanya: no [14:28:48] ok, so just a nice date then [14:29:07] Good point, though, I guess I should've picked monday :) [14:38:22] sorry net dropped [14:41:14] the syslog-ng patches works for me on deployment-bastion.eqiad.wmflabs https://gerrit.wikimedia.org/r/#/c/119256/ https://gerrit.wikimedia.org/r/#/c/119257/ [14:41:20] for prod that would impact nfs1 and nfs2 [14:41:56] hashar: ok! I will merge... [14:42:36] could use some help with some puppet class not being realized on my instances :/ [14:45:10] hashar, what are you seeing? [14:45:43] some beta instances need iptables rules to rewrite the public IP to the instance private IP equivalent [14:45:52] that is because you can't access the NATed public IP from inside labs [14:46:37] so I got a thin role class role::beta::natfix which invokes the class beta::natfix which create_resources() the ferm rules I need [14:46:40] the definition is in modules/beta/manifests/natfix.pp [14:47:03] apparently the role class is applied properly (the salt grains patch introduced yesterday does add the role::beta::natfix grain on the instance) [14:47:10] but the beta::natfix is not applied [14:47:16] ferm package is not even installed :/ [14:48:04] hashar, what's an example of an instance that applies that role? [14:48:11] or, I mean, tries to apply that role :) [14:48:11] ah of course [14:48:16] deployment-parsoid04.eqiad.wmflabs [14:49:07] where's the logic that would be installing ferm? [14:49:17] modules/beta/manifests/natfix.pp [14:49:25] which is a hash map invoked by create_resources() [14:49:29] This stuff is all merged, right? Or do you have a different version in progess? [14:49:34] beta::natdestrewrite [14:49:42] it is all merged and working on pmtpa [14:49:57] the ferm::rules are realized with vim modules/beta/manifests/natdestrewrite.pp [14:50:08] which include base::firewall [14:50:25] gripping for firewall in /var/lib/puppet/state/* yields nothing :-/ [14:52:17] maybe the puppetmaster does not recognize create_resources() and bails out [14:52:33] Yeah, I'm not very familiar with create_resource but I've seen it used elsewhere... [14:53:17] that is very powerfull [14:53:34] apparently the include beta::natfix is never included :/ [14:54:07] oh my god [14:54:14] I think I know why [14:54:25] class role::beta::natfix { [14:54:25] include beta::natfix [14:54:25] } [14:54:37] the puppet scope might interpret beta::natfix as being role::beta::natfix [14:54:41] so it would include itself [14:54:47] oh, yeah, that's probably right [14:54:57] So I think include ::beta::natfix ? [14:55:01] trying with a :: prefix include ::beta::natfix [14:55:15] damn refactoring broke it [14:55:53] fixed! [14:58:22] !log deployment-prep fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} [14:58:25] Logged the message, Master [14:58:28] and that of course breaks ssh access [15:00:47] is remapping IPs strictly needed? Could it be done via hostnames instead? [15:01:20] that is done in the network definitions listing all bastion hosts [15:01:30] then ferm by default list of those hosts and allow ssh access from them [15:01:37] bastion-eqiad.wmflabs.org never got added to the list of bastion hosts [15:01:42] patch https://gerrit.wikimedia.org/r/121379 [15:01:54] I tweaked it when alexandros did the ferm patch [15:02:39] hashar: there are several other bastions, do you mind adding them all? [15:02:59] amend my patch? :-] [15:03:25] I would happily copy paste though [15:04:11] 10.68.16.68, 10.68.16.67, 10.68.16.66, 10.68.16.5 [15:04:13] thanks [15:07:34] Oh, FFS. [15:07:49] Hey, that's the sound of Coren fixing a bug! [15:08:08] andrewbogott: Yeah, so the issue is client-side caching after all; but my google-fu fails at figuring out how to turn it off. [15:08:29] andrewbogott: The clients don't even /hit/ the mountd once it decided it failed. [15:08:43] andrewbogott: Reboot does fix, otoh [15:08:56] So, clearly, there is some evil caching going on. [15:08:57] ok… superm401, mind if we reboot toro? [15:09:04] andrewbogott: I just did. [15:09:10] ok then :) [15:09:19] andrewbogott: And, unsurprisingly, it fixed things. [15:09:51] Yeah, that's a mixed blessing. [15:09:54] So now, I really need to find a way to prevent the attempt at the mount until we know it's there -- so I'll be implementing that ugly thing we discussed. [15:10:14] ok :) [15:10:18] andrewbogott: amended https://gerrit.wikimedia.org/r/#/c/121379/2/manifests/network.pp [15:10:23] with all four EQIAD bastions [15:10:33] Gotta go see dentist though, will be back in ~3h [15:11:05] possibly we could get the three public bastions to share the same ssh host key then get bastion-eqiad.wmflabs.org to round robin to all three IP [15:12:31] andrewbogott: and you need the previous patch https://gerrit.wikimedia.org/r/#/c/121378/1 [15:12:48] andrewbogott: which fix up the include beta::natfix , I have picked it on beta puppet master and that works [15:13:22] hashar: ok, merged [15:16:12] my hero [15:16:24] so beta in eqiad is almost competed now [15:16:37] I will craft a mail and find out whether zeljkof is available tomorrow to test it out [15:16:46] I guess we can do the switch on monday [15:16:58] which would delay pmtpa labs shutdown slightly [15:21:03] !log deployment-prep applying role::beta::natfix on deployment-bastion.eqiad [15:21:06] Logged the message, Master [15:21:45] hashar: didn't you said we will abandon that once move to eqiad? [15:21:51] matanya: this ? [15:22:23] natfix [15:22:34] na we need it [15:22:40] until labs supports DNS split horizon [15:23:06] aka have the DNS give back private instances IP instead of the public address whenever a DNS query is made from an instance [15:23:30] yeah, makes sense. i thought it was on the plan for eqiad [15:23:37] !log deployment-prep role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( [15:23:39] Logged the message, Master [15:26:38] matanya: ever migrated Augeas iptables rules to ferm ? [15:26:59] not yet, only iptables. maybe it is time to start [15:27:24] and i didn't find any puppet-lint RT tickket [15:28:54] yes, Coren and andrewbogott , it does work now :-) [15:37:36] hasteur: The "dns server" that's part of nova-network doesn't have support for split horizons. Then again, I'd really like to have a proper dns server in Labs eventually, so that might get fixed this way. [15:38:50] that is defintily a better way to solve it [15:48:23] hashar: Are you also migrating the 'integration' project? [15:49:09] andrewbogott: yeah [15:49:20] Need me to do anything? Or is it finished? [15:49:32] got patches pending for misc::package-builder [15:49:40] I haven't found out if you guys are using that class [15:49:56] it basically setup pbuilder/cowbuilders images under /var/cache/pbuilder and install all tools needed to build debian package [15:50:07] link? [15:50:18] /var/ on labs is only 2GB which is too small so I need to have the cache under /mnt/ ( which is lvm) [15:50:24] * hashar digs in his emails [15:51:01] I must not be marked as reviewer on them, I don't see anything in my gerrit list [15:51:19] Mandatory linting: [15:51:19] https://gerrit.wikimedia.org/r/#/c/120005/ [15:51:19] Fix an notify {} that was not using message => [15:51:19] https://gerrit.wikimedia.org/r/#/c/120008/ [15:51:19] Replace /var/cache/pbuilder with a variable ($pbuilder_root) [15:51:20] https://gerrit.wikimedia.org/r/#/c/120009/ [15:51:20] Wrap misc::pbuilder with role classes to vary the pbuilder_root: [15:51:21] https://gerrit.wikimedia.org/r/#/c/120013/ [15:52:01] andrewbogott: added you as a reviewer on all four changes [15:52:09] thanks [15:52:12] I made them very atomic to ease merge/deplkoy/Review (hopefully) [15:55:00] !log integration deleting integration-apache1 Was used as a proxy for other instances and as a dev box for integration.wikimedia.org/ . Freeing the public IP address while at it [15:55:02] Logged the message, Master [15:55:08] one less instance! [15:55:39] 208.80.153.222 pmtpa labs IP released. [16:07:00] petan: Two questions…. 1) need any assistance with migrating project 'nagios'? 2) interesting in working on labs ganglia at some point after the dust settles from the migration? [16:07:30] andrewbogott: sign me on #2 [16:07:45] matanya: great! [16:07:54] I set up a new instance but it doesn't quite work and I don't know why :) [16:07:56] andrewbogott: interested? or interesting? :P [16:08:03] interested [16:08:33] andrewbogott: yes I can help you with ganglia, regarding #1 I was in thought that damian was working on that [16:08:57] Damianz: are you? :P [16:12:36] hashar, I'm trying to do a before/after with the lint patch, and the 'before' puppet run is… still running. [16:15:08] andrewbogott: the package creates two cow builder images which is rather slow [16:15:13] basically download and instance Precise images [16:15:27] yeah, I'm going to have some breakfast and will check back after [16:35:45] Yay, puppet magically fixed itself overnight. Thanks to whoever fixed gluster. [16:39:34] superm401: do you need any help with the editor-engagement migration? [16:39:56] andrewbogott, don't think so. I'm working my way through recreating on eqiad right now. [16:40:08] great. Let me know if you run in to any trouble [16:42:14] andrewbogott: great that you can test on labs :] [16:42:32] alexandros is working on a script to easily compare catalogs between two sha1 [16:42:54] will most probably integrate it as a jenkins job that one can manually trigger [16:48:51] error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/mediawiki/extensions/MwEmbedSupport.git/info/refs [16:49:21] After: [16:49:22] git clone https://git.wikimedia.org/git/mediawiki/extensions/MwEmbedSupport.git [16:50:14] Using the gerrit URL works. Is that a known issue? [16:53:32] superm401: On Tools? Yes ... Let me look up the bug. [16:53:47] scfc_de, was on Labs, but not Tools Labs. [16:56:44] hashar: you saw matanya's comment on https://gerrit.wikimedia.org/r/#/c/120009/1/manifests/misc/package-builder.pp ? [16:57:20] superm401: Maybe the same issue? https://bugzilla.wikimedia.org/show_bug.cgi?id=62432 [16:59:01] scfc_de, thanks, commented. [17:00:13] andrewbogott: ah no sorry [17:00:49] having trouble logging in to a labs host -- logs me right out (after converting to local puppet master) [17:00:59] cajoel: hostname? [17:01:07] keys seem to be there, as I get a motd, but it's like the shell is /bin/false [17:01:13] andrewbogott: flow-localpuppet [17:01:59] what project? [17:02:07] netflow [17:02:09] @info [17:02:09] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [17:02:17] i-000002a3.eqiad.wmflabs [17:04:28] cajoel: mind if I reboot? [17:04:36] not at all -- I should have tried that.. [17:05:08] andrewbogott: also created a new instance, and it seems to take a long time to put my public key in place. [17:05:14] I'm not sure quite what is wrong, but there was something wrong with the nfs mounts. [17:05:24] interesting [17:05:25] So when it tried to create the homedir it failed and kicked you out. [17:05:28] A reboot should help. [17:05:39] yeah, seems better now. [17:05:45] was working yesterday -- maybe the mounts changed? [17:05:51] Coren is working on a more comprehensive fix for this. [17:05:54] sweet [17:05:56] thanks [17:10:01] hmm, i'm unable to log into bastion.wmflabs only able to log into bastion-eqiad [17:12:04] Me, too. ("Permission denied (publickey).") andrewbogott, did you close down pmtpa-bastion? [17:14:18] nah, it's just gluster still freaking out from yesterday's ldap outage [17:14:20] try now? [17:15:37] is there a way to re-order the images in instance creation so that Trusty 14.04 isn't the default selected image? [17:16:25] andrewbogott, yep, able to login to it now [17:16:36] cajoel: yes, that's a mistake, I thought that Ryan fixed it yesterday. [17:21:46] andrewbogott: I noticed region=eqiad in the url, region=pmtpa still works, undocumented emergency ability (instance creation only hidden in the UI?) Anyway, was just testing. Will delete again :) [17:22:07] Migrating now, hopefully will work [17:22:09] (cvn) [17:23:15] hello [17:23:50] I used to have a php_error.log in my tool home account [17:23:56] it's not there anymore [17:24:27] and my tool is not working properly: https://tools.wmflabs.org/ptools/ [17:24:35] is there something going on ? [17:25:51] pleclown: in the future I'd encourage you to subscribe to the labs-l list so that you can get notices of planned outages and other changes. [17:26:00] In the meantime, though, I believe Coren can help you with useful links. [17:26:14] andrewbogott: sorry got interrupted with Zuul getting wild. [17:26:21] andrewbogott: It's been a while, apparently [17:26:34] re package-builder having a scoping issue . I replied to matanya concern on https://gerrit.wikimedia.org/r/#/c/120009/1/manifests/misc/package-builder.pp [17:26:44] not sure what is the best way to fix it up :/ [17:26:56] can someone peek at this file on production /etc/ferm/conf.d/10_nrpe_5666 [17:27:21] in labs, I'm getting proto tcp dport 5666 { saddr $INTERNAL ACCEPT; } -- I expect that $INTERNAL is supposed to be replaced with something... [17:27:37] andrewbogott: I created 3 eqiad instances, the first and third one are up and running with all OK afaics. The second one seems to have /home missing. Should I wait or reboot or something else? [17:27:51] (cvn-app4 is OK, cvn-apache5 is missing /home, cvn-apache5 is OK) [17:27:57] Krinkle: wait and reboot is still the best approach. Coren is working on a better solution. [17:27:59] checked `df` [17:28:12] it has /project volume though, but not home. [17:28:21] Krinkle: I had to reboot a new instance I just created to get /home working (also check that you didn't start 14.04 instances) [17:28:23] the other two have both. I created all three of them just now. [17:28:29] yeah [17:30:00] cajoel: Playing around with ferm? $INTERNAL is a variable only defined in realm production, IIRC. [17:30:04] *ferm variable [17:30:12] ah -- that might explain it [17:30:21] yeah -- trying to get ferm working in labs [17:30:40] I spent some time trying to get ferm running in Labs, but gave up after a while. [17:31:22] wondering how I might work around that... [17:31:32] it looks like it's getting farther that that is used to. :) [17:32:34] INTERNAL is here.. [17:32:34] /var/lib/git/operations/puppet/modules/base/templates/firewall/defs.erb [17:35:09] class base::firewall { [17:35:10] ok [17:35:14] right -- been down this road before [17:35:33]