[00:00:16] gluster only actually has read-write shares [00:00:27] if you want read-only you need to enable gluster's NFS support [00:00:34] yuck [00:00:54] doesn't gluster/nfs have issues with the newest nfs and ldap around it freaking out on users 'missing' or something? [00:01:11] I reacall a cloud provider talking about an issue relating to that which forced them back an nfs version. [00:01:23] well, I'm definitely seeing issues with the global dumps nfs share [00:01:43] though I'm wondering if that was due to some upgrade issue [00:02:04] I'm thinking of wiping all of the data in that, and having it re-sync [00:02:51] Tbh my main issue with nfs apart from it sucking in performance and randomly crapping out is it's a huge gaping SPOF for the entire cluster access wise and could in theory take a load of stuff down. Which probably makes me paraniod but gluster feels safer for the current use, even if proxied over nfs to the endpoint. [00:07:13] hi, how do I sudo inside my instance, so I can install gcc and perl packages? [00:07:21] `sudo`? [00:08:31] I get reported to the authorities when I do taht [00:09:18] I'm using the labsconsole passwd (that's ldap, right?) [00:09:28] Ryan_Lane: Do you think it's acceptable to enfore 1 user to 1 db for mysql rather than permit free creation of anything (which makes cp hard)? [00:09:33] gribeco: what instance? [00:09:38] bots-salebot [00:09:47] 'Edit conflict:' oh thanks wiki [00:09:49] just thanks [00:09:56] "gribeco is not allowed to run sudo on bots-salebot. This incident will be reported." [00:11:35] 1sec [00:11:50] no problem, thanks =) [00:12:01] try now [00:12:14] we have a slightly restricted sudo policy on bots [00:12:23] seems I didn't add that instance to the general group [00:12:46] it's still prompting me for a passwd, end then returning the same error message [00:12:57] 1sec [00:13:23] now? [00:13:51] same thing [00:13:55] hmm [00:14:29] Says you can't type [00:14:41] ok now says not authorized [00:14:42] hmm [00:14:57] I just tried a bad password to be sure that was not the problem [00:15:09] Logout, back in and try again. [00:15:12] I get the normal "Sorry, try again." when I do that [00:15:15] ok [00:15:36] It should be updated and I just restarted nslcd so it should be asking opendj again... [00:16:24] Failed. [00:16:29] yup [00:16:30] Woot [00:16:58] Ryan_Lane: Any ideas why sudo ldap isn't playing ball after adding a host to the default sudo policy for a project? [00:17:15] I assume you're trying sudo ls looking at the log [00:17:23] yes [00:17:32] cache [00:17:34] probably [00:17:42] I think sudo-ldap also caches [00:17:46] bleh [00:17:50] I hate caches sometimes [00:18:37] gribeco: Are you running anything currently? [00:18:43] Doesn't look like it from the process list. [00:18:49] no, it's okay to restart the instance [00:18:57] Cool, try it in a few [00:19:08] ok [00:19:29] And here boys and girls is how to clear a cache with a sledge hammer and just hope it isn't welded to your floor boards already. [00:20:23] It's back up by the way. [00:20:51] nope still failed. [00:20:56] yeah [00:21:03] * Damianz wonders if creating a new policy works [00:21:33] Try now? [00:21:37] I don't have anything valuable inside the istance, it's okay to terminate/re-create if necessary [00:21:46] I'll be surprised if creating a new policy works... [00:22:08] still bad [00:22:42] Aside from waiting a while I'm out of ideas then [00:23:09] ok, let's give it a while [00:24:40] Damianz: it caches on the filesystem [00:25:12] Any idea where? It's pretty much impossible to strace the process as it exits too quick [00:25:22] sec [00:26:32] /var/lib/sudo/ < as it's empty, wth is the point of creating a dir for each user.. [00:27:03] dunno [00:28:07] hmm [00:28:11] gribeco: try now just for kicks? [00:28:58] Apparently -K should clear up user creds... no idea if that applies to rights as well. [00:28:58] * gribeco is still getting kicked [00:29:13] sec [00:29:44] * Damianz looks at Ryan's internet history and sees 30 google searches =D [00:29:58] is it failing immediately, or asking you to keep retry your password? [00:30:07] Damianz: actually, I'm just working on a bunch of things at the same time ;) [00:30:18] it is failing on the 1st attempt, after entering the correct passwd [00:30:21] ok [00:30:34] That's no fun [00:30:39] (it fails differently with a bad passwd) [00:30:49] Fails with sudo: pam_unix(sudo:auth): authentication failure; then sudo: gribeco : user NOT authorized on host ; [00:31:09] DENIED. [00:32:14] I enabled debugging [00:32:18] gribeco: try now [00:32:33] * Damianz imagins the computer holding up a sign with 'tits or GTFO' [00:32:37] ah [00:32:39] I see the problem [00:32:42] this is a new instance [00:32:50] It is [00:33:00] access rules on opendj? heh [00:33:00] OpenStackManager is adding the i- name into LDAP [00:33:16] the instance's host name is bots-salebot [00:33:16] not the name [00:34:02] hm [00:34:09] not terribly sure how to fix this [00:34:11] I don't really see why we can't kill i-* and use name as the hostname and name.region.wmflabs as the fqdn to save confusion [00:34:30] old instances are still using the i- names [00:34:38] I *might* be able to fix that [00:34:52] we could just change the hostname of the instance [00:34:58] ugh [00:35:13] not easily [00:35:21] it's set through dhcp [00:35:27] urgh [00:35:40] seriously, why would you set that though dhcp. it's not like you're going to change it [00:35:51] well, actually, in this situation, it's helpful [00:35:59] because we probably do want to rename all of the older instances [00:37:12] This probably explains half my issues with getting nagios/snmp traps to match up hostname wise on a sidenote :P That's a work in progress that I probably need to revise my pending change for... though maybe switching to fqdns would switch this too. [00:38:00] well, this does use fqdns [00:38:09] the hostname of the system needs to match what's in LDAP, though [00:38:30] ah [00:38:56] gribeco: until I get this worked out, what do you need installed? [00:39:14] stupid nova changing the damn host name every fucking release [00:39:15] a bunch of perl packages, some have external dependencies like gcc [00:39:25] ah. hm. [00:39:29] even easier [00:39:30] * Damianz wonders back to reading about symptoms and side effects while trying to remember what he was going to do in puppet [00:39:35] manual policy on the host [00:40:00] Ryan_Lane: Well at least they give you the tools to manage your ip ranges, I mean look on the bright side of user friendlyness. [00:40:37] gribeco: ok, you can sudo on there now [00:40:45] Damianz: :D [00:41:37] yay! [00:43:15] you know what would be sort of nice [00:43:20] project subgroups [00:43:31] which you could apply classes and sudo policies to [00:44:07] so you could have a subgroup of database, web, app and just add an instance to a group and bamn all classes/security groups/sudo polciies apply and magic ponies shit rainbows [00:44:52] you're asking for a unicorn :) [00:45:14] that really wouldn't be very easy to do [00:45:59] nope, you'd either have to path nova or write a load of supporting code ontop of it. Nice thought though for some things and saves typing. [00:46:14] I'd happily take a 'clone' option that lets me create a new instance with the same settings as an existing one though. [00:46:32] yes [00:46:37] that's planned, whenever I get time to do it [00:46:48] there's a difficulty with that too, though [00:46:57] puppt breaks builds at least [00:47:02] our puppet manifests make it pretty likely that an instance configured that way won't fully build [00:47:05] which btw I'd sorta like to remote that form [00:47:10] s/remote/remove/ [00:47:21] that's why I hid it in an "advanced" section, with a warning [00:47:48] I never noticed that heh [00:47:58] Is there a warning to say you can't change security groups after too? [00:48:31] no. probably should be [00:51:10] i am getting ssh debug output on my instance shell it seems [00:52:53] did you ssh with -vv [00:53:48] Damianz: duh, of course i did, i told andrew earlier to do that :p [00:54:03] no complaining then :P [00:55:46] Apologies for hijacking your thread ryan =\ [00:56:22] no worries. it's true [00:58:45] !log wikistats - upgrading mariadb packages (yeah:) [00:58:46] Logged the message, Master [01:03:35] grrrr wtf [01:03:45] Ryan_Lane: Don't suppose you're playing with ldap stuff right now? [01:03:50] I'm not [01:03:51] why? [01:04:09] bots-sql2 won't let me login to ssh, 1 and 3 work fine [01:04:10] * Damianz frown [01:05:36] Weird and annoying, I wonder if it's doing the strange nfs thing again [01:13:39] hmm [01:13:51] what's the line betwean module and just banging it in a role class [01:14:13] it's allmost not worth even venturing into a module as these classes are so tiny but then role seems the wrong place [01:18:04] well, the classes will get larger [01:18:10] and we want everything to be a module [01:19:39] Even stuff that's just including other classes? [01:39:55] Damianz: including whicb classes? [01:40:03] everything that isn't a role class should be a module [01:40:14] role classes should only include classes from modules [01:40:28] with variables to modify behavior [01:40:49] node entries should include role classes. and set variables that can modify them [01:44:07] So what I'm doing really should then be in a role class [01:44:08] meh [01:44:13] it's going in a module for now [02:40:34] what's a rouge PhpMyAdmin install? vs non-rogue [02:43:02] You've never seen a red powdered PhpMyAdmin install? It looks like someone tried to circumcise you with bolt cutters after about a day due to all the exploits that get collected in your temp dir. [02:43:31] lol [02:43:32] ok [02:44:29] a recent version of phpMyAdmin installed intentionally w/ a reasonably-good password is OK, no? [02:48:57] * GChriss checks /tmp [02:49:30] Even an up to date version I'd suggest putting in a non-standard path behind http auth.. it's rather common for it to get exploited sadly [03:08:16] meh so my classes totally don't work, I'll revise these tomorrow I guess [03:14:51] Ryan_Lane: So I figured out the way not to do it in puppet :P [03:57:48] Seems like Gluster has its own conversion rates or something. 135MB on local storage = 163GB on Gluster, wtf?! [09:43:56] paravoid: hi :) I got an instance not responding… i--0000049c / integration-wikibugs has been assigned 10.4.0.249 but is unreachable from bastion1. 10.4.0.54 reports the destination as being unreacheable. [09:44:14] (in pmtpa) [09:45:11] maybe it is not acquiring an IP from the DHCP server (pool exhausted?) or some routing is wrong [09:45:27] other instances are reachable (such as 10.4.0.231) [10:21:55] !log bots giving rights to Mardetanha for the project [10:21:56] Logged the message, Master [10:22:13] Good you have your slaves :)) [10:25:51] 10/03/2012 - 10:25:51 - Created a home directory for mardetanha in project(s): bots [10:30:43] 10/03/2012 - 10:30:43 - User mardetanha may have been modified in LDAP or locally, updating key in project(s): bots [10:31:06] !log centralauth Created instance centralauth-mysql [10:31:07] Logged the message, Master [10:31:18] !log centralauth Installed mysql-server [10:31:19] Logged the message, Master [10:32:47] !log centralauth Install memcached on centralauth-frontend [10:32:48] Logged the message, Master [10:33:18] !log centralauth Installed MediaWiki for the main wiki [10:33:20] Logged the message, Master [10:39:42] petan: is there a reason gcc is not installed on bots-3? [10:42:18] I'm going to go ahead and install it [10:42:57] !log bots installed gcc on bots-3 [10:42:58] Logged the message, Master [11:46:24] Damianz who removed labs nfs? [11:46:27] I mean bots [11:46:37] there used to be nfs server [11:46:42] how does it work now? [11:46:50] documentation plis [13:32:55] @infobot-detail ping [13:32:55] Info for ping: this key was created at N/A by N/A, this key was displayed 25 time(s), last time at 10/2/2012 12:39:22 PM (1.00:53:33.1341650 ago) [13:33:41] @infobot-detail petan [13:33:41] Info for petan: this key was created at N/A by N/A, this key was displayed 0 time(s), last time at N/A [13:33:50] :o [13:34:42] !ping [13:34:42] pong [13:39:29] Change on 12mediawiki a page Developer access was modified, changed by Das Schäfchen link https://www.mediawiki.org/w/index.php?diff=590153 edit summary: [13:45:15] quit [13:45:22] damn [14:20:13] hi Damianz. I'd like to obtain access to the bots project on WMFLabs, can you help me please ? [14:20:47] 10/03/2012 - 14:20:47 - Created a home directory for quentinv57 in project(s): bastion [14:20:58] 10/03/2012 - 14:20:58 - Creating a home directory for quentinv57 at /export/keys/quentinv57 [14:21:32] Quentinv57 I can [14:21:41] ah, thank you petan [14:21:54] I'll be really happy if you could do it [14:22:05] !log bots adding Quentinv57 [14:22:07] Logged the message, Master [14:22:46] done [14:22:59] wait a minute for scripts to finish [14:23:08] thanks petan, but I'll need some help [14:23:14] ok [14:23:17] I don't know what's my LDAP password [14:23:25] same as wiki [14:23:29] and I don't know how to use the bots projects (ie if there are rules, etc) [14:23:35] !bots [14:23:35] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [14:23:38] eeh [14:23:40] !project bots [14:23:41] https://labsconsole.wikimedia.org/wiki/Nova_Resource:bots [14:23:42] this one [14:24:07] yes, there are some rules, + labs rules [14:24:15] okay [14:24:20] I'll take a look at this [14:24:26] basic rule is don't try to break others stuff :) don't steal passwords, etc [14:24:43] environment is still in beta [14:24:53] lot of stuff will differ in "production" version [14:24:54] and do other people on the bots projects access my files ? [14:25:03] yes, they can [14:25:07] admins at least [14:25:19] hmm... so where / how can I store passwords ? [14:25:31] that's quite complicated right now [14:25:44] 10/03/2012 - 14:25:44 - User quentinv57 may have been modified in LDAP or locally, updating key in project(s): bastion [14:25:53] 10/03/2012 - 14:25:53 - Updating keys for quentinv57 at /export/keys/quentinv57 [14:26:39] most safe way is to enter password each time you start the bot in terminal [14:26:45] petan, that's impossible to be the owner of a file so only me as access to it ? [14:27:06] unfortunately I can't prompt the password because some are using crontab [14:27:10] yes, but it's possible to have a file which only admins and you can access [14:27:23] ah, that's cool, that's what I need [14:27:37] anyway, admins are people working for the foundation ? [14:27:41] it would need to be either on bots-1 or a new instance for that [14:27:53] no, admins are people working for foundation as well as some volunteers [14:28:00] admins of bot project, I mean [14:28:32] ah... I don't want to be perceived as a paranoid, but I hope they are trusted [14:28:49] I hope so as well, given that I run bunch of bots there :) [14:28:50] because if my bots account would be compromised it sucks :/ [14:29:12] so that's okay to store my bot passwords on this server ? [14:29:16] there is so low number of people that it wouldn't be hard to track who it was... [14:30:40] atm only admins of bots are: me, Ryan Lane, mutant e, jeremyb, rich smith, Thehelpfulone and Damianz [14:30:46] 10/03/2012 - 14:30:46 - User quentinv57 may have been modified in LDAP or locally, updating key in project(s): bots [14:32:32] petan, okay, so that's cool [14:33:23] petan, should I access the bot project through the bastion server, or should I connect elsewhere ? [14:33:31] bastion [14:34:51] Quentinv57 I just created a new instance which should be first of "semi-secure" instances, that means instance where people have restricted access, and where it should be harder to get access to you files (impossible to non-root's) I think you could try that one [14:35:08] if you read the bots proposal you would see the architecture we want in future [14:35:10] !bots [14:35:10] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [14:35:11] okay, thanks a lot [14:35:24] how can I acccess this instance ? [14:35:33] wait a moment... [14:42:20] How does which bit work? public_html stuff? [14:42:56] !log bots root: installing updates on nr1 [14:42:57] Logged the message, Master [14:43:44] Damianz, figured out :P [14:43:50] Damianz you could tell me before killing nfs [14:43:54] I had some cron jobs in that [14:44:16] oh, oops. sorry I checked if it had anything important export wise on [14:44:39] np [14:44:54] I'm trying to get some base puppet classes together for bots that should clean up the apache/sql servers as they currently have scripts for backups etc in project storage with crons to run them. [14:45:01] ok [14:45:08] is there some base class for public_html [14:45:11] ? [14:45:15] how do I set it up [14:45:28] not at the moment [14:45:33] ok [14:45:48] user dirs should get auto created, there's a cronjob on bots-apache1 that runs a python script that does an ldapsearch and creates the dirs. [14:46:11] Also does some other stuff like fixing ownership and chmoding removed members to 0 for the purposes of if we ever need to do that. [14:47:30] Quentinv57 you should be able to login to bots-nr1 now, if you can't sudo su, everything should be safe :P [14:47:54] nr1 is a non-sudo for members box? [14:48:22] yes [14:48:24] (which btw we can't add new servers currently due to nova changing the hostnames in ldap so sudo-ldap doesn't see the host is allowed) [14:48:27] it's like "no root" [14:48:43] Damianz umm [14:48:49] Damianz what exactly you mean [14:48:53] sudo policy doesn't work? [14:48:55] it did [14:49:09] It works if you select all (like the admin group in bots) [14:49:33] For the default group that has specific hosts allowed, you can't add a new instance to the list due to a hostname mismatch until Ryan fixes it in OSM. [14:50:15] petan, excuse me, you'll say I'm a noob, but I don't understand how to log in to a project on WMF Labs [14:50:28] ssh to bastion then ssh to the instnace [14:50:50] Damianz does it mean, users will get all root, or noone get root [14:51:01] Quentinv57 ssh to bastio, then ssh bots-nr1 [14:51:06] !bastion [14:51:06] http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org which should resolve to 208.80.153.194; see !access [14:51:08] !access [14:51:08] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [14:51:22] It means if we add bots-5 even if we add it to the policy normal users won't be able to sudo. [14:51:28] Admins still will [14:51:37] Damianz ok that's good [14:51:44] better than if all had root :P [14:51:49] yeah [14:54:46] !log bots petrb: created public_html :P [14:54:48] Logged the message, Master [14:55:02] !log bots petrb: created on nr1, of course [14:55:04] Logged the message, Master [14:55:22] Quentinv57 let me know if you needed any help with setting up access to instance [15:05:48] 10/03/2012 - 15:05:48 - Deleting home directory for wikidata in project(s): wikidata-dev [15:10:49] 10/03/2012 - 15:10:49 - Deleting home directory for wikidata in project(s): wikidata-dev [15:10:59] !tunnel [15:11:04] @search tunnel [15:11:04] Results (Found 1): putty, [15:12:11] * jeremyb waves [15:14:19] hey [15:14:26] jeremyb did you get the email [15:15:26] !security | Quentinv57 [15:15:26] Quentinv57: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [15:15:33] ah nvm [15:15:36] !ssh | Quentinv57 [15:15:36] Quentinv57: https://labsconsole.wikimedia.org/wiki/Help:SSH [15:15:42] this one [15:15:47] 10/03/2012 - 15:15:47 - Deleting home directory for wikidata in project(s): wikidata-dev [15:20:46] @seenrx Ryan [15:20:47] petan: Last time I saw Ryan_Lane they were quiting the network N/A at 10/3/2012 7:31:45 AM (07:49:01.3662520 ago) (multiple results were found: Ryan_Lane1) [15:20:47] 10/03/2012 - 15:20:47 - Deleting home directory for wikidata in project(s): wikidata-dev [15:20:59] 10/03/2012 - 15:20:58 - Updating keys for quentinv57 at /export/keys/quentinv57 [15:21:38] @seenrx Ryan.*1 [15:21:39] petan: Last time I saw Ryan_Lane1 they were quiting the network N/A at 9/21/2012 5:19:28 PM (11.22:02:10.4323370 ago) [15:24:29] <^d> LocalDiskRepositoryManager is a misleading class name when it also manages remote repos over ssh :p [15:24:37] <^demon> Silly gerrit. [15:25:48] 10/03/2012 - 15:25:48 - Deleting home directory for wikidata in project(s): wikidata-dev [15:25:59] lololol [15:30:49] 10/03/2012 - 15:30:49 - Deleting home directory for wikidata in project(s): wikidata-dev [15:33:36] petan: which mail? [15:33:44] regarding wm-bot [15:33:50] i think not? [15:34:14] how do you advise i search for it? or where did you send it to / from? [15:35:49] 10/03/2012 - 15:35:48 - Deleting home directory for wikidata in project(s): wikidata-dev [15:35:52] I sent it to tuxmachine mail [15:36:26] *now* i see it [15:36:33] 8ish hrs ago [15:36:37] I am just wondering why I sent it to Damianz when he is not listed in that config file :D [15:37:11] you sent something to me? :P [15:37:15] XD [15:38:03] petan: so... you're saying there's no canonical store? [15:38:16] simple english plis [15:38:18] we can't just delete the derived data and run something to regenerate? [15:38:26] canonical = one storage for both? [15:38:31] no [15:38:32] no [15:38:47] but it's pretty simple to delete it from both :> [15:38:50] i mean 1 that's the authoritative "master" version [15:38:58] the "regenerate" process would be complicated [15:39:10] well it depends on how it works... [15:39:31] technically I could create some "master" but I don't know if such a simple thing is worth of such a thing [15:39:33] if one is actually derived from the other then it's not complicated to rerun the derivation. i think [15:39:54] I know what you mean, but it seems to me too complicated for simple irc logs [15:40:02] oh hey he did email me :P [15:40:05] ok [15:40:10] It was lost in the spam of 'toolserver is broken' emails [15:40:13] regenerate means to recreate whole file, that is problem with poor IO we have [15:40:47] 10/03/2012 - 15:40:47 - Deleting home directory for wikidata in project(s): wikidata-dev [15:40:51] nah, i don't think it's a problem with IO [15:40:56] anyway, whatever ;) [15:41:10] it would be if I had to recreate whole log file everytime someone post a message to channel :PO [15:41:28] there is about 60 channel where this bot is in [15:41:31] * channels [15:41:52] this is useful: http://bots.wmflabs.org/~petrb/db/systemdata.htm :D [15:42:18] I forgot to mention that [15:43:43] well that would be fairly crazy. not what i mean... [15:43:51] anyway, whatever [15:45:47] 10/03/2012 - 15:45:47 - Deleting home directory for wikidata in project(s): wikidata-dev [15:50:49] 10/03/2012 - 15:50:49 - Deleting home directory for wikidata in project(s): wikidata-dev [15:51:43] <^demon> "Host keys for any destination SSH servers must appear in the user’s ~/.ssh/known_hosts file, and must be added in advance, before Gerrit starts. If a host key is not listed, Gerrit will be unable to connect to that destination, and replication to that URL will fail." [15:51:48] <^demon> Well that explains a lot :) [15:55:50] 10/03/2012 - 15:55:49 - Deleting home directory for wikidata in project(s): wikidata-dev [16:00:29] paravoid: how did you handle that grub issue with the instance that wasn't booting previously? [16:00:50] 10/03/2012 - 16:00:50 - Deleting home directory for wikidata in project(s): wikidata-dev [16:00:59] <^demon> Ryan_Lane: Can I get that sudo to manganese now? [16:01:12] do you want me to just add the ssh host? [16:01:35] <^demon> Yeah, let's try that first. [16:01:40] <^demon> "github.com,207.97.227.239 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" [16:01:54] <^demon> To ~gerrit2/.ssh/known_hosts [16:02:15] restarting gerrt [16:02:28] ^demon: it's easier to just ssh there [16:02:30] it'll add itself [16:02:51] <^demon> Or ssh as gerrit. [16:02:53] <^demon> Either works. [16:03:01] damn it [16:03:07] did you push the fix in for the config file? [16:03:21] <^demon> I pushed it, did you merge it? [16:03:23] <^demon> Ib8ca6ef6 [16:03:25] git@github.com:mediawiki/core.git [16:03:26] link? [16:03:32] well [16:03:34] can't do it now [16:03:37] what's the right value? [16:03:57] <^demon> git@github.com:${name} [16:05:45] 10/03/2012 - 16:05:44 - Deleting home directory for wikidata in project(s): wikidata-dev [16:06:07] well, I don't see replication errors [16:06:22] <^demon> Yeah, but not seeing any results on github's side yet. [16:06:25] <^demon> I'll keep poking [16:06:33] hmmrg puppet why you hate so much [16:06:36] <^demon> Actually, it might be pushing right now. [16:06:46] <^demon> Initial object push is gonna suck :) [16:07:14] <^demon> Ugh [16:07:17] <^demon> Wtf is going on [16:07:17] ? [16:08:05] <^demon> That stuff might be downtime related, I'll poke Roan about it [16:08:16] which stuff? [16:08:18] <^demon> No, seems weird. [16:08:23] bad pack? [16:08:27] <^demon> Yeah [16:08:31] * Ryan_Lane nods [16:10:47]