[00:02:21] I think some of you will be happy with my new openstackmanager changes :) [00:02:38] I'm making it so that a non-sysop group can do all the manage actions [00:02:46] so that we can open that up [03:22:02] 07/31/2012 - 03:22:01 - User madman may have been modified in LDAP or locally, updating key in project(s): bots,bastion,mediawiki-dev [03:22:24] 07/31/2012 - 03:22:24 - Updating keys for madman at /export/keys/madman [03:48:42] is anyone on who can help me get my vm successfully puppetting? [03:49:42] i cant seem to get the puppetmaster::self to run successfully [03:52:15] > [03:52:31] damn you Amgine [03:52:41] * jeremyb stabs [03:52:52] is StabbeD>\ [03:52:53] jesusaurus: hey. what did you try? [03:53:02] AmginE [03:53:09] [04:05:15] * nibalizer is confused [04:06:53] [04:15:33] * jesusaurus returns to the irc [04:16:22] so, i made an instance, clicked the puppetmaster, puppetmaste::dashboard, and puppetmaster::self boxes [04:16:55] and tried various combinations of 'true' and 'false' for the textboxes [04:17:41] Those distinctly sound like things done. [04:18:02] running 'sudo puppet agent --test' used to give a bunch of failed dependencies and an error near the top related to failing to create some ssl file [04:18:17] now i get 'connection refused' [04:19:14] Well, the latter sounds like a key issue. [04:22:44] jesusaurus: you need to say my name when you talk to me ;-) [04:23:23] oh, am i talking to you? [04:23:35] * jesusaurus didnt know who to talk to [04:23:35] idk ;) [04:23:51] Amgine: was humming at me and you seemed to be responding to my question [04:24:19] jesusaurus: first off you should make an instance with no classes and then add classes after it successfully builds and reaches a stable+idle state with no puppet errors [04:24:25] jesusaurus: then add classes [04:24:27] gah, fine, i'll keep track of names and who's talking to me :P [04:24:41] [04:24:58] but it was never stable (i tried running puppet before adding any classes) [04:25:14] before adding classes was when it was unable to create some ssl file [04:25:29] * jesusaurus tries deleting the instance and making a new one... again [04:48:50] jeremyb: okay im back at square one, for better or for worse [04:49:07] err: /Stage[main]/Certificates::Wmf_ca/Exec[/bin/ln -s /etc/ssl/certs/wmf-ca.pem /etc/ssl/certs/$(/usr/bin/openssl x509 -hash -noout -in /etc/ssl/certs/wmf-ca.pem).0]/returns: change from notrun to 0 failed: /bin/ln -s /etc/ssl/certs/wmf-ca.pem /etc/ssl/certs/$(/usr/bin/openssl x509 -hash -noout -in /etc/ssl/certs/wmf-ca.pem).0 returned 1 instead of one of [0] at [04:49:08] /etc/puppet/manifests/certs.pp:198 [04:49:28] jesusaurus: when was the instance first created? [04:49:38] (this is brand new i guess but the other one?) [04:49:48] the other one was a few weeks ago [04:49:52] hrmmm [04:49:53] maybe 3 weeks ago? [04:50:09] that error looks very familiar. i can't remember if i got the instance to fix itself eventually [04:51:06] well, nothing changed during the run, so its gonna stay broken without manual intervention [04:52:53] yeah... just can't remember [04:53:55] jesusaurus: can i get on the box? [04:55:58] yeah, its 10.4.0.207 , is there a checkbox somewhere for me to check, or do you wanna just give me a pubkey [05:17:55] jeremyb: ?? [05:30:51] hi [05:33:50] jesusaurus: it seems to be in the puppet-cleanup project which i'm not in [05:34:24] jesusaurus: you could add me at the "manage projects" link (left sidebar on all pages) [05:43:44] Ryan_Lane: does your new group for manage actions effect the anyone can add to bastion issue? [05:44:15] no [05:44:19] that's coming later [05:44:45] I'm focusing on adding openstack api support, keystone support, and essex compatibility [05:44:54] I'm dropping ec2 compatibility support [05:45:19] once we upgrade I'll make the changes for self-registration [05:45:42] it's not a new group just yet. I'll keep it as sysop for right now, and add a new group soon after [05:45:45] upgrade means to essex? [05:45:48] yep [05:46:01] ok. does OSM get releases? [05:46:08] or just runs master? [05:46:38] * jeremyb still has trouble with that being openstreetmap ;P [05:48:58] OSM gets releases, but we run master [05:49:35] it's probably not amazingly usable for others but us, unless someone else puts some dev effort into it [05:49:47] heh [05:50:14] mostly because it has a pretty strong dependency on ldap [05:50:34] Ryan_Lane: just noticed the /topic... can't we just hide the test branch or rename to oldtest or something? and also disallow push [05:50:46] s/push/push for review/ [05:50:48] I'm working with the keystone ldap guy to make it possible for us to use keystone as the ldap manager, rather than OSM directly [05:51:00] I'm going to delete it [05:51:02] what's keystone then? a webapp? [05:51:03] maybe I'll do that tomorrow [05:51:18] keystone is the auth service for openstack [05:51:19] (i've heard the name but don't entirely understand) [05:51:27] right... that's what i thought [05:51:39] but where would i log in to do LDAP changes? [05:51:57] if I can add/manage users, groups, tenants, etc via keystone, then OSM could call it, rather than direct ldap calls [05:52:04] btw, 30 17:27:23 < jeremyb> anyone know ldap tricks for labs? how do i list or query properties of hosts? [05:52:07] 30 17:27:29 < jeremyb> ldaplist doesn't seem to work [05:52:14] yeah. I need to modify ldaplist [05:52:22] the ldap tools need to be more generic [05:52:29] maybe I'll release them as a library at some point [05:52:31] ahh, ok. so would be the same for the user [05:52:38] my tools are the best I've seen so far [05:52:46] which is sad [05:52:50] Ryan_Lane: ldaplist already has "hosts" support. maybe i'm using it wrong? [05:52:55] hahaha [05:53:12] well, it searches for host using an objectclass we aren't using [05:53:22] ldaplist is a reimplementation of a solaris tool [05:53:33] so how do i know what objectclass to use? [05:53:41] it's hardcoded, which is the problem [05:53:57] we use a standard dns schema [05:54:28] you can search for associateddomain in ou=hosts [05:54:35] using ldapsearch [05:54:39] ldapsearch is harder to use, though [05:55:22] associateddomain... [05:55:44] or dc, if you don't want to search the fqdn [05:56:01] dc is by the instance-id only though [05:56:32] jeremyb: are you 'jeremyb' on labs too? [05:56:41] jesusaurus: yes [05:56:44] oh, also, 30 17:51:21 < jeremyb> LeslieCarr: it is. i just can't figure out how to get a hostname or public IP listing for projects I'm not a member of [05:56:48] 30 17:52:56 < jeremyb> LeslieCarr: i was thinking they should just be normal SMW vars like everything else [05:57:14] Ryan_Lane: so, should i be able to do something like this: ldapsearch -x -LLL -z 5 '(ou=hosts,dc=wikimedia,dc=org)' [05:57:21] to get any arbitrary 5 hosts [05:57:22] ? [05:57:27] you can search in mediawiki for it [05:57:34] -z 5? [05:57:38] limit [05:57:43] what is -z? [05:57:44] oh [05:58:00] 07/31/2012 - 05:57:59 - Created a home directory for jeremyb in project(s): puppet-cleanup [05:58:03] Ryan_Lane: search for what? i tried for like 20 mins [05:58:17] ldapsearch -x -LLL -D '' -W -b 'ou=hosts,dc=wikimedia,dc=org' 'dc=' [05:58:18] or [05:58:36] ldapsearch -x -LLL -D '' -W -b 'ou=hosts,dc=wikimedia,dc=org' 'associateddomain=*' [05:58:56] -LLL isn't really necessary [05:59:01] 07/31/2012 - 05:59:01 - User jeremyb may have been modified in LDAP or locally, updating key in project(s): puppet-cleanup [05:59:03] neither is the base [05:59:35] mediawiki's search isn't wonderful for this :( [05:59:44] I search for the instance id [05:59:52] but with i- removed [06:00:11] which is silly, since it's more specific [06:00:23] I should add lucene at some point [06:01:22] are you guys aware of the 500s on labsconsole? [06:01:40] yes [06:02:01] it'll be fixed when we upgrade openstack [06:02:07] though we'll likely fix it before then [06:02:17] it's a scaling issue in the broken version of nova we're using [06:02:40] yeah, i'm not worried -- just making sure you guys knew [06:02:50] i'm pretty happy with labs [06:02:55] thanks :) [06:03:05] it'll be much more resilient after the upgrade [06:03:23] we've doubled size in like 3 months, it causes issues [06:04:16] eep [07:25:27] Ryan_Lane: around for a few mins? i think i'll need a puppet merge soon [07:25:34] still working on it [07:25:43] this is icky [07:25:44] I'm heading off to bed [07:25:59] * jeremyb wonders if the greeks are on then ;) [07:43:39] * jeremyb is ready to push now [07:47:36] pushed [07:48:20] !g I9e1b90579fba24 | jesusaurus, you may want to take a look too [07:48:20] jesusaurus, you may want to take a look too: https://gerrit.wikimedia.org/r/#q,I9e1b90579fba24,n,z [07:50:40] oh, huh [07:50:54] morning hashar [07:51:00] want to sanity check me? ;) [07:51:01] yeah, i dont really understand what that file is (or rather what that openssl command is doing) [07:51:07] morning [07:51:22] jeremyb: kind of busy fixing Jenkins :-D [07:51:36] heh. i saw no problems with jenkins! ;P [07:51:45] (i submitted stuff and saw jenkins respond) [07:52:47] I am rushing up the mw extensions testing ;) [07:52:54] aha [07:53:40] and loving to hate ant [07:53:52] I think I will rewrite the whole build script using rake instead [07:54:02] wtf [07:54:08] <3 rake [08:08:04] yeahhhhhhhhh [08:08:08] [echo] Opts....: --exclude-group Broken,ParserFuzz,Stub -- /var/lib/jenkins/jobs/Ext-Wikibase/workspace/extensions [08:08:11] I am happy [08:15:05] hashar: i want you dancing on commons then ;) [08:15:24] that would be kind of weird :-D [08:15:45] you seemed so happy [08:17:30] * Platonides gives hashar https://commons.wikimedia.org/wiki/File:Birthday_Cake.jpg [08:21:56] Platonides: thanks :) [08:22:35] :) [08:31:09] hashar: there's something wrong with the permissions on that cake [08:31:18] @help [08:31:18] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.4 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [08:33:22] @recentchanges+ mediawiki [08:33:22] This channel is already watched [08:33:29] @recentchanges+ labsconsole [08:33:29] There is no such a wiki in list of wikis [08:35:24] @RC+ mediawiki Developer access [08:35:24] Inserted new item to feed of changes [08:35:29] @RC+ mediawiki Talk:Developer access [08:35:29] Inserted new item to feed of changes [08:36:17] @RC- mediawiki Developer [08:36:17] Deleted item from feed [08:36:22] @RC+ mediawiki Developer_access [08:36:23] Inserted new item to feed of changes [08:36:34] @RC+ mediawiki Talk:Developer_access [08:36:34] Inserted new item to feed of changes [08:36:47] jeremyb it doesn't read spaces [08:36:50] haha [08:36:54] which are parameter separators [08:36:56] petan: and labsconsole? [08:37:24] labsconsole doesn't work because there is a high priority bug assigned to ryan [08:37:29] which isn't solved yet [08:37:35] @RC- mediawiki Talk:Developer [08:37:35] Deleted item from feed [08:37:36] @RC- mediawiki Talk:Developer [08:37:36] Can't find item in a list [08:38:03] i thought i saw some feed from labsconsole maybe. maybe imagined it [08:38:21] @RC+ labsconsole . [08:38:21] Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it [08:39:12] @info [08:39:12] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [08:40:11] !b 34685 [08:40:12] https://bugzilla.wikimedia.org/34685 [08:47:25] 07/31/2012 - 08:47:24 - Creating a home directory for por at /export/keys/por [08:48:23] 07/31/2012 - 08:48:23 - Updating keys for por at /export/keys/por [09:32:24] 07/31/2012 - 09:32:24 - Updating keys for purodha at /export/keys/purodha [09:32:47] Change on 12mediawiki a page Developer access was modified, changed by Matanya link https://www.mediawiki.org/w/index.php?diff=567146 edit summary: [09:33:00] Change on 12mediawiki a page Developer access was modified, changed by Matanya link https://www.mediawiki.org/w/index.php?diff=567147 edit summary: /* User: */ [09:45:06] !log deployment-prep deleting /home/wikipedia/common-prod was used to compare production with beta. Now that we are using the same files as production, the directory is useless and eat space for no purpose [09:45:35] grmblblb [09:50:00] petan: hi :) do you have any idea what deployment-wmsearch was for ? [09:50:16] petan: it got corrupted and I am not sure it has any purpose for now. I would like to simply delete it [09:50:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=38748 [10:25:41] paravoid: ping pong awake ? :-) [10:28:02] 07/31/2012 - 10:28:01 - Created a home directory for mlitn in project(s): deployment-prep [10:29:00] 07/31/2012 - 10:29:00 - User mlitn may have been modified in LDAP or locally, updating key in project(s): deployment-prep [10:29:22] !log deployment-prep Added Matthias Mullie as a sysadmin. [10:42:13] yes [10:42:22] sorry, just saw your message [10:45:03] paravoid: Matthias has issue connecting to deployment-dbdump :/ [10:45:09] he logs on bastion just fine [10:45:23] login is mlitn [10:45:43] Jul 31 10:30:09 i-000000d2 sshd[25744]: pam_access(sshd:account): access denied for user `mlitn' from `i-000000ba.pmtpa.wmflabs' [10:45:43] :( [10:46:05] matthiasmullie paravoid is Faidon from the ops team :-) [10:46:31] he is our hourly life saver for anything related to Labs, Debian and generally anything related to nix :) [10:49:25] ok cool, because it looks like we'll need help :) [10:49:35] looking [10:50:04] were you recently added to the group? [10:53:51] I have added it roughly 20 minutes ago [10:54:14] [12:28:01pm] [10:54:15] 07/31/2012 - 10:28:01 - Created a home directory for mlitn in project(s): deployment-prep [10:54:16] [12:29pm] [10:54:16] 07/31/2012 - 10:29:00 - User mlitn may have been modified in LDAP or locally, updating key in project(s): deployment-prep [10:55:09] okay, fixed [10:55:35] it's too expensive to do ldap queries whenever you check for group membership etc., so there's a daemon that caches that [10:55:44] out for lunch [10:55:49] matthiasmullie: I will be back in roughly one hour [10:55:57] paravoid: hooo [10:55:58] matthiasmullie: should work now though :) [10:56:01] paravoid: good to know for next time [10:56:04] thanks for fixing his access!! [10:56:54] ok cool, works indeed - thanks [13:05:29] <^demon> !svn is SVN is a chocolate chip cookie: [13:05:29] Key was added [13:09:09] <^demon> hashar: I was glancing over your work on gerrit manifest. Why aren't we creating the gerrit2 account on all hosts running gerrit? [13:09:35] must be something I have not spotted [13:09:45] <^demon> Yeah, it appears to not be created for labs instances. [13:09:58] I mainly moved the code back to manifest/gerrit.pp and create some basic/lame instances [13:10:07] what is gerrit2 account used for? [13:10:14] <^demon> So gerrit has a system account to run as. [13:10:23] <^demon> Oh wait, I think Ryan said the package takes care of the user now. [13:10:30] wonderbar! [13:10:35] <^demon> So we could remove that entirely. [13:10:53] that is exactly why I love puppet + labs [13:11:11] let you easily start a fresh instance, deploy entirely form puppet and fix the nasty stuff :) [13:11:27] <^demon> I tried to make a new instance yesterday so we could build our own gerrit.war. But Java sucks :\ [13:11:42] maybe puppet can be informed that the gerrit install provide a User['gerrit2'] [13:11:52] and you can then require User['gerrit2'] [13:11:53] you pinged me :D [13:12:10] ^demon: what happens with java ? [13:12:15] <^demon> hashar: Na, if we know the package does it, just require the package. [13:12:18] not like I could help, but I can show you some support :) [13:12:21] puppet and [13:12:39] <^demon> I installed the openjdk stuff, but maven didn't like that one bit [13:13:41] <^demon> Bellyached that it couldn't find JAVA_HOME (what sane packager left that out?!?), and after I set that, yelled about not being able to find the jdk at all. [13:13:47] <^demon> Stupid openjdk. [13:14:38] <^demon> I'll probably just rebuild and install the sun jdk instead. [13:17:18] ^demon: we had a discussion back in november about sun jdk [13:17:25] one of the problem is that there is no more any Ubuntu package for the sun jdk [13:17:37] I think Oracle make it illegal to provide such a package [13:17:44] <^demon> Arghhh, fucking oracle. [13:17:47] <^demon> I hate them. [13:18:45] ^demon: http://sylvestre.ledru.info/blog/sylvestre/2012/02/29/java_package_replacement_of_sun_java6 [13:19:08] I would not use such statement on a public channel [13:19:12] but yeah [13:19:19] fuck oracle [13:20:44] yeah, they discontinued the DLJ [13:20:49] but I think this is a good thing nowadays [13:21:03] it will give the final push for people to move to the OpenJDK [13:22:01] ^demon: maybe you need a recent openJdk version? [13:22:02] ^demon: which machine is that? [13:22:06] with the JAVA_HOME issue? [13:22:19] <^demon> It's a vm I spun up yesterday [13:22:50] <^demon> I deleted it after I got mad at openjdk :) [13:24:05] I don't understand why [13:24:16] the openjdk packaging was the same with the sun-* packages iirc [13:24:39] anyway, if you'd be willing to work on it again [13:24:45] I can work with you and fix those issues [13:25:17] <^demon> I'd be willing to try again. Using packages from ubuntu rather than manually installing sun-jdk is nicer if we can make it work. [13:25:41] I'd prefer it if we didn't mess with the Sun JDK [13:26:14] note that our repo is public [13:26:37] which means that if we package the Sun^WOracle JDK (e.g. with java-package as mentioned above) and put it in our apt repo [13:26:46] we'll be distributing something that we're not legally permitted to [13:27:41] same if you distribute it via other methods [13:28:04] so, let's avoid this proprietary shit as much as we can please :) [13:30:10] going to get an hair cut [13:30:22] will be back in a few [13:30:27] <^demon> paravoid: Well, the plan wasn't to package it or put it anywhere other than on the host in question. But you're right, yes. [13:30:57] <^demon> paravoid: This is what I installed http://p.defau.lt/?lAlIJnU0Xvv6wkzlKe8I7Q [13:31:12] <^demon> As well as git, but that's not an issue. [13:37:53] Change on 12mediawiki a page OAuth/status was modified, changed by Guillom link https://www.mediawiki.org/w/index.php?diff=567293 edit summary: new status update [13:38:36] ^demon: so you mean this is just for building the war, not actually running it [13:38:53] <^demon> Yeahh, just for building it. [13:39:09] <^demon> We want to build our own, so in case we want to pull in fixes without waiting for a release. [13:39:35] <^demon> Plus, gives us the option of making any other custom hacks (hopefully few and far between) [13:49:13] <^demon> paravoid: Ok, so after installing the openjdk stuff + maven, I cloned gerrit to /home/demon/gerrit. [13:49:23] <^demon> Tried `mvn package` and it complained about JAVA_HOME. [13:54:09] instance name? [13:56:00] 07/31/2012 - 13:56:00 - Created a home directory for jan in project(s): gerrit [13:57:00] 07/31/2012 - 13:57:00 - User jan may have been modified in LDAP or locally, updating key in project(s): gerrit [13:57:16] <^demon> gerrit-build [13:59:02] sec. [14:03:16] <^demon> I'll be back in a few minutes. [14:14:25] Change on 12mediawiki a page Developer access was modified, changed by Rotsee link https://www.mediawiki.org/w/index.php?diff=567313 edit summary: [14:15:49] <^demon> Ok, back [14:53:32] back [14:54:50] :O [14:58:52] I love labs [14:58:53] [pid 4681] rename(".git/ORIG_HEAD.lock", ".git/ORIG_HEAD") = -1 EACCES (Permission denied) [15:00:04] seriously [15:01:32] <^demon> Seriously? I'm caught up with new repo creations? [15:01:35] * ^demon does a happy dance [15:03:42] * Damianz gives chad a cookie [15:09:53] I am cursed :( [15:10:13] I got a file permission error when doing a git pull [15:10:20] although the perms looks fine [15:10:22] :/ [15:11:17] paravoid: I got a bug with /data/project , got some permission denied :/ [15:12:11] http://dpaste.org/0qesd/ :-( [15:12:31] doing a cp -a of the dir to a local directory works just fine [15:14:25] [2012-07-31 15:13:30.781221] I [client3_1-fops.c:1999:client3_1_rename_cbk] 0-deployment-prep-project-client-0: remote operation failed: Permission denied [15:14:27] yeahhh [15:21:10] !log deployment-pre rebooting dbdump to see if it help with a weird GlusterFS issue [15:22:44] propoganda :o [15:23:00] * aude amused by the unicorn [15:23:28] indeed [15:23:59] <^demon> robla put it best. "You can't win against a unicorn, you just can't." [15:24:07] heh [15:25:48] <^demon> We don't have any freely licensed pictures of sad unicorns on commons. [15:25:52] <^demon> This must be rectified. [15:27:42] °-------<:(---/----/, [15:27:49] hmm [15:28:01] no idea how to properly represent an unicorn using just one line of ASCII [15:29:27] grrrbmblblbl [15:30:52] <^demon> ônn [15:30:59] <^demon> Well, he's lacking a tail. [15:31:07] <^demon> ônn- [15:31:15] <^demon> Now he looks like a pig with a horn [15:31:19] <^demon> Or a hat [15:32:05] ^demon: there are some nice ones at http://www.ascii-art.de/ascii/uvw/unicorn.txt [15:32:13] ^demon that's a beatifull worm [15:32:31] beautifull [15:34:41] <^demon> hashar: omg yay https://gerrit-review.googlesource.com/#/c/37102/ [15:39:39] / [15:39:40] .7 [15:39:41] \ , // [15:39:42] |\.--._/|// [15:39:43] /\ ) ) ).'/ [15:39:44] /( \ // / [15:39:45] /( J`((_/ \ [15:39:46] / ) | _\ / [15:39:48] /|) \ eJ L [15:39:49] | \ L \ L L [15:39:50] / \ J `. J L [15:39:51] | ) L \/ \ [15:39:52] / \ J (\ / [15:39:53] _....___ | \ \ \``` [15:39:54] ,.._.-' '''--...-||\ -. \ \ [15:39:55] .'.=.' ` `.\ [ Y [15:39:56] / / \] J [15:39:57] Y / Y Y L [15:39:58] | | | \ | L [15:39:59] | | | Y A J [15:40:00] | I | /I\ / [15:40:01] | \ I \ ( |]/| [15:40:02] J \ /._ / -tI/ | [15:40:03] L ) / /'-------'J `'-:. [15:40:04] J .' ,' ,' , \ `'-.__ \ [15:40:05] \ T ,' ,' )\ /| ';'---7 / [15:40:06] \| ,'L Y...-' / _.' / \ / / [15:40:07] J Y | J .'-' / ,--.( / [15:40:08] L | J L -' .' / | /\ [15:40:09] | J. L J .-;.-/ | \ .' / [15:40:10] J L`-J L____,.-'` | _.-' | [15:40:12] L J L J `` J | [15:40:13] J L | L J | [15:40:14] L J L \ L \ [15:40:15] | L ) _.'\ ) _.'\ [15:40:16] L \('` \ ('` \ [15:40:17] ) _.'\`-....' `-....' [15:40:18] ('` \ [15:40:19] `-.___/ [15:40:29] really needed to spam the channel [15:40:31] :D [15:43:30] :) [15:47:09] ^demon: yeahh moar prolog :-D [15:47:36] soo [15:47:37] hmm [15:47:41] labs is broken [15:47:50] well beta is [15:48:00] git does not work over Gluster :( [15:48:18] <^demon> !paste | petan [15:48:18] petan: http://pastebin.com [15:48:43] that would be so boring [15:49:42] hashar: doesn't surprise me [15:49:55] mysql doesn't work either there [15:50:56] 1 [15:51:02] eek [15:54:36] <^demon> Sweeeeet, the search suggestions are live on the upstream gerrit :D [15:54:42] <^demon> https://gerrit-review.googlesource.com/#/ - try typing stuff in the box :) [15:54:46] Petan: is there an invisible color for IRC? [15:55:12] Amgine depends on client [15:55:23] you can configure any colors [15:55:25] [15:55:34] at least in client I am using I can :D [15:56:45] aww [15:56:57] I thought petan was pasting the postgre logo then [15:57:31] LOL [15:57:43] Damianz you don't see it as unicorn? [15:57:49] if so, delete your irssi [15:57:52] Irc is made tiny atm :P [15:57:59] bad you [15:58:13] sublimetext2 is using my screen space! [15:58:27] ^demon: only autocomplete for the search term? [15:58:48] topic:.... and then it stops suggesting? [15:59:00] <^demon> Yeah, it doesn't look for possible values. [15:59:04] <^demon> Except status, since they're known. [15:59:11] better than now, though [15:59:12] <^demon> This was round 1 of suggestions. [15:59:22] <^demon> I think there's ideas to make other things searchable there. [15:59:28] good [15:59:43] and of course, i'd like to just type something in the box, like my name [15:59:49] and everything related to me come up [16:01:17] aude: that implies contextual meta data about the user. [16:01:25] ala google [16:02:49] !log deployment-prep enabled wmgClicktracking which is a dependency for AFTv5. That magically enabled AFTv5! Thanks to Matthias Mullie for spotting the error. [16:02:55] matthiasmullie: thanks for the click tracking error :) [16:03:31] no problem [16:04:09] I was just taking another look at it, seeing the error had disapeared, but not seeing aft on any of the pages I had hit so far [16:04:34] and it magically does not appear anymore :/ [16:04:40] lol [16:05:29] mwscript eval.php --wiki=enwiki [16:05:29] > var_dump( $wmgClickTracking ); [16:05:30] bool(true) [16:05:31] :( [16:05:46] http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 - doesn't seem to exist [16:06:00] where it did before, rightM [16:06:04] ?* [16:06:38] !log deployment-prep chown -R mwdeploy:svn /home/wikipedia/common/wmf-config [16:07:01] it worked once on http://en.wikipedia.beta.wmflabs.org/wiki/Rock_balancing?debug=true [16:07:11] grmblblbl [16:09:51] Amgine: i can in fact search on google for "aude site:bugzilla.wikimedia.org" and it works but not "aude site:gerrit.wikimedia.org" [16:09:56] we could at least fix that? [16:10:18] <^demon> We disabled robots on gerrit. [16:10:32] https://gerrit.wikimedia.org/robots.txt :( [16:10:36] ^demon: why? [16:10:41] <^demon> Because they were being stupid, indexing gitweb, and causing it to crash. [16:10:50] heh. [16:10:51] sad [16:11:10] <^demon> Also, everything's delivered with JS via JSON-RPC calls, so I doubt gerrit itself would index well. [16:11:25] or maybe once we can mirror stuff to github.... [16:11:30] * aude can search github [16:11:51] of course doesn't include all the code review stuff [16:11:53] hashar: > var_dump( $wmgUseArticleFeedbackv5 ); [16:11:54] bool(false) [16:12:20] <^demon> aude: Replication to github is high on my priority list, once 2.5 comes out. [16:12:26] ^demon: good [16:12:33] <^demon> s/high/highest/ [16:12:39] <^demon> It's really really important. [16:12:40] * aude appreciated when avar set it up before [16:12:50] and he did for osm and that gets used [16:13:06] <^demon> That was a proof of concept and was replicated from SVN, so the move to git changed how it'll happen. [16:13:11] <^demon> But yes, I will be working on it. [16:13:12] right [16:13:18] good [16:13:28] matthiasmullie: well spotted. Let me enabled it again on labs :) [16:13:29] * aude can be somewhat patient :) [16:13:30] hashar: also [16:13:31] var_dump( $wmgArticleFeedbackNamespaces ); [16:13:32] bool(false) [16:13:37] grr [16:13:45] seems like a few config things are gone [16:13:49] yeah [16:13:50] and mediawiki settings stuff soon i hope? [16:13:54] <^demon> Step 1 is just the replication. Step 2 is some kind of hacky way to get pull requests into gerrit. [16:13:59] I tried to fix up the wmf-config dir that got broken thanks to gluster [16:14:04] * aude nods [16:14:11] <^demon> Step 3 is some magic future in which stuff like review moves pretty seamlessly between gerrit/github. [16:14:17] though that last one shouldn't matter - i think [16:14:20] magical :) [16:14:28] <^demon> I love magic futures :) [16:15:22] ...PROFIT! [16:16:46] <^demon> Oh man, I have to turn a profit here? [16:16:48] <^demon> :( [16:18:24] matthiasmullie: apparently setting wmgArticleFeedbackNamespaces to false is meant to load $wgContentNamespaces [16:19:20] hashar: yes, my bad - i just seem to have misread "'eswiki' => array( NS_MAIN )" for "enwiki" [16:19:30] so it is indeed perfectly normal that it's false [16:21:01] matthiasmullie: well I give up for today :/ [16:21:09] http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 is still not around [16:21:19] despite having $wmgUseArticleFeedbackv5 to true [16:24:31] matthiasmullie: got it :) [16:24:57] the commonsettings-wmflabs.php is loaded to late so its settings are not always available in CommonSettings.php :( [16:25:01] need to fix that up one day [16:25:44] matthiasmullie: got something at [16:26:15] !log deployment-prep apparently AFTv5 is somehow enabled intermittently on beta. Got it loading several time via http://en.wikipedia.beta.wmflabs.org/wiki/%C4%B0%CC%83?debug=true . I am stopping altering any configuration for n ow. [16:26:40] !log deployment-prep example: http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:ArticleFeedbackv5/%C4%B0%CC%83&ref=cta#2 [16:27:06] chrismcmahon: AFTv5 is enabled on beta now. Thanks to matthiasmullie who found out the root cause (click tracking extension was not enabled). http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:ArticleFeedbackv5/%C4%B0%CC%83&ref=cta#2 [16:27:08] ;) [16:27:11] I am off for now [16:27:13] see you later [16:28:28] hashar matthiasmullie I'm seeing AFTv5 on beta labs, thanks! [16:28:55] aight cool, looks like we made it :) [16:45:18] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=567363 edit summary: /* User:Matanya */ done [16:45:19] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=567364 edit summary: /* User:Rotsee */ done [17:17:03] 07/31/2012 - 17:17:03 - Creating a project directory for codereview [17:17:04] 07/31/2012 - 17:17:03 - Created a home directory for catrope in project(s): codereview [17:18:02] 07/31/2012 - 17:18:02 - User catrope may have been modified in LDAP or locally, updating key in project(s): codereview [17:34:12] paravoid: virt1 seems to be unhappy [17:34:29] I'd also like to migrate all instances from virt5 first, if we can [17:34:35] then we can unmount the gluster storage [17:34:42] and use it as another host to migrate to [18:26:12] how many virts are there? [18:26:38] !nagios | jesusaurus [18:26:39] jesusaurus: http://208.80.153.210/nagios3 http://nagios.wmflabs.org/nagios3 [18:27:10] !load [18:27:10] http://ganglia.wikimedia.org/2.2.0/graph_all_periods.php?h=virt2.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1327006829&g=load_report&z=large&c=Virtualization%20cluster%20pmtpa [18:27:20] that should point to virt hosts [18:27:38] I don't know if you wanted virtual hosts or vm's [18:29:19] i wanted a list of nodes (not instances) [18:29:29] second link then [18:31:05] cool [18:33:09] JeroenDeDauw: I'm just following up from Ryan's email… are you currently blocked, or is it clear how to proceed? [18:51:13] <^demon> hashar: I fixed the syntax errors and cleaned up the conditional include of gerrit::account. [18:51:22] <^demon> Still failing at the mysql stuff, but hey, progress :) [18:51:30] Ryan_Lane: if you are still around. I got a serious issue with cluster :/ [18:51:34] err Gluster [18:51:38] oh? [18:51:39] what's up? [18:51:56] ^demon: you can strip out all of the mysql stuyff [18:51:57] *stuff [18:52:03] that should be a separate thing [18:52:04] $ touch FOOBAR && mv FOOBAR NEWFILE [18:52:04] mv: cannot move `FOOBAR' to `NEWFILE': Permission denied [18:52:09] <^demon> Ryan_Lane: For labs? Where should it be pointed at? [18:52:15] <^demon> We need a database somewhere :) [18:52:21] ^demon: meaning, mysql should be included for the node as a role [18:52:30] not stuck in with the gerrit stuff [18:52:34] that is on dpeloyment-dbdump:/home/wikipedia/common which itself is a eerie of symlinks to /data/project/apache/common-local :/ [18:52:53] hashar: you are sharing NFS from gluster? [18:52:58] <^demon> Ryan_Lane: We cleaned that up, it's conditional. [18:53:03] ah [18:53:03] ok [18:53:18] <^demon> Node can set $self_db => true if needed. [18:53:22] cool [18:53:31] ^demon: there is a labs database server puppet class somewhere but I don't think it setup the root password [18:53:42] hashar: …? [18:53:46] <^demon> Yeah, I started poking at the mysql classes, but meh. [18:53:48] I wanted to use generate( ) to create a password sand save it somewhere [18:53:53] but not managed to go anywhere :/ [18:53:54] I'm in the middle of programming, if you need help, please be responsive [18:54:02] Ryan_Lane: sorry [18:54:10] Ryan_Lane: trying to figure out what you have asked [18:54:15] hashar: you are sharing NFS from gluster? [18:54:44] Ryan_Lane: I don't think we have any NFS mount anywhere [18:54:53] ok. so everything is accessing gluster directly [18:54:57] Ryan_Lane: we have ended up doing basic symbolic links instead that points to /data/project [18:55:07] so, the issue is on deployment-dbdump? [18:55:23] rename("FOOBAR", "NEWFILE") = -1 EACCES (Permission denied) [18:55:31] yeah on deployment-dbdump [18:55:31] which user is running that [18:55:34] in which directory? [18:55:47] that is user "hashar" in directory /data/project/apache/common-local [18:55:57] I have triple checked the user rights but did not manage to find out the culprit [18:56:37] the dir is: drwxrwxr-x 11 mwdeploy svn 8192 2012-07-31 18:55 ./ [18:56:37] and I am in group 'svn' [18:56:40] interesting [18:56:45] I moved the file to test and it worked [18:56:49] just NEWFILE doesn't [18:56:53] <^demon> paravoid: I fixed my java problem. Instead of just installing 'maven' and praying it had the right dependencies, I also installed 'openjdk-6-jdk' works now. [18:57:17] Ryan_Lane: iirc there is not the same issue when being root [18:57:22] wow [18:57:23] weird [18:57:29] I can also create NEWFILE as you [18:58:15] I am wondering if that instance got corrupted somehow. Trying on another one [18:58:25] no [18:58:30] it's not a corrupted issue [18:58:32] it's some gluster issue [18:59:08] let me look on the servers [18:59:44] [2012-07-31 18:57:54.425305] W [dht-rename.c:480:dht_rename_cbk] 0-deployment-prep-project-dht: /apache/common-local/FOOBAR: rename on deployment-prep-project-replicate-0 failed (Permission denied) [18:59:45] [2012-07-31 18:57:54.428691] W [fuse-bridge.c:1351:fuse_rename_cbk] 0-glusterfs-fuse: 11993: /apache/common-local/FOOBAR -> /apache/common-local/NEWFILE => -1 (Permission denied) [18:59:49] from /var/log/glusterfs [18:59:49] yeah [19:00:06] which also have the very nice log files named data-.git.log and data-HEAD.log [19:00:20] seems to have been generated when I attempted a git pull of some extensions [19:01:04] use case: cd /data/project/apache/common/php-master/extensions/VipsScaler/ && git pull [19:01:10] error: Couldn't set ORIG_HEAD [19:01:10] fatal: Cannot update the ref 'ORIG_HEAD'. [19:01:19] strace show git cannot rename an ORIG_HEAD.lock file to ORIG_HEAD [19:01:27] so... [19:01:32] gluster had an issue [19:01:32] Ryan_Lane: i feel so sorry about interrupting your programming :-/ [19:01:40] I deleted FOOBAR [19:01:42] recreated it [19:01:44] oh [19:01:44] wait [19:01:46] I was root [19:01:57] yeah as root it seems to work too [19:05:01] well, this surely looks like a gluster bug [19:06:55] then we end up with the Gluster upgrade drama ? :/ [19:07:25] I am so angry to not have noticed that before [19:07:36] would have kept the NFS instance if I knew about that limitation [19:07:56] well, hopefully this will be solved by a gluster update [19:08:00] but it seems it's broken for now [19:08:11] is this affecting git? [19:08:20] yup [19:08:32] it does put metadata in a temporary file then use rename() on it [19:08:33] is this happening on all nodes, or just this one? [19:08:46] ah [19:08:47] checking on a Precise instance [19:08:53] [11598.158941] testgluster[12903]: segfault at 2f07d451 ip 00007f532ef7ee32 sp 00007fff45fe2dd0 error 4 in libc-2.11.1.so[7f532ef36000+17a000] [19:09:00] (note the obvious reproduction is to use mv ) [19:10:45] Ryan_Lane: same issue on a precise box ( deployment-apache33 ) [19:11:01] are there any lucid boxes around? [19:11:31] deployment-dbdump is A Lucid one [19:11:59] ah [19:12:34] i have a machine (vanadium.eqiad.wmnet) that only has an internal ip -- is there a way that it can connect to / accept connections from labs machines? [19:12:47] ori-l: no [19:12:58] Ryan_Lane: ahh the testgluster shell is a C program I did to reproduce the issue . But mv works as well [19:13:09] ah. I see [19:13:15] we really need to upgrade gluster [19:13:19] bummer [19:13:28] we're already on 3.3, so I think we can do it with no downtime [19:13:47] I'll schedule it for tomorrow [19:14:03] hopefully :) [19:17:56] seems something was added to the deployment page and not the engineering calendar again [19:18:01] fuck it, I'm not changing my slot [19:21:00] 07/31/2012 - 19:21:00 - Created a home directory for ori in project(s): reportcard [19:21:58] andrewbogott: Right now I really don't feel like doing that setup again (not my responsibility or job) - I might do it at some later point anyway to see how the puppet stuff works [19:22:01] 07/31/2012 - 19:22:01 - User ori may have been modified in LDAP or locally, updating key in project(s): reportcard [19:22:35] JeroenDeDauw: well, technically it's no one's job [19:22:39] JeroenDeDauw: OK. Feel free to ask me questions when you take it on (or refer whoever does take it on to me.) [19:23:33] andrewbogott: Thanks, will do :) [19:24:27] Ryan_Lane: it's no ones job to fix stuff that people break on labs? [19:24:40] nope [19:25:05] Ryan_Lane: thanks for scheduling the Gluster upgrade :-) [19:25:10] hashar: you're welcome [19:25:27] JeroenDeDauw: it's someone's job to ensure the infrastructure doesn't continue to do that kind of thing [19:25:47] and I've offered to help you recover any files from the instance [19:26:16] Ryan_Lane: thanks - there is no critical data on there though, I'll just have to run the import script again [19:26:23] * Ryan_Lane nods [19:26:41] it's generally a dev's job to set up demo installation of things, though [19:33:08] petan, wm-bot hates me (I don't appear to be a global admin anymore) [19:41:02] out for now see you tomorrow [19:43:43] !admins [19:43:46] ,admins [19:43:51] * Damianz kicks wm-bot [19:45:12] hey Ryan_Lane, could i request an additional public ip for project editor-engagement? i need it for a setup that needs to be world-cURLable for a short while (couple of weeks). [19:46:16] Is it weird that ssh -A bastion is wmflabs and ssh bastion -A is work in my bash history? [19:52:24] hmm [19:57:46] * JeroenDeDauw also kicks wm-bot, for good measure [19:58:32] Thehelpfulone what [19:59:11] totally actually got around to writing a backup script, now Ryan might hate me every few hours [19:59:36] http://outlook.com < what the freaking hell [19:59:41] Though, who uses hotmail these days [20:00:13] Thehelpfulone I don't even know if you ever were a global admin, anyway what's going on [20:03:11] petan, hmm, I thought you added me back when we were trying to scrub some IRC logs, I was trying to enable recentchanges-on on a channel but it told me permission denied [20:03:30] Ocaasi: hey [20:03:36] we're trying to get a meeting room right now [20:03:47] you have shell access but you didn't list yourself in user db [20:03:57] Thehelpfulone ^ [20:04:14] let me fix it [20:04:17] We all know petan's password is 'I<3PinkUn1corns' [20:04:23] ok, I thought you had to restart it or something to add me? [20:04:25] !password [20:04:25] gfgjoagaewhgAW#YAU_#Y$*U*U^*^%Q#Tqyhe [20:04:27] that's it [20:04:45] Thehelpfulone yes but, there are hot restarts implemented now [20:04:50] so it restart but it doesn't dc [20:05:11] As long as it doesn't take longer than a ping? [20:05:23] it takes like 0.2 sec [20:05:29] for it to boot [20:05:30] :D [20:05:32] Hmm [20:05:41] Compiling the stuff then just loading the modules? [20:05:43] it's c not a python [20:05:54] I just upload a new binary [20:06:04] wtf [20:06:05] C takes a long time to compile, python is really fast at byte compiling :P [20:06:18] Damianz but I compile it before I kill the current one [20:06:20] :P [20:06:29] so I just swap the binaries heh [20:06:40] I should fix my python bot, could reload the modules even down to the socket handling interface. [20:07:34] !log bots petrb: synchronizing wiki list and adding tho to admins file [20:07:41] !log bots petrb: I am talking of wm-bot, of course [20:07:45] petan, ah yes you were telling me, it's on a bouncer now or something? [20:08:15] !log bots petrb: and i will fix my grammar [20:08:26] !log bots petrb: as soon as Damianz fix teh logbot [20:11:22] ok [20:11:25] @whoami [20:11:26] You are root identified by name .*@wikimedia/Petrb [20:11:41] @trustdel .*@wikimedia/Thehelpfulone [20:11:41] User was deleted from access list [20:11:45] @trusted [20:11:45] I trust: petan!.*@wikimedia/Petrb (2admin), .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikimedia/Ryan-lane (2admin), .*@wikipedia/.* (2trusted), .*@nightshade.toolserver.org (2trusted), .*@wikimedia/Krinkle (2admin), .*@[Ww]ikimedia/.* (2trusted), [20:11:55] Thehelpfulone try @whoami [20:12:05] @whoami [20:12:06] You are admin identified by name .*@wikimedia/Thehelpfulone [20:12:06] I just rebooted the bot [20:12:07] @whoisthat [20:12:13] @imurdaddy [20:12:17] lol [20:12:20] thanks [20:12:46] yw [20:13:35] poor bot, it had like 30 days of uptime [20:13:57] I could use it as a proof that c# doesn't leak memory by design [20:16:06] php does [20:16:14] php doesn't matter [20:16:22] it's not long time running thing [20:16:32] I don't know .php scripts that are running 30 days [20:16:39] I do [20:16:41] :( [20:16:44] wikipedia? [20:16:45] :P [20:16:56] I don't even want to see the wp task server stuff :P [20:17:00] sometimes it loads slowly, but it never was 30 days [20:17:22] Reminds me, I need to go write the watermarking thing for work (in python :D) [20:17:37] enjoy the performance :D [20:17:57] maybe you would faster do these watermarks by hand [20:18:15] what's going to be the next topic on the OS list? topposting/bottomposting? [20:18:18] ;-) [20:19:28] Totally not watermarking 100 images with 500+ different watermarks by hand :P [20:19:47] my cat will do that [20:20:15] Lacking pussy power [20:20:21] just let it swim in a color for a while, then jump over and it will ahve unique marks [20:24:12] * Damianz sneaks petan's cat a kilo of catnip and lols in the corner [20:41:58] Ocaasi: hm. I don't see a project [20:42:26] so, we'll need to create one [20:43:13] !resource auth-gateway | Ocaasi [20:43:14] Ocaasi: https://labsconsole.wikimedia.org/wiki/Nova_Resource:auth-gateway [20:44:01] 07/31/2012 - 20:44:01 - Creating a project directory for auth-gateway [20:49:20] Ryan_Lane: thanks, got it [20:49:32] Ryan_Lane: have time to review a change for puppet? [20:49:43] new instance creation is hurting [20:49:48] this will fix it [20:55:49] jeremyb: which change? [20:56:08] !g 17065 [20:56:08] https://gerrit.wikimedia.org/r/#q,17065,n,z [20:56:24] ah [20:58:49] jeremyb: merges [20:58:51] *merged [20:59:04] I haven't forgotten about your others ;) [20:59:17] Ryan_Lane: errrr.... i did have a -1 on it. [20:59:21] I'm neck deep in work right now, but I promise I'll get to them [20:59:24] you had a -1? [20:59:24] needs a followup i guess [20:59:46] yeah, 1 line was a mistake i guess [20:59:49] faidon didn't respond [20:59:50] removing the source file? [20:59:53] yeah [21:00:10] * Damianz gives ryan a sovel and goes back to digging his own pile of work [21:02:03] Ryan_Lane: thanks. /me will do a puppet run now [21:02:04] ;) [21:02:22] ok. did a followup [21:03:02] i saw [21:05:11] JeroenDeDauw: did you get wep sorted? was corrupt [21:05:39] jesusaurus: should be working now. just did a puppet run on your instance. want to try building from scratch again? [21:06:15] * Damianz finds jeremyb a cookie [21:06:59] jeremyb: awesome, yeah, i'll do that in a minute [21:07:17] great [21:18:32] jeremyb: the -1 comment was the one about where c_rehash comes from, right? [21:18:41] paravoid: nope [21:19:01] ah [21:19:04] paravoid: i was just making sure my prior -1 wasn't discarded [21:19:08] the rm source [21:19:14] that was indeed an error on my part [21:19:15] did it get fixed? [21:19:18] yes [21:19:19] thanks a lot for spotting this btw [21:19:24] yay for code review [21:19:27] and yay for jeremyb [21:19:31] heh [21:20:11] paravoid: for some reason the c_rehash seems to use both hash values (the precise and lucid values i had both get linked) [21:20:54] link_hash_cert($fname) if($cert); [21:20:54] link_hash_cert_old($fname) if($cert); [21:21:08] sub link_hash_cert_old { link_hash_cert($_[0], '-subject_hash_old'); [21:21:08] } [21:21:18] oh, is it just perl? huh [21:21:31] yup [21:23:15] even better. [21:23:23] that makes sense [21:23:35] because you can have both libssl1.0.0 and libssl0.9.8 running at the same time [21:23:58] not that likely, but not completely unlikely either [21:24:21] ^demon: sorry that I forgot about you [21:24:37] <^demon> It's ok, I ended up figuring it out so it's all good :) [21:24:55] yeah, I would have immediately thought of -jdk so I could have saved you some time :( [21:25:03] I'll make it up to you! [21:26:23] jeremyb: nope, I'm working on other stuff [21:28:12] JeroenDeDauw: data can be extracted from it if you like. you'll need to rebuild the instance [22:17:02] jeremyb: new precise instance is puppeting successfully