[00:16:33] 07/24/2012 - 00:16:32 - Deleting home directory for platonides in project(s): incubator [00:17:03] I'll clean up my logs as soon as I can get on. Promise. [00:17:15] * Ryan_Lane nods [00:17:26] one sec [00:17:36] fixing it now [00:18:05] Coren: should work now [00:18:34] Yeays. Thanks. off to clean things up [00:18:57] thanks [00:19:34] Erm. I use all of 36K on bastion. But I can't log on bots-3 where my stuff is. [00:19:54] gimme a min [00:20:01] kk. [00:20:12] it's going to take about 30 mins for all instances to get that fix [00:20:22] * Coren nods. [00:22:48] try now [00:23:07] Coren: ^^ [00:23:32] It seems to be trashing the disk, but I'm in [00:23:49] * Ryan_Lane nods [00:24:08] Ryan_Lane: o_O? [00:24:12] ? [00:24:23] 66M /home/marc [00:24:40] hm [00:24:49] maybe it's not your bot that's eating tons of space, then [00:25:03] I /do/ have too much logs, but some 50M of 'em. :-) [00:27:18] How did you arrive at the 1G out of 18 figure? Quotas? Do I have gunk outside my home? [00:28:39] I looked at du for all home dirs [00:28:44] it may not be you [00:29:05] if your home dir is small, then ignore me :) [00:30:45] CSBot is back in business. [00:32:32] you should ask roan to review this [00:32:50] or tim [00:32:52] either or [00:33:04] I +1'd it, but I'd prefer one of the devs to also look at it [00:38:40] Ryan_Lane: can you add me to bastion and bots? [00:44:44] giftpflanze: what's your username? [00:44:49] and why do you want access to bots? [00:48:25] gifti [00:48:51] because i want to run my bot on labs (at least test it) [00:54:26] giftpflanze: ok. I'll add you to bastion, but you really need to get one of the bots community people to add you to the project [00:54:34] and walk you through setting the bot up in the environment [00:54:47] petan: ^^ [00:54:57] ok :) [00:56:35] 07/24/2012 - 00:56:35 - Created a home directory for gifti in project(s): bots [00:57:37] 07/24/2012 - 00:57:37 - User gifti may have been modified in LDAP or locally, updating key in project(s): bots [01:43:14] I can ssh to spage@bastion.wmflabs.org and thence ssh spage@e3 (our instance), both without a password prompt, but on our instance sudo asks for a password. What's it looking for? [01:56:08] spagewmf: Your password for the web interface on labsconsole / Gerrit [05:50:59] 07/24/2012 - 05:50:58 - User ori may have been modified in LDAP or locally, updating key in project(s): editor-engagement [05:51:58] 07/24/2012 - 05:51:58 - User ori may have been modified in LDAP or locally, updating key in project(s): editor-engagement [06:46:19] still can't access my lab instance [09:05:05] hm... [09:51:27] still can't access my lab instance [09:51:44] should I file a bug report or what? [09:56:16] Nikerabbit: i can't really do anything myself [09:56:25] Nikerabbit: maybe paravoid can debug it ? [09:56:46] I know I have some troubles when I create an instance add add puppet classes at creation time [09:56:56] Nikerabbit: do you get access to other instances? [10:04:14] hashar: yes my two other instances have been working fine [10:05:25] Nikerabbit: so that might be a local issue to that instance [10:05:40] Nikerabbit: is that a new instance ? If so you could try deleting it and recreating it without adding any puppet class [10:05:47] and add the class once you get access to the instance [10:08:02] hashar: I didn't modify any puppet classes [10:08:12] only thing different is that I chose the precise image [10:24:20] i'm still not a bastion member [12:33:53] paravoid: there is something severely screwed in labs :/ [12:34:00] ? [12:34:02] some instance I can't sudo on it [12:34:05] my pass get rejected [12:34:14] some other its puppet complaining about LDAP Search failed [12:34:21] hm [12:34:22] same here [12:34:26] ffs [12:34:36] got the message at https://bugzilla.wikimedia.org/show_bug.cgi?id=38639 [12:35:19] nagios: virt0 LDAP CRITICAL 07-24-2012 12:33:45 0d 10h 27m 55s [12:36:04] yeah [12:36:07] that's what I'm looking at [12:36:50] fixed. [12:37:30] fcking opendj and its non-FHS paths [12:38:13] where the fuck are its logfiles [12:38:45] /var/opendj, ooof coourse [13:06:18] paravoid: thanks faidon! [13:06:27] that fixed puppet [13:06:37] :) [13:35:24] !log deployment-prep Running "apt-get dist-upgrade" on apache{32,33} to fix PHP5 using ubuntu packages instead of wmf packages. Upgrade kernel. [13:35:26] Logged the message, Master [13:36:49] I think that's problematic as it is [13:36:54] puppet complains [13:40:03] !log deployment-prep Rebooting all apaches [13:40:04] Logged the message, Master [13:40:24] paravoid: yeah apparently apt-get install returns an exit code of 100 because it is going to downgrade package [13:41:10] yeah [13:41:16] I wonder what to do with that [13:41:28] maybe dpkg can be passed an option to explicitly allow downgrade [13:41:40] or we pin our packages :-] [13:42:15] !log deployment Ran dist-upgrade on deployment-dbdump and rebooting. Will break udp2log loggers. [13:42:15] deployment is not a valid project. [13:42:26] !log deployment-prep Ran dist-upgrade on deployment-dbdump and rebooting. Will break udp2log loggers. [13:42:28] Logged the message, Master [13:45:21] our packages are pinned already, if there weren't, you wouldn't be able to do apt-get upgrade [13:45:28] and have them be downgraded [14:45:09] !log deployment-prep banned, at squid level, a crawler hosted on OVH. Just added the IP to squid.conf blacklist :) [14:45:10] Logged the message, Master [15:00:30] !log deployment-prep banned another /22 at squid level. [15:00:31] Logged the message, Master [15:19:58] paravoid: can you help me to figure out why I can't access my instance? [15:20:05] sure [15:20:07] what's the problem? [15:21:49] Nikerabbit: ^ [15:26:42] !log deployment-prep root@deployment-nfs-memc:/mnt/export# rsync -a --progress --inplace /mnt/export/upload6 /data/project [15:26:43] Logged the message, Master [15:29:35] paravoid: I'm using ssh proxy method to connect [15:29:37] but it says [15:29:37] channel 0: open failed: administratively prohibited: open failed [15:29:37] ssh_exchange_identification: Connection closed by remote host [15:30:17] could you tell me the instance name and give me the relevant section from .ssh/config? [15:32:18] paravoid: translation-memory-3 [15:32:56] paravoid: http://pastebin.com/Kr0xCv6p [15:32:57] !log deployment-prep rerunning rsync with --delete : root@deployment-nfs-memc:/mnt/export# rsync -a --progress --delete --inplace /mnt/export/upload6 /data/project [15:32:58] Logged the message, Master [15:33:48] Nikerabbit: maybe your local username is different from your labs ssh username? I had to add "User hashar" to my sshconfig [15:34:05] hmm it is there .. [15:34:32] Nikerabbit: here is my conf http://dpaste.org/sea9Q/ [15:34:34] similiar [15:34:42] hashar: that works fine for translation-memory-2 and -1 [15:35:16] so, [15:35:20] the hostname doesn't exist [15:35:23] Host translation-memory-3.pmtpa.wmflabs not found: 3(NXDOMAIN) [15:35:30] I suspect the instance was created when LDAP was down [15:35:36] yesterday [15:35:39] tried twice [15:35:44] hm [15:36:12] I choose precise as the image, which should be only difference to other instances [15:36:17] chose [15:36:29] there's also no Nova_Resource page [15:36:34] sounds like a bug in OpenStackManager [15:36:42] and I'm afraid I know little about it [15:36:45] Ryan Lane would know [15:36:58] do you mind pinging him, or even better open a bug in bugzilla about it? [15:37:12] please don't delete the instance, so that he'll be able to debug it further [15:37:28] the symptoms are: no hostname created for the host, no Nova_Resource wiki page [15:38:05] okay [15:38:17] sorry I can't be of more help :/ [15:38:30] thanks! [15:39:17] I'll file a but and Ryan already likes me [15:41:50] !log deployment-prep on deployment-integration, applied {{gerrit|15545}} patchset 7 to te [15:41:51] Logged the message, Master [15:41:52] azeaze [15:42:13] !log deployment-prep on deployment-integration, applied {{gerrit|15545}} patchset 7 to test out the symlinks from /data/project/upload6 to /mnt/upload6 . [15:42:15] Logged the message, Master [15:49:40] New review: Hashar; "Patch set 2: Squash in I4199c995ac37b87a81dc03a0800f718d67622220" [operations/puppet] (test) C: 0; - https://gerrit.wikimedia.org/r/6541 [15:52:04] New review: Hashar; "Ported it to the production branch with https://gerrit.wikimedia.org/r/16514" [operations/puppet] (test) C: -1; - https://gerrit.wikimedia.org/r/6541 [15:54:55] New review: Hashar; "Clean change is https://gerrit.wikimedia.org/r/16515" [operations/puppet] (test) C: -1; - https://gerrit.wikimedia.org/r/6541 [16:12:24] Hi [16:13:00] 07/24/2012 - 16:12:59 - User laner may have been modified in LDAP or locally, updating key in project(s): mobile [16:15:15] Any project about an extension can have an instance to launch the unit test ? [16:15:43] Hi ^demon [16:16:20] thanks for the repository [16:16:43] <^demon> You're welcome. Everything working ok? [16:17:46] for moment yes but the harder will be to create the unit test and I need to create an instance with a triplestore. It's possible ? [16:18:27] <^demon> I don't know about making labs instances, I'm just a mortal there. [16:19:01] lol and my a baby... [16:19:52] You know if an instance can execute test with selenium ? [16:20:03] or only phpunit [17:11:58] 07/24/2012 - 17:11:58 - User laner may have been modified in LDAP or locally, updating key in project(s): mobile [18:12:22] j^: hm [18:12:28] lemme see [18:13:20] fatal: Access denied for user j by PAM account configuration [preauth] [18:13:22] interesting [18:13:53] hm [18:13:57] it doesn't see your groups [18:15:12] wtf [18:15:59] ugh [18:16:11] it seems that validname is screwing something else up [18:16:35] I wonder if the regex is bad [18:16:56] /^[a-z][a-z0-9\-_]*$/ [18:17:19] and here's an example group: project-testlabs [18:18:52] or I wonder if nslcd was just fucking up [18:19:08] because others with the same config are working fine [18:19:29] I think nslcd was just broken [18:20:13] Jul 24 18:18:42 i-0000035a nslcd[23091]: [ef438d] error writing to client: Broken pipe [18:20:16] yeah nslcd was borked [18:20:33] such a buggy service :( [18:21:21] hm [18:21:31] seems like it's broken again [18:22:10] ah [18:22:15] Jul 24 18:21:52 i-0000035a nslcd[24898]: [495cff] group entry cn=project-swift,ou=groups,dc=wikimedia,dc=org name denied by validnames option: "project-swift" [18:22:23] lol [18:22:31] ah hah [18:22:37] the other precise box doesn't have the regex [18:23:29] /^[a-z][a-z0-9_-]*$/ <— that works [18:25:13] Shouldn't that be \- not -? [18:25:39] !log deployment-prep Instances send their syslog again! To deployment-dbdump for now {{gerrit|14090}} [18:25:40] Logged the message, Master [18:26:16] ryan_lane or andrewbogott: if you poking at virt and partman I just ran puppet on brewser and it changed netboot.cfg. [18:26:42] * Ryan_Lane nods [18:26:54] I'm working on nova upgrade, so I'm not touching that stuff right now :) [18:27:00] me too [18:27:07] But not in ways that should matter to either of you. [18:27:19] Oh, when I said 'me too' I meant, 'I am touching that stuff'. [18:27:26] Which I guess is the opposite of what I said [18:27:41] Ryan_Lane: FYI we had an issue with OpenDJ not listening on port 389. Faidon restarted during the SF night ( https://bugzilla.wikimedia.org/show_bug.cgi?id=38639 , assigned to you for investigation, probably low priority) [18:27:46] oh and hello :) [18:28:17] andrewbogott: so I probably just undid or deployed whatever it is you're doing. [18:28:21] * Damianz thinks andrewbogott just blew his mind [18:28:56] hashar: on which box? [18:28:57] Ryan_Lane: also do you know if the API metadata slowdown is fixed? If so may i close the related bug ? https://bugzilla.wikimedia.org/show_bug.cgi?id=38473 [18:29:19] Ryan_Lane: I think it was virt0 , at least nagios reported it as not listening to ldap [18:29:28] maplebed: hm. [18:29:37] it's strange that a single ldap server going down caused issues [18:30:22] maybe instances cache the DNS resolution or something, anyway Nagios reported it being down for 10+ hours [18:30:25] andrewbogott: no big deal, just letting you know. double check netboot.cfg nad make sure it's doing what you want. [18:31:45] maplebed: git production is still how I want it. So, maybe I failed to merge on brewster and puppet just caught you up with my change? [18:32:14] could be. [18:33:09] the change was putting virt100[1-9] onto ceph (before only 1-3 were on ceph) [18:33:34] that's the one. [18:33:48] Hm, there's a commit hook on sockpuppet that merges onto brewster as well. [18:34:00] But maybe that failed for some reason and I didn't notice :( [18:34:21] I've always just run puppet on brewster after making a change to dhcp or partman. [18:36:20] yes, it was virt0 [18:38:37] Ryan_Lane: why do we have NAT rules for 389? [18:38:42] paravoid: do you know if we can find out in puppet which version of a package got installed? I have an issue with openssl behaving differently (0.9.8 vs 1.0.1). [18:38:49] paravoid: opendj runs as non-root [18:39:11] it's port-forward rules [18:39:18] in the nat table [18:39:21] have they ever heard of setuid? [18:39:38] <^demon> So one of the complaints against gerrit is "no stable public API." That is being fixed--they're actively building out some RESTful APIs to expose. [18:39:42] * ^demon grins [18:39:50] Thank the Lord, srsly [18:39:53] ^demon: :-] [18:40:04] (I mean starting as root and dropping privileges, not the perm bit) [18:40:10] yeah, I know [18:40:20] <^demon> RoanKattouw: Ask, and ye shall receive :) [18:40:24] jesus, iptables [18:40:36] we can probably workaround it with fs capabilities [18:40:47] and giving it CAP_NET_ADMIN, although that's a bit too powerful [18:41:13] there's nothing wrong with port forwarding [18:41:19] as long as the table doesn't get purged [18:41:32] it's the solution that requires the least privilege for the service [18:41:34] I wonder why every other service doesn't do it then :) [18:41:47] because it's more of a burden on the admin [18:42:02] and not everyone wants iptables [18:42:12] and because you're needlessly doing NAT [18:42:32] meh [18:42:52] systems have been owned by services that run as root and drop privileges [18:43:07] so, I'd say in the other solution that daemons are needlessly given root [18:43:14] both have their pros and cons [18:44:27] we could also use redir for this [18:44:44] if you really hate the NAT rules ;) [18:48:28] we can use privbind :) [18:48:52] Hello Ryan ? [18:49:01] paravoid: could do that too [18:49:13] paravoid: I'm open to changing it. I'm not married to the port redirection [18:49:17] karima: howdy [18:49:27] karima: so, what's your extension do again? [18:49:34] anyway, next time it fails I'll have a look at iptables [18:49:37] didn't know about it [18:49:42] * Ryan_Lane nods [18:49:44] I did netstat -nap |grep :389 [18:49:48] ah [18:49:48] and didn't see it listening [18:49:53] yeah, it doesn't listen on that [18:50:14] do again? (you want a example ?) sorry for my english [18:50:31] Ryan_Lane: saw the bug I filed? [18:50:38] Nikerabbit: which one? [18:50:50] karima: what is the purpose of your extension? [18:52:22] Ryan_Lane : Ok for the moment I print the Linked Data in a page for example http://en.sparql.pro/wiki/Find_school_in_england [18:53:09] so it's a semantic extension? [18:53:59] I'll make you a labs project called linkeddata [18:54:15] Ryan_Lane : the aim will be after to save data in page of one wiki for all the wikis in several year (and with a lot coffee) [18:54:17] karima: what's your labsconsole username? [18:54:38] https://bugzilla.wikimedia.org/show_bug.cgi?id=38644 [18:54:54] ah. yeah [18:55:31] Ryan_Lane : username : karima [18:56:35] Nikerabbit: weird [18:56:46] it seems the job ran during a race condition [18:56:58] so the page wasn't updated, and the dns entry wasn't added [18:57:57] Ryan_Lane: I tried twice [18:58:06] I'm going to fix it [18:58:08] gimme a se [18:58:09] *sec [19:01:51] Nikerabbit: ok. I fixed it [19:02:13] I need to purge the database of the deleted instances [19:05:47] Ryan_Lane: somehow you added me to bots but not to bastion. mistake? forgotten? [19:08:11] oh? [19:08:16] I thought I added you to bastion and not bots [19:08:17] heh [19:08:25] :) [19:10:47] Ryan_Lane : it's ok ? What the next step ? (and doc to read ?) [19:11:10] I'm working on a problem with a developer right now. give me a little bit [19:11:23] ok no problem [19:50:31] * Ryan_Lane groans [19:51:09] seems the config reverted [19:51:11] which is weird [19:51:43] wtf [19:51:45] ah [19:51:46] that's why [19:51:55] you need to cherry-pick my change in [19:52:07] https://gerrit.wikimedia.org/r/#/c/16537/ [20:00:21] j^: ok [20:00:26] you should be able to now [20:06:54] because puppet ran again [20:07:13] ok, try now [20:30:45] Ryan_Lane: have you time to add me to bastion, now? [20:30:52] oh [20:30:53] sorry [20:30:53] yeah [20:30:56] :) [20:30:58] what's your labsconsole name? [20:31:02] gifti [20:31:17] eh? it works forme [20:31:30] ugh [20:31:32] or does it? [20:31:38] j^: mind pushing in a change? [20:32:10] actually, I'm there already [20:34:03] https://gerrit.wikimedia.org/r/16549 [20:34:06] j^: ^^ [20:38:58] Ryan_Lane: almost ready to get an ext IP for Echo :) [20:39:36] werdna: ok, let me up your quota [20:39:41] thx :) [20:39:56] paravoid: ok. I have the steps down [20:40:03] all command line, and scriptable :) [20:40:14] it'll be an easy bash script [20:40:56] I have the steps! `history | tail -n 20 | paste` :D [20:40:59] I just ran a migration on an instance and it worked perfectly [20:42:17] *suspire* [20:42:48] giftpflanze: sorry. keep forgetting to add you [20:43:00] yeah [20:43:12] just added you [20:43:18] it'll take a min or so for your key to be added [20:43:38] *hug* thank you [20:43:57] * Damianz gives giftpflanze a cookie and a seat [20:43:59] 07/24/2012 - 20:43:59 - Created a home directory for gifti in project(s): bastion [20:44:05] :) [20:44:59] 07/24/2012 - 20:44:58 - User gifti may have been modified in LDAP or locally, updating key in project(s): bastion [20:46:02] it even works, great [20:47:11] heh [20:54:22] 07/24/2012 - 20:54:22 - Updating keys for purodha at /export/keys/purodha [20:56:08] paravoid: ok. there's a script in /root on virt0 called cold-migration [20:56:33] you need to ssh directly into virt0 with a forwarded agent to use it [20:56:47] it's ./cold-migration [20:57:19] instance i-00000057 would be just 00000057 [20:57:40] so, ./cold-migration 00000057 virt8 [21:14:39] Not sure what the case is on labs, but the message caches will likely need rebyuilding [21:38:23] Ryan_Lane : I was disconnect [21:41:09] Hi All. Can I ask how you create instances (i.e virtual machines) for a project? Is it automated or do you manually create the VMs? [21:41:32] go to a project, hit add instance if you have rights to do so [21:41:51] basically you need sysadmin rights in a project, from tehre you create instances manually as needed [21:42:04] In theory applying puppet classes so they automagically get installed with st00f [21:43:38] Okay. Where can I take a look at the Puppet class for that? Just out of curiosity. [21:44:13] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=summary are the actual puppet manifests [21:44:18] They are added to instances on the configuration page [21:47:03] karima: no worries [21:47:29] adminxor: well, you go to manage instances, then add an instance in the project [21:47:34] !instances | adminxor [21:47:34] adminxor: need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [21:47:53] !puppet | adminxor [21:47:54] adminxor: learn: http://docs.puppetlabs.com/learning/ troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html [21:47:55] ok ;) [21:47:59] hm [21:48:06] that's not what I meant [21:48:11] @search puppet [21:48:11] Results (Found 6): $realm, $site, ryanland, instanceproject, puppet, git-puppet, [21:48:19] ryanland? [21:48:20] !ryanland [21:48:20] in case you want to get to wonderfull land of labs use portal we call bastion, you will see amazing world where vm's runs happily and nfs friends with ntfs, puppets are fresh and gerrit is ugly :O [21:48:26] hahaha [21:48:33] !git-puppet | adminxor [21:48:33] adminxor: git clone ssh://gerrit.wikimedia.org:29418/operations/puppet.git [21:50:08] Ryan_Lane: Thanks! :) [21:50:16] yw [21:54:20] Well, I just wanted to have a look at the actual code(do you call it code or something else?) for puppet. [21:58:26] !log deployment-prep j: run mwscript rebuildLocalisationCache.php --wiki=aawiki --threads=2 [21:58:31] Logged the message, Master [22:03:54] Ryne_Lane: Is this project abandoned? I was wondering if you have centralized syslog servers. Is this project about that? [22:03:56] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Syslog-collection [22:04:14] umm [22:04:16] not terribly sure [22:04:21] lemme see [22:04:26] oh [22:04:27] I made that [22:04:28] hahaha [22:04:40] yes, it was for centralized log collection of some variety [22:04:45] I think I was looking at logstash [22:05:14] adminxor: is that something you're interested in? [22:05:30] Yes. [22:06:51] well, let me add you to that [22:07:12] Thank you so much! [22:07:33] But, I don't think it has any instance. [22:07:37] it doesn't [22:08:00] so, try out some different solutions, and demo some [22:08:09] then we can puppetize whichever one is best [22:08:29] I was hoping for some kind of system that would collect the logs, then let us do saved searches, analytics, etc [22:08:40] saved searches being the big one [22:08:58] 07/24/2012 - 22:08:58 - Created a home directory for adminxor in project(s): syslog-collection [22:09:00] we have greps in production against log files, and that's kind of crappy [22:09:49] Let me dive into it then :) [22:09:58] 07/24/2012 - 22:09:58 - User adminxor may have been modified in LDAP or locally, updating key in project(s): syslog-collection [22:10:10] cool. have fun with it [22:10:17] let me know if you need any help [22:10:31] remember, open source only ;) [22:12:15] Of course! Do I have enough privilege to add a description to the project? I was having hard time browsing through the peojects without description. [22:20:55] What time is it in US ? [22:21:18] depends on time zone i guess [22:21:45] 2221 [22:22:01] (+/- timezone modifiers) [22:22:10] hmm... New york for example [22:22:24] is -4, iirc [22:22:39] so, 1822 [22:22:54] right now, yes [22:23:02] of course, it depends on daylight savings [22:23:18]