[00:01:24] If my tool (the zoomviwer) needs to store large amounts of data (the processed images) I would use /data/project for that, right? [00:01:56] also for the upcoming webtools project will the tools be shared or in the users' home dirs? [00:02:07] I guess that is an unlaid egg, but anyways [00:02:26] yeah. /data/project [00:02:31] it's quota'd at 300GB [00:02:39] if you need that quota raised, let me know [00:02:46] ok [00:03:00] I'd *much* prefer tools used shared space, rather than home directories [00:03:34] me too, but groups may be necessary to control access [00:03:40] there are groups :) [00:04:12] yeah, I noticed I can add them locally on my instance (I was worried about the group management being ing LDAP entirely) [00:04:20] well, there are also project groups [00:04:36] you can add users and groups locallty [00:04:38] *locally [00:04:39] this is getting confusing. Do you mean nova project? [00:04:43] yes [00:04:54] on your instance, type "groups" [00:04:57] as your users [00:04:59] *user [00:05:03] yeah [00:05:07] I saw those [00:05:16] there's a posix group for every nova project [00:05:40] so per-tool groups would be loacl to the instance the tool is running on [00:05:54] I imagein groups like tool-zoomviewer [00:06:09] yeah, that should likely be managed by puppet, though [00:06:09] with a few maintainers [00:07:14] Damianz, on the bots project I cannot see any bots except cluebot in the /data/project tree [00:07:22] oh right, he noms [00:07:30] I think they just started moving to project storage [00:07:38] well I gotta go, too [00:07:39] ok [00:07:41] it wasn't reliable until about a couple montha sho [00:07:42] see you later [00:07:43] *ago [00:07:45] see ya [00:07:51] thanks for the help [00:23:28] Is there someone here I can talk to about getting a labs account? [00:28:26] Ryan_Lane: do you know who I can talk to about getting a labs account? [00:31:52] legoktm: yep [00:31:58] @search account [00:31:58] Results (Found 4): credentials, account, account-questions, accountreq, [00:32:03] !accountreq [00:32:03] in case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [00:32:23] !accountreq | legoktm [00:32:23] legoktm: in case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [00:32:37] Thanks I'm reading that now [00:33:28] Do I need to set up a git/gerrit account first? [00:33:54] legoktm: it's the same thing (gerrit/wiki/labs account) [00:34:04] it's all in LDAP [00:35:28] So would you be able to create an account for me mutante? You're listed as one of the people who is recommended to contact.. [00:35:34] my preferred username would be "legoktm" [00:35:38] (for everything) [00:36:08] yes [00:36:11] email: "legoktm.wikipedia@gmail.com" [00:37:03] legoktm: and you never had an svn account, right [00:37:10] nope [00:41:42] yes [00:41:49] err wrong channel [00:41:53] legoktm: A randomly generated password for Legoktm has been sent to legoktm.wikipedia@gmail.com. [00:42:29] * legoktm checks his email [00:42:44] got it [00:43:45] * Damianz tries to figure out how much drugs some people on toolserver-l are smoking [00:44:23] lol [00:44:32] mutante: So does this give me shell access to run bots/set up tools? Or do I have to do something else for that? [00:47:21] legoktm: we needed 2 additional things, just gave you the shell permission and added you to the bastion project [00:47:51] legoktm: now go here https://labsconsole.wikimedia.org/wiki/Help:Access#Initial_log_in_and_password_change [00:48:04] like the password reset and SSH key uploading part [00:48:30] Ok [00:49:02] after the SSH key part you can try logging on to bastion.wmflabs.org [00:49:04] Just curious, why should the password I set be as strong or stronger than my SSH key? [00:49:05] ok [00:49:10] oh, actually, let me also add you to the bots project [00:49:40] Because it's used to get priveleged access, your key is for user level access [00:49:58] ok [00:50:54] Successfully added legoktm to bots. [00:51:03] !log bots added new member legoktm [00:51:05] Logged the message, Master [00:52:02] "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you. " [00:52:07] andrewbogott: 2012-09-26 00:51:24 TRACE nova.notifier.list_notifier ClassNotFound: Class WikiStatus could not be found: No module named mwclient [00:52:14] legoktm: please log out and back in [00:52:20] it's a known bug on initial login [00:52:21] ok [00:53:08] Ryan_Lane: Which host is throwing that? [00:53:14] virt5 [00:53:17] I just added it to the pool [00:55:46] 09/26/2012 - 00:55:46 - Created a home directory for legoktm in project(s): bots,bastion [00:55:56] 09/26/2012 - 00:55:56 - Creating a home directory for legoktm at /export/keys/legoktm [00:57:05] Awesome, I'm in [00:57:07] legoktm: see above, the bot tells [00:57:13] :) [00:57:44] legoktm: now you want to connect to an instance within the bots project from there [00:58:00] legoktm: don't run bots on the bastion host directly [00:58:38] ok [00:59:07] legoktm: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots [00:59:27] thanks [00:59:32] Ryan_Lane: Did you try restarting nova-compute? As far as I can see, all the modules are present. I can load mwclient and WikiStatus from cmdline python, no problem. [00:59:44] hm [00:59:45] All I can think is that Nova is running with a stale python path [01:00:02] Although in theory the puppet class restarted nova-compute after installing things [01:00:03] is the module being added without restarting nova-compute? [01:00:06] h [01:00:07] err [01:00:07] ah [01:00:45] 09/26/2012 - 01:00:44 - User legoktm may have been modified in LDAP or locally, updating key in project(s): bots,bastion [01:00:56] 09/26/2012 - 01:00:56 - Updating keys for legoktm at /export/keys/legoktm [01:01:26] restarting it [01:01:35] I'll reboot the instance to see if it fixes things [01:01:55] Wait, I'm wrong... [01:01:58] it doesn't bump nova-compute. [01:02:05] So just do a service start nova-compute, I bet that'll fix it. [01:02:10] And meanwhile I'll fix the puppet class. [01:02:22] ah. cool [01:02:23] thanks [01:02:39] 2012-09-26 01:02:15 DEBUG nova.notifier.list_notifier [req-f581a41e-56d5-4db8-a396-8f870e5fb1eb laner testing] wikistatus: Writing instance info to page http://labsconsole.wikimedia.org/wiki/Nova_Resource:i-0000044c from (pid=20901) notify /usr/local/lib/python2.7/dist-packages/wikinotifier.py:228 [01:02:39] would be nice if we could have a button on console that went 'HEY NOVA, UPDATE ME' and it ran the hook again for old stuff that's just wrong. [01:02:52] that may be possible [01:03:09] !log bots test [01:03:10] Logged the message, Master [01:03:11] it needs to be something that can trigger an event [01:03:26] a reboot will for sure do it [01:03:41] That's like using a sledge hammer to eat your breakfast with [01:03:44] Yeah. I tried resizing an instance from size x to size x but it was too clever for me. [01:04:01] heh [01:04:11] andrewbogott: is an event fired on metadata changes? [01:04:36] we can update a metadata key with the current date and time [01:05:01] Nope. I updated all the running instances by hacking in a special event that happens when nova-compute starts up and enumerates instances. [01:05:07] ah [01:05:10] (Which I then removed.) [01:05:12] heh [01:05:16] yeah. that could get spammy [01:05:36] Hmm does bind seriously still lack geoip support -.- [01:05:43] yes [01:05:46] It would be reasonable to add an event for metatdata changes. I'm just reluctant to have our running code diverge from the dpkg [01:05:54] yeah [01:06:00] well, we should upstream it [01:06:16] * Damianz stabs bind and goes to look at nsd [01:06:17] not into backports. I doubt they'd take that [01:06:26] Damianz: In theory the pages should just always be magically up-to-date. If you find an event that isn't hooked properly I can (maybe) add it to the list of things to track. [01:06:43] they are way more reliable now than before [01:06:49] andrewbogott: Even for instances that have/had missing pages and couldn't even be rebooted from labsconsole? [01:07:27] Damainz: I guess not, athough in that case all the page would say is "I am not an instance" [01:07:39] \o/ virt5 is working properly [01:07:46] now to clean some space on virt6 [01:07:48] lol [01:07:56] so removing/adding pages is still done in the wiki? [01:08:21] only for a very small amount of info [01:08:40] mutante: I'm trying to `ssh bots-3` but I'm getting "Permission denied (publickey)." [01:08:42] Still should diaf and moved to nova :) [01:09:00] puppet info can't be [01:09:14] until we handle puppet via nova [01:09:29] Damianz: That which nova knows about is updated when 'interesting' things happen (e.g. reboots, resizes, moves) [01:09:42] legoktm: did you see the part about ProxyCommand yet? https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [01:09:44] Things outside of Nova's view are updated by OSM [01:10:08] legoktm: if you use that you should be able to directly ssh bots-3 from your home shell [01:10:20] I still dislike OSM being anything more than an api client, the logic is just meh there... progress I guess though [01:10:34] agreed [01:10:54] mutante: I was already logged into bastion.wmflabs.org, and from that shell i was trying to ssh into bots. Is that not possible? [01:11:07] legoktm: Did you forward your key? [01:11:19] How do I do that? [01:11:22] !access | legoktm [01:11:22] legoktm: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [01:11:31] Damianz: i was trying to make him use ProxyCommand instead of forwarding key [01:11:42] it's more secure [01:11:43] He still needs to forward the key or have an agent. [01:11:52] not with proxycommand [01:12:00] Really? [01:12:03] really [01:12:05] So which one should I be using? [01:12:07] you don't forward yourkey at all [01:12:14] proxycommand is more secure [01:12:25] I'd go with that if you can get it working [01:12:29] Alright I'll try that [01:12:29] meh [01:12:39] both ways are described on wiki though [01:12:43] https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [01:12:45] I forward my key to my gateway host and use proxycommand because I'm too lazy to ssh into it all the time. [01:12:46] yep [01:13:15] Also too lazy to setup openvpn, though I'd love openvpn to labs because socks is a PITA without a custom extension which is horrid. [01:13:30] heh [01:13:34] I like foxyproxy [01:13:44] I couldn't get it working properly in chrome :( [01:13:52] Also dns forwarding over socks in chrome sucked. [01:14:09] i keep hearing all these things that do not work in chrome.. [01:14:23] I couldn't get it working in chrome either [01:14:31] mutante: At least it loads, unlike firefox. [01:14:43] firefox works perfectly for me ;) [01:14:47] it does [01:14:52] Do you have like 100000000gb of ram? [01:15:07] I have 8GB [01:15:19] Yeah I have 4... firefox will happily crash my mbp. [01:15:22] heh [01:15:46] It works "ok" on my desktop with 12. [01:16:03] I was able to login with ProxyCommand [01:16:28] http://www.tomshardware.com/reviews/macbook-air-chrome-16-firefox-9-benchmark,3108-13.html [01:16:38] Chrome on Mac actually uses more than Firefox :p [01:16:44] Who has an air? It doesn't even have a cd drive :P [01:17:24] too thin to plug in ethernet [01:17:32] legoktm: great [01:17:35] Maybe with 1 tab open, but it loads instantly not in 5min and it doesn't memory leak like a whore [01:18:00] if my pages dont load, its the network connection :p [01:18:06] especially over here [01:18:26] I need better home internets tbf [01:18:30] Ryan_Lane: Why do we 'merge' on sockpuppet rather than rebasing? Doesn't merging cause sockpuppet to have a different commit history from gerrit? [01:18:40] Especially as the office has gb connectivity now... [01:20:31] paravoid: -rw------- 1 root root 227G Sep 26 01:19 console.log [01:20:42] LOL [01:20:44] andrewbogott: yeah. it does [01:20:45] Only little [01:21:12] Ryan_Lane: If only I could remember where the wiki page is with those instructions, I would change them [01:21:14] paravoid: [2504862.835828] request_module: runaway loop modprobe binfmt-0000 [01:21:24] Damianz: does it?:) [01:22:22] Well it's like 800mb, gb from the pop just over the road and down our backbone to the other offices [01:22:55] andrewbogott: paravoid: merge rather than rebase? [01:22:58] paravoid: any ideas? [01:23:47] I feel like merge is only for 'upstream' reps, and I think of sockpuppet as being downstream... [01:24:07] Actually the cleanest thing might be to check out a fresh branch, even. [01:24:09] Damianz: ooh..you are not talking about wmf office then:) [01:24:25] It shouldn't have any changes anyway so merge should just rebase no? [01:24:35] mutante: Heh no, you'd never hire me :P [01:24:40] Ryan_Lane: about? [01:24:57] another: -rw------- 1 libvirt-qemu kvm 231G Sep 26 01:24 console.log [01:25:30] wouldn't using logrotate on those be acceptable ;P [01:25:59] Damianz: I would think so, and yet I feel like it doesn't... [01:26:09] Maybe it does and I'm just confused by all the nonsense that gerrit adds to the log [01:26:34] Gerrit does add a load of crap [01:27:27] Damianz: the giant console logs were due to a bug [01:27:34] Confirmed: Every time we merge to sockpuppet we get a 'Merge remote-tracking branch 'origin/production' into production' entry in the git history [01:27:45] And, also confirmed, that git tree is 'clean' w/respect to the upstream. [01:27:54] Hmph [01:43:32] Few questions since I couldn't exactly find them/understand it in the help pages: Do I need to use puppet to install things (like python3 for example)? Is that something I can do? [01:45:11] Also, to run a bot, is there any specific job queue like the toolserver has? Or can I just run it in a screen? [01:45:46] no, but it's advisable [01:45:52] no, write an init script or use screen [01:46:13] maybe when I get some time I can put some effort into this [01:46:27] we really do need a scheduler [01:46:30] Who votes we chop Ryan_Lane in half? [01:46:43] Yeah we really need to sort bots as well :P [01:46:46] Sorry Damianz, so how do I install things? [01:47:09] Can I just sudo apt-get install package? [01:47:25] If you have sudo access, if you don't ask someone for it or to install stuff. [01:47:34] Hmm it's nearly 3am, bedtime me thinks [01:47:43] I'm guessing I don't have sudo access... [01:48:08] oh. i do [01:50:05] are my home directories shared between instances? [01:50:41] in the same project, yes [01:50:55] ah, how convenient [01:53:24] please don't use home directories, though [01:53:27] use project storage [01:53:31] it's at /data/project [01:53:57] it's automounted, so if it doesn't show up, just try to access it [01:53:59] it'll mount itself [01:54:05] Can I just create a new folder there and stick my stuff in it? [01:54:07] it's also shared between instances [01:54:14] if you have root, yes [01:54:47] should stop looking at varnish documentation and go to sleep, seems a more boring option though [02:00:01] Ryan_Lane, was that directed at me? [02:00:32] I was mainly worried about my ssh configuraration and stuff like that [02:04:05] dschwen: yeah, meant for you [02:04:14] you can create directories as root [02:04:18] I got the message [02:04:25] like two hours ago ;-) [02:04:34] heh [02:08:22] this is neat, my native parts of wikiminiatlas now build on labs [02:08:33] just need DB access now [02:08:40] and replicated dbs [02:08:48] and the OSM stuff [02:08:57] almost there ;-) [02:10:32] heh [02:10:46] we're going to be doing OpenStreetMap in production [02:12:54] we're talking about it in the office right now, in fact. heh [02:13:16] replicated dbs should be available in 2-3 months or so [02:42:20] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=587279 edit summary: [02:43:28] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=587280 edit summary: [02:46:24] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=587281 edit summary: [02:58:45] I hope you render nicer maps than the default osm style [03:36:37] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Hazard-SJ link https://www.mediawiki.org/w/index.php?diff=587285 edit summary: Fix typo [05:15:42] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=587296 edit summary: de-strike homes [05:20:04] is it a good time to move some tools from toolserver to labs? [06:07:16] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587305 edit summary: [06:09:16] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587309 edit summary: [06:09:47] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587311 edit summary: [06:11:50] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587314 edit summary: [06:15:58] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587316 edit summary: [06:46:53] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=587375 edit summary: [07:07:59] Change on 12mediawiki a page Developer access was modified, changed by Twikibstud link https://www.mediawiki.org/w/index.php?diff=587391 edit summary: [07:24:13] !log test [07:24:13] Message missing. Nothing logged. [07:24:39] hello [07:24:44] hi [07:25:52] Damianz how did you silenced bot? [07:25:55] !ping [07:25:55] pong [07:26:01] !logging [07:26:01] To log a message, use the following format: !log [07:26:13] @infobot-ingore- log [07:26:20] @infobot-ignore- log [07:26:20] Item log was not found in list [07:26:23] lol [07:26:28] @infobot-ignore+ log [07:26:28] Item log was inserted to ignore list [07:28:06] !l [07:28:07] There are multiple keys, refine your input: labs, labsconf, labsconsole, labsconsole.wiki, labs-home-wm, labs-morebots, labs-nagios-wm, labs-project, leslie's-reset, link, linux, load, load-all, [07:28:10] hmm [07:28:11] weird [07:28:16] !!log [07:28:16] petan needs a new hobby :P [08:48:51] !lo [08:48:51] There are multiple keys, refine your input: load, load-all, [08:50:27] @help infobot-ignore+ [08:50:27] Info for infobot-ignore+: Insert a new string to ignore list for infobot [08:50:33] Damianz ^ [08:51:21] :P [08:59:07] !log integration manually installing gerrit on -jenkins [08:59:08] Logged the message, Master [09:12:10] !meep is [09:12:11] Key was added [09:12:13] !meep [09:12:13] [09:12:16] !meep del [09:12:16] Successfully removed meep [09:12:27] !meep is || [09:12:27] Invalid symbol in the key [09:12:37] !meep is blah | blah [09:12:37] Invalid symbol in the key [09:14:32] !log integration Starting Gerrit Code Review: OK !!!!! [09:14:33] Logged the message, Master [09:16:42] @help [09:16:42] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.22.1 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [09:16:48] !meep is blah | blah [09:16:48] Key was added [09:16:50] !meep [09:16:51] blah | blah [09:16:54] !meep del [09:16:54] Successfully removed meep [09:16:58] !mee|p is test [09:16:58] Invalid symbol in the key [09:17:33] !log bots fixed bug in wm-bot [09:17:33] Logged the message, Master [09:27:06] !log integration yeah gerrit on http://integration.wmflabs.org/gerrit/#/q/status:open,n,z !!! :-] [10:46:56] anyone around ? I created a new security group with a rule in it [10:47:04] I can't find out how to apply the security group to an instance [10:47:17] it is available on instance creation, but can't find out how to change it later [10:48:26] I don't think you can right now [10:48:39] going to add it to the project default group so [11:06:22] hashar you can't [11:08:19] that works also [11:12:32] bah [11:12:37] any idea how to apply the rule ? [11:12:45] I have made the modification but the instance does not have the rule :/ [11:30:22] anyone available that can help me clarify access rights to projects and instances? [11:32:34] out to get a take away [11:46:37] Bleh he left [11:46:54] dan-nl: As in user access? [11:51:09] if hashar re-appears before I get back from lunch tell him to reboot the instance and it will probably work [11:52:08] Damianz: yes, trying to sort out how user rights are assigned ... working on the Help:Getting_Started page. you first get an account and are assigned to the bastion project, but what if i want to be assigned to an existing project and help out, what do i do? [11:52:42] Damianz: contact anyone on the current project? [11:52:45] Ok, to get assigned to an existing project you just need to ask a member. Any member can add another member to a project - by default that gives them ssh and sudo access [11:52:54] Damianz: or is there a special request page? [11:53:33] Damianz: great that's clear now ... and finding the members? the best place to look is here? [11:53:39] Damianz: https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D/searchlabel%3Dprojects/offset%3D0 [11:53:42] Some projects have custom sudo rules to say for example on sysadmins can sudo up, there are also 2 other 'levels' sysadmin and netadmin. The first lets you add/delete/configure instances, the later add/manage ips. You need to be a member and ask someone in the group to add you. [11:53:53] Yeah [11:54:05] Damianz: perfect ... thanks [11:54:09] On some projects if you click the name it gives more info [11:54:18] for example https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots [11:54:33] says to bug petan or sometimes I give people access if I'm not busy :D [11:54:41] hi [11:55:05] Damianz did you read about ignore list [11:55:11] @infobot-ingore+ [11:55:13] etc [11:55:14] Yeah, the @ignore thing [11:55:16] :D [11:55:28] @help infobot-ignore- [11:55:28] Info for infobot-ignore-: Remove item from ignore list of infobot [11:55:31] Didn't know that existed so I just deleted the key as it was spamming [11:55:39] you deleted 3 keys :D [11:55:42] Still need to add logging to labs-morebots as it randomly can't edit the wiki [11:56:15] Damianz in worst case use @suppress-on [11:56:25] :D [11:56:25] That mutes it for the channel? [11:56:30] @suppress-on [11:56:30] I will not talk in here until you disable this [11:56:34] !ping [11:56:36] wm-bot: HEY [11:56:36] @help [11:56:38] :D [11:56:43] @suppress-off [11:56:43] Output will be no longer suppressed now [11:57:09] Now really going for lunch as it's 1pm heh [12:14:51] Damianz: here's the page with the revisions, hopefully this is clearer now ... https://labsconsole.wikimedia.org/wiki/Help:Getting_Started [12:25:46] 09/26/2012 - 12:25:46 - User hashar may have been modified in LDAP or locally, updating key in project(s): testswarm,bots,bastion,gerrit,deployment-prep,jenkins,wikidata-dev,integration [12:25:56] 09/26/2012 - 12:25:56 - Updating keys for hashar at /export/keys/hashar [13:23:21] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=587559 edit summary: /* User:Twikibstud */ [13:28:06] o.O [13:28:19] sumanah you have new account name on wiki? [13:28:26] petan: yes [13:28:36] I think sumana was easier to remember :) [13:28:45] petan: I know. It was. [13:28:53] why you decided to change it [13:29:27] Until 24 September 2012, I was using my Sumanah user account for both official editing (mostly to talk pages) and personal edits to articlespace. Starting 24 September 2012, User:Sharihareswara (WMF) is my user account for all my editing in my WMF capacity. [13:29:28] I am one of few people who remember your full name :D so I don't really have problem with that, just wondering [13:29:42] It's better to separate out personal and WMF accounts [13:29:43] aha [13:29:45] ok [13:29:54] I guess you didn't see that note on my userpage [13:30:00] no... [13:30:04] I just noticed on irc [13:34:00] It seemed like it was a good idea but maybe I should have just used "Sumanah (WMF)" because it's shorter and easier for people to type [13:34:28] but it's easier to accidentally select 1 instead of the other; "Sharihareswara (WMF)" is more distinct [13:40:37] Sharihareswara is rather a mouth full :P [13:41:48] Not as bad as my colleague 'Unai' who's name is pronounced as it's written [13:47:41] !log glam-gwtools added initial instance for the project [13:47:42] glam-gwtools is not a valid project. [13:48:06] !log glam added initial instance for the gwtools project [13:48:07] Logged the message, Master [14:14:33] sumanah: i have trouble typing "Sharihareswara (WMF)" without cut & paste :( [14:14:43] * aude can type Sumanah (WMF) [14:19:48] I type "su" then press tab [14:20:03] liangent: not on the wiki [14:20:04] sumanah is usually the only person here whose name starts with "su" [14:20:27] yeah, I'm not changing my IRC nick thank goodness [14:20:39] :) [14:22:10] I typed harihareswara in enwiki trying to find source of this name but found some more "Sumana Harihareswara"s instead... [14:22:25] how many are there? [14:22:50] Uncle Morty's Dub Shack ... author Sumana Harihareswara [14:22:57] Yep, that's me [14:23:14] I should say: yes, that's me [14:23:26] and Spamusement! ? [14:24:02] Yes, I enjoyed Spamusement and possibly wrote about it [14:25:10] * sumanah looks [14:25:17] yes, I did mention it in that newspaper column I wrote [14:25:43] hmm [14:26:05] and [[Ken Liu]] ? [14:26:16] "Single-Bit Error", Thoughtcrime Experiments, edited by Sumana Harihareswara and Leonard Richardson, 2009 (read) (buy). [14:26:25] Yes, I co-edited that anthology [14:26:52] and you're so famous :) [14:27:10] ha! I'm not notable enough to have *my own* page [14:27:20] (I assume) [14:28:13] It can be arranged [14:28:16] However, I believe I am the only Sumana Harihareswara in the world, so anything you see about me is either about me or it is a lie [14:28:55] I feel like changing my name by deed poll now :P [14:29:12] Damianz: you mean someone could artificially inflate my own Notability? I guess if you have insider access to the Nobel or Pulitzer committee :) [14:29:51] Technically notability is pretty much the same as seo in theory so we'd just have to have everyone talk about you as being an expert :P [14:30:07] it'd be easy for her to increase her notability [14:30:17] just enter with at and start shooting [14:30:25] Aieee! NEver! [14:30:30] you may not like the consequences of such crazy action, though ;) [14:30:34] Platonides: Only at a bat man premier [14:30:43] I just realized that it might be taken the wrong way if I said I "recoiled" from that suggestion [14:32:43] the Kannada Wikipedia page about my dad is of course longer than the en.wp page about my dad [14:32:44] https://kn.wikipedia.org/wiki/%E0%B2%B6%E0%B2%BF%E0%B2%95%E0%B2%BE%E0%B2%B0%E0%B2%BF%E0%B2%AA%E0%B3%81%E0%B2%B0_%E0%B2%B9%E0%B2%B0%E0%B2%BF%E0%B2%B9%E0%B2%B0%E0%B3%87%E0%B2%B6%E0%B3%8D%E0%B2%B5%E0%B2%B0 [14:33:58] Pretty squares [14:34:19] why aren't those photos at commons? [14:34:25] * sumanah does not know [14:34:29] we should delete them for lack of license,,, [14:34:30] * sumanah does not edit Kannada Wikipedia [14:34:41] Damianz: doesn't webfont work for you? [14:34:56] Apparently not [14:35:19] maybe he is using IE and disabled webfonts on purpose [14:35:25] totally [14:35:36] Actually I do have ie under wine. [14:35:56] I was refering to a recent vulnerability [14:36:16] Ie is ful of them :P [14:36:22] well, only the publication is recent, it has probably been there for a long time :) [14:36:29] Best recent one was resetting phones to factory defaults via tel: links [14:37:25] lol [14:37:40] Damianz: ha, originally I did not understand when you said "pretty squares" - I thought you were saying that my dad looked "pretty square", like he was a nerd (which he was) [14:37:46] xD [14:37:55] Nah, he does look pretty old though :P [14:38:07] yeah, he was [14:38:09] harihareswara / harihareshwara? [14:38:10] the images look old [14:38:20] liangent: the transliteration from Kannada is inconsistent [14:39:10] and with "Harihareshwara" I can find more on enwp [14:40:23] by the way, I do think "Single-Bit Error" and the rest of "Thoughtcrime Experiments" are pretty good, so if you like short scifi and fantasy stories, I recommend them [14:40:39] and liangent Ken Liu writes both in English and Chinese, in case you prefer to read stuff in Chinese [14:41:52] it's funny, I don't think about my dad often, but looking at his page right now makes me miss him a little [14:42:43] In another 10years you'll be able to print a full 3d render and have ANN powered conversations [14:42:51] * Platonides hugs sumanah [14:42:58] Aww, thanks [14:43:02] * sumanah hugs Platonides back [14:43:46] Damianz: ha! he used to say he wished he had a wax statue of me around to talk with when I wasn't there (Dad had a weird sense of humor; I probably pun a lot because of him) [14:43:50] * Damianz puts the mood music on and lights the candles [14:44:07] hi Merlissimo [14:44:23] hi sumanah [14:44:33] Merlissimo: thanks for responding re https://bugzilla.wikimedia.org/show_bug.cgi?id=40428 [14:44:46] sumanah: I tried to look for the Chinese title of "Single-Bit Error" but then realized a literal translation works [14:44:55] Bleh, I swear BZ is impossible to search [14:45:38] sumanah: if there would be a servby in the reponse it would also be in the header [14:45:50] liangent: also check out http://kenliu.name/ .... you may have already seen http://kenliu.name/stories/single-bit-error_chinese [14:46:24] How can it be so hard to 'get all entries I've commented on'!? seriously [14:46:37] Damianz: it's on the Advanced search page, want me to run it for you? [14:46:37] sumanah: I just remember I saw this title somewhere [14:46:55] I tired the advanced page... it said none which is wrong [14:47:10] Merlissimo: you should probably just say that in the bug since I'm out of my depth here :) [14:47:16] Damianz, which is your mail? [14:47:22] Damianz: hm, are you sure you deselected all the other constraints? [14:47:31] Damianz: like product, component, resolutions, etc? [14:47:45] the solution is probably to have someone run that query to all the servers [14:47:53] but it needs to be done from the cluster [14:48:02] Bleh, I see. If I use my email it works, was trying my username .... [14:48:24] bugzilla identifies you with the email [14:48:46] I really wish it just used openid to identify me with CentralAuth [14:48:57] Damianz: I would like single sign-on [14:49:07] Damianz: it would be a lot friendlier to new volunteers [14:49:32] Yeah, it just sucks because we have a non-standard, hacked together system for auto logging people into wikis. [14:49:33] Damianz: imagine it: no more "get a wiki account, get a Labs/Gerrit account, get a Bugzilla account" [14:50:11] Labs is harder because of the ldap requirement, though there's no reason we couldn't add an account then when they wanted shell ask for the other details (shell name etc). [14:50:47] Stupid systems like TUSC or w/e it's called are just annoying heh [14:51:22] as I said to Erik, that ball is on WMF floor [14:51:27] or rather, roof [14:51:57] It's been in WMF's carpark for like 2years AFAIK [14:52:21] it's a step forward that Chris Steipp is actually working on it [14:52:29] Which is stupid since it's really designed as an 'open' platform, it's the most closed open platform ever when it comes to building tools to grow community and data [14:52:34] yet he was listing it yesterday as something bad about the toolserver :P [14:53:13] TUSC should have never been created, centralauth should support openid and oauth as endpoints from when they where popularized [14:55:06] Damianz: what was so alternative to TUSC in 2007? [14:55:32] yeah, I was about to ask: hey Damianz, why didn't you help them build something better in 2007? :) [14:55:53] I believe the saying is that It's easy to play Monday Morning Quarterback :) [14:56:01] I hate php :P [14:56:12] TUSC exist before openID was created [14:56:27] let's honor the people who *made something work* even if it was suboptimal, because everything is suboptimal in the long run [14:57:13] It did the job, I just don't see wtf the api is still so crappy when we've had nice open and widly used standards for years. Especially when the entire principle is to be open. [14:57:49] did you see Dantman mails about OAuth and similar? [14:57:58] the conclusion was similar to "all of them suck" [14:58:47] I didn't, I don't mind the implimentation sucking slightly if it's at least vaugly user friendly though :P [14:58:54] Damianz: so, you're backing off from "should have never been created" right? [15:01:15] I don't think something designed for auth should be implimented in a shared, not hugly reliable environment. Something along the lines of that implimented as an extension for mw possibly would have been better and more open. [15:03:17] assuming it gets deployed [15:10:16] hello and good morning [15:10:40] hi dschwen [15:11:03] i would like to point out again, that ssh connections from labs to the toolserver are blocked somewhere along the way [15:11:11] any chance of finding out where? [15:11:21] i can do outound ssh fto othr servers [15:11:28] oh dear [15:11:34] oh lord, let me try that again [15:11:41] I can do outbound ssh to other servers [15:12:10] and of course i can ssh onto the TS from other servers [15:12:32] I'd like to get some testing done and would like to set up tunnels for the DBs [15:13:17] hmm [15:16:45] dschwen: I'll come back to you, Leslie is looking [15:16:52] thanks [15:17:54] !log wikidata-dev wikidata-dev-3: Did some modifications on the script for demo preparation [15:17:56] Logged the message, Master [15:26:31] dschwen, parhaps that's related to the rules to block access to wmf internal network ? [15:26:42] hm [15:26:49] ok, sounds plausible [15:26:56] Platonides: Yeah, I asked Leslie to check just now [15:27:23] hey Platonideswhile you are here, I would like to get onboard the webtools project [15:27:34] I suspect ssh is probably blocked by default for all wmf ranges. [15:41:56] dschwen, sure [15:42:14] I haven't done anything on it yet [15:42:20] I noticed that [15:42:35] I would like to get started on porting the zoomviewer [15:42:45] it is not running very well on the toolserver [15:43:20] can we get an instance up? should there be one common instance to start things off [15:43:37] I think we should make a couple of instances [15:43:45] a web server plus a bastion one [15:44:03] what does the bastion instance do? [15:44:13] it's a login server [15:44:32] well yeah, why do we need another one? [15:44:43] sumanah: can you take the lead on acct creation for the next 20ish hrs? oh, i see you got a rename? (or new acct) [15:44:45] sorry if I'm being a bit thick here [15:44:51] jeremyb: sure, I can [15:44:53] * jeremyb runs away ;-) [15:45:17] dschwen, do you prefer to develop the files on the web server ? [15:45:31] that would be easiest [15:46:47] is it the right thing? [15:47:08] ?! [15:47:12] oh, I see [15:47:59] no, I thought be 'develop' files you mean the conversion process that generates a multiresolution pyramid for the zoomviewer [15:48:24] no, a login server with a shared project directory that is used by the webserver would be fine [15:48:48] I'd just need some packages installed on the webserver [15:49:01] namely libvips-tools [15:50:52] anyone available that might be able to help with ssh'ing into an instance? have tried adjusting my ~/.ssh/config per the instructions https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option, but get the following - ssh: Could not resolve hostname glam-gwtools.pmta.wmflabs: nodename nor servname provided, or not known [15:50:54] so looking at the help pages it seems there is no central git available for labs users (yet), right? [15:51:22] dschwen: not sure what you mean -- all labs users have access to gerrit.wikimedia.org [15:51:27] maybe you mean a certain git repository? [15:51:30] push access? [15:51:38] make my own repos for my tools [15:51:59] "Note: Push access is currently limited to staff developers and operations engineers. This will change soon. " [15:52:45] no, you can request a repository [15:52:55] for a tool, and have push access to it [15:53:10] where's that pharse? [15:53:16] maybe it refers to mediawiki repository [15:53:34] dan-nl, can you provide what you put there? [15:53:35] https://labsconsole.wikimedia.org/wiki/Git [15:54:14] that's very outdated [15:54:20] :-( [15:54:22] in the .ssh/config i've got the following [15:54:29] Host bastion1.pmtpa.wmflabs [15:54:29] Hostname bastion.wmflabs.org [15:54:30] ProxyCommand none [15:54:30] Host bastion1.eqiad.wmflabs [15:54:31] Hostname bastion2.wmflabs.org [15:54:31] ProxyCommand none [15:54:32] Host *.pmtpa.wmflabs [15:54:32] ProxyCommand ssh -a -W %h:%p bastion1.pmtpa.wmflabs [15:54:32] Host *.eqiad.wmflabs [15:54:32] ProxyCommand ssh -a -W %h:%p bastion1.eqiad.wmflabs [15:54:33] Host *.wmflabs [15:54:33] User Dan-nl [15:54:33] IdentityFile ~/.ssh/wikilabs/id_rsa [15:54:34] IdentitiesOnly yes [15:55:05] oops ... didn't know how to paste in several lines at once ... [15:55:07] try running glam-gwtools.pmtpa.wmflabs [15:55:11] instead of glam-gwtools.pmta.wmflabs [15:55:21] *ssh glam-gwtools.pmtpa.wmflabs [15:56:11] that's better, but got illegal option -- W [15:56:19] dschwen, take a look to https://www.mediawiki.org/wiki/Git [15:56:22] does that w need to be lowercase? [15:56:26] no [15:56:31] you have an old ssh version [15:56:45] it's a relatively recent option [15:56:47] <^demon> That "push access is limited" nonsense is very misleading. In practice: A) Ops doesn't do direct pushing to the puppet repo, and B) Anyone with an account can push a change to *any* repo [15:56:56] <^demon> But yes, that whole page it outdated from back when we only had like 4 repos. [15:57:26] ok, will look into updating it ... using snow leopard [15:57:34] Platonides: thanks [15:57:40] you're welcome [15:58:18] <^demon> dan-nl: Really, I don't like there being git docs on labsconsole. We also needed it on mediawiki.org, so it makes sense that it's all in once place. [15:58:30] <^demon> (And there's more docs on mw.org, so I'd be inclined to remove it from labsconsole. [15:58:53] ^demon, would it be possible to automatically create repositories? [15:59:10] ^demon: i completely agree ... i find the entire documentation confusing ... that's why i started to edit [15:59:13] I'd like something like a setuid command run by users which automatically made them a repository in gerrit [16:00:03] <^demon> Well there's GUI and CLI interfaces to do it. The problem is almost no repo works *out of the box* when you create it, there's additional steps that need doing. [16:00:16] <^demon> Having a form that allows you to request it, then fires off the "setup" commands would be nice. [16:00:31] <^demon> Setup stuff being permissions, etc. [16:01:02] ^demon: was planning to review the docs on git/gerrit after i get access to my instance and then figure out the best way of referring to them ... [16:01:45] ^demon: dschwen : like dschwen i want to create a repo in the git/gerrit system for the gwtoolset [16:03:11] ^demon: but find it unclear as to how to go about doing it “properly” ... will get to that next though [16:03:48] if it's documented somewhere, seems worth a try [16:04:22] dan-nl, you ask ^demon [16:04:40] <^demon> You ask on-wiki and I create it. [16:04:47] there's a page for that, https://www.mediawiki.org/wiki/Git/New_repositories [16:04:54] <^demon> In practice, there's other people who do have create rights but nobody exercises it. [16:06:04] right, saw that earlier and wrote an email to sumanah since she's on that list but she didn't feel comfortable atm proceeding with the request so i put it off so that i could focus on the instance first [16:06:17] <^demon> 6 people in "Projects in Group Creators" + everyone in "Administrators" which includes everyone in "ops" [16:06:24] shall i email the request or put it in here? [16:06:33] wait, dan-nl, the only discomfort there was that *I am not experienced enough to create a repo* [16:07:16] dan-nl: and then Chad created it [16:07:42] ah, had not idea that it was actually created :) [16:07:46] thanks! [16:07:55] dan-nl: aiee - I thought I told you in IRC [16:08:09] maybe it was another dan ;) [16:08:18] :/ [16:08:21] * sumanah looks on https://www.mediawiki.org/wiki/Git/New_repositories/Requests [16:08:27] no matter ... perfect ... thanks [16:14:18] sumanah [16:21:05] dan-nl: I was confusing you with someone else [16:21:18] dan-nl: when did you add your request to the list? [16:21:32] dan-nl: via https://www.mediawiki.org/wiki/Git/New_repositories ? [16:22:42] hi Nischayn22 [16:23:10] no, i saw that you were on the list of people that could create the repo and just emailed you ... didn't see that page until you updated the info boxes [16:24:29] sumanah: so i'll just take this step-by-step and use the page you and ^demon mentioned once i connect to the lab instance ... [16:29:18] Change on 12mediawiki a page Developer access was modified, changed by Merlissimo link https://www.mediawiki.org/w/index.php?diff=587577 edit summary: [16:35:11] so the ssh problem was? [16:35:36] found? [16:39:40] ssmollett: andrewbogott - maybe you can help me with a labsconsole issue: https://labsconsole.wikimedia.org/wiki/Special:UserLogin/signup no longer includes a field for email address [16:39:44] this means I can't make new accounts [16:40:43] I see an email field right under the 'retype password' field. The page is rendering differently for you? [16:40:55] yes, it is [16:41:03] Are you logged in to labsconsole? [16:41:07] "retype password" then "Real name" [16:41:09] yes, as Sumanah [16:41:40] Huh. Well, lemme look at the source code :) [16:41:44] thanks [16:41:49] In the meantime… send me a screenshot? [16:41:52] ok [16:43:12] sending now [16:46:29] sumanah: What browser? [16:46:30] sumanah: only admins can create account for others using mail [16:47:01] or you have to set $wgEnableEmail to true [16:47:53] Merlissimo, any idea why that would've changed for Sumana since last week? [16:48:23] i does not know anything about labs mediawiki config [16:48:29] I was able to create a new account yesterday [16:48:33] andrewbogott: Firefox [16:48:47] 13.0.1 [16:49:14] i only checked that sumanah is not an sysop on this wiki [16:50:34] Merlissimo: In the config: $wgEnableEmail = true; [16:53:26] sumanah: perhaps i should have requested access earlier at hackathon (but i did not need access before) :D [16:54:21] as you know a cannot move my existing tools to labs because of license problems [16:58:14] sumanah: Well… I'm clueless. But I can, at least, create an account for Merlissimo. Link me to the request? [16:59:31] andrewbogott: https://www.mediawiki.org/w/index.php?title=Developer_access&action=edit§ion=5 [17:01:12] thx andrewbogott [17:01:22] Merlissimo: Did you get the email? [17:01:27] yes [17:01:38] that's why i wrote "thx" [17:01:45] :) [17:02:11] I wish I knew why things broke for Sumana. I want to blame her recent renaming, but I can't think of why that would be related. [17:02:36] It's totally her recent renaming [17:02:40] :D [17:05:50] how can i get a list of persons that can grant access to a special project? [17:06:05] i only found a member list [17:10:32] Merlissimo: Can you view this page? https://labsconsole.wikimedia.org/wiki/Special:NovaProject [17:10:55] If so, select projects you're interested in in the filter at the top and then you can see who's a sysadmin for a given project. [17:10:55] "No Nova credentials found for your account." [17:11:07] for me it shows only the projects where I'm an admin [17:11:18] (sysadmin) [17:11:37] Hm. In that case, probably the best way is to ask me :) Which project are you interested in, Merlissimo? [17:11:47] incubator [17:12:35] That would be Hydriz and only Hydriz. [17:13:02] oh, i also expected SPQRobin ;-) [17:13:18] Member but not admin [17:13:43] ok thanks, then i'll wait until he is online [17:15:31] do i need to be a member of the bastian project to connect to that instance after Hydriz added me? Or is this unrelated? [17:17:21] You need to be a member of bastion to connect to anything. [17:17:27] I'll add you, just a moment. [17:19:23] Um. Well, apparently I don't know how to do that anymore. [17:20:17] …there we go. [17:32:03] hi andrewbogott [17:32:26] 'morning [17:32:50] i'm figuring out basic puppet stuff and looking at https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=summary [17:33:22] i see from apache config that stuff is run as user "apache" [17:33:36] * aude wonders where this user is defined in puppet (or is it?) [17:35:01] jeremyb: do you know? [17:35:11] I'm looking. It wouldn't shock me if the apache .deb creates the user [17:36:18] ok [17:36:43] normally, with my own stuff, i've run apache with group www-data [17:36:51] * aude no expert [17:37:11] Well, also, users in labs come from ldap, so maybe the apache user is just magically everywhere thanks to ldap. [17:37:17] That seems most likely. [17:37:24] i don't know if we have that user [17:38:37] i don't see it [17:46:11] aude: On the labs instance I'm looking at now, apache is running as www-data. And also on the labsconsole server. [17:46:43] So either I'm misunderstanding what 'ps' is telling me, or it's not really the case that things run as user 'apache' [17:47:04] andrewbogott: that's what i see [17:47:20] * aude looking at how it's done in production [17:47:53] we can do it on labs differently, though and www-data is okay [17:48:05] Outside of labsconsole I don't know much about production systems. [17:48:09] ok [17:48:32] what i think we'll do is create a role account to share among our team [17:49:07] and run our mediawikis under that user (with group www-data) , if that's reasonable [17:50:00] again, no expert [17:58:02] aude, I can't remember, have you already looked at this? https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki [17:58:29] (That might not be relevant, since I don't know quite what you're doing.) [18:01:18] * aude looking [18:03:50] ok, on https://labsconsole.wikimedia.org/wiki/Special:NovaSecurityGroup i see wikidata doesn't even have the web security group yet [18:04:04] is anyone able to allow ssh/scp to *.toolserver.org? [18:04:24] i'm a bit confused [18:05:37] aude: This is in project wikidata-dev? [18:06:05] hm, clearly not [18:07:11] yes [18:07:11] aude: It could be that the firewall rules for web access were added to the 'default' group rather than to a 'web' group [18:07:20] ok [18:07:38] so the web group or whatever only relates to port and firewall access? [18:08:40] Yes, as I understand it 'security groups' are roughly the same thing as firewall rules [18:09:08] we might want role applicationserver [18:09:30] * aude looking at deployment prep to see what's there [18:13:14] there's also labsmediawiki.pp [18:15:19] labsmediawiki the file that contains role::mediawiki-install::labs which is what that previous link is about. [18:15:44] andrewbogott: ok [18:16:00] it's all a bit confusing but ok [18:16:43] it says "The MediaWiki source will be in a git tree in /srv/mediawiki, owned by root" [18:16:48] right? [18:17:07] but then it runs the mediawiki itself as user www-data? [18:17:54] That sounds right, although I don't have a running example near to hand. [18:18:25] The point of role::mediawiki-install::labs is to make everything work like magic so you don't have to think too hard. Is there a reason you're specifically concerned about permissions? [18:18:29] Hmm I hate c [18:19:00] Seems I scared lcarr off :o [18:19:38] andrewbogott: just figuring out how we should set permissions or if we use the puppet class, it does it automatically somehow? [18:19:59] we have multiple sys admins [18:20:19] and run cronjobs, etc. [18:21:16] aude: Yes, the puppet class will set up permissions needed to have the wiki work. As to whether that corresponds to the permissions you want… that's a different question. [18:21:35] Typically I would expect your team members to just sudo when they need to change thigns. [18:22:59] andrewbogott: but what about cronjobs, etc.? i suppose we can sudo and run them as www-data? [18:23:10] reasonable? or wrong? [18:23:24] use a file in cron.d and run that as the user? [18:23:26] * aude may experiment with a new instance [18:23:34] Damianz: ok [18:26:49] alright, time to eat and come back later [18:29:17] Hmm it seems I really should clean my oven, damn cleaning [18:41:51] Ryan_Lane: I'm about to add a little more information to the [[Wikimedia Labs]] page on mediawiki.org unless you want to move that whole shebang over to labsconsole [18:43:03] will you replicate wiki contents? [18:44:21] giftpflanze: I'm sorry, are you talking about my documentation right now? [18:44:32] giftpflanze: or are you talking about database replication within Labs, similarly to how TS does it? [18:44:56] the latter [18:45:51] hi chad [18:46:11] giftpflanze: aha! check out https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_2 [18:46:33] giftpflanze: in January-to-March 2013, the planned Labs milestones include "Database replication from production" [18:46:35] it's past quarter 2, at least 19:46 [18:47:36] Damianz: you're misreading the URI [18:47:47] it's the second "milestones by quarter" anchor [18:47:55] hey, no taking fun out of my day :P [18:48:12] on toolserver there's no wikitexts because they were too sensible and there are deleted texts in it afair [18:49:24] Hmm [18:49:30] Wikimedia use nginx right? [18:50:10] Damianz: I don't know whether we use nginx anywhere -- a search on wikitech.wikimedia.org would tell you though [18:50:59] I know it use to be on mobile but I think that's mostly varnishified now [18:51:10] though... maybe ssl uses it [18:51:27] Yeah used for ssl termination [18:51:57] Can I count working on nginx as 'Only Wikimedia-related work' now? Cause I'm too lazy to go install a vm on my desktop [18:56:00] Damianz: "only" for what purpose/reason? [18:56:08] Not sure I understand you [18:57:08] In the same sense that people develop extensions not deployed on prod, but are related to software wikimedia uses. [18:58:19] Damianz: ok, but I'm trying to figure out - when you say "can I count foo as bar", for what purpose? you mean using Labs for it? [18:58:51] Ryan_Lane: there's a description of Wikimedia Labs that says it's "for supporting development and operations engineering by both staff and selected volunteers" - mind if I take out "selected" in that sentence [18:59:03] Developing modules (or specifically right now, a better stats module that actually gives you backend info and core info in json rather than need-to-parse plaintext) [19:00:48] 09/26/2012 - 19:00:48 - Created a home directory for jgreen in project(s): analytics [19:05:44] 09/26/2012 - 19:05:44 - User jgreen may have been modified in LDAP or locally, updating key in project(s): analytics [19:06:33] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=587605 edit summary: update with links, goals, removing done TODOs, smoothing prose [19:08:18] volunteers not volunteer [19:08:58] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=587606 edit summary: plural [19:09:02] Damianz: fixed [19:09:06] :) [19:10:07] 'Provide a volume storage solution' is done as project storage or do you mean for instances (which is more for being able to kill hosts and not loose systems) [19:10:34] Damianz: ask Ryan? or ask on the talk page? [19:10:55] reading down, probably the later 'Gluster, ideally' is not really applicable anymore heh [19:11:06] so.. ...at the risk of getting annoying, could we get login.toolserver.org unblocked in the access rules/firewall/whatever? [19:11:08] 'Configure quotas for NFS home directories' will go soon when homedirs move to project storage [19:11:19] Damianz: {{sofixit}} [19:11:24] :) [19:11:35] dschwen: have you already filed a Bugzilla bug and/or RT ticket? [19:11:41] it would be really helpful to get started working with tunnels to get the migration of tools going without the DB replication in place [19:11:44] Damianz: what did Leslie say? [19:11:47] no, sorry, will do [19:11:47] dschwen: Probably, it's likely a network thing and Leslie dissapeared after I asked, make a bug and I'll ask again later [19:11:48] probably not only login.ts.o but all subdomains [19:11:54] sumanah: 10min, then ran off :P [19:11:59] and I would but that means logging in :P [19:12:11] dschwen: yes, I'm working on that [19:12:22] dschwen: I need mark or leslie to handle it, though [19:12:26] Wouldn't that fall into 'no vpns/tunnels' out of labs? [19:12:29] to get tunnels to the DBs set up login is enough [19:12:31] and Leslie is at a conference and mark is asleep [19:12:39] Ah, that's where she ran off too [19:12:46] oh, ok [19:12:54] login is deprecated afaik [19:12:55] TS being blocked is an oversight [19:13:15] giftpflanze, it is? [19:13:22] we block production ranges to protect them from labs, and TS's range is within that [19:13:43] hopefully you block prod from ts too :P [19:13:47] http://osdir.com/ml/toolserver-l/2010-02/msg00120.html [19:14:01] that is dumb [19:14:20] roles change, login is now willow (used to be nightshade) [19:14:27] semantic names are much better [19:14:32] why deprecate it? [19:14:43] there are multiple servers now [19:15:15] sumanah: did you try logging out and back in when you had the email address missing issue? [19:15:22] sumanah: are you still having it? [19:15:25] Ryan_Lane: no, duh, I should have tried that [19:15:26] brb [19:15:45] mediawiki core's auth is so buggy :( [19:16:19] Ryan_Lane: THAT FIXED IT. WHY DID IT FIX IT [19:16:25] sorry for bothering you andrewbogott_afk [19:16:27] heh [19:16:31] Change on 12mediawiki a page Wikimedia Labs was modified, changed by DamianZaremba link https://www.mediawiki.org/w/index.php?diff=587610 edit summary: Removing old/misc/not applicable stuff that's changed [19:16:36] because your session was somehow screwed up [19:16:43] mediawiki is full of bugs in this regard [19:16:48] Feel free to undo some of those but I think they all have different plans now due to issues/changes that have happended since then [19:17:53] https://github.com/devstructure/blueprint < that, looks, freaking, sexy. I wonder if it works [19:17:59] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=587612 edit summary: [19:18:24] ah. neat [19:18:37] mysql filters really suck for management but I guess we don't create/delete dbs every 2min heh [19:19:03] this is asher's plan right now [19:19:11] and he's the one doing it, so whatever he feels is best [19:19:23] We'll just blame him when it breaks? :D [19:19:38] well, he'd be the one fixing it, likely [19:20:20] Hmm, I wonder if filters still get filtered on the client end. I think that's why I avoided them last time. [19:23:02] oooh [19:23:07] since when was krb on the todo list [19:23:09] +++++++++ for that [19:28:20] Ryan_Lane: it'll help if I already know what from q1 https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_2 is already done [19:28:27] sorry, redundant "already" there [19:29:06] most of that, really should do fingerprints soon [19:29:53] Damianz: want to add a status to https://www.mediawiki.org/wiki/Wikimedia_Labs/status ? It's easy, just click the button at https://www.mediawiki.org/wiki/Wikimedia_Labs#Communications [19:30:18] All of Q1 except for gluster support in nova, dns support in nova, and fingerprints [19:30:27] Then people expect me to know the answers to things :P [19:30:49] in Q2 we've completed moving the hardware [19:30:53] * Damianz looks at the format of status updates [19:30:57] Ryan_Lane: "gluster support in nova, dns support in nova, and fingerprints" are sliding into Q2, right? [19:31:03] and mediawiki support was added to nova [19:31:13] Damianz: https://www.mediawiki.org/wiki/Wikimedia_Engineering/Project_documentation_howto [19:31:17] gluster support may slide to Q3 [19:31:30] gluster works, other stuff doesn't so well heh [19:31:31] andrewbogott: or is that plugin already written? [19:32:17] puppetize deployment-prep in Q3 is completed [19:32:36] Ryan_Lane: Gluster plugin is mostly written but we don't really have plugin support until folsom. [19:32:42] ah. ok [19:32:57] I think that'll slide to Q3, ten [19:32:59] *then [19:33:09] we'll likely be ready to upgrade to folsom then [19:33:32] second zone in eqiad is mostly on schedule [19:33:42] I could bring it up now, but there's a couple things I want to do first [19:34:05] user databases in Q3 are going to be be difficult [19:34:21] andrewbogott: if we're going to hit user databases by Q3, we likely need to put some dev work into salt [19:34:40] salt-api was pushed to github last week [19:34:47] so some of the work has begun on their side [19:34:59] we need to add keystone auth to that api, though [19:35:02] salt is pretty shiny [19:35:03] aren't there some user dbs already? [19:35:11] giftpflanze: well, kind of [19:35:16] giftpflanze: We're moving to dedicated hardware and more 'as a service' [19:35:23] currently we have shit on vms [19:35:27] you can install a database server in instances, but it's slow [19:35:34] hm [19:35:40] yeah, this will be dedicated hardware, with SSDs [19:36:03] and the database OS and software will be managed for you [19:36:03] why won't you support joining prod db with user db? [19:36:18] because it causes *really* nasty performance problems [19:36:23] ah, ok [19:36:37] you should ask binasher in #wikimedia-operations, though [19:36:49] he'll give you a more detailed reason as to why [19:36:59] nah [19:37:11] I think that should be app logic anyway [19:37:15] yep [19:37:16] it should be [19:37:43] that's how it works in our production cluster too [19:37:56] commons isn't on the same database server as enwiki, for instance [19:38:05] we're still in beta pending final ci workflow stuff being approved, right? [19:38:10] yep [19:39:17] Change on 12mediawiki a page Wikimedia Labs/status was modified, changed by DamianZaremba link https://www.mediawiki.org/w/index.php?diff=587617 edit summary: /* 2012-09-26 */ new section [19:39:31] now I'm going to find like 50 grammar issues [19:40:03] heh [19:40:15] you just added one :p [19:40:51] Can anyone remember who was working on logstash stuff? [19:41:27] <^demon> Is Comcast going to stay up now? [19:41:36] Doubt it. [19:41:43] <^demon> It's lasted more than 5 minutes this time, so signs point to yes. [19:41:57] It has to give you time to login ;) [19:41:58] * ^demon has been fighting Comcast since almost noon. [19:42:10] Damianz: yes [19:42:18] adminxor [19:42:37] we need to feed him logs [19:42:40] yeah [19:42:46] I keep getting too busy to do it [19:42:47] I keep meaning too and he keeps asking [19:42:54] :( [19:43:19] now that I generate many, just like 3 per edit on en wiki heh [19:43:22] s/now/not/ [19:47:56] Ryan_Lane: I was thinking earlier, is there any simple way - maybe motd or tiny quotas - of limiting bastion, I can see people using that for doing stuff who don't quite understand how projects work (because our docs still suck in places) [19:49:06] what are they using it for? [19:51:11] It was more of a theory than current issue problem. I've just seen a number of 'don't use bastion' replies to things as more people come in... Not really thinking screen etc which is a good use but someone running something that crashes the box and kills everyone elses access [19:51:22] also root has some defunct processes like svn currently [19:51:53] not sure why it's running apache either [19:53:15] it's running apache? [19:53:26] apparently [19:53:32] wtf [19:53:37] how did that happen? [19:53:42] damian@bastion1:~$ curl -sI localhost | head -n1 [19:53:42] HTTP/1.1 200 OK [19:53:51] yeah. odd [19:53:54] Dunno, I just wtf'd a few times and like checked 6 times I was on the right box. [19:54:25] I wonder if I accidentally ran the wrong command at some point [19:54:29] heh [19:54:46] well, it's gone now [19:54:48] This is where puppet needs a --really-enforce-the-manifest option that removes everything else [19:55:31] Oh ffs [19:55:36] bots-apache1 still has crap running on it [19:55:57] hm. yeah. puppet didn't install apache [19:56:15] www-data 12796 0.0 0.2 29684 4372 ? S Sep02 3:33 spamd child < totally [19:56:18] lrwxrwxrwx 1 www-data www-data 0 2012-09-26 06:38 /proc/12796/exe -> /usr/bin/perl < I hate shared servers [19:56:30] spamd? [19:56:37] exploit [19:56:41] I cleaned up 4 the other day [19:56:44] this is on labs? [19:56:44] fucking phpmyadmin install [19:56:47] yeah [19:56:58] we should likely delete the instance [19:57:00] it's already owned [19:57:01] missed killing this process [19:57:09] is it puppetized? [19:57:22] It's apache running mod_userdir so no but it's like 5min [19:57:30] delete it [19:57:31] I moved all the data to project storage anyway [19:57:47] All the exploits seemed to be udp scripts to spam irc etc and some rootkits that didn't work [19:57:52] but we probably should roll another [19:57:59] and *NOT* install phpmyadmin [19:58:04] indeed [19:58:17] or require that its accessed via a socks proxy [19:58:24] strace -p is amusing [19:58:41] when I was asking about security the other day it was in relation to this [19:59:08] basically, don't do shit, if you do then we're going to have a problem. ie write good code, don't leave things like phpmyadmin lying around [19:59:16] yeah [19:59:41] well, we're giving people access to do what they want. this is bound to happen [19:59:44] I'd like to do some WAF setup with mod_security when we do the proxy tbh [20:00:00] mod_security often just breaks things :( [20:00:02] I might install mod_sec on a new instance actually [20:00:39] It doesn't catch everything and really needs rule management but catches lots and yeah causes some issues, mostly when people are doing insane things though. [20:00:42] What does ?puppetize? mean? [20:00:44] Might just note as a 'would like' [20:00:51] Nikerabbit: write a puppet manifest for [20:00:57] * Damianz goes to get his phone to login [20:01:14] Nirvanchik: puppet is a configuration management system [20:01:19] it's a way of automating a system [20:01:45] so, if your system needs apache, php, etc, etc, you can write code that says "install these things, configure them this way, etc" [20:02:23] Can I impose a list of 'banned' software as in, these things commonly have issues, please don't run them, if you really want to run them publically then ask. Or is that really a per project thing [20:02:24] Ryan_Lane: wow. ok. now I get it. kind of. [20:02:30] * Damianz doesn't really feel like spending half his time cleaning servers [20:02:37] Damianz: yes [20:02:40] please do [20:03:17] I think certain things should be banned globally [20:03:39] phpmyadmin isn't a problem, for instance, if it isn't public [20:03:53] <^demon> Didn't we discuss this yesterday? [20:04:00] yes [20:04:07] Damianz: did you delete the instance yet? [20:04:21] You'd like my new job ;) We're basically building a team of 8 people to design a new infrastructure driven by automation, monitoring and metrics. Turning a vmware/windows based network in suse/centos/openstack based with ldap/sso/puppet (maybe chef)/graphite and lots of fun. Going to be a busy year but hell I've got some nice projects coming up like building a cdn to move to from akamai :D [20:04:28] Ryan_Lane: Just logging in now [20:04:47] nice [20:05:21] !log bots deleting bots-apache1, stuff will be down until I install another instance [20:05:22] I really need to write my talk for puppetconf [20:05:23] Logged the message, Master [20:05:50] isn't puppetconf like, comorrow? [20:05:55] yes [20:05:58] <^demon> Ryan_Lane: Would just need to edit /etc/apt/preferences and set Pin-Priority: -1 on the packages. [20:06:00] heh [20:06:01] and so is my talk [20:06:15] ^demon: it's less of an issue if its installed via apt [20:06:20] You talking at fosdem this year? (I actually might get to go this time) [20:06:22] then ubuntu is handling the security updates [20:06:30] Damianz: I may put in talk [20:06:36] Would be interesting [20:06:52] I'm going to try for a main room talk [20:07:12] <^demon> Ryan_Lane: Speaking of conferences, had you decided if you were gonna tag along Nov 10-11 to the gerrit summit? [20:07:20] I can't [20:07:23] I won't be in town [20:07:28] o.0 [20:07:28] I'll be in australia [20:07:30] <^demon> Ah ok, that's cool :) [20:07:34] Yeah, I totally couldn't do that [20:08:04] I like talking about cool stuff, but not with 500 people watching and thinking up hard questions to answer [20:08:06] <^demon> Ryan_Lane: I'm trying to come up with topics to talk on, so if you've got any burning suggestions, I'll take them :) [20:08:09] <^demon> Looking to do 2 talks. [20:08:17] heh [20:08:36] my last fosdem talk had about 500-600 people at it [20:08:50] and it was in the cloud room [20:08:54] damn, we should have at least 200 new labs users then :P [20:09:05] faidon was there, actually [20:09:11] We rolling 12 by default now? (realy should dist-upgrade instances) [20:09:18] no [20:09:21] we shouldn't [20:09:33] that eats up insane amounts of disk space [20:09:40] really? [20:09:43] can someone decipher 'SGE' ? [20:09:45] people should create new instances [20:09:53] Nirvanchik: sun grid engine [20:09:53] Sun Grid Engine [20:09:56] ideally [20:10:02] thanks. [20:10:03] which is now oracle grid engine [20:10:03] people should make things in puppet also :D [20:10:16] Hmm, doesn't paravoid live in europe? [20:10:20] yep [20:10:27] he's in the US right now, though [20:10:29] Probably should go to the wmf thing in germany if that's on next year again [20:10:46] yeah, you should [20:10:50] petan was there this year [20:11:00] I saw the pictures of petan [20:11:12] <^demon> I missed Berlin last 2 years. Dunno if I'll get to go this year. [20:11:14] sumanah kept telling me about it, happended to be busy that weekend heh [20:11:52] Damianz: remind me, what's your country? [20:12:01] England [20:13:36] if you and other English Wikimedia technologists wanted to, you could put on a small event [20:14:27] there's technologists in england? [20:14:31] :P [20:14:54] That requires like organization and time :( Would be interesting to use real space for an event though, something like madlab [20:15:07] Ryan_Lane: A more important question is when did I count as a technologist :P [20:15:11] heh [20:15:13] in fact, Damianz, if you're into Lua, the London Lua people wish someone would come by and hack with them/show them Scribunto [20:15:23] http://www.londonlua.org/ [20:15:36] Not really into lua that much tbh, I like my snakes [20:16:12] Sep 26 20:13:45 bots-apache1 puppet-agent[4045]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined in file /etc/puppet/manifests/webserver.pp at line 91; cannot redefine at /etc/puppet/manifests/webserver.pp:42 on node i-0000044e.pmtpa.wmflabs [20:16:17] Oh come on... [20:18:04] * Damianz unticks apache seen as apache + php5 makes puppet angry and waits 15min [20:18:45] heh [20:18:50] Don't suppose you want to login as root and force a puppet run? IIRC your key is in there [20:18:56] yeah. we need to fix our shitty apache manifests [20:19:00] which instance? [20:19:04] bots-apache1 [20:19:08] you don't have root there? [20:19:18] I can't login as puppet failed [20:19:23] oh [20:19:28] you installed it with that? [20:19:31] Didn't complete the first run before I ticked the box [20:19:35] oh [20:19:38] you need to delete/recreate [20:19:41] I can't log in either [20:19:46] sadface [20:19:48] yep [20:20:12] if(sysadmin) { echo $key >> /root/.ssh/authorized_keys2 } # happy [20:20:29] though security policies would freak out with that [20:20:51] * Damianz waits this time [20:21:09] console output is stupidly slow when it's building [20:21:14] yeah [20:22:04] It could be interesting if we could access the console log more actually [20:22:19] I agree [20:22:19] From a monitoring point of view, since we don't monitor syslog stuff currently (though doing that would be cooler) [20:22:38] well, it's on the instance itself too ;) [20:22:40] I totally want graphs of trends for system logs [20:22:50] logstash... [20:22:59] I'd much rather we syslog'd all of the logs to a central server [20:23:14] yeah, I watched a talk on it the other day and became more interested [20:23:23] Not sure how syslog would scale though as we branch out [20:23:31] Probably end up with relay servers per dc/project [20:23:44] Kinda fancied scribe but the deps/packaging is painful [20:24:09] Running a hadoop cluster just for syslogs would be a cool overuse of resources though :D [20:24:51] Actually sorta-surprised you havn't moved to the mapreduce route for stats stuff on prod tbh, but I can see why as it's more to manage [20:25:10] <^demon> Ryan_Lane: What I want is the console to use a refreshing textarea so I don't have to Ctrl+R. [20:25:39] [1;35merr: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Sshhostkey[gerrit-build.pmtpa.wmflabs] cannot override local resource on node i-0000044f.pmtpa.wmflabs[0m [20:25:43] Oh DIAF puppet [20:25:56] ^demon: Have you seen the build output on travisci? That's sexy [20:26:03] <^demon> Nope. [20:26:05] Damianz: they are doing that right now [20:26:15] Damianz: ou'll need to delete/recreate :( [20:26:17] wait [20:26:18] maybe not [20:26:21] it should try again [20:26:28] yeah, in like 5min [20:26:28] ^demon: patches welcome ;) [20:26:41] It failed twice [20:26:45] ugh [20:26:50] <^demon> Pfft, I can't even get all my gerrit.pp changes reviewed :p [20:26:57] we should just turn off exported resources in labs [20:26:59] they aren't used [20:27:00] One day we might actually have like, an api in labsconsole that can do osm stuff [20:27:06] <^demon> (See what I did there ;-)) [20:27:11] that's on the agenda [20:27:13] ^demon: :D [20:27:22] ^demon: Gerrit just doesn't get the love it craves [20:27:34] ^demon: link? [20:27:38] Ryan_Lane: Interesting how it will work with tokens in the current state [20:28:00] <^demon> https://gerrit.wikimedia.org/r/#/c/24440/ and https://gerrit.wikimedia.org/r/#/c/25233/ [20:28:15] <^demon> (The latter is a resubmit of I8c27b367, we hit a bug) [20:28:47] <^demon> Oh, and https://gerrit.wikimedia.org/r/#/c/24557/ [20:28:51] Hello jenkins-bot, I'd like you to reexamine a change. :) [20:28:53] .... [20:28:53] * aude amused [20:28:57] I need to make the gerrit bot output in here [20:29:10] <^demon> It outputs test branch :) [20:29:12] please not [20:29:13] Ryan_Lane: When does the instance page get created, is that a background job? [20:29:29] Damianz: should be when the instance is created [20:29:32] did it not happen? [20:29:38] well sorta [20:29:38] <^demon> aude: The name picking in that template is horribly busted...I'm thinking of customizing the e-mail templates to remove the name. [20:29:43] <^demon> And just do "Hello," [20:29:44] it's missing content/ [20:29:54] ^demon: would be nice :) [20:30:00] When I got redirected back to the list it's a red link, clicking on the red link straight away the page exists, refresh it's a blue link [20:30:06] It's like it didn't quite do it fast enough [20:30:21] <^demon> aude: It's no better in 2.5, so we won't be missing out on a fix anytime soon. [20:30:43] * aude nods  [20:31:42] <^demon> Pretty trivial to do. Just a matter of customizing all of the templates and tossing them in puppet (for the ones that aren't already) [20:32:01] Danny_B|backup: please not move the output in here? [20:32:02] heh [20:32:19] I want labsconsole relay in here/somewhere still :P [20:32:20] Damianz: ah. right [20:32:26] Damianz: it's a background job [20:32:31] I stick it into the job queue [20:32:39] That makes sense [20:32:40] <^demon> Actually, it might just be ChangeSubject. [20:32:43] * ^demon pokes [20:32:47] when the instance is networked, then it creates it [20:32:51] what happens if the nova thing triggers before the background job? [20:33:01] oh yeah the not got an ip yet issue [20:33:03] Ryan_Lane: yes. all these channels are cluttered by bots messages so it's hard to follow / find regular human talks on them [20:33:06] I remember you saying now [20:33:07] then nova will create it [20:33:15] Danny_B|backup: yeah, understandable [20:33:49] ^demon: ok. I merged them [20:33:55] ^demon: now improve labsconsole :D [20:34:05] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter ssh_key at /var/lib/git/operations/puppet/manifests/gerrit.pp:71 on node manganese.wikimedia.org [20:34:10] fail [20:34:20] <^demon> wot. [20:34:22] <^demon> I tested those. [20:35:01] Danny_B|backup: You mean it's like ops? :P [20:35:26] ^demon: am I missing one? [20:35:36] gerrit::jetty doesn't have the ssh_key param [20:36:00] andrewbogott: You still around? [20:36:11] I am! [20:36:46] Damianz: what's up? [20:37:29] With the nova-dns stuff, would it be possible to create SSHFP records as well as A records so we can tell ssh to check those for security/generally getting rid of warnings? [20:37:44] nope [20:37:54] the ldap backend doesn't support it [20:37:59] I've already checked [20:38:02] Ryan_Lane: You just like spoiling the fun :( [20:38:22] How can the ldap backend /not/ support that, the schema fixed on fields? [20:38:44] yeah [20:38:48] which sucks [20:39:14] back in a bit [20:39:38] This is why we can't have nice things [20:40:00] <^demon> Ryan_Lane: I'll poke it in a minute, on the phone. [20:40:32] * Damianz lets andrewbogott go back to hiding [20:41:24] http://notfound.org/ interesting idea [20:47:15] nice [20:54:05] Ryan_Lane: as I update stuff: is "Push OpenStackManager changes to show SSH fingerprints for instances" October, Nov, or later, you think? [20:54:31] that shouldn't be hard, the meta info is in nova already AFAIK [20:54:58] Ryan_Lane: When you get back - all instances in a project should be able to communicate with each other, right? Can't seem to connect to sql from web [20:54:58] Damianz: well, even nonhard things sometimes wait because there are higher priority things in front, right? [20:56:01] is logging enabled on this channel? [20:56:10] yes [20:57:00] !log bots bots-apache1 up again, issues with connections to sql at the moment. Everything else should be there, we need to puppetize this [20:57:01] Logged the message, Master [20:57:41] weird... I can't access anything on the other instances from here grrr [20:58:45] Damianz: the info isn't in nova [20:59:00] Damianz: and yes, it should be able ot [20:59:05] I thought it was avaible via ec2 stuff [20:59:15] nope [20:59:18] well it can't, in the default and web groups [21:01:00] I can ping it [21:01:29] hm [21:01:31] weird [21:01:56] I can ping them, I can't telnet to any ports though [21:02:30] I wonder if a compute node isn't updating [21:02:32] bots-apache1:* -> bots-sql2:3306 should go though fine [21:02:34] Nirvanchik: btw, hi! and are you on labs-l and you can log into Labs and so on/ [21:02:35] ? [21:03:57] strange. group based rules aren't working, but normal rules are [21:03:58] sumanah: hi. no. not yet. just thinking of this [21:04:08] that's annoying [21:04:19] I encourage you, Nirvanchik :) https://lists.wikimedia.org/mailman/listinfo/labs-l and https://www.mediawiki.org/wiki/Developer_access [21:05:48] I wonder if this is an issue on virt5 [21:05:50] it's a new host [21:06:36] sumanah: thanks. I think I'll try this week. It's nice I didn't get on Toolserver yet, or I would spend my time learning it in vain [21:06:54] Nirvanchik: not totally in vain [21:07:03] Nirvanchik: but I understand what you mean [21:07:04] and... it's easier to learn new thing than to forget one thing to learn another [21:07:34] unless these 2 things are alike [21:08:15] sumanah: thanks and bye. I go to sleep :) [21:08:21] bye! [21:09:44] firewall rules on virt6 didn't seem to get updated [21:09:53] I'm restarting nova-compute [21:13:21] Damianz: working now [21:13:36] <3 [21:13:43] something in nova is blocking [21:13:49] or deadlocking [21:13:54] I'll check when the vm finishes rebooting [21:14:34] hey, it wouldn't be a nova release without some annoying production bugs, eh? :) [21:14:36] Still can't connect [21:14:37] :( [21:14:45] seriously? [21:14:46] oh no [21:14:46] wait [21:14:47] I just did [21:14:48] 2 responds [21:14:58] maybe bots-sql1 just is broken atm [21:15:07] lemme see which node that's on [21:15:16] yeah [21:15:22] virt7 [21:15:24] bots-sql1 works fine locally (telnet) not from bots-apache1 [21:15:53] bots-sql3 is fine, bots-sql2 is fine [21:16:09] !log bots Ryan fixed acls, apache1 can now talk to sql again [21:16:10] Logged the message, Master [21:17:16] hmm [21:17:25] I restarted nova-compute on virt7 [21:17:27] I kinda want to put apache logs on project data, is that an insane idea [21:17:44] root@bots-apache1:~# telnet bots-sql1 3306 [21:17:44] Trying 10.4.0.52... [21:17:45] telnet: Unable to connect to remote host: Connection timed out [21:17:55] should be fine to put it there [21:18:11] yeah. it may take a little bit [21:18:25] it's still applying rules [21:18:27] ah ok [21:18:43] what does it actually do? re-read it's instance list and generate iptables rules? [21:18:46] or ebtables [21:18:59] works [21:19:04] yeah [21:19:04] indeed [21:19:06] Tyvm [21:19:11] for source group ones [21:19:24] there's really no other way to do it [21:21:10] The thought of how it loops over that makes my head hurt [21:21:38] which version of puppet was modules introduced? [21:22:38] No idea, does it really matter - everyone should run stable :P [21:23:05] I want to be sure that we started using puppet before modules existed [21:23:44] 'Added autoloading of modules – you can now ‘include’ classes from modules without ever needing to specifically load them.' was like 0.23.1 [21:23:53] http://projects.puppetlabs.com/projects/1/wiki/Release_Notes [21:24:05] yeah [21:24:08] As if it only goes back to 0.20.0 [21:24:10] no dates associated with the damn releases [21:24:25] http://projects.puppetlabs.com/projects/1/wiki/Release_Notes/history :) [21:24:32] have fun [21:24:57] actually the first version of that contains like 30 releases [21:24:58] fail [21:25:52] hahaha [21:25:56] yeah [21:26:06] the very first version is 0.20.0 in that wiki [21:28:06] Ryan_Lane: btw was everything that Damianz removed ok to remove? https://www.mediawiki.org/w/index.php?title=Wikimedia_Labs&diff=587610&oldid=587606 [21:28:14] yep [21:28:24] ok, just checking [21:28:28] I know of a large place called wikipedia that you can add that info later :P [21:28:42] heh [21:28:51] Ryan_Lane: as I update stuff: is "Push OpenStackManager changes to show SSH fingerprints for instances" October, Nov, or later, you think? [21:28:52] meh. I can't find it [21:28:55] agaffney> Damianz: it was sometime prior to 0.24, makes it <= 2009 [21:29:07] later. it's not a really high priority item [21:29:12] ok [21:29:25] yep. that's a little after we started using it [21:30:20] r1 | root | 2009-05-09 18:21:24 +0000 (Sat, 09 May 2009) | 2 lines [21:30:41] that's the first entry in svn history [21:31:19] 0.25 when they added module auto loading was 31/08/2009 [21:31:51] yeah [21:31:59] I'm sure we started using puppet before that svn repo existed too [21:32:04] http://projects.reductivelabs.com/versions/1 - 0.22.1 (2007), http://projects.reductivelabs.com/versions/2 - 0.22.2 (2007), http://projects.reductivelabs.com/versions/3 0.25 (2009) [21:32:09] THERE WAS A TIMEWARP [21:32:14] :D [21:32:42] I'm sure someone has a copy of a cvs repo around still heh [21:32:54] heh [21:34:57] Ryan_Lane: What where you thinking for fingerprints? Grabbing the ec2: data from console log on install or using a saltmodule to go grab the data? [21:35:14] the ec2 data doesn't have it [21:35:22] salt module is likely easiest [21:35:40] but for that we really need a salt api [21:35:44] Hmm, I swear on the first boot it dumps the 3 host keys out in console log, doesn't after that though [21:35:59] yeah it goes into the console log [21:36:05] the thing is, the instance creates it [21:36:09] so nova has no clue [21:36:15] mhm [21:36:33] I could have salt alert virt0 of its fingerprint via an event [21:36:56] https://github.com/saltstack/salt-api what you're redering to for salt api? Basically so you can provide restricted, abstracted running of system stuff [21:37:00] I could have a daemon that listens to events and updates labsconsole [21:37:20] yes. I want keystone authentication for it [21:37:22] Or we could make labsconsole have a nice api [21:37:31] then, I want to have roles that are allowed to run specific things [21:37:43] then I want people to be able to trigger actions through labsconsole [21:37:46] that would be interesting [21:37:55] that's how I'm going to make the database service [21:37:57] like [21:38:01] exactly [21:38:12] and means we can go to the command line and not touch labs console in theory [21:38:16] the database service is even more interesting [21:38:39] It it designed to run realtime async style? As if you where using say celery for background tasks [21:38:40] http://docs.saltstack.org/en/latest/ref/modules/all/salt.modules.mysql.html#module-salt.modules.mysql [21:38:51] Ie could we spool job output back to the user [21:38:53] realtime async, yes [21:39:01] That would be fucking cool [21:39:17] so, the database server will just be a normal server [21:39:26] http://docs.saltstack.org/en/latest/ref/modules/all/salt.modules.data.html#module-salt.modules.data [21:39:42] when a project wants a new database, labsconsole will create it [21:39:50] I didn't know salt was abstracted to the application level that much [21:40:03] Was looking at the facter style thing the other day, that seemed useful [21:40:05] and will add data to the hosts's datastore, linking the database with a project [21:40:29] so, when a user goes to modify a database, labsconsole will check the datastore to ensure that database is in that project [21:41:15] yeah. salt is fucking cool [21:41:32] How do you propose handling sharding of users for scaling out? Multiple servers and auto select different ones in labsconsole or just sort 1 and fix that later? [21:41:36] funny thing. I'm giving a puppet talk and am having a hard time finding interesting things to talk about [21:41:46] I could give a 2 hour talk on salt and we're hardly using it yet [21:41:59] yep. that's basically it [21:42:15] I could have a module that tells me usage info [21:42:20] and then select based on that [21:42:29] it could give me a weight number back [21:42:37] Would be interesting if we could get to the point that we're PaaS to the point of having a python command line script to deploy anything, anytime. Hook that up to jenkins and we make ponies poop rainbows [21:43:01] if salt has an api with keystone auth, that's doable [21:43:41] Any reason you don't just want to use the ldap backend? Or trying to abstract away from that as much as possible to keep within nova [21:43:49] hm, I may need a wrapper for the mysql stuff [21:43:54] maybe a runner [21:44:03] ldap backend for what? [21:44:43] I'm trying my best to move as much away from ldap as possible from labsconsole [21:44:48] for salt-api, as oppose to authing against keystone which proxies to ldap [21:44:57] nah. I want keystone access [21:44:59] err [21:44:59] auth [21:45:17] actualling authing via keystone could be interesting [21:45:23] we could use saltstack to deploy instances [21:45:23] because otherwise you need to provide a password every single time [21:45:26] yep [21:45:34] they actually have a utility for that [21:45:56] https://github.com/saltstack/salt-cloud [21:46:03] it needs generic openstack support [21:46:08] but it shouldn't be hard to add [21:46:27] That might be interesting for something I'm working on actually at a quick glance [21:46:41] salt overall is pretty great [21:46:52] Trying to provide an abstracted dashboard for our devs to launch instances on a few different public providers and our own openstack setup to stop them ticketing us. [21:47:13] I'm actually thinking of changing the cloudinit stuff to install salt, then have salt install puppet [21:47:26] cloudinit is a little dodgy sometimes [21:47:37] cloudinit is pretty rock-solid for us [21:47:55] we have a *very* standardized cloudinit injection routine, though [21:48:13] Yeah, you don't use openSUSE either :) [21:48:18] true [21:48:58] I think I need to look at salt [21:49:27] Currently at a 'do we deploy puppet, chef or ' for the huge push we're about to start re-vamping everything. [21:51:07] root@bots-apache1:~# grep 'PHP Notice: Undefined index:' /var/log/apache2/error.log | wc -l [21:51:10] 12782 [21:51:13] Pretty much sums up my hate for php (this is since I just re-intalled the box) [21:51:28] I think salt needs some more work to be able to totally replace puppet [21:51:35] but it's getting there quickl [21:51:36] quickly [21:51:46] hahaha [21:51:53] yeah. php is teh suck [21:52:40] I'm thinking it would be interesting for provisioning, currently the cobbler <> puppet gap is a bit meh and the gap is a bit meh, though tools like knife etc help somewhat. Going for IaaS style stuff requires so much supporting shizzle it's unreal. [21:53:05] Oh god [21:53:11] It has like 30min uptime [21:53:23] * Damianz hopes 8gb of / space is enough [21:53:29] Might have to move it to project storage heh [21:53:57] heh [21:54:39] I should probably start on the nginx module I was going to about 2hours ago... shame it doesn't provide decent proxy stats by default [21:58:07] Ryan_Lane: I don't suppose there's any chance of openid/oauth support on labsconsole being stable/widly avaible before we open up? I was thinking suff like gerrit/logstash is awesome to develop on etc but self signed ssls+ldap passwords probably isn't an awesome combo. 2fa provides some protection but still might be an idea to be slightly more secure internally. [21:58:52] unfortunately not [21:58:59] we really just need openid as a provider [21:59:05] but the openid extension is broken for that [21:59:15] I don't have the time to fix it [21:59:30] that would work as we really need authentication over anything else [21:59:34] we need oauth for the projects [21:59:45] but just openid as a provider for labs itself [22:02:14] oauth might be nice for labs later on if we move to developing tools to provision stuff via the console api. Likely it would be internal and do-able directly to the relevant service though I guess. Depends what the long term side of things for 'the loop' of vcs<>unit testing<>water testing<>mock tests<>development. [22:02:52] Partly think it would be interesting for projects to impliment management style stuff into labsconsole and have the entire backend as well as docs community managed but that takes a community I guess heh [22:06:41] oauth would be nice for bots writing to labsconsole [22:06:53] yeah [22:07:02] the community is still a little new for things that ambitious [22:10:21] It will end in one of 2 ways, like the toolserver and fall apart or like the old days where most ops stuff was done by random people that where bored heh. Possibly slightly too optimistic at this point but it'll be interesting. [22:11:08] yeah [22:11:22] well, it'll only fall apart if WMF drops it [22:11:43] it's so linked into our dev cycles now that it would likely be difficult [22:13:30] Having it as 'the place to go for testing prod changes' rather than seperating ci/cd out helps. Really the only issue would be if it ends up being a full time ops job of looking after random, seperated users wanting to run single tools with 0 input elsewhere. [22:13:57] yeah [22:14:08] Which on one side is good - they're helping projects as a whole from the user side, and on the other is bad because there's a seperation which there sorta is currently. [22:22:31] labs makes giving so many talks so much easier [22:22:48] this puppet talk is basically just a revised labs talk. heh [22:23:09] lol [22:23:24] Giving puppet talks will be easier when you can actually use our puppet repo outside of labs. [22:23:27] it's honestly our only interesting use of puppet [22:23:29] Trying to run it anywhere else just doesn't work [22:23:29] yes [22:23:32] yep [22:23:36] I'm talking about that too [22:23:50] modules should make sexy [22:24:00] yeah. we need modules for lots of reasons [22:24:17] sadly migrating to modules on labs sucks - role classes for osm compatability makes testing against a know setup hard... could just use local node definitions though I guess [22:24:44] what do you mean? [22:25:00] use puppetmaster self for creating things [22:26:41] Well as far I can tell some of the stuff used in prod can't easily be used in labs without writing a simple class around it 'role::' currently, which while migrating manifests to modules should be easyish, running both on 2 hosts them comparing them to ensure you didn't screw up is hard/not possible in some cases. [22:27:13] Allthough maybe it's just me who'd rather not move to modules, make a role class, migrate the prod node definitions, test the role works in labs and get it reviewed to find out it breaks something, somewhere, obscurly [22:28:06] we should be using roles in labs too [22:28:13] you can modify roles with variables [22:28:47] heh [22:28:55] that's what review is for [22:29:04] and if it breaks something, it's the reviewer's fault [22:29:27] Well how I personally try to use puppet is the only thing under manifests are role classes with variables that call module stuff and node definitions, but as we have a mess and can't have a clean slate it's a little more tricky to think about... just hope review is good [22:30:07] yeah. [22:30:15] again, blame goes to the reviewer [22:30:52] ugh. I need to find a way to delete keys from salt when instances are deleted [22:31:05] ah, right. faidon and I talked about this [22:31:21] delete the keys in puppet, have salt track puppet [22:31:33] I guess I could also call salt and have it delete both keys [22:31:43] need an API for that :( [22:32:00] keys as in salt keys? [23:03:14] Question for you guys, but esp Ryan_Lane. I've got a service running on a labs instance bound to 0.0.0.0:8081 [23:03:37] I have a local socks proxy set up -- ssh bastion.wmflabs.org -D 8085 [23:03:45] my browser is using the proxy. [23:04:16] however, i only get connection reset [23:04:21] no log entries. [23:04:30] i run netstat on the host, and i see: [23:05:07] "tcp6 :::8081 :::* LISTEN 9547/java" [23:05:15] sooo. [23:05:36] 1. has anybody seen 0.0.0.0 bind to ip6 by default before? [23:05:41] 2. is there a way to prevent this? [23:05:54] You can try binding to 127.0.0.1 [23:06:02] 3. did i do anything else wrong/stupid? [23:06:03] it happens sometimes [23:06:32] usually 0.0.0.0 implies both v4 and v6, might be an app specific thing [23:07:00] i vaguely recall there's a java -D that "prefers" v4 [23:07:09] but i'd rather not wade into that if i can help it [23:08:14] dschoon: yes, I've seen this [23:08:24] any advice? [23:08:27] in java specifically [23:08:35] do you have anything in /etc/hosts? [23:08:44] nothing custom [23:08:49] just whatever's there by default. [23:08:54] 127.0.0.1 didn't work, btw. [23:08:55] is the instance's hostname in there? [23:09:10] let me look. [23:09:20] or anything related to it? [23:09:23] only localhost [23:09:27] :( [23:09:36] ok [23:09:46] I blame java [23:09:48] try putting the instance's IP address and hostname in there [23:09:52] aiight. [23:09:55] Damianz: you'd be right to do so [23:10:01] because this is an issue with java [23:10:07] bound to 127.0.0.1? [23:10:10] no [23:10:13] to the ip? [23:10:15] to its actual IP [23:10:27] then try restarting the service [23:10:30] k [23:10:41] I had this problem with opendj at some point [23:10:49] and I remember tracing it down to this issue [23:10:54] it may not solve your problem, tough [23:10:56] *though [23:11:02] The random lets just not listen properly issue that breaks shit :D [23:11:05] Yeah, java's networking likes explicit interfaces. [23:16:07] hey all, anyone around that might be able to help with ssh access to an instance? [23:16:25] probably, what's your issue [23:16:28] <^demon> Ryan_Lane: Fix for ssh_key: https://gerrit.wikimedia.org/r/#/c/25453/ [23:16:33] <^demon> (Sorry for the delay) [23:17:26] Damianz: ssh seems to login but gives no further feedback or command prompt and seems to hang [23:17:39] ah -Djava.net.preferIPv4Stack=true [23:17:42] Try ssh -vv and paste what the debug output shows [23:17:52] dan-nl: which instance? [23:18:32] :Damianz how can i paste all lines in one chat post? [23:18:43] pastebin.com/ [23:18:49] we really need a labs pastebin [23:19:04] well, I started adding one [23:19:11] but we don't have openid, so I said fuck it [23:19:25] ssh glam-gwtools.pmtpa.wmflabs -vv [23:19:25] OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011 [23:19:26] debug1: Reading configuration data /Users/dan/.ssh/config [23:19:26] dan-nl: which instance is it? [23:19:26] debug1: /Users/dan/.ssh/config line 30: Applying options for *.pmtpa.wmflabs [23:19:26] debug1: /Users/dan/.ssh/config line 36: Applying options for *.wmflabs [23:19:27] debug1: Reading configuration data /usr/etc/ssh_config [23:19:27] debug2: ssh_connect: needpriv 0 [23:19:27] debug1: Executing proxy command: exec ssh -a -W glam-gwtools.pmtpa.wmflabs:22 bastion1.pmtpa.wmflabs [23:19:28] debug1: identity file /Users/dan/.ssh/wikilabs/id_rsa type 1 [23:19:28] debug1: identity file /Users/dan/.ssh/wikilabs/id_rsa-cert type -1 [23:19:28] debug1: permanently_drop_suid: 501 [23:19:31] ack [23:19:32] heh [23:19:35] or that.. [23:19:35] pastebin ;) [23:19:36] glam-gwtools [23:20:06] I can't ping it [23:20:17] seems down for me [23:20:33] did you reboot it? [23:20:53] Not on a node you 'fixed' earlier? heh [23:20:58] me? [23:21:04] :Damianz & :Ryan_Lane .. created it earlier, did not reboot ... haven't done anything else with it yet [23:21:09] hm [23:21:12] did it ever ping? [23:21:15] or ssh? [23:21:15] Did it build? [23:21:27] might just have failed... stupid scheduler bug [23:21:34] it says active [23:21:37] <^demon> Ryan_Lane: The gerrit::account refactor stuff is cool, because now all you need to receive replication is now just install gerrit::account with the public ssh key (and then configure gerrit to replicate) [23:21:38] it finished the standard build ... no puppet groups yet ... just wanted to try and log into it [23:21:50] <^demon> (Did this so jenkins can receive replication from gerrit) [23:22:06] <^demon> Without installing the full gerrit stuff :) [23:22:16] ^demon: I know. I did a really good job refactoring the gerrit stuff, eh? :) [23:22:32] * Ryan_Lane steals all of ^demon's credit [23:22:35] ^demon: Would be interesting to talk to hashar and replicate to beta so we don't have to have the sill while true; git pull; sleep 5. Not sure how well it would work with the submodule extensions repo though. [23:22:35] ^demon: err: Failed to apply catalog: Could not find dependency Package[gerrit2] for File[/var/lib/gerrit2/review_site] at /var/lib/git/operations/puppet/manifests/gerrit.pp:224 [23:22:53] <^demon> Damianz: Same thing should be possible. [23:22:56] <^demon> Ryan_Lane: Lemme debug. [23:23:27] hm [23:23:27] dan-nl: So console output showed a puppet run succeed? [23:23:33] I wonder why I can't ping it [23:23:45] It has the default group right? [23:23:48] (security) [23:23:55] lemme check its security rules [23:23:59] need to get around to fixing the blank entry on the manage instances page [23:24:07] labsconsole has some bug when creating new projects [23:24:28] hah [23:24:29] If it has no groups wouldn't puppet fail though? [23:24:29] that's it [23:24:32] Or it outbound allowed [23:24:35] s/it/is/ [23:24:37] outbound is allowed [23:24:37] Damianz: have no idea, just created the instance, waited for it to become active and tried to log into it via ssh, had to upgrade ssh and now i've gotten this far [23:24:45] there's no default rules [23:25:02] dan-nl: We're still finding/fixing random bugs that like to popup when new stuff is added :) [23:25:07] it's pinging now [23:25:09] try and ssh again [23:25:33] Ryan_Lane: oh, actually. it seems like there's a problem with the proxy. [23:25:33] ok. they are all applied now [23:25:40] dschoon: oh? [23:25:43] do you have a proxy through bastion atm? [23:25:45] 09/26/2012 - 23:25:45 - Created a home directory for laner in project(s): glam [23:25:51] can you tell me what your config says, exactly? [23:25:51] yeah [23:25:53] ^ that would help also [23:26:02] dschoon: I'm using a socks proxy [23:26:04] (in firefox) [23:26:05] yes. [23:26:11] * Damianz wonders if labs-home-wm really just created that OR if it's a silly bug [23:26:11] I'm using foxyproxy [23:26:17] i don't know what exactly is wrong, but clearly something is. [23:26:18] and using localhost:8082 [23:26:20] with socks 5 [23:26:29] when I use ssh I'm using -D 8082 [23:26:35] Ryan_Lane: Btw, did you see when labs-home-wm went insane and removed the same dir like 20times (think it was on beta for localisation stuff). [23:26:48] Damianz: still hanging ... [23:26:51] Damianz: that user doesn't exist in LDAP [23:27:06] derf. [23:27:17] dan-nl: hm. it connects for me [23:27:29] Well it should have only done it once :P And that kinda sucks when most tools are forced to /home for the prod stuff [23:27:35] Though I wish they weren't [23:27:36] apparently you need to make sure you have blank values in all the other boxes. [23:27:39] Totally should be in $PATH [23:27:45] Damianz: nah, it doesn't actually remove it [23:27:50] it just *says* it's going to do it [23:27:56] Ah... [23:27:56] I disabled that [23:27:59] that would explain the spam [23:28:05] Damianz: well, that's a good sign, could be that my upgrade of openssh didn't work out properly ... [23:28:05] but not the reporting, apparently. heh [23:28:19] dschoon: :D [23:28:25] dan-nl: is it still not working? [23:28:26] I'll be happy when it dies :D [23:28:35] Damianz: me too [23:28:38] Ryan_Lane: not yet .. [23:28:39] Well no I won't, can find something else to be unhappy about [23:28:41] hm [23:28:43] But we have progression that way [23:29:06] Need to kick labs-nagios-wm also [23:29:14] stupid ircecho bot [23:29:19] Ryan_Lane: it could be that i didn't upgrade openssh properly ... don't have any other ssh connection to test with though [23:29:19] yeah. I can connect through proxycommand too [23:29:31] dan-nl: can you connect to bastion? [23:29:36] ssh bastion.wmflabs.org [23:29:53] <^demon> Ryan_Lane: Ok, that mistake is a one line typofix. But before I commit, I'm also getting: http://p.defau.lt/?vw_i3iUwnB3r6p_HM9kt_g [23:30:09] <^demon> I want to pick a gid and uid that will work for prod && labs (I can reinstall labs if needed). [23:30:10] okay, proxy works now. but still cannot connect. [23:30:30] i tried starting the service on both 127.0.0.1 and 0.0.0.0 [23:30:47] 09/26/2012 - 23:30:47 - User laner may have been modified in LDAP or locally, updating key in project(s): glam [23:30:51] can unfortunately, i cannot find where to add that -D in the wrapper. [23:30:52] really it would be better if we didn't have gerrit2 in ldap [23:30:53] I kinda want to kick people up the ass to pull from upstream on their puppetmaster::self boxes and make monitoring work well, going to end up just looking like the grumpy guy though. [23:30:54] yarr. [23:31:12] Ryan_Lane: that one hangs on this debug line: debug1: SSH2_MSG_SERVICE_ACCEPT received [23:31:28] try this: telnet bastion.wmflabs.org 22 [23:31:43] <^demon> Ryan_Lane: I don't need gerrit2 in ldap...it can live as a system user. [23:31:51] Ryan_Lane: btw did you talk to Leslie today or is she back tomorrow? [23:31:57] ^demon: it's needed for the hooks right now [23:32:02] <^demon> Only reason it needs to be in ldap is for the commit validation on operations/puppet. This would be better served by jenkins anyway. [23:32:15] Damianz: she answered an email, but I haven't seen her online [23:32:16] <^demon> That's the only thing in the hooks that needs gerrit login. [23:32:31] ^demon: puppet lint checks use it too [23:32:41] <^demon> That's what I meant, the puppet stuff. [23:32:43] but yeah, we could use another user [23:32:47] Ryan_Lane: that gives me a few lines and finally Connection closed by foreign host [23:32:53] Ryan_Lane: Ah :) At least she knows then, I just poked her with a 'is it possible question' just before she was heading out re blocking ts for ssh. [23:32:56] <^demon> It should be done as jenkins, and reported by JenkinsBot like everything else. [23:33:00] dan-nl: so it connects? [23:33:15] telnet bastion.wmflabs.org 22 [23:33:16] Trying 208.80.153.207... [23:33:16] Connected to bastion.wmflabs.org. [23:33:16] Escape character is '^]'. [23:33:16] SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu7 [23:33:17] Protocol mismatch. [23:33:17] Connection closed by foreign host. [23:33:20] yep [23:33:22] ok [23:33:30] <^demon> Ryan_Lane: We should talk to hashar about it...I can't imagine it'd be hard to make a jenkins job to do puppet linting. [23:33:36] dan-nl: try ssh again [23:33:44] to bastion.wmflabs.org [23:33:54] ^demon: Does gerrit actually enforce that it needs jenkins to have reviewed a patchset before it can be merged? Or is it just treated 'as a reviewer' and left up to the merger to check. [23:34:17] dschoon: Sep 26 23:29:40 i-000000ba sshd[11569]: error: connect_to nexus.pmtpa.wmflabs port 8081: failed. [23:34:20] ah, found it in the jsw conf file. maybe this'll work. [23:34:29] that's on bastion [23:34:29] ja, expected as the ipv6 issue still exists [23:34:34] * Ryan_Lane nods [23:34:34] ty tho [23:34:37] yw [23:34:54] Ryan_Lane: to bastion ... still hangs ... [23:34:55] <^demon> Damianz: Gerrit requires +1 in Verified and +2 in CodeReview to submit by default. Jenkins fills in Verified with -1/+1 [23:35:33] Ah, so it's kinda of enforced that if it fails it needs another person to be happy to merge [23:35:53] <^demon> In practice, we grant people Verified +1 as well. Every once and awhile you know better than jenkins, and want to merge anyway. [23:35:58] Assuming that it can go < 0 and is just sum()'d [23:36:13] Like when you break jenkins jobs :D [23:36:14] yaus. now running on 0.0.0.0:8081. now for the moment of truth... [23:36:21] <^demon> It's not summed. A -1 (or -2 for CR) is a veto. [23:36:24] HOORAY HTTP ERROR 404 [23:36:30] That is a vast improvement [23:36:33] hokay. awesome. [23:36:35] ty, buys [23:36:37] *guys [23:36:57] yw [23:36:58] Ah, I thought a -1 would require a +2 to make it a +1 or such [23:37:16] <^demon> Nope, no summing. [23:37:32] <^demon> Which is why the {-2,-1,0,1,2} is such awful terminology. [23:37:41] Yeah [23:37:50] <^demon> 40 +1's can't take the place of a +2 :) [23:38:02] <^demon> And -2's are special, in that 40 +2s can't override a -2 [23:38:05] <^demon> :) [23:38:22] Would make more sense to drop the numbers and just go with 'There's an issue with this change' < comment > or w/e the text it inserts is [23:38:32] ^demon: you should just not add a user if in labs [23:38:47] really the package should handle this [23:38:53] and not puppet [23:39:19] <^demon> No, the package shouldn't....the whole idea is being able to create a gerrit2 system user easily so you can receive replication. [23:39:38] <^demon> Unless the package handled it, and the gerrit host just doesn't use gerrit::account :) [23:39:44] I don't see why you need a forced uid for the replication client [23:39:45] <^demon> In which case, a slave would be weird. [23:40:14] <^demon> The uid doesn't really matter. Can I just omit it and let the system pick one? [23:40:30] using a package for a slave would make little sense as the keys are meh packaging wise, it's really just a normal user as far as I understand [23:40:48] ah. right... [23:40:58] <^demon> Well the private key doesn't need to be on the slave. [23:41:02] use systemuser [23:41:03] not user [23:41:04] <^demon> We just need to make sure the public key is added. [23:41:25] then it'll assign a uid from the system range [23:44:41] <^demon> Okie dokie, testing on gerrit-dev [23:48:35] <^demon> Ryan_Lane: http://p.defau.lt/?zwJBHHhABFasutHkAk3tVA :( [23:48:58] the problem is that gerrit2 already exists in ldap [23:49:04] with a different uid [23:49:09] and it's not allowed to change the uid [23:49:29] this is why we shouldn't have gerrit2 in ldap [23:49:41] <^demon> Well, if the user already exists shouldn't it just skip creating it? [23:50:00] I'd say create a new user for the hooks, change the hook config, and then we can delete gerrit2 [23:50:12] It will skip creating it then enforce the group [23:50:28] no, because puppet is fucking stupid [23:50:34] it looks directly in the passwd file [23:50:44] rather than using getent like it should [23:50:48] this is technically a puppet bug [23:51:40] <^demon> Even better, we should separate all 3 gerrit2 users into their own. [23:51:47] <^demon> 1) For hooks (or move it to jenkins) [23:51:59] <^demon> 2) For gerrit itself (this is gerrit2 -- package should do it) [23:52:18] <^demon> 3) A separate gerrit replication user (lightweight, systemuser, only has the ssh key for replication purposes) [23:52:22] yeah. that's sane [23:53:18] <^demon> gerrit.pp needs more cleanup :( [23:53:21] <^demon> It's still messy [23:54:42] developers.slashdot.org/story/12/09/26/1422218/malicious-phpmyadmin-served-from-sourceforge-mirror [23:54:47] Damianz: ^^ [23:54:55] <^demon> Ryan_Lane: We need to talk about packaging gerrit sanely, and sooner rather than later. [23:54:59] saw that the other day [23:55:03] <^demon> I can't keep waiting forever for google to release 2.5 [23:55:19] Build from git ftw! [23:55:37] ^demon: package is there to be modified :) [23:55:40] <^demon> That's the plan, but in a way that automates the hell out of it and we still feel safe deploying the results. [23:55:48] <^demon> It's not in git. [23:55:53] <^demon> And I know zilch about packaging. [23:55:55] it isn't? [23:56:08] I find rpms far easier to package than debs :( [23:56:17] I do too [23:56:18] <^demon> https://gerrit.wikimedia.org/r/gitweb?p=operations%2Fdebs%2Fgerrit.git;a=shortlog;h=refs%2Fheads%2Fmaster [23:56:36] ah [23:56:39] I guess I never added it [23:56:39] <^demon> Every time I try to package a deb I end up buried in Debian docs that make no sense. [23:56:51] you can get the source package in labs [23:57:20] Can you even tell debian to download the source from x and just change the version like rpms? [23:57:35] I've only ever seen a dir within a source tree for the package [23:59:25] ^demon: apt-get source gerrit [23:59:30] on build-precise1 [23:59:49] lemme put that into git