[03:04:09] Coren, Ryan_Lane, et al.: i think i know the answer but I want to double check. hosting services that need prod support is not ok. what about the staging or development env for a public+libre mediawiki site hosted elsewhere? [03:04:51] jeremyb: acceptable [03:05:08] That'd need clearance from LCA. Officially, the Labs is meant to support the Wikimedia projects, Mediawiki development and "closely allied projects" (OSM being one) [03:05:09] in fact, I approved webplatform some time ago [03:05:24] Coren: anything related to MediaWiki dev is acceptable [03:05:58] I must have misunderstood the question then, because that sounded like staging for a site, not wm dev. jeremyb? [03:06:05] staging/development [03:06:17] for a mediawiki site [03:06:32] Then I'd suppose it okay. [03:06:43] in my opinion we should be doing everything in our power to support a healthy MW ecosystem [03:07:16] since the foundation does its best to destroy the third party ecosystem [03:07:58] in this particular case we need to move off the current host but first step is to actually replicate the existing instance *somewhere* and make sure it still works right, etc. [03:08:34] but there will also be work on porting to newer mediawiki, evaluating extensions they may want to use, etc. [03:08:52] yep, that sounds like an acceptable use to me [03:09:08] basically all third party sites at some point will do MW dev work [03:10:15] next question is project vs. tool [03:10:23] well... [03:10:30] I'd say project for now. [03:10:38] i was leaning toward project. right, good :) [03:10:40] we really need to make that MW project [03:11:22] so, would you make me a crisiswiki.org ? [03:11:33] or crisiswiki i guess [03:12:35] sure [03:13:50] it's only like ~5k pages total (all NS), ~17k revisions, and no uploads. per special:statistics [03:13:56] i think that's peanutes [03:14:00] peanuts* [03:14:03] you're going to import all the content? [03:14:11] maybe... [03:14:19] make sure it's not crawled [03:14:19] at least will import localsettings [03:14:21] right [03:14:29] could do htaccess or just robots.txt [03:14:58] I think the default robots.txt when using the puppet role will work [03:15:07] I'd make sure to check, though :) [03:15:11] i will [03:15:36] jeremyb: done [03:15:42] i see https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Crisiswiki&action=history :) [04:33:17] who's responsible for instance-proxy? [04:33:36] http://instance-proxy.wmflabs.org/ could give some docs maybe! [04:34:58] YuviPanda: ping? [04:35:04] i see a liangent but he's not here [04:36:11] i mean he's on the page [04:36:14] https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy [04:39:16] or should i just get a public IP? [04:39:24] http://80.crisiswiki-staging01.instance-proxy.wmflabs.org/ [04:39:27] http://crisiswiki-staging01.instance-proxy.wmflabs.org/ [04:39:37] 504 [04:39:51] andrewbogott_afk: YuviPanda: ^ [04:39:57] * jeremyb sleeps [04:47:05] jeremyb: Did you add/modify a security group to your project that allows port 80 traffic? I had a similar issue with the last project I setup do to not knowing that was needed. [04:47:19] bd808: i figured that out about a minute ago [04:47:29] :) [04:47:30] bd808: but that just gets back to what i asked above about docs [04:47:34] 08 04:33:36 < jeremyb> http://instance-proxy.wmflabs.org/ could give some docs maybe! [04:47:45] bd808: danke! [04:47:52] Yeah that would be reasonable [04:47:53] > Failed to add rule. [04:47:58] huh [04:49:45] bd808: any ideas? :) [04:49:57] *seems* like a simple rule i'm trying to add [04:50:01] 80 80 tcp 0.0.0.0/0 didn't work? [04:50:15] gah [04:50:27] i swear i tried that before [04:50:39] but when i tried again it worked... [04:50:45] was trying varioius permutations [04:50:52] labs is fickle and tricksy [04:51:43] Also, somebody told me that new security groups can't be added to existing instances. [04:52:06] jeremyb: pong [04:52:12] instance-proxy isn't my thing :D [04:52:32] YuviPanda: But you get blamed for it repeatedly [04:52:44] bd808: heh yeah [04:52:47] should rewrite it [04:52:50] bd808: i have an idea [04:53:03] they don't support ws for example [04:53:11] bd808: maybe one of the various failures was 0.0.0.0.0/0 instead of 0.0.0.0/0 [04:53:24] bd808: found that in my browser form history :P [04:53:39] Nice. IPv4.5 [04:54:35] still the error msg could be better! [04:54:39] and do some validation before submit [04:54:49] haha! [04:54:50] * jeremyb is so critical today :P [04:55:00] jeremyb: submit a patch! [04:55:20] YuviPanda: how do i patch http://instance-proxy.wmflabs.org/ ? [04:56:28] jeremyb: operations/puppet.git manifests/role/labsproxy [04:56:30] .pp [05:00:37] is there no more bot announcing when people change passwords/keys? [05:11:37] Ryan_Lane: where's that bot that announces key and password changes? [05:12:01] it never mentioned password changes [05:12:07] yes it did! [05:12:07] it just mentioned when a key had been updated [05:12:15] I don't see how that would be possible. [05:12:23] oh, right [05:12:31] maybe key updates happened when you changed your password? [05:12:34] it said "a user has either updated a key or changed a password" [05:12:41] because it was based on a user change [05:12:59] no, i don't think it said it. i just figured it out when i changed password and that resulted in a ping [05:13:02] I have no clue what happened to that bot [05:13:12] there's no way it could have known that [05:13:30] all it knew was that your account was updated [05:13:34] sure [05:13:46] but it knew enough to ping me here :) [05:13:58] also, home dir creation [05:14:07] iirc [05:14:23] well, homedir creation doesn't happen anymore [05:14:28] is it normal that get console output is so bare? [05:14:32] it's automatic via pam_mkhomedir [05:14:37] o [05:14:38] *no [05:14:41] it's like 2-3 lines of output for first run [05:14:51] no boot history [05:14:53] it's because the new images aren't directing the console output to the right tty [05:14:59] ah [05:15:20] I haven't been able to figure that out, not that I've had a lot of time for it [05:15:32] is there a bug? [05:15:50] yes [05:15:52] no [05:15:53] maybe [05:15:53] :) [05:18:04] Ryan_Lane: better details on https://bugzilla.wikimedia.org/45768 now :) [05:18:32] which i found by accident while looking for console goes to wrong tty [05:37:04] liangent: is instance-proxy your thing? [05:37:16] hah, bye :) [05:38:11] jwalling: were you able to ssh in? at least to bastion? [05:38:38] no i see you don't even have a key on wikitech yet [05:39:17] jeremyb: not yet - i am reintroducing my self to ssh - it's been awhile [05:39:38] jwalling: you should be able to copy/pasta from the wiki section i linked [05:39:42] (proxycommand) [05:44:51] jeremyb: we've removed all non-staff from the proxy project [05:45:05] it now holds the yuviproxy [05:45:12] which has a *.wmflabs.org cert [05:45:34] so, instance-proxy was managed by liangent at some point, but no longer [05:46:13] Ryan_Lane: oh, are there docs for the yuviproxy? [05:46:19] probably not [05:46:27] it's actually dark launched right now [05:46:46] we have a proxy special page that'll let you create proxies in your project [05:47:00] it manages DNS and talks to yuviproxy via an openstack-like api [05:47:01] oh, it's integrated, huh [05:48:15] next step is to start making everyone use the proxy and to remove public IPs from projects :) [05:48:20] Ryan_Lane: re non-staff i assume that's similar reasoning to the beta ssl change. what are the criteria exactly? does it need a specific type of NDA? [05:48:34] unless they are doing something more than just running an http service [05:48:44] no, any kind of NDA is likely fine [05:49:06] ok, well then i can help with either of those projects if you like :) [05:49:13] * Ryan_Lane nods [05:49:23] I'll let andrew/yuvi make that call :) [05:49:29] they've been working this [05:49:30] i just tend to read some bugmail like 2 weeks later [05:49:34] I only helped design the API [05:49:49] e.g. the beta ssl cert :) [05:49:53] yeah [05:50:10] did sudo get sorted finally? seemed uncertain [05:50:25] i guess i could look myself [05:50:55] no idea [05:51:00] I'm not doing any of that :) [05:51:11] I'm just orchestrating efforts [05:51:45] oh, is there not a way to just say "all projectadmins"? [05:51:51] i see everyone listed explciitly [05:51:55] explicitly* [05:52:18] no [05:52:24] because projectadmins isn't a group [05:52:25] it's a role [05:52:47] the instance doesn't know about it [05:57:02] good night (aka morning) [05:57:08] * Ryan_Lane waves [11:13:28] tools-webserver-01 overloaded: http://ganglia.wmflabs.org/latest/graph_all_periods.php?h=tools-webserver-01&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1383909111&g=cpu_report&z=large&c=tools [13:56:36] Coren: I need you to check / reboot instance hugglewiki [13:57:31] ssh: connect to host hugglewiki port 22: No route to host [13:57:31] Nothing to check; it's dead. [13:57:36] meh [13:57:43] maybe it will start up one day... [13:58:08] yeah it's booting up now [13:58:12] but it was stucked before [14:07:34] It's dead, Jim! [14:11:14] now it has "shutoff" status o.O [15:42:58] YuviPanda: Regarding backups… if I restore the db on a new proxy box, how will the old proxy entries get relayed to redis & nginx? [16:17:21] andrewbogott: they need to get relayed back to redis only. I'll need to write a restore script for that. Shouldn't be too hard [16:17:31] andrewbogott: I'm leaving to India today, so will definitely be able to get to it next week [16:24:02] YuviPanda: ok, thanks [17:08:31] Permission denied (publickey) error though I have added public key [17:11:39] harshkothari: what OS are you using? [17:20:53] mac os x [17:23:47] harshkothari: Can you run your ssh command with -v and then paste the command + output to dpaste? [17:28:08] andrewbogott: its working and I am pasting that output [17:28:40] andrewbogott: http://pastebin.com/bHWcH6C2 [17:30:36] harshkothari: Looks like you logged in successfully? [17:30:41] Or am I misreading? [17:31:42] andrewbogott: yes right now I am successfully logged it [17:31:43] *in [17:31:58] harshkothari: OK, so, problem solved? [17:32:56] andrewbogott: yes now problem solved. Dont know what is the problem but thanks :D [17:33:04] ok! [17:33:06] :) [17:52:06] (03PS1) 10MarkTraceur: Move the oojs repos to the VE channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/94388 [17:53:41] (03PS1) 10MarkTraceur: Add multimedia channel and move their extensions [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/94390 [17:53:55] YuviPanda: ^^ [19:18:49] hey folks – J-Mo, ashaw, halfak, myself and a bunch of other people are hosting a global hackathon tomorrow [19:19:05] we're going to have attendees requesting Labs access starting tonight and continuing over the weekend [19:19:46] is there anyone we can count on in different timezones (ideally Western Europe and Central Time in the US) who could lurk in the #wikimedia-labs2 channel and help process these requests as they come in? [19:21:18] ashaw: you should stop hacking and join #wikimedia-labs2 at some point :p [19:22:55] :) [19:22:57] sure thing~ [19:30:31] DarTar: I'm in US Central and can lurk for part of the day tomorrow… especially if you email to remind me when the time arrives :) [19:30:56] DarTar, is there a specific time window for the hackathon? [19:31:05] andrewbogott: fantastic, halfak ^^ [19:31:37] Hey andrew. Want to come hang out with us @ UMN tomorrow? [19:32:00] halfak, maybe, where/when? [19:32:18] 10:30AM 200 Union St SE [19:32:25] "Keller Hall" [19:32:34] * andrewbogott makes a note [19:32:44] You can bring your bike inside if you are brave enough to ride in this weather. [19:33:01] ooh, is it bad up there already? [19:33:09] Is it close to the #2? [19:33:16] ashaw, not very snowy, just a bit cold. [19:33:19] * andrewbogott back in a bit [19:34:11] * ashaw nods [19:42:46] Can someone help us get this Tool Labs access request approved? https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Aaronshaw [19:54:20] YuviPanda: Do you have time to help me get some people approved for Tool Labs use? [21:42:50] (03CR) 10Catrope: [C: 032] Add multimedia channel and move their extensions [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/94390 (owner: 10MarkTraceur) [21:43:02] (03CR) 10Catrope: [C: 032] Move the oojs repos to the VE channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/94388 (owner: 10MarkTraceur) [22:02:28] Is something weird going on with tool labs? I keep randomly getting 500 Internal server errors or other weird random errors when loading pages on it. [22:03:05] Now I get a 502 Proxy error [22:03:15] Coren: ^ [22:03:57] And a few minutes ago I got "The URI you have requested, appears to be non-functional at this time." [22:04:06] All from the same page [22:04:41] kaldari: OOM; sometimes happens on the shared webserver when something crawls us -- the way to protect yourself is to switch to the per-tool webservice scheme: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb [22:04:52] I'ma go see what is crawling us this time. [22:07:55] Something is hammering *hard* on /spellcheck/ [22:15:50] kaldari: Feeling better now? [22:16:45] Coren: Yeah, it seems to be getting back to normal [22:16:47] thanks! [22:18:38] * Coren hates crawlers that don't obey robots.txt [22:27:23] hello folks - looking for help getting up and running with my tool labs account [22:27:32] i believe it was just approved a few hours ago [22:27:34] ashaw, what's happening? [22:27:37] (thanks for that) [22:27:59] here's the latest verbose output from the ssh -i command: http://pastebin.com/g4Rz62tb [22:28:08] what's your on-wiki username? [22:28:12] aaronshaw [22:28:19] however, my login is ads [22:28:47] as in, my instance shell account is ads [22:28:54] (sorry for the confusion) [22:29:27] nope, that's normal to have two different names. [22:29:59] back in my days i think that was not possible [22:30:03] for sure - i was apologizing for not remembering the "name for it :) [22:30:30] giftpflanze: I mean, one name on wiki and one for shell. Usually wiki names have illegal shell characters. [22:31:31] hm [22:31:49] ashaw: did you file a tool-labs request? If so can you link me to it? [22:32:37] andrewbogott: i believe i did. let me look for it [22:33:10] If I were smarter I would know how to search [22:33:29] btw, looks like your wikiname is Aaronshaw <- case matters, sometimes. [22:34:36] ooh, sorry about that [22:34:38] yes [22:34:45] hmm, re: the tool-labs request [22:34:53] i definitely submitted the request [22:35:00] in my notifications it looks like you approved it... [22:35:17] Yeah, looks that way to me too :) I see you having a key 'ads@madrone' is that the one you'd expect? [22:35:52] (My doorbell is ringing, brb) [22:36:09] yes [22:40:10] andrewbogott: it occurs to me that perhaps I thought my request had been processed and it still hasn't been... [22:41:12] halfak was trying to help resolve that earlier - it seems that when I got the notification about being shuffled around in the queue I incorrectly thought it meant I had been processed! [22:42:02] ashaw: I'm pretty sure I approved you an hour ago. Try now while I watch the log? [22:42:23] sure [22:43:58] just tried it again. same error [22:44:02] did you see anything? [22:45:27] I don't even see you in the log, not sure what that's about. [22:45:37] Are you able to ssh ads@bastion.wmflabs.org ? [22:45:44] let me try that... [22:46:09] nope [22:46:26] would you like to check out the -vv output? [22:49:00] ashaw: Sorry if I asked you this already… you on OSX? [22:50:55] ashaw: And, if you omit the -i arg to ssh does it fail in just the same way? [22:53:16] andrewbogott: ashaw: http://www.baptiste-wicht.com/2010/07/tip-how-to-solve-agent-admitted-failure-to-sign-using-the-key-error/ [22:53:50] andrewbogott: Ubuntu 13.10 [22:54:01] and here more specifically: https://help.github.com/articles/error-agent-admitted-failure-to-sign [22:54:06] i'll try removing the -i and reading those links [22:54:13] lazowik: Yeah, I saw a page like that, but… don't understand why that would matter :) [22:54:18] Which isn't to say that it won't help [22:54:33] removing -i won't work [22:55:12] andrewbogott: looks like agent gets cluttered up in some way [22:55:32] you can either relogin or ssh-add to make it reload the key [22:55:46] aha [22:55:50] let's see about that.. [22:56:14] bingo [22:56:18] that's the winner. [22:56:23] :) [22:56:29] thank you both so much andrewbogott & lazowik [22:56:36] np [22:56:43] oh man, the solutions are always so dumb! [22:56:46] i'm off to edit halfak's tutorial for the hackathons now :) [22:56:48] totally! [22:56:54] Turning it off and on again totally would've fixed this. [22:57:05] * andrewbogott turns in his techs support credentials [22:57:06] ashaw: just added the key I presume? [22:57:37] s/added/created/ [22:57:48] that seems to explain that: https://coderwall.com/p/dbrdbg [22:57:51] yep [22:57:56] ssh-add did it [22:58:07] * ashaw bangs head on table [22:58:56] well, one could think that I'd try to ask for key passphrase if agent fails [22:59:04] mh, is that in the docs, if not it should [23:00:43] haha s/I'd/it'd/ [23:01:07] not the most informative error message... [23:01:29] anyhow, it's working now. and hopefully, a few other people can avoid this problem tomorrow during the other hackathons [23:05:03] "Requests for access are generally dealt with within the day (often faster)" [23:05:06] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools#Current_queue_.5B_link_.5D: [23:05:07] :p [23:09:25] OK… before I wander away from the keyboard again, is anyone stumped for lack of access just now? [23:09:37] me not :) [23:09:56] it's better to just ask here i guess