[00:00:00] 17 mb [00:00:27] it died [00:00:47] I can't even ssh there [00:01:05] cluebot crashed [00:03:32] RECOVERY Free ram is now: OK on bots-cb bots-cb output: OK: 87% free memory [01:49:26] * jeremyb is very late [01:50:03] petan: did you touch bots-cb? or it fixed itself? [01:50:45] box says it's up 27 days [01:55:43] * jeremyb will catch up on scrollback in a few mins [03:03:51] caught up! [05:17:18] petan: btw, i've historically been here 24/7 since my first /join i think. :) (re staying vs. using the logbot output) [05:22:27] jeremyb is, in fact, a logbot pretending to be human. [05:23:16] ouch [05:33:55] how does puppet know if it's running on a guest? [05:34:02] not bare metal? [05:34:21] (i.e. not labs vs. prod) [05:34:38] i don't see why the guests should have raid monitoring tools installed [05:57:53] PROBLEM host: bots-irc2 is DOWN address: bots-irc2 CRITICAL - Host Unreachable (bots-irc2) [06:03:03] RECOVERY host: bots-irc2 is UP address: bots-irc2 PING OK - Packet loss = 0%, RTA = 2.03 ms [06:38:23] Ryan_Lane: hey [06:39:08] * jeremyb sees `sudo puppetd -tv` @ [[access]] [06:39:23] don't work for me [06:39:33] have to do `sudo su - -c 'puppetd -tv'` [06:40:01] also, i was wondering why we don't use passwordless sudo? [06:41:13] how does puppet know if it's running on a guest not bare metal? (i.e. not labs vs. prod) i don't see why the guests should have raid monitoring tools installed [06:45:13] * jeremyb wakes Ryan_Lane ? ;-) [09:09:46] yuvipanda: hi [09:09:55] re logs [09:09:56] petan: hola! [09:10:11] if you wanted it to log everything I could implement it [09:10:21] but folks wanted it to log only text [09:10:27] not everything [09:10:34] from #mediawiki [09:10:41] petan: well, if consensus is to not log parts and joins, i'm fine with it [09:10:55] I don't care about consensus :D I can just ake a new feature [09:11:07] so you switch between full or partial logs [09:11:45] I just didn't know if you really need it or just wanted to know why it does log only a text [09:12:02] in case you really need it let me know and I will insert it there [09:13:12] petan: well [09:13:23] I was looking at logs and was trying to figure out what happened after I had quit [09:13:31] and no idea when I had, since it doesn't log parts and joins [09:14:09] right, I will consider inserting it there then, bot is using GMT time for logs [09:14:29] 9:14 right now [09:14:47] this mean I have to go :) [09:16:12] petan: :D [09:16:15] good night! [09:22:31] mornigng [09:22:37] 9am :D [10:12:21] hello [17:05:13] PROBLEM host: mobile-enwp is DOWN address: mobile-enwp CRITICAL - Host Unreachable (mobile-enwp) [17:18:53] RECOVERY host: mobile-enwp is UP address: mobile-enwp PING OK - Packet loss = 0%, RTA = 0.68 ms [17:23:52] jeremyb: have a look if you can get jenkins working on test [17:24:37] OrenBochman: why are we using jenkins? [17:24:42] * jeremyb is confused [17:25:06] anyway, how do i know the status of the import? is in a screen? [17:25:26] basicly it should get stuff from svn and then build + run tests [17:25:46] don't know [17:25:53] but it is completly independent [17:25:59] jenkins runs on tomcat [17:26:09] mw run on apache [17:41:30] notpeter: OrenBochman: hey [17:42:40] notpeter: so, first to catch you up a little: OrenBochman has created a labs instance in the Search project had some help setting it up from petan [17:42:56] notpeter: it's not running an import of simplewiki [17:43:17] notpeter: afaict it's just running in petan's bash (no screen or log) so i can't really see the progress [17:43:25] gotcha [17:43:54] OrenBochman: so, who's idea was it to run jenkins? i think that should maybe be a longer term goal? [17:44:30] i.e. first we should just get it working at all to be able to search for something. anything. and then we can work on hacking it? [17:44:44] notpeter: OrenBochman's on vacation for ~2 more days? [17:44:44] hi [17:44:50] hey! [17:44:57] jeremyb: there is no visible progress [17:45:04] huh [17:45:07] so, I've been thinking/talking to a coworker. does it seem reasonable to try to get this working with solr instead? It seems to have a lot of the functionality that's in the MW extention built in [17:45:09] you need to open wiki in browser and check stats [17:45:19] I can't see changes in terminal there is no output from script [17:45:20] disks not full [17:45:32] I mean there is a progress but I can't see it [17:46:00] notpet er: afaict it's just running in petan's bash (no screen or log) so i can't really see the progress [17:46:16] I meant, there is no console output :) [17:46:21] so you don't miss the fun [17:46:35] so, before you said how many were imported out of how many. how do i check that? [17:46:47] I am importing whole simple wiki [17:47:14] notpeter: i think that was a longer term plan. first to upgrade to use a more recent lucene in place and then to solr. but i was also wanting to investigate elasticsearch.org [17:47:17] so if you open the wiki you will see number of pages [17:47:29] now it's 104 000 from 300 000 [17:49:38] * jeremyb pokes OrenBochman [17:50:22] @RC+ mediawiki Wikimedia_Labs/* [17:50:22] Inserted new item to feed of changes [17:50:29] @RC+ mediawiki Wikimedia_Labs [17:50:30] Inserted new item to feed of changes [17:50:37] @RC- mediawiki Wikimedia_Labs* [17:50:37] Can't find item in a list [17:50:46] jeremyb: gotcha. that's a reasonable plan [17:51:51] notpeter: anyway, i guess my main question was what you've already done with search+puppet and what your current plan/knowledge of the existing infra is [17:52:17] notpeter: i.e. is there something we should be using to set up OrenBochman's search instance :) [17:52:53] I made a branch and started refactoring/templatizing the puppet setup. as for knowledge... I know more than many, but still not that much. rainman-sr is still the only one who knows it all [17:52:57] I think the answer is not yet [17:53:17] ok, where's the branch? [17:53:19] Change on 12mediawiki a page Wikimedia Labs was modified, changed by 80.188.9.191 link https://www.mediawiki.org/w/index.php?diff=480502 edit summary: /* Proposals */ new one [17:53:25] ah lol [17:53:29] I forgot to login :D [17:55:12] where is Ryan [17:55:27] btw what is meetbot :) [17:55:55] petan: it's the bot i was going to move to labs [17:55:58] we need to register them :) [17:56:04] that i haven't gotten around to yet [17:56:42] btw there is instance bots-2 which is idle most of time you could move it there [17:56:52] no one use it atm [17:57:13] we packages some bots it was used to test packages etc, no bots are running there [18:00:06] * jeremyb pulls a branch URL out of notpeter ;P [18:02:34] er, sorry, it's called testlabs/searchoverhaul. here's the gerrit infos: https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=summary [18:02:56] er, no. here is: https://gerrit.wikimedia.org/r/gitweb?p=operations%2Fpuppet.git;a=shortlog;h=refs%2Fheads%2Ftestlabs%2Fsearchoverhaul [18:09:25] *click* [18:09:43] (gerrit is mildly a mystery to me, fyi) [18:09:45] jeremyb: you are making irc bots? [18:10:27] just that wm-bot is in svn, so if you wanted you could improve it... [18:14:20] jeremyb: can you open the wiki [18:14:22] search [18:14:59] notpeter: yeah... i was in the git conversion meeting :) [18:16:15] * jeremyb fires up the wiki [18:16:23] the status is there [18:16:25] petan: just localhost:80/ or what? [18:16:32] yes [18:16:47] ./wiki [18:16:49] * /w [18:16:53] I didn't configure apache [18:17:00] it had no priority for me [18:17:13] once import is done I will fix it [18:17:30] ssmollett: hi [18:17:35] ssmollett: is Ryan around? [18:18:24] http://localhost/w/index.php?title=User_talk:127.0.0.1&diff=cur is funny :) [18:19:00] oh, *that*'s who Sara is? [18:19:23] jeremyb: Sara works on labs, is all I know [18:19:25] petan: she is 11 days idle... [18:19:39] RoanKattouw_away: yeah, i saw her in the labsconsole wiki [18:20:19] notpeter: so, when i click that link it doesn't mention the ref? [18:20:37] in the HTML [18:20:57] god damnit. ok. I'll try to hunt around for a proper link [18:21:04] * notpeter shakes first at gerrit [18:21:10] *fist [18:21:17] man, spelling... [18:21:21] it *might* be a log for that "branch" [18:21:37] it's just ... yeah, in need of fistshakes [18:22:07] oh, huh, it does on second thought mention the name. tiny print :P [18:22:12] at the top of the list [18:22:18] * jeremyb puts on glasses [18:22:24] (literally) [18:23:23] is ryan around? [18:24:32] ok, found glasses :) [18:27:05] jeremyb: ok, there is a number of pages, you see it? [18:28:41] petan: [[special:statistics]]? [18:28:44] yes [18:28:53] I believe that Ryan is traveling today. Not sure if he'll be in later. [18:28:58] ok [18:29:11] drdee: ^^ [18:29:35] OrenBochman: can you list all languages and extensions you wanted to have there? [18:29:53] you said that you wanted to create more wikis with different languages [18:31:19] jeremyb: thanks [18:31:52] it seems that my public keys are rejected for my two virtual instances on labs [18:32:18] drdee: you need to have a private key on bastion or forward it [18:32:24] @search forward [18:32:24] No results found! :| [18:32:28] @search ssh [18:32:28] Results (found 2): bastion, socks-proxy, [18:32:33] !bastion [18:32:33] ssh user@bastion.wmflabs.org [18:32:37] ah [18:32:44] I hoped it's there [18:32:57] petan: but did used to work, i get into bastion but i do not get to the actual instance [18:33:08] Ryan changes some things [18:33:13] ok [18:33:30] * changed [18:34:16] drdee: type !monitor instance name to see if ssh is ok [18:34:19] drdee: so, what are you trying now? what's in your ~/.ssh/config ? [18:34:33] !monitor search-test [18:34:33] http://nagios.wmflabs.org/cgi-bin/nagios3/status.cgi?host=search-test [18:34:33] !monitor bastion [18:34:34] http://nagios.wmflabs.org/cgi-bin/nagios3/status.cgi?host=bastion [18:34:41] petan: bastion1 [18:34:46] true [18:34:59] i thought the bot would actually check... [18:35:04] and tell me [18:35:08] heh [18:35:13] not [18:36:07] drdee: username? [18:36:19] oh, diedrik [18:36:40] """Yes! I am alive. """ [18:37:47] drdee: so where were you trying to ssh to? [18:39:06] reportcard1, master, pageviews are all green in nagios [18:39:49] drdee: ping [18:50:57] notpeter: ewwww, nfs [18:51:05] (reading your branch) [18:52:16] jeremyb: yeah, currently it's an integral part of how search works....... I'd like to change that. [18:58:40] notpeter: how is it integral? i was just seeing the crons [18:59:52] oh. in ways that aren't in puppet at all ;) [18:59:59] so, I'm not going to put them in now [19:00:02] uhuh... [19:00:21] search is almost completely not under version control [19:00:24] * jeremyb is not happy :/ [19:00:28] from an ops perspective [19:00:30] yeah [19:00:32] me neither [19:00:58] who were you talking to about solr? [19:01:19] binasher [19:01:24] ahh [19:02:08] i think i knew the diff between him and hashar and then i forgot and then i figured it out. /me tries not to forget again :P [19:02:34] hahaha [19:02:43] an understandable mistake [19:03:18] i've had people /msg me thinking i was jpostlethwaite [19:03:47] p, b, what's the difference? [19:10:17] $ python -c 'print ord("p")-ord("b");' [19:10:17] 14 [19:11:58] lulz [19:19:32] drdee: ping! [19:20:50] * jeremyb lures drdee to https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day [19:21:06] but we should fix his ssh problem before then :) [19:21:57] (assuming his hostmask is an actual indication of location) [19:57:29] OrenBochman: petan: please poke when you're around. i'll be back in ~30 mins [19:57:41] um [19:57:50] jeremyb: poke [19:57:51] :o [19:57:52] oh, hi [19:59:09] petan: so, i think someone told OrenBochman to set up jenkins but idk why. [19:59:13] 03 17:43:54 < jeremyb> OrenBochman: so, who's idea was it to run jenkins? i think that should maybe be a longer term goal? [19:59:16] 03 17:44:30 < jeremyb> i.e. first we should just get it working at all to be able to search for something. anything. and then we can work on hacking it? [19:59:23] jeremyb: no idea [19:59:30] he told me today if I could help him to set it up [19:59:36] any opinion? [19:59:37] I don't know jenkins much [19:59:55] jenkins is continuous integration aka automated tests [20:00:10] ok, I think we should first finish wiki [20:00:45] i think we can just run tests manually for the forseeable future. OrenBochman has limited time and it's concentrated all very soon. we should focus first on getting him something to hack on [20:00:47] I think we could use jenkins we have on integration [20:01:12] i.e. he's only on vacation for like 2 more days [20:01:20] right [20:01:54] if he tell me which tests he mean [20:02:15] anyway, i've acquired the robh/apergos "wait a long time to eat" condition :/ [20:02:23] i need breakfast [20:02:31] brb [20:20:45] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Petrb link https://www.mediawiki.org/w/index.php?diff=480567 edit summary: /* Proposals */ [20:46:26] petan: back [20:46:29] ok [20:46:51] what we needs now apart of wiki? [20:47:11] well, do you know much about how search works? [20:47:19] not [20:47:33] he told me that he wanted to set up some indexing jobs etc [20:47:35] there's an extension and a separate java daemon (also in svn) [20:47:44] right [20:47:52] did you check the extensions I installed? [20:47:57] nope [20:47:58] is it all he needs? [20:48:17] I installed wmsearch and openxml [20:48:23] so, i think we need to get to approximately a point where a page is indexed after it's edited [20:48:27] ok [20:48:37] * jeremyb is eating so not much typing [20:49:56] ok is that java daemon puppetized now? [20:50:10] I don't know if it's installed there [20:51:04] the java daemon is what notpeter was saying relies on lots of nfs [20:51:16] not puppetized at all [20:51:21] I can set it up [20:51:34] but I need to know what it needs [20:51:39] hwo it works [20:51:59] i only barely know [20:52:04] maybe notpeter has a laundry list [20:52:10] atm there is only one instance so I don't think there is anything nfs related [20:52:36] is he around? [20:52:37] idk what nfs is used for [20:52:43] i just hilighted him :) [20:52:52] he's around [20:52:57] let me go stalk him elsewhere [20:53:28] I don't know if we can start the daemon now [20:53:39] it could break stuff I want to fix db first [20:54:12] ok if there is a manual for that java stuff I would read it [20:54:24] petan: i asked him to come by but i think something just broke in prod so maybe he's working on that [20:54:32] ok [20:58:08] drdee: hey [20:59:10] * jeremyb wonders if drdee beeps when i say his name [20:59:20] jeremyb: hey [20:59:25] jeremyb: need help? [20:59:26] drdee: so, which host were you trying to ssh to? [20:59:34] i'm trying to help *you* :) [20:59:37] ah, it's other way round [20:59:40] :D [20:59:48] both reportcard1 and pageviews [20:59:57] permission denied, while this used to work [21:00:26] can you tell me about your local setup? is it agent forwarding or with ProxyCommand? [21:00:46] can you walk me through the process of connecting from a clean boot of your local machine? [21:00:51] 03 18:39:06 < jeremyb> reportcard1, master, pageviews are all green in nagios [21:00:58] okay, it seems that my public key is gone from bastion [21:01:06] so I ssh into bastion, no problem [21:01:19] so, even `ssh bastion1.wmflabs.org echo foo` don't work? [21:01:43] no, that works, i am in bastion1 [21:01:53] ok [21:02:03] so, walk me through your setup/steps [21:02:35] what OS are you coming from? [21:02:54] Windows7, using Putty [21:02:58] ahhhh [21:03:07] windows. :-/ :-/ [21:03:15] is paegent running? [21:03:27] but it used to work :) (before xmas) [21:03:28] no [21:04:39] you need paegeant running for the standard windows method [21:04:54] * jeremyb busts out the http://the.earth.li/~sgtatham/putty/0.62/htmldoc/ [21:06:24] C:\path\to\pageant.exe "c:\path\to\key.ssh" [21:07:50] jeremyb: that doens't help [21:07:56] i am in bastion1 [21:07:58] too bad i don't have a local putty handy [21:08:16] drdee: what is http://the.earth.li/~sgtatham/putty/0.62/htmldoc/Chapter4.html#config-ssh-agentfwd set to? [21:08:35] You've gotta check the box in putty to allow agent forwarding [21:09:28] Reedy: in case you missed it: this worked for him sometime last year. but idk if he saved the config or what [21:09:53] yes, i use the same config, i am pretty that my key is missing from .ssh on bastion1 [21:09:57] that's the problem [21:10:30] you got a ssh-agent running on bastion? [21:10:49] I'd recommend against putting your private key in .ssj [21:10:53] *.ssh [21:10:54] !pageant is C:\path\to\pageant.exe "c:\path\to\key.ssh" (it was written by Reedy, so blame him in case it fail) [21:10:54] Key was added! [21:10:56] :o [21:11:30] drdee: your key does not go onto bastion1 [21:12:07] !puttyfw is http://the.earth.li/~sgtatham/putty/0.62/htmldoc/Chapter4.html#config-ssh-agentfwd s [21:12:07] Key was added! [21:12:27] i think there is a miscommunication here [21:12:47] last year, i would ssh into bastion1 and then ssh to pageviews, or reportcard1 [21:12:54] exactly [21:12:57] now it says access denied [21:13:06] i haven't changed anything [21:13:08] it was reconfigured afaik [21:13:12] you forward your running ssh agent (pageant) to the other box [21:13:12] by Ryan [21:13:13] not the key [21:13:28] windows ain't really supported. at least as measured by having no docs for how to do windows [21:14:02] drdee: I can tell you another way to do this [21:14:04] i have port-forwarding enabled, that works [21:14:12] drdee: create a new key on bastion1 and insert it to wiki [21:14:15] but i cannot get into the server itself [21:14:22] so that you will login using labs key [21:14:24] agent forwarding is a separate and unrelated option [21:14:43] drdee: http://the.earth.li/~sgtatham/putty/0.62/htmldoc/Chapter4.html#config-ssh-agentfwd [21:15:01] !puttyfw del [21:15:01] Successfully removed puttyfw [21:20:32] petan: ok, i'll try doing that [21:21:40] drdee: just type ssh-keygen [21:21:43] on bastion [21:22:02] you will need to wait for puppet to distribute it [21:22:13] it's more safe than forwarding actually [21:22:38] because there is no need to copy private key to memory of bastion [21:22:51] so no one can steal your private key :o [21:30:54] thanks! [21:39:37] works? [21:39:43] ok [22:03:29] jeremyb: brb in 1 hour [22:03:41] if you needed anything just msg me and I will read it [22:04:46] k [22:05:14] instead of `ssh-keygen` better to do `ssh-keygen -t rsa -b 2048` [22:05:58] petan: the private key is not copied to bastion at all (not even memory) with forwarding) [22:12:41] drdee: i guess maybe you know already but dominic and katie both are here on IRC if you want to talk NARA [22:14:08] drdee: also i asked before about coming down to NY but it was probably lost. if you're in eastern .ca then maybe you want to visit https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day [22:23:04] Ryan_Lane: Back in SF? [22:23:56] andrewbogott: soon. in airport right now [22:24:05] and I have a doctor's appt tomorrow [22:24:51] Are you equipped to answer questions while @ the airport? [22:24:56] yep [22:25:06] for another 40 minutes or so :) [22:25:17] OK, here's an easy one: where is wgOpenStackManagerDNSOptions initialized? [22:25:53] all wg options are initialized in either LocalSettings.php, or in the extension's include file [22:25:59] the default is in the include file [22:26:05] LocalSettings overrides it [22:26:59] So, when it comes to the new ldap driver... where should it get all these values? Should I think about setting up a parallel LocalSettings.py that gets all the same vars? [22:27:12] it should be set in nova.conf [22:27:16] via FLAGS [22:27:28] err, and accessed via FLAGS [22:27:56] hey! [22:28:06] Ryan_Lane: where ya coming from? [22:28:07] Hm... can an individual driver define its own flags? [22:28:19] andrewbogott: yep [22:28:24] jeremyb: nola [22:28:39] jeremyb: also, you don't have sudo on the bastion host [22:28:41] no one does [22:28:43] except ops [22:28:51] Ryan_Lane: i figured. i was just curious :P [22:28:58] ok. And nova.conf will be created by puppet with all those values filled in? [22:29:05] Ryan_Lane: you probably saw what i was running :) [22:29:11] andrewbogott: yep [22:29:14] Ryan_Lane: did you see what i said about learning mode? [22:29:26] jeremyb: yeah, I updated the proposal, linking to it [22:29:34] okey [22:30:10] * jeremyb /msgs Ryan_Lane some scrollback [22:30:35] I saw the scrollback [22:30:39] that's why I answered you [22:30:54] we don't use passwordless sudo because it's less secure [22:31:17] sure but we don't care so much about security for most of these boxes? [22:31:36] of course we care about security. why even bother having projects otherwise? [22:31:47] Ryan_Lane: OK, next question. Will the dns driver ever be asked to create new zones, or only work within those defined by floating_ip_dns_zones and instance_dns_zone? [22:31:52] most of them not all of them [22:31:58] andrewbogott: create new zones as well [22:32:04] projects differentiate between them [22:32:08] andrewbogott: public and private, as well [22:32:23] anyway, at least i'd like to be able to have a different sudo password from wiki password [22:32:29] Ryan_Lane: OK, so you're expecting that this driver will handle requests other than those defined in the nova blueprint. [22:32:35] andrewbogott: we differentiate between public and private using the "location" attribute in DNS [22:32:46] yeah. I wrote the blueprint ages ago [22:32:56] jeremyb: why? [22:33:32] andrewbogott: I'm hoping they will accept features outside of the blueprint :) [22:33:51] Ah, so these extra features will also need to pass through the nova api. [22:33:56] * Ryan_Lane nods [22:34:16] yeah, I really should have updated the blueprint as I figured these things out in OSM [22:34:27] I'm sure they'll accept extra features, but we're in some danger of getting caught by a feature freeze. So we should probably have a go at enumerating the full list of features. [22:34:38] yeah [22:34:44] let's get the basic functionality in first [22:34:56] then aim for the extended feature set next release [22:35:17] unless you think it's possible to get them all in for this release [22:35:42] jeremyb: a different password isn't possible, and I don't really think it's necessary [22:35:58] jeremyb: if someone can log into the wiki as you, they can change your key [22:36:07] exactly. that's my point [22:36:17]