[00:20:08] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [00:20:38] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [00:50:08] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [00:50:38] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [00:55:38] PROBLEM Free ram is now: WARNING on nagios 127.0.0.1 output: Warning: 19% free memory [01:20:08] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [01:20:38] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [01:25:38] RECOVERY Free ram is now: OK on nagios 127.0.0.1 output: OK: 20% free memory [01:50:16] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [01:50:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [02:20:16] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [02:20:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [02:50:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [02:50:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [03:20:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [03:20:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [03:50:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [03:50:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [04:20:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [04:20:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [04:37:36] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [04:50:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [04:50:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [04:53:08] Change on 12mediawiki a page Wikimedia Labs was modified, changed by 98.210.205.93 link https://www.mediawiki.org/w/index.php?diff=504247 edit summary: /* Roadmap */ [05:20:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [05:20:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [05:50:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [05:50:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [06:20:26] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [06:20:56] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [06:51:09] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [06:51:09] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [07:15:19] PROBLEM Free ram is now: WARNING on nagios 127.0.0.1 output: Warning: 19% free memory [07:21:09] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [07:21:09] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [07:51:09] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [07:51:09] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [08:11:52] ? [08:11:55] Ryan_Lane: hi [08:12:02] howdy [08:12:10] are the deployment web instances down? [08:12:18] I changed main page a bit, I see no one removed it yet :) [08:12:25] heh. <3 the main page [08:12:30] Ryan_Lane: it seems they are, but site is up :/ [08:12:35] that doesn't make sense [08:12:46] cache? [08:12:54] logged in likely doesn't work [08:12:56] maybe I should try to edit some page [08:13:45] it must not really be down? [08:13:51] can you ssh into them? [08:14:02] no [08:14:11] let me check [08:14:13] petan: we all got a crack out of the main page, it's pretty great [08:14:37] it wasn't really hard to make it :D [08:15:00] no route to host... [08:15:10] anyway it seems that squid isn't able to connect to port 80 too [08:15:38] when you ssh to deployment-squid can you connect to port 80 of other instances? [08:15:55] I doubt it [08:16:01] because ssh can't route to the host [08:16:02] I can't [08:16:09] squid is using IP [08:16:12] and I can't ping it [08:16:29] it works [08:16:32] dns doesn't [08:16:36] 10.4.0.87 [08:16:38] is web2 [08:16:40] oh? [08:16:50] squid has ip [08:16:51] well. that's odd [08:17:04] no [08:17:06] the instance is dead [08:17:11] OOM [08:17:11] it pings [08:17:15] check the console log [08:17:15] try it [08:17:21] but it works :D [08:17:23] it's 10.4.0.37 [08:17:25] I can lynx to it [08:17:39] deployment-web is definitely dead ;) [08:17:40] lynx 10.4.0.87 [08:17:42] try it [08:17:44] it's web2 [08:17:56] that's probably why the site is up [08:17:59] yes [08:18:01] I'm talking about web1 [08:18:05] we need to reboot web1 [08:18:13] yeah [08:18:15] err [08:18:16] deployment-web [08:19:10] !log deployment-prep rebooting both web servers, starting with web1 [08:19:20] :o [08:19:22] web2 also OOM'd [08:19:23] where is bot [08:19:30] but it works [08:19:32] Logged the message, Master [08:19:37] that's weird [08:19:55] I'm putting the labs bots on their own instance [08:20:01] !log deployment-prep creating 2 more web servers to handle load [08:20:02] Logged the message, Master [08:20:11] ok [08:20:42] funny that apache stayed up on one of the boxes even after most processes were OOM'd [08:20:59] RECOVERY SSH is now: OK on deployment-web deployment-web output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [08:20:59] RECOVERY Disk Space is now: OK on deployment-web deployment-web output: DISK OK [08:20:59] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 92% free memory [08:20:59] RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 0 users currently logged in [08:20:59] RECOVERY Total Processes is now: OK on deployment-web deployment-web output: PROCS OK: 108 processes [08:21:11] RECOVERY host: deployment-web is UP address: deployment-web PING OK - Packet loss = 0%, RTA = 0.86 ms [08:21:11] PROBLEM host: deployment-web2 is DOWN address: deployment-web2 CRITICAL - Host Unreachable (deployment-web2) [08:21:11] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [08:21:38] heh [08:22:34] I need to make set up of web servers somehow automatic it suck to do it by hand [08:23:01] like I need to insert them inside squid [08:23:27] heh [08:23:29] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.07, 0.26, 0.13 [08:23:37] we do that via a deployment script [08:23:52] I think part of this has now been puppetized too [08:24:11] I still need to bring the deployment scripts over, though [08:24:54] well, off to bed. [08:24:56] * Ryan_Lane waves [08:24:57] ok [08:25:01] later :) [08:25:01] * Damianz tucks Ryan_Lane in [08:25:09] yay [08:33:57] PROBLEM Current Load is now: CRITICAL on deployment-web4 deployment-web4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:34:37] PROBLEM Current Users is now: CRITICAL on deployment-web4 deployment-web4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:38:07] meh [08:38:57] RECOVERY Current Load is now: OK on deployment-web4 deployment-web4 output: OK - load average: 0.40, 0.11, 0.04 [08:39:37] RECOVERY Current Users is now: OK on deployment-web4 deployment-web4 output: USERS OK - 0 users currently logged in [08:47:23] RECOVERY host: deployment-web2 is UP address: deployment-web2 PING OK - Packet loss = 0%, RTA = 1001.94 ms [08:48:23] RECOVERY Current Users is now: OK on deployment-web2 deployment-web2 output: USERS OK - 0 users currently logged in [08:48:23] RECOVERY Current Load is now: OK on deployment-web2 deployment-web2 output: OK - load average: 0.14, 0.04, 0.01 [08:49:43] RECOVERY SSH is now: OK on deployment-web2 deployment-web2 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [08:51:03] RECOVERY Disk Space is now: OK on deployment-web2 deployment-web2 output: DISK OK [08:51:03] RECOVERY Free ram is now: OK on deployment-web2 deployment-web2 output: OK: 80% free memory [08:51:04] RECOVERY Total Processes is now: OK on deployment-web2 deployment-web2 output: PROCS OK: 113 processes [08:51:08] RECOVERY dpkg-check is now: OK on deployment-web2 deployment-web2 output: All packages OK [09:15:23] RECOVERY Free ram is now: OK on nagios 127.0.0.1 output: OK: 20% free memory [09:36:51] petan: You need some moar servers [10:59:26] PROBLEM Free ram is now: WARNING on nagios 127.0.0.1 output: Warning: 19% free memory [11:49:26] RECOVERY Free ram is now: OK on nagios 127.0.0.1 output: OK: 21% free memory [14:55:43] hi Jeff_Green [14:56:02] hey sumanah [14:57:26] Jeff_Green: how's work? [14:57:58] decent. still kinda reorienting after shifting off of the fundraiser and ensuing cleanup [15:00:22] the switch to full-time telecommuting is a shift too of course [15:02:11] Hi I have a requiest ,I need a sysop permissions at http://bn.wikipedia.beta.wmflabs.org or what is the process to get it... I need some testing at there...! [15:03:28] jayanta: Annoy petan [15:03:56] Jeff_Green: I'm a fulltime telecommuter and yes, it does take adjusting [15:04:37] sumanah: where are you based? [15:04:47] Jeff_Green: New York City [15:05:14] Jeff_Green: you're in the Twin Cities? I cannot recall [15:05:16] oh you're in the East Coast contingent too! [15:05:26] I'm just north of you--Northampton Mass! [15:07:46] Oh right! [15:08:21] we're going to have to get a Northeast even on the schedule [15:09:36] Damianz: thanks ! but couldnt get u:) [15:10:38] I suppose I could do some stuff with that but meh busy person, we really need a requests page per projects for this stuff though I see from the mailing list its heading that way [15:16:57] should I start a RFA page at test?, Presently I am sysop in Bengali Wikipedia, user:jayantanth! [15:18:34] Also I need  all mediawiki configuration from bn.wikipedia.org to bn.wikipedia.beta.wmflabs.org [15:33:27] !accountreq [15:33:27] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [15:33:34] !questions [15:33:53] !account-questions [15:33:53] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [15:34:05] sumanah: [15:34:08] hi liangent [15:34:14] !account-questions | liangent [15:34:14] liangent : I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [15:36:15] sumanah: isn't the email addr that default one @users.mediawiki.org ? [15:36:27] hi ping [15:36:56] liangent: I do not know what you're talking about. [15:37:11] Does MediaWiki provide an email forwarding service using that domain name? [15:37:28] sumanah: 3. Your preferred email address. // Done Every commit needs to be rewritten to give name/email pairs to SVN users. We are using username@users.mediawiki.org for a unified e-mail address scheme for all old commits. [15:38:18] liangent: that's for OLD commits. I think. [15:40:03] sumanah: so among those 3 answers to those questions, which can be changed later? [15:40:17] email [15:41:36] sumanah: so 1. Liangent 2. liangent 3. liangent@gmail.com [15:41:56] Damianz: and if I change it later will existing commits be rewritten? [15:42:15] (I guess no) [15:42:31] Not sure how that ties in with svn tbh, I just know the other 2 are a bit of a PITA to change due to certain things [15:44:32] liangent: ok, give me a moment. [15:47:56] liangent: ok, now go to labsconsole.wikimedia.org and try to log in with the temp password you've been emailed [15:48:06] liangent: then, https://labsconsole.wikimedia.org/wiki/Special:PasswordReset [15:48:11] then Use the temporary password to set a password [15:48:25] then, liangent, https://gerrit.wikimedia.org/ - log in and add your ssh key. [15:48:26] got it? [15:49:00] sumanah: I didn't get a mail [15:49:09] liangent: wait 5 min. [15:53:07] hmmm! all are busy for other stuff...:'( [15:53:18] jayanta: you could email the labs mailing list. [15:54:16] sumanah: it's still not coming [15:54:45] liangent: that's odd. [15:54:57] Just wait one more minute, see if it went to spam folder [15:55:18] it's not in spam box either [15:55:49] sumanah: thx... [15:56:53] liangent: ok, this is odd. try just logging in at the webpage, labsconsole.wikimedia.org, and tell it you lost your password [15:58:23] sumanah: after passwordreset a temp password is there [16:00:47] I added my ssh key -- and what's the next step [16:01:04] can I clone production core&extension git repo now? [16:02:27] eh? labsconsole doesn't have interwiki [16:02:31] https://labsconsole.wikimedia.org/w/index.php?title=User:Liangent [17:49:26] PROBLEM Free ram is now: WARNING on nagios 127.0.0.1 output: Warning: 19% free memory [17:54:18] awjr: actually i'm not really sure who. wiki sysops are needed for some things but i don't think they necessarily all have access to make LDAP accounts [17:54:28] groovy thanks jeremyb [17:54:29] awjr: most (or all) of ops can [17:54:34] anyone around who can set me up with a labs account? [17:54:35] !account-questions [17:54:36] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [17:54:53] awjr: will speed things up if you answer the questions ;) [17:55:15] answers to 1 and 2 are 'awjrichards', 3 is arichards@wm.o [17:56:42] ssmollett is super idle [17:58:03] mut-away is even more idle [17:58:11] ryan's not here [17:58:53] maybe andrewbogott, maplebed or Jeff_Green? [17:59:38] * maplebed defers to ryan unless you can point me at docs to follow. I haven't done it before. [18:00:04] maplebed: i think i can actually! [18:00:12] awjr: I'm interested in trying, as long as we agree ahead of time that Ryan will have to pick up the pieces... [18:00:20] I think you can just do it via labs console. [18:00:37] maplebed: http://wikitech.wikimedia.org/view/OpenStack#Access_control [18:00:59] maplebed: we just add him to a project, right? [18:01:04] jeremyb: I have seen that interface work correctly at least once. Give it a try :) [18:01:20] Jeff_Green: no, you have to make his account first. he doesn't have one [18:01:47] I think he can do that part himself, via the UI [18:02:00] the part about 'on formey, run "add-labs-user --wikiname="" --mail="" " [18:02:04] we have to do that. [18:02:19] claiming formey to run that. [18:02:32] oh b/c he already has the SVN account [18:03:07] Jeff_Green: no, i think no scenario offers DIY. the docs cover people without accounts too [18:03:52] !log creating awjr's labs account by running add-labs-user --wikiname=awjrichards --mail="arichards@wikimedia.org" awjrichards [18:03:56] without svn accounts that is [18:03:59] creating is not a valid project. [18:04:02] labs-morebots: ping [18:04:06] there we go [18:04:11] lame. interface fail. [18:04:21] strange. this doesn't look like the procedure Ryan and I used to set up my account [18:04:34] but then half of that may have been done before the docs existed [18:04:49] awjr: you need ssh access into labs, right? [18:04:53] Jeff_Green: maybe you were already in the accountcreators group? (using a role account?) [18:05:08] maybe so, I'm not sure [18:06:01] !log bastion creating awjr's labs account by running add-labs-user --wikiname=awjrichards --mail="arichards@wikimedia.org" awjrichards [18:06:02] Logged the message, Master [18:07:01] * jeremyb suddenly wonders if awjr has 2 middle names [18:07:05] awjr: I added you to the bastion project; you should be able to log in there now. [18:07:12] 02/27/2012 - 18:07:12 - Creating a home directory for awjrichards at /export/home/bastion/awjrichards [18:07:16] !log bastion added awjrichards to the bastion project [18:07:17] Logged the message, Master [18:08:11] 02/27/2012 - 18:08:10 - Updating keys for awjrichards [18:18:21] * Thehelpfulone pokes petan|wk and petan [18:54:19] !account-questions | tfinc [18:54:19] tfinc: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [18:54:25] bots ftw! [18:54:36] nice [18:54:37] does you channel have a bot like this? [18:54:40] *your [18:54:45] @help [18:54:45] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.1.4 source code licensed under GPL and located in wikimedia svn [18:55:10] nope. we don't usually get questions that are as scriptable [18:55:18] Ryan_Lane: wm-bot is in #wikimedia-mobile as well, but we don't have anything setup for him/her [18:55:26] ah [18:55:31] it'll log the channel for you [18:55:33] we just use for logging [18:55:34] yeah [18:55:39] yuvipanda: we could set it up for how to create a mobile home page [18:55:43] it can also monitor changes on wiki pages [18:55:44] that one is pretty easy [18:56:25] ah. it's already logging your channel [18:56:28] http://bots.wmflabs.org/~petrb/logs/%23wikimedia-mobile/ [18:59:34] its in our chat room .. stealing our chats :D [18:59:42] heh [19:19:48] jeremyb: i do indeed have two middle names [19:20:08] maplebed, Jeff_Green: i see from the backscroll you guys started getting my labs account set up. thanks!!!!!! is it all set to go? [19:20:57] awjr: maplebed was the man on the job, and he's currently on the floor in our Ops meeting, not sure how far he got [19:20:57] !account-question | awjr [19:21:06] !account-questions | awjr [19:21:06] awjr: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [19:21:15] 27 17:55:15 < awjr> answers to 1 and 2 are 'awjrichards', 3 is arichards@wm.o [19:21:21] ah [19:21:38] Ryan_Lane: thanks, i already started the process and i believe maplebed started getting things set up [19:21:43] ah ok [19:22:01] Ryan_Lane: i know that johnduhart is also looking for an account [19:22:09] not sure he's actually around now tho [19:22:15] huh? he surely already has one [19:22:15] !initial-login | awjr [19:22:15] awjr: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [19:22:23] johnduhart already has one [19:22:31] ok cool [19:22:37] he set up part of deployment-prep [19:22:45] right [19:22:46] he was actually one of the initial guinea pigs :) [19:24:12] 02/27/2012 - 19:24:11 - Updating keys for awjrichards [19:24:29] Ryan_Lane: i got my jon's mixed up. i mean jon robson [19:24:36] * Ryan_Lane nods [19:24:36] although i dont see him online now [19:24:39] I need the same answers from him [19:24:41] i'll give him info on what he needs to do. [19:24:42] yeah [19:27:12] !account-questions | jdlrobson [19:27:12] jdlrobson: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [19:29:04] Ryan_Lane: do i recall correctly that you were asking people to use ssh keys for labs that are different from what they use on the cluster? [19:29:10] yes [19:29:17] awjr, Ryan_Lane: I did the ldap stuff and added awjr to the bastion host. I didn't do anything else. [19:29:26] I saw a bot here declare that awjr's homedir got created. [19:29:27] that's all you need to do [19:29:40] everything else is automated [19:29:44] then i add my ssh pubkey through the console, right? [19:29:47] yep [19:29:50] groovy [19:29:52] thanks guys! [19:29:55] !initial-login | awjr [19:29:55] awjr: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [19:31:09] jdlrobson3's multiplying [19:33:10] 02/27/2012 - 19:33:10 - Updating keys for awjrichards [19:34:29] Ryan_Lane: jdlrobson, jdlrobson, jrobson@wm.o (he /msg'd) [19:35:42] Ryan_Lane: do i need to do something special for a gerrit account? [19:38:19] awjr: no. did you read the initial log in doc? [19:38:22] awjr: try just logging in? you'll have to give it an ssh key in a different place than where you give it to labsconsole [19:38:28] it mentions to log into gerrit and set your key [19:40:19] Ryan_Lane: i have read it, but i am having trouble logging into gerrit - what are my creds supposed to be? [19:40:25] same as wiki [19:40:48] oh i had caps lock on for my pw. [19:40:50] :( [19:40:51] heh [19:41:38] cool i'm in thanks Ryan_Lane and jeremyb [19:41:41] yw [19:44:07] haha, caps lock ;) [19:47:45] hi all [19:47:51] ok so now i need to get an instance set up - and it looks like i also need a project set up? is that right? [19:48:10] or be added to one [19:48:19] what project are you going to be working on? [19:48:48] Ryan_Lane: mobile [19:49:05] if possible, i'd like to have my own instance for testing and blah [19:49:56] * Ryan_Lane nods [19:50:22] !projects [19:50:22] https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D [19:50:35] does a project already exist that you should be added to? [19:50:46] mobile exists iirc [19:51:00] mobile, mobile-sms, mobile-stats [19:51:06] hah [19:51:27] * jeremyb goes afk for a bit [19:54:25] Ryan_Lane: mobile i presume for now, not sure about mobile-sms or mobile-stats [20:10:58] <^demon|away> Ryan_Lane: Lots of new repos ;-) https://gerrit.wikimedia.org/r/#admin,projects [20:11:04] heh [20:11:19] nice [20:19:12] !account-questions | Jarry1250 [20:19:12] Jarry1250 : I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [20:23:20] sumanah: are they case sensitive? [20:23:25] <^demon|away> Ryan_Lane: We'll need to configure the filtering very soon, or ops channels are gonna get spammed with mediawiki/ repos :p [20:23:33] oh. let's do that now [20:23:52] <^demon|away> Unless you know of any exceptions, I'd say just filter everything in mediawiki/ [20:24:09] Jarry1250: I believe Ryan_Lane knows for sure, but : SVN/shell accounts should be all in lowercase, and wiki user name is probably case-sensitive. [20:24:17] Wiki username will start with a capital letter. [20:24:22] indeed [20:24:25] that's true [20:24:37] Who has power to update what wm-bot spouts? [20:24:39] we only accept characters, numbers, _ and - in usernames [20:24:51] and prefer them to be all ascii [20:25:00] sumanah: you should [20:25:07] anyone with a wikimedia-like flag [20:25:08] err [20:25:09] cloak [20:25:16] what do you want to change? [20:25:18] @help [20:25:18] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.1.4 source code licensed under GPL and located in wikimedia svn [20:26:18] Jarry1250 for the wiki then, and jarry1250, and jarry1250@gmail.com [20:27:19] hey Ryan_Lane are you there? [20:28:02] yep [20:28:21] ok, thanks Jarry1250, just a moment [20:28:31] Sure :) [20:29:55] did you receive my email about the labs account? [20:30:19] nope [20:30:21] as arthur poked me suggesting you hadn't [20:30:22] :/ [20:30:25] !account-questions | jdlrobson [20:30:25] jdlrobson: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [20:30:29] wird [20:30:58] just pm'ed you ryan [20:31:26] not sure what happened there then.. [20:33:12] !initial-login | jdlrobson [20:33:12] jdlrobson: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [20:33:13] 02/27/2012 - 20:33:12 - Creating a home directory for jdlrobson at /export/home/bastion/jdlrobson [20:33:29] you'll need to have one of the mobile folk add you to the right projects [20:33:41] if you need a new project, after talking with them, let me know and I'll create one for you [20:34:12] 02/27/2012 - 20:34:12 - Updating keys for jdlrobson [20:34:25] k thanks [20:38:12] 02/27/2012 - 20:38:11 - Updating keys for jdlrobson [20:51:03] 02/27/2012 - 20:51:03 - Updating keys for robh [20:51:04] 02/27/2012 - 20:51:03 - Updating keys for robh [20:51:06] 02/27/2012 - 20:51:06 - Updating keys for robh [20:51:12] 02/27/2012 - 20:51:12 - Updating keys for robh [20:51:16] 02/27/2012 - 20:51:15 - Updating keys for robh [20:52:02] 02/27/2012 - 20:52:02 - Updating keys for robh [20:52:03] 02/27/2012 - 20:52:03 - Updating keys for robh [20:52:06] 02/27/2012 - 20:52:05 - Updating keys for robh [20:52:12] 02/27/2012 - 20:52:12 - Updating keys for robh [20:52:17] 02/27/2012 - 20:52:16 - Updating keys for robh [20:55:46] New patchset: Ottomata; "Amending previos commit: installing python-mysqldb from deb. Not requiring mysql client class, this should be installed by the .dev if needed manifests/misc/analytics.pp - some comments" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2755 [20:56:31] I really need to modify that script to only send out one log message for updating keys [20:56:40] and it should show which project is doing so [21:03:59] PROBLEM Total Processes is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:04:39] PROBLEM dpkg-check is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:05:19] PROBLEM Current Load is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:06:09] PROBLEM Current Users is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:06:49] PROBLEM Disk Space is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:07:29] PROBLEM Free ram is now: CRITICAL on robh-1 robh-1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:22:08] !account-questions [21:22:08] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [21:23:51] wm-bot: 1. GrafZahl 2. grafzahl 3. Graf.Zahl@gmx.net [21:24:28] * GrafZahl_ leans back and lets wm-bot do its magic?! [21:24:43] No, that's not how it works [21:24:47] You need to send those things to Ryan_Lane [21:25:08] no. I was *just* saying they don't need to go to me :) [21:25:19] RoanKattouw: anyone that can make svn accounts can make gerrit accounts [21:25:36] I'm happy to help, but I can't make every single gerrit account [21:26:11] 02/27/2012 - 21:26:11 - Creating a home directory for jdlrobson at /export/home/mobile/jdlrobson [21:27:10] 02/27/2012 - 21:27:10 - Updating keys for jdlrobson [21:27:39] Oh OK [21:27:49] Do we have a queue for these things or something? [21:27:53] Hmm, should I perhaps send the data to a random person on the list https://meta.wikimedia.org/wiki/System_administrators#List ? [21:28:52] Probably best if you send it to Sumana [21:29:39] RECOVERY dpkg-check is now: OK on robh-1 robh-1 output: All packages OK [21:30:05] I have no clue how they are handling the switchover [21:30:14] I just know I don't want to link 200 accounts :) [21:30:19] RECOVERY Current Load is now: OK on robh-1 robh-1 output: OK - load average: 0.14, 0.07, 0.07 [21:31:09] RECOVERY Current Users is now: OK on robh-1 robh-1 output: USERS OK - 0 users currently logged in [21:31:49] RECOVERY Disk Space is now: OK on robh-1 robh-1 output: DISK OK [21:32:29] RECOVERY Free ram is now: OK on robh-1 robh-1 output: OK: 90% free memory [21:33:59] RECOVERY Total Processes is now: OK on robh-1 robh-1 output: PROCS OK: 80 processes [21:39:59] Still no luck trying to clone :( [21:43:15] Sumana was the first to tell me to ask someone else, so I've come full circle. What if I create a subpage of https://www.mediawiki.org/wiki/Git/New_repositories where everyone who wants a gerrit account leaves the info and the SVN admins migrate people at their leisure? [21:43:34] New review: Bhartshorne; "yay test branch" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2754 [21:43:35] Change merged: Bhartshorne; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2754 [21:43:55] New review: Bhartshorne; "yay test branch" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2755 [21:43:56] Change merged: Bhartshorne; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2755 [21:44:10] 02/27/2012 - 21:44:10 - Creating a home directory for awjrichards at /export/home/mobile/awjrichards [21:44:15] Slightly concerned I'm being firewalled by my uni [21:45:11] 02/27/2012 - 21:45:10 - Updating keys for awjrichards [21:46:29] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [21:48:00] Jarry1250: in which way is it failing? [21:48:19] Server hung up unexpectedly. [21:48:32] did it start the clone at all? [21:48:43] It creates some folders, .git I think [21:48:43] it's likely the server is just overloaded [21:48:51] can you post the output? [21:48:58] Possibly. I've tried like 15 times. [21:49:41] well, I can tell you if it's trying at all if you post the output [21:49:45] Well, the output is simply ssh: connect to host gerrit.wikimedia.org port 29418: Bad file number fatal: The remote end hung up unexpectedly [21:50:19] But for a while the appropriate .git subfolder - though no actual content AFAICT - appears. [21:50:34] bad file number? [21:50:47] can you please post the output? [21:50:56] with the command you are running and everything [21:52:31] hi all [21:52:40] Ryan_Lane, oh, sure. [21:52:42] One sec [21:53:23] quick question: how do I test some of my puppet changes on an instance? I haven't managed to figure it out and don't see anything on the wiki about it [21:53:39] danakim: you'll need to push the changes into the test branch for review [21:53:48] we'll review them, then merge them, then they will be available [21:53:51] ok, tried that. [21:53:53] aah [21:53:57] ok, that's why [21:53:57] it's an unfortunate model right now [21:54:00] Ryan_Lane: Simply http://pastebin.com/ipjdsZJN from git-bash on Windows [21:54:00] yeah [21:54:06] we'll eventually use a per-project branch [21:54:28] is it possible to set up environments in a way? per user environments? [21:54:39] or are there far too many users to try that? [21:54:49] can't use environments [21:54:54] and users would push to their private branch which is checked out on the master [21:54:57] ah [21:55:01] can't use a centralized puppet server, at all [21:55:22] so, we're going to switch it to where the repo will live on every instance [21:55:30] oh? you guys are not using a centralized puppet server in prod? [21:55:36] we are [21:55:44] but depending on how this works, we may kill that off too [21:56:00] hitting performance issues? [21:56:07] yep [21:56:42] yeah, a decentralized model would certainly help performance [21:57:41] fatal: '/test/mediawiki/extension': not a Gerrit project [21:57:43] but just to put it out there, we have had good performance by switching to nginx + unicorn on the puppet master. not sure what you guys are using now [21:58:02] still serving ~2000 clients [21:58:16] and we are not reaching capacity [21:58:38] Ryan_Lane: sorry, that would be the linebreak it's auto-inserted [21:58:47] just wanted to throw it out there, maybe you would take it into consideration [21:59:04] Jarry1250: git clone ssh://laner@gerrit.wikimedia.org:29418/test/mediawiki/extensions/examples.git [21:59:11] ah [21:59:14] it works for me [21:59:16] and it's quick [21:59:30] type this: telnet gerrit.wikimedia.org 29418 [22:00:12] danakim: we're using apache+passenger [22:00:53] Hmm, don't have telnet, have PuTTY though? [22:01:06] Will use that. [22:01:31] Jarry1250: windows has telnet by default [22:01:33] in the command line [22:01:43] nginx + unicorn might help a lot. we have also switched from apache + passenger [22:01:45] start -> run cmd.exe [22:01:58] Tried that, not found. [22:02:08] seriously? no telnet? [22:02:25] It would seem not. [22:02:28] wow, that is odd. I thought all windowses had it [22:02:36] I can't imagine it doesn't exist [22:02:44] Jarry1250, what version of windows are you using? [22:02:52] Jarry1250: is this your computer, or something given to you by work? [22:03:41] found out how to enable it. [22:03:43] Windows 7. It's mine, but I've had problems in the past because it's a share (university) network [22:03:44] danakim: Ah, [22:04:04] danakim: hm. we'll have to look into trying nginx + unicorn [22:04:21] Apparently it was disabled by default in Vista and later [22:04:27] * Ryan_Lane groans [22:04:36] dear microsoft: go to hell [22:05:14] disabling such a useful troubleshooting tool makes me want to stab [22:05:26] Okay, telnet enabled. [22:05:35] nice one [22:05:52] so you have to troubleshoot to get a troubleshooting tool [22:05:55] ...could not connect. Okay, definitely blocking me like with IRC. [22:06:01] lame [22:06:21] Jarry1250: tell your university that they are blocking the default port for every open source project using gerrit [22:06:31] And all IRC ports. [22:06:40] IRC doesn't surprise me much [22:06:51] though I think it's also stupid [22:07:18] Right, thanks anyway all, I'll be back soon enough when I get access :) [22:07:19] they are blocking their students from participating in open source projects, and they fail for that [22:07:21] Jarry1250, is ssh disabled to the outside world? and do you have a ssh box somewhere outside? [22:07:31] you could run some ssh tunnels [22:07:40] to bypass the limitations [22:07:40] he could clone the repo on bastion [22:07:45] or that :) [22:08:04] I had a look at tunneling, but no, I don't think I can quite manage it. [22:08:09] * Ryan_Lane nods [22:08:09] Anyway, I'm going to go winge. [22:08:29] Thanks all for your diagnostic help. [22:08:33] * Ryan_Lane nods [22:08:33] yw [22:09:34] Ryan_Lane, one last question. Where could I request some options to be added to the nova instance creation page [22:10:12] for example when creating a memcache instance you don't have an option to pass the listen address or the port [22:10:40] although the class supports those parameters, and I am guessing you use them in the production site.pp [22:10:59] just saying that some people might find it useful to be able to modify those [22:11:28] ah [22:11:35] danakim: you can modify the "puppet groups" [22:11:45] use the "Manage puppet groups" interface [22:11:55] in your project you can create a new group [22:12:07] then in that group you can add a classes and variables [22:12:21] they'll only show up for your project. if you are ready to them move them global, I can do so [22:13:10] ah, great! [22:14:23] thanks a lot for the info. I kind of got stuck in the workflow and how you do testing here :) [22:14:58] it would make things faster not having to push for review to test. I did try running the puppet agent against a locally cloned repo [22:15:20] but I am guessing that repo is missing a lot of stuff, as I was not succesful [22:15:58] anyway, I will stop complaing and just get to work :) [22:17:04] I'm working on per-project branches right now :) [22:18:02] ah, sounds good! and those won't require reviews, right? whatever is pushed to them will be run on the clients [22:18:18] yep[ [22:18:26] much better! [22:18:29] indeed [22:19:19] RECOVERY Free ram is now: OK on nagios 127.0.0.1 output: OK: 20% free memory [22:19:22] alright, tuning out. have a nice day or evening :) [22:21:29] PROBLEM Free ram is now: CRITICAL on bots-3 bots-3 output: Critical: 4% free memory [22:41:29] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 57% free memory [23:25:43] Krenair: hi! [23:25:44] Preferred wiki username. This will also be the user's git username, so legal name would be reasonable [23:25:44] Preferred email address [23:25:44] Their preferred shell account name [23:25:49] I need those 3 things from you. [23:26:22] Alex Monk [23:26:29] krenair@gmail.com [23:26:32] krenair [23:26:39] * RoanKattouw hands sumanah an !account-request [23:26:56] I don't like that particular boilerplate. [23:27:02] Ah, !account-questions [23:27:04] And I haven't bothered to fix it. [23:27:07] yet. [23:27:08] Would you like the boilerplate to be changed? [23:27:18] * RoanKattouw isn't sure if the bot trusts sumanah in this channel [23:27:26] I meant to change it but have not had a single moment to do so. [23:27:43] ok, just a moment Krenair [23:27:43] OK [23:27:49] I recognize the phenomenon S: [23:27:51] * :S [23:28:49] ok, Krenair - check your email for a password. [23:29:09] Then enjoy logging in! https://labsconsole.wikimedia.org/wiki/Help:Access and https://www.mediawiki.org/wiki/Git/Workflow will help you out. [23:30:29] PROBLEM Free ram is now: WARNING on nagios 127.0.0.1 output: Warning: 19% free memory [23:31:46] !account-questions [23:31:46] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [23:32:35] !account-questions is I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [23:32:36] Key exist! [23:32:52] !account-questions del [23:32:52] Successfully removed account-questions [23:32:54] !account-questions is I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [23:32:54] Key was added! [23:40:46] heh [23:40:53] switched around for easier copy/paste? [23:41:20] Yeah. [23:41:39] hi ryan, would you have some time this week to deploy http://rt.wikimedia.org/Ticket/Display.html?id=2429 ? [23:43:07] yes [23:43:52] awesome!