[00:24:33] PROBLEM Current Users is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:25:13] PROBLEM Disk Space is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:25:54] PROBLEM Current Load is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:25:54] PROBLEM Free ram is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:27:23] PROBLEM Total processes is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:29:33] PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: Connection refused by host [00:32:23] RECOVERY Total processes is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: PROCS OK: 88 processes [00:34:33] RECOVERY Current Users is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: USERS OK - 1 users currently logged in [00:34:33] RECOVERY dpkg-check is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: All packages OK [00:35:12] RECOVERY Disk Space is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: DISK OK [00:35:52] RECOVERY Current Load is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: OK - load average: 0.27, 0.69, 0.58 [00:35:52] RECOVERY Free ram is now: OK on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: OK: 848% free memory [00:54:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [01:08:52] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 176 processes [01:13:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:22:56] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [02:37:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [02:37:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 24% free memory [02:38:03] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [03:06:03] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 17% free memory [03:10:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [03:31:03] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 36% free memory [06:23:44] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [06:26:56] 12/27/2012 - 06:26:55 - Updating keys for techman224 at /export/keys/techman224 [06:28:23] PROBLEM Total processes is now: WARNING on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS WARNING: 151 processes [06:29:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 151 processes [06:39:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 146 processes [06:48:22] RECOVERY Total processes is now: OK on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS OK: 148 processes [10:21:45] !log bots installed python-irclib on bots-4 [10:21:49] Logged the message, Master [10:23:51] !log bots autoremoved libzmq1 on bots-4 [10:23:52] Logged the message, Master [11:01:42] PROBLEM Free ram is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Critical: 5% free memory [13:01:58] 12/27/2012 - 13:01:58 - Updating keys for mwang at /export/keys/mwang [13:25:34] if I can ssh into bastion, but get 'permission denied' when trying to ssh into an instance once in bastion, what might I be doing wrong? [13:26:15] olaph: you need to configure agent forwarding [13:26:21] See https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [13:38:42] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 26% free memory [13:41:40] Hydriz: got it. thanks! [13:41:44] np [13:54:42] !log bots installed python-dev on bots-4 [13:54:44] Logged the message, Master [14:11:31] !resource i-00000558 [14:11:31] https://labsconsole.wikimedia.org/wiki/Nova_Resource:i-00000558 [15:57:25] 12/27/2012 - 15:57:25 - Creating a home directory for pbeau at /export/keys/pbeau [16:01:10] 12/27/2012 - 16:01:09 - Updating keys for pbeau at /export/keys/pbeau [16:17:15] \ [16:23:09] \/ [16:28:09] ~ [18:22:23] New patchset: Andrew Bogott; "Remove Ben's key and add mine." [labs/private] (master) - https://gerrit.wikimedia.org/r/40806 [18:23:17] Change merged: Andrew Bogott; [labs/private] (master) - https://gerrit.wikimedia.org/r/40806 [18:23:59] hi andrewbogott, how was your Christmas? [18:24:30] not bad! yours? [18:24:47] Oh, except I went to Les Miserables which… I did not enjoy. [18:25:37] :( the movie? [18:25:50] I'm sorry to hear that, I was hoping it was enjoyable and am thinking of seeing it [18:26:13] Mine was very quiet. I talked to nearly no one and I read fanfic. It was awesome. [18:26:35] sumanah: Well, I have no loyalty to the work itself, so I'm not a good judge probably. [18:26:46] I've never read the book or seen the live show [18:27:04] except, I used to be a fan of a particular sketch comedy troupe [18:27:43] and one of its actresses did this lipsync/ventriloquism thing [18:28:01] I've read much of the book, and I was a little shocked/puzzled at how they elide hundreds of pages in a single line. [18:28:06] where she had a lineup of Pez dispensers lip-sync to the song 'One Day More' [18:28:20] Like, if you're starting in the middle why am I supposed to give a shit about these characters? [18:28:40] But everyone I went with was weeping and cheering and such, so probably I'm just a heartless monster. [18:28:49] I have always known that about you [18:29:01] you eat hope for breakfast [18:29:07] and love for dessert [18:29:16] I can't deny it. [18:29:35] * sumanah is attempting to do the sarcasm/insult-as-bonding thing, is not very practiced at it [18:30:29] also, nice use of "elide" - not enough people use that [18:31:27] I recently listened to a lecture series about 19th century french history which had kind of a communist perspective. So all during the movie I had this kind of marxist commentary running in the back of my head… that probably limited my enjoyment as well. [18:32:32] * sumanah laughs pretty hard [18:32:33] yeah [18:32:35] that lecture series, is it something you'd recommend? I think Leonard is looking for his next history podcast because he's done with The History Of Rome [18:33:51] The content was interesting… the teacher has some verbal ticks which are grating. It was, let's see... [18:34:10] "France Since 1871", the lecturer is Merriman [18:34:58] * sumanah is listening to an album of US Civil War songs [18:36:30] But my favorite iTunes U thing so far is this: https://itunes.apple.com/us/itunes-u/world-economic-history-before/id405570098 [18:41:23] Oh it's free! [18:41:24] nice [18:44:37] ori-l, did you by chance try to update kubo from lucid to precise? [18:46:34] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 27% free memory [18:51:26] 12/27/2012 - 18:51:26 - Updating keys for mwang at /export/keys/mwang [18:56:52] RECOVERY dpkg-check is now: OK on kubo.pmtpa.wmflabs 10.4.0.19 output: All packages OK [19:00:07] Hi! Is it possible to give me an access to web-tools in order to in order to work on the installation of OpenRefine as suggested in the mailing list? http://lists.wikimedia.org/pipermail/labs-l/2012-December/000639.html Seb35 hasn't the time to work on it. [19:01:45] hi guys. If i request a Gerrit/Git project for my bots and webtools code on Labs - what path should I choose? /labs/ireas? [19:03:32] Tpt_: It looks like the webtools project currently is used by Platonides and Odie5533. So if you want to use that particular project you should coordinate with one of them. [19:04:13] andrewbogott: Thanks! [19:04:18] Tpt_: Either of them can provide access. Or if you want to start a new project that has nothing to do with their existing work I can create one for you. [19:04:43] PROBLEM dpkg-check is now: CRITICAL on kubo.pmtpa.wmflabs 10.4.0.19 output: DPKG CRITICAL dpkg reports broken packages [19:06:10] andrewbogott: It doesn't matter if the project is only for one instance that runs only one tool? [19:06:20] ireas: It's probably best if you use git repos that are topic-based rather than person-based… we always want things to make sense to people who come along after you (if, e.g. you are hit by a bus) [19:06:29] ireas: For examples, here's the page with all the existing repo names: https://gerrit.wikimedia.org/r/#/admin/projects/ [19:07:18] andrewbogott, okay, I understand this point. but especially the web tools are so small that splitting them up in different repositories does not make sense [19:07:32] Tpt_: That would be fine for testing and development. If it's a production or semi-production tool then… well, that's kind of an unanswered question, but it probably should not be in a project that nly one person knows about. [19:07:50] ireas: Well, having a 'webtools' repo isn't crazy… it's only having one named after you that I object to :) [19:08:41] andrewbogott: It's a semi-production tool. Seb35 will I hope maintain it when he will have more time. [19:09:09] andrewbogott, I am not the only one with web tools. but as you mention this, it would maybe be the best to have a project repositories for all webtools guys. [19:09:37] hi [19:09:40] hi itu ;) [19:09:46] ireas: I'm not the granter of gerrit repos anyway, so there is no authority behind my suggestions [19:10:05] the biggest issue with creating one big repo is that then we need to figure out how merge rights work [19:10:33] i think one repo per tool is better [19:10:37] agreed [19:11:02] giftpflanze, even if one tool is just a single file? [19:11:09] I wish ^demon was here [19:11:23] yes, sort of [19:11:40] Tpt_: OK, sounding like you do indeed want to scare up one of the webtools folks. [19:15:39] andrewbogott: Ok. So I'll ask them. Thanks. [19:18:23] the file is located at /etc/nginx/sites-available/proxy [19:22:22] PROBLEM Free ram is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Warning: 19% free memory [19:34:50] do i need ssh version >= 5.4 to log into the 'bastion' ? [19:36:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 8.95, 8.07, 6.46 [19:37:12] itu: I'm not sure. I just checked and both of my clients are newer than that. [19:37:55] because of https://labsconsole.wikimedia.org/wiki/Access#Using_ProxyCommand_ssh_option [19:38:51] itu: Oh, I don't do that anyway, I just log into bastion and then on from there. [19:39:14] ok [19:39:14] So, to reach bastion probably an older version is fine. [19:40:31] setting up agent forwarding every time is tedious, or is it? [19:41:21] I don't think so, but I'm used to it [19:41:26] proxycommand is likely easiest [19:41:30] and most secure [19:41:47] 12/27/2012 - 19:41:46 - Updating keys for mwang at /export/keys/mwang [19:41:51] I have things in my .bashrc so it just happens without me thinking about it [19:43:32] me too [19:43:45] except the part where I need to enter my password [19:44:36] What do y'all use as a pastebin for screenshots? [19:44:59] labs ;) [19:45:00] There are many options and I haven't bothered to consider the privacy/political/implications of any of them [19:45:05] andrewbogott: there's always Commons! jk. imgur? [19:45:41] imgur doesn't expire things does it? [19:45:41] bugzilla :) [19:46:03] chrismcmahon: I'm with you [19:46:21] bz [19:47:55] andrewbogott: picpaste.de ! [19:49:06] english: http://picpaste.com/ [19:50:14] OK, I'm sold -- I like their use of the words 'picturefile' and 'storetime' [19:50:26] * sumanah sighs with happiness re German [19:54:31] * Ryan_Lane wants the openid extension fixed so that we can use the paste app I tested out [19:54:43] RECOVERY dpkg-check is now: OK on kubo.pmtpa.wmflabs 10.4.0.19 output: All packages OK [19:54:58] Ryan_Lane: what bug is the blocker? I could try to scrounge up some help [19:55:22] there's a few [19:55:25] let me find them [19:55:46] https://bugzilla.wikimedia.org/show_bug.cgi?id=40067 [19:55:50] https://bugzilla.wikimedia.org/show_bug.cgi?id=40068 [19:56:48] maybe that's the only two [19:56:58] the first being a pretty big one [19:58:57] andrewbogott: so, automounts and gluster are working on kubo [19:59:06] but, for some reason some of the homedirs aren't working properly [20:00:00] I'm going to fix this stupid keystone issue today [20:00:21] maybe I'll fix the last "no nova credentials" issue too [20:02:25] hm. this gluster issue is specific to kubo, too [20:29:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 8.20, 7.32, 5.99 [20:30:18] Ryan_Lane: point me to the paste app thing? [20:30:47] Ryan_Lane: that way I can create the relevant chain of blockers/dependencies/etc., grab your partner, skip to my lou, etc. [20:30:56] all bugs, circle left! [20:31:02] * sumanah is suddenly square-dancing [20:31:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [20:34:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.41, 7.00, 5.58 [20:35:36] Ryan_Lane: mike_wang is still having craziness with his labsconsole login. [20:35:55] e.g. this happened immediately after he logged in: http://picpaste.com/2012-12-27-150530_1680x945_scrot-abMwOmSq.png [20:56:53] 12/27/2012 - 20:56:52 - Creating a home directory for reviewer-bot at /export/keys/reviewer-bot [21:00:17] Who can create projects? [21:01:12] techman224: AFAIK, s_umanah, R_yan_Lane, a_ndrewbogott, probably a handful of other people [21:01:29] i mangled the nicks so as to not page them unnecessarily [21:01:31] hi techman224 -- you mean, who can create Labs projects? [21:01:33] aw thanks ori-l [21:01:49] yes, https://labsconsole.wikimedia.org/wiki/New_Project_Request/Titleblacklist [21:01:52] 12/27/2012 - 21:01:51 - Updating keys for reviewer-bot at /export/keys/reviewer-bot [21:01:52] 12/27/2012 - 21:01:51 - Updating keys for mwang at /export/keys/mwang [21:02:07] Damianz: do you have the ability to create Labs projects? [21:02:15] Nope, sorry [21:02:45] ok, lemme see what I can do here [21:14:49] @labs-user Itu [21:14:49] Itu is member of 1 projects: Bastion, [21:15:18] when i do that in a query i get: wmib.User is member of 1 projects: Bastion, [21:15:46] ok, i'm in! [21:16:26] techman224: ok, sorry, this is going slow [21:16:36] 12/27/2012 - 21:16:35 - Creating a home directory for itu at /export/keys/itu [21:17:05] techman224: I have never created a Labs project before. You might want to email the labs-l list to get this done faster [21:17:46] sumanah: Do you have the ability to create projects? [21:17:56] so, to do extension work you have to have your own project? [21:18:51] techman224: I am not sure whether I do, and labsconsole documentation is a bit slow to read through [21:19:12] giftpflanze: you can either have your own project created or you could work within some existing project. https://labsconsole.wikimedia.org/wiki/Help:Terminology#Technical_terms may help [21:20:28] If you have the right, at the top of the manage projects page there should be a new option [21:20:29] so, that means, there is no generic one? [21:21:04] Nope! I do not see the "create a project" option at https://labsconsole.wikimedia.org/wiki/Special:NovaProject so I do not have the right, thanks Damianz [21:21:10] giftpflanze: just a moment - let us look together! [21:21:26] giftpflanze: so, on the left side of https://labsconsole.wikimedia.org you see a few links [21:21:35] giftpflanze: "list instances" and "list projects" are the ones to click on [21:22:05] 12/27/2012 - 21:22:04 - Updating keys for itu at /export/keys/itu [21:22:05] "list projects" gives you a directory of all the projects that already exist. [21:22:31] thx, i know that ;) [21:22:46] you could ask Daniel Renfro (drenfro) about the extensions-review project -- maybe that's good for a generic extensions dev space [21:23:49] giftpflanze: ooh, https://labsconsole.wikimedia.org/wiki/Nova_Resource:Mediawiki-dev might be good [21:23:52] "A project for housing MediaWiki instances and testing bug fixes." [21:25:00] and there's also "testing" [21:25:09] but overall, I think "Mediawiki-dev" is the one I would steer people towards. [21:26:20] hm [21:30:23] sumanah, I've sent a message to the labs list, but it's in the moderator's queue [21:31:10] techman224: :/ I am sure a moderator will release it soon. thanks for the note [21:31:28] Ryan_Lane: ^ release techman224's note from the queue? [21:31:41] ah [21:31:49] techman224: did you subscribe yet? [21:31:52] no [21:32:48] sumanah, techman224, sorry, I'm late to the party. Do you still need a new project? [21:34:16] yes [21:34:43] OK, I will have to DeCamelCase your project name [21:35:05] techman224, do you otherwise have your labs access sorted out? [21:35:07] 12/27/2012 - 21:35:06 - Creating a project directory for titleblacklist [21:35:44] what is the size of the project storage? [21:36:00] andrewbogott, yes, I got my ssh keys submitted and have shell access [21:37:00] techman224: OK, you should be all set then. [21:37:31] I see it [21:39:08] giftpflanze: All projects take slices from the same pie. So df shows the total available for all projects: projectstorage.pmtpa.wmnet:/mwreview-project 71T 8.7T 63T 13% /data/project [21:40:16] thx a lot [21:40:37] giftpflanze: Please don't use it all :) [21:40:43] heh [21:41:46] is that a real server? [21:42:26] (" up 9 days, 14:33," ,thats poor .... ) [21:42:29] * yuvipanda keeps reading giftpflanze as gzip-ftp-deflate [21:42:32] sorry! [21:42:46] itu: see topic [21:42:57] yuvipanda: ^^ [21:43:01] giftpflanze: It's a distributed cluster of four servers. [21:43:09] ok [21:43:59] should i change my name? [21:44:33] nah [21:46:53] projectstorage.pmtpa.wmnet:/bots-project 300G 142G 159G 48% /data/project; seems to differ [21:47:15] huh [21:47:22] * andrewbogott logs into bots to look [21:48:16] looks right to me [21:48:47] nah, not at all giftpflanze. [21:49:04] well… looks to me like on the server side each project is marked with a 50g quota. [21:49:26] So maybe df is unreliable in this case. [21:49:56] -home is 50G, -project is 300G [21:50:03] it would seem [21:50:19] Yep, you're totally right. [21:50:27] So, techman224, everything I said before was wrong. [21:50:59] andrewbogott, what exactly? [21:51:01] gluster itself reports the quotas as Daminaz said -- 50G for /home and 300G for /data/project, designated for each project. [21:55:49] andrewbogott, I can't create instances [21:56:13] techman224: Apologies, I confused two conversations... [21:56:54] techman224: Do you see the 'Add instance' link on this page? https://labsconsole.wikimedia.org/wiki/Special:NovaInstance [21:56:57] so, there are quotas for evey project, right? [21:57:11] giftpflanze: Yes, seems so. I was wrong before, df lied to me. [21:57:22] andrewbogott, nope, just toggle and show project filter [21:57:42] techman224: OK. I predict that if you log out of labsconsole and back in again that page will become more interesting. [21:58:02] it works [21:58:17] more buttons :D [21:58:48] andrewbogott: can you add me to webtools? [22:00:05] giftpflanze: I can, technically, but I would prefer an existing admin to coordinage… Platonides! you around? [22:00:16] *coordinate [22:00:35] the who-should-do-what's are confusing … [22:01:34] giftpflanze: In theory, Ryan and I are in charge of allocating resources to projects and Sumanah + person-to-be-named-later will be worrying about what happens inside those projects. [22:01:55] well, Platonides is one of the people running bots and/or webtools environments [22:01:56] ah, ok, then i should ask her instead [22:02:04] But, yes, we need to develop a policy about ownership of projects. At the moment we don't really know how to handle that and so I'm just trying to avoid toe-stepping. [22:02:15] ok [22:02:46] but in fact any project member can give out memberships as it seems [22:03:03] giftpflanze: The current sysadmins of webtools are Odie5533 and Platonides, so I take it to be their domain. [22:03:14] ok [22:13:34] Hey andrewbogott, if I want to create an instance with mediawiki installed, what rules should I have so I could access it? [22:13:56] I've created the web security group [22:13:57] 80 tcp any [22:14:33] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.39, 3.27, 4.48 [22:14:48] techman224: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups#Examples <- the first one there should do it. [22:19:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.85, 3.95, 4.84 [22:30:54] PROBLEM Current Load is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:30:54] PROBLEM Free ram is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:31:33] PROBLEM Current Users is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:32:13] PROBLEM Disk Space is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:32:23] PROBLEM Total processes is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:34:34] PROBLEM dpkg-check is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: Connection refused by host [22:41:38] andrewbognott, I can't ssh into the server [22:41:47] andrewbogott ^ [22:42:03] "Permission denied (publickey)." [22:42:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.26, 7.77, 5.97 [22:42:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.53, 7.78, 6.01 [22:42:59] techman224: OK -- it's possible your instance is still spinning up. [22:43:11] Are you able to log into other labs instances? [22:43:35] andrewbogott, I don't have any other instances [22:43:38] you spin me right round, baby, right round [22:43:43] you mean other projects? [22:44:07] techman224: Yes, either. [22:44:20] Just trying to determine if the problem is the instance or the ssh'ing [22:45:21] techman224: Are you forwarding your key when you ssh to bastion? [22:45:30] On another, I get Connection closed by UNKNOWN [22:45:37] heh. [22:45:45] OK, so you running windows, linux, mac? [22:45:53] andrewbogott, mac [22:46:12] I followed https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [22:46:12] And using "ssh -A bastion.whatever"? [22:47:24] techman224: I can't access that instance either, so I think it just isn't ready yet. [22:47:43] …unless you unchecked the 'default' security group checkbox. [22:47:45] what does the log say [22:48:03] it says active, and I put it into default and web [22:48:24] console log doesn't say active unless someone hacked the script [22:49:22] It still looks busy to me… give it a few more minutes. [22:49:33] RECOVERY dpkg-check is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: All packages OK [22:49:42] techman224: OK, try it now? [22:50:09] I'm in [22:50:24] OK. I didn't do anything other than help pass the time. [22:50:43] andrewbogott should teach line dancing lessons [22:50:53] RECOVERY Current Load is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: OK - load average: 0.22, 0.94, 0.91 [22:50:53] RECOVERY Free ram is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: OK: 678% free memory [22:51:12] * andrewbogott has a tragic allergy to belt buckles [22:51:26] how does a machine have 678% free memory :D [22:51:33] RECOVERY Current Users is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: USERS OK - 1 users currently logged in [22:51:40] lots of drugs where involved in that script [22:51:53] ^^ [22:52:13] I can't use sudo, I don't know the password [22:52:13] RECOVERY Disk Space is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: DISK OK [22:52:19] your password [22:52:20] derp [22:52:23] RECOVERY Total processes is now: OK on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: PROCS OK: 89 processes [22:52:27] it's your labsconsole password [22:53:05] Looks like I need to set sudo policies [22:53:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 7.51, 6.38, 5.41 [23:05:30] It's taking a while to complete sudo puppetd -tv [23:06:13] normal [23:07:47] I checked role::mediawiki-install::labs but srv/mediawiki is not there [23:08:06] Do I need to set the labs_mediawiki_hostname? [23:09:08] techman224: If you don't set it then things will still work, you'll just access your instance using an id rather than a name. [23:09:33] PROBLEM dpkg-check is now: CRITICAL on titleblacklist-main.pmtpa.wmflabs 10.4.1.58 output: DPKG CRITICAL dpkg reports broken packages [23:13:21] I'm getting an error when trying to install a dependency [23:13:32] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 4.54, 4.33, 4.99 [23:15:32] Hello ALL! Question: How can I setup mail forwarding i.e. to use something like "drtrigon+subster@bots.wmflabs.org" to recieve mail for my bot script? [23:15:53] techman224: Sometimes it takes a couple of puppet runs. If you get a repeatable failure then let me know. [23:16:21] not atm [23:16:23] It's fixed [23:16:27] that would be nice, but spam issues are issues [23:22:28] ping? [23:22:50] pong [23:23:12] (00:15:31) DrTrigon: Hello ALL! Question: How can I setup mail forwarding i.e. to use something like "drtrigon+subster@bots.wmflabs.org" to recieve mail for my bot script? [23:24:05] [23:16:21] not atm [23:24:10] replace not with can't as needed [23:24:45] aaa - that was an answer for me... ;)) [23:24:55] so when will this be available? [23:26:10] (it's important as data source for one of my bots...) [23:28:51] Damianz: when will this be available? [23:29:19] Right now, no idea :) [23:30:10] Damianz: Could we place it in the TODO list? ;) That would be cool! [23:31:51] *please* [23:32:51] it would be useful, I think it's been suggested before... just a time thing really oh and we need real mail relays to stop using prod [23:34:44] Damianz: Good thing to hear that is is some kind of planned... ;)) Thanks a lot!! [23:35:25] Damianz: Time for another question? Is there a backup for "/home" and "/data"? [23:36:48] Nope, bots takes backups of mysql dbs and stuff into /data/project, generally backups are do it yourself kinda thing [23:40:54] So I have to backup all home settings and data by myself to my own machine? No rollback or someting like this available? *hmmm* [23:41:19] i have put mail into the list a long time ago [23:41:33] It's stored redundantly on a off-server, server... but nope no rollback [23:41:49] Basically the idae is you version control code and that's your backup [23:43:01] are gerrit repos backed up? [23:43:31] yes, they're replicated and backed up [23:44:24] I do version control, but all the crontab, login data and lots of other config (most in home) are lost in case of server harddisk error or else? [23:48:58] Damianz: ping? [23:49:28] andrewbogott, when you configure role::mediawiki-install::labs, do you need to set labs_mediawiki_hostname? [23:52:22] techman224: It will work without it. If/when you set up a host for with a public IP you will want to set it. [23:52:39] andrewbogott, what would I put in? [23:53:46] probably titleblacklist-main.pmtpa.wmflabs [23:54:17] Because of firewalling, accessing the instance from a browser will involve some proxying, and how you access the instance affects the hostname. [23:56:30] Damianz: anyway... THANKS A LOT! Greetings! :) [23:57:40] well ideally youl'd use puppet which is in git [23:57:42] oh he left [23:58:24] Can you login to nagios?