[01:27:03] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [01:31:53] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 11.67, 10.61, 9.86 [02:46:20] RECOVERY Free ram is now: OK on robh1 robh1 output: OK: 86% free memory [02:47:20] RECOVERY Current Users is now: OK on publicdata-administration publicdata-administration output: USERS OK - 1 users currently logged in [02:47:50] RECOVERY Total Processes is now: OK on robh1 robh1 output: PROCS OK: 87 processes [02:47:55] RECOVERY Disk Space is now: OK on publicdata-administration publicdata-administration output: DISK OK [02:48:25] RECOVERY dpkg-check is now: OK on robh1 robh1 output: All packages OK [02:48:31] RECOVERY Free ram is now: OK on publicdata-administration publicdata-administration output: OK: 86% free memory [02:49:15] RECOVERY Current Load is now: OK on robh1 robh1 output: OK - load average: 0.00, 0.01, 0.00 [02:49:50] RECOVERY Current Users is now: OK on robh1 robh1 output: USERS OK - 0 users currently logged in [02:49:50] RECOVERY Total Processes is now: OK on publicdata-administration publicdata-administration output: PROCS OK: 85 processes [02:50:30] RECOVERY Disk Space is now: OK on robh1 robh1 output: DISK OK [02:50:40] RECOVERY dpkg-check is now: OK on publicdata-administration publicdata-administration output: All packages OK [02:51:30] RECOVERY Current Load is now: OK on publicdata-administration publicdata-administration output: OK - load average: 0.01, 0.12, 0.07 [04:07:00] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:00] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 10.01, 9.85, 9.90 [04:17:00] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:03] Is there any form of feed for all wikis in the beta cluster? [04:33:07] * HSJ doubts it [04:33:58] The logbot logs quickly! :O [05:56:50] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 10.62, 9.99, 9.98 [06:27:47] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [06:32:46] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 11.07, 12.17, 11.22 [06:44:47] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 7.61, 7.47, 5.81 [06:54:44] PROBLEM Disk Space is now: WARNING on aggregator1 aggregator1 output: DISK WARNING - free space: / 330 MB (3% inode=94%): [06:59:44] PROBLEM Disk Space is now: CRITICAL on aggregator1 aggregator1 output: DISK CRITICAL - free space: / 279 MB (2% inode=94%): [07:17:54] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:44] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 2.25, 3.73, 4.90 [08:57:44] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 11.24, 10.92, 10.30 [09:01:44] hi Ryan_Lane [09:02:18] I think it would be best if we had a short meeting on irc to talk about this oauth stuff with all who wants it done [09:03:37] it needs to be somehow organised I guess [09:07:08] could we just get a new project called oauth on labs? [09:07:37] we could set up a wiki there and do development there [10:13:50] @RC+ mediawiki OAuth* [10:13:50] Inserted new item to feed of changes [10:34:33] Change on 12mediawiki a page OAuth was modified, changed by 90.183.23.27 link https://www.mediawiki.org/w/index.php?diff=510710 edit summary: /* Relevant bugs */ ++ [10:35:14] Change on 12mediawiki a page OAuth was modified, changed by 90.183.23.27 link https://www.mediawiki.org/w/index.php?diff=510711 edit summary: /* Relevant bugs */ fix [13:02:13] !log nginx Doing a differnt setup, where it will act more like a real Client -> Server -> Server setup and do proper recorded testing of nginx and varnish [13:02:14] Logged the message, Master [13:07:54] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:54] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 9.78, 9.64, 9.67 [13:22:54] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [14:22:58] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 11% free memory [14:23:58] PROBLEM host: robh1 is DOWN address: robh1 CRITICAL - Host Unreachable (robh1) [14:29:08] RECOVERY host: robh1 is UP address: robh1 PING OK - Packet loss = 0%, RTA = 0.72 ms [14:32:08] PROBLEM Current Load is now: CRITICAL on robh1 robh1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:32:48] PROBLEM Current Users is now: CRITICAL on robh1 robh1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:37:09] !log deployment-prep petrb: switching en.wikipedia to older previous v [14:37:11] Logged the message, Master [14:39:12] PROBLEM host: robh1 is DOWN address: robh1 CRITICAL - Host Unreachable (robh1) [15:02:46] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 10.51, 10.25, 9.96 [15:04:26] PROBLEM Current Load is now: CRITICAL on robh2 robh2 output: Connection refused by host [15:05:06] PROBLEM Current Users is now: CRITICAL on robh2 robh2 output: Connection refused by host [15:05:46] PROBLEM Disk Space is now: CRITICAL on robh2 robh2 output: Connection refused by host [15:13:56] PROBLEM Total Processes is now: CRITICAL on robh2 robh2 output: Connection refused by host [15:14:36] PROBLEM Free ram is now: CRITICAL on robh2 robh2 output: Connection refused by host [15:14:46] PROBLEM dpkg-check is now: CRITICAL on robh2 robh2 output: Connection refused by host [16:38:56] RECOVERY Total Processes is now: OK on robh2 robh2 output: PROCS OK: 89 processes [16:39:26] RECOVERY Current Load is now: OK on robh2 robh2 output: OK - load average: 0.19, 0.27, 0.21 [16:39:36] RECOVERY Free ram is now: OK on robh2 robh2 output: OK: 89% free memory [16:39:46] RECOVERY dpkg-check is now: OK on robh2 robh2 output: All packages OK [16:40:06] RECOVERY Current Users is now: OK on robh2 robh2 output: USERS OK - 1 users currently logged in [16:40:46] RECOVERY Disk Space is now: OK on robh2 robh2 output: DISK OK [17:13:05] 03/14/2012 - 17:13:05 - Creating a home directory for jgreen at /export/home/pediapress/jgreen [17:13:06] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [17:14:06] 03/14/2012 - 17:14:05 - Updating keys for jgreen [17:16:11] Preference-->Gadget not shown at bn.wikipedia.beta.wmflabs.org. [17:17:56] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 10.38, 9.92, 9.81 [19:40:46] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:41:06] PROBLEM Disk Space is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:41:06] PROBLEM Total Processes is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:41:11] PROBLEM Current Users is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:41:11] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:41:26] PROBLEM dpkg-check is now: CRITICAL on deployment-web deployment-web output: CHECK_NRPE: Socket timeout after 10 seconds. [19:46:06] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 78% free memory [19:46:06] RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 0 users currently logged in [19:48:46] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 7.04, 7.14, 5.45 [19:50:43] PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 0.18, 12.79, 14.48 [19:50:53] RECOVERY Disk Space is now: OK on deployment-web deployment-web output: DISK OK [19:50:53] RECOVERY Total Processes is now: OK on deployment-web deployment-web output: PROCS OK: 93 processes [19:51:13] RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK [19:53:43] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 1.72, 3.78, 4.48 [19:54:03] PROBLEM Disk Space is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:54:43] PROBLEM Free ram is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:56:13] PROBLEM Total Processes is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:56:53] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:57:33] PROBLEM Current Load is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:58:23] PROBLEM Current Users is now: CRITICAL on pediapress-ocg2 pediapress-ocg2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:10:43] RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.02, 0.25, 4.00 [20:21:53] RECOVERY dpkg-check is now: OK on pediapress-ocg2 pediapress-ocg2 output: All packages OK [20:22:33] RECOVERY Current Load is now: OK on pediapress-ocg2 pediapress-ocg2 output: OK - load average: 0.09, 0.11, 0.20 [20:23:23] RECOVERY Current Users is now: OK on pediapress-ocg2 pediapress-ocg2 output: USERS OK - 1 users currently logged in [20:24:03] RECOVERY Disk Space is now: OK on pediapress-ocg2 pediapress-ocg2 output: DISK OK [20:24:43] RECOVERY Free ram is now: OK on pediapress-ocg2 pediapress-ocg2 output: OK: 86% free memory [20:26:13] RECOVERY Total Processes is now: OK on pediapress-ocg2 pediapress-ocg2 output: PROCS OK: 83 processes [20:35:23] Ryan_Lane: Shall I write to Stefano and beg an invite for Faidon? Or do you already have something in progress? [20:35:37] hm. I'll do so :) [20:36:05] ok [20:37:33] maplebed: is it true you're doing a full deployment of Swift for http://commons.wikimedia.beta.wmflabs.org? [20:38:09] I am building swift in labs. who uses it is up to the project, I suppose. [20:38:12] beta would definitely be a good place to have it. may be good to share it from a swift project, to beta [20:38:18] but yeah, eventually. [20:38:43] and "full deployment" is something of a misnomer... it'll have the bare minimum configuration necessary for swift to function correctly. [20:39:00] (1 proxy and 3 storage instances) [20:39:17] Ryan_Lane: The next thing on my list is to work towards the Essex migration. Would it be useful for me to look into making puppet classes for essex? (<- warning, if you say yes, that will commit you to an epoch of spoonfeeding in order to get me up to speed.) [20:39:27] heh [20:39:36] sure, that sounds like a good idea [20:39:46] I wonder if we should have versioned puppet config [20:40:12] if $release == "essex" { use these things } else { use these things } [20:40:46] I was imagining that I'd just duplicate the existing 'openstack' dir in the puppet rep and treat it like a different project. [20:41:01] But if diablo and essex are mostly overlapped then that's silly. [20:41:37] I think a lot will overlap [20:41:42] maplebed: could you drop me a line when that happens? [20:42:02] ugh. someone else is also writing an API for the sharedfs stuff? [20:42:21] I want to stab when people don't publish the stuff they are working on [20:42:24] I don't think they've started yet -- I read that as describing an intent. [20:42:28] ahhh ok [20:42:33] chrismcmahon: umm... sure. so long as I can figure out how to trigger a reminder on completion... [20:42:35] :P [20:42:46] And, that's gluster people! So getting them involved is good, since Gluster could use a few features in support of automation. [20:42:52] oh, sweet [20:43:09] I thought this was going to be the DNS support all over again :D [20:43:31] That had crossed my mind... but we'll see how they respond to my attempt at delegating. [20:43:38] * Ryan_Lane nods [20:44:51] Oh hey labs people [20:44:59] One of the OpenStack people succeeded in skinning Gerrit [20:45:15] Is there a wiki page with advice about puppet dev processes? (I presume that during development you use local files rather than involving puppet master in the test cycle...) [20:45:16] I'll be porting their code over to our install, hopefully today [20:45:47] RoanKattouw: does that mean that the openstack gerrit is already skinned? (It's the only gerrit I'm used to looking at.) [20:46:02] No, but it's going to be [20:46:15] http://review-dev.openstack.org/ [20:47:10] The colors! The... inoffensive background! [20:47:25] I know right [20:47:34] Plus, we can now inject JS that's written in actual JavaScript, as opposed to Java [20:51:37] Ryan_Lane: is it possible to have a wiki for testing bots? Something like http://commons.wikimedia.beta.wmflabs.org, but a more clean mw instalation [20:51:53] a wiki for testing bots? [20:52:00] I'm not sure if I know what you mean [20:52:03] og [20:52:06] *oh [20:52:11] a wiki for bots to test against [20:52:13] why not beta? [20:53:24] andrewbogott: unfortunately everything uses the test branch in labs right now [20:53:30] we don't have per-project branches yet [20:53:43] I was going to do that after doing this gluster stuff [20:53:46] But I can still do a test run with local files, right? [20:53:56] not easily :( [20:54:07] beta? [20:54:37] you just linked to it [20:54:55] Ryan_Lane: Do testlabs machines use the test branch by default? [20:55:00] yep [20:55:02] all instances do [20:55:04] Alchimista: that's the copy of production [20:55:13] in every project [20:55:32] an example: i'm developing an anti-vandalism bot, and i can't make a complete debug, if i wanna test how the bot behaves with the expressions list, i can't, because i get caught on abusefilter, or by other antvandalism bot [20:56:27] disable the abuse filter Alchimista :) [20:56:41] I can give you admin rights wherever you need it [20:57:05] ok, so my dev cycle will be... code, commit, push, test, repeat? [20:57:20] Is anyone else depending on the openstack classes not being broken in the meantime? [20:57:43] PROBLEM host: driver-dev-jumbo is DOWN address: driver-dev-jumbo CRITICAL - Host Unreachable (driver-dev-jumbo) [20:57:43] andrewbogott: yep [20:57:47] and yep [20:57:48] :) [20:57:52] well, not really [20:57:54] * andrewbogott is amused by having 'push' come before 'test' in that list. [20:57:57] I use them on nova-production1 [20:58:02] agreed, I don't like that [20:58:08] I'm working on per-project branches [20:58:14] which would let us test before push [20:58:21] Well, I really can just duplicate the directory for the time being, and then overwrite once things stabilize. [20:58:47] well, it's not a directory, but a manifest, but yeah, adding an openstack-testing.pp would work [20:58:55] you'll need to rename all of the classes, though too [20:59:01] and you'd need to handle the templates too [20:59:18] I'd say let's version it by release [20:59:33] then we could also use that inside of the templates too [20:59:50] nova_release is likely a good variable name [21:00:02] How much of a headache will it be for you if I, for example, break the openstack install and then don't fix it until the next morning? [21:00:04] Thehelpfulone: disable on pt-wiki? :P on test it's also hard, there are a lot of filters [21:00:12] not much of a problem [21:00:30] on nova-production1 I'm writing openstackmanager code [21:00:31] ok. I'll just work w/out a net. Or, rather, rely on git for my net. [21:00:40] so as long as nova continues working on my instance it's no problem [21:00:49] how many filters are there Alchimista? [21:01:24] did I come on the right day? [21:01:29] !accountreq [21:01:29] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [21:01:38] lets see who is on [21:01:52] * Ryan_Lane hides [21:01:55] Quick everyone hide. [21:01:55] Ryan_Lane: is about :D [21:02:04] lol [21:02:07] btw, Ryan_Lane, am I the only one who used nova-dev2 and nova-dev3? I think I can clean them up if so. [21:02:09] DeltaQuad: what were you looking to work on? [21:02:16] andrewbogott: I believe so, yes [21:02:20] ok [21:02:26] bots, wiki bots [21:02:34] ah [21:02:39] !account-questions | DeltaQuad [21:02:40] DeltaQuad: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [21:02:44] I should go setup my devstack box again, damn thing borked. [21:03:52] Ryan_Lane: I wouldn't trust DeltaQuad if I were you, he really wants to run spamming bots... ;) [21:04:00] heh [21:04:05] oh really? [21:04:15] * DeltaQuad trouts Thehelpfulone [21:04:20] DeltaQuad: you have a −1 on your request. have anyone to vouch for you? :) [21:04:36] * DeltaQuad calls Snowolf :P [21:04:38] ya, the toolserver must have rules against it, but labs not so much yet [21:04:47] Ryan_Lane: And... should I still be targeting Lucid? [21:04:52] we have a terms of use now :) [21:04:58] andrewbogott: precise, likely [21:05:01] oh? [21:05:03] we'll need to add that image [21:05:15] Ryan has a secret legal fetish. [21:05:16] Thehelpfulone: 105 abuse filters!! [21:05:16] it's a draft: http://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use [21:05:32] Ryan_Lane: ok [21:05:41] Alchimista: hmm, so you want to create your own instance? [21:06:08] Ryan_Lane: in all seriousness DeltaQuad should be a +2, one from me and one for Thehelpfulbot [21:06:11] heh [21:06:18] I had a feeling you were kidding ;) [21:06:31] Thehelpfulone was being ironically un-helpful. [21:06:40] * Damianz goes back to catching up on emails [21:06:45] I know right? :O [21:06:49] a own or simply a shared instance [21:07:00] Ryan_Lane: DeltaQuad = username across the board, and admin@deltaquad.info [21:07:49] PROBLEM host: nova-dev2 is DOWN address: nova-dev2 CRITICAL - Host Unreachable (nova-dev2) [21:08:08] DeltaQuad: ok, your shellaccount name will be all lowercase [21:08:15] you don't already have svn access, right? [21:08:29] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [21:08:33] figured as much. and SVN access? don't think so [21:08:43] ok [21:10:14] DeltaQuad: I've emailed you a password to your account [21:10:24] !initial-login | DeltaQuad [21:10:24] DeltaQuad: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [21:10:33] kthnx /me looks [21:10:35] ignore the part of those docs that mention the password reset [21:10:43] heh [21:10:44] since you already have a temporary password in the mail [21:13:25] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 10.08, 10.35, 10.14 [21:14:15] PROBLEM Current Load is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:14:55] PROBLEM Current Users is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:15:35] PROBLEM Disk Space is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:16:25] PROBLEM Free ram is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:16:45] petan, petan|wk poke [21:17:45] PROBLEM Total Processes is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:18:35] PROBLEM dpkg-check is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:23:31] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [21:26:15] Ryan_Lane: are you able to ssh to essex-test-lucid? [21:27:03] lemme see [21:27:15] nope [21:27:53] did it build properly [21:27:55] ? [21:28:06] I never know what to look for in the log file. The output looks ok to me; ends with a login prompt. [21:28:16] weird [21:28:26] it didn't even try to install puppet or anything like that [21:28:29] That's the only place to look, right? [21:28:54] i'd say try delete/recreate [21:29:00] ok, that's easy. [21:35:51] I have no clue what went wrong with that one [21:36:05] unless cloud-init is borked, which would be crappy [21:43:30] PROBLEM Free ram is now: CRITICAL on mobile-enwp mobile-enwp output: Critical: 2% free memory [21:45:30] PROBLEM host: essex-test-lucid is DOWN address: essex-test-lucid CRITICAL - Host Unreachable (essex-test-lucid) [21:52:56] any one an admin on en wikipedia ? [21:54:16] bad channel to ask [21:54:24] try #wikipedia [21:54:27] guess so [21:54:37] I've asked there too [21:55:50] RECOVERY host: essex-test-lucid is UP address: essex-test-lucid PING OK - Packet loss = 0%, RTA = 0.48 ms [21:56:02] Ryan_Lane: I got a long week end coming up - will you be around ? [21:56:10] ish [21:56:15] about the normal amount I am [21:56:20] I prefer to not work on the weekends, though [21:56:37] and you already have a project and such, so ideally shouldn't need me :) [21:56:59] ;-) [21:57:19] I plan to do some dev stuff and might want to ask some how to do some new things [21:58:04] I'll be trying to install that maven repository as well [21:58:22] in labs [21:58:46] it's all just regular sysadmin stuff [21:58:51] you said there you could be a permanet ip/port [21:58:58] if you need help learning puppet, I can help with that [21:59:00] PROBLEM Current Load is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:59:04] does your project not already have an IP? [21:59:24] hm, probably doesn't [21:59:29] nope [21:59:33] you can test it without a public IP, though [21:59:35] using a socks proxy [21:59:40] PROBLEM Current Users is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [21:59:47] when you are done testing, and it's ready for use, we can give it a public IP [22:00:00] that complicates things [22:00:04] why? [22:00:20] PROBLEM Disk Space is now: CRITICAL on essex-test-lucid essex-test-lucid output: Connection refused by host [22:02:06] I have not got the socks proxy ffox extension to work [22:02:18] you don't need one [22:02:23] !socks-proxy [22:02:23] ssh @bastion.wmflabs.org -D ; # [22:02:36] ssh creates a socks proxy [22:02:45] then you point your browser at it [22:02:56] when you are done testing, you stop using the proxy [22:03:00] ok I did that once or twice [22:03:17] will it work for non http too [22:03:26] port redirection will [22:03:57] ok [22:03:59] @search port [22:03:59] Results (found 3): ryanland, load, socks-proxy, [22:04:11] !port-forwarding is https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_services_using_port_forwarding [22:04:11] Key was added! [22:04:12] PROBLEM Current Load is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:04:52] PROBLEM Current Users is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:05:32] PROBLEM Disk Space is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:05:34] Ryan_Lane: are yo going to be at berlin? [22:05:44] I think so [22:05:45] lemme ask [22:06:22] PROBLEM Free ram is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:07:42] PROBLEM Total Processes is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:08:03] yes, I'll be there [22:08:32] PROBLEM dpkg-check is now: CRITICAL on essex-test-l essex-test-l output: Connection refused by host [22:08:58] great [22:09:16] any chance of you doing a pupet for dummies tutorial ? [22:09:31] in context of labs [22:12:53] either I or another ops team member likely can [22:23:24] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 9.89, 10.61, 10.52 [22:32:44] RECOVERY Total Processes is now: OK on essex-test-l essex-test-l output: PROCS OK: 104 processes [22:34:14] RECOVERY Current Load is now: OK on essex-test-l essex-test-l output: OK - load average: 2.44, 2.54, 2.27 [22:34:54] RECOVERY Current Users is now: OK on essex-test-l essex-test-l output: USERS OK - 1 users currently logged in [22:35:34] RECOVERY Disk Space is now: OK on essex-test-l essex-test-l output: DISK OK [22:36:14] RECOVERY Free ram is now: OK on essex-test-l essex-test-l output: OK: 78% free memory [22:45:03] I use nginx for static cache, and it seems that mediawiki is setting session cookies on first page load, wanted to know if there was a option about this anywhere [22:48:24] RECOVERY dpkg-check is now: OK on essex-test-l essex-test-l output: All packages OK [23:11:21] hmm labs is running quite slowly Ryan_Lane [23:11:38] I'm talking about the deployment-prep instance [23:25:13] 03/14/2012 - 23:25:12 - Creating a home directory for deltaquad at /export/home/bastion/deltaquad [23:26:12] 03/14/2012 - 23:26:12 - Updating keys for deltaquad [23:27:13] 03/14/2012 - 23:27:13 - Updating keys for deltaquad [23:38:54] PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CRITICAL - load average: 27.46, 39.29, 18.99 [23:43:34] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [23:43:54] Thehelpfulone: they need more squids [23:43:54] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.88, 14.76, 13.89 [23:45:27] how do they get more squids Ryan_Lane? [23:45:38] create more instances, and load balance to them somehow [23:46:20] ok [23:46:36] and could you add DeltaQuad to the bot group if you're a sysadmin for it? [23:46:52] I'd prefer someone in the bots project do so [23:47:13] and yes, I'm in the project, but just because I need to occasionally help out with some things [23:47:24] ok [23:47:43] then Damianz is who I need ;) [23:47:48] or petan [23:48:24] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 9.57, 9.92, 9.91 [23:49:09] yep [23:50:58] Damianz is watching tv [23:56:07] * RoanKattouw throws things at Gerrit [23:56:12] Good idea [23:58:26] Ooooh [23:58:27] OK [23:58:33] * RoanKattouw throws stuff at gitweb instead [23:58:51] * Damianz throws Gerrit at sumanah [23:58:55] Noooo [23:59:09] it'll stick to me and then I'll be all glaring-at-you and unintuitive [23:59:11] ERROR [23:59:14] SUMANA APPLICATION ERROR [23:59:26] Don't worry, Ryan has you all replicated :D [23:59:29] HA [23:59:37] nooooo merge conflict