[00:06:48] (Yay, I can log into the instance. ^^) [00:07:25] And the day is saved by puppet. [00:10:13] Hm. cvresearch-bots has the security group default, yet cvresearch-web can't connect to MySQL on it. [00:10:31] I know there's some issue with security groups, but I thought it was only for rules you add later. [00:11:00] Oh. Puppet keeps changing the bind address back. [00:11:02] * AMadman fiddles. [00:28:07] !log cvresearch Set up restart script for csbot [01:24:02] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [01:52:03] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [02:39:52] RECOVERY Free ram is now: OK on swift-be1.pmtpa.wmflabs 10.4.0.107 output: OK: 20% free memory [02:42:03] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 23% free memory [02:47:52] PROBLEM Free ram is now: WARNING on swift-be1.pmtpa.wmflabs 10.4.0.107 output: Warning: 19% free memory [03:21:22] Hm. Is the bot for !log not here? [06:02:49] * Beetstra looks around for petan, Ryan_Lane or someone else to solve the bots-3 problem ... why does it go down so easy now - conflict with bots? Could we have another instance for some bots? [06:15:02] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [06:21:13] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:21:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:21:24] PROBLEM Current Users is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:22:33] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 35% free memory [06:25:54] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [06:26:13] RECOVERY dpkg-check is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: All packages OK [06:26:23] RECOVERY Current Users is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: USERS OK - 0 users currently logged in [06:26:24] RECOVERY Total processes is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: PROCS OK: 128 processes [06:28:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 155 processes [06:31:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:36:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 147 processes [06:43:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [06:53:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [07:24:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [07:35:02] PROBLEM dpkg-check is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:34] PROBLEM Disk Space is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:34] PROBLEM Free ram is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:36:33] PROBLEM SSH is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CRITICAL - Socket timeout after 10 seconds [07:38:03] PROBLEM Current Users is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:43] PROBLEM Current Load is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:53:53] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [09:16:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [11:21:52] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [11:54:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [12:29:28] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=622557 edit summary: /* Labs wide (not only bots / tools), but available for all projects */ + Stable cron / + Server statistics, workload, status and else [12:36:15] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Giftpflanze link https://www.mediawiki.org/w/index.php?diff=622565 edit summary: +see also [13:27:58] If I wasn't so tired I'd piss some peopleoff [13:28:18] too late :-p [13:28:35] Damianz: ? [13:28:55] :P [13:29:16] like 3 different pages of features wanted/required/omg must exist forever/totally depend on our lives [13:29:24] ^^ [13:29:28] slightly amusing to go slap so people around with fish and watch them rage type a reply [13:30:38] * Beetstra sees fire, grabs oil: [13:31:02] oil is actually good at putting out fire depending what oil and fire it is [13:31:28] Damianz .. why don't you go reboot bots-3, and figure out why suddenly that instance is so sensitive [13:31:54] * Beetstra runs [13:33:36] because it's full of crap, with little specs, slow disk and ooms a lot thanks to perl/python [13:34:10] gonna eat lunch first.... after my puppet shizzle gets merged I plan to migrate most the bots stuff to new servers as it's grown organically rather than managed currently [13:35:13] * Beetstra agrees somewhat with the latter ... [14:34:43] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 42% free memory [15:07:23] happy new year [16:24:53] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [16:32:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [18:11:22] franny: Are you still fighting for access, or are things finally working? [18:16:12] 12/31/2012 - 18:16:11 - Updating keys for mwang at /export/keys/mwang [18:20:31] mwang: Are you still having access problems? [18:21:03] yes. The same problem happened just now. [18:21:16] hmm why do I reconigze mwang's nick [18:21:41] mwang: So the issue is that you can only log into labsconsole once and then after that you have to do a password reset? [18:21:56] I am using my home's computer. and I logged in as mwang [18:22:11] yes. [18:22:17] Damianz: mwang is a WMF contractor living in tampa; he'll be handling account creation and shell requests and such. [18:22:30] mwang: Any idea why your ssh keys were just now updated? Did you do anything with them? [18:22:36] oh, he's the new guy [18:22:43] swear he had a different nick [18:22:58] I also got this error since yesterday. "No Nova credentials found for your account." [18:23:47] have you re-logged in since yesterday? [18:23:52] ryan sorta broke it [18:23:55] then maybe fixed it [18:24:27] yes. I logged in 10 mins ago. every time when I log in, I need reset my password. [18:25:23] I did not do anything about my ssh key. [18:25:56] mwang: Are you using a different password each time you do this? Have you tried using an extra-simple nothing-but-lowercase-letters password? [18:26:17] (I'm wondering if there's a bottleneck someplace that is truncating your password or stripping a special character or something) [18:27:01] I use UPPER case letter and numbers. [18:27:25] Hm, that seems simple enough :( [18:27:45] yes I think it is simple enough [18:28:33] If you visit this page is there a 'disable two-factor authentication' link? https://labsconsole.wikimedia.org/wiki/Special:OATH [18:30:15] yes. I found disable two-factor authentication' link [18:31:00] Click it! [18:31:27] I need type in the token [18:31:32] afaik you need 2fa to disable 2fa [18:32:06] mwang: When you've been trying to log in (and failing) were you entering anything in the 'token' field, or leaving it empty? [18:32:27] leaving it empty [18:33:24] ok. So I think what's happening is that you have 2-factor-auth turned on, which means to log in you need to know a magic token. But since you've been leaving it blank it won't let you in /but/ when you do a password reset that follows a different code path that (probably mistakenly) does not require the token. [18:33:32] Does that seem possible/consistent with what you're experiencing? [18:34:42] you remember how cloudflare got 'hacked' via the ceo's gmail? [18:34:45] that was pretty much how :P [18:35:28] Damianz: Well, since a password reset requires email access that is legitimately a 2nd factor… just a different one. [18:35:44] well since it only requires an email, it's sorta 1fa [18:35:58] That's a very good point :) [18:36:30] If the theory is correct I could reset your pass and if I had access to your email I'd have admin access to labs [18:37:00] which btw - are you still setting up test wiki? I note the *cough* wrong *cough* password is in the configs atm [18:38:13] Damianz: By 'test wiki' do you mean nova-precise2? [18:38:24] yeah [18:41:05] mwang, still there? [18:42:22] yeah. I do have tokens created before. Let me find it. [18:43:05] mwang: Once that is turned on you have to use a time-generated token every single time you log in. [18:43:29] Since you a) don't have a smartphone and b) labs is broken in a way that requires daily logins, probably best if you just disable two-factor for now. [18:43:36] Which you still need a token for, but only once :) [18:44:32] Let me try to disable it. [18:45:33] Damianz, are you working on salt-api & keystone? [18:45:35] well... it depends what you need to do, since you can't do 'admin' stuff without 2fa... hot sure about shell [18:45:52] andrewbogott: ish, I have a sorta module that's a bit sucky but works for auth... not touched -api yet really [18:46:11] thinking about writing a class that shells out to salt-call for now so we can close buts then replace it with the real api later [18:47:09] Damianz: If you have anything you want to hand off to me I'm happy to work on it. I've had salt/keystone on my to-do list for a long time but haven't started because I wasn't sure where to start. [18:47:43] mwang: OK, wait, if Damianz is right then disabling 2fa isn't a good solution… you'll just have to live with getting your token however it is you can get it. [18:47:47] https://github.com/saltstack/salt/pull/3079#issuecomment-11781628 just got merged [18:47:52] which provides /basic/ auth [18:48:11] api side needs work and a class adding to osm - planning to hack on some tonight unless I get distracted [18:48:26] cool [18:48:31] actually [18:48:36] shellright is a group [18:48:38] so won't be 2fa [18:48:56] that's all a function call for osm stuff hmm but project adding probably is under 2fa [18:49:10] Damainz: But mwang needs to be able to handle shell requests, create projects, etc. So. [18:49:14] also the token is apaprently fixed with the last hook... but the wiki broken when Ryan pushed it so I dunno what the status is [18:49:33] andrewbogott: I used all my tokens, it do not work. [18:49:51] just kill the value in the db [18:49:53] then re-enable 2fa [18:49:55] *shrug* [18:50:08] it's the stupid auth core stuff that's missing like 23875923875982375 features [18:50:11] yay opensource [18:50:54] I hope Ryan puppetized the saltmaster install with all the controller stuff... or seperatly [18:51:04] would love to install from source and actually test this for real [18:51:29] andrewbogott: Btw, do you know if it's easy to delete a ldap user if they've never logged into anything other tha labsconsole/nova? [18:51:46] OK… well, /I/ can log in with 2fa so it's at least possible [18:51:47] might just leave my test account lying around... probably should change the password from 'test' though -.- [18:51:56] Damianz: I suspect it's easy but I haven't done it. [18:52:25] cool, I'll just keep it and change the pass... will be useful to have a basic account anyway [18:52:27] mwang: Can you tell me what process you're using to get your tokens? [18:52:52] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [18:53:19] I got the token when Ryan cerated account for me. I forgot the process. [18:54:01] i need leave for a while [18:54:08] mwang: OK! So, damianz will correct me when I get the details wrong, but here is how I think this works: [18:54:28] Somewhere (locked in your google account) is a giant million-digit key which you will never see. [18:54:54] When you need to log in, google (or, in my case, my phone) provides me with an 8-digit code which is a hash of that million-digit key and the current time. [18:55:03] Um… sorry, six-digit. [18:55:10] So, each specific token is good for only 30 seconds. [18:55:20] Anytime you want to log in you need a /new/ code that is appropriate for the current time. [18:55:29] 30 seconds? that is short! [18:55:33] If you don't have a smart-phone, you can ask google to send you a fresh token via sms [18:55:45] Maybe it's 60? It's pretty short, in any case. [18:55:51] (or even if you do have a smartphone) [18:56:08] mwang: So you are using a token that has long-since expired. You need to pry a new one out of google. [18:56:11] mwang: Details are here: http://support.google.com/accounts/bin/answer.py?hl=en&answer=185839&rd=1 [18:58:07] sumanah: In the android app each token shows a little countdown pie next to it; it's very exciting! [18:58:21] wow! [18:58:27] * sumanah feels all spy-like [18:59:01] It's surprisingly similar to that radioactive tattoo that Russell Crow had in A Beautiful Mind [18:59:45] * sumanah did not see that yet [19:02:26] pretty much [19:02:38] though you should have 6/8 backup codes that are valid for single use [19:04:17] * Damianz finds some trolling media [19:04:27] http://yle.fi/uutiset/onnistunut_ilotulituskuva_ikuistetaan_tukevilta_jaloilta/6433468 [19:04:39] ups, wrong channel [19:05:16] Hmm looks like Finnish [19:05:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [19:21:08] I must be bored, just spent an hour updating linkedin with crap [19:22:10] there is a security section on Help:Move_your_bot_to_Labs; i have a script that sends emails via mw api; i don't get how to secure it [19:22:45] or better: where the risk it [19:22:50] *is [19:23:28] define secure it? [19:24:03] "We need to ensure the environment remains secure and not to spam others." I have no idea :) [19:25:03] Anyone here experienced with LDAP? Wanted to have a go at changing some things in Gerrit but I can't do anything until I get LDAP working... [19:25:19] I figured someone here might know [19:28:40] giftpflanze: Use a nr instance, chmod your files to no world read access [19:29:02] Krenair: You'll need to use the proxy agent details to bind I think, Chad would be the best person to ask [19:29:47] ... Was that to the wrong person? [19:29:56] <^demon> Indeed, Gerrit tries to bind as the proxy agent if you provide it, or falls back to binding as the provided credentials. [19:30:03] <^demon> It can't do anonymous binding. [19:30:06] I'm not actually in Labs, but I thought people here might know the answer [19:30:12] but ~ and /data/project is shared and accessible in root-envs? [19:30:28] giftpflanze: yes [19:30:43] I could totally do su giftpflanze - [19:31:24] <^demon> Krenair: Also, easier than using full-blown LDAP, check out the contrib/ directory. A user just recently wrote a lightweight ldap server that does the bare minimum for debugging gerrit :) [19:31:52] so i use /mnt/share? [19:32:52] /mnt/share is a local dir, probably a bad idea to use that [19:32:53] :( [19:32:54] HEY [19:32:59] SOMEONE REVIEW MY MAILING LIST POST [19:33:06] It was going to be amusing, now it's lame [19:33:07] andrewbogott: now I set up my 2 steps verification for my gmail account mwang@wikimedia.org. [19:33:09] stupid mailman [19:33:18] so, what do i use then? [19:33:36] andrewbogott: how do I get the token? [19:34:11] mwang: I think the page I linked you to before has instructions for getting a token via sms [19:34:27] (Is that what you meant?) [19:36:05] No. that page is for 2 steps verification for google account [19:36:43] 2fa for labs is managed via google. But let me read a little more and see what I can learn. [19:39:26] hm… that thing I said about it being managed via google is maybe not true. [19:41:53] it's generic, google is a common provider [19:43:20] http://en.wikipedia.org/wiki/HOTP#Tokens [19:43:30] Damainz: Yeah, I'm muddling the two together in my head. Maybe 2fa on labs simpy won't work without a smartphone? [19:44:19] never tried tbh - I'd think not [19:44:44] I remember having a conversation about how it was possible via email or sms. But maybe that was about google's 2fa and not ours :( [19:45:46] I wish we supported yubikeys tbh [19:47:40] !htmllogs [19:47:40] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [19:48:25] wasn't there a proper homepage on bots.wmflabs.org? [19:48:54] together with searching logs? [19:50:43] yes [20:16:24] 12/31/2012 - 20:16:24 - Updating keys for mwang at /export/keys/mwang [20:19:48] mwang: OK, I think I disabled 2fa for your account. Can you try logging out and in and see how it treats you? In this case you should leave the 'token' field empty. [20:20:23] andrewbogott: I will try [20:21:19] Having no sleep catches up eventually :( *yawn* [20:21:23] You are right. The 2fa disabled now. I can log out and log back in. [20:21:23] 12/31/2012 - 20:21:22 - Creating a home directory for johnflewis at /export/keys/johnflewis [20:21:52] we need to fix that box, no need to do homes and keys fail randomly [20:22:03] andrewbogott: I can log out and log in now [20:22:36] mwang: great! Your account will be somewhat crippled (I don't know how crippled) until we can figure out a 2fa solution for you. In the meantime we'll leave it off. [20:23:59] mwang: So now we can resume our project from last week :) Can you visit the 'manage instances' page and create yourself an instance in the testlabs project? [20:26:49] 12/31/2012 - 20:26:48 - Updating keys for johnflewis at /export/keys/johnflewis [20:27:25] andrewbogott: Two-factor authentication required. [20:27:45] andrewbogott: this time I will use tokens [20:27:56] wait... [20:28:13] mwang: Just to reconfirm -- you don't have an android or an iphone, right? [20:30:52] andrewbogott: No. I think I need get one tomorrow. [20:31:11] mwang: I wrote ryan just now asking him about alternatives. I'll let you know if he has any suggestions. [20:31:29] thanks. [20:31:33] Also maybe we can scare you up a used/cheap android here. It doesn't actually need phone service to do this, after all :) [20:32:05] mwang: Remind me, you're on windows, mac, linux? [20:32:32] Right now on windows. most time I am on Ubuntu [20:33:57] Actually I have a android phone. But I did not use it as a phone. [20:34:34] andrewbogott: because it always need charging, I switched to a basic phone last year. [20:35:24] Oh! If you have an old/disabled phone then maybe you're all set. Can you still make the wifi work? [20:35:27] andrewbogott: Now I only use the smart phone for checking emails. [20:35:43] Perfect -- the app you need doesn't require network access anyway. [20:35:47] Do you have it with you? [20:35:48] andrewbogott: the wifi works [20:35:58] yes i have it with me [20:36:08] Install the 'Authenticator' app. [20:36:25] installed [20:37:23] OK, sorry, maybe I"m telling you stuff you already know :) When you enable 2fa on labsconsole, you can load the key into authenticator. [20:37:52] I think the page has instructions, but it's easy -- just tell the app you want to 'add account' and then point the camera at your PC screen. [20:45:11] andrewbogott: it works now. Thank you very much. i see "pmtpa [Toggle, Add instance]" [20:45:39] great! And you can log in again using the token from Authenticator? [20:47:06] yes. [20:49:00] ok, so go ahead and make yourself an instance called mwang-devel [20:49:15] size 'small' is fine, make sure you put it in the 'web' security group. [20:50:38] mwang: Give it 10 or 15 minutes to start up, then make sure you can log in... [20:51:10] And set up sudoers using https://labsconsole.wikimedia.org/wiki/Special:NovaSudoer [20:58:14] mwang: I'm going to get some lunch, back in 20 or so. If you get this far, run 'sudo puppetd -tv' on your instance and make sure everything is peaceful, then check the 'puppetmaster::self' box on the instance configuration page and run 'sudo puppetd -tv' a second time. [21:05:53] PROBLEM Current Load is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:06:33] PROBLEM Current Users is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:07:13] PROBLEM Disk Space is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:07:53] PROBLEM dpkg-check is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:08:03] PROBLEM Free ram is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:08:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [21:09:23] PROBLEM Total processes is now: CRITICAL on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: Connection refused by host [21:15:53] RECOVERY Current Load is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: OK - load average: 0.73, 1.10, 0.69 [21:16:33] RECOVERY Current Users is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: USERS OK - 0 users currently logged in [21:17:13] RECOVERY Disk Space is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: DISK OK [21:17:53] RECOVERY dpkg-check is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: All packages OK [21:18:03] RECOVERY Free ram is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: OK: 895% free memory [21:18:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [21:19:24] RECOVERY Total processes is now: OK on mwang-devel.pmtpa.wmflabs 10.4.1.61 output: PROCS OK: 87 processes [21:21:21] * andrewbogott is back [21:22:39] andrewbogott: mwang@mwang-devel:~$ sudo puppetd -tv [21:22:41] [sudo] password for mwang: [21:22:42] info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb [21:22:44] info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb [21:22:46] info: Caching catalog for i-0000055d.pmtpa.wmflabs [21:22:47] info: Applying configuration version '1356988828' [21:22:48] notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully [21:22:50] notice: /Stage[main]/Ldap::Client::Utils/Exec[checkout_user_ldap_tools]/returns: executed successfully [21:22:52] notice: /Stage[main]/Base::Puppet/Exec[puppet snmp trap]/returns: executed successfully [21:22:53] notice: /Stage[main]/Ganglia/File[gmondconfig]/content: [21:22:55] --- /etc/ganglia/gmond.conf 2012-12-31 21:12:45.622927012 +0000 [21:22:56] +++ /tmp/puppet-file20121231-23563-19w0h5d-0 2012-12-31 21:21:18.674822247 +0000 [21:22:58] @@ -41,6 +41,12 @@ [21:23:00] [21:23:01] [21:23:02] [21:23:04] +udp_send_channel { [21:23:06] + host = 10.4.0.79 [21:23:07] + port = 250302 [21:23:09] + ttl = 3 [21:23:10] +} [21:23:12] + [21:23:13] [21:23:15] [21:23:16] /* Each metrics module that is referenced by gmond must be specified and [21:23:18] info: FileBucket adding {md5}28825c3dac7fac7cc34150833bbf20a3 [21:23:20] info: /Stage[main]/Ganglia/File[gmondconfig]: Filebucketed /etc/ganglia/gmond.conf to puppet with sum 28825c3dac7fac7cc34150833bbf20a3 [21:23:24] notice: /Stage[main]/Ganglia/File[gmondconfig]/content: content changed '{md5}28825c3dac7fac7cc34150833bbf20a3' to '{md5}c9b0646f07d2f038dec9b90255186deb' [21:23:26] info: /Stage[main]/Ganglia/File[gmondconfig]: Scheduling refresh of Service[gmond] [21:23:27] info: /Stage[main]/Ganglia/File[gmondconfig]: Scheduling refresh of Service[gmond] [21:23:29] notice: /Stage[main]/Ganglia/Service[gmond]: Triggered 'refresh' from 2 events [21:23:31] notice: /Stage[main]/Ldap::Client::Utils/Exec[checkout_instance_ldap_tools]/returns: executed successfully [21:23:32] notice: /Stage[main]/Base::Tcptweaks/File[/etc/network/if-up.d/initcwnd]/content: [21:23:34] --- /etc/network/if-up.d/initcwnd 2012-12-31 21:14:23.414927011 +0000 [21:23:36] +++ /tmp/puppet-file20121231-23563-15jhls2-0 2012-12-31 21:21:21.478808325 +0000 [21:23:37] @@ -1,2 +1,3 @@ [21:23:39] #!/bin/bash [21:23:40] [21:23:42] +ip route change default via 10.4.0.1 dev eth0 metric 100 initcwnd 10 [21:23:43] info: FileBucket adding {md5}88a069a93872f62fcd7aa03b8d78ae93 [21:23:45] info: /Stage[main]/Base::Tcptweaks/File[/etc/network/if-up.d/initcwnd]: Filebucketed /etc/network/if-up.d/initcwnd to puppet with sum 88a069a93872f62fcd7aa03b8d78ae93 [21:23:46] notice: /Stage[main]/Base::Tcptweaks/File[/etc/network/if-up.d/initcwnd]/content: content changed '{md5}88a069a93872f62fcd7aa03b8d78ae93' to '{md5}6cd738276909dc27f22bb548a715fa6c' [21:23:48] info: /Stage[main]/Base::Tcptweaks/File[/etc/network/if-up.d/initcwnd]: Scheduling refresh of Exec[/etc/network/if-up.d/initcwnd] [21:23:49] notice: /Stage[main]/Base::Tcptweaks/Exec[/etc/network/if-up.d/initcwnd]: Triggered 'refresh' from 1 events [21:23:51] notice: Finished catalog run in 30.40 seconds [21:23:53] mwang@mwang-devel:~$ [21:24:50] mwang: Generally if you have more than a line or two to paste you should use a private channel or something like http://dpaste.org/ -- otherwise it floods screens around the world :) [21:25:00] That looks pretty good, though -- turn on puppetmaster::self and try again. [21:25:52] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [21:36:24] andrewbogott: puppetmaster::self turned on. This is the result http://dpaste.org/4fXUx/ [21:36:41] looks good! [21:37:29] now in your homedir do $ git clone https://gerrit.wikimedia.org/r/p/operations/puppet [21:38:42] Now you have two puppet checkouts on your instance. One is owned by you in ~/puppet and one is owned by root in /etc/puppet. [21:39:10] Anytime you run puppetd -tv, the puppet config in /etc/puppet is applied to your instance. [21:39:40] So, for development purposes… you'll be editing files in ~/puppet, then 'sudo cp'ing them into the appropriate place in /etc/puppet and rerunning puppetd -tv to see how it works. [21:39:41] Make sense? [21:42:34] Yes. Thanks. I will try and play it tonight and tomorrow. [21:43:22] great. [21:43:38] Might be worth reading a few puppet intros online as well, but don't bother reading too much about the syntax because it's baffling [21:43:52] Or, at least, for me, only tinkering with it helped me learn what to do. [21:43:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 18% free memory [21:44:08] andrewbogott: Definitely [21:52:24] Damianz, still up? [21:52:37] yeah, just killing windows [21:52:53] Do you have minute to get JohnLewis started w/bots? [21:53:45] Or should I just add him and turn him loose? [21:57:09] not really, as long as he uses like 4 then it's good [21:57:29] trying to figure out why windows decides to lock a service account every few min and kill my server :( [22:45:33] PROBLEM Free ram is now: WARNING on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Warning: 19% free memory [23:14:52] Is anyone around with Mad Putty Skillz? [23:53:54] PROBLEM host: aggregator1.pmtpa.wmflabs is DOWN address: 10.4.0.79 CRITICAL - Host Unreachable (10.4.0.79)