[01:32:07] * Damianz stretches out [03:15:22] PROBLEM Current Load is now: WARNING on incubator-bots2 incubator-bots2 output: WARNING - load average: 4.52, 5.78, 5.25 [03:17:19] * Damianz paws labs-nagios-wm_ [03:25:22] RECOVERY Current Load is now: OK on incubator-bots2 incubator-bots2 output: OK - load average: 6.48, 4.67, 4.79 [06:49:49] PROBLEM Current Load is now: WARNING on incubator-nfs incubator-nfs output: WARNING - load average: 5.90, 5.85, 5.43 [06:54:49] RECOVERY Current Load is now: OK on incubator-nfs incubator-nfs output: OK - load average: 4.47, 4.57, 4.94 [07:19:49] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [07:29:46] mmm [07:30:09] * Beetstra really has to do something about the memory leaking in some of his bots [07:30:23] Maybe this Thu/Fri [07:34:49] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 28% free memory [08:44:45] petrbena@Desktop-ws:~$ uptime [08:44:47] 09:44:25 up 74 days, 13:42, 36 users, load average: 3.30, 2.13, 1.00 [08:44:49] :) [08:45:04] my desktop has bigger uptime than wikipedia [08:45:37] yay [08:47:49] 74days? My toaster has a higher uptime :P [08:47:55] :P [08:48:28] I am wondering what all is eating the cpu now [08:48:46] I don't do much and load 4 :\ [08:48:50] spiders [08:48:53] heh [08:49:22] I need to insert few hdd's [08:49:53] extend my home [09:54:33] PROBLEM Current Load is now: WARNING on incubator-bots2 incubator-bots2 output: WARNING - load average: 4.08, 5.11, 5.08 [10:04:33] RECOVERY Current Load is now: OK on incubator-bots2 incubator-bots2 output: OK - load average: 3.28, 4.17, 4.73 [10:27:33] PROBLEM Current Load is now: WARNING on incubator-bots2 incubator-bots2 output: WARNING - load average: 3.56, 5.00, 5.08 [12:06:33] PROBLEM Total Processes is now: CRITICAL on incubator-bots incubator-bots output: PROCS CRITICAL: 1451 processes [12:20:33] PROBLEM Current Load is now: WARNING on incubator-bots2 incubator-bots2 output: WARNING - load average: 6.74, 5.78, 5.22 [12:26:43] PROBLEM Free ram is now: WARNING on incubator-bots incubator-bots output: Warning: 16% free memory [12:30:33] RECOVERY Current Load is now: OK on incubator-bots2 incubator-bots2 output: OK - load average: 3.37, 4.71, 4.99 [12:58:33] PROBLEM Current Load is now: WARNING on incubator-bots2 incubator-bots2 output: WARNING - load average: 5.18, 5.78, 5.22 [13:28:20] New review: Dzahn; "fyi: the logo change has been merged meanwhile in change 2013" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2012 [14:55:16] !log planet - raised floating_ips quota to 1, allocate one public IP for webserver [14:55:17] Logged the message, Master [15:07:18] !log planet - associate IP to instance venus, add hostname planet.wmflabs [15:07:19] Logged the message, Master [17:50:12] !accountreq [17:50:12] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [17:50:50] Hodgestar / jerith: who do I ping for an account on the Wikimedia over USSD & SMS machine? [18:08:18] hi ryan [18:08:26] howdy [18:08:29] morning [18:08:58] two quick questions, on git/gerrit, how can i browse the code? i can't find the gitweb links [18:09:14] yeah. that's not terribly easy, unfortunately [18:09:41] not easy or impossible? :) [18:10:15] you can find a gitweb link by looking at a patchset in a change [18:10:32] hm. links need url encoding [18:10:38] I wonder if the bot in here can do that [18:11:17] oh. no it doesn't [18:11:39] <^demon|away> Would it be feasible to setup our own gitweb with less crazy urls so people can find the code? [18:11:41] !gitweb is https://gerrit.wikimedia.org/r/gitweb?p=analytics/udp-filters.git [18:11:41] Key was added! [18:11:45] ye [18:11:45] *yes [18:11:57] !gitweb analytics/udp-filters [18:11:57] https://gerrit.wikimedia.org/r/gitweb?p=analytics/udp-filters.git [18:12:08] well, kind of [18:12:20] so, I can add redirects to make this slightly easier [18:12:22] thanks! [18:12:27] but the links will continue to stay a little crazy [18:12:32] that's okay [18:16:13] 2nd question [18:16:24] i have trouble setting up proxy access [18:16:30] i follow the steps as described [18:16:34] but my key is giving problems [18:16:39] while normal access just works fine [18:26:32] hi roan! [18:26:46] Hi [18:27:15] good flight back to holland? [18:27:38] Yeah, it was fine [18:27:47] nice :) [18:28:07] A few delays but I was picked up at my gate by an airline employee that rushed me past all the lines :D [18:28:16] ummhhhh would you have some time today to do the final review of the new udp-filter code ? [18:28:23] Ah, right [18:28:23] NICE! VIP! [18:28:37] Ah, yeah, probably tomorrow [18:28:40] That's my 20% day [18:29:12] No, wait, Wednesday [18:30:19] today is not possible? the global dev team is very anxious in collecting mobile stats for india...... [18:30:35] I'll try [18:30:56] It's 7:30pm and I won't be staying up all that late tonight :) [18:31:10] * RoanKattouw slept from 6:30pm till 7:30am last night [18:31:15] i understand, those pesky jet lags :) [18:31:36] okay, ttyl [18:41:40] diederik: hmm. if you use a non-default key, you need to specify the key per domain in your config [18:42:00] but i am using the same key [18:42:13] if he is using a ssh-agent that shouldn't matter [18:42:22] it says it's not a RSA1 key [18:42:24] RoanKattouw: so i heard you were down there and did not dare to stop by... :-/ [18:43:01] Danny_B|backup: I was where now? [18:43:59] are you providing a public key instead of the private one? [18:44:09] nope [18:44:23] what command are you using? [18:44:51] exec ssh -e none bastion1.pmtpa.wmflabs exec nc -w 3600 pageviews.pmtpa.wmflabs 22 [18:45:23] what ssh version are you using? [18:45:27] you may not need nc [18:45:40] OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011 [18:46:02] I have 5.9 [18:46:21] I use -oProxyCommand 'ssh -W %h:%p bastion.wmflabs.org' [18:46:37] !reqaccount [18:46:38] but that substitution may not be available for you [18:46:44] !log deployment-prep configuring some boxes for cluster to handle high load [18:46:45] i am on OSX [18:46:45] Logged the message, Master [18:46:46] ummm [18:47:01] hi Ryan :) [18:47:03] this should be in your config file: ProxyCommand ssh bastion1.pmtpa.wmflabs exec nc -w 3600 %h %p [18:47:04] PROBLEM Free ram is now: CRITICAL on incubator-bots incubator-bots output: CHECK_NRPE: Socket timeout after 10 seconds. [18:47:09] thanks for creating the type [18:47:12] petan: yw [18:47:23] sdehaan: Ping? [18:47:27] Hodgestar: Ping? [18:47:33] more specifically, this: [18:47:33] Host *.*.wmflabs [18:47:34] ProxyCommand ssh bastion1.pmtpa.wmflabs exec nc -w 3600 %h %p [18:47:53] also, this: [18:47:53] Host bastion1.pmtpa.wmflabs [18:47:53] Hostname bastion.wmflabs.org [18:47:53] ProxyCommand none [18:48:06] ryan: i just copied the example from the wiki [18:48:23] i think the example needs to be updated [18:48:31] why? [18:48:32] Ryan_Lane: Can you set up Labs accounts for sdehaan and Hodgestar when they're around? [18:48:33] btw Ryan_Lane is it possible to create a puppet repo for deployment project or not? [18:48:33] it works for me [18:48:36] [18:48:37] Host bastion1.pmtpa.wmflabs [18:48:38] Hostname bastion.wmflabs.org [18:48:38] ProxyCommand none [18:48:38] Host bastion1.eqiad.wmflabs [18:48:39] Hostname bastion2.wmflabs.org [18:48:39] ProxyCommand none [18:48:39] Host *.pmtpa.wmflabs [18:48:39] ProxyCommand ssh -e none bastion1.pmtpa.wmflabs exec nc -w 3600 %h %p [18:48:40] Host *.eqiad.wmflabs [18:48:40] ProxyCommand ssh -e none bastion1.eqiad.wmflabs exec nc -w 3600 %h %p [18:48:41] [18:49:02] spam! [18:49:03] petan: a repo? [18:49:04] :D [18:49:13] jerith: pong [18:49:13] petan: just push into puppet [18:49:14] Ryan_Lane: I mean a repository for config files [18:49:17] no need for another repo [18:49:22] Ryan_Lane: They're two colleagues of mine who should have access to the vumi machine. [18:49:25] I know but that would affect all projects or not? [18:49:36] yes, but it should [18:49:42] !accountreq [18:49:42] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [18:49:42] like if I wanted to create a class for apache config, only for deployment project [18:50:05] declerambaul: ask ryan to give you instructions on how to get an account [18:50:20] !account-questions | declerambaul [18:50:20] declerambaul: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [18:50:25] petan: the puppet config needs to be kind of generic [18:50:31] ok [18:50:51] so if I had a specific configuration only way to make it wide is to copy it [18:50:53] to all instances [18:50:57] also, most of the config is already in puppet [18:51:17] you guys didn't use any of the pre-existing stuff [18:51:25] I know, but let's say I wanted to change some limits in php, just for our project [18:51:29] without affecting others [18:51:38] then it should be a variable [18:51:43] ok [18:51:48] and should have a default [18:51:54] PROBLEM Free ram is now: WARNING on incubator-bots incubator-bots output: Warning: 12% free memory [18:51:57] which is whatever the default is now [18:52:35] what is puppet group [18:52:45] https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [18:52:48] !docs [18:52:48] View complete documentation at https://labsconsole.wikimedia.org/wiki/Help:Contents [18:53:03] Ryan_Lane: I started a help for console, if you had comments, let me know :) [18:53:16] ryan: i still get a publickey access denied error: [18:53:17] debug1: Executing proxy command: exec ssh bastion1.pmtpa.wmflabs exec nc -w 3600 analytics.pmtpa.wmflabs 22 [18:53:17] debug3: Not a RSA1 key file /Users/diederik/.ssh/id_rsa. [18:53:18] debug2: key_type_from_name: unknown key type '-----BEGIN' [18:53:18] debug3: key_read: missing keytype [18:53:18] debug2: key_type_from_name: unknown key type 'Proc-Type:' [18:53:18] debug3: key_read: missing keytype [18:53:19] debug2: key_type_from_name: unknown key type 'DEK-Info:' [18:53:19] debug3: key_read: missing keytype [18:53:19] debug3: key_read: missing whitespace [18:53:29] btw puppet group should be also documented, but I am not really sure what it is [18:53:39] diederik: please use pastebin or something else like it, rather than flooding the channel [18:53:47] also some parts of interface aren't accessible for me, so I can't make help for it, someone else will need to [18:53:54] PROBLEM Current Load is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [18:53:55] petan: puppet groups are just collections of variables and classes [18:54:00] ah [18:54:18] petan: per-project groups exist so that anyone can change them for their project, to test, before they are made global [18:54:25] ok, so when I create my own class I can insert it to config of instance? [18:54:34] yep [18:54:38] nice [18:54:43] I will surely use it soon [18:54:54] PROBLEM Current Users is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [18:55:44] PROBLEM Free ram is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [18:55:44] PROBLEM Disk Space is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [18:56:54] PROBLEM Total Processes is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [18:57:34] PROBLEM dpkg-check is now: CRITICAL on deployment-web1 deployment-web1 output: Connection refused by host [19:02:04] PROBLEM Free ram is now: CRITICAL on incubator-bots incubator-bots output: CHECK_NRPE: Socket timeout after 10 seconds. [19:08:01] !initial-login | declerambaul [19:08:01] declerambaul: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [19:15:02] PROBLEM Total Processes is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [19:15:22] PROBLEM dpkg-check is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [19:16:52] PROBLEM Current Load is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [19:17:42] PROBLEM Current Users is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [19:17:52] PROBLEM Disk Space is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [19:18:32] PROBLEM Free ram is now: CRITICAL on deployment-web2 deployment-web2 output: Connection refused by host [20:42:02] PROBLEM Free ram is now: WARNING on incubator-bots incubator-bots output: Warning: 11% free memory [23:08:12] PROBLEM host: nova-dev4 is DOWN address: nova-dev4 CRITICAL - Host Unreachable (nova-dev4) [23:38:33] PROBLEM host: nova-dev4 is DOWN address: nova-dev4 CRITICAL - Host Unreachable (nova-dev4) [23:42:23] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [23:43:13] RECOVERY host: incubator-bots2 is UP address: incubator-bots2 PING OK - Packet loss = 0%, RTA = 0.58 ms