[00:20:57] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [00:25:57] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 21% free memory [00:58:07] PROBLEM Puppet freshness is now: CRITICAL on aggregator2 i-000002c0 output: Puppet has not run in last 20 hours [03:35:37] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 13% free memory [03:38:56] JuanCarlos of Freenode and [[Wikipedia:User:Killiondude|Killiondude]] of Wikipedia is Brandon Killion. His personal cell # is: (530) 207-9476. [03:38:59] His dad is: [03:39:01]
Jim D Killion [03:39:03]
(530) 669-5708 [03:39:07]
248 Arlington Cir [03:39:09]
Woodland, CA 95695-2878 [03:39:11] Mom is Rhonda, and a brother is Seth. [03:39:15] His house is worth about $153,000. [03:39:17] Moreover, he likes to really piss off IP users for things that wouldn't even faze other admins. [03:39:19] ==Other lives of Brandon Killion== [03:39:21] His dad is a pastor at a megachurch also in Woodland. [http://www.cotrwoodland.org/about-us This is their "About Us" page, and Killiondude is one of the boys in the picture. He's either the tallest or the second-tallest.] [03:39:25] He may be a pastor's kid, but acts nothing like a real christian on Wikipedia and Freenode. [03:39:27] His dad also owns and runs the "Lord's Gym" in the same town, and their equipment seems badass. [03:39:29] ==Killiondude's house== [03:39:31] [http://maps.google.com/maps?q=248+Arlington+Cir+Woodland,+CA%09+95695-2878&hl=en&ll=38.68896,-121.76656&spn=0.016062,0.010214&sll=37.0625,-95.677068&sspn=65.302917,41.835938&hnear=248+Arlington+Cir,+Woodland,+California+95695&t=h&z=16&layer=c&cbll=38.689956,-121.767967&panoid=FMYJEbVmMvkHI6GWOFizHw&cbp=12,71.85,,0,10.77 His house appears to be some McMansion in a suburb, with a basketball stand and some tra [03:39:32] what's the point of this? [03:39:36] iler right behind it.] [03:39:38] Sorry to say, but just from the looks of things, Brandon may be spoiled rotten, hence he seems to think pretty little of lowly IP users like the bunch of us. [03:39:41] ==Insight on this pastor's kid== [03:39:45] A trusted, undercover associate has stated that: [03:39:47]

''I think you would be surprised at how atypical the "typical" pastor's kid actually is.'' [03:39:50]
''~10% are goody two shoes that everyone hates'' [03:39:52]
''~60% you wouldn't know they were different from any other kids'' [03:39:54]
''~30% are asshole rebels determined to distance themselves from the PK stereotype as much as possible''

[03:40:01] !ops [03:42:47] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:47:40] jorm, the point is I WANT EVERYONE TO KNOW WHO THE ASSHOLE KILLIONDUDE REALLY IS! http://en.wikipedia.org/wiki/User:Killiondude <--- THAT IS THE ADMIN WHO MISTREATS IP USERS! [03:47:45] He tried to keep his personal info private. THAT EFFORT TURNED OUT TO BE FUTILE. [03:47:55] Now that I have given him involuntary transparency, HE SHALL TREAT ALL USERS BETTER!!!!! [03:52:47] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:55:37] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [03:57:47] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:00:37] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:02:57] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [04:07:47] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:07:47] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:12:47] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:17:57] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:22:57] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [04:32:48] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 13% free memory [04:37:47] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 2% free memory [04:42:51] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:34:02] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [05:49:42] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:51:42] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 29% free memory [05:54:42] RECOVERY Current Load is now: OK on incubator-bot1 i-00000251 output: OK - load average: 0.16, 0.65, 0.61 [06:29:02] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 22% free memory [07:45:31] hello [07:47:58] Hello [07:50:03] :) [07:51:46] Could you add me to the bastion project? I've labs account 'dereckson' and wish to be able to use an instance on the newly created 'commons-dev' project where I'm sysadmin. [08:27:43] Errrrrrr [08:28:46] * Reedy waits [08:33:17] Dereckson: you might be able to get in now [08:33:37] Hopefully your keys have been updated and a home dir created [08:34:23] Thank you. [08:59:02] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [09:08:17] New patchset: Dzahn; "change cronjob back to system user - was work-around for BZ 36206 which should be resolved" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11018 [09:08:34] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/11018 [09:08:59] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11018 [09:09:02] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11018 [09:23:44] PROBLEM Current Load is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:24:24] PROBLEM Current Users is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:25:04] PROBLEM Disk Space is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:25:44] PROBLEM Free ram is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:26:24] PROBLEM HTTP is now: CRITICAL on lynwood i-000002d2 output: CRITICAL - Socket timeout after 10 seconds [09:27:34] PROBLEM Total Processes is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:28:14] PROBLEM dpkg-check is now: CRITICAL on lynwood i-000002d2 output: Connection refused by host [09:48:54] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 20% free memory [10:56:54] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [10:59:14] PROBLEM Puppet freshness is now: CRITICAL on aggregator2 i-000002c0 output: Puppet has not run in last 20 hours [12:35:54] RECOVERY Current Load is now: OK on deployment-apache23 i-00000270 output: OK - load average: 0.03, 0.08, 0.04 [12:35:54] RECOVERY Disk Space is now: OK on deployment-apache23 i-00000270 output: DISK OK [12:35:54] RECOVERY Total Processes is now: OK on deployment-apache23 i-00000270 output: PROCS OK: 139 processes [12:36:24] RECOVERY SSH is now: OK on deployment-apache23 i-00000270 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:36:24] RECOVERY dpkg-check is now: OK on deployment-apache23 i-00000270 output: All packages OK [12:36:34] RECOVERY Free ram is now: OK on deployment-apache23 i-00000270 output: OK: 97% free memory [12:36:34] RECOVERY Current Users is now: OK on deployment-apache23 i-00000270 output: USERS OK - 0 users currently logged in [12:36:54] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 22% free memory [12:37:49] hmm [12:37:56] let upgrade to precise :-] [12:42:45] !log deployment-prep creating deployment-apache30 and deployment-apache31 using Ubuntu Precise. They will replace the Lucid boxes. See {{bug|37500}}. [12:50:21] yay [12:50:29] hashar: how you update squid? [12:50:35] does it work in puppet yet [12:51:27] na [12:51:29] still hacky [12:51:47] the box is deployment-squid [12:51:50] conf in /etc/squid/ [12:51:57] /etc is a local git repo [12:52:01] so you can commit stuff [12:52:11] then to reload squid, it is something like: sudo service squid reload [12:52:20] logs are somewhere like /var/log/squid/*.log [12:52:22] ./etc is local repo :D [12:52:29] yay for system performance then [12:52:31] petan: what do you want to change from squid conf ? [12:52:35] I don't need [12:52:42] I just wanted to know if we already moved it to puppet [12:52:53] I know how it works, I was the guy who initaly installed squid [12:52:54] I wanted to do that [12:52:59] reusing conf from production [12:53:10] but it is not worth the effort / human resources needed [12:53:11] yes the config I asked Ryan to sanitize like 6 months ago :D [12:53:23] it serves no pruposes [12:53:24] p [12:53:29] production has two layers of cache [12:53:32] hm [12:53:34] with carp and some other stuff [12:53:42] that we have no use / do not care of [12:53:45] PROBLEM Current Load is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:53:45] PROBLEM Current Load is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [12:53:47] I think we sooner switch to varnish than I get the config [12:53:57] a simple squid that does the load balancing between apaches is more than enough for now [12:54:03] yeh [12:54:07] problem is updating the config [12:54:13] you need to do that by hand [12:54:21] yup [12:54:25] PROBLEM Current Users is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:54:25] PROBLEM Current Users is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [12:55:05] PROBLEM Disk Space is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:55:05] PROBLEM Disk Space is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [12:55:55] PROBLEM Free ram is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:55:55] PROBLEM Free ram is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [12:56:25] PROBLEM HTTP is now: CRITICAL on deployment-apache30 i-000002d3 output: CRITICAL - Socket timeout after 10 seconds [12:56:25] PROBLEM HTTP is now: CRITICAL on deployment-apache31 i-000002d4 output: CRITICAL - Socket timeout after 10 seconds [12:57:35] PROBLEM Total Processes is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [12:57:40] PROBLEM Total Processes is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:58:15] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:58:15] PROBLEM dpkg-check is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [13:01:35] RECOVERY Puppet freshness is now: OK on deployment-apache23 i-00000270 output: puppet ran at Tue Jun 12 13:01:32 UTC 2012 [13:06:56] PROBLEM HTTP is now: WARNING on deployment-apache23 i-00000270 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.009 second response time [13:08:14] !log bots petrb: patching wm-bot [13:31:08] I created a new instance, when I try from bastion or directly from my machine (I configured .ssh/config with agent forwarding and ProxyCommand rules) to log to the server, I got the fllowing message: "Permission denied (publickey).". Seems it doesn't recognize my SSH keys. Is there something to do to ask puppet to deploy my keys? [13:31:25] !access [13:31:25] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [13:31:34] Dereckson: no, wait [13:31:39] !console [13:31:39] in case you want to see what is happening on terminal of your vm, check console output [13:31:44] + that [13:32:03] Already checked the console, it's how I know it's well the instance I created which replied, as the host key fingerprints matched. [13:32:27] ok, did puppet run finish? [13:32:54] Good question, do you have access to https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=commons-dev&instanceid=i-000002d2 to check? [13:33:19] if you don't see that it waits for password it didn't finish [13:33:42] "Finished puppet run" [13:33:53] that doesn't necessarily means it finished [13:33:58] but below, "[1;35merr: Could not retrieve catalog; skipping run[0mJun 12 09:17:23 i-000002d2 puppet-agent[3964]: Could not retrieve catalog; skipping run" [13:34:26] unless there is login screen it's running [13:35:02] in the past it felt like it works better if you first do your initial login and after that start to apply any puppet classes [13:35:29] Okay, I note that. [13:35:39] ah I see a message error. [13:35:43] Jun 12 13:20:57 i-000002d2 puppet-agent[3941]: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template misc/initcwnd.erb: Could not find value for 'default_gateway' at 3:/var/lib/puppet/templates/misc/initcwnd.erb at /etc/puppet/manifests/generic-definitions.pp:814 on node i-000002d2.pmtpa.wmflabs [13:35:57] (then two other lines: Using cached catalog / Could not retrieve catalog; skipping run) [13:35:57] Dereckson: i don't mean to confuse more during the debug, all valid what petan said , but there is also another easy way to directly login to an instance through the bastion, without having to even forward agents, "ProxyCommand" in ssh config [13:37:16] yep I tried also to ssh 10.4.0.100 from bastion [13:37:21] (the instance IP) [13:38:14] Host myinstance [13:38:33] ProxyCommand ssh -W %h:%p user@bastion.wmflabs.org [13:38:52] in .ssh/config. and you should be able to "ssh myinstance" from home [13:46:44] mutante: here my .ssh/config file: http://pastebin.espace-win.org/23527 [13:54:43] Dereckson: i don't use wildcard hostnames as there was some warning to be careful withit because of forking. i just listed the actual instances i am interested in. and i added another line "User mylabsuser" to the Host config [13:58:56] actually: "Ciphers arcfour256" and "MACs umac-64@openssh.com" are supposed to make it faster [13:59:48] i can't tell if it is worth it or possible drawbacks [14:00:31] but it is being recommended as "fastest" when people say jumping through proxy is slow [14:01:49] http://rfc-ref.org/RFC-TEXTS/4345/chapter6.html [14:08:15] PROBLEM Total Processes is now: UNKNOWN on lynwood i-000002d2 output: Invalid host name i-000002d2 [14:08:20] PROBLEM dpkg-check is now: UNKNOWN on lynwood i-000002d2 output: Invalid host name i-000002d2 [14:28:04] !accountreq [14:28:04] case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:38:00] !accountreq del [14:38:00] Successfully removed accountreq [14:38:09] !accountreq is in case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:38:09] Key was added [14:38:13] !accountreq [14:38:13] in case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:38:15] :) [14:51:40] it's fun [15:53:38] Change on 12mediawiki a page OAuth/status was modified, changed by Guillom link https://www.mediawiki.org/w/index.php?diff=549059 edit summary: new status update [18:25:29] Ryan_Lane: I have a very weird situation on a labs VM. Neither Subbu nor myself can log into the ve-nodejs instance (we get Permission denied (publickey) errors) and the person that's likely to blame is on vacation. Do you have magic powers that you can use to log into that instance and see what got screwed up there? [18:26:53] yes [18:27:59] RoanKattouw: mind trying for me? [18:31:02] Just tried [18:31:33] I SSHed directly to its public IP (parsoid.wmflabs.org) but I tried going through bastion earlier and that failed too [18:32:58] lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory [18:32:59] weird [18:33:27] RoanKattouw: try again [18:33:41] Failed again [18:33:48] ah [18:34:10] I have no clue how it didn't get the automount [18:34:15] ok. try now [18:34:20] I made some changes last week [18:34:52] Failed [18:34:56] wow [18:34:57] really? [18:35:04] At least from the outside yeah [18:35:11] I can try via bastion if you want [18:35:53] seems none of the things done recently have been applied [18:35:59] sec [18:37:53] RoanKattouw: try now [18:38:03] Nope [18:38:14] I wonder if AuthorizedKeysFile2 doesn't work with the ssh in precise [18:38:48] RoanKattouw: try now [18:39:19] damn [18:39:19] yay! [18:39:20] Thanks [18:39:22] that's a bitch [18:39:30] well, I need to push a fix for this now [18:39:41] Is it a compat problem with precise? [18:40:04] well, not really [18:40:08] I was using a deprecated option [18:40:20] but it should still work, if it's deprecated [18:40:22] OK, next problem [18:40:32] I added a user to the project while this bug was occurring [18:40:36] And his home dir was never cretaed [18:40:41] really? [18:40:48] hm [18:40:48] Or, wait, no [18:40:51] mount says it's there [18:40:53] But ls /home disagrees [18:40:54] * Ryan_Lane nods [18:40:59] Oh, never mind [18:40:59] that makes sense [18:41:01] He managed to log in [18:41:07] And the home dir appeared [18:41:09] home is a set of automounts [18:41:14] Aha [18:41:24] they are * automounts, which means they aren't browseable [18:41:30] like /data [18:43:22] New patchset: Ryan Lane; "Precise and above dislike AuthorizedKeysFile2" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11071 [18:43:39] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/11071 [18:44:54] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11071 [18:44:57] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11071 [18:45:24] seems I broke ssh in precise in labs for a bit [19:43:03] @help [19:43:03] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.5.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [21:00:14] PROBLEM Puppet freshness is now: CRITICAL on aggregator2 i-000002c0 output: Puppet has not run in last 20 hours [21:06:14] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [21:07:37] !log deployment-prep Configuring apache30 and 31 to use applicationserver::labs and imagescaler::labs [21:11:03] !log deployment-prep Rebooting apache30 and 31 so they apply pending package updates. Off for today. [21:11:10] of course the log bot is dead [21:11:12] :-/ [21:14:29] https://bugzilla.wikimedia.org/show_bug.cgi?id=37527 -- Labslogbot is dead [21:14:32] bye bye ;) [21:14:37] thanks hashar [21:15:09] chrismcmahon: will probably have the ffmpeg thumbnails fixed this week :-D [21:15:18] would need a way to purge the invalid ones though :-] [21:15:22] I am off to bed sorry [21:15:25] great! [21:15:25] ttyl! [21:15:26] :-( [21:15:29] (timezone sucks) [21:15:41] hashar: I'm around relatively early most days. [21:17:10] ping me whenever you come online so :-] [21:17:14] off for real ! [21:24:35] RECOVERY Current Users is now: OK on deployment-apache30 i-000002d3 output: USERS OK - 0 users currently logged in [21:25:45] RECOVERY Disk Space is now: OK on deployment-apache30 i-000002d3 output: DISK OK [21:25:55] RECOVERY Free ram is now: OK on deployment-apache30 i-000002d3 output: OK: 94% free memory [21:27:35] RECOVERY Total Processes is now: OK on deployment-apache30 i-000002d3 output: PROCS OK: 110 processes [21:27:40] RECOVERY Total Processes is now: OK on deployment-apache31 i-000002d4 output: PROCS OK: 95 processes [21:28:15] RECOVERY dpkg-check is now: OK on deployment-apache30 i-000002d3 output: All packages OK [21:28:15] RECOVERY dpkg-check is now: OK on deployment-apache31 i-000002d4 output: All packages OK [21:28:45] RECOVERY Current Load is now: OK on deployment-apache30 i-000002d3 output: OK - load average: 0.16, 0.54, 0.58 [21:28:45] RECOVERY Current Load is now: OK on deployment-apache31 i-000002d4 output: OK - load average: 0.19, 0.19, 0.11 [21:29:25] RECOVERY Disk Space is now: OK on deployment-apache31 i-000002d4 output: DISK OK [21:29:35] RECOVERY Current Users is now: OK on deployment-apache31 i-000002d4 output: USERS OK - 0 users currently logged in [21:30:55] RECOVERY Free ram is now: OK on deployment-apache31 i-000002d4 output: OK: 97% free memory [22:41:20] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 21% free memory [22:44:57] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 3.83, 13.46, 7.37 [22:54:57] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.20, 2.09, 4.01 [22:59:57] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory