[00:54:40] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours
[00:58:40] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours
[01:16:02] <Ryan_Lane>	 paravoid: We're going grab some food. Want to come?
[01:16:14] <paravoid>	 sure
[01:16:15] <paravoid>	 where?
[01:16:53] <Ryan_Lane>	 Not sure. You at the office?
[01:17:00] <paravoid>	 yes
[01:17:19] <Ryan_Lane>	 We'll meet you there
[01:17:33] <paravoid>	 oh okay
[02:41:00] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on mobile-enwp i-000000ce output: Warning: 15% free memory
[02:41:40] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 7% free memory
[03:01:44] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 5% free memory
[03:04:00] <labs-nagios-wm_>	 RECOVERY Current Load is now: OK on wikidata-dev-1 i-0000020c output: OK - load average: 1.29, 1.45, 0.63
[03:06:44] <labs-nagios-wm_>	 RECOVERY Current Users is now: OK on wikidata-dev-1 i-0000020c output: USERS OK - 0 users currently logged in
[03:06:44] <labs-nagios-wm_>	 RECOVERY Disk Space is now: OK on wikidata-dev-1 i-0000020c output: DISK OK
[03:06:44] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on wikidata-dev-1 i-0000020c output: OK: 86% free memory
[03:07:20] <labs-nagios-wm_>	 RECOVERY Total Processes is now: OK on wikidata-dev-1 i-0000020c output: PROCS OK: 90 processes
[03:07:30] <labs-nagios-wm_>	 RECOVERY dpkg-check is now: OK on wikidata-dev-1 i-0000020c output: All packages OK
[03:11:10] <labs-nagios-wm_>	 PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 4.24, 5.76, 5.17
[03:16:10] <labs-nagios-wm_>	 RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 2.67, 4.45, 4.80
[03:16:50] <labs-nagios-wm_>	 PROBLEM Disk Space is now: CRITICAL on deployment-transcoding i-00000105 output: DISK CRITICAL - free space: / 38 MB (2% inode=53%):
[03:21:50] <labs-nagios-wm_>	 PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 60 MB (4% inode=53%):
[03:46:30] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 16% free memory
[03:51:30] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 17% free memory
[03:51:30] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory
[03:55:14] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 13% free memory
[04:06:24] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory
[04:11:24] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory
[04:11:24] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory
[04:11:24] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory
[04:15:14] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory
[04:16:24] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory
[04:16:24] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory
[04:16:24] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 7% free memory
[04:21:24] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory
[04:25:14] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory
[05:20:34] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours
[05:30:53] <SpComb>	 nagios </3
[06:42:25] <labs-nagios-wm_>	 PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 2242 MB (13% inode=82%): /home/petrb 2242 MB (13% inode=82%): /home/laner 2242 MB (13% inode=82%):
[06:44:15] <labs-nagios-wm_>	 PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 5.58, 6.32, 5.39
[06:47:25] <labs-nagios-wm_>	 RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK
[07:09:21] <labs-nagios-wm_>	 RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 2.08, 3.84, 4.66
[08:03:18] <hashar>	 helllo
[08:52:01] <labs-nagios-wm_>	 PROBLEM Disk Space is now: CRITICAL on deployment-transcoding i-00000105 output: DISK CRITICAL - free space: / 34 MB (2% inode=53%):
[08:57:01] <labs-nagios-wm_>	 PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 61 MB (4% inode=53%):
[10:55:31] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours
[10:59:31] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours
[12:02:00] <petan|wk>	 hi
[12:09:38] <petan|wk>	 !log
[12:23:29] <gerrit-wm>	 New patchset: Dzahn; "test multiple cronjob creation in labs where i need it anyways" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5315
[12:23:41] <gerrit-wm>	 New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/5315
[12:24:50] <gerrit-wm>	 New patchset: Dzahn; "test multiple cronjob creation in labs where i need it anyways" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5315
[12:25:04] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5315
[12:25:54] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/5315
[12:25:57] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5315
[12:28:09] <petan|wk>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=36090
[12:28:13] <petan|wk>	 DamianZ, mutante
[12:28:20] <petan|wk>	 if you have an idea how to solve it do that
[12:28:41] <petan|wk>	 the bot is configured by Ryan and in fact I don't know where it is
[12:28:50] <DamianZ>	 bots-labs is where it should live
[12:28:57] <DamianZ>	 Bit busy to look for it atm though
[12:30:16] <mutante>	 petan|wk: is it broken?
[12:30:24] <mutante>	 or just "unstable" , what means stable instance
[12:30:53] <mutante>	 you mean like the distro version?
[12:34:00] <petan|wk>	 mutante: bots-2 is most loaded instance in bots project
[12:34:07] <petan|wk>	 it's so loaded that log bot is down most of time
[12:34:13] <petan|wk>	 !log blah blah
[12:34:15] <petan|wk>	 you see?
[12:34:32] <petan|wk>	 only bots which can recover from OOM can live there
[12:34:46] <petan|wk>	 it's not for weak bots :P
[12:35:21] <Reedy>	 if oom, fork!
[12:35:37] <petan|wk>	 that's how other bots work I think
[12:35:54] <petan|wk>	 :)
[12:36:01] <mutante>	 i see what you mean but no idea why its so loaded
[12:36:22] <Reedy>	 just restart it
[12:36:22] <petan|wk>	 I don't know it either and I don't really care I just want to move it to another instance so that it works
[12:36:24] <Reedy>	 that'll sort it out
[12:36:39] <petan|wk>	 Reedy: this issue is happening for months, dozen of restarts happened in past
[12:36:55] <petan|wk>	 I just gave up on waiting for someone to fix it, so I created ticket
[12:37:19] <Reedy>	 add moar ram
[12:37:28] <mutante>	 good idea
[12:37:58] <petan|wk>	 that bot itself doesn't need much ram, so moving it to another instance is surely easiest + we can't add moar ram to existing isntance
[12:38:47] <Reedy>	 http://cdn.memegenerator.net/instances/400x/18959123.jpg
[12:38:48] <mutante>	 just create another instance?
[12:39:01] <mutante>	 you are in bots project, right
[12:39:23] <petan|wk>	 mutante: I don't have problem allocating the resources, I have problem locating where bot lives and moving it
[12:39:33] <petan|wk>	 I mean if I create new instance who move the bot? :-)
[12:39:37] <petan|wk>	 it's Ryan's bot
[12:39:40] <petan|wk>	 he knows where it is
[12:39:44] <petan|wk>	 he knows how to move it
[12:40:14] <petan|wk>	 we already have instance bots-labs which is idling and waiting for this bot
[12:40:23] <petan|wk>	 but someone need to move it there
[12:40:27] <mutante>	 thats true, just saying we are at the same level then
[12:40:35] <petan|wk>	 that's why I created the ticket
[12:40:43] <petan|wk>	 which is assigned to Ryan now
[12:40:48] <mutante>	 i dont know about the bot, and either there are public docs or we'll have to ask Ryan to make some
[12:40:51] <petan|wk>	 I just hoped someone knows more than I do
[12:41:02] <Reedy>	 Ryan does
[12:41:10] <petan|wk>	 !Ryan
[12:41:10] <wm-bot>	 man of all answers ever (but there are others :))
[12:41:20] <mutante>	 but im sure you can feel free to also just do it
[12:41:25] <petan|wk>	 problem is that he's not around when I am
[12:41:27] <mutante>	 since its broken now
[12:41:36] <mutante>	 and you have access to the instance
[12:41:47] <petan|wk>	 mutante: I'd love to do it if only I knew where the bot lives
[12:42:21] <petan|wk>	 I don't even know how to properly move it
[12:42:26] <petan|wk>	 it's packaged to .deb package
[12:42:26] <Reedy>	 search the file system?
[12:42:32] <Reedy>	 apt-get remove logmsgbot
[12:42:33] <petan|wk>	 it's installed using apt
[12:42:45] <petan|wk>	 ok that's way how to remove existing one
[12:42:55] <petan|wk>	 but I need to install it to another server too :)
[12:42:58] <mutante>	 i can look at it later, just give me a while to finish what i was at
[12:43:06] <petan|wk>	 np
[12:43:21] <mutante>	 if its just a debian package
[12:43:27] <mutante>	 apt-get install <bot> :)
[12:43:30] <petan|wk>	 yes it is .deb package no one has
[12:43:45] <petan|wk>	 it's just installed package which doesn't exist on fs
[12:43:51] <petan|wk>	 maybe it's somewhere on Ryan's pc
[12:44:07] <mutante>	 then the package is most likely installed via a puppet class you'd have to apply to the instance via labsconsole
[12:44:17] <petan|wk>	 I didn't find any
[12:44:34] <petan|wk>	 I think he didn't puppetize this so far
[12:44:43] <petan|wk>	 also it require other stuff, like having bot password
[12:44:55] <petan|wk>	 it needs to login to console so that it can edit wiki
[12:45:20] <mutante>	 hmm,ok, maybe this _is_ better to ask him about first
[12:45:36] <mutante>	 i'll take a look later..bbiaw
[12:48:19] <gerrit-wm>	 New patchset: Dzahn; "cronjobs - fix project name in command" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5318
[12:48:32] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5318
[12:48:58] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/5318
[12:49:01] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5318
[13:17:58] <petan|wk>	 hashar: if u are around https://bugzilla.wikimedia.org/show_bug.cgi?id=36011
[13:18:10] <petan|wk>	 you said that you could help with stuff on deployment site
[13:18:39] <hashar>	 yup
[13:18:55] <hashar>	 that is probably going to keep me busy during May ;-D
[13:19:00] <petan|wk>	 ok
[13:20:44] <hashar>	 petan|wk: while you are around, is there any document about the WMF Beta project ?
[13:21:45] <petan|wk>	 yes
[13:22:08] <petan|wk>	 http://deployment.wikimedia.beta.wmflabs.org/wiki/Help
[13:22:31] <petan|wk>	 http://deployment.wikimedia.beta.wmflabs.org/wiki/HetDeploy
[13:22:40] <petan|wk>	 !sal
[13:22:40] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need
[13:23:15] <petan|wk>	 now we have 6 apaches
[13:23:27] <petan|wk>	 other than that it's same
[13:23:53] <hashar>	 \o/
[13:25:50] <hashar>	 petan|wk: about memcached, seems you can enable debugging with $wgMemCachedDebug = true
[13:26:01] <hashar>	 according to doc, that will send message to $wgDebugLogFile
[13:26:27] <petan|wk>	 that file is pretty much unreadable given to number of messages
[13:26:46] <hashar>	 ohhh
[13:26:49] <petan|wk>	 we don't have such a traffic like production but we still have some :)
[13:26:52] <hashar>	 so we need to have that split somewhere else :D
[13:27:25] <petan|wk>	 but I could try to grep some errors from there if I knew how they look
[13:27:39] <petan|wk>	 but I will try it
[13:28:15] <hashar>	 there is another debug function which accept a prefix
[13:28:37] <hashar>	 then any debug message using that prefix can be written in a specific file
[13:29:07] <hashar>	 ahhh wfDebugLog()
[13:32:47] <petan|wk>	 ok tail -f errors.log
[13:32:51] <petan|wk>	 I get a load of text
[13:32:58] <petan|wk>	 :D
[13:34:13] <petan|wk>	 memcached: MemCache: sock i:0; got swwiki:sidebar:sw
[13:34:14] <petan|wk>	 memcached: get(swwiki:resourceloader:filter:minify-css:7:d5a1bf6cbd05fc6cc2705e47f52062dc)
[13:34:25] <petan|wk>	 is that what we need?
[13:37:32] <petan|wk>	 it would be really cool if mediawiki created a token for each run of php code and addded it to source code like <!-- Debug token ad4561a5425e423...
[13:37:44] <petan|wk>	 each log messages would be prefixed with that :)
[13:37:53] <petan|wk>	 so I would just grep token and get all log lines for what I need
[13:38:39] <hashar>	 re sorry
[13:38:42] <hashar>	 had a phone call
[13:38:44] <petan|wk>	 ok
[13:39:08] <hashar>	 oh prefixing the output will surely be usefull
[13:39:23] <hashar>	 and that is probably super easy to implement :-D
[13:39:27] <petan|wk>	 I mean prefix it with something what help me identify my session
[13:39:31] <hashar>	 yup
[13:39:42] <petan|wk>	 yes it's very easy to implement
[13:53:08] <hashar>	 petan|wk: you could craft one and set $wgDebugLogPrefix
[13:53:25] <petan|wk>	 of course
[13:53:41] <hashar>	 petan|wk:  $apache_hostname + $dbname + $pid  would do it maybe
[13:54:40] <hashar>	 splitting memcached debug log : https://gerrit.wikimedia.org/r/5323
[13:57:27] <petan|wk>	 php.net is down :/
[13:58:14] <petan|wk>	 it seems to be back
[14:02:50] <hashar>	 !log <big>I can log too!</big>
[14:06:46] <petan|wk>	 hashar: does site look broken to you now?
[14:06:58] <petan|wk>	 it appears to not be able to load skin now
[14:07:22] <hashar>	 indeed :D
[14:07:24] <hashar>	 no skin at all
[14:07:53] <hashar>	 parse error in  wmf-config/CommonSettings.php line 71
[14:08:06] <hashar>	 http://deployment.wikimedia.beta.wmflabs.org/w/load.php :-D
[14:09:01] <petan|wk>	 fixed
[14:09:45] <hashar>	 how do you deploy the conf to the apaches ? 
[14:09:57] <hashar>	 you should php -l  any php file before copying the file
[14:10:24] <petan|wk>	 I don't copy it
[14:10:30] <petan|wk>	 everything lives on same storage
[14:10:40] <hashar>	 oh all apaches using a shared NFS ?
[14:10:44] <petan|wk>	 yes
[14:10:59] <petan|wk>	 once you change a single file it's live
[14:11:09] <petan|wk>	 that's not really production like :)
[14:11:15] <petan|wk>	 but in the end labs isn't prod
[14:11:25] <petan|wk>	 so it doesn't matter so much if you break stuff here
[14:11:31] <hashar>	 probably
[14:11:48] <hashar>	 till we start using it to run tests continuously :-]]]
[14:11:54] <petan|wk>	 true
[14:12:11] <petan|wk>	 I am actually trying to make it stable a bit, we should probably have some deployment process
[14:12:26] <petan|wk>	 so that changes are smooth
[14:12:37] <petan|wk>	 I have no idea how does it work on prod
[14:12:50] <hashar>	 production uses https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=files/misc/scripts;h=d1ab7f89dbec956165e8f677035a135a08aeb8bf;hb=HEAD
[14:13:27] <hashar>	 specially scap
[14:13:33] * DamianZ  gives petan|wk head

[14:13:37] <hashar>	 and over time it became harder and harder :-(
[14:14:05] <petan|wk>	 heh
[14:14:38] <petan|wk>	 we need to find out a way to keep config synced with prod config
[14:14:49] <petan|wk>	 it's a bit outdated right now
[14:15:17] <petan|wk>	 we have own InitializeSettings but CommonSettings contains some modifications and these needs to be merged
[14:15:24] <petan|wk>	 in case we need to update it
[14:15:39] <hashar>	 that production part need to be migrated to git one day
[14:15:40] <hashar>	 :-/
[14:16:14] <hashar>	 how can I get access on the beta project?
[14:19:58] <petan|wk>	 ssh deployment-dbdump
[14:21:08] <mutante>	 petan|wk: ok, re, you said it's on "bots2"?
[14:21:09] <petan|wk>	 now :)
[14:21:16] <petan|wk>	 mutante: bots-2
[14:21:17] <labs-home-wm_>	 04/19/2012 - 14:21:17 - Creating a home directory for hashar at /export/home/deployment-prep/hashar
[14:21:55] <petan|wk>	 mutante: you will probably have troubles even ssh there
[14:22:17] <labs-home-wm_>	 04/19/2012 - 14:22:17 - Updating keys for hashar
[14:22:23] <hashar>	 danke
[14:22:25] <petan|wk>	 it's a bit loaded one
[14:24:04] * chrismcmahon  reads the back scroll

[14:24:10] <petan|wk>	 the hash thing works but I would rather make it temporary
[14:24:24] <petan|wk>	 hashar: is there a way to append the text to bottom of html?
[14:24:30] <petan|wk>	 like some $footer variable
[14:24:56] <petan|wk>	 right now I do echo in CommonSettings, that is pretty ugly and run before header is sent to browser
[14:25:17] <petan|wk>	 as a temporary workaround it's ok though
[14:25:19] <hashar>	 maybe the Skin object let you add HTML comments
[14:25:28] <hashar>	 another solution is to use the debug toolbar maybe
[14:25:43] <hashar>	 http://www.mediawiki.org/wiki/Manual:$wgDebugToolbar
[14:25:52] <petan|wk>	 I would rather use ugly comment than debug toolbar :D
[14:26:00] <mutante>	 petan|wk: anything else running on that? other bots besides logbots? will rebooting kill anything still alive?
[14:26:06] <petan|wk>	 I want something what doesn't affect output
[14:26:13] <petan|wk>	 mutante: of course
[14:26:22] <petan|wk>	 mutante: did you think that log bot is what cause heavy load? :D
[14:26:28] <hashar>	 petan|wk: you could use  wgOut->addHTML( 'foo' );  or something like that
[14:26:30] <petan|wk>	 mutante: bunch of bots run there
[14:26:37] <petan|wk>	 mutante: I have no idea who own them
[14:26:49] <petan|wk>	 mutante: rebooting instance might blow up wikipedia
[14:26:58] <mutante>	 petan|wk: i have no idea, could have been all other bots are also dead anyways
[14:27:11] <mutante>	 blow up wikipedia? wth :)
[14:27:12] <petan|wk>	 mutante: I don't think so last time I sshed they were running
[14:27:30] <petan|wk>	 mutante: you know that wikipedia is edited by bots most
[14:27:42] <petan|wk>	 killing bots might as well bring whole project down :P
[14:28:28] <petan|wk>	 hashar: does it work from CommonSettings? :P
[14:28:29] <mutante>	 meh, thats why i'm asking if other bots are already down as well
[14:28:37] <petan|wk>	 I don't think si
[14:28:44] <petan|wk>	 if they were down the instance wouldn't be loaded at all
[14:28:46] <mutante>	 something that is already broken cant be broken more..well usually :)
[14:28:56] <petan|wk>	 you would be surprised
[14:29:08] <petan|wk>	 some of the bots run in hundreds of threads
[14:29:21] <petan|wk>	 it seems to me that they spawn more threads as long as they can
[14:29:44] <petan|wk>	 or processes
[14:29:59] <mutante>	 if it was true that stopping an edit bot breaks the site that should be changed quick ;)  ... ah i got on the shell now
[14:30:00] <petan|wk>	 some incubator bot was spawning more than 20 000 processes of itself
[14:30:46] * mutante  sees a bunch of perl by beetstra

[14:31:01] <petan|wk>	 yes that's it
[14:31:05] <petan|wk>	 whatever it does
[14:33:05] <mutante>	 he put a "minimanual.txt" there
[14:33:23] <mutante>	 looks like the bots are called "linkwatcher", "coibot", "xlinkbot" and "unblockbot"
[14:33:37] <mutante>	 but just linkwatcher is on bots-2, the others on bots-3
[14:37:06] <petan|wk>	 ok good
[14:37:12] <petan|wk>	 it doesn't really matter
[14:37:20] <petan|wk>	 important is to get log to some other instance
[14:37:56] <hashar>	 petan|wk: ahhhhh found it
[14:38:03] <petan|wk>	 hashar: ?
[14:38:10] <hashar>	 petan|wk: before the debug toolbar, we had  $wgShowDebug
[14:38:23] <petan|wk>	 that display debug to comments in source code
[14:38:23] <hashar>	 petan|wk: that adds some debug below the main content
[14:38:33] <petan|wk>	 that isn't something I would like to use :O
[14:38:43] <petan|wk>	 I used it before it affects performance badly
[14:39:00] <petan|wk>	 also I don't need to see debug output in most cases
[14:39:12] <petan|wk>	 only when some problem happens in that case I am fine with doing grep token error.log
[14:39:52] <petan|wk>	 making that token a bit more cute would be useful though :)
[14:45:27] <petan|wk>	 wtf
[14:45:39] <petan|wk>	 Special:Preferences doesn't show the tabs
[14:45:47] <petan|wk>	 is that supposed to be like that?
[14:45:52] <petan|wk>	 or it's a bug
[14:46:13] <labs-nagios-wm_>	 PROBLEM Free ram is now: CRITICAL on mobile-enwp i-000000ce output: Connection refused by host
[14:46:31] <mutante>	 !log bots - installing locate and updatedb to "find" stuff :)
[14:46:49] <petan|wk>	 mutante: we are fixing log bot, actually :)
[14:46:52] <mutante>	 yeah, its redundant to log fixing the logbot:)
[14:46:57] <petan|wk>	 hehe
[14:47:52] <mutante>	 petan|wk: cant find any "logmsgbot" on that instance
[14:48:14] <petan|wk>	 that's not how it's called
[14:48:20] <mutante>	 that's what it is in prod.
[14:48:37] <petan|wk>	 I know that service adminbot restart fix stuff
[14:48:42] <petan|wk>	 or something like that
[14:48:46] <mutante>	 aha!
[14:48:47] <petan|wk>	 just ls /etc/init.d
[14:48:52] <petan|wk>	 you will see
[14:49:13] <petan|wk>	 I don't know if that' name of bot
[14:49:22] <mutante>	 "adminbot" :) i looked for "bot-*" :)
[14:49:23] <mutante>	 there we go
[14:49:28] <petan|wk>	 ok
[14:49:32] <petan|wk>	 but we need to move it
[14:49:35] <petan|wk>	 not restart it :D
[14:49:40] <petan|wk>	 because it's gonna die soon anyway
[14:49:41] <mutante>	 !log bots - restarted adminbot 
[14:49:45] <mutante>	 heh
[14:49:57] <mutante>	 good demo:)
[14:50:10] <petan|wk>	 oh it died because of message?
[14:50:17] <petan|wk>	 haha
[14:50:18] <mutante>	 not sure
[14:50:32] <mutante>	 wait a minute
[14:50:37] <petan|wk>	 it's written in python so it's natural that it crashes
[14:50:48] <DamianZ>	 pfft
[14:51:13] <labs-nagios-wm_>	 RECOVERY Free ram is now: OK on mobile-enwp i-000000ce output: OK: 52% free memory
[14:51:37] <mutante>	 !log bots - test killing labs-morebots by logging!?
[14:51:41] <mutante>	 bah:)
[14:52:00] <mutante>	 not load related
[14:52:13] <petan|wk>	 I think it was crashing anyway for many reasons
[14:58:37] <mutante>	 !log bots strace it
[15:03:44] <mutante>	 !log bots try fixing labs-morebots
[15:05:17] <mutante>	 !log bots now?
[15:05:47] <mutante>	 getting closer though:) ignore this for a little while longer
[15:07:59] <mutante>	 !log lala
[15:08:00] <labs-morebots>	 Message missing. Nothing logged.
[15:08:03] <mutante>	 :)
[15:08:36] <mutante>	 !log bots fixed labs-morebots. (mkdir /var/run/adminbot and fix permissions to write to cache dir)
[15:08:41] <mutante>	 haha, cry:)
[15:09:15] <labs-nagios-wm_>	 PROBLEM Free ram is now: WARNING on mobile-enwp i-000000ce output: Warning: 7% free memory
[15:09:41] <DamianZ>	 Yeah she should do that her self
[15:09:49] <petan|wk>	 she?
[15:09:53] <petan|wk>	 we have girls on internet now?
[15:09:54] <DamianZ>	 She
[15:10:06] <petan|wk>	 where are they
[15:10:12] <petan|wk>	 \o/
[15:11:00] <mutante>	 !log bots why die?
[15:11:04] <petan|wk>	 !log bastion we want moar girls!
[15:11:07] <petan|wk>	 damn
[15:11:22] <mutante>	 theres more permission denied
[15:11:27] * DamianZ  gives LCarr and smooth to petan|wk

[15:11:41] <petan|wk>	 LCarr is too dangerous
[15:11:49] <petan|wk>	 she could hack my pc
[15:11:49] <mutante>	 hehehe
[15:11:49] <DamianZ>	 she bites
[15:12:03] <petan|wk>	 and knows nmap
[15:14:50] <petan|wk>	 !log .
[15:14:50] <labs-morebots>	 Message missing. Nothing logged.
[15:14:54] <petan|wk>	 !log bots
[15:14:55] <labs-morebots>	 Message missing. Nothing logged.
[15:15:00] <petan|wk>	 !log blah blah
[15:15:00] <labs-morebots>	 Can't contact LDAP for project list.
[15:15:00] <labs-morebots>	 blah is not a valid project.
[15:15:12] <petan|wk>	 !log
[15:15:16] <petan|wk>	 !log blah blah
[15:15:17] <labs-morebots>	 blah is not a valid project.
[15:15:22] <petan|wk>	 !log bots test
[15:15:22] <labs-morebots>	 bots is not a valid project.
[15:15:27] <petan|wk>	 here we go
[15:15:28] <mutante>	 !log bots fixed labs-morebots. (mkdir /var/run/adminbot and fix permissions to write to cache dir)
[15:15:29] <labs-morebots>	 bots is not a valid project.
[15:15:34] <petan|wk>	 :D
[15:15:40] <petan|wk>	 I think you should test it before you type fixed
[15:15:53] <mutante>	 its the same thing in this case, if it works its ok:)
[15:15:56] <petan|wk>	 but on other hand once it get logged it's fixed
[15:16:10] <mutante>	 well, it's not crashing anymore! wee
[15:16:28] <petan|wk>	 haha, if that is success in world of python, then well
[15:16:33] <mutante>	 now i did not expect another problem with project names at all
[15:16:41] <mutante>	 did that happen before?
[15:16:45] <petan|wk>	 no
[15:16:50] <petan|wk>	 it worked before :-)
[15:18:46] <mutante>	 !log wikistats test
[15:18:46] <labs-morebots>	 wikistats is not a valid project.
[15:19:20] <mutante>	 we need a file project.cache from somewhere
[15:19:29] <petan|wk>	 <labs-morebots> Can't contact LDAP for project list.
[15:19:37] <petan|wk>	 mutante: did you overlook it?
[15:21:30] <mutante>	 hrmm, and a whole 'nother story pops up after fixing one, hi murphy:)
[15:21:33] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours
[15:22:45] <labs-nagios-wm_>	 PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 7.23, 6.79, 5.72
[15:44:07] <hashar>	 petan|wk: I have merged https://gerrit.wikimedia.org/r/5323  "group memcached debug logs"
[15:44:42] <hashar>	 so you can split the $wgMemCachedDebug logs by using:    $wgDebugLogGroups['memcached'] = '/some/path/memcached-debug.log';
[15:44:54] <petan|wk>	 great
[15:45:08] <petan|wk>	 but still how do I filter out my session from logs?
[15:45:29] <petan|wk>	 I can use the token workaround for now
[16:28:28] <labs-nagios-wm_>	 PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 5.24, 19.68, 11.50
[16:43:21] <labs-nagios-wm_>	 RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.54, 1.45, 4.72
[16:48:21] <labs-nagios-wm_>	 PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 977 MB (5% inode=82%): /home/SAVE 977 MB (5% inode=82%): /home/jgreen 977 MB (5% inode=82%):
[16:54:01] <labs-nagios-wm_>	 PROBLEM host: pediapress-ocg3 is DOWN address: i-000001be CRITICAL - Host Unreachable (i-000001be)
[16:58:05] <mutante>	 petan|wk: fyi, the LDAP issue is probably related to another unrelated issue with the NFS server, so not related to the bot itself or the bots instance having load
[16:58:20] <mutante>	 got in a 1:1 in between, but Jeff_Green just noticed that NFS issue
[16:59:19] <mutante>	 we are not touching it anymore at this second for fear of data loss.. but it will be checked
[17:04:15] <labs-nagios-wm_>	 RECOVERY host: pediapress-ocg3 is UP address: i-000001be PING OK - Packet loss = 0%, RTA = 2.93 ms
[17:17:04] <petan|wk>	 mutante: ok
[17:42:12] <mutante>	 !log bots foo
[17:42:12] <labs-morebots>	 Can't contact LDAP for project list.
[17:42:12] <labs-morebots>	 bots is not a valid project.
[18:17:48] <gerrit-wm>	 New patchset: Sara; "Still another iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5340
[18:18:01] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5340
[18:34:26] <mark>	 !ryan
[18:34:47] <DamianZ>	 Ryan is really pink and pokadotty
[18:34:56] <mutante>	 !Ryan
[18:34:56] <wm-bot>	 man of all answers ever (but there are others :))
[19:04:14] <labs-home-wm_>	 04/19/2012 - 19:04:13 - Creating a home directory for robih at /export/home/bastion/robih
[19:05:05] <labs-nagios-wm_>	 RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 5.16, 4.51, 4.95
[19:05:14] <labs-home-wm_>	 04/19/2012 - 19:05:13 - Updating keys for robih
[19:12:20] <Jarry1250>	 Incidentally, is there an SVG/high res labs logo hanging around somewhere that could go onto Commons?
[20:51:49] <gerrit-wm>	 New review: Sara; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/5340
[20:51:52] <gerrit-wm>	 Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5340
[20:56:11] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours
[20:59:27] <gerrit-wm>	 New patchset: Sara; "Minor iteration for adding ganglia to labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5357
[20:59:40] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5357
[20:59:56] <gerrit-wm>	 New review: Sara; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/5357
[20:59:59] <gerrit-wm>	 Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5357
[21:00:11] <labs-nagios-wm_>	 PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours
[21:11:58] <gerrit-wm>	 New patchset: Sara; "Another minor iteration for adding ganglia to labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5359
[21:12:11] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5359
[21:14:12] <gerrit-wm>	 New review: Sara; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/5359
[21:14:59] <gerrit-wm>	 Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5359
[21:33:21] <labs-nagios-wm_>	 PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d output: DISK CRITICAL - free space: /export 480 MB (2% inode=82%): /home/SAVE 480 MB (2% inode=82%):