[01:08:52] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 182 processes
[01:13:52] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 106 processes
[01:36:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory
[01:46:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 23% free memory
[03:12:53] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: Critical: 5% free memory
[03:25:08] <gerrit-wm>	 New patchset: Hydriz; "Adding the memcached layer for the Incubator project." [labs/incubator] (master) - https://gerrit.wikimedia.org/r/27387
[03:30:14] <gerrit-wm>	 New patchset: Hydriz; "Adding the memcached layer for the Incubator project." [labs/incubator] (master) - https://gerrit.wikimedia.org/r/27387
[03:32:17] <Hazard-SJ>	 Hydriz: Any reason why the bot repeated itself?
[03:32:59] <Hydriz>	 yes, I submitted a second patch
[03:33:05] <Hydriz>	 I forgot to add a file
[03:36:29] <Hydriz>	 right, all is good, now to merge and deploy
[03:36:40] <gerrit-wm>	 Change merged: Hydriz; [labs/incubator] (master) - https://gerrit.wikimedia.org/r/27387
[04:02:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 16% free memory
[04:23:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 19% free memory
[04:27:12] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory
[04:32:13] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory
[04:56:22] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK CRITICAL - free space: /export 421 MB (2% inode=59%): /home/SAVE 421 MB (2% inode=59%):
[05:41:42] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory
[05:49:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 17% free memory
[05:51:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory
[05:56:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 31% free memory
[06:14:43] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory
[06:41:10] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Y109 link https://www.mediawiki.org/w/index.php?diff=592131 edit summary: 
[07:19:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[07:43:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory
[08:51:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory
[08:51:39] <hashar>	 lo
[08:56:19] <Platonides>	 morning, hashar
[09:11:13] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 41% free memory
[09:37:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on dumps-bot3 i-000003ef.pmtpa.wmflabs output: OK: 46% free memory
[09:44:22] <labs-logs-bottie>	 !log bots petrb: rebooting nr1
[09:44:24] <labs-morebots>	 Logged the message, Master
[09:53:52] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory
[09:59:13] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[10:54:46] <labs-logs-bottie>	 !log bots root: patched wm-bot
[10:54:47] <labs-morebots>	 Logged the message, Master
[10:54:53] <petan>	 @help
[10:54:53] <wm-bot>	 Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.24.5 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot
[10:57:44] <petan>	 @commands
[10:57:44] <wm-bot>	 Commands: there is too many commands to display on one line, see http://meta.wikimedia.org/wiki/wm-bot for a list of commands and help
[10:57:46] <petan>	 !ping
[10:57:46] <wm-bot>	 pong
[10:57:46] <petan>	 this is a problem
[10:58:07] <petan>	 !ping
[10:58:07] <wm-bot>	 pong
[10:58:10] <petan>	 !ping
[10:58:10] <wm-bot>	 pong
[10:58:13] <petan>	 quite slow
[10:58:18] <petan>	 !ping
[10:58:18] <wm-bot>	 pong
[11:03:32] <petan>	 @help
[11:03:32] <wm-bot>	 Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.24.10 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot
[11:06:04] <petan>	 @configure style-rss=[$name]: $title ($author) $link $description
[11:06:04] <wm-bot>	 Value [$name]: $title ($author) $link $description was stored into style-rss to config
[11:06:59] <wm-bot>	 [bugzilla]: [Bug 40895] This is a test bug, ignore me (Peter Bena) https://bugzilla.wikimedia.org/show_bug.cgi?id=40895 
      
      <table>
      <tr>
        <th>Field</th><th>Value</th>
      </tr><tr class="bz_feed_product">
        <td>Product</td>
        <td>Wikimedia Labs</td>
      </tr><tr class="bz_feed_component">
        <td>Component</td>
        <td>General</td>
      </tr><tr class="bz_feed_assignee">
        <td>Assignee</td>
        <td>Nobody</td>
      </tr><tr class="bz_feed_reporter">
        <td>Reporter</td>
        <td>Peter Bena</td>
      </tr><tr class="bz_feed_bug_status">
        <td>Status</td>
        <td>RESOLVED</td>
      </tr><tr class="bz_feed_resolution">
        <td>Resolution </td>
        <td>FIXED</td>
      </tr><tr class="bz_feed_priority">
        <td>Priority</td>
        <td>Unprioritized</td>
      </tr><tr class="bz_feed_severity">
        <td>Severity </td>
        <td>minor</td>
      </tr><tr class="bz_feed_target_milestone">
        <td>Target Milestone</td>
        <td>---</td>
      </tr><tr class="bz_feed_creation_date">
        <td>Creation date</td>
        <td>Tue 15:53</td>
      </tr><tr class="bz_feed_changed_date">
        <td>Last changed date</td>
        <td>11:06:53</td>
      </tr>
      </table>
    
[11:10:26] <petan>	 really would like to see the traffic...
[11:11:21] <petan>	 !ping
[11:11:21] <wm-bot>	 pong
[11:11:45] <petan>	 !ping
[11:11:45] <wm-bot>	 pong
[11:11:46] <petan>	 ..
[11:11:46] <petan>	 freenode sucks
[11:11:53] <petan>	 !ping
[11:11:54] <wm-bot>	 pong
[11:11:54] <petan>	 !ping
[11:11:55] <wm-bot>	 pong
[11:11:58] <petan>	 o.o
[11:11:59] <petan>	 !ping
[11:11:59] <wm-bot>	 pong
[11:12:18] <wm-bot>	 [bugzilla]: [Bug 40895] This is a test bug, ignore me (Peter Bena) https://bugzilla.wikimedia.org/show_bug.cgi?id=40895 
      
      <table>
      <tr>
        <th>Field</th><th>Value</th>
      </tr><tr class="bz_feed_product">
        <td>Product</td>
        <td>Wikimedia Labs</td>
      </tr><tr class="bz_feed_component">
        <td>Component</td>
        <td>General</td>
      </tr><tr class="bz_feed_assignee">
        <td>Assignee</td>
        <td>Nobody</td>
      </tr><tr class="bz_feed_reporter">
        <td>Reporter</td>
        <td>Peter Bena</td>
      </tr><tr class="bz_feed_bug_status">
        <td>Status</td>
        <td>RESOLVED</td>
      </tr><tr class="bz_feed_resolution">
        <td>Resolution </td>
        <td>FIXED</td>
      </tr><tr class="bz_feed_priority">
        <td>Priority</td>
        <td>Unprioritized</td>
      </tr><tr class="bz_feed_severity">
        <td>Severity </td>
        <td>minor</td>
      </tr><tr class="bz_feed_target_milestone">
        <td>Target Milestone</td>
        <td>---</td>
      </tr><tr class="bz_feed_creation_date">
        <td>Creation date</td>
        <td>Tue 15:53</td>
      </tr><tr class="bz_feed_changed_date">
        <td>Last changed date</td>
        <td>11:12:08</td>
      </tr>
      </table>
    
[11:28:23] <petan>	 !ping
[11:28:23] <wm-bot>	 pong
[11:28:25] <petan>	 this is creepy
[11:35:40] <wm-bot>	 [bugzilla]: [Bug 40895] This is a test bug, ignore me (Peter Bena) https://bugzilla.wikimedia.org/show_bug.cgi?id=40895               <table>       <tr>         <th>Field</th><th>Value</th>       </tr><tr class="bz_feed_product">         <td>Product</td>         <td>Wikimedia Labs</td>       </tr><tr class="bz_feed_com
[11:37:13] <petan>	 @configure style-rss=[$name]: $title ($author) $link
[11:37:14] <wm-bot>	 Value [$name]: $title ($author) $link was stored into style-rss to config
[11:37:19] <petan>	 !ping
[11:37:19] <petan>	 o.o
[11:37:19] <wm-bot>	 pong
[11:37:39] <wm-bot>	 [bugzilla]: [Bug 40895] This is a test bug, ignore me (Peter Bena) https://bugzilla.wikimedia.org/show_bug.cgi?id=40895
[11:37:52] <petan>	 hmm
[11:37:58] <petan>	 let's consider it working
[11:38:13] <petan>	 hashar is there anything you would improve?
[11:42:20] <petan>	 !petan
[11:42:20] <wm-bot>	 Petr Bena - http://enwp.org/User:Petrb
[11:42:22] <petan>	 !ping
[11:42:22] <wm-bot>	 pong
[11:43:49] <petan>	 @configure style-rss=[$name]: 2$title (4$author) $link
[11:43:50] <wm-bot>	 Value [$name]: 2$title (4$author) $link was stored into style-rss to config
[11:43:56] <petan>	 @configure style-rss=[$name]: 3$title (4$author) $link
[11:43:56] <wm-bot>	 Value [$name]: 3$title (4$author) $link was stored into style-rss to config
[11:54:25] <hashar>	 petan: busy today sorry
[11:54:37] <petan>	 ok
[11:54:38] <petan>	 :D
[12:15:49] <petan>	 @seen ^demon
[12:15:49] <wm-bot>	 petan: Last time I saw ^demon they were quiting the network N/A at 10/9/2012 9:50:29 PM (14:25:19.8619730 ago)
[12:16:01] <Damianz>	 petan: It will look ugly until all the names get switched as they are mixed currently. The switch is in SAL because nparser broke due to a format change in the JSON as you wern't really parsing the JSON. Still a few bits to fix on nagios but I'm in Ireland until thursday so won't get to it.
[12:16:11] <Damianz>	 Jan_Luca Could do, it's a bit of a pain.
[12:16:43] <petan>	 Damianz: how mixed?
[12:16:44] <Damianz>	 petan: Define sniff? If you're using sockets sure, if you're using IPC with shared memory hell.
[12:17:01] <petan>	 Damianz don't need this now
[12:17:04] <petan>	 back to nagios :)
[12:17:12] <Damianz>	 petan: We're suppose to be moving from i-xxxx.<region>.wmflabs to <name>.<region>.wmflabs.
[12:17:26] <petan>	 ok, but still JSON provides both host and name
[12:17:34] <petan>	 why you don't parse fancy name too
[12:17:35] <Damianz>	 Since they changed the naming in nova, it's named stupidly at the moment as we now have multiple regions that conflict naming wise.
[12:17:49] <petan>	 no wonder nparser broke then
[12:17:57] <petan>	 I have stuff changing meh
[12:17:59] <petan>	 * hate
[12:18:02] <Damianz>	 heh
[12:18:06] <Damianz>	 Yeah, it's a right pain.
[12:18:30] <Damianz>	 Like 'bots-1' can exist in 2 regions now, so until we move to change all the fqdns it sucks.
[12:18:39] <petan>	 do you really want to maintain your new script or I should fix nparser? because I don't understand python, but if you are going to maintain it, then it's fine
[12:18:43] <Damianz>	 New instances are named correctly, we just havn't re-named old ones.
[12:19:00] <petan>	 also, do groups work
[12:19:15] <petan>	 it does :)
[12:19:37] <Damianz>	 I was going to maintain it as it seems easier long term for me, could fix nparser if needed. I prefer a real template over the mangling fetch.sh does.
[12:19:54] <Damianz>	 And groups should work, apart from project groups there's only like web and db I think currently.
[12:30:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[12:35:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 23% free memory
[13:06:45] <aude>	 hi paravoid and mutante 
[13:06:56] <aude>	 do you have gerrit superpowers?
[13:07:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[13:08:09] <aude>	 Reedy: hashar do you have superpowers?
[13:08:38] <hashar>	 aude: what do you mean?
[13:08:45] <aude>	 https://gerrit.wikimedia.org/r/#/admin/groups/32,members
[13:08:51] <aude>	 we need to add a team member to our group
[13:09:05] <aude>	 Silke_WMDE_ needs to be added
[13:11:20] <aude>	 maybe it's just these people who have superpowers: https://gerrit.wikimedia.org/r/#/admin/groups/1,members ?
[13:11:33] <aude>	 also ops people
[13:12:02] <aude>	 and https://gerrit.wikimedia.org/r/#/admin/groups/119,members
[13:12:32] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes
[13:13:23] <hashar>	 aude: I can do it
[13:13:31] <aude>	 hashar: yay! :)
[13:13:33] <hashar>	 aude: did Silke_WMDE_ logged on labs already?
[13:13:42] <aude>	 yes
[13:13:42] <hashar>	 yeah
[13:13:48] <hashar>	 aude: {{done}} :-]
[13:13:55] <aude>	 happy! :)
[13:14:06] <aude>	 thank you!
[13:14:34] <hashar>	 aude the admins are indeed "Project and Group Creators" : https://gerrit.wikimedia.org/r/#/admin/groups/119,members
[13:14:55] <hashar>	 and admins of course :-)
[13:15:30] <aude>	 hashar: ok
[13:15:34] <aude>	 good to know
[13:23:44] <Silke_WMDE_>	 hashar: thx! :)
[13:24:19] <hashar>	 Silke_WMDE_: welcome in the wikidata team I guess :-]
[13:25:17] <Silke_WMDE_>	 hashar: Hehe, I've actually been there from the beginnig. Just nothing to do with git tags (and the gerrit world) until today. :)
[13:25:45] <hashar>	 I should fly to Berlin for a whole week to meet you all
[13:25:46] <hashar>	 :-D
[13:29:05] <Silke_WMDE_>	 hashar: Yeah, indeed.
[13:33:43] <Silke_WMDE_>	 hashar: And if I am in the gerrit group wikidata does that mean that I can push git tags? I keep getting an error.
[13:34:04] <Silke_WMDE_>	 This one:
[13:34:06] <Silke_WMDE_>	 To ssh://smeyer@gerrit.wikimedia.org:29418/mediawiki/extensions/Wikibase.git
[13:34:06] <Silke_WMDE_>	  ! [remote rejected] test -> test (can not create new references)
[13:34:06] <Silke_WMDE_>	  ! [remote rejected] test2 -> test2 (can not create new references)
[13:34:06] <Silke_WMDE_>	 error: failed to push some refs to 'ssh://smeyer@gerrit.wikimedia.org:29418/mediawiki/extensions/Wikibase.git'
[13:34:13] <Silke_WMDE_>	 Any idea?
[13:35:30] <hashar>	 Silke_WMDE_: seems you can't create tags
[13:35:32] <hashar>	 :/
[13:35:37] <hashar>	 need to look it up
[13:35:38] <hashar>	 give me asec
[13:36:52] <Silke_WMDE_>	 ok
[13:43:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory
[13:43:32] <Reedy>	 Permissions are indirect, but they look right
[13:47:09] <hashar>	 ah grant Create Reference for reference name refs/tags/*
[13:48:25] <hashar>	 looks like you should be able to push indeed
[13:48:34] <hashar>	 maybe it is an annotated tag
[13:48:39] <hashar>	 which uses a different permission
[14:35:35] <Platonides>	 Silke_WMDE, have you checked if the tag got created?
[14:35:59] <Silke_WMDE>	 locally, yes, it got created.
[14:35:59] <Platonides>	 it may have done so despite denying the fact
[14:36:06] <Platonides>	 I mean in the remote
[14:36:29] <Silke_WMDE>	 ah, no, I can't see it in gerrit afterwards.
[14:36:35] <Platonides>	 :(
[14:37:09] <Platonides>	 no, there's no test tag in the wikibase repo
[14:40:56] <^demon>	 Ohhhh man. We totally can't upgrade to gerrit 2.5 yet :(
[14:41:50] <RogueMadman>	 I hate to bother ops having already sent an e-mail, but that was on Saturday; would it be possible to create the cvresearch project with a public IP and add westand and myself? ._.
[14:46:32] <Hydriz>	 RogueMadman: I would suggest waiting for a guy called andrewbogott to help you create the project. He is currently AFK, so try again later :)
[14:47:18] <RogueMadman>	 Will do. Much appreciated. :)
[14:56:17] <Jan_Luca>	 Hydriz: Hi
[14:56:25] <Hydriz>	 Ohai Jan
[14:56:40] <Hydriz>	 Gave you the dev bit on the beta cluster, btw :)
[14:59:02] <Jan_Luca>	 Did you already test the centralauth-project
[15:00:03] <Hydriz>	 hmm, not yet for now, I am trying to fix up CentralAuth on another project :P
[15:01:55] <Jan_Luca>	 Oh, there is db error for centralauth.wmflabs.org
[15:02:13] <Hydriz>	 yeah, ContentHandler
[15:02:34] <Jan_Luca>	 no, git update and forgot to run update.php
[15:02:53] <Jan_Luca>	 I should add this to the cron job ...
[15:03:07] <Hydriz>	 cron job?
[15:03:36] <Jan_Luca>	 I add a cron job for updating MW core from Gerrit
[15:04:11] * Damianz  bounces up and down

[15:04:36] <Damianz>	 <3 512kb/s wifi
[15:05:14] <RogueMadman>	 <3
[15:05:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[15:05:59] <Damianz>	 You know what could be cool rather than using cronjobs? If we actually told gerrit to replicate the repos to a read only share and just symlinked that on beta/other stuff wanting latest mw.
[15:08:30] <wm-bot>	 [bugzilla]: 3[Bug 40819] Config viewer is a 404 (4Damian Z) https://bugzilla.wikimedia.org/show_bug.cgi?id=40819
[15:08:36] * Damianz  sends hashy some spam

[15:08:49] <Damianz>	 wm-bot: oh hai 
[15:08:49] <wm-bot>	 Hi Damianz, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-)
[15:09:14] <Jan_Luca>	 Hydriz: Do you need any global rights?
[15:09:22] <Hydriz>	 phew, spent the whole day debugging why variables like wmgUseLastModified doesn't work in $wgConf
[15:09:39] <Hydriz>	 Jan: Would be great, though I can modify it later :P
[15:09:51] <Damianz>	 lol
[15:10:12] <Hydriz>	 yeah, just one goddamn line: extract ($globals);
[15:10:42] <Damianz>	 I'm half tempted to clean up the mess that is the common config file but a) I have no idea what bizzare hacks are still needed and b) don't really want to take the site down :P
[15:10:52] <Damianz>	 Reading it is like stabbing yourself with a spoon 
[15:11:00] <Hydriz>	 I just did haha
[15:11:01] <Jan_Luca>	 Hydriz: Do know the MySQL pass, it is not blank :P
[15:11:26] <Hydriz>	 Jan: Sure, thanks for setting up the project though! :)
[15:11:39] <Hydriz>	 Damianz: I was hoping that we can arrange stuff alphabetically
[15:11:39] <Damianz>	 Jan_Luca: probably 'puppet'
[15:12:05] <Damianz>	 Hydriz: Or just fix the random formatting and comment blocks :)
[15:12:19] <Hydriz>	 hmm, true about random formatting
[15:12:21] <Damianz>	 I was thinking defaults followed by cluster specific settings.
[15:12:32] <Hydriz>	 I see a lot of include(blah) and require(blah) and require_once(blah)
[15:12:45] <Damianz>	 and lined up ='s, not lined up ='s etc
[15:13:11] <Hydriz>	 Not sure if the labs/incubator repository can be a good inspiration for you? :P
[15:13:37] <Damianz>	 I really hate there's so many cluster/cluster specific blocks... there should just be 1, at the end!
[15:13:39] <Hydriz>	 I am trying to keep things nice, though its sadly broken haha
[15:14:01] <Damianz>	 s/cluster/wiki/
[15:14:11] <Damianz>	 Well db backend since by wiki would make too much sense.
[15:14:44] <Damianz>	 oh also
[15:14:46] <Damianz>	 08:16 < Damianz> Jan_Luca Could do, it's a bit of a pain.
[15:14:53] <Damianz>	 In relation to ip request form.
[15:15:05] <Hydriz>	 hmm, SiteMatrix seems to be ordered randomly (by project)
[15:15:10] <Damianz>	 Imo if we're going down that route, we should just have 1 request queue for everything.
[15:15:31] <Hydriz>	 why is news coming before quote?
[15:15:41] <Damianz>	 Magic
[15:15:51] <Hydriz>	 though other than wikt and w, the others are ordered alphabetically
[15:16:02] <Hydriz>	 but quote came first bah
[15:16:27] <Hydriz>	 oh good, there is space for travel too
[15:16:30] <Jan_Luca>	 Damianz: Thank you and yes, there should be 1 request queue
[15:16:42] <Damianz>	 On the bright side the config file doesn't come with tests, so break all the things!
[15:17:10] <Hydriz>	 lint tests?
[15:17:36] <Jan_Luca>	 Hydriz: You have the rights. Btw, I have created a Gerrit repo for the project: https://gerrit.wikimedia.org/r/#/admin/projects/labs/centralauth,branches
[15:17:46] <Hydriz>	 yep, saw it :)
[15:17:54] <Jan_Luca>	 The config is in the shared-wiki-config branch
[15:18:11] <Hydriz>	 heh, not master
[15:18:33] <Hydriz>	 is it merge review, or open push?
[15:18:42] <Hydriz>	 ...and is it immediately deployed?
[15:18:47] <Jan_Luca>	 At moment merge review
[15:19:57] <Jan_Luca>	 I want to use cron (or puppet when I learned it ;-) ) to download it from Gerrit. That is the reason for using branches, too
[15:20:17] <Hydriz>	 bloody hell: PHP Notice:  Undefined index: format in /data/project/public/w/includes/Import.php on line 664
[15:20:42] <Hydriz>	 there is some recent commit that broke the hell of importDump.php
[15:24:06] <Jan_Luca>	 Now, let's broke everything: First pull of the config in Gerrit ...
[15:25:19] <Hydriz>	 Oh yes, Jan: Can you try if you can use importDump.php successfully or something?
[15:25:32] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes
[15:26:29] <Jan_Luca>	 Do you have a dump for testing?
[15:26:53] <Hydriz>	 there are lots in /public/datasets
[15:27:15] <Hydriz>	 go for aawiki or something, and on a DB that can be easily reverted on
[15:31:14] <Hydriz>	 Jan: o_O you are using the method of wgConf stated in the wiki, its different from Wikimeda's, if you are trying to follow them :P
[15:32:07] <Jan_Luca>	 The project is for developping CentralAuth in general but where are the diffs?
[15:32:43] <Hydriz>	 hmm, the diff is more of the language and site of the wiki or something
[15:33:01] <Hydriz>	 its tailored for things like $site and $lang, if you understand what it is :P
[15:34:41] <Jan_Luca>	 You love ":P" ...
[15:35:12] <Hydriz>	 lol
[15:36:46] <Jan_Luca>	 Hydriz: No importDump.php does not work
[15:36:54] <Hydriz>	 you get the same error?
[15:37:17] <Hydriz>	 I have a feeling its ContentHandler, its breaking too many things
[15:37:22] <Jan_Luca>	 yes
[15:39:37] <Damianz>	 Hydriz: Imo linting should be done before even committing. The only real way to test your config isn't screwed is to go provision a cluster to a spec you expect and watertest/mock against that to check it's really working. Sucks but hey this is why we have staging.
[15:46:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on hume i-000003cc.pmtpa.wmflabs output: DISK CRITICAL - free space: / 23 MB (1% inode=45%):
[15:51:13] <labs-nagios-wm>	 PROBLEM Disk Space is now: WARNING on hume i-000003cc.pmtpa.wmflabs output: DISK WARNING - free space: / 51 MB (3% inode=45%):
[15:53:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 27% free memory
[16:11:26] <Jan_Luca>	 Is there no support in Git for gluster?
[16:12:13] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[16:19:39] <Jan_Luca>	 !log centralauth Use config files from Gerrit now: https://gerrit.wikimedia.org/r/gitweb?p=labs%2Fcentralauth.git;a=shortlog;h=refs%2Fheads%2Fshared-wiki-config
[16:19:46] <Jan_Luca>	 !log centralauth Use config files from Gerrit now: https://gerrit.wikimedia.org/r/gitweb?p=labs%2Fcentralauth.git;a=shortlog;h=refs%2Fheads%2Fshared-wiki-config
[16:20:12] <Jan_Luca>	 !log centralauth Use config files from Gerrit now: gerrit.wikimedia.org/r/gitweb?p=labs%2Fcentralauth.git;a=shortlog;h=refs%2Fheads%2Fshared-wiki-config
[16:20:13] <labs-morebots>	 Logged the message, Master
[16:27:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 27% free memory
[16:45:27] <Damianz>	 support in Git for gluster?
[16:45:36] <Damianz>	 it shouldn't really care, the only thing that doesn't work is mysql
[16:52:57] <andrewbogott>	 Damainz:  I've had some pretty crazy git behaviors lately which I suspect are due to the file being on a gluster share.  I'm not certain that gluster is to blame though.
[16:53:37] <andrewbogott>	 It /mostly/ works in any case.
[16:58:51] <^demon>	 Damianz: git won't care.
[16:59:34] <Damianz>	 That was in reply to Jan_Luca :P
[16:59:47] <^demon>	 Oh, missed that :)
[16:59:49] <Damianz>	 Git will happily work on a totally failing disk that's randomly loosing data
[18:15:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[18:16:09] <Jan_Luca>	 Damianz: Thank you for your answer, I forgot to write that I found the cause (wrong command).
[18:17:32] <Damianz>	 that would probably cause an un-expected result
[18:18:57] <Jan_Luca>	 I tried to use "git pull" insteed of "git clone" and there was a message about /data/... so I thought the first moment that this is a problem with gluster
[18:19:12] <sumanah>	 hi Jan_Luca 
[18:19:16] <Jan_Luca>	 Hi
[18:20:09] <Damianz>	 I assume it said HEY, DATA IS NOT A REPO, GO AWAY (or point the ref somewhere else, because you know git is cool and can store .git outside of the data)
[18:20:26] <sumanah>	 Jan_Luca: I realized I should point you to https://www.mediawiki.org/wiki/Testing_portal in case that would be helpful to you, but you may have already seen it ....
[18:21:58] <Jan_Luca>	 The messages was: fatal: Not a git repository (or any parent up to mount parent /data)
[18:22:00] <Jan_Luca>	 Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[18:22:19] <Damianz>	 mhm
[18:22:28] <Damianz>	 'repo or gtfo'
[18:22:56] <Jan_Luca>	 The command was: git pull https://gerrit.wikimedia.org/r/p/labs/centralauth.git
[18:22:58] <^demon|lunch>	 What a completely unhelpful error message.
[18:23:00] <^demon|lunch>	 Thank you git.
[18:23:46] <Jan_Luca>	 When I looked at the command the second time I saw that this must be "clone"
[18:23:53] <^demon|lunch>	 Jan_Luca: pull requires an existing clone. A pull is really just a fetch & merge :)
[18:24:00] <Damianz>	 git isn't the most user friendly tool but it's freaking flexiable
[18:24:01] <^demon|lunch>	 Yep, clone for a fresh repo.
[18:24:14] <Damianz>	 Though as I recall svn gives some really weird errors when you break it's metadata
[18:24:40] <^demon|lunch>	 Damianz: Which is why the only way to recover a broken svn working copy is to re-checkout :(
[18:25:09] <Damianz>	 Oh it's more fun when you break it on the server and have to restore from backups
[18:25:13] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 23% free memory
[18:25:30] <Damianz>	 At least with git even if you trash the branches as long as you have the commit data you can just about put it back together.
[18:25:45] <Jan_Luca>	 ^demon|lunch: I know, but at this moment I was distracted
[18:26:01] <^demon|lunch>	 At least you figured it out :)
[18:26:03] <Damianz>	 Btw, osx sucks ass
[18:26:17] <Damianz>	 I ran a sed to replace tabs with spaces AND IT BROKE LIKE 800 files
[18:26:19] <Damianz>	 sadface
[18:26:26] <Damianz>	 stupid non-standard-gnu-toolsets
[18:26:45] <Jan_Luca>	 :-D
[18:26:48] <^demon|lunch>	 We've hit that before with MW.
[18:26:52] <^demon|lunch>	 Actually, it was with grep.
[18:27:03] <^demon|lunch>	 grep -e instead of grep -P
[18:27:12] <Damianz>	 On the bright side a simple find+while+mv == restored back
[18:27:13] <Damianz>	 And I think it's dinner time
[18:27:32] <Damianz>	 Oh grep can be fussy too
[18:28:20] <^demon|lunch>	 Why it uses -e for "pattern" instead of -P I've never figured out.
[18:29:00] <Damianz>	 Probably along the same reason egrep exists with a man page that says 'use grep -E'
[18:30:51] <^demon|lunch>	 On OSX? I can't find that.
[18:31:33] <^demon|lunch>	 Meh, I use ack for source code anyway :p
[18:32:03] <Damianz>	 Hmm I have it on here, wouldn't surprise me if some random thing installed it though... could do with standardising my laptop install.
[18:32:54] <Damianz>	 Terminal on osx really needs to support left/right arrow keys properly without patching the config
[18:35:01] <Jan_Luca>	 Does anyone know why "git pull origin master" needs so much time for MW-core?
[18:35:02] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[18:35:42] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[18:37:18] <^demon>	 Jan_Luca: Huge branch was just merged, repacks probably aren't efficient at the moment.
[18:37:37] <Jan_Luca>	 Ok, thank you
[18:37:52] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[18:38:12] <labs-nagios-wm>	 PROBLEM SSH is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Server answer:
[18:38:32] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[18:38:52] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[18:49:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[18:53:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS WARNING: 198 processes
[18:54:52] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on worker1 i-000004b4.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages
[18:59:53] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on worker1 i-000004b4.pmtpa.wmflabs output: All packages OK
[19:04:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[19:07:52] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: DISK OK
[19:08:12] <labs-nagios-wm>	 RECOVERY SSH is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[19:08:32] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: All packages OK
[19:08:52] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: PROCS OK: 125 processes
[19:08:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 40% free memory
[19:10:43] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[19:10:53] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: WARNING - load average: 0.35, 3.42, 5.39
[19:12:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on worker1 i-000004b4.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages
[19:15:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.33, 1.46, 3.99
[19:34:42] <Damianz>	 mmm steak
[19:34:53] <Damianz>	 Byte services
[19:34:57] <Damianz>	 s/te/e/
[19:39:44] * Damianz  tries to figure out webplatform vs stackoverflow and decides he still doesn't understand so but it has a nicer interface for collaboration.

[19:48:47] <Damianz>	 'Please select a rating for the seller based on these questions; Item arrived by 8 October 2012?' Options; 5 (Excellent), 4 (Good), 3 (Fair), 2 (Poor) or 1 (Awful)
[19:48:50] <Damianz>	 That's clearly a yes or no question amazon....
[19:50:12] <^demon>	 5 is on time or early.
[19:50:14] <^demon>	 4 is a day late.
[19:50:22] <^demon>	 3 is 2-3 days late.
[19:50:24] <^demon>	 So on :p
[19:51:18] <Damianz>	 On that scale I think 'Awful' would have to be 'never turned up'... this is why star ratings suck. Like YAY 5 stars.. wait it has 1 rating.
[19:52:34] <^demon>	 Damianz: Also, http://xkcd.com/1098/ :)
[19:53:05] <^demon>	 hashar: Building 2.4 is going to be a pain in the rear...the stable-2.4 branch is missing ~5 commits that are in the 2.4.2 tag :(
[19:53:58] <Damianz>	 Yep - also there's the human factor that if it's got a rating of 5 you feel like an ass rating it with 1 so will probably rate it around 3 instead.
[19:55:29] <^demon>	 hashar: Granted, Shawn conceded that the LDAP problem needs fixing, so hopefully that'll make it into stable-2.5? *sigh*
[19:56:26] <hashar>	 ^demon: doooh
[19:56:39] <hashar>	 ^demon: good to know the LDAP regression is considered as … a regression
[19:56:40] <Damianz>	 How do they even work versioning? a 2.x release every x months, 2.x.y release for bugs and an x.0.0 release when they feel bored?
[19:56:53] <^demon>	 Damianz: Something like that.
[19:56:58] <hashar>	 ^demon: can't we build from 2.4.2 and cherry pick the commit we need ?
[19:57:06] <hashar>	 ^demon: I think that is what openstack has done
[19:57:12] <^demon>	 I could try that.
[19:57:24] <^demon>	 I was hoping to work from stable-2.4 :)
[19:58:24] <Damianz>	 You know you secretly want to fork it and maintain and wikimedia branch long term :D
[19:58:25] <hashar>	 you also might want to cherry pick the commits from 2.4.2 and submit them to stable-2.4
[19:58:43] <hashar>	 so they can release a 2.4.3 security issue without missing the few commits in between
[20:04:33] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes
[20:06:32] <^demon>	 hashar: I tried to push them to refs/for/stable-2.4 but upstream told me "No new changes" :(
[20:06:36] <^demon>	 Pestering on repo-discuss.
[20:07:07] <hashar>	 ^demon: even if it is in 2.4.2 ? ???
[20:07:28] <hashar>	 ^demon: does not make any sense to me but they might use a different scheme and consider 2.4.2 to be the branch
[20:07:36] <^demon>	 I'm thinking 2.4.2 got tagged from master.
[20:07:42] <hashar>	 so a 2.4.3 might be relased by "forking" 2.4.3
[20:07:50] <hashar>	 ah or that
[20:08:08] <Ryan_Lane>	 ^demon: I thought there was a workaround for the ldap issue
[20:08:32] <hashar>	 so I would get 2.4.2 and cherry-pick the needed commit, push that to the WMF repo and use that.
[20:09:34] <^demon>	 Ryan_Lane: That was a different issue. This is "Internal groups can't inherit members from external groups"
[20:09:40] <^demon>	 Which I considered a regression.
[20:09:57] <Ryan_Lane>	 do we do that right now?
[20:10:03] <^demon>	 Yes, in several places.
[20:10:06] <Ryan_Lane>	 really?
[20:10:09] <Ryan_Lane>	 it doesn't work
[20:10:16] <^demon>	 The groups inherit.
[20:10:25] <Ryan_Lane>	 just cant add them as reviewers?
[20:10:25] <^demon>	 You just can't use them for suggestions (that's always been broken)
[20:10:27] <Ryan_Lane>	 ah
[20:11:00] <^demon>	 Case in point: Administrators inherits ops.
[20:11:31] <Ryan_Lane>	 ah
[20:11:31] <Ryan_Lane>	 right
[20:12:04] <^demon>	 So, upstream has conceded that this needs fixing, but I don't know when that'll be.
[20:12:10] <^demon>	 Definitely don't want to upgrade without it fixed.
[20:13:38] <Ryan_Lane>	 yeah
[20:13:40] <^demon>	 However, hashar needs something from stable-2.5 for his Zuul work (and we can't wait on 2.5 forever), so I'm going to build us a custom 2.4.2-wmf that we're gonna deploy between now and 2.5 :)
[20:16:00] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Arcane21 link https://www.mediawiki.org/w/index.php?diff=592333 edit summary: /* account request */ new section
[20:19:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes
[20:25:21] <hashar>	 ^demon: before I head to bed, openstack has a patch for 2.4.2. Although it does not have all the fix from the patch in 2.5, they have been using it for a few months so should be fine.
[20:25:45] <hashar>	 repo is https://review.openstack.org/p/openstack-ci/gerrit.git   commit: b5a28fb  in branch openstack-2.4.2
[20:25:56] <^demon>	 I got the 2.5 patch to merge with just a minor tweak for 2.4
[20:26:03] <hashar>	 ahh nice :-]
[20:28:43] <hashar>	 have a good night!
[20:29:45] <^demon>	 http://noc.wikimedia.org/~demon/gerrit-2.4.2-2-g4f4c36e.war :) 
[20:30:07] <^demon>	 It's already 4:30. I'll test this build out on labs tomorrow and then we can maybe deploy Friday, Ryan?
[20:30:21] <Ryan_Lane>	 sounds good to me
[20:30:35] <^demon>	 This is just 2.4.2 + hashar's patch, so impact is minimal :)
[20:33:23] <^demon>	 Put us on the deployments page on wikitech.
[20:39:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS WARNING: 158 processes
[20:45:35] <kiko>	 Ryan_Lane, yo!
[20:45:41] <Ryan_Lane>	 kiko: howdy
[20:45:49] <kiko>	 Ryan_Lane, have time for a privmsg?
[20:46:01] <Ryan_Lane>	 sure. we can likely discuss things in here too
[20:49:33] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes
[21:14:43] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS OK: 145 processes
[21:28:43] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: PROCS CRITICAL: 207 processes
[21:29:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes
[21:33:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on syslogcol-srv i-000003a9.pmtpa.wmflabs output: PROCS WARNING: 197 processes
[21:36:51] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Wtmitchell link https://www.mediawiki.org/w/index.php?diff=592354 edit summary: 
[21:38:42] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: PROCS CRITICAL: 230 processes
[21:38:42] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages
[21:43:54] <ori-l>	 hey Ryan_Lane / other lab admins -- i'd like to request a public ip allocation for editor-engagement. you can de-allocate the public ip in "phabricator", since it isn't used atm.
[21:44:18] <Ryan_Lane>	 is the ip disassociated?
[21:44:30] <Ryan_Lane>	 ori-l: you can release the ip
[21:44:31] <ori-l>	 i'm not sure. let me dissociate it if it isn't, sec
[21:44:41] <Ryan_Lane>	 we only handle the quotas
[21:44:42] <ori-l>	 i don't think i can reallocate it to another project tho
[21:44:59] <Ryan_Lane>	 if you need an ip again later in that project, you can allocate it
[21:45:09] <Ryan_Lane>	 you need to disassociate it, remove the dns entries, then unallocate it
[21:45:18] <Ryan_Lane>	 well
[21:45:20] <Ryan_Lane>	 "release" it
[21:46:14] <ori-l>	 i've dissociated it, not sure how to "unallocate" -- i only see an option to allocate additional ips
[21:48:18] <ori-l>	 editor-engagement has no more ips to allocate :/
[21:49:44] <ori-l>	 ^ Ryan_Lane
[21:50:00] <Ryan_Lane>	 there's an option to release it
[21:50:17] <Ryan_Lane>	 ori-l: editor-engagement needs to have the quota upped
[21:50:22] <Ryan_Lane>	 it has nothing to do with a lack of ip addresses
[21:50:52] <ori-l>	 yeah, i know.. the two are kind of unrelated, just hoping that by pointing out there being a spare some kind of cosmic balance is maintained
[21:51:03] <ori-l>	 it makes the request easier to digest :)
[21:51:51] <ori-l>	 can you bump the quota for ee by 1?
[21:53:23] <Ryan_Lane>	 you guys already have 3
[21:53:27] <Ryan_Lane>	 why do you need so many :(
[21:53:43] <Ryan_Lane>	 I just bumped it btw
[21:53:53] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: Connection refused by host
[21:53:59] <ori-l>	 thanks. i still want to explain, tho. so for one, editor-engagement kind of lumps a lot of sub-teams
[21:54:06] <Ryan_Lane>	 ah. right
[21:54:11] <Ryan_Lane>	 no need to explain further
[21:54:14] <Ryan_Lane>	 I forgot about that
[21:54:32] <ori-l>	 cool, thanks
[21:55:12] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: Connection refused by host
[21:55:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: Connection refused by host
[21:55:42] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on syslogcol-srv i-000003a9.pmtpa.wmflabs output: Connection refused by host
[22:00:42] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on syslogcol-srv i-000003a9.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[22:00:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on syslogcol-srv i-000003a9.pmtpa.wmflabs output: OK - load average: 1.65, 1.35, 1.17
[22:03:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on syslogcol-srv i-000003a9.pmtpa.wmflabs output: OK: 654% free memory
[22:04:32] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes
[22:05:01] <mutante>	 !log apt-get dist-upgrade on marmontel (blog)
[22:05:02] <labs-morebots>	 apt-get is not a valid project.
[22:05:11] <mutante>	 heh, wrong chan, thanks bot:)
[22:05:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on syslogcol-srv i-000003a9.pmtpa.wmflabs output: DISK OK
[22:13:42] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on syslogcol-srv i-000003a9.pmtpa.wmflabs output: All packages OK
[22:17:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on sfero i-000004b3.pmtpa.wmflabs output: DISK CRITICAL - free space: / 0 MB (0% inode=52%):
[22:26:22] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK
[22:29:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes
[23:14:02] <kiko>	 Ryan_Lane, actually, looks like I have a good grasp of the technical bits I wanted to know from the docs you provided -- thanks!
[23:14:07] <kiko>	 I'll come back if there's more 
[23:14:19] <Ryan_Lane>	 cool
[23:14:23] <Ryan_Lane>	 kiko: sounds good
[23:20:39] <ori-l>	 i keep getting "dnsdomainname: Temporary failure in name resolution", which prevents sudo, puppetd, /mnt mountpoint, etc
[23:28:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory
[23:53:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 20% free memory
[23:57:48] <Ryan_Lane>	 ori-l: oh?
[23:58:22] <ori-l>	 when running puppetd -tv (in a rootshell i sudo'd into earlier): dnsdomainname: Temporary failure in name resolution
[23:58:26] <Ryan_Lane>	 hm
[23:58:41] <ori-l>	 when trying to sudo, some error about the ldap server
[23:58:55] <ori-l>	 and the /mnt netfs isn't mounted
[23:59:19] <Ryan_Lane>	 restarting ldap on virt0
[23:59:22] <ori-l>	 all i did was "cat /dev/urandom > /etc"
[23:59:28] <ori-l>	 (j/k)
[23:59:32] <Ryan_Lane>	 heh
[23:59:49] <Ryan_Lane>	 try now