[00:57:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [01:07:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes [01:12:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 39% free memory [01:12:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 94 processes [02:31:12] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [02:39:13] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 73 MB (5% inode=51%): [02:39:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:57:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:29:22] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK CRITICAL - free space: /export 516 MB (2% inode=59%): /home/SAVE 516 MB (2% inode=59%): [03:51:12] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [04:11:12] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 5% free memory [04:17:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [04:21:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:31:52] PROBLEM Disk Space is now: WARNING on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK WARNING - free space: / 569 MB (5% inode=91%): [04:47:53] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 41% free memory [05:36:14] 10/13/2012 - 05:36:14 - Updating keys for devunt at /export/keys/devunt [06:03:52] PROBLEM Current Load is now: CRITICAL on cvresearch-bots i-000004bf.pmtpa.wmflabs output: Connection refused by host [06:04:32] PROBLEM Current Users is now: CRITICAL on cvresearch-bots i-000004bf.pmtpa.wmflabs output: Connection refused by host [06:05:13] PROBLEM Disk Space is now: CRITICAL on cvresearch-bots i-000004bf.pmtpa.wmflabs output: Connection refused by host [06:08:53] RECOVERY Current Load is now: OK on cvresearch-bots i-000004bf.pmtpa.wmflabs output: OK - load average: 0.11, 0.58, 0.45 [06:09:33] RECOVERY Current Users is now: OK on cvresearch-bots i-000004bf.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [06:10:12] RECOVERY Disk Space is now: OK on cvresearch-bots i-000004bf.pmtpa.wmflabs output: DISK OK [06:49:22] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 556 MB (3% inode=59%): /home/SAVE 556 MB (3% inode=59%): [07:01:52] RECOVERY Disk Space is now: OK on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK OK [07:46:00] 10/13/2012 - 07:45:59 - User csbot has been renamed, moving home directory in project(s): cvresearch [07:50:59] 10/13/2012 - 07:50:59 - User csbot has been renamed, moving home directory in project(s): cvresearch [08:19:22] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK CRITICAL - free space: /export 516 MB (2% inode=59%): /home/SAVE 516 MB (2% inode=59%): [08:43:30] It's ubuntu which has the option to dist upgrade, it juses uses large amounts of disk and if you drop your connection it can kill the box [08:59:52] PROBLEM host: i-000004c0.pmtpa.wmflabs is DOWN address: i-000004c0.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000004c0.pmtpa.wmflabs) [09:08:53] PROBLEM Current Load is now: CRITICAL on cvresearch-web i-000004c1.pmtpa.wmflabs output: Connection refused by host [09:09:33] PROBLEM Current Users is now: CRITICAL on cvresearch-web i-000004c1.pmtpa.wmflabs output: Connection refused by host [09:13:52] RECOVERY Current Load is now: OK on cvresearch-web i-000004c1.pmtpa.wmflabs output: OK - load average: 0.04, 0.52, 0.44 [09:14:32] RECOVERY Current Users is now: OK on cvresearch-web i-000004c1.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [09:34:14] http://packages.ubuntu.com/precise/php5-intl says it's using libicu48 [09:34:23] but on labs, it installs libicu42 for me [09:34:33] but I can't find a way to use libicu48 [09:48:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [09:58:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 20% free memory [10:24:38] any apt/dpkg expert? [13:41:31] [bz] (8RESOLVED - created by: 2Andre Klapper, priority: 4Unprioritized - 6normal) [Bug 41000] User_talk page does not display section heading icons - https://bugzilla.wikimedia.org/show_bug.cgi?id=41000 [13:47:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [14:02:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 46% free memory [14:24:37] [bz] (8NEW - created by: 2Chris McMahon, priority: 4Normal - 6normal) [Bug 35918] no preview thumbnails upon uploading video via UploadWizard - https://bugzilla.wikimedia.org/show_bug.cgi?id=35918 [14:46:14] there's like 3 bugs open for that heh [14:46:23] liangent: Define expert [14:51:32] petan, can you make the bot show resolution maybe? [14:52:02] And who modified it? [14:52:14] does the rss feed provide that? [14:52:22] (currently the bot just spits out the feed) [14:59:40] manager [15:00:05] some packs wm debs [15:00:06] * Damianz looks at liangent [15:00:23] Packaging debs not so much meh, I like rpms for packaging [15:01:22] Damianz: I want to find someone who understands why and how wmf's debs are configured in its current way... [15:01:25] [bz] (8NEW - created by: 2Krinkle, priority: 4Unprioritized - 6critical) [Bug 41004] squid001 gives "Zero Sized Reply" error on POST request - https://bugzilla.wikimedia.org/show_bug.cgi?id=41004 [15:01:47] Probably best asking in tech or ops or find someone who wrote one of the current packages and ask them directly [15:02:05] there's rough docs on how they are managed but not packaged, there's debian docs on packaging [15:03:49] [bz] (8REOPENED - created by: 2Sumana Harihareswara, priority: 4Unprioritized - 6major) [Bug 40991] "Zero Sized Reply" when saving page edit - https://bugzilla.wikimedia.org/show_bug.cgi?id=40991 [15:04:35] Damianz: where're current docs located? [15:04:46] wikitech [15:04:54] bleh I need a spanner, brb [15:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:30:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 96 processes [15:35:38] Damianz: for "how packaged" I just wonder why php5-intl depends on libicu42 instead of libicu48 [15:35:50] it's the latter case in main ubuntu repo [15:36:05] Probably because it was build on a box that has 42 installed [15:36:19] We really need to get buildbot/jenkins building rpms out of git for consistancy [15:36:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [15:41:53] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 46% free memory [16:11:12] PROBLEM Disk Space is now: WARNING on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK WARNING - free space: / 573 MB (5% inode=78%): [16:14:32] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 152 processes [16:17:17] Anyone know how to re-generate il8n files? [16:49:35] Damianz: "Where you put shit" instead of storage [16:49:56] Reedy: You're suppose to put shit in project storage! [16:50:14] Really anything 'storage' is only needed for mysql servers since gluster screws them over badly :( [16:53:39] [bz] (8ASSIGNED - created by: 2Sumana Harihareswara, priority: 4Normal - 6critical) [Bug 40991] squid001 gives "Zero Sized Reply" error on POST request - https://bugzilla.wikimedia.org/show_bug.cgi?id=40991 [16:53:40] [bz] (8NEW - created by: 2Damian Z, priority: 4Unprioritized - 6normal) [Bug 40943] Fix the instance types - https://bugzilla.wikimedia.org/show_bug.cgi?id=40943 [16:58:44] [bz] (8UNCONFIRMED - created by: 2Damian Z, priority: 4Unprioritized - 6normal) [Bug 39792] Labsconsole session is omnipresent - https://bugzilla.wikimedia.org/show_bug.cgi?id=39792 [17:00:47] awesome, serverfault is down [17:01:08] 'Server Error in '/' Application.' causes undescribed happiness due to the irony involved [17:01:12] also ewww aspx [17:08:03] [bz] (8NEW - created by: 2Damian Z, priority: 4Unprioritized - 6normal) [Bug 41008] Clear out NFS - https://bugzilla.wikimedia.org/show_bug.cgi?id=41008 [17:18:49] 'Every minute spent on discussing gender issues in technology is a wasted minute that should be better spent building free software' -- Randi Harper < rather like that [18:44:20] Change on 12mediawiki a page Developer access was modified, changed by Chuscade link https://www.mediawiki.org/w/index.php?diff=593317 edit summary: [18:46:12] RECOVERY Disk Space is now: OK on testing-arky i-0000033b.pmtpa.wmflabs output: DISK OK [18:50:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [18:54:12] PROBLEM Disk Space is now: WARNING on testing-arky i-0000033b.pmtpa.wmflabs output: DISK WARNING - free space: / 74 MB (5% inode=51%): [18:55:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [19:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [20:00:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [20:14:09] paul_irish: What're you up to? [20:15:26] paul_irish: Long time no see, haven't seen you in this channel before though. Are you working on a project in labs? [20:15:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [20:18:15] * Damianz wonders if paul is irish or paul has a second name irish but assumes he's working on webplatform st00fs [20:18:35] hey Krinkle! [20:18:35] also hi krinkly [20:18:46] working on webplatform stuff. yeah my surname is irish [20:19:01] mr ryan_lane told me to join here so i obeyed :) [20:19:13] This is the one and only Paul Irish in the house. [20:19:24] :o you mean he's hiding rather than reviewing my changes [20:19:28] paul_irish: Ah, you know Ryan? [20:19:41] Ryan is like the only op for webplatform currently :P [20:19:53] Ah, right. Ryan works on that too. [20:20:04] paul_irish: Are you going to the summit in dc next week? [20:20:08] (jquery) [20:20:26] * Damianz goes back to stabbing backbone js [20:21:35] Well, in Dulles, not washington, but whatever. [20:21:46] paul_irish: I kinda thought you'd be there, though I haven't seen your name anywhere yet. [20:21:54] Krinkle: unfortunately i cant make it [20:22:06] i'll be in tokyo [20:22:20] aye, unfortunate indeed. You'll be missed! [20:22:27] paul_irish: another conference in Tokyo? [20:24:10] visiting the chrome team there [20:27:04] paul_irish: that's cool too. [20:28:13] paul_irish: this summit will be my first jQuery event. Really looking forward to it ('ve been FOSDEM, OSCON and various Wikimedia events so far) [20:29:56] you'll love it. everyone is awesome [20:30:29] * Damianz wonders if Krinkle is doing FOSDEM next year [20:30:44] I went in 2011. Didn't go in 2012 [20:31:06] I must say, it was an awesome event, but content wise somewhat of a disappointment. [20:31:34] Overall quality of talks wasn't what I expected, didn't pick up a lot of stuff relevant to me and/or what I didn't know yet. [20:31:41] I keep meaning to go, wasn't that impressed with the videos last year... year before had more interesting talks imo [20:32:08] We gave one talk at OSCON in 2011 (basically the reason I went) [20:32:30] FOSDEM was a lot more satisfying [20:32:56] Primarily because of the Mozilla track (Mozilla had a room for 2 days, back-to-back presentations all day long) [20:33:22] Sat their almost continously on both days. A lot of useful stuff I picked up there. [20:37:37] I like Mozilla from the point of standards and what they try to do but firefox really sucks compared to what it use to be so they loose points there. [20:37:49] Even if chromium still doesn't have as decent plugins for some stuff still [20:58:53] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [20:59:41] I agree Firefox has been on the slow / bloat size lately, I had already switched to Safari a few years ago for the speed and web developer tools [20:59:52] and as soon as Chrome came out of the Mac, I switched again [21:00:06] except that time I also switched for casual browsing, I haven't touched Firefox in months (other than for testing) [21:00:17] I'm all on Chrome now, and loving it :) [21:00:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [21:00:36] As far as I'm concerned we could do with making WebKit the new standards [21:00:37] Lately? I was having trouble with Firefox taking up huge amounts of memory in 2010 [21:00:42] let it be part of the HTML6 standard [21:00:55] HTML6, ammendment 1: The engine must be WebKit [21:01:12] wrapper around: do whatever you want, nobody cares [21:01:26] Chromium, Safari, Chrome, IE, Mobile Safari, Opera [21:01:29] Firefox :P [21:01:39] paul_irish: thoughts ? [21:01:41] :D [21:02:32] maybe also specify that the next DOM API requires the V8 implementation of ecmascript. [21:03:29] I suppose it would spare microsoft a lot of wasted resources [21:03:53] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 43% free memory [21:03:55] They like ripping off things, right? [citation needed], nothing better than just embedding an open-source project. [21:04:46] I like v8, actually have chrome on here (osx) but chromium everywhere else. [21:05:14] m$ is build on ripping off others work :) [21:05:19] s/build/built/ [21:05:43] Yes, but microsoft does get kudos for being so good as to also reverse engineer all bugs of others work [21:05:58] most famously with the first JScript [21:06:12] and (so the legend goes) also demanded at ECMA that these bugs become the standard. [21:06:35] from Netscape's LiveScript at the time [21:10:39] there is that [21:10:52] should watch the code rush documentry again [21:30:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 189 processes [21:35:16] Krinkle, I guess you know about WebSQL, which specified that you needed to implement the SQL dialect of Sqlite 3.6.19. [22:00:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [22:03:07] Change on 12mediawiki a page Developer access was modified, changed by Hendrik Brummermann link https://www.mediawiki.org/w/index.php?diff=593358 edit summary: [22:39:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [22:44:53] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 43% free memory [22:47:06] Change on 12mediawiki a page Developer access was modified, changed by Nx link https://www.mediawiki.org/w/index.php?diff=593374 edit summary: [23:07:13] PROBLEM Disk Space is now: CRITICAL on deployment-jobrunner06 i-0000031d.pmtpa.wmflabs output: DISK CRITICAL - free space: / 286 MB (2% inode=82%):