[00:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:14:14] OK, the sudoers page should work now (for the moment) [00:14:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:16:52] andrewbogott: so how to install MediaWiki? [00:17:24] Let's see if you can sudo now [00:17:47] andrewbogott: I can [00:18:07] * Damianz gives andrewbogott a cookie [00:18:23] Just 'cause I fixed it doesn't mean I know why it was broken :( [00:18:30] * PiRQuadratus gives andrewbogott another cookie but it's http-only [00:18:43] PiRQuadratus: Try 'sudo puppetd -tv' and let's see what it has to say. [00:18:59] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Sshhostkey[abusefilter-global-main.pmtpa.wmflabs] cannot override local resource on node i-00000512.pmtpa.wmflabs [00:19:09] that's normal [00:19:12] um… for context, 'puppet' is the tool that's configuring mediawiki, using scripts that I wrote. [00:19:18] andrewbogott: Sound like a standard php developer ;) [00:19:31] Damianz: Yeah :( [00:20:01] Is there any place I can get a beginner's tutorial for WM labs? I have an instance set up, but no idea what to do next. [00:20:37] blackjack48: I take it you read the 'getting started' guide and it clarified nothing? [00:22:38] Yep. :) I have a project on git that I'd like to use, but the whole puppet thing seems confusing. [00:24:20] There isn't any tutorial beyond what you've seen. Since labs is basically just 'here are some computers for you' it's hard to approach as a general subject. [00:24:39] Do you have a specific question? Or a specific thing you're trying to do? [00:24:44] see that's the point where project documentation would be helpful [00:24:54] Sure. [00:25:25] Also why is it half midnight :( [00:26:01] Damianz, you could fix that problem by switching continents. [00:26:16] That's true [00:26:16] PiRQuadratus, did puppet tell you anything interesting? [00:26:20] Mediawiki-related? [00:26:30] nope [00:26:34] But then I'd have to be up at 3am for my 4month infrastructure meeting ;P [00:28:22] andrewbogott: Sumana said I should set up a prototype site for a feature I was working on. [00:28:38] PiRQuadratus: That's 'cause it's working! [00:28:46] * andrewbogott just checked again [00:29:00] blackjack48: a mediawiki feature? [00:29:19] Yep, can I create a wiki on my instance somehow? [00:29:21] blackjack48: Imagine that I don't know any of the context for what you're doing. Because I don't. [00:29:58] blackjack48: This might be useful: https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki [00:30:25] But that page is full of terminology and links and it's pretty important that you read it and understand before launching in. [00:31:23] Ajraddatz: ^ working [00:31:38] linksss [00:31:41] andrewbogott: that does look potentially useful - thanks. [00:32:09] blackjack48: The part that everyone skips and then regrets is the part about setting up security groups before you build your instance. [00:32:22] (PiRQuadratus can testify about the other parts he regrets :) ) [00:32:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.81, 6.07, 5.31 [00:33:47] wait, does labs use centralauth and separate wiki databases? [00:33:55] I wish you could change security group to instance mappings :( [00:34:10] Makes sense not to for 'cloud' throw away instances but for long running ones like ours it's a pita. [00:34:23] Ajraddatz: no and yes [00:34:33] seperate in the sense of empty and on the instance you make [00:34:44] Most people bulk load in part of a en wiki dump [00:34:50] * andrewbogott is punching out for a bit. [00:35:13] * Damianz is going to bed so he can get in early and finish his estimates [00:35:32] PiRQuadratus, I'll be back online tomorrow if you have more questions. I'm in the Minneapolis right now, that's UDT-6. [00:35:46] Um… Minneapolis, not 'the' Minneapolis. [00:36:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:42:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.94, 6.71, 5.68 [00:43:58] okay, bye [00:44:03] see you tomorrow [00:44:36] Ajraddatz: I'll finnish [sic] it tomorrow [00:44:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:45:54] Ajraddatz: https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy <- you can use this to access it too [00:46:06] except you're not in this project [00:47:14] doesn't matter [00:47:16] well, you don't need to be in the project, as long as the project is allowing http access to all of labs [00:47:32] * Ryan_Lane goes back to lurking [00:47:54] * Damianz notes the sighting of a wild ryan [00:57:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.48, 3.77, 4.68 [00:57:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.96, 4.27, 4.83 [01:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:09:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:14:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:25:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.02, 5.39, 5.11 [01:30:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.07, 4.47, 4.78 [01:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:31:19] sumanah: hello [02:31:44] Hi PiRQuadratus [02:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:41:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:54:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:39:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:02:22] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 574 MB (5% inode=70%): [04:05:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.08, 5.22, 5.22 [04:05:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.63, 6.26, 5.69 [04:09:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:10:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.05, 4.30, 4.82 [04:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:21:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.78, 5.99, 5.35 [04:24:15] Change on 12mediawiki a page Developer access was modified, changed by Bruceburge link https://www.mediawiki.org/w/index.php?diff=607131 edit summary: [04:35:05] 11/19/2012 - 04:35:04 - Creating a project directory for video [04:36:03] Change on 12mediawiki a page Developer access was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=607134 edit summary: [04:36:19] Change on 12mediawiki a page Developer access was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=607135 edit summary: [04:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:41:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.12, 3.73, 4.65 [04:48:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:50:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 3.78, 5.11, 5.11 [04:51:25] * BruceBurge slaps Ryan_Lane1 around a bit with a large trout [04:51:40] you must be using the web client [04:52:29] you should likely read these docs [04:52:32] !security [04:52:32] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [04:52:36] !instances [04:52:36] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [04:55:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.09, 4.42, 4.82 [04:57:45] !log video created video project for development and testing of an external video extension for MediaWiki [04:57:47] Logged the message, Master [04:57:52] !resource video [04:57:52] https://labsconsole.wikimedia.org/wiki/Nova_Resource:video [05:00:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.01, 3.68, 4.84 [05:08:32] !logging | BruceBurge [05:08:32] BruceBurge: To log a message, use the following format: !log [05:08:48] though you probably aren't working on stuff with others, it's a good habit [05:10:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:18:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:40:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:48:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:18:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:23:43] Every time I try to create a security group, it says "Failed to create security group." - any idea why this is happenning? [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:30:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:33:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:40:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:48:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:18:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:40:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [07:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:48:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:18:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:43:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:18:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:33:10] why is that ~petrb/logs no longer accessible did someone play with apache [09:33:16] Damianz, jeremyb ? [09:33:20] on bots [09:33:31] it gives 403 [09:36:01] i didn't [09:36:22] !htmllogs [09:36:22] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [09:37:16] well it's accessible for me... [09:37:23] ~wm-bot is [09:37:26] but ~petrb is not [09:37:29] it should be same [09:37:38] and i'm saying it is [09:37:38] http://bots.wmflabs.org/~petrb/logs/%23wikimedia-labs/ [09:37:47] You don't have permission to access /~petrb/logs/#wikimedia-labs/ on this serve [09:37:54] oh, maybe the individual dirs aren't [09:38:04] but ~petrb/logs is accessible [09:38:14] You don't have permission to access /~petrb/logs/ on this server. [09:38:17] neither that [09:38:50] it's inconsistent for me [09:38:57] i got 200 then 403 then 200 [09:39:19] lol [09:39:55] wtf [09:40:33] weird apache [09:40:43] or weird gluster [09:40:44] :D [09:41:07] I am wondering why the performance of website got so bad after we moved to gluster [09:41:32] now refresh of single dir takes 20 seconds [09:41:35] before it was 1 second [09:44:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:49:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:11:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.92, 4.57, 5.00 [10:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:33:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [10:38:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [10:38:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 144 processes [10:43:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [10:44:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:49:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:06:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 154 processes [11:14:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:19:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:45:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:49:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:45:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:16:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:46:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:46:45] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=607261 edit summary: /* User:Ilonakz */ not done [13:47:09] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=607262 edit summary: /* User:TK-999 */ [13:47:39] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=607263 edit summary: /* User:יוסף שמח */ :{{done|Account created.}} ~~~~ [13:48:33] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=607265 edit summary: /* User:Aracele */ created [13:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:16:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:46:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:51:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:34:17] Change on 12mediawiki a page Developer access was modified, changed by Martsniez link https://www.mediawiki.org/w/index.php?diff=607362 edit summary: [15:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:51:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:55:28] andrewbogott_afk hashar OK, I deleted the files that are no longer needed from the 5th patchset. (Also the apache template that I do not really need at the moment.) [15:58:18] Silke_WMDE_: nice :-] [15:58:36] Silke_WMDE_: try to reach andrew now :-] [15:58:45] eventually drop him an email since you are in different timezone [15:58:50] yep [15:58:53] I am off to get my daughter [15:59:05] Thx! CU! [16:17:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:46:24] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [16:47:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:51:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [16:51:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:59:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [17:17:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:21:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:25:52] PROBLEM dpkg-check is now: CRITICAL on gerrit-build i-0000050d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [17:35:52] RECOVERY dpkg-check is now: OK on gerrit-build i-0000050d.pmtpa.wmflabs output: All packages OK [17:47:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:09:16] petan, Damianz: Any chance we could get curl installed on bots.wmflabs.org? [18:18:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:23:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:26:03] Change on 12mediawiki a page Developer access was modified, changed by Ilonakz link https://www.mediawiki.org/w/index.php?diff=607424 edit summary: [18:44:29] Change on 12mediawiki a page Developer access was modified, changed by MarkTraceur link https://www.mediawiki.org/w/index.php?diff=607435 edit summary: /* User:Ilonakz */ [18:49:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:53:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:19:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:23:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:23:49] kaldari: {{done}} [19:23:59] Yay! [19:24:15] owe you a wikibeer :) [19:24:45] * Damianz adds it to the list with sumanah [19:49:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:53:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:23:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:49:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:54:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:55:23] 11/19/2012 - 20:55:22 - Updating keys for jeremyb at /export/keys/jeremyb [20:55:45] danke labs-home-wm [21:02:41] Damianz: Any idea why I would be getting: Fatal error: Call to undefined function curl_init() [21:04:41] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607479 edit summary: /* User:Ilonakz */ rm duplicate req [21:05:04] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607481 edit summary: /* User:Ilonakz */ done [21:05:32] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607482 edit summary: /* User:Ilonakz */ grrrrr, fingers are too fast! [21:06:06] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607483 edit summary: /* User:Martsniez */ done [21:06:29] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607484 edit summary: /* User:Ilonakz */ 3rd time maybe? [21:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:24:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:50:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:56:04] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 19% free memory [22:20:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:35:16] Every time I try to create a security group, it says "Failed to create security group." - any idea why this is happenning? [22:35:24] hi blackjack48 [22:35:49] blackjack48: Are you sysadmin/netadmin for the project? [22:35:49] Hi Sumana [22:35:55] It might just be broken, I'll check [22:36:11] I'm not sure, how do I check? [22:37:15] 'manage projects' should say what the roles are in a project [22:38:05] blackjack48: I just created a secgroup without trouble. So at least the web interface is working OK. [22:39:16] andrewbogott: it says I am both a sysadmin and a netadmin for the project. [22:39:28] what's the name of the security group you're creating? [22:39:59] I've tried several variations of "web" [22:41:05] what project? [22:41:37] "watchlist-groups" [22:42:12] worked for me :( [22:42:47] Want me to back it out and figure out what it didn't work for you, or just forge ahead? [22:43:31] are there any other groups I need to create for a simple web server? Otherwise, I'd be fine forging ahead... [22:43:39] *mediawiki server [22:44:02] Nope. You will need to add rules to the group, though… right now it's empty. [22:45:46] yay forging ahead [22:45:49] I'm able to add rules, so I think I'm set for now - thanks. [22:45:53] :) [22:46:12] Sure. No idea why it spurned you :( [22:51:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:58:52] PROBLEM Current Load is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [22:59:32] PROBLEM Current Users is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [23:00:13] PROBLEM Disk Space is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [23:01:03] PROBLEM Free ram is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [23:02:23] PROBLEM Total processes is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [23:02:53] PROBLEM dpkg-check is now: CRITICAL on wlgroupsdemo i-00000513.pmtpa.wmflabs output: Connection refused by host [23:12:23] RECOVERY Total processes is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: PROCS OK: 84 processes [23:12:53] RECOVERY dpkg-check is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: All packages OK [23:13:53] RECOVERY Current Load is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: OK - load average: 0.22, 0.98, 0.89 [23:14:33] RECOVERY Current Users is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [23:15:12] RECOVERY Disk Space is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: DISK OK [23:16:02] RECOVERY Free ram is now: OK on wlgroupsdemo i-00000513.pmtpa.wmflabs output: OK: 1053% free memory [23:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:46:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 163 processes [23:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:52:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3710672 [23:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)