[00:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:04:39] hi Emw, I left a msg about browser test automation on your talk page [00:06:02] chrismcmahon: ah, thank you. i'll begin working with that tonight [00:06:41] Emw: the automation can't reach inside the tag but it can tell that the tag is there or not. [00:08:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.81, 4.08, 4.98 [00:08:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.68, 4.14, 4.82 [00:10:04] chrismcmahon: hmm. there are a few things that would be nice to test in the interactive object within the canvas itself, but knowing that the canvas is there is good at least [00:12:30] Emw: there might be tricks I'm not aware of for canvas tags, I'll do some research on that, it's pretty exciting [00:12:32] maybe i could make innocuous changes to some DOM object upon certain interactions and inspect that via selenium in lieu of some way to inspect the actual 3d model [00:13:20] noone's bitten my question yet over at #selenium [00:14:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:14:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:15:13] Emw: this looks accurate: http://www.theautomatedtester.co.uk/blog/2011/selenium-advanced-user-interactions.html [00:21:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.97, 6.73, 5.67 [00:21:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.63, 7.19, 5.97 [00:24:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.64, 6.25, 5.41 [00:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:44:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:44:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:00:12] ugghhhh [01:00:14] home is filled again [01:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:04:15] !resource mobile-sms [01:04:15] https://labsconsole.wikimedia.org/wiki/Nova_Resource:mobile-sms [01:05:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 181 processes [01:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:15:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:18:33] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK [01:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:43:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [01:44:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:54:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.90, 4.74, 4.90 [02:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:04:16] can i get another public IP for a project i'm working on? it's https://labsconsole.wikimedia.org/wiki/Nova_Resource:Pdbhandler [02:05:57] Damianz, Ryan_Lane or whomever [02:06:06] ah [02:06:20] sure, what do you need it for? [02:06:26] and why in addition to the one you have? [02:06:57] i'd like to be able to test the extension with MW 1.21 in addition to the working deployment i've got with MW 1.19 [02:07:37] you don't have both on the same server? [02:07:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.28, 5.47, 5.22 [02:08:28] no, is there documentation available on how to do that? [02:08:31] ah [02:08:32] right [02:08:36] ok [02:08:54] I've upped your quota [02:09:03] thanks! [02:09:07] yw [02:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:33:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:37:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.83, 4.26, 4.74 [02:38:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:39:43] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3271104 [02:44:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:51:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [02:56:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [03:02:43] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3375484 [03:03:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:03:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [03:03:53] PROBLEM Free ram is now: WARNING on techvandalism-bot i-00000421.pmtpa.wmflabs output: Warning: 8% free memory [03:08:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [03:08:53] RECOVERY Free ram is now: OK on techvandalism-bot i-00000421.pmtpa.wmflabs output: OK: 40% free memory [03:12:43] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3372328 [03:14:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:20:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.09, 6.30, 6.52 [03:30:15] Anyone here now? [03:30:27] odie5533: yep [03:30:30] what's up? [03:31:08] Ryan_Lane: Regarding Bug #42387, I seem to be able to login to the bots-1 through 4 now so it must be created the home directories. [03:31:15] https://bugzilla.wikimedia.org/show_bug.cgi?id=42387#c1 [03:31:21] ah. ok. cool [03:31:30] However, I'm now having a different problem [03:31:35] oh? [03:31:41] The instance I created, webtools-odie, seems to have dropped off the map [03:31:56] you created it in the webtools project? [03:32:08] I can still sort of connect to it in bastion (it still has the same home directory bug), but it isn't showing in the webtools project or list of instances any more. [03:32:09] Yes [03:32:19] And it used to show up in the project instance list [03:32:50] let me see [03:33:00] I see it [03:33:13] Where? [03:33:23] go to "Manage instances" in the sidebar [03:33:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:33:43] It is not showing there for me either. [03:33:52] and I no longer have the option to create a new instance from the Manage instances. [03:33:55] do you have webtools in your project filter? [03:33:58] ah [03:33:59] Yes [03:33:59] hm [03:34:07] I wonder if someone removed you [03:34:16] It doesn't appear so [03:34:21] I think you may need to log out and back in [03:34:24] Additionally, I do not see the instance on this page: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Webtools [03:34:43] now that's weird [03:34:49] ah, that did it [03:35:11] my manage instances now works [03:35:18] now let's see why the instance kicks people out when they ssh in [03:35:25] but it is not showing on the project page. And I did a search for all instances, and it still does not show up. [03:35:54] i.e. https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&offset=0&limit=500&q=[[Resource+Type%3A%3Ainstance]]&p=searchlabel%3Dinstances%2Fformat%3Dbroadtable&po=%3FInstance+Name%0A%3FInstance+Type%0A%3FProject%0A%3FImage+Id%0A%3FFQDN%0A%3FLaunch+Time%0A%3FPuppet+Class%0A%3FModification+date%0A%3FInstance+Host%0A%3FNumber+of+CPUs%0A%3FRAM+Size%0A%3FAmount+of+Storage%0A [03:36:04] hm [03:36:10] webtools isn't shared out [03:36:35] I think webtools does not have a project volume [03:36:41] perhaps that would be an issue? [03:36:53] ah [03:36:58] I see a few issues [03:39:02] something failed when the instance was created [03:39:12] I'm fixing it [03:39:28] ok. works now [03:39:47] let me make it appear in the project, too [03:40:03] It is working now! [03:40:16] What failed during the creation? [03:40:40] dns record wasn't populated properly [03:41:09] now it shows in the project oo [03:41:12] *too [03:41:17] !resource Webtools [03:41:17] Does the webtools project have a shared volume? [03:41:17] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Webtools [03:41:23] yes [03:41:33] How can I access it? [03:41:36] /data/project [03:41:59] ls: cannot access /data/project: No such file or directory [03:42:05] hm [03:42:59] it may take a little longer for that to show up after the fix [03:43:03] gimme a sec [03:44:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:45:04] hm. weird [03:45:31] https://labsconsole.wikimedia.org/wiki/Special:NovaVolume Does not show any volumes for webtools, and would not let me create one. But it doesn't show me any volumes for bots or bastion either. [03:45:42] ignore novavolume [03:45:46] it's not implemented [03:45:53] ah, okay [03:47:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:50:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.11, 4.08, 4.81 [03:52:11] seems gluster is having some issues [03:52:34] restarting the processes [03:56:36] Are volumes created for projects when the projects are created? I think I saw a 300 GB allocation mentioned somewhere [03:56:44] yeah [03:57:01] seems we're hitting some kernel limits on the storage servers, though [03:57:07] I'm going to need to increase them [03:57:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 17% free memory [03:59:24] Ryan_Lane: Do you know if the DNS problem is a bug in the creation process or a one-time error? [03:59:33] one-time error [03:59:47] most likely related to a bug in memcache we're having [04:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:14:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:18:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:21:28] Ryan_Lane: Any luck with the project storage? [04:21:53] not yet [04:21:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.01, 4.41, 4.89 [04:22:25] we have too many volumes [04:22:37] the process is trying to open too many sockets [04:22:42] and is hitting ulimits [04:22:59] and it seems gluster has no way of adjusting that, even though it runs as root [04:23:47] so, I'm trying other ways of handling it [04:25:03] Alright. I'm still able to test everything I'd want basically since there is plenty of storage for the instance and I can just move stuff at a later point if necessary. Thank you again for the help. [04:26:02] yw [04:26:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.28, 3.57, 4.66 [04:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:34:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.65, 5.49, 5.23 [04:37:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.70, 4.05, 4.65 [04:46:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:48:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:53:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3406096 [04:59:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.29, 3.90, 4.68 [05:04:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:16:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:18:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:34:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:36:42] PROBLEM Free ram is now: WARNING on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: Warning: 15% free memory [05:41:42] RECOVERY Free ram is now: OK on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: OK: 37% free memory [05:46:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:48:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:56:41] odie5533: it's fixed now [06:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:17:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:18:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:31:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:31:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:47:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:48:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:51:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [06:56:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [07:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [07:17:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:20:43] Maybe we should have a #wikimedia-labs-log channel [07:21:01] so that we can chat in this one, and people that like seeing spam can visit the other one ;) [07:21:08] well, realistically, those logs should go away. those hosts are likely dead [07:21:14] meaning they should be deleted [07:21:41] the logs are telling us there's a problem. the nagging should make people do something about it. heh [07:21:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [07:21:57] But many of the notices are for different people. [07:22:05] For instance, there is nothing I can do to fix those. [07:22:06] of course, it would likely be more helpful if it showed the instance names rather than ids [07:22:08] yeah [07:22:23] so they are important to only a select number of people [07:22:31] unless they are your instances [07:23:02] True. I guess I can just set the bot to ignore. [07:23:23] ahh, peace and quiet. [07:23:26] :D [07:36:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:47:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:51:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [08:07:22] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 323 MB (3% inode=66%): [08:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:12:22] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DISK CRITICAL - free space: / 283 MB (2% inode=66%): [08:17:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:48:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:49:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:08:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.89, 4.11, 4.81 [09:13:20] petan: did you do any of the configuring of the bots instances? [09:13:40] yes, all of them [09:13:53] Did you use puppet? [09:14:03] depends [09:14:10] for all puppetized stuff that worked yes [09:14:13] otherwise not [09:14:22] we are definitely going to use it though [09:15:29] petan: It seems quite complicated to use. How did you learn to use puppet? [09:15:55] I didn't, that's why it's not being used for everything right now [09:15:59] :) [09:16:12] I am still learning and yes, it's very complicated, more than it should be [09:18:04] petan: Can any user edit other user's public_html files on bots? [09:18:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:18:32] odie5533: yes but only on testing environment [09:18:59] How will permissions be set so that they can't edit others' files? [09:19:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:19:52] I guess I am thinking I want the webtools to have a web server that serves files from each user's own private area to a similar /~user as the bots project has. [09:20:23] And I'd like it so users can't mess with other people's files. [09:22:30] petan: I guess you have a cron set up to create the public_html directories for the users [09:22:49] this was configured by Damianz [09:23:01] but the public_html doesn't have permissions set so other users can't edit them [09:23:06] I think that puppet does it, not sure... [09:23:32] odie5533 it's not supposed to be editable by other users, but on testing environment all users have root [09:24:01] petan: Well, for a new instance, how can you set it so that, given not everyone has root, users can only edit their own files? [09:24:54] you mean how to create a folder where everyone can write to, but newly created objects are writable by owners only? [09:25:21] ehm, no. a folder where only a specific user can write to. [09:25:58] so that the /data/project/public_html/username is the only directory they can write to (for their username) [09:26:17] create a folder then "chown user folder" [09:26:21] chmod 700 folder [09:26:32] write or even read? [09:26:40] if others can read, then chmod 755 folder [09:26:41] write, read, execute [09:26:48] I mean others [09:27:03] with chmod 755 owner can rwx, others r-x [09:27:11] okay [09:27:51] petan: I didn't realize it was that easy, thank you. [09:28:17] so about puppet.... how do you use it? :) [09:29:37] !puppet [09:29:38] learn: http://docs.puppetlabs.com/learning/ troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html [09:29:50] @regsearch puppet [09:29:50] Results (Found 7): $realm, $site, instanceproject, puppet, git-puppet, ryanland, puppetmasterself, [09:30:08] there is no labs guide yet [09:30:25] and I am the worst person to ask regarding that [09:30:26] :D [09:30:32] I don't understand it [09:30:36] heh [09:31:24] I guess I'll just use apt-get for now and document what packages I download so that it could be switched over to puppet later. [09:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:43:37] odie5533 there is command log for that [09:43:50] log "apt-get these things..." [09:44:04] can I delete stuff from the log? [09:44:08] yes [09:44:10] !sal [09:44:10] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [09:44:35] I am looking at the users on the instance, and I don't appear to have an entry in /etc/passwd [09:48:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:49:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:18:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:38:09] is there a problem with http://en.wikipedia.beta.wmflabs.org/ ? [10:38:22] I am getting 403 forbidden [10:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:39:26] looks like the entire wmflabs.org is down [10:39:32] http://isup.me/wmflabs.org [10:39:41] It's not just you! http://wmflabs.org looks down from here. [10:48:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:49:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:49:19] zeljkof: works for me [10:49:37] oh, wait, wmflabs.org is down nevermind! [10:49:45] I thought you meant the labconsole [10:49:59] odie5533: ok, then it is not just me [10:50:22] odie5533: how do I report the problem, in bugzilla? [10:52:02] I have no idea. [10:52:16] There's also a mailing list [10:52:49] odie5533: let me check if I am already subscribed to labs mailing list [10:53:13] odie5533: do you know the name of the list? [10:53:29] !log wikidata-dev wikidata-dev-3 Changes in config: require_once( "$IP/extensions/Wikibase/lib/WikibaseLib.php"); is no longer needed, using experimental branch on both repo and client: require_once( "$IP/extensions/Wikibase/client/WikibaseClient.experimental.php"); on devclient: added config line $wgWBSettings['repoDatabase'] = "dev_repo"; for upcoming changes [10:53:58] hm [10:54:00] odie5533: found it: https://lists.wikimedia.org/mailman/listinfo/labs-l [10:54:17] !list [10:54:17] we have a mailing list labs-l@lists.wikimedia.org feel free to send a message there, don't forget to subscribe [11:09:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:19:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:39:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:49:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [12:09:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:19:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:39:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:49:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:09:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:17:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 16% free memory [13:19:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:19:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:22:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 21% free memory [13:39:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:55:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [14:10:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:19:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:19:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:11:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:19:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:19:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:21:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 12% free memory [15:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:59:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 155 processes [16:00:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [16:03:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [16:08:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [16:08:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 15% free memory [16:11:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:19:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:19:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:28:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 21% free memory [16:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:04:12] RECOVERY Disk Space is now: OK on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK OK [17:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:19:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:20:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [17:40:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [17:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:49:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:49:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:03:27] ^demon: how are you deploying gerrit.war to gerrit-dev? puppet? do you have your repo checked out somewhere? [18:03:49] <^demon> I have the repo checked out to /gerrit/core, and have been deploying manually from there. [18:09:13] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [18:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:19:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:39:48] Ryan_Lane: fyi, http://bugs.debian.org/694114 [18:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:43:01] 3.0? [18:43:58] ah. mediawiki-extensions package 3.0 [18:44:21] <^demon> saper: Running init --batch tends to misconfigure gerrit, so running puppet afterwords is a good idea. [18:44:33] <^demon> So my process is build -> init -> puppet -> start gerrit [18:45:48] jeremyb: something special I should be seeing there? [18:48:42] Ryan_Lane: i guess not, just thought you might be interested [18:49:09] meh [18:49:20] I don't use the packages [18:49:36] it only interests me if they find a bug I haven't found yet [18:49:42] since they often do and don't report upstream [18:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:56:59] andrewbogott_afk: reviewed your OSM changes [19:09:42] PROBLEM Free ram is now: WARNING on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: Warning: 8% free memory [19:13:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:14:42] RECOVERY Free ram is now: OK on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: OK: 33% free memory [19:17:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [19:18:01] andrewbogott: you want to deploy the changes, or me? [19:18:17] I really need to upgrade labsconsole at some point [19:18:34] I need to upgrade webplatform too [19:18:39] damn mediawiki. heh [19:18:40] Ryan_Lane: I'm going to fix another couple little things today, then we can do it all at once. [19:18:44] cool [19:18:48] sounds good [19:18:52] When you say 'upgrade labsconsole' you mean the whole mediawiki install? [19:18:55] thanks for working on openstackmanager [19:18:56] yeah [19:19:02] it's a decent bit of work [19:19:17] I think I'll wait on the release of the next version of SMW [19:19:34] since we're using some bastardized version [19:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:21:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:33:34] Change on 12mediawiki a page Developer access was modified, changed by Bene* link https://www.mediawiki.org/w/index.php?diff=611064 edit summary: [19:44:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:45:34] Ryan_Lane, do you think you meant 'instance id/name' instead of 'image id/name' in this one? https://bugzilla.wikimedia.org/show_bug.cgi?id=40065 [19:47:48] oh, nm, I see what you mean [19:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:57:42] PROBLEM Free ram is now: WARNING on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: Warning: 11% free memory [20:14:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:19:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:44:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:49:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:51:22] PROBLEM Current Load is now: WARNING on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: WARNING - load average: 2.15, 6.83, 5.61 [20:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:56:23] RECOVERY Current Load is now: OK on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: OK - load average: 0.47, 2.81, 4.19 [21:00:57] !resource maps [21:00:57] https://labsconsole.wikimedia.org/wiki/Nova_Resource:maps [21:04:53] so many OSM fixes! :) [21:04:56] \o/ [21:08:59] andrewbogott: gluster got itself into a fucked up state recently [21:09:18] when? [21:09:31] seems we had enough volumes that it was opening too many files for sockets [21:09:38] I fixed it last night [21:09:47] it's likely been broken for new volumes for a couple weeks [21:09:56] someone reported it last night [21:10:15] mmmm gluster [21:10:19] yeah [21:10:21] heh [21:10:39] I'm starting to wonder if we should only have one volume per project [21:10:46] How did you fix it? [21:10:55] have project storage and home directory storage in the same share [21:11:04] and quota them as directories under the share [21:11:14] I changed the ulimit [21:11:20] which wasn't the easiest thing in the world [21:11:24] seems upstart in lucid is buggy [21:11:34] and gluster can't change it itself [21:11:50] Hm… so merging into the same project would only give us 2x as many volumes. [21:11:58] so, I changed the upstart from using an exec by itself to launch gluster into a script with ulimit and exec inside of it [21:12:06] yeah [21:12:18] but that's a lot of ports [21:12:24] and a lot less volumes [21:12:39] sharing is the exact same [21:12:49] same sets of nodes and such [21:12:54] less to break [21:13:02] hm [21:13:08] I'm not sure how mounting that would work, though [21:13:15] I don't think you can mount subvolumes in gluster [21:13:20] wait, is each homedir a different volume? [21:13:24] nah [21:13:42] but we have project and home for every project [21:13:52] so if we combine them it's 1/2 as many volumes [21:13:59] Yep, OK, that's what I thought. [21:14:10] but if we combine them we'd need to mount them somehow [21:14:18] I'm not sure it's possible [21:14:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:14:24] Is there a theoretical upper bound on number of sockets? Can you just change the ulimit to 'many' and have done? [21:14:43] there's a hard limit in the kernel I believe [21:14:46] but it's really, really high [21:14:55] I changed it from 1024 to 40000 [21:15:14] Oh, well… in that case it doesn't seem worth thinking about for a while :) [21:15:24] it's one socket per brick per volume [21:15:39] more if you use NFS [21:16:02] Is there a reason we don't want project storage mounted all the time? [21:16:19] yes [21:16:28] if you need to make changes to the mount, autofs makes things way easier [21:16:33] though we should likely make the timeout longer [21:16:39] it's really short right now [21:17:08] lemme find the change I made [21:17:11] ok. Otherwise we could just have project storage be /home/projectstorage and then mounting would be a solved problem [21:17:47] https://gerrit.wikimedia.org/r/#/c/35854/ [21:18:02] that would be a mount under a mount [21:18:09] it works, but it's not usually a good idea [21:18:36] though, realistically, we could make /data/project a direct mount, rather than an indirect one [21:18:42] then people could see it [21:18:48] oohhhhh [21:18:50] I see what you mean [21:18:54] yeah, that could work [21:18:54] No, I mean it wouldn't be a mount at all. We'd just have one shared volume /home which... [21:18:54] yeah. [21:19:14] that's dirty :D [21:19:31] let me see if it's possible to mount subdirs [21:19:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:20:03] Hm… for some reason having shared storage in an 'account' dir like /home/xfer or /home/shared feels normal to me. [21:20:27] it feels like going into someone else's home directory to me [21:21:31] understandable [21:21:43] can't mount subdirs [21:21:44] :( [21:21:50] you can with nfs [21:21:54] one more reason to hate gluster [21:22:23] I don't think there's a need for anyone to need more than a single project volume [21:22:27] I should make /data/project direct [21:22:41] if they need more space we can increase the quota [21:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:27:44] there we go [21:27:52] now /data/project shows when you do ls /data [21:28:00] now to remove the * entry [21:28:54] done. [21:30:17] Ryan_Lane: Could you also ping me when you have a second, regarding my project request? [21:30:25] I have a second now [21:30:29] Awesome. [21:30:31] I wrote a response [21:30:35] did you respond back? [21:30:44] I saw your response, thank you. [21:30:52] yw [21:31:51] I was just wondering how much trouble it would be to set that up (looking forward to centralized user databases, BTW). Short version: PHP inserts into SQL, then bot pulls and edits Wikipedia. Bot is in Python. I do have usernames in the SQL, so I hope that won't be a problem. [21:32:32] someone in bots needs to do it for you [21:32:52] they would need to add your IP address in the security group, and add your permissions via mysql [21:32:58] it should be relatively straightforward [21:33:11] petan, Damianz: ^^ [21:33:20] That's great. And no security concerns with the usernames? [21:33:47] petan, Damianz: does that make sense to you guys? some tools in webtools project may occasionally need access to databases in the bots projects [21:34:15] where's the split betwean bots web and webtools then? surly they're seperate [21:34:18] Matthew_: well, other people will have access to your bot's username/password [21:34:41] Damianz: permissions on who can manage instances, network rules, etc [21:34:51] Damianz: it's common for bots and tools to need access to the same databases [21:35:03] Ryan_Lane: Hmmm... since it's only doing one task, and a minor one at that (I'm just getting ready to port to labs pending the closer of Toolserver), it's not that big of a deal. [21:35:06] some tool writers will never write bots and some bot writers will never make tools [21:35:09] I think the split makes sense [21:35:26] But on the flip side, Toolserver users are used to them being in one place... [21:35:39] Matthew_: yeah. the only people that can see it are the bots admins, btw [21:35:42] other users can't [21:35:47] so it's somewhat limited [21:35:56] and in the future it will be even more limited [21:36:13] (from the perspective of who can see username/password) [21:36:33] OK, that's fine. Like I said, since it's a minor task, I'm not as worried about the username and password. The bot can be blocked for making any mainspace edits. [21:36:42] * Ryan_Lane nods [21:36:52] if your bot gets blocked make sure to tell us :) [21:37:00] we always want to know if something fishy is going on [21:37:11] Of course. [21:37:15] not that I expect that [21:37:40] Damianz: do you think it should be a combined project? [21:38:04] I kind of like that it's split [21:38:08] I don't either. I've found all of the devs I've met to be trustworthy. [21:38:15] it's easier to deal with security with split projects [21:39:05] I'm of a split opinion [21:40:17] hmm [21:42:20] "split" opinion :D [21:42:48] * Damianz throws a brick at Ryan [21:42:51] I am also of the opinion that it should be split! [21:43:12] I think if it's split bots shouldn't have a webserver [21:43:25] no? [21:43:28] why not? [21:43:32] But if we're doing webtools for serious it should be managed properly (via puppet or labsconsole) [21:43:45] well, bots should too :) [21:43:47] Because it seems we're duplicating efforts [21:43:51] true [21:44:01] bots is a pile of shit right now >.> [21:44:09] yeah, bots needs some love [21:44:10] there's a limiting things called time involved [21:44:16] indeed [21:44:20] always... [21:44:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:49:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:50:26] Ryan_Lane: I have to go to class now, I'll be avalible later. Please let me know if you need anything else from me. [21:50:45] we'll need to give you access to webtools [21:50:53] you'll need to create an instance [21:50:58] then we'll need to make the mysql changes [21:51:52] OK. [21:52:42] ttyl [21:52:51] cya [21:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:55:21] Change on 12mediawiki a page Developer access was modified, changed by Ezeroğlukancer link https://www.mediawiki.org/w/index.php?diff=611109 edit summary: [21:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [21:57:02] Change on 12mediawiki a page Developer access was modified, changed by Ezeroğlukancer link https://www.mediawiki.org/w/index.php?diff=611111 edit summary: /* User:{{Kancer Ezeroğlu}} */ [21:58:09] Change on 12mediawiki a page Developer access was modified, changed by Ezeroğlukancer link https://www.mediawiki.org/w/index.php?diff=611113 edit summary: /* User:Kancer Ezeroğlu */ [22:06:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [22:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:30:12] petan: you like C#, I think there's someone in #webplatform you should talk to :) [22:30:53] petan: fr0zenice is working on a bot framework in javascript, that's run via a C# javascript engine [22:31:03] where you can submit javascript to the engine, and it'll run it [22:31:37] the idea would be to have a set of admin tools for common functions (like pywikipediabot) [22:36:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [22:42:13] PROBLEM Disk Space is now: WARNING on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK WARNING - free space: /tmp 0 MB (4% inode=99%): [22:45:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:51:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:53:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:57:12] PROBLEM Disk Space is now: CRITICAL on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK CRITICAL - free space: /tmp 0 MB (2% inode=99%): [23:11:19] Ryan_Lane: In labs ldap, uid is the same as shell name, right? [23:11:30] yep [23:11:35] shell account name [23:11:41] great. [23:15:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:22:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:23:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:45:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:52:18] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=611170 edit summary: /* User:Bene* */ done [23:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:52:35] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=611171 edit summary: /* User:Kancer Ezeroğlu */ done [23:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)