[00:12:04] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 350 minutes) [00:25:33] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 364 minutes) [00:39:11] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 377 minutes) [00:52:42] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 391 minutes) [01:06:16] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 404 minutes) [01:19:46] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 418 minutes) [01:33:20] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 432 minutes) [01:46:51] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 445 minutes) Acnetj (waiting 2 minutes) [02:00:21] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 459 minutes) Acnetj (waiting 15 minutes) [02:13:50] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 472 minutes) Acnetj (waiting 29 minutes) [02:27:18] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 486 minutes) Acnetj (waiting 42 minutes) [02:40:39] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 499 minutes) Acnetj (waiting 55 minutes) [02:54:09] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 512 minutes) Acnetj (waiting 69 minutes) [03:07:34] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 526 minutes) Acnetj (waiting 82 minutes) [03:20:59] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 539 minutes) Acnetj (waiting 96 minutes) [03:34:30] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 553 minutes) Acnetj (waiting 109 minutes) [03:47:55] Warning: There are 2 users waiting for shell, displaying last 2: Steinsplitter (waiting 566 minutes) Acnetj (waiting 123 minutes) [04:01:16] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 579 minutes) [04:05:59] YuviPanda: so we got limn installed and the server thing runs, I just can't figure out where the public endpoint is... [04:06:18] legoktm: you need to get an IP assigned to your labs instance [04:06:22] legoktm: and then add a DNS entry [04:06:26] errr [04:06:33] tools.wmflabs.org/stats/ [04:06:41] why isnt that good enough? [04:06:45] bleh [04:06:46] tools labs [04:06:47] right [04:06:51] i forgot [04:06:59] so what do you mean by 'public endpoint'? [04:07:17] it should've just opened up a port and started serving? [04:09:03] ok i just started running it [04:09:06] with npm start [04:09:14] except http://tools.wmflabs.org/stats/ is still nothing [04:09:27] legoktm: which port is it running on? [04:09:37] Limn Server (port=8081, env=development, rev=HEAD, vardir=./var, data=./var/data) [04:09:44] legoktm: i'm pretty sure you need to setup some form of reverse proxying for that to work properly. [04:09:49] >.< [04:10:02] one of the annoying things about tools labs, IMO [04:10:26] where is yours running from? [04:14:46] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 593 minutes) [04:15:15] Good night everyone. [04:15:58] legoktm: just a normal labs instance [04:16:02] not tools labs [04:16:10] oh hmmm [04:18:09] Maybe it would be easier to just do that…. >.< [04:19:24] legoktm: yeah, I'm not sure how good tools labs is going to be supporting non PHP stuff [04:19:31] * YuviPanda shakes fist at CGI [04:28:08] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 606 minutes) [04:30:16] legoktm: I suggest talking to Coren about this [04:30:25] legoktm: obviously we'll like ts to support nodejs too [04:30:28] err [04:30:29] toolslabs [04:30:31] i mean [04:30:35] legoktm: and not just PHP [04:30:36] tl*? :P [04:30:37] yeah [04:41:30] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 620 minutes) [04:54:54] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 633 minutes) [05:08:24] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 647 minutes) [05:21:54] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 660 minutes) [05:35:21] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 674 minutes) [05:48:52] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 687 minutes) [06:02:25] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 701 minutes) [06:15:50] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 714 minutes) [06:29:15] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 727 minutes) [06:42:41] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 741 minutes) [06:56:08] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 754 minutes) [07:09:34] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 768 minutes) [07:22:55] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 781 minutes) [07:36:17] @requests [07:36:17] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 795 minutes) [07:36:25] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 795 minutes) [07:38:23] @requests [07:38:23] Warning: There is 1 user waiting for shell: Steinsplitter (waiting 797 minutes) [07:38:34] meh [07:39:06] @requests [07:39:06] There are no shell requests waiting [08:07:02] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: High - major) [Bug 41104] glusterfs log files are not rotated - https://bugzilla.wikimedia.org/show_bug.cgi?id=41104 [08:29:07] [bz] (NEW - created by: anubhav, priority: Low - normal) [Bug 46951] Unable to Create a New Developer Account - https://bugzilla.wikimedia.org/show_bug.cgi?id=46951 [09:06:26] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Normal - enhancement) [Bug 45617] update Precise image to get the latest patches - https://bugzilla.wikimedia.org/show_bug.cgi?id=45617 [09:07:20] Warning: There is 1 user waiting for shell: Dominiol (waiting 0 minutes) [09:20:54] Warning: There is 1 user waiting for shell: Dominiol (waiting 13 minutes) [09:26:04] Coren: around? [09:34:16] Warning: There is 1 user waiting for shell: Dominiol (waiting 27 minutes) [09:45:49] addshore: what u need hm? [09:46:05] i need my project on labs called 'proposals' to blow up [09:46:07] ;p [09:46:11] not needed any more :) [09:46:19] blow up? :P [09:46:57] just delete it ;p [09:47:20] ok so u need to delete anything or not? btw can I remove that wikidatastats now? [09:47:50] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 40 minutes) If11026 (waiting 5 minutes) [09:52:11] petan: yeah go for it [10:01:21] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 54 minutes) If11026 (waiting 19 minutes) [10:14:56] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 67 minutes) If11026 (waiting 33 minutes) [10:28:27] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 81 minutes) If11026 (waiting 46 minutes) [10:41:53] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 94 minutes) If11026 (waiting 60 minutes) [10:55:20] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 108 minutes) If11026 (waiting 73 minutes) [11:08:51] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 121 minutes) If11026 (waiting 86 minutes) [11:22:22] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 135 minutes) If11026 (waiting 100 minutes) [11:35:52] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 148 minutes) If11026 (waiting 114 minutes) [11:49:27] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 162 minutes) If11026 (waiting 127 minutes) [12:02:56] Warning: There are 2 users waiting for shell, displaying last 2: Dominiol (waiting 175 minutes) If11026 (waiting 141 minutes) [12:03:27] haha :) [12:03:41] !sr is https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=[[Category%3AShell+Access+Requests]][[Is+Completed%3A%3Afalse]]&p=format%3Dtemplate%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3DOutstanding-20Requests%2Fdefault%3D%28no-20outstanding-20requests%29%2Ftemplate%3DShell-20Request%2Fintrotemplate%3DShell-20Request-2Fhead%2Foutrotemplate%3DShell-20Request-2Ffoot&po=%3FShell+Request+User+Name%0A%3FShell+Justification%0A%3FModification+ [12:03:41] Key was added [12:03:47] !sr [12:03:47] https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=[[Category%3AShell+Access+Requests]][[Is+Completed%3A%3Afalse]]&p=format%3Dtemplate%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3DOutstanding-20Requests%2Fdefault%3D%28no-20outstanding-20requests%29%2Ftemplate%3DShell-20Request%2Fintrotemplate%3DShell-20Request-2Fhead%2Foutrotemplate%3DShell-20Request-2Ffoot&po=%3FShell+Request+User+Name%0A%3FShell+Justification%0A%3FModification+ [12:03:52] :/ [12:05:18] !sr del [12:05:19] Successfully removed sr [12:05:25] !requests is http://bit.ly/10eZZoa [12:05:25] This key already exist - remove it, if you want to change it [12:05:30] !requests [12:05:30] https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests [12:05:46] !r [12:05:46] on is http://www.mediawiki.org/wiki/Special:Code/MediaWiki/$1 [12:06:03] !shellrequests is http://bit.ly/10eZZoa [12:06:03] Key was added [12:06:06] !sh [12:06:06] http://bit.ly/10eZZoa [12:06:09] :) [12:07:01] hey Dominiol [12:07:09] hi petan [12:07:32] you have a shell now [12:08:32] petan, do you have any ideas when to expect SQL replication? [12:08:46] Cyberpower678 only Ryan_Lane and binasher have [12:09:08] @notify binasher [12:09:08] I will notify you, when I see binasher around here [12:10:00] petan, I've got MediaWiki setup, I don't know where to start. [12:10:12] start with what [12:10:48] I mean, I don't what I page I should create first. [12:11:00] neither I do [12:11:16] Maybe the Getting Started Page [12:20:26] how i can login into shell? [12:34:30] what's host is available to connect on shell account? [12:34:44] (sorry poor english) [12:34:47] petan, curiousity, what is the execute permission? [12:35:22] that give you ability to browse folders and execute files [12:35:40] petan, then what does read do? [12:36:41] read bit allows the affected user to list the files within the directory [12:36:46] execute bit allows the affected user to enter the directory, and access files and directories inside [12:37:35] with +r only you should be able to ls the folder, but you won't be able to cd into it, neither access any files in it [12:37:58] petan, so say I just disabled read and left execute. Inside the folders are configuration files with passwords in them. Would anybody be able to access them? [12:38:57] addshore, do me a favor? [12:38:59] depends on the permission set on these files [12:39:21] if the folder is only +x and files are +r then it is likely possible to read them [12:39:43] but - it's not that simple as if the folder was +rx [12:40:09] because one wouldn't be able to read the list of files, so you would need to know the exact path, but I never really tried it [12:40:23] it's very uncommon to use only +r or +x on folders [12:40:36] If you go to the folder and tried it wouldn't let you but you can type the direct path to the file in them and still access them correct? [12:41:00] if the files have +r then /probably/ yes [12:41:06] this may differ per filesystem [12:41:32] Cyberpower678: addshore is afk right now. [12:41:49] When a program runs and access a file, does it use read or execute? [12:42:26] Read at the very least. [12:42:47] legoktm, do me a favor then? [12:43:01] sure? [12:43:42] goto /data/project/cyberbot/Peachy and try to enter configs [12:44:27] Cyberpower678 programs only need +r [12:44:56] legoktm, ^^ [12:46:11] local-legobot@tools-login:/data/project/cyberbot/Peachy$ cd Configs/ [12:46:11] -bash: cd: Configs/: Permission denied [12:46:39] Excellent [12:47:17] I had a security flaw. All of my passwords were exposed. [12:47:39] Thanks legoktm [12:47:44] np :) [12:48:30] legoktm, do you think I'm crazy for trying to revive a framework that was abandoned almost 3 years ago? [12:50:36] @notify Coren [12:50:36] This user is now online in #wikimedia-labs so I will let you know when they show some activity (talk etc) [12:50:58] Coren is there any guide to remove existing project [12:51:07] Cyberpower678: Here I am trying to get people to adopt a framework that was proposed in 2007. [12:51:25] legoktm, what framework is it? [12:51:38] https://www.mediawiki.org/wiki/Manual:Pywikipediabot/2.0 :) [12:52:55] legoktm: why are you promoting the evil re-write [12:53:07] * legoktm huggles Betacommand  [12:53:28] legoktm: it needs to burn in hell [12:53:52] Ah. The todo list is significantly smaller than my Peachy to do list. There are so many extensions from MediaWiki that I need to make compatible with Peachy. [12:54:09] I'm writing a getting started page. [12:55:27] Betacommand: When was the last time you even tried it? [12:55:33] There are sooo many benefits to using it. [12:56:01] less than a month [12:56:07] legoktm, I don't know Python. [12:56:21] I'll let you two argue. :p [12:56:44] Betacommand: Well if you file bugs or tell me specific problems in #pywikipediabot I can fix them :) [12:57:26] legoktm: easy bug fix, delete the rewire as it needs a complete re-write [12:57:55] I hope you realize that doesn't help you one bit. [12:58:14] legoktm: here is one major issue, requires install [12:58:24] no it doesnt [12:58:42] legoktm: what OS are you runing? [12:58:46] as long as pywikibot/ is in the same directory as you are, you're fine [12:59:04] *nix based systems [12:59:11] But the packaging is python standard [12:59:17] it works just fine on windows [13:00:00] legoktm: it throws user-config in some assine %APPDATA% path that is a real pain in the ass to avoid [13:00:38] so then why don't you file a bug about it? [13:00:44] legoktm: it does not run under windows [13:00:58] can you please file bugs on these things? [13:01:18] * Cyberpower678 is somewhat amused. [13:01:19] legoktm: do you know how many bugs Ill need to file? [13:01:31] easily 20+ [13:01:34] No I don't, but I'm ready to fix them [13:01:42] LOL [13:02:17] trunk works, and doesnt cause the headache of the rewrite [13:03:11] it works, but it can be much better [13:03:22] legoktm: I work on 5 different operating systems, trunk requires no hoops [13:04:05] I have yet to get rewrite working on a non-*nix system [13:05:08] Well I'll try and get a hold of a windows system [13:06:07] legoktm: what is the svn of rewrite? [13:06:47] https://svn.wikimedia.org/svnroot/pywikipedia/branches/rewrite/ [13:16:55] legoktm: error one: http://toolserver.org/~betacommand/rewrite/Error%201.png [13:17:07] I shouldnt need to modify my path vars [13:17:17] or use some -param every time [13:17:24] you dont [13:17:36] thats if you want to have multiple user-config.py's [13:18:13] legoktm: unless its in %appdata% you need to use either a param or or modify the path vars [13:18:26] check the screenshot [13:18:37] oh [13:18:40] i see what you mean [13:18:49] hm [13:18:56] Like I said its shit [13:18:59] The sleeper has awakened! [13:19:13] hey Coren [13:19:15] Coren is there any guide to remove existing project [13:19:18] Oh no! Coren's awake. Everybody run [13:19:20] petan: "Existing project" or "Existing tool"? [13:19:24] tool [13:19:24] good morning Coren [13:19:41] legoktm wanted to rename one tool to another one [13:19:51] well [13:19:52] so I just created a new one, and we moved data from old to new [13:20:00] now the old can be probably removed - not sure though [13:20:03] legoktm, addshore? [13:20:05] yeah [13:20:06] you can delete it [13:21:23] petan: It can be removed from the project page on wikitech [13:21:24] * Cyberpower678 is never using PyWikipediaBot. :p [13:21:34] ok what about the db and so [13:21:52] !log tools removing local-wikidatastats from ldap [13:21:53] Logged the message, Master [13:22:52] hey hashar, how is the private puppet repo organised on Labs? is it used at all? :) [13:24:03] petan: There's no provision for automatically deleting anything; I'll have to write a "cleanup" script eventually. [13:24:06] Cyberpower678: trunk works well [13:24:24] Betacommand, I'm no Python expert. [13:24:39] The best I can do is barely read it [13:30:28] MaxSem: it is used to fill the passwords:: placeholders in our puppet manifests [13:30:40] yeah [13:30:45] and where it is? [13:31:00] Betacommand, legoktm: Weren't you guys arguing about something? :p [13:31:00] is it zomgsecret?:) [13:31:08] MaxSem: labs/private.git [13:31:12] cause I need to add to it [13:31:22] MaxSem: I am not sure what is the permission scheme though [13:31:23] Cyberpower678: Now I'm thinking about how to fix it :) [13:32:32] MaxSem: https://gerrit.wikimedia.org/r/#/admin/projects/labs/private,access apparently anyone in the project-bastion project [13:32:56] yeah, i've just cloned it [13:32:57] thanks [13:33:21] Cyberpower678: I just threw one example of how broken the rewrite is and now legoktm needs to fix it, since he put his foot in his mouth and said he would [13:33:30] :D [13:33:44] I'm currently trying to get GeoData and EventLogging to work on Labs [13:33:52] :D [13:34:04] Cyberpower678: and I wasnt even looking for an issue, I just checked out the code and the first command gave me issues [13:35:02] Betacommand, are you familiar with PHP? [13:39:15] im here whoever was pinging me :P [13:39:30] me a while ago. [13:42:45] New patchset: MaxSem; "EventLogging credentials" [labs/private] (master) - https://gerrit.wikimedia.org/r/60839 [13:43:22] addshore I was [13:43:35] addshore, so was I [13:43:51] addshore I like to poing u [13:44:02] but I don't really need u :D [13:44:06] !addshore [13:44:06] lolz [13:44:19] !legoktm [13:44:19] shh [13:44:32] !petan [13:44:32] Petr Bena - http://enwp.org/User:Petrb - he is a bit septic about Coren's project :) [13:44:55] damn FBI knowz me [13:45:02] !maxsem is AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA! [13:45:02] Key was added [13:45:13] this bot is connected to databases of FBI [13:45:21] CIA and all that [13:45:35] * MaxSem reports petan to ZOG [13:47:26] !Cyberpower678 [13:47:35] meh [13:47:52] !Cyberpower678 is awesome! [13:47:52] Key was added [13:48:06] !Cyberpower678 [13:48:06] awesome! [13:51:07] Betacommand, you never answered my question. :p [13:51:51] Cyberpower678: I can read PHP but not write it [13:52:00] Ah, [14:05:53] petan: did you make 'proposals' go away? :) [14:06:10] ? [14:06:20] where [14:06:35] on labs [14:10:40] got no idea what u mean [14:10:47] is that a page or project or huh [14:11:10] https://wikitech.wikimedia.org/wiki/Nova_Resource:Proposals [14:11:42] The whole project is now not needed :) [14:11:55] * Cyberpower678 deletes addshore  [14:11:59] Oops [14:12:00] oh... I don't have permissions to remove projects or I don't know that :P [14:12:03] you need to ask Ryan_Lane [14:12:17] he is a big boss here :P [14:12:35] * Cyberpower678 restores addshore from a backup [14:12:44] :p [14:12:57] will do :) i try to avoid bugging ryan where I can and bug you and coren instead ;p [14:13:08] Actually, I think deleting projects is troublesome/complicated and needs some black magic. [14:13:17] What about creating new ones? [14:13:25] addshore and I need one created [14:13:29] ryan does that too ;p [14:13:31] legoktm: That part I can do. :-) [14:13:41] ahh :D [14:14:05] * Cyberpower678 uploads addshore to Wikipedia and watches what happens. :p [14:14:21] O_o [14:14:21] legoktm: What's needed? [14:14:22] Ok, is the name "stats" available? [14:14:43] legoktm: Yes. [14:15:00] Coren: Well we're trying to set up limn, but thats not going to be possible on tools it seems since its running its own nodejs webserver [14:15:49] Also we'd need a public ip and domain name…can you do that too? [14:16:25] addshore: Stats exists now [14:16:29] =] [14:16:33] :) [14:16:51] legoktm: I think I have the rights to up the IP quota; sadly, I have no idea /how/ it's done. That'll need Ryan or Andrew [14:17:17] :D [14:17:35] Coren: give it a try by trying to remove the public ip from 'proposals'? ;p [14:18:05] addshore: if I do, I'm not going to be able to allocate it back to stats; its quota is currently 0 [14:18:34] i mean try and change the quota on proposals? :P [14:19:22] addshore: I have no idea how to do that. :-) It's done by hand on the backend, AFAIK, and doesn't seem to be documented anywhere. I'll bug Ryan for a HOWTO. :-) [14:19:54] ahh ;p [14:20:12] hi, sorry, I'm late to the party. Anyone mention Limn? [14:20:17] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 0 minutes) [14:20:18] hi :) [14:20:26] that would be addshore and I [14:20:46] :D [14:21:01] how..? xD [14:21:36] hi addshore and legoktm [14:21:40] how can I help? [14:21:42] hi milimetric :) [14:21:45] I'm one of two Limn maintainers [14:23:04] well right now we're just waiting to get a public IP so we can set it up [14:23:32] right now we have a bunch of charts on stathat that we want to move to labs [14:23:38] oh ok [14:23:44] are the charts in Limn already or? [14:23:51] nope [14:24:14] stuff like https://www.stathat.com/stats/D6QK which I think we can download csv's of [14:24:50] ok, cool. Well, we're working on documentation, but someone has a draft about adding datasources and graphs to a running Limn instance here: [14:25:01] https://wikitech.wikimedia.org/wiki/EE_Dashboard [14:25:30] it's specific to their instance but we'll genericize it. [14:25:55] but i guess first we'll have to set up the Limn instance. You guys wanna join me in #wikimedia-analytics? [14:25:58] legoktm: You can still set it up while you're waiting for the public IP, can't you? [14:26:08] yep [14:26:22] right. [14:26:43] Coren: I'm not listed as a member of the project on https://wikitech.wikimedia.org/wiki/Nova_Resource:Stats ? [14:27:03] legoktm: Beg addshore to let you in? Think it'll work? :-) [14:27:10] * legoktm stabs addshore  [14:27:16] * addshore dies [14:27:22] RIP [14:27:52] oh no! [14:28:10] man. now i lost another Limn customer [14:28:11] :( [14:28:30] legoktm: done :) [14:28:47] addshore: great. you wanna create the instance? [14:30:08] Wooooow. You didn't even make me an admin :x [14:30:36] legoktm: i did :O [14:30:41] o.O [14:31:07] not listed https://wikitech.wikimedia.org/wiki/Nova_Resource:Stats but it looks like i can create instances so [14:31:09] make sure your looking at stats and not local-stats [14:31:38] i think teh admin lists are hackily added to those pages :P [14:31:43] so it might take a while for you to appear ;p [14:31:51] milimetric: is limn puppetized? [14:31:54] yes [14:32:05] ottomata was just about to school us all on the ways of puppetized limn [14:32:16] :P [14:32:49] yeah so, its real easy to set up a limn instance [14:32:50] hi guys! [14:33:04] but you'll need to do a weeee little puppet [14:33:09] there's a readme that has some docs [14:33:22] addshore: Created instance i-000006e0 with image "ubuntu-12.04-precise" and hostname i-000006e0.pmtpa.wmflabs. [14:33:23] https://github.com/wikimedia/operations-puppet/tree/production/modules/limn [14:33:34] lovely :) [14:33:49] !log stats legoktm Created instance i-000006e0 with image "ubuntu-12.04-precise" and hostname i-000006e0.pmtpa.wmflabs. ;p [14:33:51] Logged the message, Master [14:33:53] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 13 minutes) [14:34:55] ok, doesn't seem that difficult ;) [14:35:02] so once you have that set up addshore/legoktm, then you'll need to deploy limn to that instance. [14:35:42] the puppet doesn't deploy it because of some restrictions on deploying to production boxes [14:35:55] so it's decoupled in this fabric project: https://github.com/wikimedia/limn-deploy [14:36:09] oh, I didn't know that existed, does that weork? [14:36:30] yea, of course, that's how I deploy these dozen or so limn instances we have now [14:36:31] :) [14:36:45] you guys can also just clone limn to your box as well, as the puppet module README says [14:36:47] either way [14:36:47] just clone that, sudo pip install fabric, and let me know where your data repository is so I can set up a stage for you [14:37:40] yeah, you can do that, and make sure the data is at that path described in the README [14:37:55] addshore: [14:37:58] legoktm@bastion1:~$ ssh limn001 [14:37:59] Permission denied (publickey). [14:38:10] mhhm lemme try [14:39:09] milimetric: so we just need to use the fabric limn-deploy tool? [14:39:30] yes, but you still need ssh access [14:39:44] that deploy tool basically runs all the commands you need over ssh [14:39:46] legoktm: {{worksforme}} also legoktm probably should have called the instance stats-limn001 ;p [14:40:12] oh oops [14:40:14] i got in [14:40:38] so the fabric deployer is run like this: "fab your_stage deploy" [14:40:52] for that to work, I've gotta set up "your_stage" in fabric/stages.py [14:40:56] !log stats installing pip [14:40:57] Logged the message, Master [14:41:21] do you guys have a data repository yet? [14:41:41] data repositories look like this: https://github.com/wikimedia/limn-editor-engagement-data [14:41:42] !log stats pip install fabric [14:41:43] Logged the message, Master [14:41:57] milimetric: no.... [14:42:26] ok, so I guess do you want to version control your graphs/datasources/etc.? [14:43:34] well so our goal was that we would be able to create new graphs and add datasources on the fly [14:43:46] maybe a simple python cgi script to create the necessary files [14:45:35] oh, creating datasources and graphs can be done from the UI [14:45:48] but the question is, once you create them, would you like to version them [14:46:15] let's just say no for now and I can show you how to do that later :) [14:46:20] sure :) [14:46:51] ok :) so once you get limn configured through that puppet, let me know what instance it's on / what folder you put it in, and I'll commit a new stage for you [14:46:57] should I call it stats_limn001? [14:47:25] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 27 minutes) [14:47:28] * addshore pokes legoktm to get rid of the instance he has and name things prepended with stats- ;p [14:47:37] addshore: fine [14:47:45] ;p [14:47:53] oh, and I need to know what URL it's going to serve at (because I have to register the instance with google for authentication) [14:49:56] something like stats.wmflabs.org? :) [14:50:21] !log stats deleted limn001 and created stats-limn001 so addshore will be happy [14:50:22] Logged the message, Master [14:50:34] :D [14:51:46] yes addshore, exactly [15:00:53] !log stats installed python-pip and pip install fabric on the new instance [15:00:54] Logged the message, Master [15:00:59] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 40 minutes) [15:03:04] ok so now.... [15:04:03] !log stats sudo pip install -U Fabric paramiko path.py [15:04:04] Logged the message, Master [15:04:16] ok one sec legoktm [15:04:38] so what's the instance called that you used with the puppet install? [15:04:41] ugh [15:04:43] not there yet [15:05:08] need to install python-dev so we can get the C libraries so we can install pycrypto which is a dependency of Fabric. [15:05:17] oh! [15:05:18] sorry! [15:05:22] you only need that locally [15:05:25] not on the labs instance [15:05:29] i'm sorry! [15:06:24] o.O [15:06:26] ok [15:06:50] lemme do that then [15:08:24] milimetric: Please select a staging target ['prod', 'test', 'gp', 'dev', 'mobile', 'mobile_dev', 'ee_dashboard', 'debugging']: [15:08:36] yep, so now i'm setting up your stage [15:08:40] and will push that [15:08:43] ok :) [15:08:46] so - what are you going to call your instance? [15:09:03] (i need that to set up the stage) [15:10:03] instance is called stats-limn001 [15:10:10] but the url would probably be stats.wmflabs.org [15:10:35] no, this is the instance name mentioned in the puppet docs: https://github.com/wikimedia/operations-puppet/tree/production/modules/limn [15:10:40] in that case it's "reportcard" [15:10:58] so in this case you could set it to "stats", does that work? [15:11:03] oh. yeah sure [15:12:11] ottomata: does puppet expect limn to be at /usr/local/share/limn or /usr/lib/limn? [15:14:30] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 54 minutes) [15:15:08] oh i see it in instance.pp, i assume that's what's deployed [15:15:20] (it's /usr/local/share/limn) [15:18:53] ok legoktm, I've made the stage so now the command would be "fab stats_limn001 deploy" [15:19:04] so you can git pull that limn-deploy repo [15:19:06] and it'll be there [15:19:57] oh, and what's your email address? I have to send you the "secret" config file you need [15:20:13] legoktm.wikipedia@gmail [15:20:41] does fabric assume that my local machine's username is the same as my remote username? because thats not the case [15:21:13] [stats-limn001.pmtpa.wmflabs] Login password for 'km': <-- not really sure which password it wants... [15:22:07] yes, i think it does [15:22:20] -u USER, --user=USER username to use when connecting to remote hosts [15:22:21] aha [15:22:21] erosen found an option to pass it the username, I think it was -u? [15:22:22] yea [15:22:23] lemme try that [15:23:17] http://dpaste.de/s8Keh/raw/ [15:23:47] do we need to create that user manually? [15:24:49] no [15:24:55] the puppet does all that [15:25:07] right, so that's why we need to do the puppet stuff first [15:25:24] btw, thanks for this - the more people go through this process the more I know about how to make it better [15:26:17] :D [15:26:21] yay improving processes! [15:26:32] legoktm: you fix bug 1 yet? [15:26:57] Betacommand: not yet, got distracted with other stuff. I'll ping you when I do :) [15:27:54] milimetric: ok, i thought the deploy tool did that. lemme see how to set up the puppet stuff... [15:27:55] legoktm: dont worry Ill keep you busy with issues for the next few years [15:28:00] :D [15:28:07] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 68 minutes) [15:28:27] i gotcha, yeah, the puppet *used* to do most of this stuff for you [15:28:35] but we're not allowed to do that in production so we had to take it out [15:29:35] so for the puppet setup, you just have to follow the instructions at: https://github.com/wikimedia/operations-puppet/tree/production/modules/limn [15:29:40] and use "stats" as the instance name [15:29:50] ok [15:29:50] let me know if you hit trouble [15:30:05] !log stats sudo git clone https://github.com/wikimedia/limn.git /usr/local/share/limn [15:30:06] Logged the message, Master [15:32:40] milimetric: does http://dpaste.de/wCr7q/raw/ look right? [15:36:05] looks good to me legoktm [15:36:20] but i'm a total puppet noob [15:36:25] heh, me too [15:36:35] so how do i get puppet to actually run the file? [15:38:22] ottomata: once they've set up class role::analytics::stats, how do they use it on that box? [15:38:53] ok and i got your email and set up the file [15:39:06] cool, that should be good [15:39:21] we just have to wait for ottomata to help with the puppet and we're good to go ;) [15:39:29] You can add your class in labs console [15:39:58] Manage Puppet Groups -> check your project -> submit [15:40:23] Add group (if you need to). you can give it an arbitrary name [15:40:25] then [15:40:29] Add class [15:40:34] role::analytics::stats [15:40:37] or whatever your class is [15:40:38] then [15:40:40] go to [15:40:44] Manage Instances [15:40:56] and click 'configure' in the actions column for your instance [15:41:06] check the box to include your new role class [15:41:07] and submit [15:41:10] where should the puppet file be saved? right now its just in ~/puppet/thing.pp [15:41:11] then run puppet on the node [15:41:32] are you using self-hosted puppetmaster? [15:41:37] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 81 minutes) [15:41:46] im not sure…? [15:41:49] <-- puppet noob [15:41:52] then probably not [15:41:57] ok you have to do that first :p [15:42:07] on the configure instance page [15:42:13] actually, i'll just send you the instructions [15:42:19] :D [15:42:27] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [15:42:36] do that first [15:42:43] once you do that [15:42:47] you can edit files in /etc/puppet/manifests [15:42:50] i would just create a new file [15:42:51] in [15:43:11] /etc/puppet/manifests/role/whatever.pp [15:43:15] and add your class there [15:43:27] i guess cp your ~/puppet/thing.pp there [15:43:44] ok [15:44:36] !log stats enabled the puppetmaster::self class, now going to force a puppet run [15:44:38] Logged the message, Master [15:49:02] addawa, you there? [15:49:14] addawa? [15:49:26] lol [15:49:53] !log stats supposedly enabled role::analytics::stats, now forcing a puppet run [15:49:55] Logged the message, Master [15:51:13] err: Failed to apply catalog: Could not find dependency Package[apache2] for File[10-limn-stats.conf] at /etc/puppet/modules/limn/manifests/instance/proxy.pp:35 [15:51:23] im guessing i need to enable that... [15:52:03] ottomata, milimetric: webserver::apache2? [15:52:33] yeah, i think there's something about that in the readme, one sec [15:52:54] Requirements [15:52:54] puppet-labs' apache puppet module. [15:52:58] ok [15:52:59] right [15:53:03] hmmm [15:53:35] ah yes [15:53:36] so [15:53:48] i tried to remove as many deps as possible [15:53:52] this installs a vhost file for you [15:54:01] but doesn't try to manage however you set up apache [15:54:39] do i need to enable everything in the apache group? [15:54:39] err: Failed to apply catalog: Could not find dependent Service[apache2] for File[10-limn-stats.conf] at /etc/puppet/modules/limn/manifests/instance/proxy.pp:35 [15:54:47] you could do this in a coupel of ways [15:54:47] do ummmm [15:54:59] include webserver::apache [15:54:59] webserver::apache::module { ['rewrite', 'proxy']: [15:54:59] require => Class['webserver::apache'] [15:54:59] } [15:55:03] in your role class [15:55:17] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 95 minutes) [15:55:42] does it matter in what order that part is included? [15:55:46] no [15:55:51] puppet is (for the most part) orderless [15:56:02] it will order things based on explicit dependencies [15:57:14] ok [15:57:18] running that now [15:57:58] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined in file /etc/puppet/manifests/webserver.pp at line 128; cannot redefine at /etc/puppet/manifests/webserver.pp:98 on node stats-limn001.pmtpa.wmflabs [15:58:05] lemme disable the other one [15:58:58] wait wha [15:59:33] i unchecked the webserver::apache2 box on wikitech and it seems to be going well now [15:59:40] ohh ohohohoh [15:59:42] ok that's fine [15:59:57] actually checking the boxes is probably the better way to do it [16:00:02] but if this works you can leave it [16:00:06] either way is fine [16:00:08] it did :P [16:00:22] milimetric: so now the fabric deploy? [16:00:39] one sec [16:00:54] yeah, let's try it [16:01:13] i would've thought stats.wmflabs.org would've been up (on the apache but serving nothing) [16:01:26] but let's deploy it and see [16:01:41] well we dont have a public ip yet [16:01:43] [stats-limn001.pmtpa.wmflabs] out: /bin/bash: npm: command not found [16:01:46] lemme install nodejs [16:01:57] wait [16:02:09] * legoktm waits [16:02:18] you'll probably want to do it from chris lea's repository instead of the official [16:02:25] the official is like several years behind or something crazy [16:02:35] one sec [16:03:22] ubuntu instructions here: https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager [16:03:27] Guys [16:03:31] npm won't work without a public IP [16:03:43] Oh wait, this is labs [16:03:46] Never mind, carry on [16:03:47] yep [16:03:53] thanks though :) [16:04:19] btw legoktm, I'm keeping judicious record of all this pain: https://mingle.corp.wikimedia.org/projects/analytics/cards/631 [16:04:23] and the idea is to reduce [16:04:26] the pain [16:04:27] :) [16:04:46] oh great :D [16:05:21] milimetric: "I'm keeping judicious record of all this pain" sounds like a Mountain Goats song lyric (and I approve) :) [16:05:27] nice :) [16:05:31] love the mountain goats [16:05:46] ok i did all that and [16:05:47] nodejs is already the newest version. [16:06:25] cool, and install npm too [16:07:07] {{done}}. gonna re-run fabric now [16:08:49] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 108 minutes) [16:09:40] milimetric: http://dpaste.de/Ivw0w/raw/ [16:10:09] well that's good/bad [16:10:17] means it installed but i got the wrong name for the instance [16:10:32] maybe it's just "stats" instead of "limn-stats"... [16:10:51] if you know upstart you can check for me, otherwise I'm logging in and figuring it out [16:11:10] i dont know what upstart is.... [16:11:28] do you want me to add you to the stats project? [16:11:34] oh yea that would be great [16:12:52] ok done [16:12:57] k, logging in [16:13:42] ssh milimetric@stats-limn001.eqiad.wmnet [16:13:46] that gave me permission denied [16:13:52] er its on pmtpa [16:13:57] oh [16:13:59] pmtpa.wmflabs [16:14:06] doh sorry [16:14:10] i've been working in prod cluster [16:14:22] :P [16:15:24] hello [16:15:34] huh... weird [16:15:37] i see limn-stats there [16:16:09] hi Pyfisch [16:16:51] ls -l /etc/init/limn* [16:16:53] my bot is working :) [16:16:56] limn-stats there? [16:17:06] yep [16:17:08] i started it [16:17:16] it must just be the fabric deployer legoktm [16:17:24] "start limn-stats" worked [16:18:57] ok :) [16:19:09] i'm running the deployer again to see what's wrong but you should be all set [16:19:28] curl localhost:8081 works as I'd expect it, so once you get your public IP you'll have your instance up [16:21:36] awesome [16:22:26] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 122 minutes) [16:22:55] so i think you mentioned earlier that we can create new graphs straight from the web interface.... [16:22:57] k, pushed a fix to the limn-deploy project too legoktm [16:23:03] oh great :) [16:23:17] yeah, for some reason this box didn't like sending "service limn-stats stop" over ssh [16:23:22] but was fine with "stop limn-stats" [16:23:24] go figure [16:23:27] :P [16:23:27] locally, they both work [16:23:32] ok, so graphs from the UI [16:23:40] wanna do a quick google hangout and i can show you on my local instance? [16:23:45] just so you know how to work it? [16:24:25] oh, or you can just read this: https://wikitech.wikimedia.org/wiki/EE_Dashboard [16:24:29] well would it be possible to do that server-side with a script if all of our charts are going to be time vs int values? [16:25:04] hm [16:25:10] graphs are basically json objects [16:25:27] the objects are trees of graph "nodes" [16:25:51] and the nodes which are responsible for drawing lines reference your datasources [16:26:04] you could script it all, yea, there's already a project called limnpy [16:26:20] https://github.com/wikimedia/limnpy [16:26:24] erosen wrote this [16:26:46] you can basically use it to automatically generate limn datasources and graphs from datafiles [16:27:02] and it does some pretty fancy stuff with pivoting too, but doesn't sound like you'll need [16:27:16] huh. [16:27:23] so either way - ping us in #wikimedia-analytics once you get started (i'm not always in this room and I think we've spammed them enough) [16:27:30] erosen's in there too and he can tell you more about limnpy [16:27:40] he uses it to maintain this: http://gp.wmflabs.org/ [16:28:08] oh thats cool [16:28:22] :) though it does look as though some of those graphs are broken :) I'll tell him [16:28:30] k, signing out for now, got lunch and meetings to tend to [16:28:34] later legoktm! [16:28:35] O_O. I'm away for 3 minutes and I return to a flood. [16:28:41] thanks for all the help :) [16:29:48] legoktm, whenever I see your username, I think about my childhood. [16:30:02] * legoktm <3s legos [16:30:15] * Cyberpower678 does too. [16:31:05] What's ktm though? [16:31:12] my initials [16:31:30] ohhhh. [16:35:56] Warning: There is 1 user waiting for shell: MarcoAurelio (waiting 135 minutes) [16:57:54] Is there any public DB replication to labs yet? [16:59:06] Nope....... [16:59:17] last eta I heard was by amsterdamn [16:59:26] amsterdam* [16:59:31] 1 month ) [17:00:10] I've wanted to run a couple of SQL queries recently but have no TS access :( [17:00:48] addshore, did you look at the code yet? :) [17:01:08] nope :D [17:01:17] addshore, :( [17:01:17] Krenair: I can run them for you if you want [17:01:23] or just request access? :P [17:01:28] Cyberpower678: im not going to be able to for a month :< [17:01:52] That's what you said 3 weeks ago? :p [17:01:52] legoktm, request access to TS? I don't intend to create/host tools... [17:02:07] sorry cyber so so busy :< [17:03:47] addshore, when you're busy, you're busy. [17:04:03] addshore, http://tools.wmflabs.org/peachy/wiki/index.php?title=Help:Getting_Started my first one [17:09:35] Krenair: I can run them for you [17:10:56] legoktm just ran some for me, thanks anyway betacommand :) [17:11:34] Krenair: any time [17:12:00] * Betacommand lost count of the number of reports he creates [17:12:31] * Cyberpower678 is proceeding to transition more tasks over to labs. [17:13:52] mhhm how do i loginto https://graphite.wikimedia.org/ ? [17:23:44] OperationalError: (2006, 'MySQL server has gone away') [17:23:56] is this normal on labs? [17:25:07] um [17:25:25] had you opened a connection, and then not used it for a while, and then tried to access it again? [17:30:21] legoktm: thats possible [17:32:33] Pyfisch: stale connections are probably closed after a period [17:32:58] ^ yeah [17:39:29] Hey, labsfolk [17:39:53] parsoid-spof.pmtpa.wmflabs won't let cscott in, fails at creating his home directory [17:40:15] Probably related to the fact that we can't write to our home directories anymore, fails with a message about it being a read-only filesystem [17:40:24] gluster fun [17:49:23] well, without a working parsoid login I can stay blissfully innocent and make marktraceur do all the server work ;) [17:49:28] Boooo [18:01:33] !log tools Begin maintenance (login disabled) [18:01:35] Logged the message, Master [18:02:22] * Betacommand wonders if his bots will re-start [18:03:01] Betacommand: They are supposed to. [18:05:23] Coren: If I hand a dime every time I heard that and it didnt work I would have a nice nest egg [18:06:24] :P [18:06:49] here [18:10:19] Coren you disabled even -dev o.o [18:18:40] !log tools petrb: we are getting in troubles with memory on tools-db there is only less than 20% free memory [18:18:42] Logged the message, Master [18:19:59] petan: Probably not: I did configure mysql to eat ~75% of the memory for internal buffers; start worrying if that number /changes/ :-) [18:20:52] well it is actually about 10% it was my guess [18:21:02] 87% precisely used by mysql [18:21:27] thing is that there is only 1.2g of free memory, and if gluster or something went mad, mysql might get killed :o [18:21:58] Yeah? We know gluster is insane. That's why I'm ripping it out. :-) [18:30:12] Coren, petan [18:30:13] [14:29:22] Connection attempt 5... [18:30:13] [14:29:22] Resolving host name "tools-login.wmflabs.org" [18:30:13] [14:29:22] Connecting to 208.80.153.224 Port: 22 [18:30:14] [14:29:23] SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1 [18:30:14] [14:29:23] Starting SSH session. Remote Id: "SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1" [18:30:15] [14:29:23] SSH protocol version reply. Client Id: SSH-2.0-SmartFTP [18:30:17] [14:29:23] Key Exchange Algorithm: ecdh-sha2-nistp256 [18:30:19] [14:29:23] Server "ssh-rsa" host key fingerprint: 95:48:D5:F6:7D:E5:E8:C4:2D:96:88:2A:A0:89:40:28 [18:30:21] [14:29:23] Key exchange completed. [18:30:23] [14:29:23] Host Key Algorithm: ssh-rsa [18:30:25] [14:29:23] Client to Server Encryption: aes128-ctr [18:30:27] [14:29:23] Server to Client Encryption: aes128-ctr [18:30:29] [14:29:23] Session MAC: hmac-sha1 [18:30:31] Cyberpower678 hold ur fire [18:30:31] [14:29:23] Client to Server Compression: zlib@openssh.com [18:30:33] [14:29:23] Server to Client Compression: zlib@openssh.com [18:30:35] [14:29:23] Requesting service "ssh-userauth". [18:30:37] [14:29:23] RTT: 124.984 ms [18:30:39] [14:29:23] Authentication request. Method: none [18:30:41] Yo! No need to paste your whole log! [18:30:43] Cyberpower678 there is maintenance [18:30:43] [14:29:23] Received authentication banner message. [18:30:45] [14:29:23] Server supported authentications: publickey [18:30:46] Use pastebin dammit [18:30:47] [14:29:23] Authentication request. Method: publickey [18:30:49] [14:29:23] Received authentication banner message. [18:30:51] [14:29:23] Server closed connection [18:30:53] [14:29:23] Connect failed. Error=0x80043112 [18:30:58] That's way more than I wanted to post. :p [18:31:04] lol [18:31:15] I only wanted to post the connect failed message. :p [18:31:22] Your client would be more useful if it actually showed you said authentication banner message. :-) [18:31:22] Cyberpower678 there is maintenance [18:31:33] _______________ [18:31:33] | | Sorry, the tools project is in maintenance from [18:31:33] | Maintenance | 2013-04-25 18:00 UTC to 2013-04-25 19:00 UTC and [18:31:33] |_______________| logins are disabled during this period. [18:31:33] [18:31:42] lol u both :D [18:31:49] * Krenair facedesk [18:31:59] :P [18:35:39] Coren, I'm trying to transition. :p [18:38:03] Cyberpower678: get used to it, maintenance always happens at the worst time for you [18:38:56] At least it's only another 22 minutes. [18:39:09] Unlike toolserver which usually takes 22 days. :p [18:41:24] Coren: Protip https://en.wikipedia.org/wiki/Box-drawing_characters [18:47:20] Cyberpower678: 22? must have been a good month [18:47:49] :/ [18:48:43] legoktm, I've got a lego battleship, a batmobile, and 2 hi-tech trucks. [18:48:50] All lego [18:49:12] I want to get the deathstar, [18:50:03] Krinkle|detached: Ping when you return, please. [18:56:42] 4 more minutes. [18:58:07] * Cyberpower678 wonders why his processor core number 4 is so active while all others are idle. [18:58:49] Hm. Not all tools cleanly restarted. How sad. Yours apparently did, Betacommand :-) [18:59:15] Coren: I saw [18:59:52] Damn. My FTP client is consuming 1.5GB of RAM on my computer. [19:00:13] mine didnt :( [19:00:16] !log tools Maintenance over; systems restarted and should be working. [19:00:17] Logged the message, Master [19:00:24] coren but I still get a nickel :P [19:01:06] Connection Established [19:01:07] The morebots clearly were resistant to being restarted. Is teh good. [19:02:15] Oh, wait. I note that two of those that didn't restart were /not/ on the continuous queue. [19:02:49] ... and the other one I knew about and was known to be unrestartable because of a username change. [19:02:53] * Coren wins. [19:02:58] if you just jsub script, that is continuous right? [19:03:00] I want the nickel back, Betacommand. :-) [19:03:15] duh: Not unless you specify -continuous [19:03:21] oh [19:03:23] lemme fix that [19:03:25] duh: jstart implies it, not jsub. :-) [19:03:33] Coren: how does one "become" via WinSCP? [19:03:33] blah [19:03:43] ok [19:03:43] thanks [19:04:42] Betacommand: ... hm. I don't know of a way. There's an easy workaround since you can write directly to /data/project/xxx as your normal user, or you can copy to your own home and move the stuff to the tool afterwards. [19:05:16] Betacommand: But I haven't found a satisfactory method to do it cleanly using SFTP/SCP clients. [19:09:55] Coren: how would one host multiple tools that share a common code base without duplicating the code across every tool ? [19:11:14] * Betacommand would normally just throw the path into the script environ but that wont work on labs [19:14:31] Betacommand: No, that'll work to as long as you set the permissions right to allow all the tools to reach it. [19:15:09] Betacommand: It might be a bit more complicated if you need to share the code between the tools /and/ restrict it from anyone else, but that doesn't sound like a good idea anyways. :-) [19:16:16] legoktm: BTW, that worked right; your bots are now set to restart (note the queue they are on) [19:17:21] Coren: thats kinda what needs to happen, config files contain private information (wiki login data) that should be shared across tools (this is why I just throw everything up under the same account normally) [19:18:47] Hm. [19:19:48] Well, that can be done, actually, provided you're comfortable playing with permissions. [19:21:09] It does take a bit of trickery to set up, but it's doable. If you want, I'll sit down with you and show you how sometime later today once I'm done with the maintenance and followup? [19:26:08] Betacommand: Although, strictly speaking, if your individual tools are so intertwined as to share config, credentials and maintainers, it's arguable that they aren't really deserving of seperate tool accounts either. [19:26:58] (It only make sense to make tools accounts for things separate enough that they could be maintained independently of each other. [19:27:01] ) [19:27:01] Coren: thats what I wasnt sure about [19:27:22] Coren: most of my code is interdependent [19:28:18] I suppose that's the best question to find the line where to create a tool account: "Could someone take this and maintain it" [19:28:49] So, for most of your tools "bits", it might be simplest to create just the one "betastuff" account. [19:29:48] Coren: let me rephrase that, a lot of my code re-uses parts from other scripts Ive written :P [19:31:34] If needed any one part could be isolated and run independently. But maintaining such a monster with that many instances it becomes a management headache [19:31:52] Hm [19:32:09] So, basically, you have a swiss army knife with detachable blades. :-) [19:32:16] yeah [19:32:40] Coren: how are we going to handle things like wsgi and node? [19:32:59] suphp isn't going to work for that [19:33:41] or hell, java (eww) [19:33:47] Ryan_Lane: I'm working on a few prototypes for wsgi that deploys the actual server to the grid and uses the CGI wedge to talk to it. It works fine, atm, but needs a lot of manual tweaking. [19:34:09] being able to proxy directly to them would work as well, assuming we can set the ports [19:34:26] (this is another reason virtual hosts would be nice ;) ) [19:34:46] Ryan_Lane: That's prototype B. I'm already doing per-tool proxy redirection, there is nothing that makes it impossible to be a bit more dynamic about what I'm proxying /to/ [19:34:55] * Ryan_Lane nods [19:35:46] (i.e.: rather than proxy webserver-xx:80, for tool Y, proxy to $dynamically-determined-server-on-the-grid [19:35:55] yep [19:40:47] Ryan_Lane: Not going to do virtual hosts until I can get 2.4 running so I can use mod_privileges [19:41:07] why not mod-itk? [19:41:58] Ryan_Lane: And I'd like to do a /lot/ more testing of 2.4 on Precise before we bring it back in. [19:43:23] again, why not mod-itk? :) [19:44:54] I've had bad experiences with running it. :-) It's filed in my "last resort" column. I suppose (or at least *hope*) it has gotten better in the past couple years, but it used to be brittle, unscalable, and not very secure. [19:45:18] * Ryan_Lane nods [19:45:22] And, to be honest, I *really* dislike the idea of apache running uid 0 [19:47:58] Besides, it's a non-issue for things like WSGI, node or java servers; those end up being proxied to anyways. [19:48:04] yeah, not a huge fan of that either [20:11:37] [bz] (RESOLVED - created by: Ryan Lane, priority: Low - normal) [Bug 43508] Run a script to add all members of the shell mediawiki group to the bastion project - https://bugzilla.wikimedia.org/show_bug.cgi?id=43508 [20:12:36] Ryan_Lane, if you've decided against https://bugzilla.wikimedia.org/show_bug.cgi?id=43650 isn't it wontfix instead? :) [20:13:00] heh [20:13:00] sure [20:13:33] [bz] (RESOLVED - created by: Ryan Lane, priority: Low - normal) [Bug 43650] Switch instance bootstrapping to use salt - https://bugzilla.wikimedia.org/show_bug.cgi?id=43650 [20:16:02] Coren, new crontab file is missing newline before EOF, can't install. [20:16:02] This is new to me. [20:16:56] Coren: how's the performance of the NFS vs gluster for tools, now? [20:21:44] Coren? [20:21:50] petan, ? [20:21:56] yes [20:22:06] Coren, new crontab file is missing newline before EOF, can't install. [20:22:06] This is new to me. [20:22:17] Coren add new line to end of file [20:22:40] oh, Cyberpower678 [20:22:42] not Coren [20:22:51] Cyberpower678 that was meant to u [20:22:55] just add one [20:23:13] Add ehat? [20:23:19] *what [20:24:32] Ryan_Lane: It goes *woosh*. Then again, tools is the only project on it, so it doesn't mean all that much yet. [20:25:21] heh [20:25:22] indeed [20:25:33] we should switch it for deployment-prep as well :) [20:25:45] did you do the local override via puppet? [20:25:48] or manually? [20:26:50] Ryan_Lane: Manually; but it's all of four files and a rc.local line. We're still arguing the color of the puppet bikeshed for tool labs, remember? :-) [20:27:54] rc.local?? [20:28:22] what went into rc.local? [20:28:39] I can [20:28:39] err [20:28:39] Ryan_Lane: Yeah, ugliness caused by NFS4; we have to manually turn on the use UID and not username switch which is default to off. [20:28:51] what do you mean? [20:28:54] echo 1 >/sys/module/nfs/parameters/nfs4_disable_idmapping [20:29:12] there should be a way to do that without rc.local [20:29:15] sysctl? [20:29:22] It's not there that I saw. [20:29:33] or /etc/default/ [20:29:39] If I did everything correctly, everything but adminstats should be transitioned. [20:29:40] rc.local is evil as hell [20:30:08] I know it is; it's a workaround. Honestly, I'm still stunned there is no sysctl for it. Perhaps an upstart script? [20:30:18] usually this is in a config file [20:30:54] I saw no way of setting this to on anywhere except by poking straight into /sys [20:31:05] usually /etc/default/nfs-common [20:31:15] Nope. Nothing there for that. [20:31:32] /etc/idmapd.conf ? [20:31:52] Ryan_Lane: Nope. The kernel has to know not to use it. Trust me, man, I looked everywhere. [20:31:59] the domain for the export and the domain set in idmapd just need to be the same [20:32:05] it doesn't need to be disabled [20:32:19] Ryan_Lane: That won't help -- the local-xx users do not, /can/ not exist on the NFS server. [20:34:02] are you using AUTH_SYS? [20:35:52] Coren is there some docs on how this nfs server is implemented [20:36:03] is it external server? [20:36:07] where it runs? [20:36:11] I mean dedicated hw [20:36:35] it's dedicated hardware [20:36:37] also that hostname it has for some reason doesn resolve to me to ip [20:36:38] it's labstore3/4 [20:36:42] oh [20:36:46] we shrunk all of the gluster volumes for this [20:36:53] aha [20:37:01] what filesystem it uses? [20:37:07] XFS, I'd imagine [20:37:19] btw since i migrated all my boxes to btrfs I experienced huge performance improvements [20:37:21] Ryan_Lane: Yes, auth_sys, but idmapping only works iff the users exists on both sides, and the local-xx users do not (and are not unique) [20:37:31] It's XFS [20:38:06] petan: did you actually take measurements before and after? [20:38:56] well, that is not easy to do, I didnt convert the fs in order to benchmark it, and now it is too late to go back to original fs and do some tests, but I am pretty sure there are some reliable benchmarks out there [20:39:08] btrfs uses sparse files and all that [20:39:16] well, then you don't know if there was any performance gain at all, then [20:39:19] :) [20:39:39] the system just appear to me much faster [20:39:39] s/, then// [20:39:42] that could be related to lots of things [20:39:46] the ldap upgrades I did [20:39:49] even my laptop where I use it now seems faster than it was before :P [20:39:59] well I am not talking about labs [20:40:05] but my personal computers :D [20:40:14] I dont use btrfs on labs very much [20:40:21] in fact I use it just on bots... [20:40:23] things "feeling" faster could be psychological, though ;) [20:40:30] indeed [20:40:38] you need to actually benchmark things to know for sure [20:40:42] \ignore petan [20:40:46] meh [20:41:41] petan: Also, "feeling fast" is often more about early access latency than actual performance; something can very well feel faster and be considerably slower. [20:42:11] Cyberpower678 it is /ignore petan :P [20:42:38] oh right. [20:42:54] Just for future reference, how do you unignore? [20:44:24] ignore is a feature implemented in your client [20:44:28] some clients can't unignore [20:44:45] it usually lasts until you close and reopen your client [20:44:57] irc sucks :) [20:45:41] nah it just suck if you are using wrong client [20:45:45] irc is best <3 [20:46:00] Coren why you chose xfs over ext4 just out of curiosity [20:46:04] I use XChat. [20:46:44] Cyberpower678 did u try telnet? [20:46:49] that is a cool irc client :D [20:47:03] Who here is in charge of SQL Replication setup? [20:47:06] petan, no [20:48:08] i stupidly allowed a labs instance's storage to fill up to capacity and the machine is now unable to accept connections -- is there any way for me to rescue it by deleting things? this is i-00000669, btw. [20:48:16] petan: Better behaviour over LVM slices; turns out to be invaluable when doing thin snapshots. [20:48:47] ori-l: root should normally still be able to log in, at the very least. [20:48:56] ori-l: well, how do u know itś because of disk? [20:49:02] did you try to reboot it? [20:49:11] that could remove some data, at least clear /tmp [20:49:20] petan: yes. I can see in the console output various services unable to run because they can't create pidfiles [20:49:43] arent pidfiles created in tmpfs? [20:49:51] I always thought they are [20:50:08] they should be; i can't give you the exact errors because i'm waiting on the console output display to reload after closing it by accident [20:50:43] ori-l: what's the instance name? [20:50:59] I may be able to ssh in [20:52:08] 'flog' (i-00000669) [20:52:11] just noticed I can merge on labs/private - is it ok to merge myself @ https://gerrit.wikimedia.org/r/#/c/60839/ ? [20:54:05] MaxSem: ping [20:54:15] apmon, pong [20:54:21] Hi [20:54:53] hey [20:54:53] you can merge on that? [20:54:59] * Ryan_Lane goes to check its permissions [20:55:08] Do you know in what timeframe we are likely to be able test the things on the real hardware? [20:55:27] State of the map US is going to be in San Fransisco on June 9th [20:55:35] you can +2. I don't see how you can merge [20:55:41] Ryan_Lane, oh [20:55:43] and I was wondering if I could give a talk about mod_tile / renderd and scaling up [20:56:06] apmon: we have no current roadmap for postgres [20:56:16] but it would be good if we could have some results to fall back on, rather than just theory [20:56:29] <^demon> ACL on labs/private is a mess :\ [20:56:30] Ryan_Lane: This is the production stuff, not the labs [20:56:43] ah. you're asking in the wrong channel, then :) [20:56:51] :-) [20:57:02] since it involves TESTING... [20:57:04] :) [20:57:42] ^demon: gerrit's acls are perplexing to me [20:57:54] ^demon: I wouldn't mind if you took a stab at cleaning that up :) [20:58:31] <^demon> I thought about writing a script that would comb all of the project.config files and see what redundant bits could be removed. [20:58:35] apmon, we will talk with ops about getting some resources to make it move forward. ideally, we want it by end of Q4 2012/13, which is soon, but I can't guarantee that we will have anything in prod by June [20:59:38] is it rrequired for a talk?:) [21:00:46] Doesn't need to be in actual production, it would just be good to have real confidence that it would scale well, if I'd decide to talk about it... :-) [21:01:28] heh [21:01:43] yes, Labs is not about anything performance [21:02:40] But if I understood it correctly, there is already hardware available. So once it works in labs, one could test it on real hardware? [21:03:19] yes, but the threshold for stepping into porduction is high [21:04:10] OK, sounds reasonable. [21:04:29] Wait, are we talking about postgres? [21:05:01] the general tile rendering stack [21:05:02] <^demon> Getting production hardware isn't the hard part...it's making sure you've got buy in from ops and anyone else to make sure what you're doing is sound. [21:05:16] <^demon> Actually allocating the hardware is (usually) pretty easy, once you've figured that out. [21:05:58] <^demon> Buy in is half the battle. The other half is begging for code review. [21:06:18] <^demon> Ryan_Lane: See what I did there? ;-) [21:06:31] https://bugzilla.wikimedia.org/show_bug.cgi?id=47526 [21:11:26] Ryan_Lane: I checked, and double checked, and there is no way to set that tunable from sysctl; do you want me to make it into an upstart script? [21:15:43] ^demon: :D [21:16:00] ^demon: how well do you know the api and javascript? ;) [21:16:12] <^demon> MediaWiki API? [21:16:38] <^demon> Or Gerrit API? [21:16:40] yep [21:16:41] mediawiki [21:16:49] <^demon> I know the API well. Javascript not so much. [21:16:55] because I'm going to have some stuff that needs to be reviewed soon :D [21:17:23] it's about 50/50 js and mediawiki api [21:18:04] <^demon> I can certainly review API stuff, sure. [21:18:04] ok. I'm eating, but I'll review after [21:20:45] Ryan_Lane: I might do that review for you if you happen to be good at doing this thing I need from ops [21:21:19] hahaha [21:21:22] (CARP for Parsoid Varnish) [21:21:26] all this quid pro quo :D [21:21:34] ah [21:21:38] well, it won't be carp [21:21:48] Well something that makes it not duplicate the cache [21:21:50] because varnish doesn't have carp [21:21:56] but mark wrote something for this [21:22:02] And probably update everything to the new style of doing Varnish stuff in the role classes [21:22:03] I have no clue how it's implemented or configured [21:22:18] I took a very very rough initial stab at it locally but I know I'm going to hit a wall soon [21:22:28] And I don't really have much time to do it anyways [21:22:31] basically you have two varnishes running per box [21:22:46] the frontend one will be configured to talk to the backends [21:22:52] and you'll round robin the frontend ones [21:22:53] (manifests/role/cache.pp) [21:22:58] Yes [21:23:01] I think upload is already configured like this [21:23:05] So far that's the same as CARP, right? [21:23:11] yep [21:23:22] carp is a specific algorithm, though [21:23:34] (Yes, upload already has it which is why I started in role/cache.pp) [21:23:34] mark's is different [21:23:34] Oh, OK [21:23:48] Well that's fine [21:23:53] maybe I'll take a look at this to see if I can learn it [21:23:59] not a bad idea for me to know how that's working [21:24:07] The thing is we're trying to cache ~800GB of distinct stuff on two machines with 512GB each [21:24:16] the thing I'm waiting on for review right now is jquery.chosen [21:24:21] OK [21:24:25] Gerrit links? [21:24:38] https://gerrit.wikimedia.org/r/#/c/58281/ [21:24:57] the other two things I have aren't passing js linting, so I need to fix that before I ask for reviews :) [21:25:18] heh [21:25:30] Ryan_Lane: I'll make lint fixes if you unfuck my instance :P [21:25:35] hahaha [21:25:42] I'm trying to get into it [21:25:51] worst case I can mount it directly [21:25:51] <^demon> We're all very open about this bribery thing, aren't we :p [21:25:55] yep [21:26:03] jshintcoin [21:26:08] the new virtual currency [21:26:09] ori-l: you already tried a reboot? [21:26:12] Ryan_Lane: yep [21:26:27] ok. I'm going to kill it and clean it directly [21:26:32] what can I delete? [21:26:55] there's almost certainly a huge mongodb directory under /srv, though I don't remember the exact path: /srv/mongo, or /srv/mongodb, or something like that. [21:27:04] ok [21:27:22] can someone review https://gerrit.wikimedia.org/r/#/c/60839/ please? [21:27:57] <^demon> MaxSem: lgtm, do you have a change where you're using those credentials so I can compare? [21:29:41] ori-l: I don't see it under /srv at all [21:29:50] maybe mnt? [21:29:56] nope [21:30:01] that wouldn't fill / anyway [21:30:20] hrm. what's eating up all the space, then? [21:30:22] * ori-l boggles [21:30:27] ah [21:30:36] /var/lib/mongodb [21:30:40] I can kill that? [21:30:56] yep! [21:31:17] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=678606 edit summary: [+2] /* Submitting Jobs */ tpyo fiix [21:31:31] ori-l: ok. now reboot the instance [21:31:40] doing so [21:31:45] btw which change(s) aren't passing jshint? [21:31:49] ^demon, I will not use mysql yet, I just need those vars to satisfy a dependency [21:32:13] some changesets to make the labs console nicer :) [21:32:17] ori-l: https://gerrit.wikimedia.org/r/#/c/60782/ [21:32:28] ori-l: https://gerrit.wikimedia.org/r/#/c/60783/ [21:32:35] 60793 is especially cool :) [21:32:35] cool, on it [21:32:49] thanks [21:32:57] feel free to tell me what I'm doing stupid with the js, too [21:33:05] I'm a noob when it comes to js [21:33:15] Change merged: Demon; [labs/private] (master) - https://gerrit.wikimedia.org/r/60839 [21:33:16] k [21:33:20] <^demon> MaxSem: Submitted. [21:33:29] thanks:) [21:33:37] <^demon> yw [21:37:21] <^demon> Ryan_Lane: Blah, I hate the gerrit comment parser. [21:37:38] yeah. just noticed the link myself [21:37:39] heh [21:37:54] <^demon> Worth fixing this quarter. [21:40:18] now I just need to find some bribery material for csteipp for some openid reviews [21:45:16] ori-l: what's the "use strict" thing do? [21:46:07] Ryan_Lane: ES<5 allowed a lot of shenanigans that nobody thought were a good idea, like redefining 'undefined' and defining all undeclared variables in global scope [21:46:26] ugh [21:46:31] the 'use strict'; pragma is evaluated by older browsers as just a string literal that doesn't do anything. on browsers that support ES5, it triggers stricter handling of such things [21:46:39] so, everything I add should have 'use strict'? [21:46:43] "Error writing /home/maxsem/update-geodata: Read-only file system" [21:46:50] * Ryan_Lane grumbles [21:46:52] pretty much, yeah [21:46:54] I fucking hate glusterfs [21:47:01] MaxSem: which instance/project? [21:47:11] deployment-bastion [21:47:23] so deployment-prep, then? [21:47:31] yeah [21:47:43] I'm so excited we're starting to switch away [21:47:54] o_0 [21:47:59] to what? [21:48:12] NFSv4 [21:48:23] two systems attached to a shared set of shelves [21:48:39] so that we can failover between systems if necessary [21:49:38] ori-l: why data.error !== undefined rather than 'error' in data ? [21:49:46] more reliable? faster? [21:50:19] No real difference [21:50:19] "in" looks up the inheritance chain so it'll return true for properties of javascript's 'Object' [21:50:26] Oh well there's that I suppose [21:50:34] ah. ok [21:51:03] so if I accidentally used property of javascript's object that'll return true [21:51:05] got it [21:51:56] yep. var x = {}; console.log('constructor' in x); [21:52:14] and some browsers have weird nonstandard attribtues [21:52:15] well, that's a pain. [21:52:42] oh well, now I know how to properly handle that :) [21:53:52] languages are easier to deal with when they have a single implementation [21:54:51] also, languages are easier to deal with when they're not javascript [21:54:55] hahaha [21:57:30] i see your rationale for using switch -- but it'd be more idiomatic to use an object as a lookup table by defining a property for each action -- var actions = { reboot: function () { /* ... */ }, consoleoutput: function () { /* ... */ } [21:57:58] in fact, you might want to make those methods on an Instance type [21:58:51] so, then I'd call: actions[action] ? [21:59:15] right, optionally with a check that such action exists [21:59:28] what's the benefit of doing so over using a switch? [22:00:06] by itself that sort of change won't be beneficial, probably, other than getting jshint to shut up about 'switch', which it doesn't like for obscure reasons [22:00:30] but it seems like a step toward decoupling application logic from the dom [22:00:30] ahhhh. ok [22:00:56] mind making the change so I can see how it's done/ [22:01:37] sure, ok. I'm first going to update 60783 with the bare minimum changes needed to make that pass jshint [22:01:43] * Ryan_Lane nods [22:02:01] MaxSem: is it working now? [22:02:08] Ryan_Lane, yes - thanks [22:02:11] yw [22:02:18] one of the bricks was down. no clue why [22:02:42] I guess it knows we're plotting to kill it soon [22:02:49] and wants some attention [22:03:22] <^demon> "Somebody love meeeeeee." [22:08:09] Coren: did you change out /home and /data/project? [22:08:14] <^demon> Ryan_Lane: Can we get those last 2 changes in? Then I can stop bugging you. [22:08:19] oh [22:08:20] forgot [22:10:23] ^demon: all merged through [22:10:35] <^demon> \o/ [22:11:27] Coren: ah. you have the role in the module [22:11:33] oh [22:11:34] wait [22:11:35] sorry [22:11:37] no context :D [22:11:41] https://gerrit.wikimedia.org/r/#/c/59969/2/modules/toollabs/manifests/bastion.pp,unified [22:12:12] I'm more of a fan of all roles being in roles/ [22:12:18] but I'd like roles/ to be a module itself [22:12:28] <^demon> Bah, stupid class redeclaration. [22:12:37] ^demon: :D [22:13:09] <^demon> Says Class[Role::Gitblit] is being redefined. [22:13:18] o.O [22:14:13] on which system? [22:14:21] <^demon> Antimony. [22:15:15] on which line in which manifes? [22:15:36] <^demon> gitblit.pp line 12. [22:15:45] <^demon> role/gitblit.pp, rather [22:16:02] heh [22:16:12] <^demon> Oh, does it not like gitblit being called from role::gitblit? [22:16:20] <^demon> Silly. [22:16:25] welcome to puppet [22:16:28] that's very likely the issue [22:16:52] this is, of course because we aren't using modules [22:17:10] <^demon> Fixing. [22:18:51] <^demon> https://gerrit.wikimedia.org/r/#/c/60935/ [22:20:12] merged through [22:21:24] <^demon> Yep, that was it. [22:24:47] <^demon> https://gerrit.wikimedia.org/r/60936 is a minor improvement too ;-) [22:26:55] merged through [22:29:04] ^demon: merged through again [22:58:11] Warning: There is 1 user waiting for shell: Kyzz (waiting 0 minutes) [23:11:42] Warning: There is 1 user waiting for shell: Kyzz (waiting 13 minutes) [23:21:21] Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=678644 edit summary: [-90] /* LDAP */ [23:22:15] Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=678645 edit summary: [-62] /* Gluster */ [23:25:19] Warning: There is 1 user waiting for shell: Kyzz (waiting 27 minutes) [23:26:45] !log openstack upgrading openstack services to folsom release on nova-precise2 [23:26:47] Logged the message, Master [23:38:51] Warning: There is 1 user waiting for shell: Kyzz (waiting 40 minutes) [23:52:27] Warning: There is 1 user waiting for shell: Kyzz (waiting 54 minutes)