[01:46:00] Cyberpower678: poke [01:49:10] Betacommand, ouch [01:49:46] Cyberpower678: any plans to implement log stats in your supercount? [01:50:02] Betacommand, can you elaborate? [01:50:33] Count of moves, uploads, for non-admins, for admins delete, undelete, block ect [01:50:46] Betacommand, yes. [01:51:16] Cyberpower678: any idea how far down the road those features are? [01:51:42] Betacommand, I've got so much on my plate right now. Right now I'm implementing deleted edit analysis. [01:52:35] Cyberpower678: no problem just looking if you had a ballpark idea, IE a week, six months, or a year [01:53:14] Betacommand, My ballpark would be between 1-5 months, depending on RL. [01:53:24] np [11:50:29] s1 replag is > 24 hours; can anyone check what the problem is? [12:24:49] hi folks,is it possible yet to have a tool labs tool that binds to a tcp port? [12:25:34] in my specific case, for responding to HTTP requests [12:25:41] MartijnH: as in: a tcp port accessible from outside tool labs? It might be possible via instance-proxy, but I'm not completely sure. [12:26:29] snap. I don't have the knowledge to set up an instance-proxy anyway [12:26:31] Why do you want to run your own http server? [12:26:46] as in: can't the tool run via fcgi? [12:27:08] MartijnH: basically, every tool has a private lighttpd server at their disposal [12:27:43] I don't really know lighttpd. Maybe I could set it up as a proxy [12:28:12] otherwise I'd have to run something like a servlet container on top of it, and that's just a major pain [12:28:25] MartijnH: oh, you're trying to run servlets. I see. [12:28:40] There's a tomcat host for that, but I'm not completely sure how it works. [12:28:42] Coren: ^ [12:28:49] no, I'd rather not run servlets, I'd rather just respond to HTTP requests directly [12:29:03] but lacking the infrastructure, servlets are a painful but possible fallback mode [12:29:53] I really don't get your use case then. You have some code, which provides it's own HTTP server? [12:30:48] if so, I would expect it should be easy enough to run it as fastcgi server, too. [12:31:26] I could check that [12:31:53] MartijnH: As a rule, no; this is not a scenario available to tools -- partly because that would cause your tool to get IP information from endusers. You can set up your own project to do that, at need, provided you are careful about the general labs TOS and use the less comprehensive privacy policy (which, inter alia, means you cannot embed in project pages) [12:32:18] MartijnH: But setting up a FCGI responder is normally quite simple. [12:32:35] Coren, thanks for the info and background [12:32:40] I'll read up on setting up FCGI [12:32:48] MartijnH: What language is your code written in? [12:32:54] Scala [12:33:19] (Yes, I know your feelings about the JVM) [12:33:29] and I know the memory allocation is a pain [12:33:43] but it's the only language (apart from C#) I'm semi-competent at [12:33:58] Unusual choice. I don't know of an FCGI wedge for Scala offhand, but it shouldn't be to hard to whip one up. https://github.com/adamsmasher/BlogExperiment/blob/master/FCGI.scala might be a good starting point? [12:34:33] thanks, that looks promising [12:35:48] Also, if you actually need to implement HTTP yourself, you might want to ask YuviPanda for help -- he can explain to you how a tool running on the grid can communicate to the proxy to get requests sent to it. [12:37:13] well, the libraries I'm using provide the HTTP interface (much like Node doesn't require you to write your own HTTP implementation), so I don't have to whip that up myself [12:38:33] MartijnH: Talking to the proxy might be your simplest choice then. It's fairly simple conceptually: when your daemon starts, you need to open a TCP socket with the proxy and send it your URI base and port; as long as the socket remains open the proxy will... well, proxy to it. :-) [12:39:07] that does sound pretty simple [12:39:46] "the proxy" in this case is the tools lighttpd instance configured as a proxy, or something else? [12:40:12] it's the thing that routes requests to the per-tool lighttpds [12:40:25] MartijnH: It's in perl, but https://git.wikimedia.org/blob/operations%2Fpuppet/f0294a9e2de41ddcf2219fc672b9ab2fbebdf3fc/modules%2Ftoollabs%2Ffiles%2Fportgrabber at line 32 is what you want. [12:40:36] thanks! [12:42:11] whoo, perl code even I can understand \o/ [12:42:24] well, apart from the $^F = 8; et al. [12:42:57] $^F is an evil hack to change which file descriptor to keep open on exec. That /is/ obscure. [12:43:41] Lemme put it to you this way: I've been coding with perl for ~20 years and I had to look that up. :-) [12:43:45] and the $| = 1; does something with stdout? [12:43:56] [/random guess] [12:44:33] $| is more idiomatic. That turns off buffering of output. [12:45:10] assuming that $ is a sigil to indicate a variable and my declares some local variable, I understand the lines my $host = `hostname`; and exec @ARGV, $port; [12:45:10] I could have called it with the "new" name $OUTPUT_AUTOFLUSH [12:45:23] MartijnH: $ designates a scalar indeed. [12:45:43] MartijnH: my perl-fu suggests the calling convention is ./portgrabber [12:46:09] valhallasw: That's at least a green belt right there. :-) [12:46:20] @ARGV is the array-of-arguments, and the first element got popped off by line 20 [12:47:14] FYI, that's a bit more complicated than it needs for a stand alone program talking to the proxy because it's a wrapper that has to presume that the command has no idea how to talk to it. [12:49:03] The only "important" part is line 32-34 that does the real work: It connects to tools-webproxy:8282 and sends it ".*\n$host:$port\n" where "$host" expands to "where the webserver is running" and "$port" to "what port is it listening on" [12:50:02] ".*" is just a regex telling it you want all URI sent to it and not just a subdir. [12:58:35] Coren: well, I suppose the 'get a port number' part is also relevant? [12:58:45] I guess I should have a better understanding of networking to understand what's going on. but lacking that, and going the blissfully ignorant route, I should make a shell script that calls portgrabber with as arguments the name of the tool and the command to start my tool, and feed that to jstart, and my tool should be listning on port 8282? [12:59:06] ph, wait, the other way around [12:59:18] MartijnH: the tool should listen at the port portgrabber suggests [13:00:39] Right, portgrabber will actually allocate a port and give it as argument to your command (the reason why is simple: that's the only way to avoid port conflicts between tools) [13:00:59] also, you would have to make sure the job starts at a webgrid host -- not sure how to do that [13:02:09] valhallasw: Just asking for the right queue works. In this case, you'd want the non-lighttpd webgrid: -q webgrid-tomcat [13:03:11] (I didn't expect non-tomcat webservers, but it'll work just as fine - the point is to not use webgrid-lighttpd because it overcommits resources presuming they are all running the same daemon) [13:03:25] ah, right. [13:03:59] and the port is passed on as an extra argument at the end of the argument list passed to the tool? [13:04:04] correct [13:05:02] try, e.g., portgrabber test python -c "import sys; prin [13:05:02] t sys.argv" a b c [13:05:13] *lookofdisapproval* [13:05:26] portgrabber test python -c "import sys; print sys.argv" a b c [13:05:35] which returns ['-c', 'a', 'b', 'c', '4123'] [13:08:42] so that passes the arguments -c, "import sys; print sys.argv", a, b, c and the port number to python, and python takes the second argument not prefixed with a - as its script and passes the rest to the script as arguments that are available in python as sys.argv? [13:08:59] er, the first argument not prefixed with a -? [13:09:28] Just ignore the -c (which means 'run this code instead of a script') [13:09:46] the point I was trying to make was that the port number is just added to the argument list [13:09:54] it's not really all that important what python does with its arguements to me anyway, yeah [13:11:04] Coren: one thing I don't get -- how does the proxy know which urls to forward? I don't see the tool name being forwarded to the proxy, just to portgranter [13:12:41] Coren: also, portgrabber is not working on -tomcat. [13:12:52] connect to portgranter: No such file or directory [13:12:58] portgranterd not running, probably? [13:14:24] otherwise: works great [13:14:25] valhallasw: Definitely. Lemme go check -- that's a bug. [13:15:38] oh, tools-webproxy does ident back on the host to get the path name? That would make sense. [13:15:53] * Coren it does. [13:16:06] portgranter start/running, process 1334 [13:16:19] Heh. I never actually put ensure => running in puppet. D'oh! [13:21:34] woot, working \o/ [13:21:39] but via qsub, as jsub does not support -b y [13:26:44] jsub actually /only/ supports -b y. :-) It doesn't support -b because it doesn't want you to try '-b n' :-P [13:27:15] Coren: let me rephrase that. qsub doesn't try to read the command file with -b y, while jsub *does* try to read it [13:27:31] which means jsub -b y portgrabber etc works, but qsub portgrabber does not [13:27:42] not a huge issue -- just needs an intermediate script :-) [13:29:40] th other way around. I get confused by the q and j's all the time. [13:35:05] Coren, MartijnH: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Other_web_servers [13:36:20] MartijnH: once you've got your Scala webserver running, could you replace my python example with that? Thanks :-) [13:36:51] valhallasw: Made a small tweak [13:37:31] nice valhallasw :) [13:37:43] The exec eliminates an unneeded subshell. [13:37:49] Coren: ah. Good point. [14:07:33] Coren: s1 replag is > 29 hours. Any idea what's hanging it up? [14:08:50] I know exactly what it is, but it'll need a shove from our DBA. There were some system control statements issued in prod that cannot work on the replicas that have stalled the replication timeline. [14:10:43] pipeline*