[00:13:40] day six without a working dewiki database... I guess there are no news? [00:16:08] apper: what's the latest issue? externallinks too small? [00:16:50] because recentchanges definitely has recent stuff [00:18:09] yes, replication works, but most of the revisions are missing [00:18:42] page seems to be complete, but revision lacks of at least two thirds of the revisions [00:19:12] and 85% percent of the page_latest revisions [00:19:35] apper:jeremyb: latest: 41,293,437 expected: 130,682,295 [00:20:08] and I only checked page, revision and recentchanges - I don't know the state of other tables [00:20:30] but I think revision is essential for most tools (at least for mine) [00:23:13] apper: Coren answered to giftpflanzes question at 15:53 that Sean (DBA) will be on duty tomorrow (ahh today) [00:23:40] right, but if he's where i think he is then it already is day there [00:23:42] hedonil: ah, thanks [00:25:25] jeremyb: so is there a way to contact him directly? [00:25:38] well he's currently -afk... [00:25:46] * jeremyb shall poke [00:25:48] is there no emergency DBA or is the priority of this problem to low to get this fixed yesterday? [00:26:04] I just want to know ;) [00:26:25] jeremyb: yeah poke [00:28:13] labs replication is hostnely substantially more complicated than typical maria/mysql replication [00:28:19] honestly* [00:28:39] so, it (I guess) needs someone that knows how it works to fix it [00:30:36] jeremyb: He, that's what DBA's are for [00:31:10] jeremyb: maybe there's a manual for that:-D [00:31:34] jeremyb: I understand that, but in all companies I know there are emergency plans for such things and people who get paid for being on standby duty. I don't want to blame these people, I just want to know how this works at WMF [00:32:37] I'm not sure labs comes with a SLA [00:33:01] It certainly doesn't get the priority production does/would [00:33:05] grrrrrrr, so much lag (on my shell) [00:33:08] Reedy++ [00:33:18] Reedy: ah, okay [00:33:43] hedonil: my point was that it can't just be fixed by any DBA. needs someone that already knows it. whether there's a manual, unsure [00:34:19] jeremyb: sure. just kiddin' [00:34:53] jeremyb: but - 6 days is a loooong period [00:35:43] hedonil: right, but idk how broken it was or what's needed to fix it [00:36:23] jeremyb: think status was FUBAR [00:38:17] jeremyb: and that's a problem... the only thing which was done by a WMF employee at the corresponding bugzilla bug was setting the priority of the bug down... so noone knows, what has to be done or who cares at the moment... [00:40:17] * hedonil fiddles with voodoo to speed things up [00:40:43] but now I know that a DBA called Sean will work on it today, that's great news [00:41:23] apper: yeah, now we know his name :-D [00:41:35] Go find Domas [00:41:36] * Reedy grins [00:41:54] apper: we make him an offer he can't refuse8-) [00:42:07] Can someone delete the spam https://wikitech.wikimedia.org/wiki/Special:RecentChanges ? [00:42:36] PiRSquared: sure [00:42:43] damn lag [00:45:30] not sure how much to use autoblock [00:45:44] https://wikitech.wikimedia.org/wiki/Special:Contributions/Larygloria0156 [00:48:49] Greetings. I can't login to bastion. Any ideas? [00:50:30] mathbot: you should try bastion2 [00:51:06] Thank you. I wish this was documented somewhere. [00:51:24] bastion looks pretty dead [00:51:32] jeremyb: https://wikitech.wikimedia.org/wiki/Talk:Main_Page#Shell_access_for_spammers.3F.21 [00:51:39] PiRSquared: i saw [00:51:48] https://wikitech.wikimedia.org/wiki/Special:ListUsers/Larygloria?limit=3 [00:59:05] hedonil: apper: so we got a bit of status: there's a process running to compare tables between the 2 boxes and fix places that are broken but that process is repeatedly losing its connection to the labs side. hoping to find out why that's happening today. [00:59:28] jeremyb: thanks! [01:03:29] so, it's 2 AM here, I go to sleep and hopefully the database is fully working soon :) [01:03:30] bye [01:03:37] and thanks jeremy for the update [01:03:39] nacht [01:03:59] nacht ;) [01:20:15] TParis, ping [01:20:48] Hey [01:24:40] I am having problems logging in. I followed all the steps at https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances, with no luck. I am able to connect to gerritt, using ssh USER@gerrit.wikimedia.org -p 29418 [01:25:00] I cannot connect to bastion as it is down. Bastion 2 gives: [01:25:25] ssh -Av USER@bastion2.wmflabs.org [01:25:33] debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/oleg/.ssh/id_rsa debug1: Authentications that can continue: publickey debug1: Trying private key: /home/oleg/.ssh/id_dsa debug1: No more authentication methods to try. Permission denied (publickey). [01:25:38] Any ideas? Thanks! [01:26:16] mathbot: try -vvv not -Av [01:27:12] OK. Here's what I got: [01:27:13] debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/oleg/.ssh/id_rsa d [01:27:45] I just generated a fresh rsa key using ssh-keygen -t rsa. And now luc. [01:28:13] don't do a fresh key. focus on the old key [01:28:18] when did you add the old key? [01:28:30] A few days ago [01:28:49] OK, to gerritt just today, 20 minutes ago. [01:28:58] ignore gerrit [01:29:09] all we care about is [[special:preferences]] [01:29:21] you have no authorized_keys at all atm [01:29:28] but 2 keys in ldap [01:29:38] presumably you have the same problem as LuaKT [01:29:40] I added a few days ago to the other place, that is to labs [01:30:05] my. [01:30:35] Any suggestions? Miscomunication problems amog the servers perhaps? [01:31:02] for the record, ldap is: [01:31:03] 2048 b7:75:ec:46:fa:01:e0:c9:06:d7:fe:ff:4c:21:2f:1d oleg.alexandrov at gmail.com-1 (RSA) [01:31:07] 2048 55:2b:49:b7:8c:15:c1:3b:57:70:70:32:ba:65:8d:53 oleg.alexandrov at gmail.com-2 (RSA) [01:31:42] we don't know what it is. the symptom is no new keys are being pushed from ldap to the authorized_keys file store at all [01:33:48] hence i said not to make a new key. i knew it wouldn't be propagated :-] [01:34:18] Well, it failed both with the old key and the new key. :) I think I will give up for now. Perhaps it will come back in a few days? [01:34:51] But I had the old key for quite many days, maybe even a week. Oh. I'll try some other time. Thanks1 [01:35:27] well we could look when it first broke for LuaKT [01:36:10] Cool. [01:37:38] LuaKT: first came to ask about it not working 17 UTC on the 28th [01:37:59] the most recently modified authorized_keys file is Nov 26 17:35 UTC [01:38:06] so 2 days before he asked [01:38:11] mathbot: ^ [01:38:17] yep [01:39:25] Can my key be added by hand by you folks? [01:40:57] mathbot: maybe... not by me. but some of the people that could help with that also could just fix the problem :) [01:41:08] (there's not many people that could place your key manually) [01:41:27] Should I write an email to somebody? Or file a ticket? [01:50:27] gah [01:50:29] i was typing! [01:53:30] setting 701 on a file should make it readable to no one right? [01:53:36] its a cgi script [01:54:59] i set 700 and it still works fine [01:56:55] 701 or 700? [01:57:17] anyway, depends who the file owner is and who's reading it [01:57:26] 700 is certainly not noone [02:04:15] ohai notconfusing! [02:04:18] legoktm: ? [02:05:21] yes? [02:05:28] hello [02:05:47] jeremyb: well, i meant no one by my tool. 700 ended up working which is good enough for me [02:07:45] notconfusing: just hi :) [02:08:14] legoktm: i thought you were saying everyone could still read it [02:08:22] when it was 701 someone else could [02:15:11] sorry, is bastion down? [02:15:18] i'm getting connection timeout [02:15:48] yurik: yes. use bastion2 [02:17:15] jeremyb, thx [02:37:25] PiRSquared: how are we doing on spam? [02:37:37] * PiRSquared checks [02:38:03] Seems mostly handled for now. [02:38:21] except for the two spammers with shell access [02:39:15] can't do much about that. they are blocked, right? [02:39:32] yes [02:39:50] jeremyb: you can't revoke shell access once it is granted? [02:39:55] nope! [02:40:05] your link don't do much now that i deleted the pages! [02:40:11] https://wikitech.wikimedia.org/wiki/Special:PrefixIndex/User:Larygloria [02:40:20] jeremyb: https://wikitech.wikimedia.org/wiki/Special:ListUsers/Larygloria?limit=3 :P [02:40:35] the ones with four digits [02:40:39] both have shell [02:40:40] yeah, all blocked [02:41:28] At least they can't do any harm... right? [02:43:36] they could probably still [02:43:43] i.e. we should revoke shell [02:44:42] i wonder if you can update your prefs when you're blocked. e.g. add an ssh key [02:44:55] I think you can. [02:46:22] well key updating is broken anyway :-] [02:46:24] $ for u in larygloria{01,0144,0156}; do groups "$u"; done [02:46:24] larygloria01 : wikidev [02:46:24] larygloria0144 : wikidev project-bastion [02:46:24] larygloria0156 : wikidev project-bastion [02:46:41] so they have no keys set and they're not in any groups besides bastion [02:47:01] so all we need to do is revoke shell / rm from bastion :) [02:52:29] huh, block log makes me think something wrong with autoblock [02:53:03] Hopefully the spammers don't even know they were granted shell access [02:53:20] but I might be underestimating them [02:53:59] wtf. /me tests [02:54:34] huh, a few users like this including rschen7754 [02:54:43] why does custom sig make it's way to ldap? [02:55:26] huh? [02:56:05] rschen7754: your sig on wikitech? [02:56:22] or something [02:56:29] yes, it's my default signature on all other wikis [04:02:25] Coren: please !log in prod log when booting physical hardware? (or someone else voice an opinion) [04:02:34] * jeremyb is referring to virt10 [04:04:00] (I think also mail to labs-l would have been warranted) [04:34:02] does anyone here have experience with maintaining code which is shared between tools? [04:34:30] stick it in a git repo? [04:35:43] ln -s [04:36:45] currently I have my shared code in /home/magog [04:36:54] but that directory is hidden to the tools [04:37:00] should I create a shared tool? [04:37:06] (@SigmaWP) [04:37:17] Hmm [04:37:24] Magog_the_Ogre: You could do what I did [04:37:34] legoktm, I have never once used git before in my life, and I don't have the foggiest idea where I'd start with it [04:37:41] Create a tool with your username and then dump anything useful there [04:37:49] yup, I was thinking about that [04:37:52] * SigmaWP named his tool "sigma" [04:37:52] SigmaWP's idea is pretty good. just symlink the common code [04:37:57] didn't know if I would break some sort of conevntion [04:38:10] Well, nobody's chewed me up yet, so I think I'm doing fine [04:38:37] silly question then [04:38:49] how do I get my code into the tool directory? I don't think I can sftp into it. [04:39:12] I just ssh into it and then cp everything [04:39:28] why not? [04:39:39] you can sftp [04:40:11] but you shouldn't use sftp much [04:41:36] ok thanks everyone [04:41:59] jeremyb, I reiterate I've never used git before; I might as well just sftp into my home directory and then cp like SigmaWP said [04:42:15] no, you should learn git :) [04:42:58] it's almost 12 in the morning. *implying there isn't other time for me to get this done* [04:43:56] oh, good TZ you have [09:08:27] any gerrit people around? mediawiki/extensions/JsonConfig wants to be created :) [09:36:17] Hello, I can't login to bastion.wmflabs.org. Is anyone having any problem? The error I get is: "Network error: Connection timed out" [09:36:31] My internet is okay BTW. [10:09:32] is there a way to remove all mails in the account? [10:12:05] ok, found a way :) [10:42:53] Did /public/pagecounts permanently move to /mnt/pagecounts? [10:43:59] Tanvir: did you try bastion2.wmflabs.org? [10:56:11] Just tried and it worked! Thanks a bunch, Whym! [10:59:44] np [11:20:05] Meh, http://boogs.wmflabs.org down? Cannot test. :-/ [12:01:45] paravoid: if you happen to be around, instances on virt10 are apparently all in SHUTOFF state [12:01:50] which includes bastion.wmflabs.org [12:44:15] what's wrong with bastions? [12:44:24] ~> ssh jkroll@sylvester.pmtpa.wmflabs [12:44:24] ssh: connect to host bastion.wmflabs.org port 22: No route to host [12:44:24] ssh_exchange_identification: Connection closed by remote host [12:46:26] using tools-login.wmflabs.org as ProxyCommand does work. did bastion*.wmflabs.org get deprecated or something? [12:49:40] JohannesK_WMDE: don't know any details, but I heard that you should use bastion2.wmflabs.org for the time being [12:50:09] whym: ... ah. thanks. [12:50:25] o.O [13:36:40] JohannesK_WMDE: use bastion2.wmflabs.org [13:36:52] bastion.wmflabs.org points to bastion1.wmflabs.org which is dead right now [13:49:28] hashar: Thanks for adding that to my scrollback buffer. I just needed it. [14:02:44] dh_python3: error: no such option: --shebang [14:02:46] I am doomed [14:02:49] manybubbles: you are welcome [14:06:26] hashar: A reboot on the wikitech interface should suffice to wake them. [14:07:36] Coren: oh , mind rebooting bastion1 instance in the bastion project? [14:08:00] I have no idea how to use Semantic MediaWiki to select all instances hosted on virt10 though [14:08:47] Is that bit of data even exposed? [14:08:59] on my https://wikitech.wikimedia.org/wiki/Special:NovaInstance [14:09:17] whenever you click on an instance that list the virt box [14:09:23] !log bastion bastion1 and bastion3 rebooted (virt10 failure fallout) [14:09:33] that is how I found out that those bastions box were on virt10 [14:10:09] Ah, I didn't even know that. Makes sense that it'd be though. [14:10:24] It should be possible to make a #ask to get that list. [14:10:27] Coren: /public/pagecounts seems to have moved to /mnt/pagecounts at tools-login. Is this a stable setup, or is it going to change again? [14:10:32] also filled in a bug to make bastion.wmflabs.org a round robin entry pointing to all bastions [14:10:42] will have to have the ssh host key shared on all three instances though [14:11:26] krd: It's a workaround; I /will/ keep a symlink back when I get a chance to reboot the tools project though. [14:11:59] Can you also tell me at which intervals the files in pagecounts are updated? [14:12:09] * Coren didn't notice the bastion1 was down since he uses iron or tools-login. [14:12:27] krd: I /think/ it's a few times a week. [14:13:14] hmm.... Can we have it each day, probably at a fixed time? [14:14:21] Hi Coren and everybody! [14:14:26] oops, /mnt/pagecounts is nearly empty. hmm. [14:14:32] Hallo, Silke [14:14:44] What's the status of dewiki database? [14:14:47] *sigh* [14:16:30] I haven't been able to get a hold of our DBA for the past 3 days. :-( [14:16:55] It's urgent enough in my priority list that once it gets to morning in Oz, I'll give him a phone call. [14:17:41] (in ~7h) [14:18:21] a dba is a... database admin??? [14:18:26] yes [14:18:30] ok [14:18:33] Coren: could apergos helps with db stuff ? [14:19:33] hashar: Possibly, but the whole replication setup is sparesely documented. Sean is still in catch-up mode. [14:20:01] Thanks for the update! [14:20:23] Asher was a good guy, but the replication was a little rushed on the "document things dammit" side. :-) [14:20:30] yeah [14:20:39] that is known as "bug 1 - our documentation sucks" [14:51:10] Coren, did you get my email? [14:52:05] Cyberpower678: Probably, somewhere in my sea of email. Remind me the Subject: ? [14:52:35] Coren, ever since the 503 issues were resolved, the internal links in xtools are broken. [14:53:14] Ah, yes, that email. Need moar information. Give me a problematic URL? [14:53:43] Coren, A link that points to https://tools.wmflabs.org/xtools/ec ends up going to http://tools-webgrid-01:4040/xtools/ec/ [14:54:04] Coren, https://tools.wmflabs.org/xtools/pcount/index.php?name=Cyberpower678&lang=en&wiki=wikipedia and then click Edit Counter [14:55:15] Coren, what could be causing that? [14:56:23] Don't know yet, but note how the correct https://tools.wmflabs.org/xtools/ec/ doesn't do that. [14:56:47] Meaning? [14:57:13] It works if not called as a link, but when typed directly into the browser? [15:00:15] Coren, ^ [15:02:09] No, it means that it works when the final slash is there. [15:02:36] (As it must) [15:03:33] That's weird. It worked before with out the slash. Isn't the web server supposed to correct for that? [15:06:21] Coren, now the edit counter is slow. [15:10:12] Cyberpower678: It is; and that's probably the cause of the problem. The web server will create a 302 for an URI with a missing slash, but right now it looks like it's doing it /after/ the rewrite for proxying. [15:12:15] So is the problem on my end, or your end of the tool? [15:15:20] Coren, ^ [15:15:50] Cyberpower678: Alsmot certainly at the proxy. [15:16:20] Ok. Let me know when you figure out what spoiled. [15:16:45] Coren, it's taking longer and longer to process a request. [15:17:31] I can't help with 'longer and longer' unless you give me more detail about what happens that is longer. Database queries? Local processing? Transmission? [15:20:09] Coren, sorry. When I click on the edit count link, it appears the proxy is taking longer to service the request. The edit counter itself only spends 1.5 seconds to execute but I'm waiting about 10 seconds to see it. [15:22:58] Coren, now it's stuck. [15:23:07] Works fine with https [15:23:34] Coren, then you have a bug when servicing http requests. [15:24:06] It's not at the network; once the counter data arrives, the rest gets downloadded within 70ms (including, btw, a 404 on an image the webpage tries to use) [15:24:27] Cyberpower678: Of course there's a bug: HTTP is accepted. [15:25:37] Getting my edits is pretty consistent at ~2.1 sec. [15:25:46] Coren, so what do I need to do to fix this. [15:26:17] I see nothing to fix; what's wrong with 2-3 secs? [15:26:39] Because HTTPS works but HTTP times out. [15:27:14] Coren, loaded in 45 seconds. [15:27:34] Script itself "Executed in 1.22 second(s)." [15:27:46] It's working fine for me but I note the webpage is trying to load images from toolserver.org! [15:28:48] I know. That's going to be fixed at some point. The page displays before it attempts to load the image. [15:29:31] I don't mind 2-3 seconds, but 45 is insanely long. [15:29:44] I'm not seeing 45 seconds. [15:29:48] And you're using http? [15:29:55] Not https? [15:30:04] Both. [15:30:28] I'm timing it. It takes 45 seconds to display something. [15:31:44] Interestingly enough, your script does /not/ do the same thing under HTTP and HTTPS; and is much more variable under HTTP (took 14s this time for me) [15:32:12] And I think I see why: because of mixed-content prohibition, it doesn't /try/ to fiddle with toolserver.org under HTTPS. [15:32:28] So you get the result instantly. [15:32:47] Under HTTP, it tries to fiddle with stuff from toolserver.org and /those/ seem to time out. [15:33:16] So I'm thinking your javascript is misbehaving. [15:34:02] Coren, ok. Back to the proxy bug. Why are links without the / being redirected elsewhere. [15:34:05] ? [15:35:24] Cyberpower678: I'll look into it sometime today. They're not being redirected "elsewhere", their just letting through an internal hostname. In the meantime, there is an easy fix: correct your links so that they are not missing the / anymore. :-) [15:35:56] Coren, easy, but extremely tedious. [15:36:19] It's a bug that the redirect doesn't work right (clearly) but that's a safety against broken URI being typed manually -- your links should not be pointing to broken places. [15:36:22] Something I don't have time to do today. [15:36:29] https://tools.wmflabs.org/xtools/ec is an illegal, improper URL [15:37:39] Ok. [15:37:42] Not today. [15:38:00] I'll see about the redirect during the day though; it's an annoying bug regardless. [15:45:52] Coren: have you found out why public keys are not generated from ldap to labstore1.pmtpa.wmnet:/keys on /public/keys ? [15:46:17] iirc that is not documented / unpuppetized and thus either some investigation or Ryan Lane to reply [15:56:05] might be andrewbogott_afk :D [15:56:51] andrewbogott_afk: you got rid of user accounts form labstore boxes ( https://gerrit.wikimedia.org/r/#/c/98030/ ) that apparently causes the authorized keys of labs users to no more be exported on labstore1:/keys https://bugzilla.wikimedia.org/show_bug.cgi?id=57751 [16:13:01] hashar: Andrew probably knows, too. [16:13:35] hashar: But that changeset is almost certainly unrelated. [16:18:05] we will see :-] [16:18:36] well the cronjob set the UID of files to the user id, must be failing with some "user not found" or something [16:18:46] but yeah, andrew will find out :] [16:26:00] hashar: I'm getting a slow start today but will look at that soon. [16:26:13] Do you know of a specific account or key that was changed after the 28th? [16:30:01] andrewbogott: jenkins-deploy [16:30:33] and I think jenkins-slave as well :D [16:31:00] Ah, right, that's in the bug [16:31:10] andrewbogott: the files in /keys/ belong to the user so potentially removing the user from labstore would cause the script that export the keys to be failing because it can't chown or something like that [16:31:14] Did you create those the 'normal' way via wikitech account creation? [16:31:26] should spurt some errors in the cron mails though [16:31:33] jenkins-deploy I can't remember [16:31:40] jenkins-slave, yeah created via wikitech [16:31:52] the credentials for both users are on fenari in /home/wikipedia/docs/labs-jenkins [16:32:07] the users are used by the Jenkins master to connect and run jobs on labs instance [16:32:30] had to add a ssh public key which eventually never got accepted since it is never published in /keys/jenkins**/.ssh/authorized_keys [16:32:45] I am sure looking at the cronjob / cron mail will yield an obvious error message [16:33:09] the question would be where the hell is that cron being run. And ideally should be moved back in puppet :] [16:33:39] I'll have to dig. Probably unrelated to the puppet patch you saw though, since that only affected mhoover. [16:35:28] yeah was just a guess [16:35:34] andrewbogott: More data point: endusers changing their keys on wikitech no longer have their changes reflected on the shared filesystem. [16:35:36] because that is a change made on same day that impact labs [16:36:02] andrewbogott: So it's not just new users. [16:36:14] hm. [16:36:30] if you can add any status on https://bugzilla.wikimedia.org/show_bug.cgi?id=57751 that would be nice [16:36:39] even if it is to say you have no clue or can't fix it right now :] [16:37:06] I am off, will be back later on today [16:38:38] Coren: hey, what's up with labs stuff? Specifically: the de db replication thingy and andre__ mentioned accesing his bugzilla test instance is intermittent (right andre?) [16:38:56] We had one of the VM backends fail during the weekend. [16:39:08] That's fixed but will have wrecked havock with a number of projects. [16:39:23] ugh, sucky [16:39:30] The DB thing is separate (afaik, only dewiki is impacted) and will require Sean's gentle touch. [16:39:55] I'm going to pounce on him the second it's daylight in Oz. :-) [16:40:03] Coren, oh I see. Because https://bugzilla.wikimedia.org/show_bug.cgi?id=57642 gets a bit of buzz :) [16:41:01] Yes. I think we need to have Tim capture Sean and lock him in a room until there's enough documentation about replication for other opsen to help. [16:41:34] It doesn't help that he had the hit the ground running from day one in a documentation vacuum himself. I'm impressed he didn't yet run away screaming. :-) [16:49:21] Tools web is extremely slow for me. anyone else? [16:50:16] ireas: Try https, most people report it much faster. [16:50:32] ireas: The old proxy is on its way out to be replaced by something much nicer. [16:50:44] Coren, ah, yes, works better now. Thanks! :) [16:51:13] Coren: think you can review that stuff today? I wrote a replacement for portgrabber that's more generic and works for all tools on all nodes, but we can just make it work with lighty for now [16:51:39] YuviPanda: Linky? [16:51:58] YuviPanda: Does it still do the trick of holding the reservation exactly as long as the FD is open? [16:52:06] Coren: exactly :D [16:52:08] it does [16:52:12] moment [16:52:45] Coren: https://gerrit.wikimedia.org/r/#/c/98352/ [16:52:52] Coren: I'll add an upstart script for it later [16:52:52] YuviPanda: What is your proxy doing when it doesn't have a map? Shuffle off to one of the -webserver-xx? [16:53:05] Coren: yup. equal distribution on all three [16:53:23] Excellent. [16:53:35] Jenkins didn't like your stuff though. [16:53:36] Coren: and the list of 'backup nodes' is in puppet as well [16:53:45] Coren: yeah, that's a stupid stupid rule there [16:53:46] 80col limit [16:54:18] Coren: I'll fix that. that's the only reason it is complaining [16:56:31] There's a couple of other whines about whitespace too. [16:57:36] * Coren reads code now. [16:59:36] Ah, now you grant ports from the proxy, and requests are done over TCP? How do you ensure that the port is really unused? [17:00:05] Coren: so that's the tricky bit I need to figure out now. I'm trying to figure out how to do that without a race condition [17:00:18] since if the helper first probes, then starts, we can have something open up in the meantim [17:00:28] Coren: but that'll crash the app we just opened, since it'll try to bind to that path [17:00:35] so that sounds okay to me, but I wanted to run it by you [17:00:56] That'd work iff you make sure that the next request will end up with a different port. [17:01:12] Otherwise the webserver will restart, get the same port, and crash again. [17:01:48] Coren: randomize+probe? [17:02:04] probe by either connecting to it or parsing netproc? [17:03:57] Well, in the current setup, I simply use a list of ports local to the host, so I can rely on it. You could do the same, only downside is that the same port can't be used on two hosts which may become an issue with a lot of webgrid nodes. [17:04:30] Coren: yup, and it also means a random app can actually just open up a port, since not all port opening is under your full control [17:04:54] Coren: and the new one means we don't need to have 'webgrid' nodes as such. They can just run in the general exec nodes [17:05:26] YuviPanda: That's by design; I am /not/ letting people open random apps this way. :-) [17:05:36] Coren: any reason? [17:05:42] i mean [17:05:44] what reason? [17:05:47] gah, can't english [17:05:50] what's the reason? [17:05:59] YuviPanda: Privacy violations. No unproxied incoming connections. [17:06:13] Coren: err, these are all proxied. no ip is passed back [17:06:18] Oh, wait, you mean why no random app? [17:06:32] as in, I want a node app running and proxying. [17:07:08] Coren: or something of that sort. not just what lighty supports. [17:07:28] Support issues mostly. I don't want people running random webservers. If your project is complex enough that it needs a setup where you set a webserver up, then you need to spin off to your own project. [17:07:46] Coren: my problem was that the current setup completely kills any app that wants to use websockets [17:08:15] or Go. Or Scala. Or anything that doesn't have first rate support for CGI or fcgi or apache / lighty. [17:08:39] Coren: just like how you have webserver start, we can have uwsgi start or go start or whatever [17:09:18] Way outside the scope of tools. [17:09:20] Coren: not right now. I just want to make lighty now [17:09:32] Coren: what. using anything not PHP or Python is 'way out of scope of tools'? [17:10:53] Coren: wanting to use those languages on tools was the only reason I've been working on this proxy at all [17:11:22] Wait, what? Why are you under the impression that lighttpd is language dependent? [17:12:04] over anything that's not cgi or fcgi? [17:12:16] Coren: you can't really use websockets over lighttpd either [17:13:05] Coren: also I don't see why we can't have uwsgi or a node thing running instead of just lighttpd. They're just servers, and once we get the script done for lighttpd they'll be trivial to write [17:13:46] YuviPanda: Yes, but then those will also have their own grid nodes for resource allocation, and startup scripts under our control. [17:13:52] why [17:13:58] startup scripts under our control sure [17:14:01] why own grid? [17:14:03]