[03:09:12] Oh my, that sounds brilliant! Thank you, @brion [03:09:47] Woohoo [03:10:53] I'm still making additional with possible performance optimizations. If we can rig the kernels of multiple languages to call each other in-process and leave the RPC to the scaling and isolation system, I think call overhead goes way down [03:11:14] Though this may be complex to implement ;) [03:12:18] We'll take a very close look. Piggybacking on a stable technology has a lot of benefits. [03:13:30] Doing RPCs cross language boundaries is another thing we were planning to implement ourselves. If there's something we can reuse, that would be neat. [03:16:21] Funny thing, I was thinking about making Wikifunctions easily accessible from a Jupyter notebook, but the other way around didn't occur to me - but you're absolutely right [03:18:56] :) [03:19:55] The debugger protocol, is that LSP? [03:21:36] DAP I think? An extension of the same one used by vs code [03:22:45] Two questions I wanted to Google but maybe you know: is there a lua kennel? Is there a kernel that runs in there user's browser? [03:23:11] Both excellent questions which I will look into ;) [03:23:52] I really want to use a common debugger frontend for server side and client side scripts [04:51:52] Yes, yes, yes! [06:08:45] so for kernel running in the browser i guess you have to connect to a virtual message bus instead of the websocket. it sounds straightforward in principle, i just don't know if anyone's done it yet ;) [06:09:27] and full debugging of an in-browser language (like single-stepping) requires a suitable interpreter implementation or compilation to async functions (which by total coincidence i am working on ;) ) [14:57:12] Hehe :D [14:57:37] That's pretty cool! Let's keep the ball rolling on this [16:51:45] <3 [17:32:25] @vrandecic totally n00b question, but I identify as one: will there be a kind of a "Hello World", an example or set of examples that demonstrate the absolute basics of what this project can do ? [18:00:36] Yes, definitely! It's just a matter of "when" and not of "If" (re @moebeus: @vrandecic totally n00b question, but I identify as one: will there be a kind of a "Hello World", an example or set of examples that demonstrate the absolute basics of what this project can do ?) [18:02:45] What's super helpful is to say what you think it does, so we can see how we are doing with and where we need to improve our communication strategy [18:10:35] my reason for asking is really just my experience with SPARQL and Wikidata, a project that I truly love but where any kind of quality structured documentation or tutorials have been scattered about and a little hard to find, proper polished manuals have only started to appear fairly recently, a little bit like an afterthought. Every other week there is someone in Chat asking "What is the point of WD?" and blah blah. So I think [18:46:27] I have mixed feelings about this, as I don't want to be too prescriptive. Yes, we should describe and offer documentation on how it works, but a lot of it will be processes and rules that the community will develop. Also, we don't know which parts will be successful and work out how. [18:47:12] Put differently: there's a difference in the manual on MediaWiki.org and the help pages in Wikipedia [18:47:58] The development team could do the first, but shouldn't do the latter [18:52:43] fair point. Most of what you guys are doing is way over my head anyways, but if I can learn how to generate simple descriptions in languages I don't speak I will be superhappy (that's one of the things I imagine the project will empower me to do btw :) ) [19:00:09] I very much hope so, but it'll take a while until we get there [19:06:55] We have to wikicrawl before we can wikirun ;) [19:08:41] time is... not of the essence. I hope I'm the first person to ever say that 🙃 [23:21:42] Brion, i finished reading you're essay. That's great work. A few things: [23:22:27] Yes, we still plan to be purely functional. That should help with many of the things, as you mention correctly. So, we don't have to bother about state [23:23:57] Yes, I *really* want have functions and closures as values, i.e. first class function treatment [23:25:46] The "query Wikidata" is slotted for after launch, so that's future-us problem. That also includes a lot of the caching problems you mention. We're discussing these questions with the architecture team [23:27:04] One advantage of using Jupyter kernels would be that we already have them running in prod (paws), so a security review might be easier [23:27:19] (assuming paws counts) [23:28:38] *nod* [23:30:02] Reusing or not reusing execution contexts probably makes a big difference to performance, I think we should do some serious attempt to benchmark launch of jupyter kernels for invocations in separate contexts and see if it's reasonable [23:31:06] future-you is gonna grumble about past-you's decisions :P [23:31:20] Hehe [23:32:11] Very much [23:32:42] Yes, benchmarks are a good idea. If we can keep the kernels alive, that could help a lot, I'm sure [23:32:47] A thing to note about closures as first class functions is they'd be explicitly not deterministic / pure functional if provided by a language with mutable state [23:32:55] But yes, we'll need to measure! [23:33:32] I think the kernel can at least reuse the process, running new interpret contexts in new threads [23:33:48] So you don't have to dynamically link v8 on every call [23:33:58] Sounds good! [23:34:06] Unless we stay in the pure functional fragment of the language (re @wmtelegram_bot: A thing to note about closures as first class functions is they'd be explicitly not deterministic / pure functional if provided by a language with mutable state) [23:34:40] Well what's pure functional subset of js? [23:34:46] re closures, in GraalEneyj we get those almost for free, but ensuring that the implementations stay pure is expensive for performance [23:35:15] I did a test with a JS function calling an eneyj function calling another JS function, and the outer JS function has a loop with one million iterations – runs in ~3m20s with proper isolation, or 1s without it [23:35:40] Yikes [23:35:50] so for now I made that configurable, and if anyone wants to just trust that implementations don’t share state via globalThis or whatever, then they can specify that (but default is to isolate implementations) [23:35:51] Oy [23:37:41] That's a good number to know, thanks Lucas [23:40:50] Lucas is that in-process calls or RPC? I imagine even in process is expensive when setting up and tearing down interpreter state in an inner loop ;) [23:42:18] There may also be better ways to separate global contexts running in the same interpreter loop in some runtimes. I don't know whether realms have been sufficiently standardized and implemented yet... [23:42:40] Dealing with the global object in js seems like it would need special support [23:44:02] But it should be possible? Like separate frames in a web app [23:44:42] They share call stack and heap and can refer to each other's objects if they have access to them but have separate global objects, instances of constructors etc [23:45:04] This might be a more suitable high performance model [23:45:47] We'd avoid leaking objects by giving them only wrapper functions that do a serialize deserialize [23:47:15] So that might be something we can expand on top of the jupyter model, using one JS virtual machine per render session, or might be something we'd do wholly differently [23:50:05] Hmm, that's a good question. (re @wmtelegram_bot: Well what's pure functional subset of js?) [23:50:10] Either way, there's always going to be less overhead by making fewer calls outside the function's own source and dependent libraries. Which means we should have performance monitoring tools so function developers can see where their cpu time is going and refactor loops to the most efficient balance [23:50:25] Yes! [23:50:55] The trick with JS is that "a + 1" could invoke arbitrary code that modifies global state via a.valueOf() [23:52:54] it’s all in-process; call isolation is done through an “inner context”, which in JS terminology means a separate realm, I believe (re @wmtelegram_bot: Lucas is that in-process calls or RPC? I imagine even in process is expensive when setting up and tearing down interpreter state in an inner loop ;)) [23:54:07] Yeah that sounds about what I was thinking with the analogy to web frames [23:54:28] Confirms it's high overhead, though how much compared to RPC we'll have to measure! [23:58:29] So for enforcing determinism without recreating initial state, I think we're limited to a statically typed subset of JavaScript where all types are known and can be statically validated as not altering any global state. This isn't impossible but it would be significantly different from just saying "you can write JavaScript functions" - you might have trouble reusing third party code libraries. [23:58:55] But if we go down that road I would recommend looking at TypeScript for starters [23:59:38] I just don't know if it's possible to statically prevent pollution of the globals