[01:48:36] will u guys be listing the names of donors this year? don't think it was done last year but did happen in 2013 [20:00:27] #startmeeting Shared hosting technical alternatives [20:00:27] Meeting started Mon Dec 21 20:00:26 2015 UTC and is due to finish in 60 minutes. The chair is gilles. Information about MeetBot at http://wiki.debian.org/MeetBot. [20:00:27] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [20:00:27] The meeting name has been set to 'shared_hosting_technical_alternatives' [20:01:10] hi everyone, gilles from wmf performance here, hosting these two back-to-back meetings [20:01:22] o/ [20:01:29] I guess, to get this out of the way I'll say containers, containers, containers, containers [20:01:32] containers [20:02:14] which style of container? [20:02:24] exactly, and I'm pretty clueless on the topic [20:02:38] so if anyone has experience with them and the different types, that would be nice to talk about [20:02:52] * robla sees if Yuvi is available [20:03:01] is it really a viable alternative to shared hosting in terms of performance and cost? [20:03:19] #link https://phabricator.wikimedia.org/T113210 [20:03:58] in the phabricator task that led to this meeting, VPS was considered as a bad proposal in terms of cost-to-performance ratio. which is why shared hosting is till appealing. I wonder where containers are on that spectrum [20:04:12] *still [20:05:40] let me tell you: containers aren't understood by 99% of people who can upload PHP files into some directory [20:06:08] sure, but that might change if vendors of "easy hosting" adopt the technology [20:06:23] I wonder about the potential, not the practicality right now [20:06:38] (backlog available somewhere?) [20:06:59] do we have info on how good is the quality of these one click solutions is? [20:07:50] <_joe_> if you are talking about the possibility that shared hosting services will offer container support [20:08:12] <_joe_> I spoke recently with a sysop from a large shared hoster in Italy [20:08:31] uh, hello? [20:08:31] <_joe_> he told me fair and square that shared hosting is way cheaper than supporting containers [20:08:41] ori: backlog of this meeting or the last one last month? [20:08:45] this one [20:08:47] <_joe_> so that is never going to happen there [20:08:59] !logs [20:08:59] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ [20:09:02] ori: ^ [20:09:21] thanks [20:09:21] * robla looks into fixing the topic line [20:09:30] <_joe_> notwithstanding that, there are a plethora of offerings for running containers or thin VMs around [20:09:58] _joe_: did he elaborate about what ways it was cheaper? just talking about the effect of container overhead or that it's too complicated to manage right now and thus more expensive in staff, etc. [20:10:05] <_joe_> but that will /never/ be a viable substitute for the hundred of thousands of shared hosting mediawiki installation [20:10:15] hundreds of thousands? [20:10:34] <_joe_> ori: I am pretty sure I saw a figure like that some time ago [20:10:44] o/ [20:11:03] <_joe_> it was a list of the most common web apps on the internet, joomla and wp of course were far ahead, but my memory may fail me [20:11:18] <_joe_> gilles: so say I have to host 10 mediawiki instances on a single physical node [20:11:24] <_joe_> they're all very low traffic [20:11:47] <_joe_> I will just use apache-suexec and spawn fcgi workers whenever I need [20:12:05] <_joe_> I can squeeze hundreds of them on a single host [20:12:23] <_joe_> and share one apache master for all of them [20:12:55] <_joe_> to obtain the same with containers, I should furiously overcommit memory (and I still have one apache master process per wiki that will run anyways) [20:13:46] gilles: could you append this to the topic line? " | Wikimedia meetings channel | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ " [20:15:07] We really need to fix that. [20:15:07] robla: the topic is set by meetbot [20:15:15] insu it ? [20:15:22] hashar: yes, by the chair of the meeting [20:15:32] oh, I could have done so through the bot? ok [20:15:34] and it ends up in the meetbot log :D [20:15:36] anyway... [20:15:45] <_joe_> btw, I do still occasionally support a shared hosting service that hosts literally hundreds of websites per each physical node, and said nodes are incredibly thin. If we did that with containers, we'd need much more memory and resources. Or you can just containerize the fcgi execution, maybe... It's anyways going to cost people much more to get "N containers running continuously" than "N web spaces [20:15:51] <_joe_> on a shared hosting" [20:16:01] <_joe_> I hope this answered your question, at least partly [20:16:08] So what does a container get us that the current situation does not? [20:16:22] For Mediawiki adoption and admins? [20:16:42] _joe_: it does, yes. do all containers techs work like that? you mention containerizing fcgi itself. I'm unclear on where the line can stand between the host and the containers [20:16:42] <_joe_> ckoerner: it's extremely easy to distribute a set of containers and give people instructions to run them [20:17:36] would the typical scenario of "I need to install mediawiki and a few extensions I need" be easier? [20:17:50] or would that not help, still having to fidget with editing config files [20:17:57] <_joe_> I guess not [20:18:20] installing mediawiki itself and adding extensions are different topics imho [20:18:32] one big challenge with containers is keeping the containers up-to-date [20:18:42] the later could be solved by having a proper extension manager that grab files over the internet from extension distributor [20:18:43] I think we're discussing the wrong thing. Running PHP is easy, the whole containers thingie was about non-PHP services [20:18:53] <_joe_> MaxSem: that's what I got [20:19:18] for the wiki farms I have seen, it is usually just an apache, a vhost per site and a copy of mediawiki per site. Because this way it is easier to do upgrades and have a different extensions per site (sic) [20:19:49] I did ask a few sysadmin about containers. And none were interested either because they have their own รง [20:19:52] MaxSem: right, that's an undeniable advantage, but it doesn't in itself outweigh the cost/performance benefits. otherwise this audience would be using VPS right now, where they can run whatever services they want and have VE and whatnot [20:19:53] containers [20:20:04] or don't want to deal with a third party container [20:20:33] <_joe_> gilles: so what is the aim of the discussion? Installing mediawiki with VE and other additional services? [20:21:34] it's rather, what new technical solutions could help shared hosting or make that audience likely to jump ship to something else. if they're going to migrate to containers because they're suddenly a cost-effective proposal, doing to much for shared hosting might be being behind the times [20:21:52] doesn't like that disruption is likely to happen, though, given the overhead you describe [20:22:36] <_joe_> I think that shared hosting users and people that run containers in any form are two very different "customer" types [20:22:45] <_joe_> but I might be wrong ofc :) [20:22:58] if we're stuck with shared hosting being a "good deal" for a lot of people, I guess the next question is can any technology help with wiki farm hosts using non-php services that extend mediawiki's functionalities? [20:23:26] I agree, but I thought it was worth asking, since containers are so foreign to me and tend to be oddly brought up in SOA discussions [20:23:44] <_joe_> gilles: would said farms think that spending more money to offer VE will be a bargain? [20:23:51] imho as long as something can't beat the cost/performance ratio, nothing will replace shared hosting [20:23:53] gilles: here's the agenda for this hour as I understand it: https://phabricator.wikimedia.org/T113210#1877356 how much progress are you hoping to make before the hour is over? [20:23:54] <_joe_> I have zero idea [20:23:58] under "shared-hosting" kind, you can add most corporate/enterprise environments in which a container will be frowned upon [20:24:39] hashar: why is it frowned upon? too risky for security? [20:24:45] shared hosting is the pseudo-default option for folks with a desire to host as they dominate the lower end (supported tech and cost) of the 'website hosting' market. [20:25:02] It's a great thing, a foot in the door for many to get their site up. [20:25:14] gilles: that and how would you run backup / install monitoring probes etc... with your proprietary softwares [20:25:25] robla: I guess it's kind of covered... unless something has another disruptive shared hosting alternative to get out of a hat right now [20:25:29] Arguably why the previously mentioned softwares have been successfully adopted - including MediaWiki. [20:25:29] *someone [20:25:35] <_joe_> ckoerner: I agre [20:25:54] Raise your hand if your first site was on shared hosting :) [20:26:00] And look at where we are now! :_ [20:26:08] whoops, I smooshed that smiley. [20:26:09] :) [20:26:58] <_joe_> shared hosting is _the_ big strength of php [20:27:19] the few keys points for MediaWiki adoption imho were: 1) PHP/ MySQL , 2) its like Wikipedia so must be good 3) FTP, copy-paste some LocalSettings.php [20:27:20] _joe_: I'm not sure I see the cost different between a heavy pure php extension and something that offloads treatment to a non-php service [20:27:22] I seem to mention WordPress a lot, and that gets some eye rolls, but they recently worked with Bluehost (a popular shared-hosting provider in the US) to figure out a way to bring hosted sites up to date. Maybe there's something we can learn from their work? Not an apples to apples, but https://videopress.com/v/cmA03MuQ?at=1278 [20:27:25] *difference [20:28:30] <_joe_> gilles: any shared hosting that wants to do that has to figure out a ton of things first [20:28:34] some folks from hosting companies seemed interested in this debate. afaik their main issue with the kind of services we produce is that they're not easy to load-control/separate between users. [20:28:41] because that's not the use case we're building for [20:29:15] basically any service we come up with needs a way to be monitored for per-user usage, have data segregation between users, etc. [20:29:34] it's not something we need for the wmf, because on the contrary it makes more sense for us for all our wikis to talk to each other [20:29:43] <_joe_> gilles: that was my point :) [20:30:02] and a way to monitor and quota on a per user basis [20:30:06] <_joe_> with "figure out a ton of things" [20:30:08] right [20:30:16] I don't think that anyone will do that work for us [20:30:20] we've had non-php stuff for a long time (e.g. thumbnailing, captcha generation, formulas) [20:30:43] but it's all optional and a lot of them have fallbacks [20:30:43] formulas been a nightmare due to ocaml texvc etc [20:30:52] you can still thumbnail with GD if you're not faint-hearted [20:31:05] but thumbnails is really just installing Imagemagick and it was well covered by documentation [20:31:40] I guess imagemagick is a good example if shared hosts have it. because in itself it doesn't have any per-user control/quota, etc. [20:31:44] it's just an executable [20:31:49] hashar: so....could we aspire to bring other stuff to the level of ease of "just installing"? [20:32:03] I guess what makes it ok is that it doesn't keep any data [20:32:12] <_joe_> gilles: apache can run in chroots and with suexec [20:32:26] <_joe_> you can let then mediawiki shell out to imagemagick [20:32:39] <_joe_> shelling out is relatively easy to control, too [20:32:51] <_joe_> but shared hosting management is a bloodbath in general [20:33:00] robla: ideally yeah. A step by step tutorial explaining how to glue stuff. [20:33:09] right, I'm saying this is an example where IM had no effort to do to be appealing to shared hosting. no extra work on their part, being a binary was enough... [20:33:42] gilles mentioned the compatibility list https://www.mediawiki.org/wiki/Compatibility [20:33:48] <_joe_> gilles: oh ok, I agree [20:33:59] hashar: yeah that's the focus of the next hour [20:34:15] it was started by usualibility initiative iirc so we get a glance at which browser to target for and at the same time be able to state to people: "Sorry your IE 5.0 is not supported see [[Compatibility]] [20:35:24] _joe_: IM probably gets a pass because it's not a service, it doesn't run continuously, doesn't have a data store, etc. [20:35:27] so to me, it sounds folks find shared-hosting / simple hosting to not fit their development pattern hence end up proposing a container approach to be able to install whatever they want as dependencies [20:35:51] which essentially is the same as stating: we no more support shared-hosting / self hosting. Trust us and spawn that container that has everything for you. [20:36:24] I am not saying it is evil. It is just that the effort to integrate everything in containers is antinomic with trying to support shared/self-hosting [20:37:01] it's not the same audience, I don't think it solves any problem. it just adds another medium likely to be poorly maintained in the long term, imho [20:37:26] if we make service installation/management easier for shared hosting, it becomes easier on anything fancier [20:37:34] it's the smallest common denominator [20:37:36] yup [20:37:45] I think the whole container thing is different category from shared-hosting. Unless there is a knowledgeable admin watching this container and doing updates, etc. it will be vulnerable, out-of-date and unmaintaineable very soon [20:38:07] <_joe_> SMalyshev: indeed [20:38:09] so lets better support shared hosting by making it trivial to install all the dependencies, trivial to upgrade, trivial to manage extensions, trivial to tweak settings [20:38:47] then given that strong base. Folks will start building and maintaining containers as a community. [20:38:51] what would that look like? easily installing a fancy extension and the services it needs [20:39:17] my personal use case would be: drop MediaWiki core sources to a shared hosting [20:39:19] gwicke has pushed for us to get better at the "apt-get install" version of installing this stuff [20:39:21] head to /install.php [20:39:22] hashar +1 [20:39:35] have a list of recommended / most popular extensions to check for install [20:39:42] click [OK] [20:39:47] done my wiki is set with all the fanciest stuff I want [20:39:48] robla: so extensions as debian packages that have what they need as apt dependencies? [20:39:53] self-installing extensions creates an obvious vulnerability point though [20:40:05] is this where i mention npm? [20:40:09] gilles: yeah. I'm warming up to the idea [20:40:16] with limitations based on compatibility. So lets say I lack nodejs, Parsoid is not available and hence VisualEditor is not available [20:40:51] robla: that would be quite a commitment to maintaining and keeping up to date. if it stops getting attention, still serving old versions with vulnerabilities in them could be disastrous [20:41:12] gilles: yes, what technology wouldn't have that problem? [20:41:13] I don't buy debian packages either [20:41:18] that is the same issue as with containers. [20:41:23] cscott: can you describe how npm would help with that scenario? [20:41:28] .deb packages and containers are both technical means to a problem [20:41:30] robla: I don't know! [20:41:33] we still don't have the solution. [20:41:50] I guess the one with the smallest effort for us to package? maybe that can even be automatically packaged? [20:42:14] our biggest problem with .debs has been our upstream relationship with the main MediaWiki .deb packager, who I think still insists that MySQL isn't a database [20:43:05] gilles: i need to read backlog, hang on. [20:43:15] robla, we can always defeat him by removing support for his perfect database ;) [20:43:40] cscott: easily installing mediawiki + extensions and keeping everything up to date. and how much effort that would be for us on the packaging side of things [20:44:23] ...and by "our upstream relationship", it's important to define "our" here. I don't think WMF should own fixing that relationship [20:44:24] with containers it looks relatively doable: https://phabricator.wikimedia.org/T92826 [20:45:19] gwicke: it is (relatively) easy to make a container. To make it updateable/maintainable though I believe is not esy at all [20:45:22] gilles: php-node-embed contributes to per-user separation because it's 'one process per user'. Each user has its own process, it's own web server on its own port (or vhost, what have you) [20:45:37] gilles: the existing shared-node-hosting providers manage this way, it seems to work. [20:45:51] SMalyshev: updating isn't that hard from what I have seen [20:45:57] gilles: so in some ways it's simpler than the php+apache shared hosting setup, where everyone has to share an apache [20:45:59] robla: btw, that maintainer has been kicked out of debian and the mw package removed. legoktm is trying to re-package it from scratch [20:46:17] YuviPanda: oh, wow, ok [20:46:18] the current startup script always runs the latest container & stores all data outside of the containers themselves [20:46:19] gwicke: also it raises the barrier from "knows how to handle ftp" to "knows how to handle docker" which I think is a big difference [20:46:44] robla: https://gerrit.wikimedia.org/r/241690 [20:47:11] SMalyshev: agreed, install should not be harder than copy&pasting a command or clicking on a preconfigured image [20:48:11] gwicke: does it help with the next step, when people want to install specific extensions that aren't included in the initial selection of the container? [20:48:27] gilles: there is support for extra extensions, yes [20:48:49] currently I'm not including a full extensions/ checkout, but we could consider doing so [20:49:43] I do think though that we should include everything that's popular out of the box [20:49:53] Also I would imagine docker hosting is rarer and more expensive than php/ftp hosting? And, if doing inhouse, starting docker runner would be harder than apt-get install apache2 php [20:50:31] suitable vms are fairly readily available from about $5/month [20:50:40] SMalyshev: yup. In a corporate env Docker would be a dead-end [20:50:59] > SMalyshev: updating isn't that hard from what I have seen [20:50:59] we're back to the reason why people stick to shared hosting, which isn't just that it's cheap, but that you get a much better cost/performance ratio than any vm [20:51:00] https://github.com/gwicke/mediawiki-containers has a link to some [20:51:10] updating *is* pretty hard still. [20:51:28] you need to update all your packages, rebuild the container and then pull and restart. [20:51:33] while containers are a slightly better deal than vps in that respect, I don't think they'll have shared hosting users move to containers [20:51:39] +1 to gilles as well [20:51:42] YuviPanda: currently, every restart also implicitly updates [20:51:50] hashar: that's what I am worried about - if I go to corporate IT and ask them "i need a apache/php server to run my department wiki", the response would probably be "ok", if I say "I need you to support docker install" the response would be "call me in 2020 when we have free time for that" [20:51:51] it might capture a small portion of that market but it won't make it disappear [20:52:37] gilles: I don't think it's clear that shared hosting provides better performance / money right now [20:52:55] well the overhead is pretty undeniable, _joe_ explained it well earlier [20:53:06] SMalyshev: exactly :-] Though Docker seems to support Solaris zones now ! [20:53:30] gwicke: would you be willing to maintain a list of $5/month hosts that we officially support? [20:53:54] robla: there are sites like serverbear that have up-to-date lists [20:53:56] example: http://serverbear.com/compare?Sort=BearScore&Order=desc&Server+Type=VPS&Monthly+Cost=-&HDD=-&RAM=500000000-&BearScore=-&Virtualization=KVM [20:54:12] so we have 7 minutes left.. Should we record some kind of consensus regarding "Shared hosting technical alternatives"? [20:54:14] basically all of those will work [20:54:18] yes, but do they maintain a list we officially support on our behalf? [20:54:29] my experience with <$5 VPS has been that you get a VPS that works most of the time, emphasis on most [20:55:19] you get suspended the moment your site has any significant traffic and get no response from 'support' :) [20:55:30] same with shared hosting [20:55:49] not really [20:55:51] that really depends on the host [20:55:53] if you want reliability, go with reputable vendors like digitalocean [20:56:25] rather than the very cheapest deal [20:56:56] or AWS, google etc [20:57:07] hashar: my personal conclusion would be that it's not going anywhere, the case for containers hasn't convinced me that it's a good enough proposal for people to abandon shared hosting. not sure if everyone agrees [20:59:13] hashar: probably we could have a list of alternatives - like deb/rpm/whatever packages, docker - with list of people willing to work on it (e.g. for docker I guess gwicke? ) [20:59:13] therefore containers aren't an alternative as much as they're a different medium. worth pursuing for reasons other than "dealing with shared hosting" [20:59:15] shared hosting is a solution like docker [20:59:15] gilles, +1 [20:59:16] the problem we are trying to solve is making it easy to run a fully-featured wiki [20:59:16] i think we need to come up with a list of 'recommended ways to run visual editor on your host', which (at this point) can't include "traditional" shared hosting, but might include things like containers, node-based-shared hosting, etc. [20:59:16] since VE seems to be the forcing function here [20:59:16] SMalyshev: potentially. But deb/rpm/npm/docker are all the same. A technical solution to offer pre packaged stuff for people that have privileged access to a server / container. [20:59:16] yeah, VE is a good example of a feature that a) outside of php-only and b) perceived as important and not some exotic add-on [21:00:01] right now the informal thing which we've been doing is suggesting the .deb packages for parsoid, and that seems to be working pretty well [21:00:08] so i guess that's one point for the "deb" solution [21:00:16] hashar: yes. For non-privileged access the only way I see is to implement our own packaging scheme/installer/updater... [21:00:34] again, informally, parsoid's was broken in vagrant for a long while before anyone noticed [21:00:51] we should look at all the options & their ROI, including shared hosting & different solutions targeted at VMs [21:00:56] I guess that if legoktmis repackaging mediawiki from scratch, it's a good time to make extra debs for popular extensions. question is, is legoktm doing this on his free time or is this a budgeted project? [21:00:58] gilles: I took think container is a dead-end. At least now in 2015. And I don't see WMF having the resources to support proper maintenance of images with security updates etc [21:01:41] I would rank containers as significantly less effort than packages [21:01:52] i'm happy to try suggesting a container-based alternative on Parsoid/Setup or when people wander into #mediawiki-parsoid and see if it gets any uptake. [21:01:55] if docker images includes things like mysql, that means we are now responsible for keeping with mysql updates (at least security updates)? [21:02:07] SMalyshev: yup I talked about it earlier. A way to install extensions from a click of a button directly from mw web interface. With a way to check the host capabilities and gray out extensions that lack required pieces. Ie you lack nodejs, you don't get parsoid and hence VisualEditor is grayed out. [21:02:26] SMalyshev: no, normally you just use the mariadb image & leave maintenance of that to the mariadb folks [21:02:49] that's what mediawiki-containers does, anyway [21:03:03] hashar: yes, I know. as someone who implemented something like that in different app, it's possible but has a number of pitfalls and challenges to consider. [21:03:19] gwicke: ah, ok, that's good [21:03:33] (I'm letting this first section run over on purpose since it's lively, btw, I think that graded support debate is likely to be shorter) [21:04:32] gwicke: so containers support is skunkworks right now. I assume that deb packaging is the same... shouldn't we fix that first? [21:05:11] if we had a packaging person/team they could cover both... [21:05:22] you mean, like a release engineering team? [21:05:26] .deb packages have the same issues [21:05:45] at first one need a .deb system AND the ability to install a .deb [21:05:52] imho, it's a matter of doing it [21:05:58] rather than continuing to talk about it forever [21:06:12] which distro are going to be supported? Just Debian Jessie or the various forks / Ubuntu as well? [21:06:25] we should properly prioritize it, but the best way to get buy-in for that is to demonstrate some progress [21:07:20] "talking about it forever" keeps happening because it's not a specific team's mission, that's my point [21:07:31] <_joe_> I don't think a lot of people will install mediawiki off of debs. Running mediawiki + services via containers has more potential, but it needs a) significant work on tooling on our side b) a decently capable admin to maintain it [21:08:23] I found it surprisingly straightforward so far [21:08:38] <_joe_> when I say "tooling" I mean both on the ease-of-installation part and on the release-updates in a timingly and tested way [21:08:47] _joe_: do you argue that deb support won't help shared hosts? [21:09:02] from my (admittedly limited) experience most small-time mediawiki installs go as "get a linux server, deploy apache/php, unpack mediawiki, copy-paste configs, liftoff!" that was some time ago though, before VE etc. days [21:09:02] <_joe_> gilles: it will do exactly zero benefit to them [21:09:21] <_joe_> SMalyshev: it's the same with most shared hosting platforms [21:09:27] SMalyshev: still roughly the same :-} [21:09:36] * YuviPanda agrees with _joe_ on debs being 0 use for shared hosting [21:09:45] if you're using debs you're already on a VPS [21:09:45] for Parsoid / VE it is not getting installed in my experience [21:09:48] so debs probably won't help much in that scenario [21:09:51] <_joe_> for mediawiki farms, they just have their own personalized branch of mediawiki they're running anyways. [21:10:03] YuviPanda: for the hosts I mean, that make the shared hosting available [21:10:20] even for them, since a package only installs one instance of mw [21:10:25] but I guess it's more a case for having good debs for the services themselves [21:10:26] we should have a one-line installer thing, using curl | sh if necessary [21:10:45] <_joe_> gwicke: ewww no, that is an horrible security liability [21:11:00] I knew I would get you with this ;) [21:11:11] <_joe_> (curl | sh vs providing a script signed with gpg) [21:11:18] ok, then maybe it's time to move to the next part of the meeting [21:11:20] <_joe_> I'm not against the one-liner install :) [21:11:40] #endmeeting [21:11:42] <_joe_> well, maybe 5-lines [21:11:42] I can tell folks wondering what is that "GPG signature" error and just force ignoring it somehow or giving up [21:11:43] Meeting ended Mon Dec 21 21:11:40 2015 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [21:11:43] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-20.00.html [21:11:43] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-20.00.txt [21:11:43] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-20.00.wiki [21:11:43] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-20.00.log.html [21:11:43] :D [21:11:45] again, parsoid has been using debs for a while, and people seem to be happy with it. [21:11:48] #startmeeting Shared hosting support definition [21:11:48] Meeting started Mon Dec 21 21:11:48 2015 UTC and is due to finish in 60 minutes. The chair is gilles. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:11:48] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:11:48] The meeting name has been set to 'shared_hosting_support_definition' [21:12:01] * _joe_ out [21:12:52] so, is it possible to define "shared hosting support" more granularly? [21:13:20] #link https://phabricator.wikimedia.org/T113210 How should the WMF support non-technical mediawiki installs? [21:14:04] I really liked the introduction of the browser compatilibilty matrix [21:14:12] and grading them. [21:14:16] if we start talking about "graded support" definition, what would that look like? "required" "recommended"? like PHP, Apache and MySQL would fall in "required" for mediawiki core, and IM would be "recommended" for example [21:14:35] or is that the wrong way to look at it? [21:16:25] for MediaWiki for a long time we ensured lot of features run on a basic PHP compilation [21:16:37] going as far as proposing pure PHP versions for some pecl extensions [21:17:12] or having different "backend" services that should fit most needs [21:17:23] the cache for example, you can use either squid/varnish or a file based cache [21:17:42] the backend caches can be memcached/apc/database [21:18:18] yes, I think that having multiple options is a recurring problem, but I also notice that a lot of the time we have one well supported one and several other options that are actually not getting proper support. starting with alternative DBs, for example [21:18:19] I see there grading of platforms, but should there be also grading of extensions? i.e. "we should ensure CirrusSearch runs on the platform before we declare MW fully supports the platform"? [21:18:48] yes I think that extensions would need their own compatibility grids [21:18:53] ideally using the same format as core [21:20:04] gilles: that's for sure but I'm asking about something slightly different - should we also have a priorities between extensions? i.e. if we discover extension X does not work on platform Y, how is it important for WMF to invest in fixing it? [21:20:08] for things where several options are available, I think we could have a "support" level for each. "WMF-supported", "community-supported", "unsupported" for example [21:20:26] because when we say "you can use MSSQL" it's basically a joke [21:20:43] but if it was listed as one of the options, you wouldn't know that [21:21:29] SMalyshev: yes I guess that for browsers we have a universal baseline that all extensions and core abide to [21:21:33] given limited resources, I think we should put more effort into making a standard install work really well than pretending to support all kinds of impossible permutations [21:22:15] gwicke: so, standard install implies standard list of extensions? [21:22:37] definitely some, yes [21:22:48] VE is a no-brainer, for example [21:22:49] I would say right now the supported extensions are those shipped in the release tarball [21:23:49] the permuations are a good place to leverage the community. let the person who really cares about MSSQL support maintain that (or not, if nobody cares about it) [21:24:15] another option would be to say what a standard install is for different hosting platforms: shared hosting, containers, vps, etc. for example [21:24:23] for databases, eg, i'd say the things we really support are mysql and sqlite (the latter because we use it in jenkins) [21:24:46] subbu: like for a given extension you'd see if it has support for each of those? [21:24:50] if we're not regularly running tests on a particular code path, i don't think we can say it's supported. [21:24:51] so it is clear that if someone wants shared install, they won't get VE which might be okay for some wikis. [21:24:54] Postgre has lot of community support but it does not leverage any of Postgre ability so there is little point [21:24:55] gilles, yes. [21:25:19] that's still quite a few permutations [21:25:21] "shared hosting" is a bit vague, though, as it depends on the specific host's willingness to install services [21:25:46] we'd have to define what that smallest common denominator has [21:25:57] our own definition of "typical shared hosting", I guess [21:26:27] to me, it is clear that smallest common denominator is php+apache .. i haven't seen any conversation or discussion so far that has advanced it beyond that basic setup. [21:26:36] I don't think we ever firmly defined the minimum requirements [21:26:38] and mysql :) [21:26:50] yes.:) [21:26:53] and some code paths are barely tested [21:26:56] so what would that be? minimal requirements? just those 3? [21:26:58] it's really PHP and $webserver [21:27:22] Apache is not even a requirement afaik [21:27:26] but, you are getting an experience that's far from what you might have expected when you signed up for "what Wikipedia uses" [21:27:27] right [21:27:38] rather than get into these endless debates / discussions about shared hosting, containers, vps, deb .. maybe specify a small subset of hosting platforms, and what the support is for each. [21:27:46] so should we have a list of extensions which work with pure php/mysql? and maybe a list of those that is important to keep working with pure php/mysql? [21:27:46] subbu: node-mediawiki-express replaces apache w/ node. but yes: php + some web server. [21:27:55] and one could suggest using sqlite for degraded perfs :-D [21:28:27] so, any objections to have a matrix of "hosting types" with each having a support level value? for core and each extension [21:28:46] based on composer.json we have a requirement for the PHP extension iconv. We probably require some others as well but I guess they are not explicitly defined because most distro compile PHP with them [21:28:53] also, should we have a test suite that would install those on minimal envt and actually checks they work? [21:29:02] SMalyshev: +2 [21:29:42] part of the reason sqlite is so well supported is that we have been running CI with sqlite as a backend. Though CI now uses MySQL [21:29:44] I think we should approach this more from a user perspective [21:30:12] iconv is optional, we only use it for some i18n support. [21:30:12] but we first need to define some basic support matrix I guess. Then come up with test patterns to ensure we do not break our requirements [21:30:21] as a small wiki user, I don't want to spend $X per month, and don't want to deal with OS maintenance [21:30:22] if not present, php falls back on a pure-php implementation [21:30:24] tell me what to do [21:30:39] well, I'm a user, I use a VPS, I'd go to an extension page. first thing I see is that VPS is supported, good. then I'd like to see a list of software to install to make that extension work, or even a command I can copy+paste [21:30:46] same for container, etc. [21:30:57] gilles, +1 on the matrix. [21:30:58] I would suggest to start at the other end [21:31:04] the kind of host is irrelevant imho [21:31:20] it is what the host can ship which matter [21:31:24] I'm a user with budget X and skill Y, and want features A, B, C; what should I use? [21:31:52] gwicke: would you go to mediawiki.org to answer that question, though? [21:32:11] I guess we could have a wizard of sorts that answers that question for you [21:32:12] the actual support will vary between VPS hosts / shared hosts [21:32:13] gwicke: "want features" is not that clear though - if you want search, do you need CirrusSearch or not? that depends on size of the wiki, speed, etc. [21:32:24] gilles: if there were clear recommendations, then probably yes [21:32:51] and few options = good for most users [21:32:57] gwicke, gilles i think the user budget/skill/feature matrix and the hosting type matrix are orthogonal .. i think the latter is more about what the wmf is willing to support. [21:32:59] I guess then that a wizard where people can pick budget and features and gives you a list of hosts that match your criteria? [21:33:08] gilles: definitely [21:33:28] gilles: wizard sounds too complex [21:33:29] subbu: right, I think both things solve a different problem and can coexist [21:33:40] yes. [21:33:59] I really think we should focus our efforts on less than 3 options [21:34:25] just enough to cover the main use cases [21:34:35] just enough to run CI tests on all three options before commit [21:34:46] that's what i'd say. if we don't test it, we don't support it. [21:34:58] but we can't run tests on tens of different permutations [21:34:58] now we established earlier that containers are unlikely to replace shared hosting, but could containers eat the VPS market completely in a few years? [21:35:11] that would make a good case for caring about containers rather than VPS in all of this [21:35:27] containers typically run in a VPS [21:35:39] no conflict there [21:35:45] potentially but I don't think it is our role lead the migration toward containers [21:36:24] if containers live inside a VPS, then our support buckets are really just "shared hosting" and "server you have full control over" [21:36:47] and I don't see how some extension could ever not run on a server where you can do what you want... [21:37:03] so really this becomes binary about where some extension runs on typical shared hosting or not [21:37:08] whether [21:37:45] for requirements, I guess whenever you have full control of the host you should be able to get them installed [21:37:49] okay, so I guess we agree that we don't need a wizard [21:37:53] whether it is redhat/debian a VPS or a container [21:38:00] for shared hosts, it is slightly more complicated [21:38:10] different shared host will provide different set of packages / versions [21:38:25] so you would need a finer list of requirements than just the type of hosts [21:38:34] if you are willing to spend $5 / month and want full functionality, get a VPS with distro X, Y or Z and run this command in a shell [21:38:55] so if you want i18n , you will want a host that ship PHP with iconv / mbstring [21:39:01] else, follow these instructions for shared hosting [21:39:21] ...if you want to use node-based shared hosting then... [21:40:00] can't we just have a simple list of software dependencies and minimum versions? [21:40:11] I would expect to have that on the right-hand side of https://www.mediawiki.org/wiki/Extension:VisualEditor for example [21:40:18] for non-technical users, that's not very helpful [21:40:26] gilles: we have already. In /INSTALL and on https://www.mediawiki.org/wiki/Manual:Installation_guide [21:40:40] look for the gray box "requirements in a short" [21:40:54] hashar: yes but not per-extension [21:40:58] eg: [21:41:00] "Image thumbnailing and TeX require additional programs. Parsoid (required by VisualEditor) and other services have their own requirements." [21:41:24] my point is that there needs to be a simple option for most non-technical users [21:41:33] for extensions there is some effort with the extension registry system. I am not sure it has fields listing requirements [21:41:39] no reading of dependencies, versions, distros etc [21:41:48] do this & it will work [21:41:49] some extensions have composer.json which might or might not list requirements [21:42:31] gwicke: well the mediawiki installer does check for requirements and bails out when one is missing [21:42:46] gwicke: so walk us through the end-user experience [21:42:56] in some occasion we released a version for which the installer suddenly required an optional dependency which broke the install [21:42:56] what would you expect? [21:43:08] follow instructions on how to buy a vm [21:43:13] run a command in that vm [21:43:26] and start using the wiki [21:43:36] upgrades should be automated [21:43:46] ideally, ssl too [21:43:49] LocalSettings.php tweaks should be discouraged [21:43:54] https://letsencrypt.org/ [21:44:01] (as in, not required) [21:44:19] gwicke: that sounds like a good vision, but I'm not sure on what the path to that vision looks like [21:44:54] "upgrades should be automated" -- like chrome, it just updates itself without asking? [21:45:00] what are the incremental changes we can make to our software to get to that end point? [21:45:08] or automated == there's some button in the web ui the user can press to upgrade the wiki [21:45:20] cscott: yes; basically, restart & upgrade once per night [21:45:28] or $period [21:46:00] you need a little bit of care to ensure the entire world doesn't hit mediawiki.org looking for updates at 12:01am UTC every night. [21:46:16] but yeah, i'm generally in favor of upgrade-by-default [21:46:41] you might start by building the upgrade-check functionality into core [21:46:56] ckoerner: does "upgrade once per night" seem viable with the hosts you're familiar with? [21:47:14] chrome has this, downloads the upgrade in the background, then changes color of the hamburger icon to indicate "an upgrade is ready, restart when you're ready" [21:47:41] we might have some sort of echo integration to message the sysadmin when a new upgrade is ready; that would just most of the same infra that fully automated upgrades would use. [21:47:44] the startup script in https://github.com/gwicke/mediawiki-containers fetches the latest containers on each startup [21:48:01] & runs migrations in each of them [21:49:24] I think the best way to make progress with a reasonable cost is to minimize the amount of variance we need to account for [21:49:27] I would not do auto upgrade [21:49:34] but having an ability to check for upgrade would be neat [21:49:46] i think there are two separate threads here (even if related) .. (a) what should wmf devs test/develop against in terms of promised support (b) what are the installation options available to users (and ease of use, upgradation, docs, etc.)? [21:49:53] as well as one clickable upgrade. [21:50:27] Jenkins does it http://blogs.mathworks.com/developer/files/2015InstallTAPPlugin.png [21:50:29] s/wmf devs/mediawiki devs/ [21:51:06] great point, subbu ! [21:51:11] to me, auto-upgrade seems tricky with breaking changes [21:51:12] one-clickable upgrade from the web ui? given a tar bundle, presumably? [21:51:42] or a .phar :-} [21:51:44] subbu: i think auto-upgrade entails a much greater attention paid to changes which might break [21:51:45] subbu: it requires good testing before releasing a new version [21:51:46] subbu: yes supporting automatic upgrade/migration forever is quite a constraint to live with int he long term [21:52:18] but, it's a lot more tractable to do this for a fairly constrained configuration than for hundreds of possible permutations [21:52:20] plus automated testing, of course -- we should be running the automated upgrades *before* they are pushed to users, so we can check any breakage before it hits the general public [21:52:24] gwicke, cscott it is not just testing and attention but the promise of migration scripts that gilles mentions [21:52:34] i am not yet convinced we are there yet. [21:52:34] but, to circle back to the original theme, shared hosting, could auto-upgrade work in that sort of environment? [21:52:42] subbu: yeah, but mediawiki-core already runs migrations on upgrade. [21:52:50] cscott: openstack does that. They run their patches against a test suite which attempt to upgrade from the last supported versions [21:53:00] cscott, what happens when there are changes to wikitext spec? [21:53:13] hashar: yeah. and i did implemented it with the upgrade system at olpc and litl as well. [21:53:14] RB also runs migrations on each startup [21:53:15] cscott: so install MediaWiki 1.23 + extensions, update to REL1_26 packages, run update, run tests. Report [21:53:33] subbu: same thing as when chrome updates the version of HTML they support [21:53:50] subbu: they roll it out in stages, and sometimes roll back an update if it breaks too much of the web [21:54:00] the one for database migration is named "Turbo Hipster" https://wiki.openstack.org/wiki/Nova/Turbo-Hipster [21:54:05] but every feature is first available behind a opt-in flag first. [21:54:08] if you are serious about supporting upgrades for non-technical users, you need to fully automate & properly test them [21:54:18] gilles: the main problem would be non-BC changes and extension upgrades. You need to be very careful not to try and run new-version code against old-version platform and vice-versa. [21:54:29] cscott: that's an even bigger infrastructural project. how do you even check that you didn't break 3rd-party wikis? [21:54:31] i am not saying it is not doable ... but is that the discussion we are having now? [21:54:57] gilles: you roll in batches to different corpus. That is what we do when pushing code to the wikimedia site [21:55:20] gilles: with various groups of wiki. mediawiki.org being among the first since it produces useful bug reports from end users [21:55:38] and IIRC mobile is doing the same. The Play Store has different channels [21:55:55] so you can push alpha, beta, tech savvy, US only, worldwide (don't quote me) [21:55:55] docker has tags for this [21:55:56] ok, auto-upgrade sounds like a cool project, but it doesn't solve how honest and documented we are about shared hosting support right now [21:56:09] which is what we should be discussing [21:57:00] I would organize the features in tiers [21:57:04] if extensions define that they don't support shared hosting, then I guess it's a non-issue and the fancy world of having control over your server can get all the goodies [21:57:16] and depending on what you platform provides, you would be eligible to some features [21:57:20] but we should have that clearly communicated [21:57:31] hashar: as opposed to organizing our features in tears ;-) [21:57:43] and again shared-hosts have different capabilities! [21:57:56] gilles: absolutely. And for those that do declare support for shared hosting, aka minimal config, at least core ones, there should be tests to ensure that [21:57:58] robla: :-)))))))))) [21:58:00] hashar, i think we define what we mean by 'shared hosting'. [21:58:25] rather than overload the term 'shared hostng' to mean all kinds of variabilities .. let hostin gproviders market their support as shared-hosting++ or +++ or something like that. [21:58:34] hashar: I think we need to define "minimally viable shared hosting" [21:58:43] to me shared-hosting is a host on which you have limited privileges such as the inability to install additional packages or upgrade versions. [21:58:43] we could have granular software requirement definitions, but there's no easy way to map that to which hosts support them [21:58:43] aka "config we use in testing" :) [21:59:04] "it's complicated" ;) [21:59:21] me.org+VE shared hosting for ex. [21:59:30] subbu: so we define the support buckets and ask shared hosts to state which one they belong to? [21:59:40] * subbu is handwaving at this point :) [21:59:43] and for that minimal version you really only need: file access (eg: FTP), data backend (sqlite or mysql), PHP 5.3.3 [21:59:53] we won't cover 100% but if we cover 80% that's good [22:00:20] maybe have a script that you could upload to shared host and if will check if it's ok? [22:00:41] SMalyshev: that is what the mw installer does [22:00:42] gilles, i was mostly trying not to go down the path of trying to capture all kinds of shared hosting solutions. [22:00:53] and make 'shared hosting' an umbrella term/ [22:01:02] we ensure the installer works with the bare minimum. At least enough to run the various checks [22:01:02] hashar: cool then, so that's not a big problem [22:01:06] I guess we could also have a list of hosts that support a given extension. Something collapsed by default. Ideally you'd want to be able to easily figure out the common hosts that support all the extensions you want... [22:01:37] and it'd be up to them or community members to keep that up to date [22:01:42] what I would like to see for each features their requirements [22:01:48] i agree with gwicke that we need a small subset of installation options that we promise support for. [22:01:51] eg I want VE, I need nodejs (because of Parsoid) [22:02:19] I want a powerful search ? We recommend ElasticSearch, suggest whatever over super search system we have an extension for [22:02:38] hashar: but this goes back to the novice user not knowing what node is. they know what dreamhost is, however [22:02:55] gilles: you don't, you get feedback from the community. that's how the HTML/JS standard is evolving. potentially-breaking features are rolled out slowly, with good feedback paths and a quick update cycle if something actually breaks. [22:03:05] click the link, popup a text box explaining what node is ? :D [22:03:44] that's a different user, who would probably be better served with simple instructions [22:03:51] shared hosting would be too complicated for them [22:03:53] gilles: if they know dreamhost, they know heroku then. (node-based shared hosting) [22:03:58] but for the ideal case they don't need to understand that, if they pick the easiest hosting option. it'll be installed for them [22:04:07] gilles: something like "node is used to run a server but is different from PHP. To benefit from WYSIWYG support in MediaWiki your host would need to provide node support". [22:04:58] hashar: what is WYSIWYG? [22:05:19] you are targeting a fairly technical audience if you are using terms like this [22:05:34] so maybe software requirement would work for shared hosting users then, who should be tech savvy enough to at least forward that list to a host and ask "do you have that?" [22:05:56] gwicke: you got my point [22:06:13] gwicke: explain in layman terms what node :-D [22:06:47] for absolutely non tech savvy folks, I guess the "one click install" hosts is a better match for them [22:06:56] hashar: most users shouldn't need to care what node is [22:07:04] unless you want VE [22:07:44] robla (apologies for the long delay) once per night seems viable. [22:08:10] gwicke, i think hosting providers can figure out how to do the marketing though. is it our job to figure out the marketing pitch for them so as to attract users? [22:09:07] alright, I'm going to close the meeting because time is well over, but don't let that stop you from chatting on [22:09:09] #endmeeting [22:09:10] Meeting ended Mon Dec 21 22:09:09 2015 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [22:09:10] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-21.11.html [22:09:10] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-21.11.txt [22:09:10] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-21.11.wiki [22:09:11] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-12-21-21.11.log.html [22:09:23] one click install, wysiwyg, node, no node, php, whatever .. it is their job. they can figure it out .. what they need is to know what combinations are supported by the mediawiki dev. [22:09:46] yes and what is actively being supported [22:09:46] subbu: I think it's our job to provide one tested & simple solution [22:10:04] at least one [22:10:38] gilles: thank you for leading the effort [22:10:39] I don't think that a packaged VM would help a host, they need documentation to figure out what the dependencies are. sure it's neat to have a working example, but it would be extra work to them to pick at the thing to figure out the dependencies [22:10:44] right. I would say that there are probably 2 or 3 combintations ... but let us not get distracted by how to market it to users. [22:10:55] gilles, thanks for organizing the discussions. [22:10:58] and to have setup those two sessions! [22:11:47] djfinining the requirements, scope and ensuring it works is the first step [22:11:51] that is already a lot of work [22:12:12] once we reliably release a set of extensions that are known to work with a single set of requirement [22:12:15] we can expand to more extensions [22:12:21] and different flavor of requirements [22:12:48] from there, it would be easy to provide a better distribution than simply a tarball [22:12:56] eg updater / .deb/containers [22:13:19] I think we're stuck with "typical shared host" being our smallest common denominator. That shouldn't stop us from us trying to move the needle on that definition and working with the biggest hosts to get there [22:13:54] like lobbying them to install service X which would unlock support for N of our extensions [22:13:59] webhosts do upgrade [22:14:13] and they usually want the most popular soft to be installable on their platforms [22:14:18] then that becomes part of our base support definition [22:14:33] so for our basic requirements, I think we have have close to 100% host coverages [22:14:46] for core you mean? [22:14:48] we can market it better expose it though [22:14:49] yeah [22:15:01] which is really core + extensions shipped in the release tarball [22:15:46] that is in mediawiki/tools/release.git in make-release/make-release.yaml [22:16:18] my english is crap sorry :( [22:16:55] a part which is missing that cscott mentionned is ensuring the released stuff does work properly on min requirements [22:17:21] so ideally we would grab the tarball, install it in an image that has solely the min requirements and run a bunch of smoke tests on it [22:17:33] our minimum requirements should have continuous integration, yeah [22:17:41] +1 [22:17:54] that is eventually something we talked about last year [22:18:17] but mark and markus had a lot of work going with supporting the actual release and getting in touch with endusers [22:18:29] and on releng side, we had other things to handle for wmf internal stuff [22:18:30] so, what would be the best candidate for the first package we'd want to lobby hosts to have for their users? parsoid? maybe something less ambitious? [22:18:42] parsoid is too ambitious [22:18:48] it is not even a core feature [22:18:51] imho [22:18:59] maybe ambitious, but definitively most useful. and the thing that users most want. [22:19:02] i think containers; npm-based install are solutions that some hosting providers might / could pick up if it proves viable for them. i think the prototypes that gwicke and cscott have worked demonstrate that. [22:19:05] +1 [22:19:09] to parsoid [22:19:13] it is a fancy sugar cream on the top of the wiki that "just" provides WYSIWIG [22:19:20] (don't get me wrong, I love VE / Parsoid) [22:19:48] subbu: I doubt it'll make shared hosting go away [22:19:50] the fancy sugar cream is what the users want [22:19:51] so, if VE is a sought after feature, then, it will push hosting providers to figure out solutions in those constrained environments. [22:19:56] getting hosting providers to install some package of some version once won't help much in the longer term [22:19:59] The MediaWiki stack holder group is volunteer driven and published a survey recently about third party uses [22:19:59] http://mwstake.org/ [22:20:02] gilles, sure, i am not saying it will go away. [22:20:24] that is similar to the community tech team at wikimedia reaching out to the community to ask for needs [22:20:45] then based on user feedback, establish a top X list of features that are really wanted [22:21:13] https://www.mediawiki.org/wiki/MediaWiki_Stakeholders%27_Group/Tasks/Feature_wishlist [22:21:16] Said wishlist [22:21:43] ckoerner: thank you , that is exactly the link I was looking for [22:21:50] gwicke: I think they do a pretty good job at keeping packages up-to-date actually, more so than people do on their own VPS on average. because for security reasons it's their in their best interest to have the latest stable [22:22:07] so there you have the list of stuff to do for better shared-host / 3rd party hosting :D [22:22:10] gilles, i guess i am saying we shouldn't try to solve the problem of trying to provide parsoid+ve in minimally viable shared hosting environments. [22:22:31] gilles: they'd subscribe to your repo? [22:23:05] I doubt that they'd grant us root on their boxes [22:23:58] gilles: you might want to capture the post meeting discussion :D [22:24:36] cscott: yeah everyone wants sugar nowadays :-} [22:25:06] subbu: why not? shared hosts aren't something static, they regularly install new software when it becomes popular and a frequent user request [22:25:14] cscott: but having php + node is a bit challenging. A few hosts approached on the PHabriactor task related to shared-host, maybe they can be reached out to to figure out how hard it would be to get nodejs for us [22:25:29] gilles, right .. that is what i am saying. :) [22:25:29] (or we could rewrite parsoid in PHP ) [22:25:34] hashar - that is a good idea. [22:25:48] hashar, ckoerner there is not even a html5 parser in php .. so, pipedream. :) [22:26:04] Sorry, I meant reaching out to hosting providers. :) [22:26:25] ckoerner: the mwstake group might have some contacts already [22:26:58] subbu: yup and it would make no sense to rewrite anyway :D [22:27:21] anyway gotta sleep [22:27:26] gilles: thank you again for the sessions! [22:28:05] gilles, i meant: if hosting providers want to break into the VE-mw market, then some might be willing to install node.js and provide a parsoid service shared between multiple wikis, for ex. [22:28:13] honestly I think that talking to them is the lowest effort for us. imagine if we convince the 5-10 biggest PHP shared hosting providers to run parsoid, then "VE shared hosting support" becomes a non-issue [22:28:17] no writes, no packaging... [22:28:21] *rewrites [22:28:40] you'd neet one parsoid instance per user for resource isolation [22:28:48] *need [22:28:58] and, how expensive is that? [22:29:03] resource-wise [22:29:06] about $5 a month [22:29:18] lol. [22:29:36] $5 a month gets me unlimited parsoid? :) can I run enwiki on that? :P [22:30:12] "unlimited" in the "unlimited 4G internet" sense [22:30:12] on the task, one person from a hosting company stated that to get parsoid / node they would need per user limitation/accounting [22:30:18] but maybe that can be added as a feature in Parsoid [22:30:27] (no clue how parsoid works really) [22:31:22] hashar, parsoid already emits stats about parse times on a per-request basis. [22:31:41] and we can enforce resource limits on a per-request basis too. [22:31:46] hashar: the standard solution for that kind of thing is a VM [22:31:58] does parsoid need to run as a service, or could it run on-demand in the same fashion imagemagick does? i.e. call the command, which does the processing and returns [22:32:27] i think cscott gwicke and tim startling were talking about different modes of doing that. [22:32:35] because the latter seems to be tolerate more easily by hosts, because it makes it easier to contain/track per user [22:32:42] * subbu doesn't remember all the various options they brainstormed [22:32:50] *starling [22:33:04] it's a lot slower, but not impossible [22:33:18] startup is a couple of seconds [22:33:58] would it rely on "external" permanent storage of data? if so, can that be easily user-contained? [22:34:19] newer features will probably, yes [22:34:33] it's fairly easy to contain in a VM [22:34:36] gilles, i think talking to hosting providers who want to go with this option is the best way to do this .. figure out what kind of support would enable them to turn this on. [22:34:42] anyway, I should stop this silly game ;) [22:35:24] subbu: right, less hypotheticals then [22:36:11] alright then, I'll try to synthesise all these discussions into a somehow useful session for the summit. maybe turn all of these ideas into a series of proposals for people to debate [22:36:50] gilles: add a key point which would be to reach out to some popular hosts and the one mwstake have contacts with [22:36:50] my feeling so far is that there are a lot of orthogonal things [22:37:04] maybe they would be able to give some requirements for parsoid/node and a shared host infrastructure [22:37:44] I think it wouldn't hurt to have a regular meeting with technical representatives from these companies on a regular basis [22:37:53] an instance per wiki is overkill for sure :D [22:37:54] we have so little visibility in what their users need/do right now [22:37:58] maybe they have a little clue? [22:38:05] specially when the wikis only have a few hundred hits per months [22:38:26] yup [22:38:51] and they would surely be interested in how wmf runs its wikifarm [22:41:41] you know one of my pipe dream ideas, which is listed on the phab task, is that we could have a say container-based wiki farm of our own, for which we'd make point-and-click management of wikis and extensions, which could become the new infrastructure for incubator wikis for example [22:41:58] and make it so good that commercial hosts just use what we've built [22:42:34] it would also allow us to expand the scope of what the incubator is for, which is right now just wikipedias in new languages, afaik, to being self-serve hosting of open educational content wikis [22:43:17] which would actually bring us a lot of institutions and small projects who are currently paying to host their wikis whose content fit in the scope of our mission statement [22:44:14] beta could live on that infrastructure too [22:44:55] we need something critical like beta to be on it anyway, if we want it to be truly supported. and by extension, mediawiki on containers would be too [22:45:27] I think that the weakest point of container support, if we don't create a wmf need for it, it's destined for being under supported in the long term [22:47:19] gilles: catching up [22:48:18] gilles: ahah that is a good trick [22:48:43] knowledge organizations uses wiki to share knowledge [22:48:49] sharing knowledge is WMF mission [22:49:02] -> how do we support orgs willing to share knowledge on wiki [22:49:07] -> WMF wiki farm :D [22:49:18] or another subsidiary / foundation [22:49:48] I think a self-serve wikifarm for open educational content is more decentralised and more in the original spirit of the foundation [22:49:48] I am not sure how the donor base of wikimedia will react to such spendings though [22:50:17] yeah it could be done by someone other than the wmf [22:50:25] this applies pretty much to everything in this debate [22:50:48] this could easily be a mozilla-like org too. for-profit for commercial wikis, non-profit for open wikis [22:51:04] and here comes MediaWiki Foundation :-D [22:51:05] https://www.turnkeylinux.org/mediawiki [22:51:21] "The mission of the Wikimedia Foundation is to empower and engage people around the world to collect and develop educational content under a free license or in the public domain, and to disseminate it effectively and globally. " [22:51:40] nowhere here does it say "hosted on a sanctioned wikimedia-hosted website" [22:51:42] sounds like it would fit in our scope [22:52:12] gilles: sorry, I went afk for lunch. Packaging for debian is just something I'm doing in my free time (and I was promised stickers if I finish it) [22:52:12] mutante: nice:-) [22:52:27] https://hub.turnkeylinux.org/amazon/launch/mediawiki/ [22:52:37] legoktm: I suspected as much... thanks for fighting the good fight [22:52:55] that last page has a Wikimedia Logo on it [22:53:13] as if we are using that product [22:53:28] gilles: so in the end we need some kind of strategy / vision [22:53:51] mutante: I guess someone mistook mediawiki for wikimedia [22:53:56] with strong buy-in from the donors, editing, readers and other knowledge organizations [22:54:12] then figure out a rough .plan and get the funding to back it up [22:54:14] oh no wait, it is a list of companies [22:54:14] odd [22:54:34] hashar: yeah something I can just whip up next weekend, right? :) [22:55:54] gilles: you should talk to Brion about it :-} [22:56:12] as i understand it [22:56:16] we all agree something needs to be done [22:56:30] but we have different opinion , try different approach like a huge bazar [22:56:45] when we might want to build a very small church and get followers [22:56:53] and build up from there [22:57:12] the http://mwstake.org/ is a good approach [22:57:20] [14:45:27] I think that the weakest point of container support, if we don't create a wmf need for it, it's destined for being under supported in the long term <-- Yeah, this. If there is no active maintainer of it, who has comitted time to fix and respond to bugs, it's not going to be great, like how pgsql support turns out. [22:57:32] would be nice to have a bunch of wmf folks more involved in it. the two sessions we had could have been held under mwstake [22:59:42] legoktm: which left me to wonder why you are even doing the .deb work :-) [22:59:55] though it is surely a fun hacking /side project [23:01:17] I think it would be pretty sad if MW wasn't packaged for debian/ubuntu. I also think there are a lot of MW users stuck on 1.19 because they don't have an upgrade path available. [23:01:31] Also, I wanted to learn how to create deb packages :P [23:01:54] legoktm: https://qa.debian.org/popcon.php?package=mediawiki :D [23:02:47] that list # of mediawiki installs for people participating in the popularity contest [23:03:44] so roughly 500 install (*vote*) [23:03:46] :D [23:05:24] it is not worth the effort imho :D [23:05:30] anyway sleep time! *wave* [23:06:47] if you are a user who installed mediawiki from the distro package, and then you have questions and go to #mediawiki, they are probably told to switch to the latest version from git? [23:08:30] we have https://releases.wikimedia.org/mediawiki/ but we'd need that to get to https://packages.debian.org/stable/misc/mediawiki more often [23:08:50] because the distro package is 1.19 (super EOL) yeah, you'll probably get told that [23:10:21] so the people who need help are pkg-mediawiki-devel@lists.alioth.debian.org [23:10:27] to make this less EOL [23:10:38] that would be cool [23:11:24] yeah, that's what I and a few other people are working on :) [23:11:31] :)