[01:25:58] Hey all, I have several wikis load balanced across multiple web servers, with one instance of memcached per wiki. Is there any drawback or issue if I were to change to a single large memcached instance? [01:26:27] justinl: Performance should improve if you share the cache [01:26:29] FWIW, the memcached instances run on the same server, listening on 127.0.0.1 ports 11211 through 11216 [01:27:22] sorry, i meant the instances run on the same server as varnish and apache, so the apaches on each physical server only use their local memcached instances [01:27:43] I think if you have a lot of servers, something like nutcracker can help improve performance too, but I don't really know anything about it [01:27:57] not familiar with nutcracker, i'll google it [01:28:04] only 4 servers though [01:28:29] Also check out haproxy for Memcache.(Twemproxy/Nutcracker was a disaster the last time I worked with in front of Memcache.) [01:28:38] performance is generally fine as far as i can tell, i was more thinking of management of multiple instances and the memory overhead [01:28:51] MW generally assumes that you memcached is shared amongst all servers. Some features like rate limitting won't work properly unless memcached is shared amongst everything [01:29:46] I don't do any special tuning of mediawiki beyond setting the port and amount of memory for each wiki [01:29:55] sorry, tuning of memcached* [01:30:04] I would go with one Memcache instance/clustered servers of the same instance for multiple wikis. Though in my use case each key in Memcache is prefixed with the wiki's database name. [01:30:28] MW will generally prefix the cache key with db name in most cases [01:30:39] unfortunately I don't have the server resources do dedicate to memcached [01:30:53] there's a few exceptions for things that get shared amongst multiple wikis in a wikifarm which have unprefixed keys [01:31:03] eventually i'm moving my wikis to AWS so I'll use a big elasticache instance [01:31:30] If you have something like APC installed (or are using hhvm), MW will use apc for certain really hot keys [01:31:38] actually i'm not using a wiki farm, each wiki is its own copy of mediawiki, so yeah, i have 28 copies of mediawiki across my 4 servers :P [01:31:57] I do use APC (opcache with my upcoming 1.27 upgrade since it'll be php 7) [01:32:02] Well, that's still actually considered a wikifarm from a MW standpoint [01:32:18] I mean, APCu or whatever the in memory caching thing is called [01:34:51] Can't even make a virtual machine for the one Memcache instance? How big are your server instances?(VMs, Physical?) Realistically one or two VMs should handle 28 low to medium traffic wikis. [01:35:21] No, I don't have a VM platform in the environment where the servers reside. [01:35:40] Actually, I do but it's deprecated and being eventually eliminated, not really up-to-date for production services. [01:36:28] Sounds like you could consolidate your web hosts to two servers with databases on the third server and the fourth server for caching. Of course I am making assumptions about your setup here. [01:37:22] ^ Probably would be better for backups, too [01:37:25] The four web servers are blade servers to spread the load and provide sufficient redundancy and the database is a dedicated mysql master-replica pair [01:37:35] Oooh, nice. [01:37:47] justinl: If a significant portion of your traffic is not logged in users, having a varnish server set up can significantly reduce load on your apache servers [01:37:49] Plus a physical nfs server for the upload directory that the four web servers share [01:38:18] the four web servers also run varnish and we do have a lot of logged in users as well as anonymous [01:38:23] NFS/Gluster Cluster for the uploads is a good choice. [01:38:54] i'm stuck with the one nfs server as well, but again, since I'm moving to aws next year, that will move to EFS [01:39:22] If you're interested in making it real fast, it may be possible to set up varnish, too. I know Wikipedia uses it across servers, and has a little piece of software that queues the purge requests. Though may not be beneficial at your scale [01:39:38] Compared to the work required I mean [01:40:16] So you have Varnish in front of Apache or NGINX on each server? Normally I would split server duties in those cases. [01:40:21] I already have varnish in front of apache, with the memcached instances behind apache, all on each of the four servers [01:40:45] I'm not sure how it would be distributed. Varnish in itself can be a reverse-proxy I believe. [01:40:46] So would I if I had more servers :) [01:41:28] Hmm, sounds like the best way you could make this better would be switching to a VM setup to "isolate" each server's responsibilities. Like you said though, you plan on moving to AWS soon. [01:41:48] Re CZauX_ using multicast udp for varnish purges is probably a little overkill for a 4 server setup [01:42:53] One Varnish server would be enough for all of those wikis.(Two for redundancy.) The MW jobrunner service does a great job of handling purging to multiple Varnish servers anyway. [01:44:13] Since I don't have any more servers available to me, I have to keep varnish, apache, and memcached all on the same servers. [01:45:16] I am concerned, though about the job queue, I have a weird problem where apaches frequently get "stuck" waiting on responses from what I believe are job handling requests, so I end up maxing out the number of apaches on a server all the time [01:45:33] i actually have a cron job that kills apache every 5 minutes if it sees 100 apaches running (the current max setting) [01:45:39] justinl: if you're running a multiserver setup, best to run the job queue from cron [01:45:46] e.g. set $wgJobRunRate = 0; [01:45:59] and then run maintinance/runJobs.php from the command line via cron [01:46:17] tbh I don't use apache unless its an actual shared server setup where .htaccess is needed. [01:46:19] i've wondered about that, too, since i've seen it a lot, but currently we have the job run rate set to 0.1 for the two biggest wikis and 0.5 for the others [01:46:23] Nginx seems to do a better job [01:46:55] justinl: Jobrunner is a persistent service that can be used to handle job queues in real time. https://github.com/wikimedia/mediawiki-services-jobrunner [01:47:11] we can get a LOT of jobs at times since templates are very heavily used on the wikis -- i've sometimes seen well over 10,000 jobs appear from a single template update [01:47:22] I'll take a look at jobrunner, thanks [01:47:35] Wew. Have PHP7 or HHVM? [01:47:37] Oh yeah, the template propagation of server death. [01:47:53] php 5 right now since we're on ubuntu 12.04 but upgrading to 16.04 and php 7 in a couple of weeks [01:48:04] You're going to love those improvement ;) [01:48:12] I'd note that MediaWiki is better tested on hhvm than php7 [01:48:17] yeah, already running in my stage environment though i can't load test it [01:48:26] but moving to apache + php-fpm will be a big help i think [01:48:34] Nginx with PHP7.0-FPM is very fast in my setup [01:48:43] I've never used hhvm so i'm not at all familiar with it [01:48:55] I highly recommend NGINX over Apache. PHP7 FPM works wonders too. [01:48:58] Probably wouldn't bother with it, tbh. [01:49:03] hhvm I mean [01:49:09] Mostly because wikipedia uses hhvm, and everything wikipedia uses gets better tested [01:49:15] I've been using apache forever but do have nginx in a couple of places, been meaning to test it to see how it works with our wikis [01:49:38] I'll make a note to look at HHVM as well [01:50:10] Lots of people successfully use mediawiki with apache. I imagine both apache and nginx use are both fine choices [01:50:31] They are, though each have their distinctions [01:50:37] NGINX existing as competition to Apache made the Apache Foundation wake up and get working on performance. [01:51:01] I've always heard nginx is better with static content and apache with dynamic, but i imagine separating out the php to php-fpm mostly obviates that? [01:51:29] Pretty much. [01:51:36] PHP-FPM is a seperate process, so yep [01:51:44] Nginx just directs the flow of information [01:52:10] We have it in several places, several proxies and now instead of apache for the graphite servers I manage [01:52:18] Look into 'ondemand' mode for PHP-FPM, I've had some good success with performance with that. [01:52:28] will do, thanks [01:53:18] while I'm here and getting some really awesome advice, let me follow up on my earlier comment about stuck apaches [01:54:03] not sure why, but occasionally apache processes will hang awaiting responses from the external load balancer's IP address, which I believe from some straces are due to internal runjob requests [01:54:28] when that happens, apache waits forever and then eventually i hit the max number of apaches and further connections fail, so i have to restart apache [01:54:38] this happens without fail every 2-4 hours on all 4 web servers [01:55:03] That is odd, though I dont' have any experience using load balancers. [01:55:14] NGINX can run into the same issue, but the NGINX master process is good at detecting hung child NGINX process and nuking them. [01:55:26] unfortunately they're Microsoft IIS running ARR so not a lot of insight into them [01:55:48] rip [01:56:12] they're hung on read() calls from sockets to the load balancers IP front-facing IP address [01:56:35] yeah, this should go away too on AWS with their ELB, but it's still a pain in the ass [01:56:55] IIS/ARR also breaks page titles that have plus signs, apostrophes, and ampersands [01:57:17] so i've had to hack includes/GlobalFunctions.php to get those working and even ampersands still break the page editing url [01:58:11] justinl: I know why your read()'s are hanging like that. *Lots of furious typing.* [01:58:35] omg I would kiss you if you know the solution to that! [01:59:03] > Trela stopped typing ... [01:59:14] lol [01:59:22] T106793 [01:59:23] T106793: Pages with single quote in title are inaccessible by some clients (redirect loop) - https://phabricator.wikimedia.org/T106793 [01:59:35] justinl: There's also $wgLegalTitleChars which can change what's allowed in Titles [01:59:40] When MediaWiki runs jobs during the request process it flushes the PHP output before and runs the insertion into the database based job queue. However, the actual job queue process works by making a HTTP request during the PHP shutdown function phase to the wiki's public domain(Special:RunJobs) which goes out to your load balancers and back into the web servers of course. This are long running processes which can cause issues for your [01:59:40] load balancers. [01:59:58] yep, redirect loops. been there done that. if you google this issue, you'll probably find my posts all over the place asking about this [02:00:12] Trela: *furious reading* [02:00:13] In theory Special:RunJobs is supposed to exit before running the job [02:00:35] Assuming all close socket stuff happens properly. [02:00:42] So, basically you should switch to a cron job for the job queue or the MW jobrunner service. [02:01:01] at least i think its supposed to [02:01:07] Ok, that's the strongest reason I've ever read to move to the cron job method! [02:01:16] Special:RunJobs is a PITA, seriously, I highly doubt spawning a new HTTP process is more performant than running it on the same thread [02:01:46] but well, nobody cared to write a RFC for that, so that's what we get... [02:01:52] The wiki farm stack I work on was starting to experience this problem with the job queue a year ago and we switched to MW jobrunner. [02:02:06] I think the rationale is that its less performant, but the latency is supposed to be less noticable to the user [02:02:08] The question is how to balance the number of jobs to run per time period per wiki a [02:02:10] since its in a different process [02:02:51] I agree that it seems to cause problems to some users [02:02:53] So I may just need to play with the numbers. I do graph my job queue sizes for the 5 main wikis so that helps. [02:03:10] Oh I should add that we HEAVILY use Semantic MediaWiki so that results in a lot of work. [02:03:25] OH GOD NOT SMW. [02:03:27] * Trela runs screaming. [02:03:32] Whenever I do MW upgrades, I have to run the rebuildData.php script and on the biggest wiki that takes 4-5 days [02:03:50] !hss [02:03:50] https://upload.wikimedia.org/wikipedia/mediawiki/6/69/Hesaidsemanticga2.jpg [02:04:05] while if you run the job queue script, and there's no jobs, it just exits, so its ok to run it more than you need (As long as you don't go insane and run it twice a second) [02:04:13] Vulpix: LOL! [02:04:44] Yeah, I run it by hand sometimes if I see a lot of jobs taking a long time, and before upgrades to clear out the queue. [02:04:45] Whats wrong with Semantics? [02:04:52] I've never used it, tbh [02:05:29] CZauX_: The vanilla MW community and the semantic mediawiki community are kind of separate [02:05:42] There's significantly less overlap than you would think [02:05:46] Currently our largest wiki has almost 1.6 million semantic object ids so I'm sure that causes some kind of issues, probably why the rebuilds take days [02:05:49] There's nothing wrong with semantic mw [02:05:54] CZauX_: Lots of legacy code cruft that results in performance issues. So many performance issues that I have a corner of a room wide white board dedicated to making graphs describing them. [02:06:06] Hehehehe, alright. [02:06:10] Although some developers disagree with some of the things they do [02:06:11] What does it provide, though? [02:06:14] we also have legacy DynamicPageList stuff that will probably never disappear either, but everything new uses SMW [02:06:17] SEMANTICS! [02:06:40] justinl: The "Wikimedia" variant, or the third party variant? [02:06:47] Because the third party variant is a shit show [02:06:52] justinl: If you ever want to migrate away from SMW. https://www.mediawiki.org/wiki/Extension:Cargo [02:07:09] I've actually been meaning to look into contacting someone for MW professional consulting, there's probably a ton that could be done a lot better both with our MW and SMW architecture [02:07:10] bawolff: Which third party variant? DPL2 or the DPL3 that I rewrote entirely out of DPL2? [02:07:34] I think the one labelled DPL2 [02:07:40] DPL2 [02:08:02] The one with the 100,000 line function in it [02:08:06] looks like we're on 2.3.0 [02:08:25] DPL3 is a drop in replacement for DPL2 that fixes lots of performance issues. https://www.mediawiki.org/wiki/Extension:DynamicPageList3 [02:08:38] I should check on that for the upgrades in a couple of weeks! [02:08:56] Older versions of DPL2 had massive XSS issues in it [02:09:03] So be warned about that [02:09:18] good to know, thanks! [02:09:29] They literally set $wgRawHTML = true; for the wiki... [02:09:45] Flawless security [02:09:55] Trela: That's interesting, I haven't seen your rewrite before [02:10:17] I'm sure its much better than DPL2 [02:10:57] I lost two weeks of my life rewriting that code base. It is a blur. [02:11:11] TBQH, I'm not a MW developer or even editor, I'm a Unix systems engineer, so I just provide the platform and our large userbase does all of the content editing, including all of the templates and SMW stuff [02:11:26] justinl: Of course, please keep in mind that all DPLs/SMW/cargo whatever else allows your users to make really expensive SQL queries [02:11:50] if you have concerns about being DOS'd, you should not install any of those extensions [02:12:04] On the topic of Mediawiki things, would it ever be possible to have a yml configuration file? With frameworks like Drupal, I can pre-make the server configurations, and check the values already on a wiki. But because LocalSettings.php is... php, Its a lot more difficult. [02:12:16] yeah, occasionally we get huge spikes on our database master but in general it handles it pretty well (lots of cores and memory) and we use memcached (hence my original question) [02:12:21] Also Cargo is scary unless you have it use a different SQL user with different rights [02:12:37] CZauX_: Not YML, but MediaWiki is moving towards that direction with JSON configuration for extensions and skins. [02:13:02] CZauX_: we don't natively provide support for that, however, since config files are code, you could probably do that in a couple lines [02:13:26] We've used DPL and SMW for years and they're tightly integrated. Our big wiki has 58k pages, 156k articles, and 1.7m semantic values. [02:13:53] Thing is, I've been wanting to create a kind of MW deployment tool. Wordpress has EasyEngine, Drupal has Drush, MW doesn't have anything like that to my knowledge. But, the nature of mediawiki makes it difficult currently [02:13:56] e.g. Have LocalSettings.php basically just read a yaml file and extract all values to be globals [02:14:31] CZauX_: I built that tool, but I am not allowed to open source it. :( [02:14:41] :( [02:14:48] I plan on writing a deploy tool for myself to streamline patches, updates, config changes, etc. I use SaltStack so I may just leverage that. [02:15:35] Its certainly something we could use [02:15:57] Making MW easy to install is an area that deserves more attention than it gets [02:16:38] Oh I know, this spawned all from me trying to make a wiki for a community [02:16:41] The default installer is great for single wikis, but not past that. [02:16:44] When I upgrade, I update one system and rsync to the others :P [02:16:48] Tryign to setup a mediawiki site propertly is the biggest PITA. [02:17:49] justinl: You should look into Chef for server deployment. [02:17:56] ^ [02:18:17] Composer is another mixed bag. I was surprised by MW 1.25s change to it and I'd never used it, and since only some extensions, including SMW and some of its extensions that we use, it's really frustrating. [02:18:36] Trela: I switched from Puppet to SaltStack last year. :) [02:18:37] justinl: The semantic people really like composer [02:18:50] Composer is pretty cool, not everything uses it yet, though. [02:18:54] Most of the rest of the MW extension community don't really [02:18:56] Composer is great if you don't have SMW installer. SMW 100% screwed up their Composer usage which breaks anything else that uses Composer. [02:19:03] *installed. [02:19:24] MW core itself really likes composer for installing dependencies, but not really for extensions [02:19:30] The dream is to have a pure composer updated site, so you can just run Composer Update, and d0ne [02:19:39] Composer isn't that bad, but what sucks is that only some of our extensions use it, so I need it for some and then manual pulls from the extension distributor, such a painful manual process since we use like 25 extensions. [02:20:35] CZauX_: One update would be nice, but then I run into the multiple server issue. From what I've seen online it seems most MW installation are single server except for Wikipedia itself. I may be wrong, but that's what my research tended to turn up. [02:21:03] Most guides/configs are for single server setup, and multi site setup gets pretty complex [02:21:08] *multi-server multi-site [02:21:12] There are a few medium size installs [02:21:16] So I want to set up a single deploy server, update it, and then have SaltStack or some script rsync or whatever that content to all of the live servers [02:21:35] Yeah, being multi-server and multi-site is painful at times. [02:21:39] Don't use composer to update on all servers. Use composer(Or anything else like a respository) to update a copy on a staging server. THEN once that is update and confirmed stable push those changes out to all servers using a deploy system. [02:21:42] But you're right, there docs are dominated by the single server case [02:22:30] That's what I do, I update a stage server and then when ready, rsync to a live server without overwriting config files (with rsync rules) and then rsync from the one live serve to the rest. [02:22:39] It's ugly and I'm working to improve that. [02:22:42] Even with single server case, a lot of guides on setting up MW are... dated, or don't really explain how to get what you /want/. A lot of it has to do with the complex tech, though. Just having the Math extension requires installing and configuring a Node.JS server, lol [02:23:17] There's a portion of WMF that really likes node [02:23:27] they want to make everything into node.js services [02:23:28] Yeah, some users are asking about being able to upload and have sound files available in the wiki pages and I recall that requires some external stuff. [02:23:37] There's another half of the foundation that hates this idea [02:23:38] Use a git repository for deploys. Commit changes locally then the deploy script runs the necessary "git fetch", "git checkout blaaaaah", and such commands in the repository checkout on each server. [02:23:39] Oh I love node, but its an absolutely memory hog with things like Visual Editor [02:23:46] The fight has been ongoing for like 4 years now [02:23:55] Visual Editor in idle eats up 400MB of ram last time I tried XD [02:24:07] Then you do not have to worry about rsync rules and overwriting files. [02:24:08] O_O [02:24:30] I've thought about integrating my process with our internal Perforce server. [02:25:39] Like I said, I have a lot of work still on the MW config management, revision control, and deployment fronts [02:26:19] The more wikis you add the more you convince your bosses the time investment to make it easier is worth it. :P [02:26:24] *you can convince [02:26:41] Better to do it right, than to fail, and then do it right. [02:26:59] atleast in that case [02:27:12] Fortunately I don't see the number of wikis growing, just their sizes. But I definitely concur with that sentiment. :) [02:28:29] I'm actually having to upgrade the operating systems to move to MW 1.27 due to the PHP requirement and without more servers to do it in parallel, I've come up with a process to minimize downtime but care must be taken. :P [02:29:58] There are some differences between 12.04 and 16.04, I can tell you that [02:30:19] Mainly good performance differences. :D [02:30:37] Could use some help. Posted a detailed technical question to the support desk at https://www.mediawiki.org/wiki/Topic:Tfly0768242qozhc Appreciate any assistance. [02:30:50] and you can run HTTP2 with the OS handling the openssl version [02:31:22] Yeah, I've upgrade my stage environment to 16.04 and I've built/upgraded other internal systems to 16.04 and I'm getting used to a lot of the big changes, like systemd, but overall I'm really liking it. [02:31:32] IowaJason: How did you install MediaWiki? [02:31:46] Did you unpack the tarball into an empty directory? [02:32:42] bawolff - Thanks - I did not. I overlaid on existing directory. [02:32:48] IowaJason: Can you try unpacking on a fresh directory just in case some of the old files are mixed in with the new? [02:33:21] bawolff - sounds reasonable. I will try. [02:33:22] justinl: In theory it should be possible to do it without downtime, although probably not the easiest thing to do [02:34:34] justinl: Is your downtime due to having to update the db? [02:34:34] I mean schema updates? [02:35:16] bawolff: For general maintenance I do it one server at a time, taking it out of the load balancer server farm, do maintenance, add it back, and move to the next one, but since upgrading requires database schema updates that probably wouldn't be compatible with the version on as-yet non-upgraded servers, that won't work. [02:35:59] bawolff: Yep! [02:36:17] And then I do the SMW rebuildData.php script for each wiki [02:36:36] Why do you rebuild SMW data? You don't have to. [02:36:39] so some things can look broken until those complete [02:36:56] Oh, because I'm also upgrade SMW 2.3 to 2.4 at the sam time [02:37:04] Shouldn't have to. [02:37:05] You can do things like take the slave db server out of rotation, do the schema update to the slave server, make the slave the new master, take the old master out of rotation, do the schema update there, and then set it back up [02:37:05] upgrading*, same* [02:37:15] or so I'm told [02:37:26] I'm not upgrading the database servers, just the wiki web servers. [02:37:27] I'm never administered a multi-db MW farm [02:37:33] Which MW version are you currently on? [02:37:48] but the mediawiki upgrade process is basically just upgrading the db server [02:37:57] or at least that's what update.php does [02:38:10] obviously there's still changing all the mediawiki files [02:38:10] 1.26.4 [02:38:52] Oh, suuuuuper easy to upgrade to 1.27 with no down time then. [02:39:14] So I'm upgrading four web servers running Ubuntu 12.04 to 16.04, MW 1.26.4 to 1.27.1, and SMW 2.3 to 2.4 but not touching the database servers except for the update.php and rebuildData.php scripts [02:39:43] Lines 278 and 279 in includes/installer/MysqlUpdater.php. Temporarily remove the [ 'dropTable', 'msg_resource_links' ], and [ 'dropTable', 'msg_resource' ], lines. [02:39:52] Plus it's moving from Apache with mod_php to Apache + PHP-FPM [02:40:17] MW 1.26 will continue to run happily and MW 1.27 will run in harmony with 1.26 on the other servers with the rest of the updates in place. [02:40:17] Make sure to backup everything [02:40:23] multiple times :^) [02:40:25] bawolff - success. Thank You! I deleted only the includes and vendor directory, recopied 1.27.1 versions, ran composer update and started Apache. Working as designed. [02:40:52] Note, if installing from the tarball you normally don't have to run composer update [02:40:55] I backup the databases three times per day and always do my upgrades right after a backup. :) [02:41:14] all the composer dependencies should be included in the tarball [02:42:04] IowaJason: btw, do you mind posting on the support desk thread what the solution was, in case others have the same issue? :) [02:42:24] bawolff- I will do within next 10 minutes. [02:42:31] thanks [02:44:41] Trela: So I've never even looked at that file before, I'd definitely need time to test that kind of tweak to change the upgrade procedure, but it may be worth investigating. However, the upgrade day is already scheduled (having been delayed already) so changing the procedure now might be difficult. [02:45:42] I did that to upgrade the farm I work on and it worked beautifully. So it is tested, technically. :P [02:46:48] Trela: :) I'd have to convince our community manager [02:47:19] I'm actually supposed to be out on PTO and vacation all next week with the upgrades happening the following week, so it's definitely tight on time. [02:47:48] You're like a mirror of me. I am out all next week on PTO too. :O [02:48:20] We get Thu and Fri off so I figured I have plenty of PTO saved and I'd just burn 3 days to get a whole week off, it'll be awesome! [02:48:40] Best plan! [02:48:58] Trela: Question about that no-downtime upgrade procedure: Could SMW complicate matters? [02:49:18] Or any other extensions for that matter, we use a couple dozen. [02:49:24] You really do not have to run rebuildData.php upgrading from SMW 2.3 to 2.4. [02:49:34] From the 1.x series to 2.x series? Definitely. [02:49:43] Any extension could potentially have schema changes that are not back compatible, although its kind of weird [02:50:20] you could run php update.php --dry-run to see what the update script does [02:50:34] Really, even though they suggest it? I'm very ignorant of SMW internals so I always do since it's the recommendation. I'd really like to understand MW and SMW internals better, though it's hard to sit down and spend the time doing so since I have so much other stuff on my plate, too. [02:50:34] And check for anything destructive (dropping tables, columns) [02:51:14] Actually I'm very ignorant of MW internals, too, I don't even really do any editing, so I don't even know most MW syntax. [02:51:43] SMW suggests a lot of things that do not need to be done and/or are bad practices. 1.) You can do the full upgrade on all the bits and pieces then let the SMW rebuilt go. 2.) SMW will rebuild automatically when people edit articles. [02:51:54] bawolff - Support desk question marked as resolved with the technical details you provided. Thanks. [02:52:00] :) [02:52:30] justinl: In fairness, MW syntax is pretty stupid in a lot of places :P [02:53:25] bawolff: Maybe, I'll take your word for that since I just don't know enough to judge, but nevertheless I feel I should really know it to better support the platform we provide our community. :) [02:54:16] Definitely feel that understanding SMW better would help due to the crazy size of our database and number of SMW objects and properties, trying to improve overall performance, data rebuild times, etc. [02:55:04] And the job/stuck Apache issue has been such a pain and I've felt my lack of knowledge about the internals hampered my ability to troubleshoot better, despite the reading I did just to get as far as I did. [02:57:52] Speaking of, how do you find work to sysadmin a wiki farm? Just a standard web sysadmin job that turns out to be for mediawiki? [02:58:46] Video game studio, the wikis and some other stuff (more now thanks to me) run on Linux. [02:59:30] Have you tried other storage backends for SMW? [02:59:45] I know very little about SMW performance, but i imagine that could help [03:00:07] CZauX_: Are you a sysadmin? [03:00:59] I've never investigated other backends at all. I inherited two wikis when I started the job and they were already on MySQL so when I had to stand up new ones, I just used that same backend and never thought about it. [03:01:12] Not proffessionally, yet. I've learned a lot about how to host webservers and create sites with different frameworks, but haven't foudn a way to market that yet :P [03:02:00] There are people who do make a living administering MediaWiki installs [03:02:20] Although I don't think its super easy to eek out a living that way [03:03:02] I think there's especially a lot of short term jobs - install the wiki for someone, and upgrade it ocassionally [03:03:05] The wikis are a significant portion of my workload but there's a lot of other stuff too. By themselves, they probably wouldn't be enough to justify a full-time engineer salary. [03:03:31] See for example contractor job boards like upwork.com [03:03:56] If you can do small extension development work (e.g. modify things to meet people's needs) there is a market for that [03:04:05] but not a huge one [03:04:58] I'm familiar with making extensions, recently tried to make a universal oEmbed solution, but won't be possible until V2 of the oEmbed standard is drafted and adopted, which is years from now XD [03:04:59] I would kill for a software engineer that already knows Mediawiki before being hired. [03:06:45] Thats the odd thing, really. A lot of people say stuff like that, but I can never find actual applications that have to do with the work being done. Heck, even some of wikimedias intern positions aren't descriptive XD [03:07:37] CZauX_: Oh that's cool. I always thought it'd be neat if commons supported oEmbed [03:08:14] CZauX_: Yeah, it seems like people complain both that there is noone to hire, and people complain that nobody will hire them [03:08:18] I don't know what's up with that [03:08:47] CZauX_: Do you mean actual WMF intern positions, or GSOC/Outreachy "interns" [03:08:58] My problem with hiring the people goes, "So, would you be willing to move to Alabama?" *Clicking noise of phone hanging up.* [03:09:39] WMF intern, I believe. There is a Technical Design Intern position currently [03:09:45] right [03:09:55] Usually those things aren't actually MW development [03:10:07] It tends to be other teams at the foundation that hire interns [03:11:09] The technical intern position currently up there is an outreach or grants thing I think [03:11:20] Ahh, that makes sense. [03:11:34] I have no idea what they are doing, but I doubt they are actually working on MediaWiki [03:11:43] They might be writing some scripts and creating pages [03:12:50] Honestly, its kind of hard to get hired by the foundation as a junior developer [03:13:15] They do often hire out of the volunteer developer pool though [03:13:23] Yeah I'm not sure what that job entails really. I know they want some front-end stuff and widget development, hence the Javascript and Lua. Dunno why they want Python, and would be odd to have MySQL requirement unless you're actually manually migrating stuff [03:13:47] So contributing to MW core can be a good way to get a foot in the door [03:13:58] if you're a student, GSOC can be a very good way to get yourself known [03:14:33] I find it kind of weird they list MySQL as a programming language instead of just SQL [03:14:47] although technically i guess mysql is slightly different from generic sql [03:15:01] Originally it was for students only, so I'm not sure if some of the requirements are even right *shrug* [03:15:20] I'm building Drupal sites now anyways, so :^) [03:15:34] Probably they want MySQL so that the person can run one-off ad-hoc queries to get statistics about whatever it is they are making the person do [03:16:37] Yeah I haven't taken a lot of time to learn actual SQL commands. more so how to sanitize input and secure the users. [03:17:41] Well honestly, I think that's a preferable skill set [03:17:48] but that's just me [03:17:56] And I currently work on MW security [03:18:27] so I'm biased [03:19:28] Also, wtf is with all the veteren stuff on https://boards.greenhouse.io/wikimedia/jobs/487209?gh_src=w3ekxu1 [03:20:13] I believe you have to indicate the kind of veteran, not sure if thats a state thing or not [03:20:43] That just seems weird [03:20:53] Law bit where employers are required to ask, but veterans are not actually required to answer. ?_? [03:21:06] Maybe its because I'm not an american [03:22:24] Basically it comes down to data collection to validate the government efforts to make sure veterans are getting jobs in society after returning from active duty.(Very small short cited overall of what it all entails.) [03:59:21] UpWork requires you have an education listed with a degree. I hope they appreciate Wizard school. [04:00:04] lol [04:00:11] Do they really, that seems stupid [04:00:48] In the application process if you don't fill it out it says "You must have atleast one educative experience" [04:00:48] Keep in mind also, that lots of jobs on upwork don't pay all that well (but some do) [04:00:54] lol [04:01:49] I do Point of Sale contracting, so I'm familiar with atleast some of it. [04:04:18] ah PoS, where both full versions of the abbreviation can mean the exact same thing >.> [04:05:02] In my experience, that is quite accurate. [04:31:18] Hey all, me again. Any idea why if I run the runJobs script with say --maxjobs=1000 on a wiki whose job table has like a handful of entries will still run 1000 jobs? [19:15:46] hi. If anyone has time could you please check out https://phabricator.miraheze.org/T1000 ? (Comments / BlogPage errors) [19:16:25] Reception|away: IT's very obvious [19:16:31] Some schema change hasn't been applied [19:16:51] Or, the query is just wrong [19:17:28] What do you mean by "wrong"? What can I do to fix it? [19:17:41] !update [19:17:41] update.php is a script that updates the database tables. You usually need to run it after upgrading MediaWiki or installing certain extensions. For details, see [19:17:49] ^ Reception|away [19:18:12] I see no database tables in the blogpage extension [19:18:15] if that doesn't work, read the documentation of the extension, since it may require manual intervention [19:18:34] Reedy: the database tables are from the Comments extension [19:19:09] And Comment_Plus_Count doesn't exist in the schema for that either [19:19:47] So it looks like BlogPage is potentially working against an old schema version of Comments? [19:20:19] This is the Comments version we currently have: https://github.com/wikimedia/mediawiki-extensions-Comments/tree/REL1_27 [19:20:31] https://github.com/wikimedia/mediawiki-extensions-Comments/blob/REL1_27/sql/comments.sql [19:20:42] There's no Comment_Plus_Count column [19:21:19] and there should be? Where is this Comment_Plus_Count column coming from? Is it the BlogPage extension? [19:21:32] https://github.com/wikimedia/mediawiki-extensions-BlogPage/blob/REL1_27/BlogPageClass.php#L948-L949 [19:21:40] BlogPage is trying to use it [19:21:44] Should there be? I've no idea [19:21:47] Yes.. I was just looking at that.. [19:21:59] You'd be best asking ashley probably [19:22:12] Something looks out of sync [19:22:16] Or it's depending on a patch never merged [19:22:30] Ok. Thank you very much for your help! [19:24:06] Reception|away: The blogPage code looks weird [19:24:21] column aliases shouldn't be done like that [19:24:24] lol ashley [19:24:39] Ok.. [19:24:42] I've seen https://gerrit.wikimedia.org/r/#/c/275198/ [19:24:49] would this be a potential fix? [19:25:31] It doesn't change the Comments DB, and it doesn't change the SQL query BlogPage is trying to run, so I'd say no [19:30:48] also note that extensions maintained by Ashley don't support branches https://phabricator.wikimedia.org/T149508#2754752 so REL1_27 may be very well broken [19:31:22] Vulpix: it's broken on master in the same way [19:42:31] matrix failed xD [19:55:03] Question: So lets say I created a page on my Wiki that had a link to a non-existent file, I then uploaded a file after the page was created, however on after updating the page and said file the file is still showing "There are no pages that link to this file." I have tried using the Purge Extension with no effect, so how would I go about fixing the file to show it actually is linked on a page? I was looking @ Rebuildall.php and [19:56:20] probably a Job queue issue again [19:57:39] I'm still new to mediawiki lol, what would that be? and also all of my images are showing they aren't linked anywhere even though I know that they are, because they were all uploaded after I created the pages for them.. [19:58:15] Iomega0318: can you check if there's a large amount of pending jobs by running https://www.mediawiki.org/wiki/Manual:ShowJobs.php (from a command line) [19:58:38] Let me see if I can, I'm on shared hosting. [19:59:09] I think the api will also show you the number of jobs [19:59:19] or just query the job table [20:02:03] bleh well I will have to check the table then, I'll have to fix something to use SSH on MediaWiki later grr.. one sec [20:05:09] Iomega0318: from the api you can also check it, something like https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=statistics (but pointing to your wiki) [20:06:03] awesome that was way faster lol, "jobs": 1496 [20:06:27] I'm guessing that's not good haha [20:07:05] Reception|away: yeah, it's broken on master, there's a bug about it as well as a proposed patch (by me); as a workaround, you can set $wgBlogPageDisplay['comment_of_day'] = false; in your wiki's LocalSettings.php [20:07:37] Thank you very much! [20:09:23] https://gerrit.wikimedia.org/r/#/c/275198/ is indeed the patch which I should clean up and merge before the end of the year or something...the code is kinda sucky and complicated and whatnot, hence why I still haven't gotten around to that [20:09:46] of course it sucks that this odd quirk isn't documented anywhere *and* it's been "broken" this way for quite some time... [20:12:16] ashley: I did the change (https://github.com/miraheze/mw-config/commit/16d2abf6a410823f0c0ef2b88355182a8a6dd067) but still getting the error [20:13:36] ah, sorry, it's 'comments_of_day' (plural, not singular), my bad (also you should probably be using wfLoadExtension( 'BlogPage' ); instead of require_once()'ing a deprecated PHP entry point) [20:15:27] Iomega0318: the job queue has some issues in MediaWiki 1.27, you can see some background on https://www.mediawiki.org/wiki/Manual:Job_queue [20:16:06] ashley: thank you very much! It now works fine [20:16:36] yw [20:16:36] Iomega0318: your only option until 1.27.2 is released is to set $wgRunJobsAsync = false; in your LocalSettings.php since you're on a shared host and I guess setting up cron jobs may not be possible [20:17:23] Is there a way to tell if a rendering is for a preview or for a page save? I'd like some expanding/collapsing boxes to be expanded if preview, collapse by default in "real" [20:18:01] Ahh okay gotcha, and I can set up cron jobs through my hosting page so there's that. [20:21:39] Iomega0318: ok, well, setting a cron job to run runJobs.php periodically might help to clear the queue [20:22:16] well, it will absolutely help in cleaning the job queue, not "may" :) [20:24:52] haha I will see if I can set one up, not sure if it will work though I had to use an htaccess file to change my PHP version to use MediaWiki which was my issue earlier using SSH, bleh to shared hosting.. [20:34:47] Job created successfully, now just need to get it to work lol.. same issue as SSH.. [20:36:19] $wgRunJobsAsync = false should help, then on each wiki request a job might be picked depending on the value of $wgJobRunRate [20:44:10] Well my issue is that /usr/bin/php points to 5.4, so trying to figure out where 5.6 is located lol so it will run lol.. if I refresh my main wiki page the job queue does go down by one each time if that makes a difference lol.. [20:45:38] yeah, that's expected, the wiki will pick one job on each request, by default [20:46:17] (unless it's broken as it could beyour case before setting $wgRunJobsAsync = false) [20:49:45] Roger that and that is now set as false, still working on the cron job though.. [20:59:10] sheesh finally figured out where the php version was stored lol.. Arvixe hosting sucks especially in documentation.. been going to SiteGround to read theirs since the hosting is similar lol.. [20:59:23] also got the cron job runnin correctly and guess what!! "jobs": 0 [21:00:45] just checked and my file is now showing it is linked somewhere, and on top of that my site seems MUCH faster, I guess those jobs were impacting performance haha.. [21:01:06] so thank you very very much for your help I never would have been able to do that without it! [21:05:53] yw :)