[06:56:54] (03PS1) 10Gilles: Add mysterious table to the union [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132920 [06:57:26] (03CR) 10Gilles: [C: 032] Add mysterious table to the union [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132920 (owner: 10Gilles) [06:57:58] (03CR) 10Gilles: [V: 032] Add mysterious table to the union [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132920 (owner: 10Gilles) [08:52:59] (03PS1) 10Gilles: Force move on write-protected files [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132923 [08:55:17] (03CR) 10Gilles: "Adding Ori who told me about window.Geo in the first place. See Gergo's last inline comment." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131435 (owner: 10Gilles) [08:56:43] (03CR) 10Gilles: [C: 032 V: 032] "Self-merge because without this change the next daily performance tsv build will fail." [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132923 (owner: 10Gilles) [09:22:30] (03CR) 10Gilles: UploadWizard: Add support for PostgreSQL (031 comment) [extensions/UploadWizard] - 10https://gerrit.wikimedia.org/r/132328 (https://bugzilla.wikimedia.org/64067) (owner: 10Jjanes) [10:21:05] (03PS1) 10Gilles: Global-specific section is no longer needed [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132930 [10:21:20] (03CR) 10Gilles: [C: 032 V: 032] Global-specific section is no longer needed [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132930 (owner: 10Gilles) [10:35:53] (03CR) 10Gilles: [C: 032] Fix url encoding in URL guesser [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132585 (owner: 10Gergő Tisza) [10:37:09] (03Merged) 10jenkins-bot: Fix url encoding in URL guesser [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132585 (owner: 10Gergő Tisza) [13:58:18] (03PS1) 10Gilles: Optimise slow queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132943 [13:58:58] (03CR) 10Gilles: [C: 032 V: 032] Optimise slow queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/132943 (owner: 10Gilles) [14:48:14] hey gi11es, when you have a moment … what i'm noticing in regards to thumbnail generation is that none of the code is available via the mw framework; it looks like it’s all in /core/thumb.php [14:49:39] dan-nl: in production it runs on different servers than the web ones anyway, that are behind varnish, etc. [14:49:41] as far as i can tell that won't work for what i'm trying to do. [14:50:04] dan-nl: even if right now imagemagick is available on web servers, it's generally a bad idea and long term I expect that to change [14:50:09] it would mean that i'd need to copy the code inside it, don't like that idea [14:50:40] and an http request to thumb.php also wouldn't work ... [14:51:09] why wouldn't an http request to the thumb's url work? [14:51:30] i guess for now i could try and copy the code in thumb.php that's necessary into gwtoolset … but are you already creating a Thumbnail class? [14:51:59] no, at the moment we're trying to fix specific things in the existing code, there is no big rewrite effort [14:52:20] as far as i can tell an http from the job to thumb.php could end up on any server and the idea is to use the tmp file that's on the same server as the job that's running [14:52:22] even if you could generate thumbnails elsewhere, I don't see how you could populate them [14:53:26] unless i kick off the http to thumb.php after the file is created [14:53:43] I see, it does sound like the plan is to manually push the thumb to swift, then. from a job server instead of an image scaler server, right? [14:53:49] i guess that would take care of it, i was thinking to create the thumb from the tmp file [14:54:32] yes, that was the initial thought, but i don't know how i'd push the thumb into swift [14:54:52] imho the best option is if there's a point in time where the image scalers can find the image to resize it for the thumb, and the file isn't in Special:Newest [14:54:54] and i'd have to copy code from thumb.php ... [14:55:10] then you just http query to the various thumb sizes and wait for the http requests to finish [14:55:16] one after the other [14:55:31] right that looks like it would be the best approach with what is available [14:55:34] this way the job takes care of priming the thumbs, and it doesn't have to deal with any of the thumb generation code itself [14:55:47] but not sure when the item would appear in Special:NewFiles [14:56:05] that's the critical part, I'm not familiar with the file lifecycle, I don't know when that happens [14:56:13] I thought you had that figured out when you mentioned that plan [14:57:00] could be that's what aaron thought, but it wasn't clear at the time … [14:57:14] and brion was pointing me to thumb.php [14:57:30] sounds like we should start a mailing list thread to figure it out in more detail [14:57:32] he was showing me the code i'd need to use [14:58:45] k, wikitech-l? [14:59:52] i think i'll try out the idea though … complete the mediafile upload in gwt uploadhandler and then http to thumb.php [15:00:18] the critical piece will be whether or not the title will appear immediately in Special:NewFiles [15:06:57] gi11es: do you know if the tmp file created during the job would be available to all servers? or would it only be available on the server the job is running on? if only on the job server, which i assume would be the case, then is it possible to http thumb.php only on that job server? [15:07:49] dan-nl: I'd say wikitech-l + multimedia [15:08:11] the tmp file would definitely only exist on the server handling the request, and I'm pretty sure that job servers and image scalers aren't the same [15:10:31] k, in that case i figure i'd have to complete the upload of the original file first before issuing an http to thumb.php, which would probably defeat the purpose of creating this patch since it would probably be in Special:NewFiles as soon as the upload completed. will look further and see if i can sort out an email for those lists to see if my assumptions are wrong … thanks [15:12:35] (03CR) 10Gilles: [C: 04-1] "Looking at these new functions, it looks like you could have all these new functions in a separate file pretty painlessly. Which would be " (031 comment) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132133 (owner: 10Gergő Tisza) [15:15:20] dan-nl: I know that there's a thing called the "stash" where uploaded files can live before being sent to NewFiles [15:15:35] I think its goal is mainly for multi-page documents that need to be combined on the server into one [15:15:45] but there's a possibility this could be used [15:16:18] I think it's also worth looking at the problem from the other end and seeing if we could easily mark files so that they don't appear on NewFiles until a separate call to do so is made [15:16:52] that's the real source of the problem. generating the thumb on other servers sounds pretty heavy im comparison [15:17:26] ja, that sounds like a better approach … have you and tgr been looking into that? [15:19:19] dan-nl: if I understand correctly, part of the problem was that lots of large files had to be moved between servers and that overloaded Swift [15:19:21] not yet [15:19:42] so any approach that requires copying of files between servers might run into the same problem [15:20:15] tgr: I think they're looking to have more control over generating the thumbnails before the file appears in Special:NewFiles [15:20:19] tgr, from what aaron told me, i understood that it was more about too many requests to create thumbnails from large tiffs [15:20:40] reducing swift reads might be achieved on the image scalers if for example we support requesting more than one size of thumbs at once [15:21:19] those thumbnail jobs filled the pool and some failed up to 5 times [15:21:46] dan-nl: yes, and the scalers started requesting the original files, and Swift ran out of bandwidth [15:21:53] that was one problem [15:22:06] the other was that the scalers could not handle the load [15:22:15] I'm surprised that faidon hasn't replied to my thread about capping said bandwidth, which would greatly mitigate the issue [15:23:07] the hackathon was quite busy … maybe he'll get a chance to catch-up this week [15:24:09] imho the reference thumb/chaining proposal would also help greatly. the biggest reference thumb would be orders of magnitude smaller than the original [15:24:16] so if you try fetch files as soon as they are uploaded, that might help with the CPU load issue but not with the Swift issue [15:24:16] but I'm just guessing, not familiar with Swift [15:24:16] but the Swift capacity issue is something that an improved scaler logic might also not solve [15:25:14] although creating the thumbnail sizes for both special pages which show new uploads would probably cut the traffic in half [15:25:17] seems that even if we hide the files from Special:NewFiles that might not take care of the issue … [15:26:13] it might help, but it is certainly not a safe solution [15:26:15] with chaining thumbnails, at least instead of reading the original every time we need to generate a new thumb size, we'd only read the original once [15:26:30] if 16 new large tiff files are created they still need to have thumbnails created and kicking them all off in the "background" would still be the same as in foreground [15:26:30] for example, someone might open the category page [15:27:08] I think the point would be that the file doesn't appear anywhere until a specific call is made [15:27:36] oh yeah, that would be better [15:27:48] which would give us control over when the thumbnail generation happens. if we chain thumbnails, it would allow us to generate the reference thumb befire the file appears on the wiki [15:27:49] right so that would limit the number of pool requests to create a thumbnail for a specific mediafile [15:27:51] that can be done by using the upload stash probably [15:28:23] when the mediafile is placed in the stash is it then available to all image scalers? [15:28:28] the easiest would be if thumbs of something in the upload stash can be generated by the image scalers [15:28:37] so an http call to thumb.php would work on the stash file? [15:28:45] no clue [15:28:56] sounds like that's the first thing we need to investigate [15:29:21] the user who uploaded the file to the stash can preview it, but whether that includes thumbnails or just the original... [15:31:55] one thing i'm unclear on atm, the image scalers, are they different than the regular webservers? for example a call to http://commons.wikimedia.org/w/thumb.php - is that a call to an image scaler? [15:33:42] image scalers are invoked through varnish when you hit urls like https://upload.wikimedia.org/wikipedia/commons/thumb/2/23/Multimedia_Vision_2016.pdf/page3-666px-Multimedia_Vision_2016.pdf.jpg [15:33:56] AFAIK commons uses a 404 handler to redirect requests for non-existing thumbnails to thumb.php [15:34:47] varnish checks if it has that thumb in its cache, if not then it hits thumb.php with the appropriate parameters on the image scaler cluster [15:35:12] or something that in turn calls thumb.php [15:35:36] then the php side will check if the thumb for that size exists in swift, return it if it exists, generate it, store it in swift and return it otherwise [15:35:53] so the gwtoolset job could hit http://commons.wikimedia.org/w/thumb.php with appropriate parameters referring to the stash file? [15:36:29] first time, create the reference file, then use it to generate the remaining thumbs? [15:37:17] the question is whether the final location for the thumb is known when the file is in the stash [15:37:51] or, alternatively, if it would be easy to move the thumbs generated in advance to their final location when the file graduates from the stash [15:38:23] what is the final location? [15:39:06] there is some sort of load balancing within the scalers, I suppose? [15:39:34] if so, guessing the right one might be hard [15:39:50] you don't have to guess if hitting a magic url does the trick [15:40:07] either the same url as the published file's thumb [15:40:15] or a stash-specific url [15:40:46] it all depends on whether or not at the stash stage you can tell at which url the thumbnail of the published file will be [15:41:33] guessing the eventual public thumbnail url should not be hard [15:42:33] then this might end up being a possilbe solution [15:42:56] i'll take a closer look at it tomorrow … thanks for your thoughts! [15:43:00] you just call the right function on the media handler class, IIRC it is doTransform which with the right parameters does not actually do the transform but does give you the URL [15:43:53] yeah, with self::TRANSFORM_LATER [15:46:38] hmm, haven't used mediahandler yet .. [15:50:27] you probably won't need to use it directly, there is a File::transform() which calls it [15:50:43] than again, who knows [15:50:47] :) [15:50:56] that part of the codebase is rather horrible [15:52:35] MediaWiki can be set up in several different ways, like generate the thumbnail on 404, generate it when the wikitext is parsed, serve it via PHP for private wikis... [15:52:37] hi gi11es [15:53:02] a user approached me today, and has an issue with media viewer [15:53:04] all on the same code path with lots of obscure conditions [15:53:15] using IE 9 he gets a blank screen [15:53:27] (on win 7) [15:53:57] in hebrew I presume, every single image or specific ones? [15:54:09] in hebrew, every image [15:54:13] screenshot? [15:54:42] https://imgur.com/BX879VL [15:55:40] do you have a link to the file he was trying to open, by any chance? [15:55:43] thanks both … i focus on this again tomorrow and see if i can get further otherwise i'll create an email thread as gi11es suggested earlier [15:56:03] i'll poke him gi11es [15:56:39] I'm giving it a try with me ie9/win7 vm right now [15:56:45] dan, did you see the bug i added you to ? [15:56:50] dan-nl: ^ [15:57:13] confirmed, matanya, I can reproduce [15:57:16] not yet, which one? [15:57:17] no need to bug the user further [15:57:26] thanks [15:57:57] dan-nl: https://bugzilla.wikimedia.org/show_bug.cgi?id=20444 [15:58:37] gi11es: does it need a bugzilla ticket ? [15:59:08] matanya: sure, we track our work on mingle, but a bugzilla ticket doesn't hurt [15:59:33] matanya: are you thinking of another dan? [15:59:44] dan-nl: possibly :) [15:59:50] dan garry ? [15:59:58] ja, that's another dan :) [16:00:10] sorry then :) [16:00:22] np [16:02:59] gi11es: https://bugzilla.wikimedia.org/show_bug.cgi?id=65225 [16:03:56] media viewer isn't triggering at all on beta for me right now... interesting [16:04:38] on any browser [16:05:08] tgr: can you get it to work on beta? [16:05:33] it works on mediawiki.org [16:06:24] same for me [16:07:10] works fine locally [16:07:13] doesn't seem to be loaded at all [16:07:15] it's as if it's been disabled site-wide [16:07:46] it shows up in the preferences, and I have it turned on [16:09:32] Error: Could not load thumbnail data. could not load image from http://upload.wikimedia.org/wikipedia/commons/4/42/DidoSotiriou.jpg [16:10:29] works here [16:13:31] it works fine after calling mw.loader.using('mmv.bootstrap') [16:14:21] which is weird, because I can see the call to load it in the page source [16:15:41] nevermind, the call is there on some pages but not others [16:15:48] some weird caching bug? [17:16:24] (03PS1) 10Gerrit Patch Uploader: Apply coding conventions [extensions/UploadWizard] - 10https://gerrit.wikimedia.org/r/132957 [17:16:27] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [extensions/UploadWizard] - 10https://gerrit.wikimedia.org/r/132957 (owner: 10Gerrit Patch Uploader) [17:56:32] gi11es: I did an edit on the lighbox demo page and it works now, so probably some sort of caching bug [18:03:00] gi11es: I've been trying to get access to EventLogging data for a weekend project; Ori said I need cluster access first and that should go through you [18:44:44] (03PS4) 10Gergő Tisza: Do not log a metadata event when the user presses the wrong key [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131586 [18:46:39] (03CR) 10Se4598: "note: the external tool must be moved to tool labs or a other location. "On June 30th, the tools on Toolserver will be stopped." (see http" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132133 (owner: 10Gergő Tisza) [18:47:22] ^ nothing new at this comment, just summed up what I already mentioned [18:47:39] (03CR) 10Gergő Tisza: "Just a move." (031 comment) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 (owner: 10Gergő Tisza) [18:49:53] (03CR) 10Gergő Tisza: "Well, the tests were changed slightly, but that is in the parent commit now." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 (owner: 10Gergő Tisza) [18:58:32] (03PS3) 10Gergő Tisza: Extract metadata panel scroll/log/animate logic into separate component [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 [18:59:03] (03CR) 10jenkins-bot: [V: 04-1] Extract metadata panel scroll/log/animate logic into separate component [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 (owner: 10Gergő Tisza) [19:18:32] gi11es: I can see some strange cache behavior on my local box as well [19:18:55] the JS code does not change after a branch switch and Ctrl-F5 [19:19:16] maybe there was some change in how resourceLoader caching is handled? [19:20:52] (03PS5) 10Gergő Tisza: Minimal zoom implementation [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132133 [19:21:13] (03CR) 10Gergő Tisza: "Rebase + took care of the inline comment." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132133 (owner: 10Gergő Tisza) [19:23:11] (03PS4) 10Jjanes: UploadWizard: Add support for PostgreSQL [extensions/UploadWizard] - 10https://gerrit.wikimedia.org/r/132328 (https://bugzilla.wikimedia.org/64067) [19:25:19] (03CR) 10Gergő Tisza: "Gilles: how would you split up the responsibilities of mmv.js? I agree it is bloated, but handling user actions (i.e. all the functions th" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/132133 (owner: 10Gergő Tisza) [21:26:42] (03PS4) 10Gergő Tisza: Extract metadata panel scroll/log/animate logic into separate component [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 [21:26:44] (03CR) 10jenkins-bot: [V: 04-1] Extract metadata panel scroll/log/animate logic into separate component [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 (owner: 10Gergő Tisza) [21:29:27] (03PS5) 10Gergő Tisza: Do not log a metadata event when the user presses the wrong key [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131586 [21:29:29] (03PS5) 10Gergő Tisza: Extract metadata panel scroll/log/animate logic into separate component [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/131618 [22:45:58] Re: Possibly the "blacklist" mentioned in https://www.mediawiki.org/wiki/Extension:PageImages#Configuration would be useful? (I just saw that, whilst searching for something else. i know nothing more!) [22:46:12] Re: "disable MediaViewer for some images" * that is. [22:57:55] quiddity: using image lists from MediaWiki pages sounds like a very inflexible approach to me [23:01:01] tgr, I agree. I was mostly pointing it out, in case it provided helpful supplementary lists, or might qualify as the "additional use case" that the mail-thread was speculating about, or could be skimmed for "edge-cases we haven't already thought of". [23:01:38] yeah, thanks, it was interesting [23:01:42] (Or something. I don't know database-fu, so can't even find the lists! No time for deeper digging.) [23:02:08] although it looks very experimental [23:02:39] https://en.wikipedia.org/wiki/MediaWiki:Pageimages-blacklist would be the blacklist, probably just a test [23:02:54] indeed. (It's used in the Beta Hovercards, which is what I'm looking at it for.) [23:08:14] thanks, quiddity !