[00:12:55] (03CR) 10MarkTraceur: "Patch with fix incoming." (032 comments) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [00:31:02] (03PS4) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 [00:31:03] (03CR) 10jenkins-bot: [V: 04-1] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [00:31:14] (03PS5) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 [00:31:14] My bad. [00:31:15] (03CR) 10jenkins-bot: [V: 04-1] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [00:31:17] ...huh [00:33:39] (03PS6) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 [00:33:48] Fucking git and its rebase complaints [00:37:11] ...what the eff [00:37:20] https://meta.wikimedia.org/wiki/Tech/News/2013/49 [00:37:31] I'm not sure why this is going out only now [00:37:37] Because it was there for a month IIRC [00:37:47] (the CMD entry) [00:48:18] marktraceur: This thread about retrospectives on the teampractices list may be worth reading -- http://lists.wikimedia.org/pipermail/teampractices/2013-December/000191.html [00:51:12] Thanks! [00:59:39] (03PS4) 10MarkTraceur: Create smoke tests for the LightboxImage class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100427 (owner: 10Aarcos) [00:59:48] (03CR) 10MarkTraceur: [C: 032] Create smoke tests for the LightboxImage class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100427 (owner: 10Aarcos) [01:00:07] bd808: I think we may need to have more formal implementations of loads of scrum principles before it would work very well [01:00:35] Probably that would take the team getting together and having a sane conversation about structure [01:00:36] (03Merged) 10jenkins-bot: Create smoke tests for the LightboxImage class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100427 (owner: 10Aarcos) [01:01:49] They might be a little more productive when the team has a more structure to how it takes on work, but the general theory is applicable to any team folloing any process IMO [01:02:36] (03CR) 10Aarcos: [C: 031] "Just a suggestion otherwise, LGTM." (031 comment) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99786 (owner: 10MarkTraceur) [01:02:36] Celebrating success is hugely important and often forgotten by mgmt [01:03:24] * marktraceur sings Shock me like an electric eel [01:03:56] But yeah [01:04:12] And the idea of triaging the things that suck periodically is important for making things better in the future [01:04:56] Not having a whine fest, but making a list of process/procedure bugs and pain and tracking how attempts to correct help, hurt or don't change the situation [01:06:07] (03PS5) 10MarkTraceur: Add performance logging [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99786 [01:06:23] I guess I'll merge this because aarcos +1'd. [01:06:40] Yeah [01:07:12] (03CR) 10MarkTraceur: [C: 032] "Per aarcos." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99786 (owner: 10MarkTraceur) [01:07:40] (03Merged) 10jenkins-bot: Add performance logging [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99786 (owner: 10MarkTraceur) [01:13:17] (03CR) 10Aarcos: [C: 031] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [01:15:10] I wonder if that's a hm-we-should-merge-this +1 [01:16:01] I've heard that +1 is the new +2 :) [01:16:30] Nobody +2's after labour day [01:17:21] In core we only +2 if the day, month, hour, minute, and year evenly divide the change id [01:18:41] So *that's* what that jenkins job was [01:18:54] It all makes sense now [01:20:33] (03CR) 10MarkTraceur: "Is this a "Merge-this-now" +1 or just a "hm that's a nice idea" +1?" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [01:39:59] (03CR) 10Aarcos: "Sorry, still don't have +2 button. Looks ready for HEAD if you ask me, ;-)." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [01:40:50] (03CR) 10MarkTraceur: [C: 032] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [01:41:19] (03Merged) 10jenkins-bot: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [02:08:58] (03PS1) 10Aarcos: Smoke test for MultiLightbox class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100523 [02:09:23] Wooooo [02:14:07] (03CR) 10MarkTraceur: [C: 04-1] "One liiiiittle thing." (031 comment) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100523 (owner: 10Aarcos) [02:14:14] Y'all coming up? [02:19:00] (03PS2) 10Aarcos: Smoke test for MultiLightbox class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100523 [02:20:03] (03CR) 10MarkTraceur: [C: 032] "Not exactly what I meant but OK! :)" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100523 (owner: 10Aarcos) [02:20:31] (03Merged) 10jenkins-bot: Smoke test for MultiLightbox class. [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100523 (owner: 10Aarcos) [02:32:19] (03CR) 10Jforrester: "Any chance we could back-port this, at least to wmf6? It's reeeeeeeally annoying. :-)" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/99722 (owner: 10MarkTraceur) [03:34:38] zomg, need agora button at the bottom of https://commons.wikimedia.org/wiki/Special:UploadWizard -that "Next" button is nigh-invisible after the bold illustration. [03:34:48] Hah [03:34:58] quiddity: Can it wait 'til AFTER the presentation [03:44:34] marktraceur: https://www.mediawiki.org/wiki/Extension:PronunciationRecording - Rahul Maliakkal [04:00:10] https://commons.wikimedia.org/wiki/Category:Lake_Tahoe - 288 files, and 15 subcategories. [04:00:35] File:Oh man, I rocked that half-pipe.jpeg [05:22:26] quiddity: https://commons.wikimedia.org/wiki/MediaWiki:Gadget-Favorites.js [14:35:35] (03PS1) 10Gergő Tisza: WIP rewrite template parsing with DOMDocument [extensions/CommonsMetadata] - 10https://gerrit.wikimedia.org/r/100569 [15:27:03] (03Abandoned) 10Gergő Tisza: Trim HTML-based metadata values [extensions/CommonsMetadata] - 10https://gerrit.wikimedia.org/r/97743 (owner: 10Gergő Tisza) [16:16:43] thanks bd808. do we need anyone else to approve to merge? [16:19:36] dan-nl: Let's give tgr a chance to look at it, but if Siebrand didn't spot anything wrong it's probably ok [17:12:29] (03PS2) 10Gergő Tisza: WIP rewrite template parsing with DOMDocument [extensions/CommonsMetadata] - 10https://gerrit.wikimedia.org/r/100569 [17:21:16] bd808: you can see the activity for a 3000 item metadata file http://ganglia.wmflabs.org/latest/?c=deployment-prep&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [17:21:28] the category is at http://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Category:Images_from_Royal_Botanic_Gardens_Edinburgh [17:22:14] thanks tgr [17:22:34] Even more clear at http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=deployment-prep&h=deployment-jobrunner08&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [17:25:46] Load should spread out better in prod where there are 16 job runner nodes instead of just 1 [17:28:31] dan-nl: I only see one error from that run [17:28:34] MWException: There is a serious problem with the gwtoolsetUploadMediafileJob job queueThere are > 1000 gwtoolsetUploadMediafileJob’s in the job queue. gwtoolsetUploadMetadataJob has attempted 5 times to add additional gwtoolsetUploadMediafileJob’s to the job queue, but cannot because the gwtoolsetUploadMediafileJob’s are not clearing out. [17:28:49] yep, we ran into a throttle limit [17:28:51] Happened at 2013-12-10T17:04:59.000Z [17:30:07] the combination of media file jobs added vs cleared reached the throttle limit of 1000 so the tool stopped creating media file jobs and issued that errror [17:31:32] bd808: i originally wanted to delay those metadata jobs using time, but when i looked that part of the job code was disabled … will look more closely at it now [17:32:02] bd808: are you okay with merging https://gerrit.wikimedia.org/r/#/c/100548/ now? it'll need to be rebased [17:32:23] Sure. I'll push some buttons [17:33:31] bd808: thanks! [17:35:23] bd808: was that server load okay in your opinion? [17:36:00] dan-nl: IN my completely non-binding opinion, yes. :) [17:36:13] :) [17:36:25] who is supposed to give the official okay? [17:36:34] The job queue is supposed to take care of itself as far as I'm concerned [17:36:57] In prod things will take longer because there are lots and lots of jobs in queue at any given time [17:37:05] i wonder if we could just up the throttle to 10000 instead of 1000 [17:37:51] Aaron or folks in the #wikimedia-operations channel could give you better tuning advice [17:38:00] better than I can [17:38:31] It's humbling to adit but I still don't understand 90% of what happens on the prod cluster [17:38:36] *to admit [17:39:22] As the guy who knew everything about everything in most of my prior jobs it's frustrating at times [17:50:29] bd808 thanks. i'll see what i can find out over there ... [17:55:43] (03PS3) 10Gergő Tisza: Rewrite template parsing with DOMDocument [extensions/CommonsMetadata] - 10https://gerrit.wikimedia.org/r/100569 [18:00:00] Stand-up! [18:00:00] Order: marktraceur -> tgr -> bd808 -> fabriceflorin [18:00:45] * bd808 waves an unspecified number of fingers at wmfmedia-standup  [18:05:59] Hah [18:53:15] (03PS1) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100610 [18:53:16] (03CR) 10jenkins-bot: [V: 04-1] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100610 (owner: 10MarkTraceur) [18:53:48] (03Abandoned) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100610 (owner: 10MarkTraceur) [19:16:04] (03PS1) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] (wmf/1.23wmf6) - 10https://gerrit.wikimedia.org/r/100611 [19:16:34] (03CR) 10jenkins-bot: [V: 04-1] Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] (wmf/1.23wmf6) - 10https://gerrit.wikimedia.org/r/100611 (owner: 10MarkTraceur) [19:19:52] (03PS2) 10MarkTraceur: Move clearInterface things to mw.LightboxInterface [extensions/MultimediaViewer] (wmf/1.23wmf6) - 10https://gerrit.wikimedia.org/r/100611 [19:19:57] Let's try that again [19:20:12] (I tore out a test, but it's OK, we can put it back when we push the latest master) [19:32:20] (03CR) 10Aarcos: "I think this is ready for HEAD." [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [19:37:09] (03CR) 10MarkTraceur: [C: 032] "Chaaaaarge" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [19:37:15] We need to get you +2 [19:39:09] (03CR) 10jenkins-bot: [V: 04-1] Fix fileLink initialisation [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [19:39:22] Arrrghhhhh [19:44:50] bd808: committed https://gerrit.wikimedia.org/r/#/c/100617/1. are you okay +2'ing it or would you rather wait for aaron to +1 it first? [19:52:10] dan-nl: Let's see what it does. [19:52:23] thanks! [19:54:50] (03PS1) 10MarkTraceur: Add caption to the interface in place of description [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100618 [19:56:59] (03PS7) 10MarkTraceur: Fix fileLink initialisation [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 [19:57:05] Don't do it Jenkins [19:57:07] Don't do it [19:57:19] Don't you even dare [20:00:54] Muahaha [20:01:13] aarcos_: 98593 is passing tests now, you can take another look and I can +2 if you still think it's good [20:01:38] marktraceur: k [20:10:24] (03CR) 10Aarcos: [C: 031] "Green !" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [20:13:19] chrismcmalunch: fyi, david and i did not get that 503 error today when we tested gwtoolset on the beta cluster [20:15:23] dan-nl: it's mysterious for sure [20:17:10] (03CR) 10MarkTraceur: [C: 032] "Ultragreen" [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [20:17:21] (03CR) 10Gergő Tisza: Fix fileLink initialisation (031 comment) [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [20:17:41] (03Merged) 10jenkins-bot: Fix fileLink initialisation [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/98593 (owner: 10MarkTraceur) [20:17:59] hey bd808 kicked off the 3000+ file again [20:18:12] * bd808 crosses fingers [20:18:42] looking at the runJobs.log i see some issues with source media files, but they error out as elected and do not interfere with other media files [20:18:54] the idea is to later use that info to create a message for the user [20:18:58] tgr: Good point...hm. I think it's minor enough that it can be a when-we-notice-it change [20:19:43] dan-nl: "error=GWToolset\Jobs\UploadMediafileJob::run: HTTP request timed out." [20:19:54] yes that's the one i've been noticiing [20:19:54] It would be nice if that message said the URL that timed out [20:20:28] it's in the array [20:20:46] or i thought it should be [20:21:15] * bd808 goes to look at raw logs instead of logstash [20:21:24] no, it only indicates the record, 53 [20:22:18] going to look at the original metadata file [20:22:24] Ah. There is a lot of data there that logstash can't correlate [20:22:41] The print_r of your array. Hmm. [20:24:04] boy that lativia server is timing out a lot [20:24:15] * bd808 wishes we had structured logging already [20:26:40] error=GWToolset\Jobs\UploadMediafileJob::run: Error writing temporary file. [20:27:06] bd808 looks like the server is also having a hard time storing the temp file … ? [20:31:20] dan-nl: Not a lot of useful data in that error message. For instance "which temp file" and "what was the actual error" [20:32:02] ja, not sure where that message is coming from [20:32:17] here is one that has the url [20:32:18] error=GWToolset\Jobs\UploadMediafileJob::run: The media file URL could not be evaluated. The URL delivers the content in a way that is not yet handled by this extension. URL given was "http://data.rbge.org.uk/images2/23073/-1". [20:32:40] so that's a bad url example [20:34:15] The url returns an image/jpeg file [20:34:38] bd808: i imagine the temporary file error is coming from UploadFromUrl [20:37:24] strange with http://data.rbge.org.uk/images2/23073/-1 … a moment ago it return no image, not it does ... [20:37:32] maybe their server had a hiccup? [20:38:20] That's possible. You may be inadvertently DOSing them with the downloads [20:40:45] hmm ... [20:41:42] Huh. Loading the image outright appears to give me a very big image. [20:42:03] that Error writing temporary file message is coming from UploadFromUrl->reallyFetchFile() [21:38:15] hey bd808, committed https://gerrit.wikimedia.org/r/#/c/100684/. hopefully it may shed more light on those media file urls [21:43:06] thanks bd808 [21:47:05] hey bd808 it's at least showing the original and evaluated urls now! [21:47:30] still, this is the only message from uploadformurl Error writing temporary file. [21:47:45] Argh, I hate computers sometimes [21:47:52] Cool. I'm still poking around trying to figure out how to make my log viewer reassemble your messages :) [21:48:16] uh ooh, don't know if you want to do that ... [21:50:07] dan-nl: I'm working on a project to use http://logstash.net/ and Kibana to put logs into an ElasticSearch cluster for searching. RIght now I'm not putting multiline messages back together properly. Debugging GWT is a good excuse to figure that out. [21:50:28] :) [21:50:41] I'd give you the URL, but for now its [21:50:46] "secret" [21:51:13] Mostly because we don't have a good method to control access in labs [21:57:59] np [22:20:43] bd808: figured a way to expose the tmp path in the erroe output https://gerrit.wikimedia.org/r/#/c/100692/ [22:22:27] thanks! [22:27:00] (03PS1) 10MarkTraceur: Load images normally [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100693 [22:27:11] * marktraceur not sure how this is going to work [23:00:00] Hm, OK, that sorta works. [23:00:06] I need to make some unrelated fixes. [23:02:54] (03PS1) 10MarkTraceur: Extend the post-image div height for the drag affordance [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100706 [23:11:21] hey bd808, added the additional exception msg info to saveMediafileAsContent in https://gerrit.wikimedia.org/r/#/c/100711/1 [23:11:58] i really don't understand why we're getting this tmp write error. there was no issue with it earlier today [23:15:26] jobrunner08 is out of disk space: Filesystem Size Used Avail Use% Mounted on [23:15:45] /dev/vda1 9.9G 9.4G 0 100% / [23:19:26] I just cleaned up 4.2G of files from /tmp there. That may help for a bit [23:23:08] (03PS1) 10MarkTraceur: Fixes for the scroll-up affordance [extensions/MultimediaViewer] - 10https://gerrit.wikimedia.org/r/100713 [23:23:51] dan-nl: I'm not sure but there may be some bug you are hitting that leaves files in /tmp when something goes wrong. We'll need to keep an eye on that. [23:24:02] bd808: :( is that an issue with mediawiki or with GWT? [23:26:22] fabriceflorin: I have a feeling that loads of the stuff "in testing" in mingle could reasonably be accepted now [23:28:27] fabriceflorin: You're correct. In the absence of a scrum master to update the board, let's all try to move things forward when we know a task has progressed. I will take a quick stab now at moving and/or accepting the ones I am aware of. Perhaps aarcos could also help us with this part of our agile development workflow. [23:29:10] Meanwhile, I have invited you all to a meeting to plan our work on the upload pipeline, and have started this Etherpad to get the conversation started: https://etherpad.wikimedia.org/p/upload-pipeline-plan [23:30:07] marktraceur: Where is the best place to test the latest Media Viewer code right now? alpha? beta? [23:30:39] Hm [23:32:43] fabriceflorin: MediaWiki.org is always "latest stable", en.wikipedia.beta.wmflabs.org is always "latest master", alpha is just "latest stuff that Mark wanted to test" for now [23:32:56] alpha is kinda a silly place to point people at right now [23:33:17] Cool. For the purposes of approving tickets on Mingle, which one should I use? MediaWiki.org? [23:33:36] Sure. [23:33:39] Or Beta [23:33:42] OK, thanks. [23:33:46] Actually, Beta, because it's more recent. [23:43:07] OK, I will test these on Beta. For now, I have to leave early for a meeting out of office. So I will start on this tonight, and finish this review work tomorrow afternoon, since my morning is pretty much shot already :( [23:43:46] Any questions for me before I leave? [23:44:26] Naw [23:44:30] Have a good night [23:44:53] Sounds good. You too :) [23:50:55] chrismcmahon: do you think these errors may have something to do with that 503 error? [23:50:56] 2013-12-10 23:41:50 deployment-jobrunner08 enwiki: ParsoidCacheUpdateJobOnEdit 0.6558450978185469 type=OnEdit t=126420 error=Failed connect to 10.2.2.29:80; Connection timed out [23:51:43] dan-nl: I sort of doubt it, but I don't really know [23:51:43] gwicke in #wikimedia-labs just told me that the ip should be $wgParsoidCacheServers = array( 'http://10.4.0.61' ); on beta [23:52:22] have no idea where to open the bug for that … seems like it would be a setting hasher might be able to investigate [23:53:31] dan-nl: yes, file a bug under Wikimedia Labs/deployment-prep and assign to hashar please