[00:19:31] (03PS1) 10Dduvall: Run mwext-mw-selenium on MobileFrontend test/gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/229042 (https://phabricator.wikimedia.org/T107587) [00:19:33] (03PS1) 10Dduvall: Run mwext-mw-selenium on Gather experimental pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/229043 [00:27:09] (03CR) 10Jdlrobson: [C: 031] "would merge if i could!" [integration/config] - 10https://gerrit.wikimedia.org/r/229042 (https://phabricator.wikimedia.org/T107587) (owner: 10Dduvall) [00:28:21] (03CR) 10Dduvall: [C: 032] Run mwext-mw-selenium on MobileFrontend test/gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/229042 (https://phabricator.wikimedia.org/T107587) (owner: 10Dduvall) [00:29:56] (03Merged) 10jenkins-bot: Run mwext-mw-selenium on MobileFrontend test/gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/229042 (https://phabricator.wikimedia.org/T107587) (owner: 10Dduvall) [00:30:52] !log Reloading Zuul to deploy I18fa56da9a27a4efeb061ceba773c8b50bc4a8f4 [00:30:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:51:46] (03CR) 10Krinkle: "I am not active maintainer of CI configuration. Please stop adding me as reviewer to all your integration/* related patches." [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [02:05:26] Yippee, build fixed! [02:05:26] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #773: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/773/ [04:29:40] Yippee, build fixed! [04:29:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #520: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/520/ [06:08:01] (03CR) 10Polybuildr: [C: 032] Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:12:25] (03CR) 10Polybuildr: "Some comments use a \t instead of spaces by accident. Maybe we should replace the "\t" with a " " rather than just prepending a space?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228993 (owner: 10Legoktm) [06:14:03] (03CR) 10Polybuildr: "Jenkins?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:14:22] Anybody know why a CR+2 isn't making jenkins do anything? [06:14:23] https://gerrit.wikimedia.org/r/#/c/228996/ [06:15:46] (03CR) 10Polybuildr: [V: 032] Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:16:40] Krinkle_: ^^ ? [06:17:23] I'm currently planning to take away CR+2 V+2 and then give CR+2 again... [06:18:14] (03CR) 10Polybuildr: Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:18:31] (03CR) 10Polybuildr: [C: 032] Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:24:17] grrrit-wm: Why won't jenkins merge? [06:26:02] polybuildr: hi [06:26:12] legoktm: hello! [06:26:19] (03CR) 10Legoktm: [C: 032] "/me taps mic" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:26:24] oh lol [06:26:28] because [06:26:42] polybuildr: it has a dependency! https://gerrit.wikimedia.org/r/#/c/228993/1 [06:26:56] (03CR) 10Legoktm: "It has an unmerged dependency." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:27:11] Yippee, build fixed! [06:27:11] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #701: FIXED in 8 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/701/ [06:27:19] (03CR) 10Legoktm: [C: 04-1] "Yes." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228993 (owner: 10Legoktm) [06:27:28] legoktm: Doesn't it say [06:27:28] (03PS3) 10Legoktm: Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 [06:27:36] polybuildr: try now [06:27:37] Depends On: Automatically fix warnings caught by SpaceyParenthesisSniff (MERGED) [06:27:41] I cherry-picked on master [06:27:42] hmm [06:27:43] weird. [06:29:43] (03CR) 10Polybuildr: [C: 032] "But it said the dependency was merged! Anyway, +2 again." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:30:00] (03Merged) 10jenkins-bot: Automatically fix warnings caught by SpaceAfterControlStructureSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228996 (owner: 10Legoktm) [06:30:26] That worked. [06:30:42] legoktm: so, pbpcbf seems to work pretty nicely. :D [06:32:12] Yay [06:32:27] We just need tests for it [06:32:38] Which addshore is working on :D [06:33:29] yeah :D [06:36:53] o/ [07:04:13] Project browsertests-VisualEditor-production-linux-firefox-sauce build #75: STILL FAILING in 2 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-production-linux-firefox-sauce/75/ [08:26:26] (03PS6) 10Paladox: Add jenkings tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 [09:22:26] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [09:27:27] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1505843 (10fgiunchedi) @brion @hashar I've uploaded ffmpeg `7:2.7.2-1~wmf1` to `trusty-wikimedia`... [09:40:18] (03CR) 10Paladox: [C: 031] Add jenkings tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [09:40:42] (03PS7) 10Paladox: Add jenkings tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 [09:45:12] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1505869 (10fgiunchedi) also `ffmpeg2theora` has been rebuilt with ffmpeg and uploaded to `trusty-w... [11:48:45] 10Beta-Cluster, 6Labs: Have a public IP & DNS for Beta's HTTP proxy - https://phabricator.wikimedia.org/T107866#1506039 (10mobrovac) 3NEW [13:02:21] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #738: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/738/ [13:44:29] 10Beta-Cluster, 6Labs: Have a public IP & DNS for Beta's HTTP proxy - https://phabricator.wikimedia.org/T107866#1506208 (10akosiaris) Couple of questions: * Why would applications what to check if they are about to do an internal or external request ? Got a use case in mind ? * Even if there is a legitimate u... [14:00:19] 10Beta-Cluster, 6Labs: Have a public IP & DNS for Beta's HTTP proxy - https://phabricator.wikimedia.org/T107866#1506272 (10mobrovac) 5Open>3Invalid a:3mobrovac >>! In T107866#1506208, @akosiaris wrote: > Couple of questions: > > * Why would applications what to check if they are about to do an internal... [15:23:26] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:07:20] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Wikimedia code repository browser in Phabricator - https://phabricator.wikimedia.org/T752#1506623 (10Nemo_bis) [16:07:23] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1506620 (10Nemo_bis) 5Open>3Resolved a:3Nemo_bis First attempt: https://www.mediawiki.org/wiki/Module:Callsigns + https://www.mediawiki.org/w/index.php?tit... [16:17:28] release gents when https://gerrit.wikimedia.org/r/#/c/226049/ is merged and deployed from tin....is there any special thing to do for codfw things? [16:25:06] 10Beta-Cluster, 6Scrum-of-Scrums, 6Security-Team, 6operations, and 3 others: Setup a dedicated mediawiki host in Beta Cluster that we can use for security scanning - https://phabricator.wikimedia.org/T72181#1506701 (10dduvall) [16:56:58] Is dduvall here? And if so what is his nick? [17:01:11] andrewbogott: marxarelli [17:01:56] huh [17:02:11] ok, marxarelli, please comment on https://gerrit.wikimedia.org/r/#/c/228890/ ? [17:14:38] chasemp: for now, nothing special needs to be done for cdfw on a trebuchet deploy. Everything uses tin as the master [17:14:56] ok thanks just making sure [17:15:19] mira will be the deploy server in codfw, but to my knowledge nobody has thought about how cross-dc repl will work for trebuchet [17:15:32] I have a WIP idea for scap [17:16:00] it has some problems in beta cluster but I haven't ruled out weird config there [17:16:22] * bd808 pretends to be a releng engineer on the weekends [17:16:47] :) [17:25:40] 6Release-Engineering: Create KPIs for #releng-201516-Q2 - https://phabricator.wikimedia.org/T107905#1506866 (10greg) 3NEW a:3greg [17:27:08] 6Release-Engineering: Make matrix of skills/people - https://phabricator.wikimedia.org/T107906#1506883 (10greg) 3NEW a:3Ettorre [17:27:24] 6Release-Engineering: Make matrix of skills/people - https://phabricator.wikimedia.org/T107906#1506901 (10greg) a:5Ettorre>3greg [17:47:47] imma gonna apply 2 logstash cherry-picks to validate that they work outside of my testing projects [17:48:39] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/226991/ [17:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:49:03] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/227175/ [17:49:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:52:38] 6Release-Engineering, 6Team-Practices: Organize "testing: where does it hurt?" workshop for the end of July - https://phabricator.wikimedia.org/T102713#1506991 (10greg) 5Open>3Resolved a:3greg Donezors. Any follow-up is in other tasks already, yes? [17:52:57] 6Release-Engineering, 6Team-Practices: Organize "testing: where does it hurt?" workshop for the end of July - https://phabricator.wikimedia.org/T102713#1507001 (10greg) a:5greg>3dduvall [17:54:49] 6Release-Engineering: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1507013 (10greg) a:5greg>3None [17:55:32] greg-g: I sent you an email requesting a deploy window for ogv.js & other TimedMediaHandler fixes; let me know when makes sense and if I need to do anything else to prep for it :D [17:56:47] and i'll probably have more tmh fixes next week to clean up transcode job bugs, which'll be less exciting but useful [17:56:53] 6Release-Engineering: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1507024 (10greg) a:3demon Alright, basic matrix is at: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Skill_matrix Now, let's go around and fill it out. I'll assign to Chad (first alphabetica... [17:57:06] PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL 55.56% of data above the critical threshold [0.0] [17:57:48] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/227175/21 [17:57:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:01:33] !log Upgraded logstash to 1.5.3 on deployment-logstash2 [18:01:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:02:44] !log Upgraded elasticsearch to 1.7.1 on deployment-logstash2 [18:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:10:00] brion: [18:10:13] thedj: hey [18:10:49] brion: is it all in master and can it just ride the train rollout? or does it need a new extension/something that isn't just, essentially "do a git pull"? [18:11:09] (submodule update, rather ;) ) [18:11:29] greg-g: it's all in master for the extensions, so should be a simple pull & commit yeah [18:11:43] there is one small fix in MwEmbedPlayer (for IE fullscreen), the rest is in TimedMediaHandler [18:12:05] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [18:12:33] brion: cool, so a new branch is being cut and deployed now/soon [18:12:42] Cut already, deployed soon. [18:12:48] that [18:13:21] brion: so, if you wanna make sure that all is good in the new branch (wmf17) then we'll just let mukunda do the hard part [18:13:29] spiff [18:13:48] lol [18:13:51] :) [18:15:32] brion: when did your change go into master? as long as it was more than ~45 minutes ago then it'll be in the wmf/1.26wmf17 branch which I am about to deploy any minute now [18:16:02] twentyafterfour: merged yesterday so may already be in \o/ [18:16:32] indeed, wmf17 just branched from master [18:16:48] brion: so yeah, master is pushed to prod every week, starting with test wikis on Tues, non-wikipedias on Wed, then everywhere on Thurs [18:16:49] (03PS1) 10Paladox: Update Slate skin test [integration/config] - 10https://gerrit.wikimedia.org/r/229176 [18:17:19] brion: so, as long as you did all the right testing in beta/etc, you should be good :) [18:17:22] 6Release-Engineering, 6Team-Practices: Organize "testing: where does it hurt?" workshop for the end of July - https://phabricator.wikimedia.org/T102713#1507097 (10ggellerman) @greg Dan and I are meeting on Weds Aug 5 to work on next steps for the actionable stuff we've already identified in the meeting notes [18:17:29] woot [18:17:40] brion: iow: it works on beta, right? :) [18:18:13] ok yeah i see the updated TMH bits in origin/wmf/1.26wmf17 [18:19:34] (03PS1) 10Paladox: Add jshint to check: to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229178 [18:20:44] brion: can you find a test video on the test wikis (one of: test.wp, test2.wp, mw.org) to make sure we're good? [18:21:03] checking... [18:21:40] (03PS1) 10Paladox: Add composer-test to Slate Skin [integration/config] - 10https://gerrit.wikimedia.org/r/229180 [18:22:12] greg-g: to test after the update? mw.org and test.wp.org are still on wmf16 right? [18:22:29] right now, but not in a few minutes :) [18:22:34] greg-g: yeah try https://www.mediawiki.org/wiki/File:Sneak_Preview_-_Wikipedia_VisualEditor.webm [18:22:52] well, I can't, I own now apple products :) [18:22:53] you, too can have "I'm Trevor, and I'm working on a Visual Editor for Wikipedia" stuck in your head ;) [18:22:57] heh [18:23:01] ok i'll keep it open :D [18:26:25] brion: :-) [18:29:14] 10Deployment-Systems, 6Services: Evaluate Ansible as a deployment tool - https://phabricator.wikimedia.org/T93433#1507175 (10GWicke) [18:30:22] 10Deployment-Systems, 6Services: Evaluate Ansible as a deployment tool - https://phabricator.wikimedia.org/T93433#1136938 (10GWicke) We have now started to look into configuration deployment support as well. See {T107532} for the details. [18:57:06] it's worrrrrrrrking :D [18:59:52] marxarelli: labnodepool1001 is now reimaged and rebooted. It looks to me like nodepool is dieing on each run — I’ll leave that for you to investigate. [19:01:08] brion: hhhhow? mw.org is still wmf16 [19:01:15] greg-g: on test [19:01:18] andrewbogott: sounds good. thanks [19:01:37] yay! [19:01:39] sorry i'm getting ahead of myself ;) [19:02:13] :) [19:03:17] 10Browser-Tests, 3Reading-Web: MobileFrontend Selenium tests do not use page object pattern - https://phabricator.wikimedia.org/T65620#1507420 (10Jdlrobson) anyone? @dduvall? [19:08:30] 6Release-Engineering: Get RelEng team members greater access - https://phabricator.wikimedia.org/T107926#1507451 (10greg) 3NEW a:3greg [19:15:53] 10Browser-Tests, 3Reading-Web: MobileFrontend Selenium tests do not use page object pattern - https://phabricator.wikimedia.org/T65620#1507508 (10dduvall) There are still a handful of steps where I see references to selectors, but many are necessary as they take variable arguments. ``` $ ag 'element\(' featur... [20:04:08] (03Abandoned) 10JanZerebecki: [WIP] add DOC_SUBPATH parameter [integration/config] - 10https://gerrit.wikimedia.org/r/226452 (owner: 10JanZerebecki) [21:03:21] 6Release-Engineering, 10Continuous-Integration-Config, 3Mobile-App-Sprint-63-Android-Europium, 3Wikipedia-Android-App: Create jenkins slave instance dedicated to Android runs - https://phabricator.wikimedia.org/T107336#1507919 (10thcipriani) Moved this machine over to use Ubuntu Trusty while we're trying t... [21:18:26] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:38:46] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [21:49:50] aude: offtopic response to a couple night's ago discussion: are you still using identi.ca/pump.io? [21:54:20] greg-g: no and wasn't aware they still exist? [21:54:40] legoktm: have you experimented at all with jenkins matrix builds? [21:54:51] I have not. [21:55:11] what are those? [21:55:19] i'm researching possible ways to run the mw-selenium job in parallel, and that seemed like one option [21:55:22] oh, darn [21:55:29] aude: yeah, pump.io is still going, essentially maintenance mode, sadly. [21:55:34] :( [21:55:56] marxarelli: not sure if you saw my correction: I misremembered it being lego, it was actually tyler [21:56:17] marxarelli: why does it need to run in parallel? [21:56:24] is twentyafterfour still around today? [21:56:25] legoktm: https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Combinations+Plugin [21:56:26] legoktm: I'm now misattributing CI fixes to you. Good work. :) [21:56:36] legoktm: it's very slow :) [21:56:37] aude: I am [21:56:37] :P [21:56:50] marxarelli: slower than mw-core-phpunit-zend? :P [21:57:02] twentyafterfour: no idea how, but wikidata is still not on the right branch :( [21:57:08] legoktm: up to 13 minutes for 16 scenarios [21:57:16] it's as if git submodule update ... was forgotten [21:57:23] where the git clones are done afresh [21:57:31] 7 minutes otherwise [21:57:37] which isn't too bad [21:57:38] aude: that might be... [21:57:51] that's not terrible [21:58:40] marxarelli: it's already running on trusty right? [21:58:46] not terrible but not great [21:58:50] yeah [21:59:27] marxarelli: FWIW commit 44aa6f12b34cde2267b107d2359a5288610d5636 in integration/config is where I removed the one matrix job we had [22:00:11] thcipriani: ooooh, neat [22:00:38] here was the bug/reason why I moved it off of the configuration matrix: https://phabricator.wikimedia.org/T96199 [22:00:59] aude: I'm updating it [22:01:38] twentyafterfour: ok [22:01:49] thcipriani: doesn't sound promising :/ [22:02:32] marxarelli: yeah, matrix builds definitely seemed to be the root-cause of a lot of jenkins lockups [22:03:01] we haven't had too many since we've moved away from using them, but that may be due to other factors/changes as well. [22:03:51] maybe i should just slow my roll and be glad that the mw-selenium job is stable [22:06:05] marxarelli: which job is this? [22:06:21] thcipriani: https://integration.wikimedia.org/ci/job/mwext-mw-selenium/ [22:11:55] marxarelli: hmm, you could just run cucumber in parallel breaking up tasks, but I guess (a) I'm not sure how state dependant all the cucumber tests are and (b) merging junit results wouldnt' [22:12:08] be awesome, but I'm sure someone has figured it out [22:12:26] thcipriani: yeah, i tested out the parallel_tests gem locally that basically does the multiplexing for you [22:13:09] but you think it's pretty hacky you said? [22:13:16] but there are some major downsides, mostly organizing output in a sane way, and race conditions with video recording of the same Xvfb display [22:13:35] oh right, same display..... [22:13:49] * greg-g is just following along, ignore me [22:15:25] marxarelli: around? [22:15:29] the parallel_tests runner sets an environment variable for the group number that's being executed [22:15:33] (And free?) [22:15:55] it could potentially be used to switch on the display port [22:15:57] jdlrobson: yeah [22:16:00] what's up? [22:16:09] marxarelli: just keen on your thoughts around https://phabricator.wikimedia.org/T107770 [22:16:37] there's been lots of back and forth but right now i'm just keen to setup _something_ (cc @tgr|away @bd808 ) [22:16:59] jdlrobson: so there's an open patch i believe [22:17:01] (note blocked task https://phabricator.wikimedia.org/T104561) [22:17:09] there are 2 existing "something" ways [22:17:23] you mean https://gerrit.wikimedia.org/r/#/c/226230/ ? [22:17:25] jdlrobson: https://gerrit.wikimedia.org/r/#/c/226230/ [22:17:29] yup [22:17:48] yeh so right now i just want to make a decision on which something way to use :) [22:18:03] it looks sane for the most part, expect for the ~ 4M article content :) [22:18:05] it can be wrong - and we can revisit/rewrite it - but it would be good to get a starting point :) [22:18:25] *except* for [22:18:41] a starting point is good, i just don't want that starting point to bloat the repo [22:18:44] remember, once things go in git... hard to get it out :) [22:18:45] there's no going back from that [22:18:54] sure - bloating things could be circumvented by using external urls? [22:18:54] werd. what greg-g said [22:19:10] or by initiating the role - so it doesn't bloat the default instance? [22:19:11] jdlrobson: that was one thought, yeah [22:19:19] greg-g: yeh i agree [22:19:31] so is this the only issue - the file size problem? [22:19:37] or a separate repo for test content maybe? [22:19:42] it would be pretty easy to add a puppet define that knows how to fetch a dump via the api I think [22:19:51] jdlrobson: my only gripe, yeah [22:19:53] and put it in the VM [22:20:01] via the api for en.wikipedia.org ? [22:20:04] then make the load depend on that [22:20:07] yeah [22:20:11] bd808: ah, that's a good idea [22:20:13] or whatever wiki [22:20:23] so an instance would essentially be a list of api urls to mediawiki instances? [22:20:44] sorry /role/ would essentially be a list of api urls to mediawiki instances? [22:21:17] jdlrobson: list of API URL + titles probably [22:21:54] marxarelli: makes sense. This would however mean having to work out all the templates in an article too.. :-S [22:22:09] but i guess although that's a pain that could work [22:22:34] It would be just like https://gerrit.wikimedia.org/r/#/c/226230 except instead of having the dump in the repo there would bea bit in the role that says "got get [[:en:Barack_Obama]]" [22:22:53] can't we automate a full dump fetch including tempaltes? [22:23:06] bd808: not sure if api supports that? [22:23:19] screen scraping would :) [22:23:46] * marxarelli shudders [22:23:48] bd808: errrrrr [22:24:02] you can fetch a full dump with templates, but [22:24:16] we don't want a full dump [22:24:18] 1) you can't specify the revision (I think) [22:24:31] it would slow down the average instance too much [22:25:01] 2) all the template transclusion, lua evaluation and what not would have to be done on the vagrant instance, which in my experience is a very painful thing [22:25:24] the question is, do you need to be able to edit the imported page? [22:25:56] if not, just fetch the expanded contet via action=raw&templates=expand [22:26:07] that supports revisions and does all the work on the servers [22:26:15] and is almost not screen scaping :) [22:26:46] it won't fetch the (non-Commons) images though [22:27:10] but you can just set up enwiki as a remote image repo if that's a problem [22:27:11] that seems pretty minor [22:27:22] tgr: that's a good idea [22:27:35] getting the fully expanded pages seems smart to me [22:27:37] what do you mean by editing the imported page? [22:27:48] do you just mean no template editing - just article content? [22:27:52] these tests aren't stress tests for the parser [22:27:53] running browser tests on the edit interface [22:27:54] if so i think that makes sense [22:28:26] since the expanded wikitext will be completely different from the original wikitext of the article [22:28:39] tgr: but i'd still be able to edit it in my local wiki? [22:28:47] you might care about that for, say, VisualEditor template editing browser tests [22:29:16] you would be able to edit it but it would be an unreadable wikitext/HTML soup [22:29:29] tgr: this seems like a good starting point. We could always add support for importing articles without expanded templates [22:29:30] as infoboxes would be mostly expanded to HTML, for example [22:29:53] tgr: if anything that would make reading web's lives easier [22:30:06] bonus, the fully expanded Barack_Obama is *only* 400K [22:30:22] we can work with the raw html, work out the problem with the template and then spend time on the wiki working out which template has the problem [22:30:30] https://www.irccloud.com/pastebin/MhHjoG7a/ [22:30:36] so this is what i'm expecting ^ [22:30:39] it's not raw HTML exactly [22:30:42] Something along those lines [22:31:08] jut download and look -- https://en.wikipedia.org/wiki/Barack_Obama?action=raw&templates=expand -- it's wikitext with the tempaltes expanded [22:31:09] whoa. that output is way too readable for puppet's taste :P [22:31:22] marxarelli: ;-) [22:32:07] 6Release-Engineering, 10VisualEditor: browsertests-VisualEditor-language-screenshot jenkins job failing - https://phabricator.wikimedia.org/T1321#1508258 (10Jdforrester-WMF) [22:32:24] on general principal I think that should be doable [22:32:31] ugh, apparently templates inside are not expanded... [22:32:46] * jdlrobson tests [22:33:27] lots of {{cite}} [22:33:57] 10Browser-Tests, 10VisualEditor: browsertests-VisualEditor-language-screenshot jenkins job failing - https://phabricator.wikimedia.org/T1321#1508275 (10greg) [22:34:13] 6Release-Engineering, 10Continuous-Integration-Config, 3Mobile-App-Sprint-63-Android-Europium, 3Wikipedia-Android-App: Create jenkins slave instance dedicated to Android runs - https://phabricator.wikimedia.org/T107336#1508276 (10thcipriani) After fiddling with some android flags, I'm now back to this poin... [22:37:06] seems like https://phabricator.wikimedia.org/T26774 [22:41:16] which is apparently an epic bug from the dawn of western civilization -- https://phabricator.wikimedia.org/T4700 [22:46:15] I like how the first comment is questioning how hard it will be to fix [22:48:38] bd808: i just imported barack obama and i think that the output would be suitable for reading web. I assume we can address https://phabricator.wikimedia.org/T26774 separately? [22:48:55] given they are API requests we can always change the API method used.. [22:49:13] I'm guessing T26774 will live until the heat death of the universe [22:49:35] possibly, but it would get more attention if it was surfaced in a vagrant role ;-) [22:50:04] heh. I don't think a lot of our parser hackers are running mw-v yet [22:50:32] doesn't look that difficult actually, more like an issue of no one caring [22:51:05] so any objections to making this the way forward? [22:51:08] and whether the change will break someone's workflow [22:51:17] jdlrobson: not from me, no [22:51:49] jdlrobson: if you are find with red links in the references, I don't see any problems [22:51:53] Are there any example code implementing this kind of role? [22:52:01] tgr: since anomie isn't here to do it, obligatory xkcd -- https://xkcd.com/1172/ [22:52:18] jdlrobson: well no since you are making the first one ever ;) [22:52:45] jdlrobson: I can work on it if you'd like [22:52:51] for a role importing pages, you can look at multimedia [22:53:09] Ideally I'd like the list of articles to be extendable by anyone e.g. A csv file which looks like so: [22:53:11] for the actual import command, import_text and import_dump are both somewhat similar [22:53:14] https://www.irccloud.com/pastebin/vyXwx31z/ [22:53:17] jdlrobson: no [22:53:28] puppet is not going to read random csv files [22:53:31] depending on whether you want to import page-by-page or via Special:Export [22:53:34] it can likely reuse import_text [22:53:52] 6Release-Engineering, 10Continuous-Integration-Config, 3Mobile-App-Sprint-63-Android-Europium, 3Wikipedia-Android-App: Create jenkins slave instance dedicated to Android runs - https://phabricator.wikimedia.org/T107336#1508346 (10thcipriani) This may explain why the android emulator plugin is failing with... [22:54:14] but Special:Export would need to be amended a bit so cloning the import_text class is the easier way to go [22:54:30] https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/mediawiki/manifests/import_text.pp [22:55:18] bd808: yeh i've got zero experience with puppet - forgive my ignorance. [22:55:31] i'm just trying to get a crystal clear image of what this role would look like [22:56:04] i.e. mediawiki::export { 'Barack_Obama': wiki_api => '...', file => 'foo' } -> mediawiki::import_text { 'Barack_Obama': sources => 'foo' } [22:56:35] something like that [22:56:59] jdlrobson: something a lot like -- https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/role/manifests/gwtoolset.pp#L36-L120 [22:57:30] bd808 marxarelli thanks [22:57:36] in other words, you could implement a resource that retrieves the content via the API and saves it to file. chain that with import_text and wrap it up in another resource like `mediawiki::import_from_api` [22:58:01] yeah [22:58:26] I think at the same time I would refactor to mw::import::* naming [22:58:33] marxarelli: would it be possible to pair with you on this at some point - i've never written a puppet role and i figure it would be a useful activity [22:58:34] or as a prequel [22:58:43] jdlrobson: sure [22:58:47] awesome [22:59:21] okay thanks a bunch this has been helpful [22:59:30] seems like we have a path forward :) [23:00:07] jdlrobson: w00t! [23:00:07] okay updated implementation notes in https://phabricator.wikimedia.org/T104561 to capture this discussion [23:03:07] okay next problem - https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/13179/console < anyone know why this is failing for all MobileFrontend patches? [23:07:56] is anyone other than @Krinkle familiar with this stuff greg-g ? [23:08:04] what? [23:08:13] the jenkins qunit jobs [23:08:25] i'm not sure why it's failing as it doesn't fail locally [23:09:33] that's probably mainly timo, yeah [23:10:06] doh [23:10:10] jdlrobson: The 'qunit-mobile' job is not my work, it's yours afaik. It was a custom job that uses the querystring useformat=mobile [23:10:18] once the target bypass stuff works, that job is obsolete and should be removed. [23:10:23] i guess hashar set it up then... [23:10:39] Krinkle: it's failing right now blocking some code getting merged and i'm not sure why [23:10:39] We run tests on /plain which is target and skin agnostic. Useformat has no effect there [23:10:49] jdlrobson: I'd say remove it from the zuul trigger then. [23:10:51] I'll deploy [23:11:04] Krinkle: the job isn't quite obselete yet [23:11:10] it doesn't run the mobile tests [23:11:21] (the new one) [23:11:52] ? [23:11:57] Define 'the mobile tests' [23:12:01] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #220: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/220/ [23:12:36] Krinkle: it doesn't seem to be running anything in the MobileFrontend repository because of the patch I merged for ori yesterday [23:13:18] This is the last working run [23:13:19] https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/13175/console [23:13:25] It just runs qunit, right. nothing else? [23:13:28] In what way is it not obsolete? [23:14:10] now i'm confused... [23:16:20] So confusion point 1) I don't understand why mwext-MobileFrontend-qunit-mobile is now failing for everything and what caused that 2) In reverting Ori's patch https://gerrit.wikimedia.org/r/#/c/229268/ the MobileFrontend tests run again but with errors so i am working to fix that, so that we can use mediawiki-extensions-qunit / mwext-MobileFrontend-qunit [23:16:20] instead of mwext-MobileFrontend-qunit-mobile [23:16:32] (not sure how mwext-MobileFrontend-qunit and mediawiki-extensions-qunit differ at this point) [23:23:04] bd808, marxarelli: any tip on how to debug "Cannot reassign variable name" errors in puppet? [23:23:46] tgr: try `PUPPET_DEBUG=1 vagrant provision` to get more info [23:23:57] hopefully it will tell you which manifest/variable [23:24:04] I did but it did not [23:24:10] *sigh* [23:24:50] does it not tall you the name at all? [23:24:53] *tell [23:25:06] only the name of the node [23:25:20] which is not super helpful for vagrant [23:25:30] i fought with this recently, and i can't remember if i found a good to debug it [23:25:34] heh. no it is not [23:26:27] wait. it says "Cannot reassign variable name"? [23:26:36] if so then $name is the variable [23:26:37] Error: Cannot reassign variable name on node mediawiki-vagrant.dev [23:26:49] oh, ha [23:26:56] name is reserved [23:26:58] so you are at least looking for something named $name [23:27:01] `name` is reserved [23:27:09] that's what it was for me [23:27:10] and yeah that's reservered (dup of $title) [23:27:21] duh [23:29:27] thanks, that would have taken a loong time to figure out [23:29:59] yw. I noticed it because I googled for the general error and saw how they format the message [23:30:08] jdlrobson: The main difference between qunit and qunit-mobile is that qunit is the standard grunt-karma with Chrome and QUnitjs run that uses special:javascipttest/qunit/plain, the other uses phantomjs witha legacy grunt plugin to open Special:JavaScripttest/qunit?useformat=mobile (not plain). Which has skin HTML and other things it shouldn't depend on [23:30:12] and uses phantomjs instead of a real browser [23:30:56] Krinkle: so any ideas why it has broken suddenly? [23:31:09] No idea, and can't look into it right now [23:31:17] What kind of error are you seeing? [23:31:42] https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/13191/console [23:32:12] just a time out.. but we've had no issues merging so either someone force merged something that broke it or there's been a change in another repository that broke it or somethings wrong with the job [23:32:15] i'm not sure how to debug it [23:33:44] jdlrobson: Have you tried locally? [23:33:57] yeh of course [23:33:59] works fine [23:34:06] with latest mediawiki and MF installed, view index.php?title=Special:JavaScriptTest/qunit&useformat=mobile [23:34:09] exactly that [23:34:20] yup already tried... no problemos [23:34:25] No other extensions enabled [23:34:44] will try disabling some extensions [23:36:06] jdlrobson: It's probably broken because the page loads QUnit asynchronously now [23:36:26] there's nothing we can do about that because it's a regular mediawiki special page view (with mobilefrontend even), not like /plain, so we have no control. [23:36:40] And the old phantomjs client expects QUnit to have finished at dom-ready [23:36:42] which isnt the case anymore [23:36:47] I can't verify but I expect that's the problem [23:36:52] Krinkle: if only i could replicate it locally... [23:37:16] jdlrobson: I can replicate that locally. You can see in the waterfall that dom ready happens before qunit is loaded [23:37:29] you can replicate the broken tests? [23:37:50] No, and it never will because you don't stop looking at dom browser. We're humans that will keep the tab open. [23:38:06] A CI job will quit the browser when dom is ready and there is no qunit. It wouldn't know when it will finish. [23:38:14] QUnit has a callback but in this case QUnit is undefined [23:38:37] and mw is undefined, too. [23:38:50] jdlrobson: What tests are running in qunit-mobile that don't run in qunit? [23:39:01] I'd recommend just fixing or moving that so that we can disable the job [23:39:19] synchronous loading is no longer supported. The qunit-mobile/phantomjs job was meant to be obsolete 6 months ago when we moved to karma. [23:39:50] well any help moving to karma and deprecating this job would be much appreciated - it seems we still have some failing VE tests though.. [23:40:03] Where? [23:40:10] mediawiki-extensions-qunit is passing [23:40:10] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/6273/console [23:40:29] Only when MobileFrontend tests are enabled [23:40:37] VE and its tests are working [23:41:42] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/6273/consoleFull [23:43:05] Yeh... so basically I have no coverage for MobileFrontend tests right now with that broken job [23:44:12] and i'm not sure how to enable the MobileFrontend tests without this breakage - i'm not sure what's causing these VE tests to fail (i suspect there is a badly written MobileFrontend test in there but i can't isolate it) [23:44:31] jdlrobson: Does mw-ext-qunit pass on MF master? [23:44:37] Where is the commit that enables them? [23:44:46] this jenkins url is triggered for an unrelated commit