[02:52:15] (03CR) 10Krinkle: [C: 032] "Deployed mwext-MobileFrontend-qunit-mobile, mwext-Mantle-qunit-mobile, mwext-Thanks-qunit-mobile." [integration/config] - 10https://gerrit.wikimedia.org/r/198655 (https://phabricator.wikimedia.org/T93511) (owner: 10Krinkle) [02:55:44] (03PS3) 10Krinkle: Migrate mobile-qunit jobs to new CI slaves in labs [integration/config] - 10https://gerrit.wikimedia.org/r/198655 (https://phabricator.wikimedia.org/T93511) [02:55:50] (03CR) 10Krinkle: [C: 032] Migrate mobile-qunit jobs to new CI slaves in labs [integration/config] - 10https://gerrit.wikimedia.org/r/198655 (https://phabricator.wikimedia.org/T93511) (owner: 10Krinkle) [03:05:07] (03Merged) 10jenkins-bot: Migrate mobile-qunit jobs to new CI slaves in labs [integration/config] - 10https://gerrit.wikimedia.org/r/198655 (https://phabricator.wikimedia.org/T93511) (owner: 10Krinkle) [03:05:24] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #521: FAILURE in 23 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/521/ [03:39:53] (03PS1) 10Legoktm: Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 [03:40:06] (03CR) 10Legoktm: [C: 032] Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 (owner: 10Legoktm) [03:41:11] 10Continuous-Integration: Replace project-specific "{name}-thing" jobs with generic "thing" ones - https://phabricator.wikimedia.org/T91997#1139867 (10Legoktm) [03:41:16] (03CR) 10jenkins-bot: [V: 04-1] Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 (owner: 10Legoktm) [03:41:28] eh? [03:41:52] oh [03:41:53] epic fail [03:42:14] (03PS2) 10Legoktm: Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 [03:44:46] Yippee, build fixed! [03:44:46] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #387: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/387/ [03:47:15] (03CR) 10Legoktm: [C: 032] Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 (owner: 10Legoktm) [03:51:58] (03Merged) 10jenkins-bot: Document why {name}-{ext-name}-npm still exists [integration/config] - 10https://gerrit.wikimedia.org/r/198680 (owner: 10Legoktm) [04:00:58] (03PS1) 10Legoktm: zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 [04:39:33] (03PS2) 10Legoktm: zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 [04:43:25] (03PS3) 10Legoktm: zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 [04:46:09] (03PS4) 10Legoktm: zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 [04:47:57] (03CR) 10Legoktm: [C: 032] zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 (owner: 10Legoktm) [04:49:23] (03Merged) 10jenkins-bot: zuul: Create extension-jslint template, remove jslint from extension-unittests [integration/config] - 10https://gerrit.wikimedia.org/r/198681 (owner: 10Legoktm) [04:50:14] !log deploying https://gerrit.wikimedia.org/r/198681 (no-op) [05:59:48] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1139927 (10Amire80) Could this failure in ContentTranslation also be related? - https://gerrit.wikimedia.org/r/#/c/198187/ [06:18:17] 10Continuous-Integration: Add publishing macro for integration.wikimedia.org/cover - https://phabricator.wikimedia.org/T93558#1139942 (10Krinkle) 3NEW [06:18:35] 10Continuous-Integration: Migrate mediawiki-core-code-coverage job to labs - https://phabricator.wikimedia.org/T93559#1139948 (10Krinkle) 3NEW [06:18:43] 10Continuous-Integration: Add publishing macro for integration.wikimedia.org/cover - https://phabricator.wikimedia.org/T93558#1139942 (10Krinkle) [06:18:44] 10Continuous-Integration: Migrate mediawiki-core-code-coverage job to labs - https://phabricator.wikimedia.org/T93559#1139948 (10Krinkle) [06:19:15] 10Continuous-Integration: Add publishing macro for integration.wikimedia.org/cover - https://phabricator.wikimedia.org/T93558#1139942 (10Krinkle) [06:19:16] 10Continuous-Integration, 10OOjs: Publish QUnit coverage on integration.wikimedia.org - https://phabricator.wikimedia.org/T87490#1139958 (10Krinkle) [06:21:49] Krinkle: running phpunit coverage is soo slow I don't think it's feasible to do it post-commit [06:22:05] legoktm: We're not. It's on a daily cron [06:22:18] but just having it run faster is nice in general. [06:22:30] oh, I think I read the bug title wrong [06:22:56] legoktm: I just kicked off a build with a hacked configuration to run it on a trusty slave (and publishing to a directory in the workspace) [06:22:59] I did [06:23:03] It's running like 10x faster. [06:23:07] :DDD [06:23:16] because hhvm? [06:23:25] Yeah, that and other things [06:23:31] but I imagine it's mostly hhvm [06:23:48] it runs a lot slower than normal phpunit run because of all the coverage inspection phpunit does [06:23:58] I guess that particular type of inspection is more efficient in hhvm [06:24:19] it's extremely CPU intensive which hhvm is better at [06:24:41] Hm.. int.wm.o/ci requests are timing out [08:23:18] ContentTranslation tests are failing. Pointers where the issue is would be helpful: https://integration.wikimedia.org/ci/job/mwext-ContentTranslation-qunit/1694/console [08:25:53] Nikerabbit: fwiw I already reported the issue [08:26:06] https://phabricator.wikimedia.org/T93510 [08:39:09] but nobody knows the cause yet [08:46:52] Nikerabbit: We may need hashar to fix that. [08:47:48] kart_: impossible to say until the cause is known [08:50:23] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: QUnit tests for ContentTranslation fail - unable to merge commits - https://phabricator.wikimedia.org/T93510#1140027 (10Nikerabbit) p:5High>3Unbreak! [08:53:05] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: QUnit tests for ContentTranslation fail - unable to merge commits - https://phabricator.wikimedia.org/T93510#1140031 (10Nikerabbit) There are errors such as `ERROR: '[earthquake] Url exceeds maximum length'` in the log but I am not sure if th... [08:57:39] hashar: morning! [08:58:04] You've T93510 in breakfast ;) [09:02:18] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: QUnit tests for ContentTranslation fail - unable to merge commits - https://phabricator.wikimedia.org/T93510#1140054 (10hashar) ext.uls Common languages FAILED Would it mean it is an issue in ULS? The console log shows an `TypeError: undefin... [09:02:24] kart_: not much I can do, seems to be in the code [09:02:35] gave some clue on it. Will not be able to investigate for sure :( [09:03:50] hashar: thanks [09:04:14] kart_: try to reproduce it locally [09:04:20] seems some function got renamed somehow [09:04:28] or maybe an extension needs to be added as a dependency [09:11:05] zeljkof: I was playing around with "https://gist.github.com/zeljkofilipin/0ee89385c2819d71d686"; I was not able to find documentation for visit method [09:11:19] vikasyaligar: sorry, in a meeting [09:11:35] zeljkof: ah ! ok; never mind found it here => http://www.rubydoc.info/github/cheezy/page-object/master/PageObject/PageFactory [09:11:38] z and I are in a conf call for an hour :D [09:12:11] OK ! enjoy :) [09:16:31] vikasyaligar: we are actually pairing using that gist :D [09:18:53] hashar: awesome; I am also digging on it here :) [09:48:56] hey, how do I request that a gerrit repo is renamed [09:49:42] (what project do I attach to the phabricator task?) [09:50:08] werdna: Wikimedia-Git-Gerrit [09:50:47] thanks hashar [09:51:04] werdna: that would notify the appropriate people. I am not sure we ever rename repositories though :( [09:51:13] that is somehow painful, having to do a bunch of sql queries [09:51:44] well, could always just create a new one and import the old data [09:52:02] yeah eventually [09:52:05] that is not ideal though [09:57:43] werdna: that’s what we usually do, I think [09:57:45] werdna: chad would know [09:57:52] only problem would be pending patchsets [10:02:12] werdna: and the Gerrit change history. But maybe Chad already has all the appropriate sql commands to properly rename the repo [10:02:22] anyway, need a task to be filled :) [10:02:41] hashar: I just took a quick look at http://www.dzone.com/snippets/watir-logging [10:02:45] it is from 2008 :) [10:02:50] it is obsolete 99% [10:02:55] zeljkof: yo [10:02:58] busy? [10:03:05] aharoni: no, coming to the meeting [10:03:09] ok [10:03:16] hashar: I filed a task, so we’ll see :) [10:04:03] zeljkof: ah damn Google [10:04:14] bringing food to my kid then I will be back in roughly half an hour [10:20:01] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1140218 (10polybuildr) The two open microtasks for this project have pending patches. It... [10:20:03] zeljkof: I got this(https://gist.github.com/zeljkofilipin/0ee89385c2819d71d686#file-page-object-selenium-local-rb-L8) working just by changing browser to @browser; reference: https://github.com/cheezy/page-object/blob/master/lib/page-object/page_factory.rb#L74 [10:20:26] vikasyaligar: thanks, in another meeting now :) [10:20:42] zeljkof: Oh OK :) [10:21:00] vikasyaligar: oh, I can not believe the mistake was such a basic one :) [10:31:12] (03PS1) 10Adrian Lang: Switch wikidata qunit jobs from qunit to qunit-karma [integration/config] - 10https://gerrit.wikimedia.org/r/198699 [10:35:51] vikasyaligar: hm, I get the same error message when I change browser to @browser [10:36:03] vikasyaligar: can you share your code and terminal output? [10:41:56] zeljkof: ah ! one more thing; argument of visit method should be the class name [10:42:04] so it will be visit SaucePage [10:42:36] vikasyaligar: argh [10:42:43] vikasyaligar: I am stupid :) [10:42:51] vikasyaligar: thanks, will try [10:42:56] zeljkof: :) [11:27:32] 7Blocked-on-RelEng, 6Release-Engineering, 6Multimedia, 6Scrum-of-Scrums, and 2 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1140290 (10Gilles) I've turned what I had done manually into a shell script that will automatically package the latest version of these djang... [11:52:50] hashar: https://integration.wikimedia.org/ci/job/mwext-ContentTranslation-qunit/1700/console - possible to get debug run? [11:53:02] ie tests are passing locally. [11:54:08] https://dev.translatewiki.net/wiki/Special:JavaScriptTest/qunit?hidepassed&filter=ext.cx [11:56:07] 10Continuous-Integration, 10VisualEditor, 3VisualEditor 2014/15 Q3 blockers, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#1140333 (10adrianheine) The reported number of tests ra... [11:58:47] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: QUnit tests for ContentTranslation fail - unable to merge commits - https://phabricator.wikimedia.org/T93510#1140338 (10Nikerabbit) The `TypeError: undefined is not a function` refers to `checkCssHandles` in `mediawiki.js`. No very recent cha... [12:03:00] 10Continuous-Integration, 10VisualEditor, 3VisualEditor 2014/15 Q3 blockers, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#1140354 (10adrianheine) More issues I had when running... [12:08:16] zeljkof: it's alive: https://commons.wikimedia.org/wiki/File:VisualEditor_category_item-hr.png [12:09:01] aharoni: woohoo [12:12:50] zeljkof: 23 minutes per language. [12:13:04] aharoni: not bad [12:13:11] 10Continuous-Integration, 10VisualEditor, 3VisualEditor 2014/15 Q3 blockers, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#1140385 (10adrianheine) Sorry for the noise, but after... [12:13:13] I thought it was more [12:13:57] zeljkof: and we now have 36 languages. [12:14:05] 828 minutes [12:14:11] aharoni: wow [12:14:18] over 13 hours [12:15:30] if we'll have 60 languages, it will be 23 hours. [12:15:59] Ah, well. I can live with it. I say, when we get to 60 languages, I'll be happy about having 60 languages :) [12:16:39] green green https://integration.wikimedia.org/ci/view/BrowserTests/view/VisualEditor/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/22/console [12:21:31] EVERYTHING IS GREEN [12:21:33] https://integration.wikimedia.org/ci/view/BrowserTests/view/VisualEditor/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/22/console [12:23:27] awesome ! aharoni zeljkof :) [12:23:43] vikasyaligar: we added two more screenshots today [12:23:50] I'm adding them to the user guide now [12:24:05] aharoni: does it have padding ?? [12:24:07] aharoni: vikasyaligar: great news [12:24:13] vikasyaligar: not yet [12:24:21] some time soon I'll add padding [12:24:41] awesome ! [12:37:34] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:52:59] 7Blocked-on-RelEng, 6Release-Engineering, 6Multimedia, 6Scrum-of-Scrums, and 2 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1140424 (10Gilles) I've updated P421 and it now works with all missing packages except MySQL-python. I haven't tried the "sentry" package its... [12:56:18] zeljkof: hey :) Seems you have created a test job and forgot to delete it : https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-186947/ [13:03:04] hashar: oops, sorry, deleting now [13:05:09] !log Jenkins: deleting unused jobs mwext-.*-phpcs-HEAD and mwext-.*-lint [13:07:21] pff [13:07:27] I am no more using !log [13:07:30] the bot keeps dieing [13:10:55] qa-morebots: [13:10:55] I am a logbot running on tools-exec-07. [13:10:55] Messages are logged to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL. [13:10:55] To log a message, type !log . [13:10:58] !log Jenkins: deleting unused jobs mwext-.*-phpcs-HEAD and mwext-.*-lint [13:11:03] Logged the message, Master [13:11:25] !log and I restarted qa-morebots a minute or so ago (see https://wikitech.wikimedia.org/wiki/Morebots#Example:_restart_the_ops_channel_morebot ) [13:11:28] Logged the message, Master [13:27:02] hashar: looks like all I had to do is change "visit sauce_page" to "visit SaucePage" :) thanks to vikas [13:27:07] hashar: https://gist.github.com/zeljkofilipin/0ee89385c2819d71d686/revisions [14:12:40] (03PS1) 10Amire80: Add krc and om to language screenshot tests [integration/config] - 10https://gerrit.wikimedia.org/r/198717 [14:22:00] (03PS2) 10Amire80: Add krc, om and tr to language screenshot tests [integration/config] - 10https://gerrit.wikimedia.org/r/198717 [14:28:08] 10Continuous-Integration, 6operations, 3Continuous-Integration-Isolation, 7Upstream: Create a Debian package for NodePool - https://phabricator.wikimedia.org/T89142#1140534 (10hashar) [14:28:23] 10Continuous-Integration, 6operations, 3Continuous-Integration-Isolation, 7Upstream: Create a Debian package for NodePool - https://phabricator.wikimedia.org/T89142#1028174 (10hashar) I have filled a Debian intent to package: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027 [14:32:16] Yippee, build fixed! [14:32:17] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » it,contintLabsSlave && UbuntuTrusty build #23: FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=it,label=contintLabsSlave%20&&%20UbuntuTrusty/23/ [14:47:10] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #195: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/195/ [14:50:12] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ilo,contintLabsSlave && UbuntuTrusty build #23: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ilo,label=contintLabsSlave%20&&%20UbuntuTrusty/23/ [14:52:53] hashar: any change to look in https://phabricator.wikimedia.org/T93510 [14:53:09] kart_: no clue :( [14:53:21] I havent even looked at it sicne my comment yesterday [14:53:34] kart_: timo might know. He made a bunch of changes last week [14:54:58] hashar: thanks. Added him as subscriber. will poke him. [14:55:06] Krinkle|detached: ^^ https://phabricator.wikimedia.org/T93510 [15:01:21] anyone around who can setup jenkins to verify https://gerrit.wikimedia.org/r/#/c/198723/ ? its in a new project - its maven. [15:14:23] (03CR) 10Zfilipin: [C: 032] Add krc, om and tr to language screenshot tests [integration/config] - 10https://gerrit.wikimedia.org/r/198717 (owner: 10Amire80) [15:16:56] off [15:17:01] will be back in the evening [15:19:19] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests, 5Patch-For-Review: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1140607 (10Tobi_WMDE_SW) 5Resolved>3Open Reopening because the limit is too low for the Wikidat... [15:21:39] (03Merged) 10jenkins-bot: Add krc, om and tr to language screenshot tests [integration/config] - 10https://gerrit.wikimedia.org/r/198717 (owner: 10Amire80) [15:28:07] (03CR) 10AndyRussG: "Hmmm... any idea about what's happening here? Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/193556 (https://phabricator.wikimedia.org/T86092) (owner: 10AndyRussG) [15:37:14] 10Deployment-Systems: Think about using screen for deployments - https://phabricator.wikimedia.org/T93588#1140669 (10demon) 3NEW [15:51:16] 10Deployment-Systems: Think about using screen for deployments - https://phabricator.wikimedia.org/T93588#1140726 (10greg) p:5Triage>3Low [15:52:47] 6Release-Engineering, 10Ops-Access-Requests, 6Phabricator, 6operations, 5Patch-For-Review: Chad H. needs access to iridium (Phabricator host) to manage repos - https://phabricator.wikimedia.org/T92564#1140824 (10chasemp) [15:55:12] 10Continuous-Integration, 10MediaWiki-extensions-MultimediaViewer, 6Multimedia: Chromium 41.0.2272 (Ubuntu) mmv.ui.ProgressBar jumpTo()/hide() FAILED / animateTo() FAILED - https://phabricator.wikimedia.org/T93540#1141041 (10Raymond) [15:55:13] 10Continuous-Integration, 10MediaWiki-extensions-Survey: Fix test failures: ResourcesTest::testFileExistence + ResourcesTest::testMissingDependencies - https://phabricator.wikimedia.org/T93542#1141040 (10Raymond) [15:57:29] 10Continuous-Integration, 10MediaWiki-extensions-ContentTranslation: QUnit tests for ContentTranslation fail - unable to merge commits - https://phabricator.wikimedia.org/T93510#1141308 (10Raymond) [16:01:24] 10Continuous-Integration, 10Thanks: PhantomJS timed out, possibly due to a missing QUnit start() call. - https://phabricator.wikimedia.org/T93594#1141331 (10Raymond) 3NEW [16:01:44] 10Continuous-Integration, 10Thanks: PhantomJS timed out, possibly due to a missing QUnit start() call. - https://phabricator.wikimedia.org/T93594#1141340 (10Raymond) [16:20:25] 6Release-Engineering, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Gerritbot shouldn't post "Change merged by jenkins-bot:" messages any more - https://phabricator.wikimedia.org/T91766#1141406 (10chasemp) p:5Triage>3Low Can someone link to a comment of the type referenced here? [16:23:28] 6Release-Engineering, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Gerritbot shouldn't post "Change merged by jenkins-bot:" messages any more - https://phabricator.wikimedia.org/T91766#1141427 (10Jdforrester-WMF) >>! In T91766#1141406, @chasemp wrote: > Can someone link to a comment of the type referenced he... [16:28:46] 6Release-Engineering, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Gerritbot shouldn't post "Change merged by jenkins-bot:" messages any more - https://phabricator.wikimedia.org/T91766#1141459 (10Jdforrester-WMF) >>! In T91766#1095851, @Krenair wrote: > I have a feeling I've asked this before, but - do we ha... [16:35:22] anyone around who can setup jenkins to verify https://gerrit.wikimedia.org/r/#/c/198723/ ? its in a new project - its maven. [16:44:31] 6Release-Engineering, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Gerritbot shouldn't post "Change merged by jenkins-bot:" messages any more - https://phabricator.wikimedia.org/T91766#1141511 (10chasemp) a:3QChris >>! In T91766#1141427, @Jdforrester-WMF wrote: >>>! In T91766#1141406, @chasemp wrote: >> Ca... [16:50:33] 10Staging: Create staging-terbium - https://phabricator.wikimedia.org/T91543#1141530 (10demon) I'm not sure this will be terrible useful actually. [16:52:20] ^d: YuviPanda twentyafterfour can we talk about how this is going to work? https://phabricator.wikimedia.org/T91553 Somewhat familiar with swift, but can we have a swift be _on_ an openstack libvirt instance? [16:52:47] <^d> So I haven't tried on labs before [16:52:54] <^d> I've had it running in a local vm before though [16:52:55] thcipriani: aaaaaaah. I think it can be, yeah. godog is your man. [16:53:02] you have to use swift_new module [16:53:44] <^d> So, now that we have tin we need to create a wiki [16:53:50] <^d> We should use addwiki like prod :) [16:53:54] kk, just wanted to be sure it was possible. Seems like, that's one of the few remaining big ones (in my mind at least) [16:54:08] <^d> config's all busted probably [16:54:10] thcipriani: ’tis the biggest one [16:54:11] <^d> should be fun [16:54:22] ^d: so I'm rebuilding mw01 right now, I think apachectl _should_ start [16:54:24] 10Staging: Create staging-terbium - https://phabricator.wikimedia.org/T91543#1141542 (10yuvipanda) Doesn't it also run crons? do we want crons? because that's how we get crons... [16:54:41] thcipriani: btw, mediawiki roles take two puppet runs usually... [16:54:46] I went through the list of errors I was seeing earlier, I think I got them all. [16:54:54] YuviPanda: I noticed :) [16:55:03] heh. Joe had given up on trying to fix that, iirc [16:55:04] <^d> YuviPanda: Yeah but I'm not sure if all the crons are useful in labs [16:55:10] <^d> Also, they're all in the site.pp, no role [16:55:20] we can move them but yeah, you’d know better, etc [16:55:40] <^d> node 'terbium.eqiad.wmnet' { [16:55:40] <^d> include role::mediawiki::common [16:55:40] <^d> include role::db::maintenance [16:55:42] <^d> include role::peopleweb [16:55:44] <^d> include misc::monitoring::jobqueue [16:55:46] <^d> include scap::scripts [16:55:48] <^d> include role::noc [16:55:50] <^d> include role::mediawiki::searchmonitor [16:55:52] <^d> include admin [16:55:54] <^d> include ldap::role::client::labs [16:55:56] <^d> include misc::maintenance::pagetriage [16:55:58] <^d> include misc::maintenance::translationnotifications [16:56:00] <^d> include misc::maintenance::updatetranslationstats [16:56:02] <^d> include misc::maintenance::wikidata [16:56:03] * YuviPanda slaps ^d with a large paste trout [16:56:04] <^d> include misc::maintenance::echo_mail_batch [16:56:06] <^d> include misc::maintenance::parsercachepurging [16:56:08] <^d> include misc::maintenance::cleanup_upload_stash [16:56:10] <^d> include misc::maintenance::tor_exit_node [16:56:12] <^d> include misc::maintenance::update_flaggedrev_stats [16:56:14] <^d> include misc::maintenance::refreshlinks [16:56:16] <^d> include misc::maintenance::update_special_pages [16:56:18] <^d> include misc::maintenance::update_article_count [16:56:20] <^d> include misc::maintenance::purge_abusefilter [16:56:22] <^d> include misc::maintenance::purge_checkuser [16:56:24] <^d> # Revert of https://gerrit.wikimedia.org/r/74592 per request from James Alexander. [16:56:26] <^d> class { '::misc::maintenance::purge_securepoll': [16:56:28] <^d> ensure => absent, [16:56:30] <^d> } [16:56:32] <^d> # (bug 15434) Periodical run of currently disabled special pages [16:56:34] <^d> # to be run against PMTPA slaves [16:56:36] <^d> include misc::maintenance::updatequerypages [16:56:38] <^d> package { 'python-mysqldb': [16:56:40] <^d> ensure => installed, [16:56:42] <^d> } [16:56:44] <^d> include role::backup::host [16:56:46] <^d> backup::set {'home': } [16:56:48] <^d> } [16:56:52] <^d> IRC was closer than Phab for pasting [16:56:57] ^d: gotta turn off rate limiting in your client :) [16:57:34] <^d> hmm? [16:58:42] ^d: I was just getting on line every 2 seconds from your paste, now that I think of it, could be on the freenode side, too. [16:59:00] <^d> It went quickly for me :p [17:00:26] relativity [17:00:45] mw01 would be going more quickly if IO wasn't so slow right now... [17:01:32] 10Continuous-Integration: make Jenkins voting for the wikidata/query/rdf project - https://phabricator.wikimedia.org/T93601#1141576 (10greg) 3NEW [17:17:08] 10Continuous-Integration: Jenkins: Set up PHPUnit testing on MySQL backend - https://phabricator.wikimedia.org/T37912#1141640 (10Krinkle) [17:18:27] 10Continuous-Integration, 10Thanks: PhantomJS timed out, possibly due to a missing QUnit start() call. - https://phabricator.wikimedia.org/T93594#1141656 (10Krinkle) Actual error: https://integration.wikimedia.org/ci/job/mwext-Thanks-qunit-mobile/31/ -> https://integration.wikimedia.org/ci/job/mwext-Thanks-qu... [17:21:14] Yippee, build fixed! [17:21:15] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sl,contintLabsSlave && UbuntuTrusty build #23: FIXED in 3 hr 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sl,label=contintLabsSlave%20&&%20UbuntuTrusty/23/ [17:23:16] hashar: Why do we need to allow large code dumps, or core dumps at all? [17:23:18] # allow the dumping of corefiles, up to 2GB [17:23:18] ulimit -c 2097152 [17:23:29] hashar: Are they going to a specific directory? Are they purged automatically> [17:34:28] Krinkle: git blame should say ? [17:34:42] hashar: That points to you and faidon [17:34:42] Krinkle: it is probably there to debug Zend PHP or HHVM crashes [17:34:46] Some kind of debugging activity [17:34:58] I am sure we had issues with Zend regex at some point [17:35:18] I'm currently working on global-set-env, global-setup and global-teardown (mainly to fix TMPDIR) [17:35:21] (for non-mw jobs) [17:35:32] Just curious if I should move to corelimit or if it's not important anymore [17:37:34] 10Continuous-Integration, 7Regression: /tmp/npm-* directories left behind on Jenkins slaves - https://phabricator.wikimedia.org/T90836#1068897 (10Krinkle) [17:37:41] (03PS1) 10Krinkle: Implement global-set-env, global-setup, global-teardown [integration/jenkins] - 10https://gerrit.wikimedia.org/r/198757 (https://phabricator.wikimedia.org/T90836) [17:40:10] (03PS2) 10Krinkle: Implement global-set-env, global-setup, global-teardown [integration/jenkins] - 10https://gerrit.wikimedia.org/r/198757 (https://phabricator.wikimedia.org/T90836) [17:40:54] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #45: ABORTED in 26 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/45/ [17:41:38] (03PS3) 10Krinkle: Implement global-set-env, global-setup, global-teardown [integration/jenkins] - 10https://gerrit.wikimedia.org/r/198757 (https://phabricator.wikimedia.org/T90836) [17:43:13] (03PS4) 10Krinkle: Implement global-set-env, global-setup, global-teardown [integration/jenkins] - 10https://gerrit.wikimedia.org/r/198757 (https://phabricator.wikimedia.org/T90836) [17:44:04] 10Continuous-Integration, 10Thanks: MediaWiki Thanks extension tests failing (TemplateParser syntax error) - https://phabricator.wikimedia.org/T93594#1141888 (10Krinkle) [17:47:36] 10Continuous-Integration, 10Thanks, 7Mobile: MediaWiki Thanks extension tests failing (TemplateParser syntax error) - https://phabricator.wikimedia.org/T93594#1141920 (10EBernhardson) [17:47:47] 10Continuous-Integration: make Jenkins voting for the wikidata/query/rdf project - https://phabricator.wikimedia.org/T93601#1141922 (10Manybubbles) Cool, thanks for making the task. [17:49:11] 10Continuous-Integration, 10Thanks, 7Mobile: MediaWiki Thanks extension tests failing (TemplateParser syntax error) - https://phabricator.wikimedia.org/T93594#1141936 (10Krinkle) This was fixed and duplicate of {T93436}. However, it's now failing with something else: ``` 17:48:00 Testing http://localhost:94... [17:49:47] 10Continuous-Integration, 10Thanks, 7Mobile: MediaWiki Thanks extension test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1141948 (10Krinkle) [17:51:55] hashar: Could you do a sanity check on https://gerrit.wikimedia.org/r/#/c/198757/ in the next few hours? Would like to deploy later tonight. [17:51:59] Bash stuff etc. [17:52:38] 10Continuous-Integration, 10VisualEditor: Concurrent builds using local Chromium/Firefox browsers on Linux host fail - https://phabricator.wikimedia.org/T90673#1141972 (10Krinkle) p:5High>3Low [17:53:10] 10Continuous-Integration: Consider running tmpreaper on Jenkins jobs' tmpfs - https://phabricator.wikimedia.org/T89327#1141978 (10Krinkle) p:5High>3Low [17:53:11] (03PS4) 10Awight: WIP: Set to voting once the job passes [integration/config] - 10https://gerrit.wikimedia.org/r/195343 (https://phabricator.wikimedia.org/T91895) [17:53:15] (03CR) 10jenkins-bot: [V: 04-1] WIP: Set to voting once the job passes [integration/config] - 10https://gerrit.wikimedia.org/r/195343 (https://phabricator.wikimedia.org/T91895) (owner: 10Awight) [17:53:41] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #46: STILL FAILING in 7 min 13 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/46/ [17:53:52] 10Continuous-Integration: Consider running tmpreaper on Jenkins jobs' tmpfs - https://phabricator.wikimedia.org/T89327#1033472 (10Krinkle) My `TMPDIR` sub-scoping should mitigate the 90% case of our `/tmp` overflowing issues. Putting this back on the shelf for now. [17:54:08] 10Continuous-Integration: Consider running tmpreaper on Jenkins jobs' tmpfs - https://phabricator.wikimedia.org/T89327#1141987 (10Krinkle) 5Open>3declined p:5Low>3High [17:54:09] 10Continuous-Integration: Jenkins: Figure out long term solution for /tmp management - https://phabricator.wikimedia.org/T74011#1141989 (10Krinkle) [17:54:33] (03PS5) 10Awight: wikimedia-fundraising-civicrm tests are voting [integration/config] - 10https://gerrit.wikimedia.org/r/195343 (https://phabricator.wikimedia.org/T91895) [17:54:35] (03CR) 10jenkins-bot: [V: 04-1] wikimedia-fundraising-civicrm tests are voting [integration/config] - 10https://gerrit.wikimedia.org/r/195343 (https://phabricator.wikimedia.org/T91895) (owner: 10Awight) [17:54:47] (03PS16) 10Awight: Jenkins job builder definition for CRM job [integration/config] - 10https://gerrit.wikimedia.org/r/195063 (https://phabricator.wikimedia.org/T91895) [17:54:54] (03PS6) 10Awight: wikimedia-fundraising-civicrm tests are voting [integration/config] - 10https://gerrit.wikimedia.org/r/195343 (https://phabricator.wikimedia.org/T91895) [17:56:43] will be back around 8pm utc [18:02:31] Yippee, build fixed! [18:02:31] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #47: FIXED in 7 min 11 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/47/ [18:02:55] <^d> How did role::cache::text ever work in beta? [18:03:35] <^d> stuff like backend_match hardcoded :p [18:05:25] Yippee, build fixed! [18:05:25] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #522: FIXED in 24 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/522/ [18:09:49] <^d> lol, tin has no extensions yet. [18:10:01] <^d> staging-tin that is [18:10:59] ^d: whenever I find bugs where I'm like, "How did this ever work?" it breaks immediately :| [18:11:39] <^d> A copy of your installation's LocalSettings.php must exist and be readable in the source directory. Use --conf to specify it. [18:11:41] <^d> lol. [18:11:44] <^d> no wikis exist yet [18:11:51] also, we should add some extensions to staging-tin, mw01 is serving...ish...weird hhvm errors, but this is to do with the code on tin, so.. [18:12:13] ^d: well, that'd probably be a problem :P [18:12:38] ^d: yeah, that’s where I left tin... [18:12:45] on account of not knowing jack about multiversion or anything... [18:12:52] * YuviPanda has only like, 2 patches to mwcore ever... [18:13:07] <^d> luckily multiversion has nothing to do with core :D [18:13:11] <^d> It's all in wmf-config [18:13:18] right [18:13:24] I have even less patches in wmf-config :P [18:15:17] <^d> I think I can tweak checkoutMediaWiki to also do extensions for a master setup [18:15:21] * ^d fiddles [18:16:18] <^d> Gah, it doesn't know how to figure out the extensions unless they're in the upstream branch [18:16:20] <^d> Lameeee [18:16:21] (03PS1) 10Legoktm: Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 [18:16:59] https://github.com/wikimedia/mediawiki/commits/master?author=yuvipanda <-- for the record!!! [18:17:39] > Fix Typo in CREDITS [18:17:57] (03CR) 10Krinkle: Create generic jshint and jsonlint jobs and use them (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/198762 (owner: 10Legoktm) [18:18:34] <^d> I have 13 pages of that, just from git [18:19:24] Krinkle: that's not confusing at all >.> [18:20:08] 10Continuous-Integration, 6Mobile-Web, 10Thanks, 7Mobile: MediaWiki Thanks extension test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1142143 (10Krinkle) [18:20:24] 10Continuous-Integration, 6Mobile-Web, 10Thanks, 7Mobile: MediaWiki Thanks extension test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1141331 (10Krinkle) It's now causing MobileFrontend qunit-mobile job to fail as well. Same error. [18:20:59] (03CR) 10jenkins-bot: [V: 04-1] Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 (owner: 10Legoktm) [18:21:26] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1142151 (10Krinkle) {T93594} is causing MobileFrontend qunit-mobile job to fail still. [18:22:14] (03PS2) 10Legoktm: Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 [18:22:33] (03CR) 10Legoktm: Create generic jshint and jsonlint jobs and use them (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/198762 (owner: 10Legoktm) [18:24:01] 10Continuous-Integration, 6Mobile-Web, 10Thanks, 7Mobile: MediaWiki Thanks extension test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1142176 (10Jdlrobson) [18:24:02] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1142177 (10Jdlrobson) [18:24:30] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1138991 (10Jdlrobson) [18:25:44] (03PS1) 10Krinkle: Move mwext-*-testextension-zend to UbuntuPrecise slaves in labs [integration/config] - 10https://gerrit.wikimedia.org/r/198770 (https://phabricator.wikimedia.org/T93143) [18:27:45] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1142202 (10Raymond) [18:27:48] 10Continuous-Integration, 6Mobile-Web, 10Thanks, 7Mobile: MediaWiki Thanks extension test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1142204 (10Mattflaschen) Neither of these have anything to do with the Thanks extension, except that MobileFronten... [18:28:13] 10Continuous-Integration, 5Patch-For-Review: Migrate .*testextension-zend jobs to labs slaves - https://phabricator.wikimedia.org/T93143#1142207 (10Krinkle) a:3Krinkle [18:28:21] 10Continuous-Integration, 6Mobile-Web, 7Mobile: MediaWiki MobileFrontend test failure: Failing to assert that session ID is 16 chars long - https://phabricator.wikimedia.org/T93594#1142211 (10Mattflaschen) [18:28:24] 10Staging, 5Patch-For-Review: Create staging-mw-app* (MW App servers) - https://phabricator.wikimedia.org/T91548#1142214 (10thcipriani) Got some warnings setting up first app server. I think most of this will go away with T1256 : **Warnings**: # `wikispecies.conf` – live handles this in redirect.conf and r... [18:30:22] (03PS1) 10Krinkle: Migrate prepare-mediawiki to MySQL (affects testextension + mwext-qunit) [integration/config] - 10https://gerrit.wikimedia.org/r/198773 (https://phabricator.wikimedia.org/T37912) [18:32:27] (03PS3) 10Legoktm: Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 [18:33:16] (03PS4) 10Legoktm: Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 [18:35:06] (03CR) 10Legoktm: [C: 032] Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 (owner: 10Legoktm) [18:39:42] (03Merged) 10jenkins-bot: Create generic jshint and jsonlint jobs and use them [integration/config] - 10https://gerrit.wikimedia.org/r/198762 (owner: 10Legoktm) [18:40:43] Yippee, build fixed! [18:40:44] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » gl,contintLabsSlave && UbuntuTrusty build #23: FIXED in 4 hr 32 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=gl,label=contintLabsSlave%20&&%20UbuntuTrusty/23/ [18:41:52] !log deploying https://gerrit.wikimedia.org/r/198762 [18:41:58] Logged the message, Master [18:56:43] <^d> thcipriani: Whee. https://gerrit.wikimedia.org/r/#/c/198783/ [18:58:03] ^d: nice! [18:58:45] <^d> I got it running on staging-tin now [18:58:51] <^d> Should give us a php-master + extensions [18:59:17] <^d> Ok, checkoutMediaWiki is fine. updateBranchPointers still explodes [19:00:41] that's the on the just loops through extensions...? [19:02:11] no, I'm thinking of deploys MakeWMFBranch...many php files to keep straight. [19:13:12] (03PS1) 10Legoktm: Only run jshint/jsonlint jobs when relevant files are touched [integration/config] - 10https://gerrit.wikimedia.org/r/198792 [19:19:29] !log disabled puppet on staging-palladium to test a puppet patch [19:19:31] Logged the message, Master [19:19:48] thcipriani: ^d do you have any new instances to create? I’ve a patch for the autosigner I need to test... [19:20:35] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #445: FAILURE in 9 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/445/ [19:21:35] (03PS2) 10Legoktm: Only run jshint/jsonlint jobs when relevant files are touched [integration/config] - 10https://gerrit.wikimedia.org/r/198792 [19:22:30] YuviPanda: not just yet. Probably just spin up a throw away one, I guess :\ [19:22:52] thcipriani: alright. maybe I should spin up rdb01 and so can get rid of current rdb1? [19:23:12] that'd be good. I think there are a few list that still. [19:25:43] thcipriani: hah! the current rdb instances are trusty anyway - in prod they’re precise... [19:27:04] YuviPanda: then please destroy away! [19:27:14] thcipriani: yeah doing [19:28:22] !log created staging-rdb01.eqiad.wmflabs [19:28:25] Logged the message, Master [19:38:27] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1142515 (10Florian) [19:51:16] (03CR) 10Jforrester: [C: 04-1] "Good except for the comments. :-)" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/198792 (owner: 10Legoktm) [19:57:44] <^d> Slightly better failure now, no longer complaining about a lack of LocalSettings [19:57:44] <^d> :) [20:00:52] 10Beta-Cluster: m.wikidata.beta.wmflabs.org should point to a mobile IP - https://phabricator.wikimedia.org/T85469#1142625 (10yuvipanda) a:5yuvipanda>3None [20:25:22] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #558: FAILURE in 44 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/558/ [20:56:25] twentyafterfour: are you going to deploy the scap change you merged to beta and prod? [20:58:58] !log 20:41 cscott deployment-prep updated OCG to version 11f096b6e45ef183826721f5c6b0f933a387b1bb [20:59:00] Logged the message, Master [20:59:23] !log Last log copied from #wikimedia-labs [20:59:25] Logged the message, Master [21:26:38] bd808: I don't know how to deploy scap :-o [21:28:15] I guess I can probably find the dox [21:34:48] twentyafterfour: I think that should just be: cd /srv/deployment/scap && git deploy start && git deploy sync on tin [21:34:59] (03PS3) 10Legoktm: Only run jshint/jsonlint jobs when relevant files are touched [integration/config] - 10https://gerrit.wikimedia.org/r/198792 [21:35:13] once you've got the right revision checked out in /srv/deployment/scap/scap [21:35:27] thcipriani: yup. Git fetch && git rebase origin/master in between start & sync [21:35:45] (03CR) 10Legoktm: Only run jshint/jsonlint jobs when relevant files are touched (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/198792 (owner: 10Legoktm) [21:36:25] cd /srv/deployment/scap/scap && git deploy start && git checkout master && git fetch && git rebase origin/master && git deploy sync [21:36:38] needs to happen both on deployment-bastion and tin [21:36:55] (03CR) 10Jforrester: [C: 031] Only run jshint/jsonlint jobs when relevant files are touched [integration/config] - 10https://gerrit.wikimedia.org/r/198792 (owner: 10Legoktm) [21:57:07] bd808: why bastion and tin? [21:57:17] beta and prod [21:57:32] beta isn't auto-updated [21:58:46] 10Continuous-Integration, 6operations: fix failures of jenkins job operations-puppet-puppetlint-strict - https://phabricator.wikimedia.org/T93642#1143178 (10Krenair) [22:00:59] <^d> Also staging-tin :p [22:01:05] <^d> because we now have 3 environments! [22:04:15] :) [22:07:50] Yippee, build fixed! [22:07:50] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fr,contintLabsSlave && UbuntuTrusty build #23: FIXED in 7 hr 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fr,label=contintLabsSlave%20&&%20UbuntuTrusty/23/ [22:16:47] 10Continuous-Integration, 6operations: fix failures of jenkins job operations-puppet-puppetlint-strict - https://phabricator.wikimedia.org/T93642#1143262 (10hashar) [22:27:44] 10Continuous-Integration, 10MediaWiki-extensions-MultimediaViewer, 6Multimedia: Chromium 41.0.2272 (Ubuntu) mmv.ui.ProgressBar jumpTo()/hide() FAILED / animateTo() FAILED - https://phabricator.wikimedia.org/T93540#1143353 (10Tgr) From https://gerrit.wikimedia.org/r/#/c/198608/ [22:36:19] 10Continuous-Integration, 10MediaWiki-extensions-ImageMetrics, 6Multimedia: Karma failing on an ImageMetrics test referencing a value set in a head script - https://phabricator.wikimedia.org/T93459#1143377 (10Tgr) 5Open>3Invalid a:3Tgr Thanks! I didn't know `Special:JavaScriptTest/qunit/plain` was a th... [22:36:58] 10Continuous-Integration, 10MediaWiki-extensions-ImageMetrics, 6Multimedia: Karma failing on an ImageMetrics test referencing a value set in a head script - https://phabricator.wikimedia.org/T93459#1143383 (10Tgr) 5Invalid>3Open (I'll rather keep it open, the test still needs fixing.) [22:38:58] 10Continuous-Integration, 10MediaWiki-extensions-MultimediaViewer, 6Multimedia: Chromium 41.0.2272 (Ubuntu) mmv.ui.ProgressBar jumpTo()/hide() FAILED / animateTo() FAILED - https://phabricator.wikimedia.org/T93540#1143388 (10Tgr) There have been no changes to MediaViewer recently; I assume this is the same i... [22:39:31] eh what [22:39:32] 22:26:09 git.exc.GitCommandError: 'git remote update origin' returned exit status 1: fatal: Unable to create temporary file '/srv/ssd/jenkins-slave/workspace/mwext-MapSources-testextension-zend/src/.git/objects/pack/tmp_pack_XXXXXX': No space left on device [22:39:40] on lanthanum [22:40:58] /srv/ssd is pretty much full [22:41:10] <^d> ugh [22:41:12] <^d> no good [22:41:18] <^d> let's see [22:42:00] https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1427150513.262&target=servers.lanthanum.diskspace._srv_ssd.byte_free.value [22:42:19] went from 17gig free to 0 in the past 30 minutes? [22:42:31] <^d> dafuq? [22:42:32] er, hour or so [22:42:48] it's been proccessing a bunch of l10n-bot commits [22:42:56] many of them for extensions that haven't been tested in a looong time [22:43:04] hrrrm [22:43:05] so it probably had to re-clone core for them [22:43:10] but that shouldn't be 17gb [22:43:29] *shouldn't* [22:44:14] https://integration.wikimedia.org/ci/computer/lanthanum/ do I just press "mark temporarily offline" to take it out of rotation? anything running on it is going to fail [22:45:03] <^d> I think? [22:45:29] sure [22:45:36] !log took lanthanum offline in jenkins [22:45:38] Logged the message, Master [22:45:43] I just hope it doesn' thave many jobs pinned ot it that can't go on another slave [22:45:50] they'll go to gallium [22:46:09] which is going to cause most extension merges to slow down due to https://phabricator.wikimedia.org/T93143 [22:47:20] ugh [22:48:50] !log deleting mwext-*-lint workspaces on lanthanum, shouldn't be needed any more [22:48:52] Logged the message, Master [22:48:53] <^d> https://phabricator.wikimedia.org/P424 - breakdown on wasted space [22:49:09] that freed up ~440MB [22:49:52] what are the @# directories for? [22:50:06] <^d> fuck if i know [22:52:40] wonderful [22:52:41] 183M .git [22:53:03] anyways [22:53:11] qunit-mobile runs on labs now so we can just delete all of it [22:53:14] 10Continuous-Integration, 10MediaWiki-General-or-Unknown, 6Mobile-Web, 5Patch-For-Review: MobileFrontend commits fail in qunit in Jenkins - https://phabricator.wikimedia.org/T93511#1143421 (10Jdlrobson) Should be working now? [22:53:18] legoktm: see -dev [22:53:44] ugh [22:54:00] that's one way to empty the queue [22:54:50] !log deleting mwext-*-qunit-mobile workspaces on lanthanum, shouldn't be needed any more [22:54:53] Logged the message, Master [22:55:17] ok [22:55:20] 2.2G free now [22:55:50] but... 17gig to 2.2gig is also crazy... [22:56:14] how big is core? [22:56:58] my checkout is 441meg [22:57:31] it's not unreasonable for lanthanum to have cloned it 34 times today [22:57:33] er [22:57:35] it is unreasonable [22:57:37] but not impossible [22:58:06] !log deleting mwext-*-qunit* workspaces on lanthanum, shouldn't be needed any more [22:58:08] Logged the message, Master [22:58:10] right [22:58:42] ok [22:58:45] 13G back [22:58:52] we're at 92% [22:59:21] 1) why wasn't there an alert? [22:59:28] 2) I'll think of more points later [23:00:19] !log lanthanum is now online again, with 13G free disk space [23:00:22] Logged the message, Master [23:00:47] legoktm: \o/ [23:01:31] RECOVERY - Disk space on lanthanum is OK: DISK OK [23:01:32] it sounds like there was one? [23:01:52] legoktm: you can do a similar cleanup on gallium as well [23:01:54] much earlier there was this: PROBLEM - Disk space on lanthanum is CRITICAL: DISK CRITICAL - free space: /srv/ssd 5454 MB (3% inode=86%): [23:02:09] Krenair: oh, fine, just prove me wrong :P [23:02:49] * greg-g should get his bouncer set back up [23:02:57] and, we should respond to those alerts [23:03:59] <^d> alerts are for squares [23:04:06] * ^d prefers being a trapezoid [23:04:20] now I'm putzing aroud in icinga.wmflabs so see if there's anything useful to watch [23:04:46] http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=deployment-prep&style=detail&nostatusheader [23:05:02] betalabs is down ? [23:05:18] beta cluster* [23:05:19] hasharDinner: I think zuul is having issues [23:05:24] hasharDinner: it's complaining that nothing can be merged [23:05:32] ohh [23:05:32] thedj: no, is it for you? [23:05:46] legoktm: that would be the zuul-merger being broken [23:05:55] legoktm: it runs on gallium.wikimedia.org as zuul [23:05:57] http://en.wikipedia.beta.wmflabs.org redirects to http://www.en.wikipedia.beta.wmflabs.org [23:06:14] hasharDinner: of course, gallium is out of space too [23:06:14] giving: PHP fatal error: [23:06:15] Invalid host name (www.en.wikipedia.beta.wmflabs.org). [23:06:23] and https is unreachable for me [23:06:28] legoktm: log files are /var/log/zuul/merger.log and /var/log/zuul/merger-debug.log [23:06:31] right, no https for beta cluster [23:06:41] error: unable to create temporary sha1 filename .git/objects/37: No space left on device [23:06:42] error: unable to create temporary sha1 filename .git/objects/37: No space left on device [23:06:42] yeah i suspected as much. [23:06:43] and I'm not being redirected [23:06:48] hasharDinner: yeah, going to delete stuff now [23:06:51] /dev/sdb1 149G 149G 24K 100% /srv/ssd [23:06:52] :( [23:07:35] 6Release-Engineering, 6Engineering-Community: Lyon -> Annecy Transportation Info to RelEng Team - https://phabricator.wikimedia.org/T93686#1143467 (10Rfarrand) 3NEW a:3MNeisler [23:07:37] !log deleting mwext-*-lint workspaces on gallium, shouldn't be needed [23:07:39] Logged the message, Master [23:07:49] I am not sure why it filled up though [23:08:16] hasharDinner: my hunch is that l10n-bot triggered jobs for extensions that didn't ahve workspaces yet, so it cloned a bunch of mw-core's [23:08:30] :=( [23:08:37] that is surely a killer [23:08:52] 6Release-Engineering, 6Engineering-Community: Lyon -> Annecy Transportation Info to RelEng Team - https://phabricator.wikimedia.org/T93686#1143486 (10MNeisler) 5Open>3Resolved [23:08:52] btw, beta cluster icinga info sorted by severity: http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?nostatusheader&hostgroup=deployment-prep&style=detail&sorttype=2&sortoption=3 [23:09:10] lots of gigantic .git folders as well [23:09:18] 6Release-Engineering, 6Engineering-Community: Lyon -> Annecy Transportation Info to RelEng Team - https://phabricator.wikimedia.org/T93686#1143487 (10Rfarrand) 5Resolved>3Open [23:09:31] <^d> deployment-lucid-salt.eqiad.wmflabs [23:09:35] <^d> lol [23:10:03] legoktm: yeah we should first git clone from a local copy of mediawiki/core on the same disk. Git would creates hardlink which save up a ton of space and io [23:10:11] then run zuul cloner to refresh the clone [23:10:26] ^d: keep it around please [23:10:37] ^d: that is used to test out the salt upgrades for production [23:11:06] !log deleting mwext-*-qunit* workspaces on gallium, shouldn't be needed [23:11:08] Logged the message, Master [23:12:00] /dev/sdb1 149G 146G 3.6G 98% /srv/ssd [23:12:27] hasharDinner: what does the @2 after the workspace name mean? [23:14:06] legoktm: that is when more than one build of a job occur on the same node [23:14:21] jenkins append @N to prevent clashes between concurrent builds [23:14:24] can we delete the @ ones with minimal impact then? [23:14:30] not really [23:16:19] !log deleting mwext-*-lint* workspaces on gallium, shouldn't be needed [23:16:22] Logged the message, Master [23:18:46] !log Stopping Jenkins for an upgrade [23:18:48] Logged the message, Master [23:41:04] (03PS1) 10Hashar: zuul: ensure gate can runs at least 12 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/199169 [23:42:30] (03CR) 10Hashar: [C: 032] zuul: ensure gate can runs at least 12 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/199169 (owner: 10Hashar) [23:42:55] hasharDinner: Why is Jenkins being upgraded now? [23:43:08] Everything is stalled in Zuul. What's going on. [23:43:46] OK. Seems like that is already back up [23:43:58] Zuul is showing gate jobs that are completed or aborted, but it's not doing anything [23:44:13] Why is not reporting back to Gerrit and moving on to the next [23:44:48] In fact it has reported back to Gerrit already! It's just stuck in Zuul. [23:45:15] see pm [23:45:58] OK. But that's not causing the lock [23:46:13] it is deadlocked apparently :( [23:46:19] I think it's time we seriously give a look to our use of Zuul/Gearman there is a *fundemantal* logic problem (more than one) as to why it very frequently gets into a state where it's just not doing anything. [23:46:26] We've ruled out quite a few factors. [23:46:41] even fixed a few :) [23:46:52] here it is [23:48:03] I havent done anything though [23:48:12] I'm not blaming you :) [23:48:19] so the current state [23:48:23] (03Merged) 10jenkins-bot: zuul: ensure gate can runs at least 12 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/199169 (owner: 10Hashar) [23:48:25] gate-and-submit is a dependent pipeline [23:48:41] and such pipelines have a limit to the number of jobs they can run in parallel [23:48:44] It's just that the frequently and time wasted on restoring Jenkins functionality is imho unacceptable. If this were a consumer product, I'd have asked my money back, sued the sales guy and his entire famliy and wrote my own. [23:48:44] it defaults to 20 [23:48:57] that is linearly increased as long as jobs are passing / changes merged [23:49:03] on failure it is exponentially descreased [23:49:13] down to a minimum of only 3 jobs [23:49:23] I have raised it up to 12 with https://gerrit.wikimedia.org/r/199169 [23:49:24] Yeah, I've seen that before, but that's not the core problem. [23:49:37] so with the change above, we should always have 12 jobs runnable [23:49:49] (have *at least* 12 jobs) [23:49:52] There is a deeper problem that allows it to be in a state where Jenkins is running fine, Gearman is fresh and up, all slaves are connected, with empty slots, and nothig is happening. [23:50:28] Which happens at least twice a day, and I've yet to find a solution to the problem except for force-restarting Zuul. I've tried all the common tricks and things, none of them work. [23:50:33] When that particular scenario happens. [23:51:07] I suspect it may have to do with changes being force-merged and it losing track of changes somehow. [23:51:13] Nothing specific I found so far. [23:51:34] oh force merges would surely disrupt zuul [23:51:47] Why? [23:51:50] though it might handle the merge event properly not sure [23:52:15] well the change at the tip of the gate queue would not merge in Gerrit [23:52:22] since it is already merged (forcefully / manually) [23:52:28] so that would most probably trigger an error [23:52:33] cause the change to be dropped [23:52:37] Just now for example, there were 2 jobs at the top of the gate queue (within the 3 it allowed) that had all jobs completed. It even ran the postmerge thing already. But it stays "busy" in Jenkins console build, there is no way to abort it, and it just keeps spinning in Zuul. [23:52:42] and all the changes in the queues to be reenqueued / retested [23:52:51] Well, that's what should happen. [23:52:53] so if you have A <- B <- 50 jobs [23:52:58] you force merge A and B [23:52:59] Or it should assume success and continue. [23:53:14] A fails to merge and thus Zuul reenqueue B + 50 jobs [23:53:16] It also happens quite often with only a few things in the queue. And not releated to the gate queue per se. [23:53:21] once B tests are finished, it fails to merge [23:53:28] and zuul reenqueue again 50 jobs [23:53:28] Test queue as well. I've seen it plenty of times with nothing in the gate queue. [23:53:31] (in theroy) [23:53:36] Yeah [23:54:06] we should show the gate-and-submit window on the status page :) [23:54:46] we should reduce the stuff that shares one queue. I think the downstream/upstream optimisation is costing more time than it saves in practice. Or at least I'd like to challenge it with research. [23:54:57] We've always just assumed it [23:55:52] having projects find out their master is not compatible with mediawiki master after the change merges in master when they next submit a patch is acceptable. That's imho normal CI practice and not significant. That's not where our time is spent. [23:56:12] 10Continuous-Integration: Zuul status page should show the pipelines "window" value - https://phabricator.wikimedia.org/T93701#1143656 (10hashar) 3NEW [23:56:19] Anyway, got other stuff to fix :) [23:56:57] :D [23:57:05] So far my count is at 13 for the number of unrelated instrastructure components that required changing and were able to block adoption of Karma and MySQL. Including 1) sqlite database blocks in Trusty, 2) Unix path limit of 103 characters. [23:57:12] Computers suck. [23:57:15] well [23:57:22] you also overhauled the whole CI :) [23:57:26] It's unbelievable. [23:57:37] Yeah, but it was supposed to be a simple switch. [23:57:46] you migrated the mw jobs to use mysql as a backend which worth the news [23:57:58] Yeah, I'm writing a tech post about that to wikitech-l this week. [23:58:03] !!!!!!! [23:58:12] But it's still half-finished [23:58:25] Stuff keeps breaking [23:59:07] hasharDinner: April 1st I'm going to re-create the instances again. Please make sure the https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup page is up to date with any hacks you may had to apply the past month. [23:59:20] It's sometimes needed, I know it, I do it too. Just make sure it's all written up :) [23:59:28] Hopefully it will take only a day this time. [23:59:33] 6Release-Engineering, 6Engineering-Community: Lyon -> Annecy Transportation Info to RelEng Team - https://phabricator.wikimedia.org/T93686#1143673 (10Rfarrand) a:5MNeisler>3Rfarrand [23:59:35] hmm [23:59:46] beside the Zuul install madness, all should be in puppet :) [23:59:49] I barely do any hack