[00:00:29] Krenair: there's been a few we just did a release [00:00:38] Im more confused with how that got through Jenkins https://integration.wikimedia.org/ci/job/mwext-mw-selenium/2567/console [00:01:12] https://gerrit.wikimedia.org/r/#/c/260680/ [00:01:16] that's on the dev branch, not master [00:01:19] beta runs master [00:01:27] of everything [00:02:01] https://gerrit.wikimedia.org/r/#/c/259860/ [00:02:03] ah, but then it was abandoned and replaced with a master patch [00:02:04] that ran on master [00:02:07] sorry had wrong url [00:02:13] https://integration.wikimedia.org/ci/job/mwext-mw-selenium/2569/console [00:02:20] only touches extension.json though? [00:02:41] and no selenium jenkins job should run on dev - marxarelli ? [00:04:18] Krenair: drinksurvey should not be loading on beta labs which is also concerning me [00:04:55] oh i have a theory [00:05:15] Anyone looking at the Beta Cluster issues: http://en.wikipedia.beta.wmflabs.org/w/load.php?debug=false&lang=en&modules=ext.cite.style|ext.flow.board.styles|ext.flow.board.topic.styles|ext.flow.mediawiki.ui.form%2Cmodal%2Ctext%2Ctooltips|ext.flow.styles.base|ext.gadget.PagesForDeletion%2CwmfFR2011Style|ext.tmh.thumbnail.styles|ext.uls.nojs|ext.visualEditor.desktopArticleTarget.noscript|ext.wikimediaBadges|mediawiki.legacy. [00:05:16] commonPrint%2Cshared|mediawiki.sectionAnchor%2Cui|mediawiki.skinning.content.parsoid|mediawiki.skinning.interface|mediawiki.ui.anchor%2Cbutton%2Cicon%2Cinput%2Ctext|mediawiki.widgets.styles|oojs-ui.styles|oojs-ui.styles.icons%2Cicons-alerts%2Cicons-content%2Cicons-editing-core%2Cicons-interactions%2Cicons-layout%2Cicons-moderation%2Cicons-movement%2Cindicators%2Ctextures|skins.vector.styles|skins.vector.styles.responsive& [00:05:18] only=styles&skin=vector ? [00:05:23] Krenair: sigh fix on way [00:05:25] config change [00:05:30] yes matt_flaschen, see this channel [00:05:54] wait.. wtf [00:05:59] Sorry, I'm not auto-joined. I will in the future. [00:06:10] okay [00:06:13] reverting that patch fixed beta [00:06:22] Krenair: it's not that [00:06:33] what is $wgQuickSurveysConfig defined as on beta cluster? [00:06:36] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39787 bytes in 0.981 second response time [00:06:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31066 bytes in 1.465 second response time [00:07:02] jdlrobson, array(5) [00:07:12] what is $wgQuickSurveysConfig[0] [00:07:14] or var_dump it [00:07:25] as the one enabled on cluster has platforms so i'm a little confused [00:07:52] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39432 bytes in 0.705 second response time [00:08:00] it's an array(16) [00:08:15] > var_dump( $wgQuickSurveysConfig[2]['name'] ); [00:08:15] string(12) "drink-survey" [00:08:48] Krenair: i need to see all the contents of it [00:09:03] it should match the value in wmf-config/CommonSettings-labs.php [00:09:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39437 bytes in 0.984 second response time [00:09:51] jdlrobson, https://phabricator.wikimedia.org/P2451 [00:10:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39454 bytes in 9.780 second response time [00:10:27] ahhh i see [00:12:33] Krenair: so this should fix it: https://gerrit.wikimedia.org/r/260696 [00:12:37] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 3045 bytes in 0.626 second response time [00:12:37] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 3148 bytes in 0.896 second response time [00:13:55] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 2655 bytes in 0.636 second response time [00:14:10] that looks like it, thanks jdlrobson [00:15:07] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 2655 bytes in 1.015 second response time [00:15:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 2655 bytes in 0.829 second response time [00:17:44] beta is still unreachable for me, is there another step to do? [00:20:17] no [00:20:22] waiting for it to finish deploying the fix [00:23:51] ah [00:23:54] jdlrobson, [00:23:56] [9be15ddf] /wiki/Main_Page?debug=true InvalidArgumentException from line 101 of /srv/mediawiki/php-master/extensions/QuickSurveys/includes/SurveyFactory.php: The "drink-survey" survey has specified an invalid platform. Please specify one or more of the following for the "mobile" platform: stable, beta. [00:24:06] different error [00:24:10] argg [00:24:15] fixing [00:25:11] Krenair: https://gerrit.wikimedia.org/r/260699 [00:26:26] 10Deployment-Systems, 6operations, 5Patch-For-Review: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585#1899836 (10Dzahn) a:5Dzahn>3None [00:33:24] jdlrobson, Luke081515: looks back up now [00:33:34] Yeah, thanks [00:33:45] Krenair: thanks sorry about that. These config issues are a pain. [00:33:54] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39446 bytes in 0.815 second response time [00:34:00] np, didn't affect me [00:34:06] just trying to make sure beta doesn't break so horribly [00:35:04] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39454 bytes in 1.026 second response time [00:35:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39453 bytes in 0.687 second response time [00:37:38] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39773 bytes in 0.761 second response time [00:37:38] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31065 bytes in 0.728 second response time [00:53:51] marxarelli: are you there? Can you remind me how extension tags work on browser tests? [00:54:24] marxarelli: e.g. https://gerrit.wikimedia.org/r/#/c/257410/2/tests/browser/features/read_more.feature [01:08:14] Project beta-scap-eqiad build #83320: 04FAILURE in 3 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83320/ [01:18:22] jdlrobson: yeah, sorry. stepped out [01:19:50] jdlrobson: mw-selenium will skip tests that are tagged with @extension-{name} if {name} is not an enabled extension (according to the MW API siteinfo) [01:25:15] so it doesn't work locally? [01:25:44] jdlrobson: it should [01:25:59] since it simply queries the api, no magic stuff [01:26:09] mm weird. [02:15:53] Yippee, build fixed! [02:15:54] Project beta-scap-eqiad build #83323: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83323/ [08:25:50] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #822: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/822/ [08:51:47] 10Continuous-Integration-Config: Mark UserDailyContribs as archived in CI - https://phabricator.wikimedia.org/T122279#1900397 (10Legoktm) 3NEW [09:29:54] 10Continuous-Integration-Config, 10Wikidata: increase timeout of composer for php-composer-test - https://phabricator.wikimedia.org/T121291#1900438 (10thiemowmde) It seems this got worse just a few days ago. Possibly because of https://gerrit.wikimedia.org/r/259485. I see jobs constantly failing now. As a wor... [09:45:38] 10Continuous-Integration-Config, 10Wikidata: increase timeout of composer for php-composer-test - https://phabricator.wikimedia.org/T121291#1900457 (10Addshore) >>! In T121291#1900438, @thiemowmde wrote: > It seems this got worse just a few days ago. Possibly because of https://gerrit.wikimedia.org/r/259485. I... [10:13:41] (03PS1) 10JanZerebecki: composer test: increase process timeout [integration/config] - 10https://gerrit.wikimedia.org/r/260727 (https://phabricator.wikimedia.org/T121291) [10:31:44] (03PS1) 10Paladox: [UserDailyContribs] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/260729 (https://phabricator.wikimedia.org/T122279) [10:54:06] (03CR) 10Thiemo Mättig (WMDE): [C: 031] composer test: increase process timeout [integration/config] - 10https://gerrit.wikimedia.org/r/260727 (https://phabricator.wikimedia.org/T121291) (owner: 10JanZerebecki) [11:01:17] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review: increase timeout of composer for php-composer-test - https://phabricator.wikimedia.org/T121291#1900564 (10thiemowmde) I wanted to benchmark this anyway. The question I wanted to answer was: Is there a golden bullet? Is there a specific PHPCS sni... [11:20:49] (03PS1) 10Paladox: [Newsletter] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/260741 [11:21:24] (03PS2) 10Paladox: [Newsletter] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/260741 (https://phabricator.wikimedia.org/T121352) [11:37:41] (03PS1) 10Paladox: Update mw-checks and mw-checks-test templates [integration/config] - 10https://gerrit.wikimedia.org/r/260743 [11:39:23] Project beta-scap-eqiad build #83373: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83373/ [12:11:01] (03CR) 10Addshore: [C: 031] composer test: increase process timeout [integration/config] - 10https://gerrit.wikimedia.org/r/260727 (https://phabricator.wikimedia.org/T121291) (owner: 10JanZerebecki) [12:55:37] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #698: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/698/ [12:59:36] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #887: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/887/ [14:05:55] (03CR) 10JanZerebecki: [C: 032] "Deployed to Jenkins: (['php-composer-test', 'php-composer-test-hhvm', 'php-composer-test-zend'])" [integration/config] - 10https://gerrit.wikimedia.org/r/260727 (https://phabricator.wikimedia.org/T121291) (owner: 10JanZerebecki) [14:07:58] (03Merged) 10jenkins-bot: composer test: increase process timeout [integration/config] - 10https://gerrit.wikimedia.org/r/260727 (https://phabricator.wikimedia.org/T121291) (owner: 10JanZerebecki) [14:11:20] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-12-01: increase timeout of composer for php-composer-test - https://phabricator.wikimedia.org/T121291#1900857 (10JanZerebecki) 5Open>3Resolved [15:04:50] 10Continuous-Integration-Config: translatewiki.net phplint job should allow PHP 5.4 syntax - https://phabricator.wikimedia.org/T97889#1900934 (10Paladox) Would using composer-test then installing php then run the phplint test would that work. [15:36:38] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [15:46:37] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.20 ms [16:53:08] jzerebecki: Could you review and merge https://gerrit.wikimedia.org/r/#/c/260743/ please. [17:01:54] (03PS1) 10Bmansurov: Setup RelatedArticles browser tests job [integration/config] - 10https://gerrit.wikimedia.org/r/260764 (https://phabricator.wikimedia.org/T120715) [17:20:50] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [17:31:00] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [18:11:20] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [18:40:53] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.14 ms [19:07:33] 10MediaWiki-Releasing: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#1901529 (10GWicke) In the latest version, there is now support for the (in)famous `curl | bash` one-line install pattern: ``` curl https://raw.githubusercontent.com/wikimedia/mediawiki-containers/master/m... [19:08:19] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [19:30:57] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [19:49:19] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [20:00:54] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.64 ms [20:31:34] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [20:55:52] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [20:58:22] euh: Build run failed: [Errno 28] No space left on device: '/mnt/jenkins-workspace/puppet-compiler/1544' [20:59:24] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/1544/console [21:05:22] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1901987 (10mobrovac) 3NEW [21:12:01] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902019 (10Dzahn) a:3Dzahn [21:13:36] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1901987 (10Dzahn) I checked /mnt/jenkins-workspace/puppet-compiler/output# for especially large ones, and: root@compiler02:/mnt/jenkins... [21:15:04] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902035 (10Dzahn) 5Open>3Resolved these really large ones must have been running on a LOT of nodes (maybe it was attempted to use th... [21:17:45] 10Continuous-Integration-Infrastructure, 6Labs, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902074 (10Dzahn) [21:37:19] marxarelli: around? [21:37:38] jdlrobson: yeah but heading out for lunch in a few [21:37:40] what's up? [21:37:55] https://gerrit.wikimedia.org/r/#/c/260764/1 < can i get a review? [21:38:04] also keen to get MultimediaViewer running browser tests on a per commit basis [21:38:15] i can write the patches but just wanted to get someone who could merge them and see it through to completion :) [21:38:34] cool. have you created new jjb jobs before? [21:38:57] Yeh but I can't remember exactly ;-) [21:39:04] i'll just need to refresh my memory [21:39:24] no worries. looks pretty good so far [21:42:54] marxarelli: remind me.. did you document this on wiki? [21:43:28] jdlrobson: i didn't but, yeah, it's documented [21:43:36] * marxarelli searches [21:43:44] Trying to remember all steps involved [21:44:04] jdlrobson: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated [21:44:23] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [21:44:24] ? [21:44:25] sorry, that's not it :) [21:44:32] yeh :) [21:45:31] jdlrobson: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder [21:45:35] 5Release-Engineering-Epics, 10Browser-Tests-Infrastructure, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1902205 (10Jdlrobson) [21:46:04] 5Release-Engineering-Epics, 10Browser-Tests-Infrastructure, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#541910 (10Jdlrobson) >>! In T55697#1646493, @Jdlrobson wrote: > The description is a little confusing. Does VisualEditor do this alrea... [21:46:34] jdlrobson: k. gotta jet but i'll review your patch and create the job if all is well when i get back [21:47:56] marxarelli: was hoping for documentation on https://gerrit.wikimedia.org/r/#/c/221313/5 :/ [21:52:41] (03PS1) 10Jdlrobson: Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) [21:53:08] (03PS2) 10Jdlrobson: Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) [21:54:15] (03CR) 10jenkins-bot: [V: 04-1] Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) (owner: 10Jdlrobson) [21:55:32] (03PS3) 10Jdlrobson: Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) [21:55:56] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [21:57:35] (03PS1) 10Jdlrobson: Enable the jjb browser test job for RelatedArticles [integration/config] - 10https://gerrit.wikimedia.org/r/260867 (https://phabricator.wikimedia.org/T120715) [22:35:49] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [23:09:24] jdlrobson: https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests [23:11:09] (03CR) 10Dduvall: [C: 032] Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) (owner: 10Jdlrobson) [23:12:30] (03Merged) 10jenkins-bot: Run MultimediaViewer browser tests on each commit [integration/config] - 10https://gerrit.wikimedia.org/r/260866 (https://phabricator.wikimedia.org/T113620) (owner: 10Jdlrobson) [23:15:35] !log Reloading Zuul to deploy I78727ce68b45f3a6305291e6e1e596b62069fc21 [23:15:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:15:56] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms [23:26:24] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [23:30:10] (03PS2) 10Dduvall: Enable the jjb browser test job for RelatedArticles [integration/config] - 10https://gerrit.wikimedia.org/r/260867 (https://phabricator.wikimedia.org/T120715) (owner: 10Jdlrobson) [23:30:26] (03CR) 10Dduvall: [C: 032] Enable the jjb browser test job for RelatedArticles [integration/config] - 10https://gerrit.wikimedia.org/r/260867 (https://phabricator.wikimedia.org/T120715) (owner: 10Jdlrobson) [23:31:38] (03Merged) 10jenkins-bot: Enable the jjb browser test job for RelatedArticles [integration/config] - 10https://gerrit.wikimedia.org/r/260867 (https://phabricator.wikimedia.org/T120715) (owner: 10Jdlrobson) [23:37:20] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit in the short term - https://phabricator.wikimedia.org/T94322#1902682 (10Legoktm) Is it time to re-evaluate this given that we can now use the `Depends-On` header to mark cross-repo de... [23:37:26] !log Reloading Zuul to deploy I39b9f292e95363addf8983eec5d08a0af527a163 [23:37:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:50:52] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 3.01 ms [23:56:50] Yippee, build fixed! [23:56:50] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce build #275: 09FIXED in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce/275/