[00:06:34] PROBLEM - Puppet failure on integration-slave-trusty-1011 is CRITICAL 22.22% of data above the critical threshold [0.0] [00:06:56] PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL 50.00% of data above the critical threshold [0.0] [00:14:15] PROBLEM - Puppet failure on integration-slave-trusty-1015 is CRITICAL 44.44% of data above the critical threshold [0.0] [01:17:56] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [0.0] [01:30:11] 10Continuous-Integration-Infrastructure: jobs on core wmf branches use branch names of dependencies instead of the submodules - https://phabricator.wikimedia.org/T106777#1477842 (10JanZerebecki) 3NEW [02:46:14] 10Continuous-Integration-Infrastructure: Continuous Integration should offer a LESS linter - https://phabricator.wikimedia.org/T106780#1477932 (10Spage) 3NEW [03:02:42] 10Continuous-Integration-Infrastructure: Continuous Integration should offer a LESS linter - https://phabricator.wikimedia.org/T106780#1477986 (10Legoktm) When the MediaWiki structure tests run for extensions (not skins yet), all less files that are part of a resourceloader module have their less files compiled... [03:05:37] 10Continuous-Integration-Infrastructure: Continuous Integration should offer a LESS linter - https://phabricator.wikimedia.org/T106780#1477987 (10Spage) FWIW after `npm install recess` ``` $ ./node_modules/.bin/recess --includePath=resources/src/mediawiki.less resources/src/mediawiki.ui/components/utilities.les... [04:02:36] 10Continuous-Integration-Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - https://phabricator.wikimedia.org/T72049#1478082 (10yuvipanda) [04:35:01] 10Beta-Cluster, 10Pywikibot-OAuth: Investigate process for setting up an OAuth client on the Beta cluster - https://phabricator.wikimedia.org/T104764#1478153 (10VcamX) [05:34:20] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<10.00%) [05:35:34] Yippee, build fixed! [05:35:34] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #489: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/489/ [06:39:20] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:59:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 40.00% of data above the critical threshold [0.0] [07:09:29] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:33:30] 10Beta-Cluster, 10Pywikibot-OAuth: Set up a Pywikibot OAuth test client on the Beta cluster - https://phabricator.wikimedia.org/T104764#1478294 (10jayvdb) [07:39:54] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [08:02:47] (03PS16) 10Paladox: Update tests for Vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [08:03:12] (03PS14) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [08:04:20] (03PS8) 10Paladox: Update tests in vector extension [integration/config] - 10https://gerrit.wikimedia.org/r/225029 [08:04:48] (03PS12) 10Paladox: Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 [08:04:59] (03PS7) 10Paladox: Update farmer tests [integration/config] - 10https://gerrit.wikimedia.org/r/225042 [08:05:06] (03CR) 10Paladox: [C: 031] Update farmer tests [integration/config] - 10https://gerrit.wikimedia.org/r/225042 (owner: 10Paladox) [08:05:18] (03CR) 10Paladox: [C: 031] Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 (owner: 10Paladox) [08:08:06] (03PS7) 10Paladox: Update BreadCrumbs tests [integration/config] - 10https://gerrit.wikimedia.org/r/225291 [08:08:43] (03PS7) 10Galorefitz: Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) [08:13:46] (03CR) 10Paladox: "Ok I will file a bug for discussion not sure what project it should be under." [integration/config] - 10https://gerrit.wikimedia.org/r/226418 (owner: 10Paladox) [08:24:57] PROBLEM - Puppet failure on integration-slave-precise-1013 is CRITICAL 20.00% of data above the critical threshold [0.0] [08:26:16] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL 60.00% of data above the critical threshold [0.0] [08:27:16] (03CR) 10Polybuildr: "Looks good. Two minor issues with indentation that I'm not sure of." (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [08:28:52] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 40.00% of data above the critical threshold [0.0] [08:30:12] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 33.33% of data above the critical threshold [0.0] [08:30:55] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 50.00% of data above the critical threshold [0.0] [08:31:05] PROBLEM - Puppet failure on integration-publisher is CRITICAL 22.22% of data above the critical threshold [0.0] [08:31:17] PROBLEM - Puppet failure on integration-puppetmaster is CRITICAL 50.00% of data above the critical threshold [0.0] [08:35:34] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 50.00% of data above the critical threshold [0.0] [08:35:36] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 60.00% of data above the critical threshold [0.0] [08:35:42] PROBLEM - Puppet failure on deployment-salt is CRITICAL 50.00% of data above the critical threshold [0.0] [08:35:52] PROBLEM - Puppet failure on integration-saltmaster is CRITICAL 30.00% of data above the critical threshold [0.0] [08:36:14] (03PS1) 10Hashar: zuul: revert label back to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/226679 (https://phabricator.wikimedia.org/T106531) [08:37:17] (03CR) 10Hashar: [C: 032] zuul: revert label back to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/226679 (https://phabricator.wikimedia.org/T106531) (owner: 10Hashar) [08:38:41] (03CR) 10Hashar: [C: 032] zuul: revert label back to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/226679 (https://phabricator.wikimedia.org/T106531) (owner: 10Hashar) [08:38:43] (03PS1) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [08:38:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 60.00% of data above the critical threshold [0.0] [08:38:54] (03PS2) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [08:39:08] PROBLEM - Puppet failure on deployment-test is CRITICAL 44.44% of data above the critical threshold [0.0] [08:39:20] (03PS3) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [08:39:29] (03PS4) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [08:39:47] !log upgrading zuul [08:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:40:01] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 60.00% of data above the critical threshold [0.0] [08:40:18] (03CR) 10Paladox: "Npm can be added later once the files are merged for the extension." [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [08:40:21] !log upgrading zuul to zuul_2.0.0-327-g3ebedde-wmf3precise1 to fix a regression ( https://phabricator.wikimedia.org/T106531 ) [08:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:40:36] (03Merged) 10jenkins-bot: zuul: revert label back to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/226679 (https://phabricator.wikimedia.org/T106531) (owner: 10Hashar) [08:43:29] (03Restored) 10Hashar: Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [08:43:42] PROBLEM - Puppet failure on integration-slave-precise-1011 is CRITICAL 60.00% of data above the critical threshold [0.0] [08:44:06] PROBLEM - Puppet failure on integration-slave-precise-1012 is CRITICAL 33.33% of data above the critical threshold [0.0] [08:44:24] (03PS2) 10Hashar: Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 [08:44:39] PROBLEM - Puppet failure on deployment-upload is CRITICAL 40.00% of data above the critical threshold [0.0] [08:44:57] (03CR) 10jenkins-bot: [V: 04-1] Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [08:45:14] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 22.22% of data above the critical threshold [0.0] [08:45:15] (03CR) 10Hashar: [C: 032] Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [08:45:47] (03CR) 10jenkins-bot: [V: 04-1] Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [08:46:02] (03Abandoned) 10Hashar: Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [08:46:29] 10Continuous-Integration-Infrastructure, 6operations: Upload new Zuul .deb package on apt.wikimedia.org for precise-wikimedia - https://phabricator.wikimedia.org/T106499#1478406 (10hashar) [08:46:31] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Zuul: Changes voted CR+2 do not enter gate-and-submit unless Verified is +2. - https://phabricator.wikimedia.org/T106531#1478403 (10hashar) 5Open>3Resolved a:3hashar Fixed and tested on https://gerrit.wikimedia.org/r/#/c/226275/ [08:46:35] (03PS5) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [09:01:14] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK Less than 1.00% above the threshold [0.0] [09:03:52] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [09:04:56] RECOVERY - Puppet failure on integration-slave-precise-1013 is OK Less than 1.00% above the threshold [0.0] [09:05:11] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [09:05:55] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [09:06:19] RECOVERY - Puppet failure on integration-puppetmaster is OK Less than 1.00% above the threshold [0.0] [09:08:48] (03CR) 10Hashar: [C: 04-1] "See inline comments :-)" (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [09:10:34] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [09:10:35] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [09:10:43] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [09:11:07] RECOVERY - Puppet failure on integration-publisher is OK Less than 1.00% above the threshold [0.0] [09:13:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [09:14:10] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [09:15:02] RECOVERY - Puppet failure on integration-zuul-server is OK Less than 1.00% above the threshold [0.0] [09:15:52] RECOVERY - Puppet failure on integration-saltmaster is OK Less than 1.00% above the threshold [0.0] [09:17:24] (03PS8) 10Galorefitz: Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) [09:18:44] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK Less than 1.00% above the threshold [0.0] [09:19:13] (03CR) 10Galorefitz: "Please ignore this patch." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [09:19:40] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [09:19:46] (03PS6) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [09:24:06] RECOVERY - Puppet failure on integration-slave-precise-1012 is OK Less than 1.00% above the threshold [0.0] [09:25:15] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [09:25:37] (03PS1) 10Hashar: Create tox template for Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) [09:25:47] (03CR) 10Hashar: [C: 032] Create tox template for Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) (owner: 10Hashar) [09:27:46] (03PS7) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [09:27:51] (03PS2) 10Hashar: Create tox template for Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) [09:28:00] (03CR) 10Hashar: [C: 04-2] Create tox template for Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) (owner: 10Hashar) [09:30:25] (03Abandoned) 10Paladox: SyntaxHighlight_GeSHi run old test for branchs REL1_23, REL1_24, REL1_25 [integration/config] - 10https://gerrit.wikimedia.org/r/226418 (owner: 10Paladox) [09:30:27] (03PS3) 10Hashar: Create tox template for Jessie + common jobs [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) [09:34:23] (03CR) 10Hashar: [C: 032 V: 032] "I forgot to create the generic tox jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) (owner: 10Hashar) [09:36:22] (03Merged) 10jenkins-bot: Create tox template for Jessie + common jobs [integration/config] - 10https://gerrit.wikimedia.org/r/226686 (https://phabricator.wikimedia.org/T103977) (owner: 10Hashar) [09:38:39] (03PS9) 10Galorefitz: Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) [09:40:27] (03PS1) 10Hashar: Migrate operations/software/conftool tox to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226688 (https://phabricator.wikimedia.org/T103972) [09:40:56] special abusefilter page on en.wp seems broken [09:41:05] oh, wrong han [09:42:23] (03PS2) 10Hashar: conftool tox-flake8 is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/222293 [09:43:02] (03CR) 10Hashar: [C: 032] Migrate operations/software/conftool tox to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226688 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [09:43:51] Yippee, build fixed! [09:43:52] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #558: FIXED in 6 min 50 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/558/ [09:44:39] (03Merged) 10jenkins-bot: Migrate operations/software/conftool tox to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/226688 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [09:56:28] (03CR) 10Giuseppe Lavagetto: [C: 032] "Changes have been merged, we should enable this." [integration/config] - 10https://gerrit.wikimedia.org/r/222293 (owner: 10Hashar) [09:56:37] (03CR) 10Hashar: [C: 032] "Upstream change merged :)" [integration/config] - 10https://gerrit.wikimedia.org/r/222293 (owner: 10Hashar) [09:57:40] (03CR) 10Polybuildr: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [09:58:19] (03Merged) 10jenkins-bot: conftool tox-flake8 is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/222293 (owner: 10Hashar) [10:07:37] (03CR) 10Polybuildr: [C: 032] ":D" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [10:08:02] (03Merged) 10jenkins-bot: Sniff to check for space in single line comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [10:14:59] 10Continuous-Integration-Infrastructure: Pin tox to a specific version - https://phabricator.wikimedia.org/T106799#1478588 (10hashar) 3NEW a:3hashar [10:33:52] CFisch_WMDE1: argh, looks like my internet is behaving [10:34:05] just a second, connecting via different network [10:34:09] zeljkof: ok [10:45:08] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Pin tox to a specific version - https://phabricator.wikimedia.org/T106799#1478633 (10hashar) I used integration-slave-trusty-1011 as a playground area. I uninstalled pip and ran puppet: ``` Debug: Executing '/usr/bin/pip install -q tox' Notice: /Stage... [10:46:33] RECOVERY - Puppet failure on integration-slave-trusty-1011 is OK Less than 1.00% above the threshold [0.0] [10:48:58] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Pin tox to a specific version - https://phabricator.wikimedia.org/T106799#1478588 (10hashar) Deployed https://gerrit.wikimedia.org/r/#/c/226694/ on the integration puppetmaster and that works. Now pending for ops to merge in the change. [10:54:12] RECOVERY - Puppet failure on integration-slave-trusty-1015 is OK Less than 1.00% above the threshold [0.0] [10:59:55] RECOVERY - Puppet failure on integration-slave-trusty-1016 is OK Less than 1.00% above the threshold [0.0] [11:00:49] RECOVERY - Puppet failure on integration-slave-trusty-1021 is OK Less than 1.00% above the threshold [0.0] [11:04:11] RECOVERY - Puppet failure on integration-slave-trusty-1013 is OK Less than 1.00% above the threshold [0.0] [11:08:54] RECOVERY - Puppet failure on integration-slave-trusty-1012 is OK Less than 1.00% above the threshold [0.0] [11:11:56] RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK Less than 1.00% above the threshold [0.0] [11:32:34] (03Abandoned) 10Hashar: make-wmf-branch: include Wikibase and Wikidata extensions [tools/release] - 10https://gerrit.wikimedia.org/r/226059 (owner: 10Mjbmr) [11:34:06] RECOVERY - Puppet failure on integration-dev is OK Less than 1.00% above the threshold [0.0] [11:56:53] 10Continuous-Integration-Infrastructure: Jenkins: install tox on Precise labs instances - https://phabricator.wikimedia.org/T46443#1478822 (10hashar) [11:56:55] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Pin tox to a specific version - https://phabricator.wikimedia.org/T106799#1478821 (10hashar) 5Open>3Resolved [11:57:46] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Zuul: Changes voted CR+2 do not enter gate-and-submit unless Verified is +2. - https://phabricator.wikimedia.org/T106531#1478825 (10hashar) [12:07:17] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:19:09] 10Browser-Tests, 10Wikidata: adapt wikidata_api gem to be compatible with mediawiki_api gem version 0.4 - https://phabricator.wikimedia.org/T106811#1478899 (10WMDE-Fisch) 3NEW [12:46:04] !log Jenkins: switching gearman plugin from our custom compiled 0.1.1-9-g08e9c42-change_192429_2 to upstream 0.1.2. They are actually the exact same versions. [12:46:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:46:37] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #726: ABORTED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/726/ [12:49:33] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: Increase number of Jenkins slaves to spread load and prevent browser test failures on beta - https://phabricator.wikimedia.org/T72049#1478963 (10hashar) that was a request to #labs-infrastructure to bump the project quota. [13:24:22] 10Beta-Cluster: Puppet keeps restarting jobrunner service - https://phabricator.wikimedia.org/T76999#1479019 (10hashar) [13:24:49] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1479021 (10Aklapper) >>! In T616#1320300, @hashar wrote: > Seems more have been migrated since @demon last comment. Mind adding a status update?... [13:35:31] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<30.00%) [13:57:33] PROBLEM - Puppet failure on integration-slave-trusty-1011 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:07:25] 10Continuous-Integration-Infrastructure: Integration tests for Wikibase dependencies - https://phabricator.wikimedia.org/T54278#1479097 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki This is solved for dependencies on Wikibase by using the dependency mechanism. Also there mediawiki-extensions-hhvm is to... [14:07:34] RECOVERY - Puppet failure on integration-slave-trusty-1011 is OK Less than 1.00% above the threshold [0.0] [14:07:50] 10Continuous-Integration-Infrastructure, 10Wikidata: Integration tests for Wikibase dependencies - https://phabricator.wikimedia.org/T54278#1479101 (10JanZerebecki) [14:37:30] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1479149 (10WMDE-Fisch) [14:38:07] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #202: FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/202/ [14:43:14] twentyafterfour: That import into diffusion I was asking about, for mediawiki/extensions/SmiteSpam, any chance it could be done soon? Linking to commits on gerrit manually is tiring when writing out weekly reports. :/ [14:56:37] ostriches: maybe you could help? ^ [15:27:42] !log upgrading pip to 7.1.0 via pypi ( https://gerrit.wikimedia.org/r/#/c/226729/ ). Revert plan is to uncherry pick the patch on the puppetmaster and: pip uninstall pip [15:27:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:29:07] !log removing pip obsolete download-cache setting ( https://gerrit.wikimedia.org/r/#/c/226730/ ) [15:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:32:10] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL 20.00% of data above the critical threshold [0.0] [15:34:43] PROBLEM - Puppet failure on integration-slave-precise-1011 is CRITICAL 40.00% of data above the critical threshold [0.0] [15:37:54] PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL 50.00% of data above the critical threshold [0.0] [15:39:44] all lies [15:43:10] :) [15:47:20] (03Restored) 10Hashar: Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [15:47:26] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [15:48:48] (03Abandoned) 10Hashar: Invalid testing change [integration/config] - 10https://gerrit.wikimedia.org/r/226275 (owner: 10Hashar) [15:49:41] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK Less than 1.00% above the threshold [0.0] [15:51:48] PROBLEM - Puppet failure on integration-slave-trusty-1021 is CRITICAL 20.00% of data above the critical threshold [0.0] [15:52:13] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK Less than 1.00% above the threshold [0.0] [15:52:55] RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK Less than 1.00% above the threshold [0.0] [15:54:53] PROBLEM - Puppet failure on integration-slave-trusty-1012 is CRITICAL 20.00% of data above the critical threshold [0.0] [15:55:03] (03PS1) 10Hashar: conftool tox-py27-jessie is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/226733 (https://phabricator.wikimedia.org/T103972) [15:55:13] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL 22.22% of data above the critical threshold [0.0] [15:55:56] PROBLEM - Puppet failure on integration-slave-precise-1013 is CRITICAL 30.00% of data above the critical threshold [0.0] [15:55:57] PROBLEM - Puppet failure on integration-slave-trusty-1016 is CRITICAL 60.00% of data above the critical threshold [0.0] [15:56:49] (03CR) 10Hashar: [C: 032] conftool tox-py27-jessie is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/226733 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [15:58:11] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL 60.00% of data above the critical threshold [0.0] [15:58:25] (03Merged) 10jenkins-bot: conftool tox-py27-jessie is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/226733 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [16:08:32] PROBLEM - Puppet failure on integration-slave-trusty-1011 is CRITICAL 40.00% of data above the critical threshold [0.0] [16:08:57] PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL 60.00% of data above the critical threshold [0.0] [16:10:42] PROBLEM - Puppet failure on integration-slave-precise-1011 is CRITICAL 30.00% of data above the critical threshold [0.0] [16:15:06] PROBLEM - Puppet failure on integration-slave-precise-1012 is CRITICAL 44.44% of data above the critical threshold [0.0] [16:15:15] PROBLEM - Puppet failure on integration-slave-trusty-1015 is CRITICAL 33.33% of data above the critical threshold [0.0] [16:18:21] so [16:18:23] I broke puppet [16:18:42] I lost count of hours I wasted due to it [16:19:16] Error: Could not update: Could not locate the pip command. [16:19:16] Error: /Stage[main]/Contint::Packages::Python/Package[pip]/ensure: change from absent to 7.1.0 failed: Could not update: Could not locate the pip command. [16:23:48] :( [16:25:05] PROBLEM - Puppet failure on integration-dev is CRITICAL 33.33% of data above the critical threshold [0.0] [16:28:19] hashar: which hosts are you seeing that on? [16:30:08] thcipriani: I pushed a change earlier on to install pip with ... pip [16:30:16] https://gerrit.wikimedia.org/r/#/c/226729/ [16:30:42] seems there is some intermittent / random failures now http://shinken.wmflabs.org/problems?search=hg:integration [16:31:00] integration-slave-precise-1011 potentially has it [16:31:54] maybe they are just some old transient errors [16:32:09] can you install puppet with puppet? ;) [16:32:14] hmm [16:32:15] yeah [16:32:20] there is a gem provider in puppet [16:32:38] (03PS1) 10Mobrovac: Add the npm job for the MobileApps service repository [integration/config] - 10https://gerrit.wikimedia.org/r/226736 (https://phabricator.wikimedia.org/T106831) [16:32:38] so something like: package { 'puppet': provider => 'gem', ensure => 'latest', } [16:32:43] would self update puppet from ruby gems :} [16:35:14] RECOVERY - Puppet failure on integration-slave-trusty-1015 is OK Less than 1.00% above the threshold [0.0] [16:36:26] !log puppet on Jenkins slaves might have some intermittent issues due to pip installation https://gerrit.wikimedia.org/r/226729 [16:36:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:38:39] marxarelli: I talked with zeljkof this morning [16:38:48] about defining a ruby entry point [16:39:04] seems you guys consider rake to be overkill forcing to devs to have to learn something new [16:39:14] sounds like we would settle for make ? :} [16:39:27] make4life [16:39:33] I can imagine jenkins jobs just doing 'make test' 'make lint' [16:39:41] hashar: it's not that rake is overkill [16:40:14] it just doesn't seem necessary to implement yet another entry point, when we could just pick a single polyglot entry point [16:40:29] yeah [16:40:31] after all those years [16:40:37] I am still not smart enough to refactor [16:40:41] https://www.mediawiki.org/wiki/Continuous_integration/Entry_points [16:40:52] we went with entry points per languages [16:41:18] with some dispredancies. Javascript uses 'npm test' [16:41:33] but python repos have a job per tox env [16:46:13] PROBLEM - Puppet failure on integration-slave-trusty-1015 is CRITICAL 55.56% of data above the critical threshold [0.0] [16:47:07] ... [16:48:33] hashar: oh, i forgot that this patch wasn't merged https://gerrit.wikimedia.org/r/#/c/224456/2 [16:48:42] if you have a moment ... :) [16:49:08] I think I sneaked at it last week and it looked fine :-} [16:50:46] hashar: sweet! i'll merge it then [16:51:19] (03CR) 10Dduvall: [C: 032] Separate mw-selenium setup/teardown and be strict [integration/jenkins] - 10https://gerrit.wikimedia.org/r/224456 (owner: 10Dduvall) [16:51:36] twentyafterfour: Sorry, had to leave then. Also sorry because this is getting repetitive. Should I stop now? :P [16:52:16] marxarelli: yeah I guess you can self merge [16:52:26] you have plenty of time ahead today to amend / refine it as needed [16:52:32] and seems to have limited impact :-) [16:52:43] (03CR) 10Hashar: [C: 031] Separate mw-selenium setup/teardown and be strict [integration/jenkins] - 10https://gerrit.wikimedia.org/r/224456 (owner: 10Dduvall) [16:54:59] !log Might have fixed the puppet/pip mess on CI slaves by creating a symlink from /usr/bin/pip to /usr/local/bin/pip ( https://gerrit.wikimedia.org/r/#/c/226729/1..2/modules/contint/manifests/packages/python.pp,unified ) [16:55:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:55:05] thcipriani: I think I solved it [16:55:17] puppet is run from a cron and /usr/local/sbin/puppet-run [16:55:24] which is probably missing /usr/local/bin in its PATH [16:55:31] and 'pip install pip' get rid of /usr/bin/pip [16:55:33] so pip is not found [16:55:39] and hmm puppet bails out [16:56:08] (03PS4) 10Dduvall: Create and cleanup TMPDIR for mw-selenium builder [integration/config] - 10https://gerrit.wikimedia.org/r/224179 (https://phabricator.wikimedia.org/T103039) [16:57:55] (03PS5) 10Dduvall: Create and cleanup TMPDIR for mw-selenium builder [integration/config] - 10https://gerrit.wikimedia.org/r/224179 (https://phabricator.wikimedia.org/T103039) [16:58:40] (03Merged) 10jenkins-bot: Separate mw-selenium setup/teardown and be strict [integration/jenkins] - 10https://gerrit.wikimedia.org/r/224456 (owner: 10Dduvall) [17:01:17] RECOVERY - Puppet failure on integration-slave-trusty-1015 is OK Less than 1.00% above the threshold [0.0] [17:01:21] ohh [17:01:53] gotta leave :-( [17:02:10] did you get pip figured out? [17:02:13] yeah [17:02:17] see above :} [17:02:21] like ~ 7 minutes ago [17:02:21] neat :) [17:02:23] hashar: have a good weekend! [17:02:30] going to mass run /usr/local/sbin/puppet-run on all slaves [17:03:27] !log integration-saltmaster : salt -b 1 '*slave*' cmd.run '/usr/local/sbin/puppet-run' & && disown && exit [17:03:31] I should use screen really [17:03:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:04:14] !log cancelled last command [17:04:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:04:33] !log integration-saltmaster, in a '''screen''' : salt -b 1 '*slave*' cmd.run '/usr/local/sbin/puppet-run'|tee hashar-massrun.log [17:04:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:04:44] (03CR) 10Dduvall: [C: 031] Create and cleanup TMPDIR for mw-selenium builder [integration/config] - 10https://gerrit.wikimedia.org/r/224179 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [17:04:47] :) [17:04:55] well [17:05:01] we should have migrated to tmux already [17:05:02] :D [17:05:24] bah, GPL4evah [17:05:27] !log running `jenkins-jobs update config/ 'mediawiki-selenium-integration'` to deploy and test Ib289d784c7b3985bd4823d967fbc07d5759dc756 [17:05:29] :P [17:05:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:05:50] (03CR) 10Dduvall: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/226653 (owner: 10Dduvall) [17:06:35] marxarelli: https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/SAL :-D [17:06:46] bd808 Inc. latest pet tool [17:06:53] oooooh! [17:07:00] neat [17:07:39] I have added the nickname to the event list [17:07:55] yeah that is great [17:08:00] finally we can ... search! [17:08:32] look at you! number #1 [17:08:36] :) [17:09:29] (03CR) 10Dduvall: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/226653 (owner: 10Dduvall) [17:09:43] I'm going to setup a logstash host in my labs project jsut for irc indexing and make it public [17:10:01] then I'm going to try and write a pretty tool jsut for SAL [17:10:18] sounds like great goals [17:10:22] thank you Bryan! [17:10:38] now I am heading to dinner && bed ! have a good friday folks [17:10:40] and since I just got 8 hours of my day back from the logstash cluster upgrade I might get that done today :) [17:10:50] might look at puppet runs on CI later tonight [17:10:55] ahah [17:10:56] clever [17:11:02] hashar: bon appetit! [17:11:14] bd808: yes yes yes! 20% time from over-delivering is always warranted [17:11:31] merci! [17:12:22] very cool. can we have icinga events in logstash? [17:12:56] jzerebecki: https://phabricator.wikimedia.org/T7 [17:13:03] someday [17:13:12] thx :) [17:13:22] (03CR) 10Dduvall: [C: 032] "Deployed and tested by running the mediawiki-selenium-integration job against I639895ee8840564c28b060d8b65055772ad28694." [integration/config] - 10https://gerrit.wikimedia.org/r/224179 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [17:14:27] T7? wow [17:15:25] (03Merged) 10jenkins-bot: Create and cleanup TMPDIR for mw-selenium builder [integration/config] - 10https://gerrit.wikimedia.org/r/224179 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [17:17:13] PROBLEM - Puppet failure on integration-slave-trusty-1015 is CRITICAL 55.56% of data above the critical threshold [0.0] [17:17:37] !log running `jenkins-jobs update config/ 'mediawiki-selenium-integration' 'mwext-mw-selenium'` to deploy Ib289d784c7b3985bd4823d967fbc07d5759dc756 [17:17:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:18:11] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK Less than 1.00% above the threshold [0.0] [17:19:53] RECOVERY - Puppet failure on integration-slave-trusty-1012 is OK Less than 1.00% above the threshold [0.0] [17:20:05] RECOVERY - Puppet failure on integration-slave-precise-1012 is OK Less than 1.00% above the threshold [0.0] [17:20:57] RECOVERY - Puppet failure on integration-slave-precise-1013 is OK Less than 1.00% above the threshold [0.0] [17:25:17] PROBLEM - Puppet failure on nodepool-t105406 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:28:26] shit balls. and i broke ci... [17:28:38] "unbound variable SKIP_TMPFS" [17:31:55] (03PS1) 10Dduvall: Fix unbound variable SKIP_TMPFS error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226747 [17:33:11] (03CR) 10Dduvall: [C: 032] Fix unbound variable SKIP_TMPFS error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226747 (owner: 10Dduvall) [17:33:32] RECOVERY - Puppet failure on integration-slave-trusty-1011 is OK Less than 1.00% above the threshold [0.0] [17:33:48] (03Merged) 10jenkins-bot: Fix unbound variable SKIP_TMPFS error [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226747 (owner: 10Dduvall) [17:33:56] RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK Less than 1.00% above the threshold [0.0] [17:35:31] !log updating integration slave scripts from integration-saltmaster to deploy I6906fadede546ce2205797da1c6b267aed586e17 [17:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:36:46] marxarelli: :D [17:37:04] I was just coming looking for who to poke ;) [17:37:27] :) [17:37:50] it should be fixed now [17:38:01] *rechecks* [17:38:14] i forgot that jenkins runs its shell jobs in a very strict mode [17:38:26] * marxarelli should do the same when testing slave scripts [17:39:28] greg-g: I was a very early tester of the phab setup. https://phabricator.wikimedia.org/project/profile/1/ [17:40:18] RECOVERY - Puppet failure on integration-slave-trusty-1013 is OK Less than 1.00% above the threshold [0.0] [17:41:49] RECOVERY - Puppet failure on integration-slave-trusty-1021 is OK Less than 1.00% above the threshold [0.0] [17:43:14] yup, all fixed for me marxarelli :) [17:43:27] oh good! [17:43:36] sorry about that :) [17:43:50] no worries :D [17:43:56] bd808: :) [17:45:59] RECOVERY - Puppet failure on integration-slave-trusty-1016 is OK Less than 1.00% above the threshold [0.0] [17:46:57] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480101 (10polybuildr) Could we please please have an update on this? The extension I'm currently writing as a GSoC project (phab: #mediawiki-ext... [17:49:06] (03PS2) 10Dduvall: Configure video capture for headless mw-selenium jobs [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226651 (https://phabricator.wikimedia.org/T104583) [18:02:54] marxarelli: do you know of any docs about the strict mode jenkins uses for shell jobs? i was always wondering if I need to add set -eu in inline jjb shell snippets. or doesn't it actually and it is just this scipt has #!/bin/bash -eu ? [18:03:42] jzerebecki: i couldn't find anything specific [18:03:57] ok thx [18:04:48] jzerebecki: it'd be great to find some though, and implement a linter test for our slave scripts [18:05:43] marxarelli: the only thing i could find was the setting for which shell to use which we have set to just /bin/bash [18:06:02] (03PS1) 10Mjbmr: Add Mjbmr to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/226753 [18:10:46] marxarelli: i created https://phabricator.wikimedia.org/T106384 to fix inline scripts missing it. but having a test for it might be even better. [18:11:11] or we could just stop using bash/shell [18:11:28] its broken. [18:11:40] for programming. [18:16:14] jzerebecki: heh. not my favorite 'language' either but we have a lot of existing code to consider [18:33:08] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480329 (10mmodell) @polybuildr: rSSPM [18:36:46] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480349 (10Ricordisamoa) >>! In T616#1480329, @mmodell wrote: > rSSPM https://www.mediawiki.org/wiki/Phabricator/Diffusion/Callsign_naming_conve... [18:39:30] (03CR) 10BearND: [C: 031] "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/226736 (https://phabricator.wikimedia.org/T106831) (owner: 10Mobrovac) [18:41:07] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480363 (10JanZerebecki) >>! In T616#1424341, @QuimGil wrote: > How to keep up with new repositories? Change https://www.mediawiki.org/wiki/Gerr... [18:42:28] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480368 (10mmodell) Our callsign naming conventions make no sense and limiting callsigns to 4 characters serves no purpose at all. [18:53:29] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#1480403 (10Aklapper) >>! In T616#1480368, @mmodell wrote: > Our callsign naming conventions make no sense and limiting callsigns to 4 characters... [19:01:52] 10Beta-Cluster, 10Pywikibot-OAuth: Set up a Pywikibot OAuth test client on the Beta cluster - https://phabricator.wikimedia.org/T104764#1480430 (10jayvdb) [19:06:25] 10Beta-Cluster, 10Pywikibot-OAuth: Set up a Pywikibot OAuth test client on the Beta cluster - https://phabricator.wikimedia.org/T104764#1480454 (10jayvdb) Rather than wait for {T100797}, I think this test should have a dedicated account "Pywikibot-oauth". Travis can have a separate environment variable for th... [19:06:39] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1480470 (10brion) The state of video scaling in Trusty seems to be completely fubar if my testing... [19:18:11] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1480526 (10brion) >>! In T103335#1480470, @brion wrote: > The state of video scaling in Trusty see... [19:35:44] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK Less than 1.00% above the threshold [0.0] [19:40:48] ah [19:41:07] so I think the magic /usr/bin/pip symlink to /usr/local/bin/pip fixed the puppet cron [19:41:09] at last [19:59:45] greg-g: Would it be OK to do an emergency deploy for VE? https://phabricator.wikimedia.org/T106696 UBN caused by us not fixing a deprecation that turned into a removal. Pretty key workflow. :-( [20:01:04] (/cc Krenair.) [20:01:20] marxarelli: did we update chrome driver? [20:01:47] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/ has started failing consistently [20:16:37] James_F: sorry, was afk, it says resolved now? [20:16:51] greg-g: Resolved in master. Still an issue in prod. [20:18:08] James_F: maybe put back the deprecated call ? [20:18:18] in case it is still used elsewhere [20:18:28] hashar: We announced the breaking change three months ago. [20:18:32] hashar: We just missed one. [20:18:41] ok ok :-} [20:18:44] just checkin hehe [20:18:48] hashar: Also, the breaking change is in a library. We really don't want to revert the library. :-) [20:19:04] James_F: doit [20:19:07] Kk. [20:19:12] Krenair: You want to do it? [20:20:15] jdlrobson: marxarelli changed that job on 2015-07-23_23-10-43 [20:20:23] jdlrobson: https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/jobConfigHistory/showDiffFiles?timestamp1=2015-07-01_08-39-25×tamp2=2015-07-23_23-10-43 [20:20:52] jdlrobson: seems the repo moved to a new version of mediawiki-selenium [20:21:18] marxarelli: can you let me know when you change versions in selenium? I'll bring this up in Monday's session :) hashar thanks for letting me know [20:22:05] jdlrobson: well the version bump is done in the extension source repo so it should be noticed [20:22:15] jdlrobson: in this case the last two builds fail on the same test "Notification.Opening notifications" [20:22:21] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/203/testReport/ [20:22:29] unknown error: Element is not clickable at point (941, 26). Other element would receive the click: ... [20:22:30] jdlrobson hashar: the version of mediawiki_selenium is specified in your Gemfile [20:22:32] no clue what that could mean [20:22:42] when i change it, i add you as a reviewer :P [20:22:44] but http://saucelabs.com/jobs/dc219f1fb613468999cfc91f99a02ab4 might tell [20:22:55] marxarelli: you are so sweet to our devs! [20:23:22] marxarelli: sounds good. would be also good to verify these changes manually straight after a merge. [20:23:47] i hadn't connected the dots and thought we'd somehow broken something [20:23:59] James_F, okay.. [20:24:11] apparently it tries to click on an element with id secondary-button and class user-button [20:24:16] but the span notification count is over it [20:24:26] jdlrobson: it likely has nothing to do with the upgrade [20:24:36] so something triggered that user to have a notification which the test does not take in account :/ [20:24:39] marxarelli: but it hasn't been failing before... [20:24:41] Krenair: Core patch incoming. [20:24:44] so i'm confused why it would suddenly stop failing [20:24:49] Krenair: https://gerrit.wikimedia.org/r/#/c/226907/ [20:25:07] jdlrobson: there are many many moving parts [20:25:07] jdlrobson: there is notification red box with number [1] [20:25:13] that's integration testing [20:25:15] jdlrobson: that seems to overlap the bell icon [20:25:30] hashar: yeh i see why it's failing but i don't know what's caused that - this red box has always been there [20:25:33] in this particular case, it's not even our chromedriveer [20:25:36] it's saucelabs [20:25:40] ahh [20:25:40] that's why i wondered if chromedriver on saucelabs got updated or something [20:25:48] marxarelli: ahh okay.. so i guess that's my question [20:25:54] how do i find out when chromedriver gets updated on saucelabs [20:26:36] so, i think we're jumping to conclusions here. the first step to debugging this is to reproduce it [20:27:14] i.e. run the scenario through saucelabs, specifying the latest chrome version and working backwards [20:27:19] marxarelli: sure that's next on my list after lunch [20:27:25] i'm just trying to compile theories [20:27:42] totally, it's a decent theory [20:27:51] we just need to verify it [20:28:12] jdlrobson: do you have a saucelabs api key? [20:30:41] ok time for a late lunch [20:30:45] nope i dont have a saucelabs key [20:30:53] saucelabs is a mystery to me :) [20:31:39] hashar: do you know who has the login for the main saucelabs account? [20:33:03] i have my own subaccount of the `wikimedia` account. we should create some for each team or at least vertical [20:39:10] (03PS1) 10Hashar: (WIP) Generic debian-glue (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/226911 [20:41:19] greg-g: All clear. [20:50:24] (03PS1) 10Paladox: Update WikidataPageBanner tests [integration/config] - 10https://gerrit.wikimedia.org/r/226913 [20:50:42] (03PS2) 10Paladox: Update WikidataPageBanner tests [integration/config] - 10https://gerrit.wikimedia.org/r/226913 [21:03:15] (03PS3) 10Paladox: Update WikidataPageBanner tests [integration/config] - 10https://gerrit.wikimedia.org/r/226913 [21:06:09] (03PS2) 10Hashar: (WIP) Generic debian-glue (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/226911 [22:25:07] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1481198 (10MoritzMuehlenhoff) >>! In T103335#1480470, @brion wrote: > The state of video scaling i... [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:39:58] 10Beta-Cluster, 10Sentry, 10Wikimedia-Logstash: Channel PHP errors from Logstash to Sentry on the beta cluster - https://phabricator.wikimedia.org/T85239#1481572 (10Tgr)