[00:09:06] (03CR) 10JanZerebecki: "Deployed Jenkins jobs (['mwext-MobileFrontend-doxygen-publish', 'mwext-MobileFrontend-publish'])." [integration/config] - 10https://gerrit.wikimedia.org/r/223527 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [00:10:12] (03CR) 10JanZerebecki: [C: 032] Publish php docs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/223527 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [00:12:20] (03Merged) 10jenkins-bot: Publish php docs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/223527 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [00:14:56] !log reloading zuul 369e6eb..73dc1f6 for https://gerrit.wikimedia.org/r/#/c/223527/ [00:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:16:50] (03CR) 10Hoo man: [C: 04-1] "We had this discussion before and decided we don't want to go on a one-week deployment cycle." [tools/release] - 10https://gerrit.wikimedia.org/r/226059 (owner: 10Mjbmr) [00:20:00] (03PS1) 10JanZerebecki: [WIP] add DOC_SUBPATH parameter [integration/config] - 10https://gerrit.wikimedia.org/r/226452 [00:35:05] (03PS1) 10JanZerebecki: Move job into correct file. [integration/config] - 10https://gerrit.wikimedia.org/r/226454 [00:41:10] !log clean up doc dir after job changes gallium:~$ sudo -iu jenkins-slave rm -r /srv/org/wikimedia/doc/MobileFrontend/master/{app-0c945a27f43452df695771ddb60b3d14.js,data-500abda2bcb0df13609e38707dfa7f4e.js,eg-iframe.html,extjs,favicon.ico,index.html,member-icons,output,resources,source,styles-3eba09980fa05ead185cb17d9c0deb0f.css} [00:41:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:44:13] (03CR) 10JanZerebecki: [C: 032] Move job into correct file. [integration/config] - 10https://gerrit.wikimedia.org/r/226454 (owner: 10JanZerebecki) [00:46:38] (03Merged) 10jenkins-bot: Move job into correct file. [integration/config] - 10https://gerrit.wikimedia.org/r/226454 (owner: 10JanZerebecki) [01:09:00] !log beta-mediawiki-config-update-eqiad jobs stuck [01:09:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:43:51] Yippee, build fixed! [01:43:51] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #322: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/322/ [01:48:29] legoktm: are you going to unstuck them? [01:48:44] no, I don't know how [01:50:43] !log trying https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update [01:50:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:57:31] !log reconnected slave and needed to kill a few pending beta jobs, works again [01:57:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:10:56] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10JanZerebecki) [02:13:38] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1473884 (10JanZerebecki) [02:17:16] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1473890 (10JanZerebecki) How about running them fully on Wikimedia CI infrastructure as long as they don't work reliably on SauceLabs? [02:19:25] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1473891 (10JanZerebecki) >>! In T92619#1381725, @WMDE-Fisch wrote: > The issues seem fixed for Chrome. Firefox still fails with some of our t... [03:42:09] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n: l10nupdate fails since 2015-07-14 - https://phabricator.wikimedia.org/T106460#1473917 (10greg) From tonight: ``` 02:03 logmsgbot: LocalisationUpdate failed (1.26wmf14) at 2015-07-23 02:03:02+00:00 02:03 logmsgbot: LocalisationUpdate fa... [03:45:38] 10Deployment-Systems, 6Release-Engineering: Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#1473918 (10greg) [03:52:32] 6Release-Engineering, 6Analytics-Backlog, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1473921 (10greg) Sorted differently (branch order): ``` 1.26wmf1 3 1.26wmf4 8 1.26wmf5 1 1.26wmf7 5 1.26wmf8 1 1.26wmf10 2 1... [04:38:24] (03CR) 10Florianschmidtwelzow: "Oops, thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/226454 (owner: 10JanZerebecki) [05:28:10] (03PS1) 10Florianschmidtwelzow: Separate MobileFrontend doc links into JS and PHP [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) [05:45:39] 6Release-Engineering, 6Analytics-Backlog, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1473986 (10mmodell) I think it's clear that we need to abandon the practice of branching & changing URL prefixes each week. [05:47:02] 6Release-Engineering, 6Analytics-Backlog, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1473987 (10mmodell) One naive solution would be to replace old branches with symlinks to a current branch. This would mostly s... [06:20:12] 10Deployment-Systems, 6Release-Engineering, 5Patch-For-Review, 5WMF-deploy-2015-07-14_(1.26wmf14): No auto-submodule updates to 1.26wmf14 - https://phabricator.wikimedia.org/T105994#1474035 (10mmodell) 5Open>3Resolved [06:23:16] (03CR) 10Legoktm: [C: 04-1] Separate MobileFrontend doc links into JS and PHP (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:30:03] (03CR) 10Florianschmidtwelzow: [C: 04-1] Separate MobileFrontend doc links into JS and PHP (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:30:44] hahah oops [06:30:57] (03CR) 10Legoktm: "oops, that's my fault :P" [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:32:17] (03PS2) 10Florianschmidtwelzow: Separate MobileFrontend doc links into JS and PHP [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) [06:33:02] (03CR) 10Legoktm: [C: 032] "Thanks" [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:33:04] (03CR) 10Florianschmidtwelzow: "fixed :D" [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:33:14] (03Merged) 10jenkins-bot: Separate MobileFrontend doc links into JS and PHP [integration/docroot] - 10https://gerrit.wikimedia.org/r/226479 (https://phabricator.wikimedia.org/T105134) (owner: 10Florianschmidtwelzow) [06:43:29] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [06:46:29] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1474061 (10mmodell) >>! In T99096#1458885, @Krinkle wrote: > If that seemed possible I would've done... [06:51:19] 10Deployment-Systems, 6operations, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1474062 (10Joe) No, this specific type of locking has gone away with HHVM 3.6.x; Closing the ticket. [06:51:26] 10Deployment-Systems, 6operations, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1474063 (10Joe) 5Open>3Resolved [06:53:29] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:41:02] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1474126 (10Legoktm) @mmodell how hard do you think this will be to implement? I'm trying to figure out whether it's worth adding support to {T99084} for non-semver "1.26wmf1... [07:42:55] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1474129 (10WMDE-Fisch) >>! In T92619#1473891, @JanZerebecki wrote: >>>! In T92619#1381725, @WMDE-Fisch wrote: >> The issues seem fixed for Ch... [07:43:32] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1474130 (10mmodell) @legoktm: The only tooling related issue I can think of is the "php-" prefix is assumed in several places. Everything after the php- is pretty much arbit... [07:44:56] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1474131 (10Legoktm) >>! In T67306#1474130, @mmodell wrote: > Maybe we should try using a semantic version for next week's branch and see if anything breaks? Sounds like a g... [07:51:01] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1474133 (10WMDE-Fisch) As mentioned in https://phabricator.wikimedia.org/T95291 Firefox still resizes randomly but it does not break our t... [08:03:30] (03CR) 10Hashar: "Jan as a similar need for the Wikidata repos, that is why I added him as a reviewer." [integration/config] - 10https://gerrit.wikimedia.org/r/221310 (owner: 10Awight) [08:07:51] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474139 (10hashar) 3NEW [08:18:30] !log creating deployment-puppetmaster m1.medium :D [08:18:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:26:13] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474158 (10hashar) Created deployment-puppetmaster m1.medium https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-puppetmaster.deployment-prep.eqiad.wmflabs [08:34:34] PROBLEM - Puppet failure on deployment-puppetmaster is CRITICAL 30.00% of data above the critical threshold [0.0] [08:39:35] !log Disabling puppet agent on ALL beta cluster instances [08:39:36] RECOVERY - Puppet failure on deployment-puppetmaster is OK Less than 1.00% above the threshold [0.0] [08:39:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:42:10] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474187 (10hashar) Disabled puppet agent on all beta cluster instances. Changed puppet master in wikitech hiera https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=171624... [08:52:07] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474206 (10hashar) Reapplying the cherry-picked changes: https://gerrit.wikimedia.org/r/#q,Ib206f3820e5aa11ff6d26e777609b5692f74dd4f,n,z https://gerrit.wikimedia.org/r/#q,I5c0a1fd283cc545ab073a5... [09:07:46] 10Browser-Tests, 10VisualEditor, 10Wikimania-Hackathon-2015, 5Patch-For-Review, 5WMF-deploy-2015-07-21_(1.26wmf15): add true internationalized content support to auto-screenshots - https://phabricator.wikimedia.org/T105466#1474255 (10Aklapper) What is the status of this task, now that Wikimania 2015 is o... [09:07:50] PROBLEM - Puppet failure on mira is CRITICAL 50.00% of data above the critical threshold [0.0] [09:08:09] 6Release-Engineering, 10Wikimania-Hackathon-2015: Sprint: Wikimedia Site Requests triage/cleanup/process - https://phabricator.wikimedia.org/T90468#1474270 (10Aklapper) What is the status of this task, now that Wikimania 2015 is over? Did this hacking project take place and was successfully finished? If yes: P... [09:19:31] (03CR) 10Nemo bis: "That's fixed and the extension is being used by several wikis now. See e.g. http://strategywiki.org/wiki/Special:PopularPages" [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) (owner: 10Nemo bis) [09:22:26] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [09:24:42] PROBLEM - Puppet failure on deployment-salt is CRITICAL 60.00% of data above the critical threshold [0.0] [09:25:12] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 33.33% of data above the critical threshold [0.0] [09:25:40] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 20.00% of data above the critical threshold [0.0] [09:25:47] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 20.00% of data above the critical threshold [0.0] [09:26:09] PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL 66.67% of data above the critical threshold [0.0] [09:26:53] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 30.00% of data above the critical threshold [0.0] [09:29:41] mobrovac: puppet fails on beta cluster for deployment-restbase01 and deployment-restbase02 :( [09:30:00] some salt calls fail [09:30:01] * mobrovac sighs [09:30:08] KeyError: 'cassandra/metrics-collector' [09:30:08] :D [09:30:16] seems to be missing some config [09:30:16] ah [09:30:35] PROBLEM - Puppet failure on deployment-puppetmaster is CRITICAL 10.00% of data above the critical threshold [0.0] [09:30:37] it's probably cherry-picked or sth [09:30:48] mobrovac: note I am currently moving all instances to use deployment-puppetmaster as the puppetmaster [09:30:58] I cherry picked the patches from the old puppetmaster (deployment-salt) [09:31:00] ah ok [09:31:09] i'll let you finish and then take a look [09:31:41] I am just mass changing instances, so feel free to look at it right now :) [09:32:54] !log puppet broken on deployment-fluorine : Error: Could not request certificate: Neither PUB key nor PRIV key:: header too long [09:33:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:33:51] 10Beta-Cluster: puppet run broken on deployment-fluorine - https://phabricator.wikimedia.org/T106655#1474324 (10hashar) 3NEW [09:35:12] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [09:35:35] RECOVERY - Puppet failure on deployment-puppetmaster is OK Less than 1.00% above the threshold [0.0] [09:36:25] mobrovac: filled and CCed you: https://phabricator.wikimedia.org/T106657 [09:36:43] 10Beta-Cluster, 10RESTBase: puppet broken on deployment-restbase01 and deployment-restbase02 - https://phabricator.wikimedia.org/T106657#1474343 (10hashar) [09:36:50] kk thnx hashar [09:37:49] RECOVERY - Puppet staleness on deployment-restbase01 is OK Less than 1.00% above the threshold [3600.0] [09:38:43] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474365 (10hashar) Migrating all instances using: ``` sudo -s find /var/lib/puppet -type d -name ssl -exec rm -fR {} \; find /var/lib/puppet -type d -name ssl -exec rm -fR {} \; puppet agent --en... [09:41:54] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [09:43:18] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 22.22% of data above the critical threshold [0.0] [09:44:06] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 22.22% of data above the critical threshold [0.0] [09:44:42] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [09:44:46] 10Beta-Cluster: Can not ssh to beta cluster instance deployment-apertium01 - https://phabricator.wikimedia.org/T106658#1474417 (10hashar) 3NEW [09:45:34] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 50.00% of data above the critical threshold [0.0] [09:45:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [09:45:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [09:47:21] 10Beta-Cluster, 6operations: puppet fail on deployment-mx - https://phabricator.wikimedia.org/T106660#1474435 (10hashar) 3NEW [09:48:08] PROBLEM - Puppet failure on deployment-test is CRITICAL 33.33% of data above the critical threshold [0.0] [09:49:56] hashar: the deployment host is still deployment-bastion ? [09:53:21] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [09:54:04] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [09:56:11] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [09:56:34] 10Beta-Cluster, 10RESTBase: puppet broken on deployment-restbase01 and deployment-restbase02 - https://phabricator.wikimedia.org/T106657#1474462 (10mobrovac) The `cassandra/metrics-collector` repo is missing on `deployment-bastion`. I guess once You migrate the puppet master @hashar we'll be able to force a pu... [09:58:11] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [10:10:55] (03PS2) 1020after4: Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) [10:11:21] (03CR) 10jenkins-bot: [V: 04-1] Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [10:15:30] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [10:20:29] (03PS3) 1020after4: Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) [10:21:27] (03CR) 1020after4: "Revised to significantly improve the error messages." [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [10:23:34] !log running apt-get upgrade on deployment-parsoidcache02 [10:23:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:30:21] !log regenerated puppet cert on deployment-salt , the old puppetmaster now a puppet client [10:30:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:31:33] 10Beta-Cluster, 10Parsoid, 7Varnish: deployment-parsoidcache02 fails varnish VCL compilation - https://phabricator.wikimedia.org/T106662#1474493 (10hashar) 3NEW [10:37:44] 10Beta-Cluster: puppet fails on deployment-bastion attempts to access puppet:///private/ssh/tin/mwdeploy_rsa - https://phabricator.wikimedia.org/T106663#1474507 (10hashar) 3NEW [10:40:11] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 88.89% of data above the critical threshold [0.0] [10:40:53] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474514 (10hashar) 5Open>3Resolved a:3hashar All instances have been switched: ``` root@deployment-salt:~# salt '*' cmd.run 'grep "server =" /etc/puppet/puppet.conf' deployment-fluorine.dep... [10:43:36] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474528 (10hashar) 5Resolved>3Open Forgot to migrate the labs/private.git local hacks :-( [10:49:31] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1474548 (10fgiunchedi) update on this, I was talking to @MoritzMuehlenhoff and since ffmpeg will b... [10:51:44] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474551 (10hashar) Applied on labs/private: 0001-LOCALHACK-Keys-for-mwdeploy-user.patch 0002-Hack-touch-releases-id_rsa.upload.patch [10:52:41] !log Beta cluster puppetmaster is now deployment-puppetmaster.deployment-prep.eqiad.wmflabs . Migrated all instances (solves https://phabricator.wikimedia.org/T106649 ) [10:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:54:12] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474558 (10hashar) [10:54:14] 10Beta-Cluster: puppet fails on deployment-bastion attempts to access puppet:///private/ssh/tin/mwdeploy_rsa - https://phabricator.wikimedia.org/T106663#1474555 (10hashar) 5Open>3Resolved a:3hashar I forgot to migrate the labs/private.git puppet patches when migrating to the new puppet master. Applied:... [10:58:34] !log beta : salt '*' cmd.run 'rm /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist' [10:58:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:01:02] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1474576 (10hashar) 5Open>3Resolved Cleaned puppetmaster self config from deployment-salt https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-salt.deployment-prep.eqiad.w... [11:04:08] done [11:04:10] and lunch [11:05:11] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [11:27:50] RECOVERY - Puppet failure on mira is OK Less than 1.00% above the threshold [0.0] [11:59:29] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [12:07:58] 10Continuous-Integration-Infrastructure, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review, 7Zuul: gerrit review --label verified=1 causes a SQL error - https://phabricator.wikimedia.org/T106596#1474670 (10hashar) select * from patch_set_approvals where change_id=226272 ; |-------|---------------------|--------... [12:17:37] 10Continuous-Integration-Infrastructure, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review, 7Zuul: gerrit review --label verified=1 causes a SQL error - https://phabricator.wikimedia.org/T106596#1474705 (10hashar) The case insensitive collation was introduced by https://gerrit.wikimedia.org/r/#/c/6439/1/templat... [12:29:32] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1474740 (10mmodell) >>! In T67306#1474131, @Legoktm wrote: > Sounds like a good idea to me :) Will this require any changes to make-wmf-branch? Maybe the command line argum... [12:54:54] Yippee, build fixed! [12:54:54] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #542: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/542/ [13:12:16] 10Browser-Tests, 10VisualEditor, 10Wikimania-Hackathon-2015, 5WMF-deploy-2015-07-21_(1.26wmf15): add true internationalized content support to auto-screenshots - https://phabricator.wikimedia.org/T105466#1474863 (10Amire80) 5Open>3Resolved [13:13:00] 10Browser-Tests, 10VisualEditor, 10Wikimania-Hackathon-2015, 5WMF-deploy-2015-07-21_(1.26wmf15): add true internationalized content support to auto-screenshots - https://phabricator.wikimedia.org/T105466#1444372 (10Amire80) Basic support is done by Vikas, further work will be done in the coming weeks. Thanks! [13:23:42] !log integration puppetmaster in bad shape: Warning: Error 400 on SERVER: Cannot allocate memory - fork(2) [13:23:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:24:55] !log apt-get upgrade integration-puppetmaster and rebooting it [13:25:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:30:18] PROBLEM - Puppet failure on integration-puppetmaster is CRITICAL 20.00% of data above the critical threshold [0.0] [13:40:16] RECOVERY - Puppet failure on integration-puppetmaster is OK Less than 1.00% above the threshold [0.0] [13:43:14] PROBLEM - Puppet failure on integration-slave-trusty-1015 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:46:00] (03CR) 10JanZerebecki: "See jjb/wikidata.yaml to see how that works. The plan is to generalize those so they can be applied to all extensions." [integration/config] - 10https://gerrit.wikimedia.org/r/221310 (owner: 10Awight) [14:05:35] (03CR) 10Hashar: [C: 032 V: 032] "Merge to gbp-pq export the patch in the debian/precise-wikimedia branch" [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/226415 (https://phabricator.wikimedia.org/T106531) (owner: 10Hashar) [14:07:56] RECOVERY - Puppet failure on integration-slave-precise-1013 is OK Less than 1.00% above the threshold [0.0] [14:21:26] (03PS1) 10Hashar: Fix gerrit review case mismatch errors with labels [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/226532 (https://phabricator.wikimedia.org/T106531) [14:23:14] RECOVERY - Puppet failure on integration-slave-trusty-1015 is OK Less than 1.00% above the threshold [0.0] [14:26:23] (03PS2) 10Hashar: Fix gerrit review case mismatch errors with labels [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/226532 (https://phabricator.wikimedia.org/T106531) [14:45:25] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1475128 (10JanZerebecki) [14:49:40] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1475140 (10JanZerebecki) [14:49:49] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10JanZerebecki) [14:50:46] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10JanZerebecki) [15:04:47] (03PS1) 10Paladox: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 [15:05:16] (03PS2) 10Paladox: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 [15:05:35] (03PS3) 10Paladox: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 [15:06:42] (03PS4) 10Paladox: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 [15:18:15] (03CR) 10Hashar: [C: 032 V: 032] "Created at https://people.wikimedia.org/~hashar/debs/zuul_2.0.0-327-g3ebedde-wmf3precise1/" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/226532 (https://phabricator.wikimedia.org/T106531) (owner: 10Hashar) [15:20:22] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Zuul: Changes voted CR+2 do not enter gate-and-submit unless Verified is +2. - https://phabricator.wikimedia.org/T106531#1475260 (10hashar) The new package is at https://people.wikimedia.org/~hashar/debs/zuul_2.0.0-327-g3ebedde-wmf3precise1/ I hav... [15:20:55] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1475266 (10Eevans) [15:20:58] 10Beta-Cluster, 10RESTBase: puppet broken on deployment-restbase01 and deployment-restbase02 - https://phabricator.wikimedia.org/T106657#1475263 (10Eevans) 5Open>3Resolved a:3Eevans I added the deploy repo to: https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep (https://wikitech.wikimedia.org/w/ind... [15:21:34] thcipriani: I took some notes regarding Zuul packaging at https://www.mediawiki.org/wiki/Continuous_integration/Zuul#new_package [15:21:44] but it misses a lot of bits still :-( [15:23:28] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:29:15] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [15:38:43] 10Beta-Cluster, 10RESTBase: puppet broken on deployment-restbase01 and deployment-restbase02 - https://phabricator.wikimedia.org/T106657#1475360 (10hashar) Thank you ! [15:39:57] !log applied wip logstash & cassandra changes (https://gerrit.wikimedia.org/r/#/c/226025/) to deployment-prep [15:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:41:06] hashar: I think this test failure was a fluke and I'm trying to SWAT: https://gerrit.wikimedia.org/r/#/c/226541/ how can I make it recheck? [15:45:06] 10Continuous-Integration-Infrastructure, 7HHVM: Upgrade HHVM related packages on Trusty Jenkins slaves - https://phabricator.wikimedia.org/T106699#1475388 (10hashar) 3NEW [15:46:13] thcipriani: commenting "recheck" should do it [15:46:33] hashar: yeah gave that a shot https://gerrit.wikimedia.org/r/#/c/226541/ [15:46:56] thcipriani: at least it did for me :-}} [15:47:40] it is in https://integration.wikimedia.org/zuul/ [15:47:58] hashar: I noticed that, thanks [15:48:10] I have no idea why your recheck did not work though [15:48:19] OHH [15:48:20] yeah [15:48:21] of course [15:48:28] you changed the verified note [15:48:36] but zuul matches the comment message using a regex [15:48:43] that does not take in account someone changing a vote as well [15:48:54] your message has an extra " -Verified" [15:49:02] which cause the regex to fail matching [15:49:09] thcipriani: ^^ [15:49:42] hashar: heh, regexes: the bane of my existence :) [15:52:16] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Migrate all debian-glue jobs to Jessie slaves - https://phabricator.wikimedia.org/T95545#1475418 (10hashar) [15:52:19] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Update jenkins-debian-glue packages on Jessie to v0.13.0 - https://phabricator.wikimedia.org/T102106#1475416 (10hashar) 5Open>3Resolved Package has been uploaded on apt.wikimedia.org: ``` apt-cache policy jenkins-debian-glue je... [15:52:49] thcipriani: :-} [15:53:10] thcipriani: ask aaron schulz about regex backtracking and unicode properties. That is usually interesting talks [15:53:18] meanwhile, off for picnic with kids [15:53:25] hashar: have fun! [15:53:37] and thanks for your help! [15:54:25] :-) [16:01:28] 6Release-Engineering, 10Wikimania-Hackathon-2015: Sprint: Wikimedia Site Requests triage/cleanup/process - https://phabricator.wikimedia.org/T90468#1475456 (10Krenair) >>! In T90468#1474270, @Aklapper wrote: > What is the status of this task, now that Wikimania 2015 is over? Still going. I managed to look thr... [16:18:35] 10Continuous-Integration-Infrastructure, 5MW-1.26-release, 5Patch-For-Review: Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches - https://phabricator.wikimedia.org/T90303#1475517 (10JanZerebecki) [16:22:07] 10Continuous-Integration-Infrastructure, 6Release-Engineering: automatically build and commit mediawiki/vendor (composer) - https://phabricator.wikimedia.org/T101123#1475531 (10JanZerebecki) [16:29:53] (03PS1) 10Zfilipin: Run RuboCop in test and gate-and-submit pipelines. [integration/config] - 10https://gerrit.wikimedia.org/r/226551 (https://phabricator.wikimedia.org/T106220) [16:33:21] (03PS2) 10Dduvall: Run RuboCop for MW-Vagrant on test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/226551 (https://phabricator.wikimedia.org/T106220) (owner: 10Zfilipin) [16:35:29] (03CR) 10Dduvall: [C: 032] Run RuboCop for MW-Vagrant on test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/226551 (https://phabricator.wikimedia.org/T106220) (owner: 10Zfilipin) [16:37:10] (03Merged) 10jenkins-bot: Run RuboCop for MW-Vagrant on test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/226551 (https://phabricator.wikimedia.org/T106220) (owner: 10Zfilipin) [16:38:13] !log Reloading Zuul to deploy I96b6218a208f133209452c71bcf01a1088305aea [16:38:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:16:46] Differential beta currently does not support running any tests at all, right? [17:35:09] twentyafterfour: what's the status of https://phabricator.wikimedia.org/T616? I haven't seen ^d around to annoy him about this. (I want to see my extension in Diffusion! :P) [17:53:31] polybuildr: which repo? [17:53:44] twentyafterfour: mediawiki/extensions/SmiteSpam [17:53:54] It's on the list in the task too. [17:56:32] polybuildr: you haven't seen ^d because he's known as ostriches now ;) [17:56:40] * twentyafterfour just outed him [17:56:46] :O [17:57:07] * polybuildr /whois ostriches [17:57:13] [ostriches] (demon@mediawiki/demon): Chad H. [17:57:25] How did this happen? [17:57:28] This is black magic. [17:58:22] Did something unleash his inner demon? [17:58:28] And then it turned out to be an ostrich? [18:03:04] heh I don't know [18:07:47] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL deployment-prep.deployment-fluorine.diskspace.root.byte_percentfree (<100.00%) [18:07:49] twentyafterfour: Also, the repository? Will the phab gods be kind and migrate my repo? :P [18:09:31] polybuildr: I have to deploy mediawiki right now. I might be able to look into it in a few mintues [18:09:45] Oh oh. No hurry. Thanks! :D [18:11:50] (03PS1) 10Florianschmidtwelzow: Add composer test to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/226587 [18:11:53] 10Beta-Cluster, 10Continuous-Integration-Config, 5Patch-For-Review: beta-update-databases-eqiad should get list of DB to update from mediawiki-config all-labs.dblist - https://phabricator.wikimedia.org/T98342#1476045 (10fgiunchedi) and now for a random drive-by comment: looking at the code review, it seems t... [18:14:05] (03CR) 10BryanDavis: "Looks pretty good by eyeball. Somehow I messed up my testing VM so that sync-wikiversions fails with and without this patch. :/" (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [18:23:47] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1476090 (10Krinkle) >>! In T99096#1474061, @mmodell wrote: >Can someone enlighten me about how the st... [18:26:53] 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted, 6operations: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#1476096 (10Mattflaschen) 5Open>3Resolved a:3Mattflaschen It hasn't recently, but I don't know that it's fixed either. I'll m... [18:27:00] 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted, 6operations: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#1476101 (10Mattflaschen) a:5Mattflaschen>3None [18:41:32] (03CR) 10Legoktm: "Code looks good, just nit-picky style issues." (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/224648 (https://phabricator.wikimedia.org/T101872) (owner: 10Galorefitz) [18:46:11] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1476167 (10mmodell) >>! In T99096#1476090, @Krinkle wrote: > One thing to keep in mind as well is tha... [19:20:15] (03PS1) 10Dduvall: Let MW-Selenium 1.x use the local test suite configuration [integration/config] - 10https://gerrit.wikimedia.org/r/226603 [19:26:23] mmm, zuul stuck? [19:28:17] I'm seeing stuff move on https://integration.wikimedia.org/zuul/ [19:30:37] PROBLEM - Free space - all mounts on integration-slave-trusty-1015 is CRITICAL integration.integration-slave-trusty-1015.diskspace._mnt.byte_percentfree (<10.00%) [19:32:48] (03CR) 10Sbisson: [C: 031] Let MW-Selenium 1.x use the local test suite configuration [integration/config] - 10https://gerrit.wikimedia.org/r/226603 (owner: 10Dduvall) [19:33:00] yeah it looks better now [20:10:33] PROBLEM - Free space - all mounts on integration-slave-trusty-1015 is CRITICAL integration.integration-slave-trusty-1015.diskspace._mnt.byte_percentfree (<10.00%) [20:23:49] PROBLEM - Puppet staleness on deployment-apertium01 is CRITICAL 50.00% of data above the critical threshold [43200.0] [20:27:25] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1476503 (10mmodell) p:5Lowest>3High [20:27:36] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1476504 (10mmodell) a:3mmodell [20:28:35] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #205: FAILURE in 5.5 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/205/ [20:30:54] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n: l10nupdate fails since 2015-07-14 - https://phabricator.wikimedia.org/T106460#1476511 (10mmodell) I'm not sure how to tell. It is difficult to debug issues with LocalisationUpdate [20:31:43] PROBLEM - Puppet staleness on deployment-fluorine is CRITICAL 30.00% of data above the critical threshold [43200.0] [20:45:12] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 25.00% of data above the critical threshold [0.0] [21:10:16] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [21:12:25] (03PS1) 10Paladox: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 [21:12:48] (03PS2) 10Paladox: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 [21:13:18] (03PS3) 10Paladox: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 [21:14:41] I'd like to deploy a CentralNotice configuration change to betawiki. Looks like there are no toes to step on at the moment? [21:18:26] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:19:46] hashar: i'm seeing a lot of qunit jobs failing [21:19:54] oh [21:19:56] seemingly falsely [21:20:14] if they fail on the same test [21:20:25] some code is breaking / has a race condition :( [21:20:34] PROBLEM - Free space - all mounts on integration-slave-trusty-1015 is CRITICAL integration.integration-slave-trusty-1015.diskspace._mnt.byte_percentfree (<20.00%) [21:20:53] marxarelli: the last i have seen was comparing two timestamps that could potentially be generated at different seconds [21:20:54] hashar: https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/5036/consoleFull [21:21:27] i can't tell what the real failure is. it seems like maybe just a warning but `grunt karma:main` is returning non-zero [21:21:46] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/buildTimeTrend :( [21:22:11] that jobs clone several extensions, so there is a lot of potential candidates :( [21:22:44] liuggio/statsd-php-client: 1.0.16 installed, 1.0.12 required. [21:23:32] seems to be a dependency conflict between mw/core and mw/vendor :( [21:25:31] marxarelli: ah I see what you mean [21:25:48] seems chromium dies at some point [21:26:01] "WARN [Chromium 43.0.2357 (Ubuntu 0.0.0)]: Disconnected (1 times), because no message in 60000 ms." [21:26:02] yeah [21:26:17] PROBLEM - Puppet failure on deployment-cache-text03 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:26:18] maybe it is a race condition when multiple qunit jobs run on the same instance [21:26:36] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1476813 (10JanZerebecki) [21:26:50] I wish we had log of XVFB usage [21:27:33] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1116439 (10JanZerebecki) [21:27:56] 10Browser-Tests, 10Wikidata, 7Tracking: Wikidata Browsertests (tracking) - https://phabricator.wikimedia.org/T88541#1014338 (10JanZerebecki) [21:29:00] marxarelli: what did you do to gather browser tests?! ;) [21:29:05] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1476823 (10JanZerebecki) [21:29:39] jdlrobson: heh. sorry :) [21:29:44] jdlrobson: needs this https://gerrit.wikimedia.org/r/#/c/226626/1 [21:29:47] they were soooo green! [21:29:54] magic! [21:30:09] marxarelli: be careful with that.. [21:30:18] i seem to remember some MobileFrontend tests need to run on desktop and mobile [21:30:23] but the gersh dern failing qunit job is holding it back [21:30:35] i'm not sure if this interferes with that [21:30:39] e.g. switching to desktop [21:30:47] (gather shouldn't have this problem but worth considering) [21:30:59] jdlrobson: en.wikipedia.beta.wmflabs.org doesn't seem to work with the mf_useformat cookie [21:31:31] jdlrobson: why would mobilefrontend test the desktop site? [21:32:41] jdlrobson: anyway, it will be no different than how it was running before [21:32:50] looking at it now we don't seem to any more, but previously when there was no api we accessed workflows there. Hypothetically we might want a switch to desktop test. [21:33:14] since the jjb job was setting MEDIAWIKI_URL=http://en.m.wikipedia... [21:33:26] i just wanted to point out that as a flag, no concrete concerns form me right now :) [21:33:33] cool, cool [21:34:32] hashar: is there a difference between mwext-qunit and mediawiki-extensions-qunit? [21:34:45] both are running for Gather it seems [21:34:46] apparently Xvfb doest not die :-) [21:34:54] yup [21:35:07] mwext-qunit only has Gather + its extensions dependencies [21:35:23] the dependencies should be injected by Zuul itself via a python function somewhere in integration/config.git:/zuul/ [21:35:37] mediawiki-extensions-qunit clones a bunch of extensions and is triggered by all of them [21:35:44] so that is slightly different [21:35:51] i see [21:36:06] the idea behind mediawiki-extensions-* jobs is to make sure the extensions participating works well together [21:36:44] one day I will grab the list of such extensions from mediawiki/tools/release to have us test all extensions deployed on WMF cluster [21:37:17] anyway for chromium being disconnected, I Cant find anything obvious [21:37:24] I guess you will have to recheck / +2 again [21:37:49] for now I am heading to bed. Had a though evening [21:38:00] it seems like mediawiki-extensions-qunit should only run a subset of tests [21:38:06] integration tests, not unit tests [21:38:33] yeah [21:38:39] since unit tests shouldn't be interoperating with any other extension (even dependencies) [21:38:47] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 20.00% of data above the critical threshold [43200.0] [21:38:48] if we had a split between unit and integration :-/ [21:39:10] for PHP, I would like to eventually have jobs that just run phpunit [21:39:14] with no dependency added [21:39:16] something to talk about on Monday :) [21:39:21] yup [21:39:24] ow! that hurts! [21:39:37] jdlrobson: https://gerrit.wikimedia.org/r/#/c/226626/ [21:40:00] jdlrobson: should fix the problem [21:40:47] marxarelli: you can talk to Krinkle_ about splitting qunit tests in unit/integration tests :D [21:41:05] the thing is the mediawiki PHP backend would probably still need the dependencies anyway [21:42:33] +++ [21:43:33] (03PS2) 10Dduvall: Let MW-Selenium 1.x use the local test suite configuration [integration/config] - 10https://gerrit.wikimedia.org/r/226603 [21:48:56] greg-g: wait, was that a joke? [21:49:00] :D [21:49:03] marxarelli: jerk [21:49:13] IS TESTING A JOKE TO YOU?! [21:49:28] life is a joke man [21:49:58] it's like the Man's joke, man [21:50:11] there are overpaid hipsters running around in capes outside, it's so much a joke [21:50:23] haha [21:50:27] what's going on over there? [21:50:42] https://twitter.com/g_gerg/status/624308312733921280 [21:50:44] see replies [21:50:56] I don't know, but something yelp and costumes and buses [21:53:06] thor is kind of a bad winker [21:53:17] i expect more from a superhero [22:01:27] (03PS4) 1020after4: Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) [22:03:07] (03CR) 1020after4: "ok now it passes the 3rd argument" [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [22:03:49] (03CR) 1020after4: Check for l10n cache before sync-wikiversions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [22:09:38] 6Release-Engineering: Review team ownership of projects/things listed on the Developers/Maintainers page - https://phabricator.wikimedia.org/T106751#1476996 (10greg) 3NEW a:3greg [22:10:12] qunit is frequently timing out [22:10:14] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/5055/console [22:14:04] legoktm: was just chatting with hashar about that [22:14:20] seems like chromium kicks the bucket randomly [22:28:26] 10Deployment-Systems, 6Release-Engineering, 6Language-Engineering: Spot check on-wiki messages against prior day's translatewiki updates - https://phabricator.wikimedia.org/T93308#1477109 (10Nemo_bis) [22:29:24] 10Deployment-Systems, 6Release-Engineering, 6Language-Engineering: Spot check on-wiki messages against prior day's translatewiki updates - https://phabricator.wikimedia.org/T93308#1134012 (10Nemo_bis) > all of the messages that were merged prior to 03:00Z Not all of them. Only the changes to non-English mes... [22:37:22] marxarelli: https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/5061/console looks like it died there [22:37:26] test is still running though [22:37:31] oh, ignore me [22:37:35] I'm impatient :P [22:40:05] legoktm: it was being flakey earlier. perhaps it was due to a high number of jobs running all at once [22:40:11] possibly [22:40:19] we could lower the number of concurrent qunit jobs [22:40:30] but maybe it's just any job? [22:48:50] Yippee, build fixed! [22:48:50] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #207: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/207/ [22:49:20] 6Release-Engineering, 6Engineering-Community: Review team ownership of projects/things listed on the Developers/Maintainers page - https://phabricator.wikimedia.org/T106751#1477256 (10Aklapper) [22:50:07] 6Release-Engineering: Review team ownership of projects/things listed on the Developers/Maintainers page - https://phabricator.wikimedia.org/T106751#1477259 (10greg) [22:50:34] 6Release-Engineering: Review team ownership of projects/things listed on the Developers/Maintainers page - https://phabricator.wikimedia.org/T106751#1476996 (10greg) @aklapper: this is a task for me to do it for my team, not ECT ;) [22:59:55] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #689: FAILURE in 8 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/689/ [23:03:49] !log running `jenkins-jobs update config/ 'browsertests-*'` to deploy I2d0f83d0c6a406d46627578cb8db0706d1b8655d [23:03:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:09:43] (03PS3) 10Dduvall: Let MW-Selenium 1.x use the local test suite configuration [integration/config] - 10https://gerrit.wikimedia.org/r/226603 [23:15:42] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #557: FAILURE in 7 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/557/ [23:20:46] (03CR) 10Dduvall: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/226603 (owner: 10Dduvall) [23:23:13] (03Merged) 10jenkins-bot: Let MW-Selenium 1.x use the local test suite configuration [integration/config] - 10https://gerrit.wikimedia.org/r/226603 (owner: 10Dduvall) [23:28:11] marxarelli: you upset barry :/ [23:36:37] (03PS1) 10Dduvall: Adjust build/artifact retention for mw-selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/226650 (https://phabricator.wikimedia.org/T104583) [23:41:34] (03PS1) 10Dduvall: Configure video capture for headless mw-selenium jobs [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226651 (https://phabricator.wikimedia.org/T104583) [23:46:47] (03PS1) 10Dduvall: Release minor version 1.5.0 [selenium] - 10https://gerrit.wikimedia.org/r/226653 [23:47:32] (03PS2) 10Dduvall: Release minor version 1.5.0 [selenium] - 10https://gerrit.wikimedia.org/r/226653 [23:50:48] PROBLEM - Puppet failure on integration-slave-trusty-1021 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:53:54] PROBLEM - Puppet failure on integration-slave-trusty-1012 is CRITICAL 30.00% of data above the critical threshold [0.0] [23:54:03] I just noticed the Changed Files directory list on the left in Diffusion, eg: https://phabricator.wikimedia.org/rMSELa6d7bf6b23f6de2f3ee5f11bdb6c778a521a71c2 [23:54:08] PROBLEM - Puppet failure on integration-dev is CRITICAL 22.22% of data above the critical threshold [0.0] [23:54:16] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL 33.33% of data above the critical threshold [0.0] [23:54:17] pretty nice [23:54:57] PROBLEM - Puppet failure on integration-slave-trusty-1016 is CRITICAL 60.00% of data above the critical threshold [0.0] [23:56:00] poor puppet