[02:41:09] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:53:55] 10Continuous-Integration, 10VisualEditor, 3Editing Department 2014/15 Q4 blockers: Concurrent builds using local Chromium/Firefox browsers on Linux host fail - https://phabricator.wikimedia.org/T90673#1180484 (10Krinkle) [09:53:57] 10Continuous-Integration, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins jobs using MediaWiki frequently fail due to database locks - https://phabricator.wikimedia.org/T89180#1180485 (10Krinkle) [09:53:59] 10Continuous-Integration, 10VisualEditor, 5WMF-deploy-2015-03-25_(1.25wmf23): Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#1180483 (10Krinkle) [10:01:28] 10Continuous-Integration: gallium and lanthanum disks full (tracking) - https://phabricator.wikimedia.org/T91211#1180487 (10Krinkle) 5Open>3Resolved a:3Krinkle The on-going efforts on T86659 and T91396 have finally brought gallium and lanthanum to stable levels in terms of disk usage. {F108921} {F108922}... [10:04:57] 10Continuous-Integration, 10Wikidata: PHP fatal errors are not visible in jenkins output in mwext-Wikibase-client-tests job - https://phabricator.wikimedia.org/T92397#1180495 (10Krinkle) Is this still an issue? This might be related to T85799 / 0a35ec5d1f370cee4ebb9be0c386978d6cfcaabb. [10:20:54] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1180499 (10Krinkle) [10:21:06] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1175877 (10Krinkle) [10:22:27] !log Creating integration-slave-trusty-1001-1005 per T94916. [10:22:31] Logged the message, Master [10:33:46] 10Continuous-Integration, 5Patch-For-Review: Update puppet for packages having different names in Jessie - https://phabricator.wikimedia.org/T95000#1180504 (10Krinkle) > firefox -> iceweasel > chromium-browser -> chromium This also affects the global binaries from shell. We'll need individual projects to ensu... [10:39:46] (03CR) 10Krinkle: [C: 032] Publish oojs code coverage to integration.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [10:44:08] (03Merged) 10jenkins-bot: Publish oojs code coverage to integration.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/201694 (https://phabricator.wikimedia.org/T93558) (owner: 10Krinkle) [10:48:23] !log Puppet on integration-puppetmaster has been failing for the past 2 days: "Failed when searching for node i-0000063a.eqiad.wmflabs: You must set the 'external_nodes' parameter to use the external node terminus" (=integraton-dev.eqiad.wmflabs) [10:48:26] Logged the message, Master [11:06:31] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1180520 (10Krinkle) [11:06:32] 10Continuous-Integration: Downgrade intergration-puppetmaster back to Ubuntu Precise (re-create instance) - https://phabricator.wikimedia.org/T94927#1180518 (10Krinkle) 5Resolved>3Open Puppet on integration-puppetmaster has been failing for the past 2 days since April 3: > Failed when searching for node i-00... [11:12:49] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1180522 (10Krinkle) All the new instances are failing to provision with the assigned role from the new puppet master. ``` SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify... [11:13:30] !log New integratin-slave-trusty-1001-1005 must remain unpooled. Provisioning failed. details at https://phabricator.wikimedia.org/T94916#1180522 [11:13:34] Logged the message, Master [11:29:20] (03PS1) 10Krinkle: macro: Create npm-run-test and npm-run-doc [integration/config] - 10https://gerrit.wikimedia.org/r/201938 [11:44:32] (03PS2) 10Krinkle: macro: Create npm-run-test and npm-run-doc [integration/config] - 10https://gerrit.wikimedia.org/r/201938 [11:46:55] (03CR) 10Krinkle: [C: 032] macro: Create npm-run-test and npm-run-doc [integration/config] - 10https://gerrit.wikimedia.org/r/201938 (owner: 10Krinkle) [12:17:40] (03PS1) 10Krinkle: Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 [12:18:32] (03PS1) 10Krinkle: Publish UnicodeJS code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201946 [12:18:55] (03CR) 10jenkins-bot: [V: 04-1] Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 (owner: 10Krinkle) [12:19:49] (03CR) 10jenkins-bot: [V: 04-1] Publish UnicodeJS code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201946 (owner: 10Krinkle) [12:20:10] (03PS2) 10Krinkle: Publish UnicodeJS code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201946 [12:22:33] (03PS2) 10Krinkle: Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 [12:56:32] (03CR) 10Krinkle: [C: 032] "Deployed unicodejs-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/201946 (owner: 10Krinkle) [13:07:10] (03Merged) 10jenkins-bot: Publish UnicodeJS code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201946 (owner: 10Krinkle) [13:07:30] (03PS3) 10Krinkle: Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 [13:13:15] (03CR) 10Krinkle: [C: 032] "Deployed oojs-core-publish, oojs-ui-publish." [integration/config] - 10https://gerrit.wikimedia.org/r/201945 (owner: 10Krinkle) [13:17:30] (03CR) 10jenkins-bot: [V: 04-1] Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 (owner: 10Krinkle) [13:58:59] (03PS4) 10Krinkle: Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 [13:59:05] (03CR) 10Krinkle: [C: 032] Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 (owner: 10Krinkle) [13:59:55] (03PS1) 10Krinkle: Remove outdated zuul filter for postmerge jsduck-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/201949 [14:00:03] (03CR) 10Krinkle: [C: 032] Remove outdated zuul filter for postmerge jsduck-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/201949 (owner: 10Krinkle) [14:03:21] (03Merged) 10jenkins-bot: Publish OOjs UI code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/201945 (owner: 10Krinkle) [14:03:23] (03Merged) 10jenkins-bot: Remove outdated zuul filter for postmerge jsduck-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/201949 (owner: 10Krinkle) [15:16:59] (03PS1) 10MarkAHershberger: Update ignore with some emacs files. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201955 [15:17:01] (03PS1) 10MarkAHershberger: Start up php mess detector rule set [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201956 [15:20:07] (03CR) 10MarkAHershberger: [C: 032] "self merging gitignore" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201955 (owner: 10MarkAHershberger) [15:20:33] (03Merged) 10jenkins-bot: Update ignore with some emacs files. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201955 (owner: 10MarkAHershberger) [16:04:36] 10Beta-Cluster, 10Deployment-Systems, 6MediaWiki-Core-Team, 5Patch-For-Review: scap failed: IOError [Errno 13] Permission denied: '/srv/mediawiki-staging/wikiversions-labs.cdb.tmp' - https://phabricator.wikimedia.org/T94261#1180713 (10bd808) 5Open>3Resolved [16:15:56] 10Staging, 6operations: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1180735 (10Tgr) [16:38:12] PROBLEM - Citoid on deployment-sca01 is CRITICAL: Connection refused [17:35:40] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1180803 (10Krinkle) [17:43:25] 10Continuous-Integration: Can't view per-slave build history - https://phabricator.wikimedia.org/T95024#1180806 (10Krinkle) Note that while we reduced the number of unique jobs, the number of overall builds are more or less the same. The primary concern with viewing history is the sheer number of builds and how... [19:17:43] 10Deployment-Systems: Investigate and fix synchronization problem of MediaWiki:Extdist-created-extensions/zh-hant message between TWN and repository - https://phabricator.wikimedia.org/T95127#1180872 (10Ciencia_Al_Poder) 3NEW [19:33:10] 10Continuous-Integration: Zuul fails "Function build:integration-zuul-layoutdiff is not registered (negative ttl in effect)" - https://phabricator.wikimedia.org/T94657#1180888 (10Krinkle) p:5Triage>3Low [19:46:36] 10Continuous-Integration: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit in the short term - https://phabricator.wikimedia.org/T94322#1180900 (10Krinkle) Upstream has foreseen the same problem. They're working on a way to configure queues manually instead of the current automated queuing inf... [19:57:00] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180919 (10Krinkle) As long as the "number of failures" (e.g. errors/warnings, not the complete failures) are counted in Graphite and result in Shinken e-mails. Is that... [19:58:54] 10Continuous-Integration, 6Release-Engineering, 5Patch-For-Review: Zuul-cloner forgets to clear workspace - https://phabricator.wikimedia.org/T76304#1180921 (10Krinkle) Probably backport https://review.openstack.org/148121, or upgrade our Zuul to a newer upstream. [19:59:54] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180922 (10yuvipanda) They are counted in graphite, and present in the emails just not particularly clearly. [21:42:03] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1180998 (10Krinkle) 5Open>3Resolved a:3Krinkle [22:02:37] (03PS1) 10Krinkle: misc: Re-order definitions [integration/config] - 10https://gerrit.wikimedia.org/r/201974 [22:02:39] (03PS1) 10Krinkle: Create cdb-publish job (Doxygen) [integration/config] - 10https://gerrit.wikimedia.org/r/201975 (https://phabricator.wikimedia.org/T75530) [22:02:51] (03CR) 10Krinkle: [C: 032] "Deployed https://doc.wikimedia.org/cdb/master/." [integration/config] - 10https://gerrit.wikimedia.org/r/201975 (https://phabricator.wikimedia.org/T75530) (owner: 10Krinkle) [22:02:57] (03CR) 10Krinkle: [C: 032] misc: Re-order definitions [integration/config] - 10https://gerrit.wikimedia.org/r/201974 (owner: 10Krinkle) [22:07:48] (03Merged) 10jenkins-bot: misc: Re-order definitions [integration/config] - 10https://gerrit.wikimedia.org/r/201974 (owner: 10Krinkle) [22:07:49] (03Merged) 10jenkins-bot: Create cdb-publish job (Doxygen) [integration/config] - 10https://gerrit.wikimedia.org/r/201975 (https://phabricator.wikimedia.org/T75530) (owner: 10Krinkle) [22:07:58] (03PS1) 10Krinkle: Add CDB [integration/docroot] - 10https://gerrit.wikimedia.org/r/201976 (https://phabricator.wikimedia.org/T75530) [22:08:25] (03CR) 10Krinkle: [C: 032] Add CDB [integration/docroot] - 10https://gerrit.wikimedia.org/r/201976 (https://phabricator.wikimedia.org/T75530) (owner: 10Krinkle) [22:08:27] (03Merged) 10jenkins-bot: Add CDB [integration/docroot] - 10https://gerrit.wikimedia.org/r/201976 (https://phabricator.wikimedia.org/T75530) (owner: 10Krinkle) [22:08:48] 10Continuous-Integration, 5Patch-For-Review: Publish cdb documentation to doc.wikimedia.org - https://phabricator.wikimedia.org/T75530#1181019 (10Krinkle) 5Open>3Resolved https://doc.wikimedia.org/cdb/master/ [23:00:22] (03PS1) 10Krinkle: Separate jslint from mwext-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 [23:16:37] (03PS2) 10Krinkle: Separate jslint from mwext-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 [23:17:54] (03CR) 10jenkins-bot: [V: 04-1] Separate jslint from mwext-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 (owner: 10Krinkle) [23:19:41] (03PS3) 10Krinkle: Separate jslint from mwext-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 [23:24:05] (03PS4) 10Krinkle: Separate jslint from mwext-check-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 [23:24:13] (03CR) 10Krinkle: [C: 032] "No-op." [integration/config] - 10https://gerrit.wikimedia.org/r/201978 (owner: 10Krinkle) [23:28:33] (03Merged) 10jenkins-bot: Separate jslint from mwext-check-jobs [integration/config] - 10https://gerrit.wikimedia.org/r/201978 (owner: 10Krinkle) [23:33:51] 10Staging, 6operations: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1181077 (10ori) >>! In T90229#1054901, @faidon wrote: > It should be trivial to build for jessie and include into jessie-wikimedia, at least until jessie-backports comes to life. Done. [23:34:02] 10Staging, 5Patch-For-Review: Setup staging-palladium as puppetmaster and saltmaster - https://phabricator.wikimedia.org/T88304#1181079 (10ori) [23:34:03] 10Staging, 6operations: Package geoipupdate for jessie - https://phabricator.wikimedia.org/T90229#1181078 (10ori) 5Open>3Resolved [23:56:32] (03PS1) 10Krinkle: Delete old jshint jobs from repos with no *.js files (W-Z) [integration/config] - 10https://gerrit.wikimedia.org/r/201982 [23:56:34] (03CR) 10jenkins-bot: [V: 04-1] Delete old jshint jobs from repos with no *.js files (W-Z) [integration/config] - 10https://gerrit.wikimedia.org/r/201982 (owner: 10Krinkle) [23:56:49] (03PS2) 10Krinkle: Delete old jshint jobs from repos with no *.js files (W-Z) [integration/config] - 10https://gerrit.wikimedia.org/r/201982