[00:11:50] (03CR) 10BryanDavis: [C: 032] add deploy-promote script [tools/release] - 10https://gerrit.wikimedia.org/r/220292 (owner: 1020after4) [02:43:54] marxarelli|away: Nice work on the mediawiki-selenium upgrade guides. Very useful! [02:44:11] Krinkle: oh thanks [02:44:27] we run through it every time we do one, so it gets better and better :) [02:48:24] Krinkle: thanks for getting that generic MW setup for puppet/JJB in place [02:49:44] Krinkle: unfortunately i wasn't able to do any work on my patch today because of a family emergency but i should have time to rebase and deploy my patch tomorrow [02:50:00] Aye.. sure. No worries :) [03:32:20] (03PS2) 1020after4: Add a delete-stale-branch script. [tools/release] - 10https://gerrit.wikimedia.org/r/220500 [03:32:50] (03CR) 1020after4: [V: 032] Add a delete-stale-branch script. [tools/release] - 10https://gerrit.wikimedia.org/r/220500 (owner: 1020after4) [03:33:27] (03CR) 1020after4: [C: 032] add deploy-promote script [tools/release] - 10https://gerrit.wikimedia.org/r/220292 (owner: 1020after4) [03:33:40] (03Merged) 10jenkins-bot: add deploy-promote script [tools/release] - 10https://gerrit.wikimedia.org/r/220292 (owner: 1020after4) [03:38:57] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1403617 (10GWicke) [07:02:57] (03PS1) 10Nemo bis: Bundle HitCounters extension [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) [10:08:44] (03PS1) 10Hashar: Experimental tox jobs for operations/software/conftool [integration/config] - 10https://gerrit.wikimedia.org/r/221088 (https://phabricator.wikimedia.org/T103972) [10:11:22] (03CR) 10Hashar: [C: 032] "That will let us trigger the jobs manually by commenting 'check experimental' in Gerrit. Eg on https://gerrit.wikimedia.org/r/#/c/221087/" [integration/config] - 10https://gerrit.wikimedia.org/r/221088 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [10:13:17] (03Merged) 10jenkins-bot: Experimental tox jobs for operations/software/conftool [integration/config] - 10https://gerrit.wikimedia.org/r/221088 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar) [10:34:25] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#1404223 (10hashar) [10:35:45] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Install etcd on Jessie CI slaves - https://phabricator.wikimedia.org/T103976#1404229 (10hashar) 3NEW [10:35:57] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Install etcd on Jessie CI slaves - https://phabricator.wikimedia.org/T103976#1404229 (10hashar) [10:38:01] lunchhh [12:01:12] 10Beta-Cluster, 10Mathoid, 7Monitoring, 5Patch-For-Review, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1404558 (10Krenair) 5Open>3Resolved a:5hashar>3Krenair [12:29:10] 10Beta-Cluster, 10Mathoid, 7Monitoring, 5Patch-For-Review, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1404602 (10hashar) 5Resolved>3Open It is still falling on labs Shinke http://shinken.wmflabs.org/service... [12:45:13] hashar: euh, doesn't shinken accept ldap user/pass ? [12:45:26] (in labs) [13:07:15] mobrovac: is there a pass ? [13:07:27] mobrovac: ah : guess/guess [13:11:28] 10Beta-Cluster, 10Mathoid, 7Monitoring, 5Patch-For-Review, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1404716 (10mobrovac) >>! In T103595#1404602, @hashar wrote: > It is still falling on labs Shinke > > http://... [13:50:35] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 888 bytes in 0.010 second response time [13:51:04] RECOVERY - Mathoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 888 bytes in 0.015 second response time [13:59:45] ah [14:00:39] 10Beta-Cluster, 10Mathoid, 7Monitoring, 5Patch-For-Review, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1404780 (10hashar) 5Open>3Resolved Shinken caught up with the change (time is UTC): ``` [13:50:35] mobrovac: Mathoid monitoring is fixed on beta ^^^^ :-} [14:01:14] 10Beta-Cluster, 10Mathoid, 7Monitoring, 5Patch-For-Review, 7Shinken: Shinken is showing HTTP 404 warnings for deployment-mathoid/sca02 mathoid services - https://phabricator.wikimedia.org/T103595#1404782 (10Krenair) Yuvi has pressed buttons and this now seems fixed. [14:01:48] yey [14:01:52] hashar: merci [15:11:23] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release5, 3LE-Sprint-88: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1405016 (10Amire80) Do we need any more testing environments? If not... [15:15:22] ostriches: good mornning! [15:15:29] o/ [15:15:40] I found a good one earlier today [15:15:46] my git fetch are slow [15:15:56] turns out Gerrit git upload-pack sends ALL references :( [15:16:01] i.e. all of refs/changes/ [15:16:25] long detail at https://phabricator.wikimedia.org/T103990 [15:16:25] meh [15:16:40] tl;dr; we want to set on Gerrit side for all repo: [15:16:40] uploadpack.hiderefs=refs/changes [15:16:41] uploadpack.hiderefs=refs/cache-automerge [15:16:41] uploadpack.allowtipsha1inwant=true [15:16:45] maybe in gerrit2/.git/config [15:16:52] if jgit actually supports that [15:16:59] (untested obviously) [15:17:32] meh [15:17:56] yeah I know it is going to curse you :-] Luckily it seems the jgit version we have supports the settings [15:19:17] I thought Christian did something (ages ago) to resolve this. [15:21:19] ostriches: maybe for the replication ? [15:23:43] No, I don't think so [15:23:52] It was 2-3 years ago, I forget. [16:31:12] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 55.56% of data above the critical threshold [0.0] [16:56:13] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [16:58:26] 10Beta-Cluster, 10VisualEditor, 10VisualEditor-MediaWiki: On Beta Cluster, switching from VisualEditor to edit source mode intermittently loads the wikitext editor without any CSS - https://phabricator.wikimedia.org/T86624#1405382 (10greg) Still an issue? [17:02:29] 10Beta-Cluster, 10VisualEditor, 10VisualEditor-MediaWiki: On Beta Cluster, switching from VisualEditor to edit source mode intermittently loads the wikitext editor without any CSS - https://phabricator.wikimedia.org/T86624#1405404 (10hashar) 5Open>3declined a:3hashar Not much we can do after a few mont... [17:04:25] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-103: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1405408 (10hashar) + #ci-isolation so I get it on my radar. That is not needed for that project though. [17:04:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 20.00% of data above the critical threshold [0.0] [17:13:10] twentyafterfour: Curiosity question about phab. Can herald be used to add a standard comment to all bugs when they are flagged for security? [17:13:40] csteipp: I don't think it has an 'add a comment' option [17:14:02] The security extension probably could be [17:14:18] since it edits the transaction list on edits to a task [17:14:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [17:15:01] We could use a custom field to place a standard text template in the task header [17:15:17] csteipp: does it specifically need to be a comment? [17:16:51] twentyafterfour: Hmm, that's an interesting idea. And that could only be shown for security tasks? [17:17:39] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 10.00% of data above the critical threshold [0.0] [17:19:03] twentyafterfour, csteipp: IIRC Herald works on any edit to a task, not just creation? [17:19:13] at least for personal rules [17:21:45] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-103, 5Patch-For-Review: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1405470 (10Andrew) a:3Andrew [17:27:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [17:35:11] Does anybody know what's going on with this test? -- https://integration.wikimedia.org/ci/job/mwext-GWToolset-testextension-zend/73/console [17:35:32] ResourcesTest::testMissingDependencies is failing with "The module 'ext.visualEditor.mwcore' required by 'ext.geshi.visualEditor' must exist" [17:35:51] geshi depends on VE [17:36:01] but gwtoolset doesn't have VE as a dependency [17:38:17] legoktm: what's the fix? [17:38:33] https://integration.wikimedia.org/ci/job/mwext-GWToolset-testextension-zend/74/console [17:38:40] (03PS1) 10Legoktm: Add VisualEditor as a dependency for GWToolset (through GeSHi) [integration/config] - 10https://gerrit.wikimedia.org/r/221154 [17:39:37] the GWToolset job hasn't been converted over to use the sane-ish dependency system through zuul yet [17:39:50] ah [17:39:59] thanks for looking [17:44:33] https://integration.wikimedia.org/ci/job/mwext-GWToolset-testextension-zend/74/console woot, passed [17:44:40] (03CR) 10Legoktm: [C: 032] Add VisualEditor as a dependency for GWToolset (through GeSHi) [integration/config] - 10https://gerrit.wikimedia.org/r/221154 (owner: 10Legoktm) [17:44:40] awesome [17:47:16] (03Merged) 10jenkins-bot: Add VisualEditor as a dependency for GWToolset (through GeSHi) [integration/config] - 10https://gerrit.wikimedia.org/r/221154 (owner: 10Legoktm) [17:48:39] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 30.00% of data above the critical threshold [0.0] [17:50:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 40.00% of data above the critical threshold [0.0] [17:57:54] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 50.00% of data above the critical threshold [0.0] [18:08:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [18:10:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [18:20:58] (03PS8) 10Dduvall: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) [18:21:00] (03PS1) 10Dduvall: Refactor bundle builder to allow for more options [integration/config] - 10https://gerrit.wikimedia.org/r/221164 [18:22:31] csteipp: I could theoretically show the message only on security tasks. [18:27:51] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [18:46:56] !log running `jenkins-jobs update '*bundle*'` to deploy Icb31cf57bee0483800b41a2fb60d236fcd2d004e [18:47:01] Logged the message, Master [18:48:00] (03CR) 10Dduvall: [C: 032] Refactor bundle builder to allow for more options [integration/config] - 10https://gerrit.wikimedia.org/r/221164 (owner: 10Dduvall) [18:50:39] (03PS2) 10Dduvall: Refactor bundle builder to allow for more options [integration/config] - 10https://gerrit.wikimedia.org/r/221164 [18:53:24] (03PS9) 10Dduvall: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) [18:56:46] (03CR) 10Hashar: [C: 032] "Yeah sounds legit :-} Eventually we will want to migrate to rake I guess." [integration/config] - 10https://gerrit.wikimedia.org/r/221164 (owner: 10Dduvall) [18:58:34] (03Merged) 10jenkins-bot: Refactor bundle builder to allow for more options [integration/config] - 10https://gerrit.wikimedia.org/r/221164 (owner: 10Dduvall) [19:22:13] (03PS1) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [19:22:44] (03PS2) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [19:23:19] (03CR) 10Paladox: [C: 04-1] "Need to upload patch to metrolook first so -1." [integration/config] - 10https://gerrit.wikimedia.org/r/221175 (owner: 10Paladox) [19:30:36] (03PS3) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [19:30:46] (03CR) 10Paladox: [C: 031] Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 (owner: 10Paladox) [19:31:41] (03CR) 10Paladox: "@Krinkle not sure if I should ask you to review since you said before to only add you to the projects you activily edit like mediawiki cor" [integration/config] - 10https://gerrit.wikimedia.org/r/221175 (owner: 10Paladox) [20:57:10] 10Deployment-Systems, 6Release-Engineering, 10RESTBase, 6Services, 3releng-201516-q1: Create or improve the RESTBase deploy method (tracking) - https://phabricator.wikimedia.org/T102667#1406074 (10GWicke) [20:58:31] 10Deployment-Systems, 6Release-Engineering, 10RESTBase, 6Services, 3releng-201516-q1: Create or improve the RESTBase deploy method (tracking) - https://phabricator.wikimedia.org/T102667#1406078 (10GWicke) I changed the title to 'Create or improve' to reflect that our current Ansible-based system already... [21:20:59] !log running `jenkins-jobs update` to deploy I7affe14e878d5c1fc4bcb4dfc7f2d1494cd795b7 [21:21:03] Logged the message, Master [21:22:12] * marxarelli has forever consecrated his lunch status in the sal [21:25:50] (03CR) 10Dduvall: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [21:42:27] (03PS1) 10Legoktm: Re-add jsonlint to check pipeline for some skins [integration/config] - 10https://gerrit.wikimedia.org/r/221295 (https://phabricator.wikimedia.org/T104028) [21:42:44] (03PS2) 10Legoktm: Re-add jsonlint to check pipeline for some skins [integration/config] - 10https://gerrit.wikimedia.org/r/221295 (https://phabricator.wikimedia.org/T104028) [21:42:48] (03CR) 10Jforrester: "Meh" [integration/config] - 10https://gerrit.wikimedia.org/r/221295 (https://phabricator.wikimedia.org/T104028) (owner: 10Legoktm) [21:42:50] (03CR) 10Legoktm: [C: 032] Re-add jsonlint to check pipeline for some skins [integration/config] - 10https://gerrit.wikimedia.org/r/221295 (https://phabricator.wikimedia.org/T104028) (owner: 10Legoktm) [21:44:30] (03Merged) 10jenkins-bot: Re-add jsonlint to check pipeline for some skins [integration/config] - 10https://gerrit.wikimedia.org/r/221295 (https://phabricator.wikimedia.org/T104028) (owner: 10Legoktm) [21:45:11] !log deploying https://gerrit.wikimedia.org/r/221295 [21:45:16] Logged the message, Master [21:48:46] (03PS10) 10Dduvall: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) [21:49:26] (03PS4) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [21:50:41] (03PS11) 10Dduvall: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) [21:51:51] (03PS5) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [21:52:40] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Automatic account creation for MW-Selenium test user in integration environment - https://phabricator.wikimedia.org/T103135#1406188 (10dduvall) 5Open>3Resolved [21:52:42] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Experiment with JJB builder for running a subset of integration MW-Selenium tests - https://phabricator.wikimedia.org/T103039#1406189 (10dduvall) [21:53:34] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Define JJB builder for running a subset of integration MW-Selenium tests - https://phabricator.wikimedia.org/T103039#1406194 (10dduvall) [22:12:28] (03PS12) 10Dduvall: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) [22:17:09] (03CR) 10Dduvall: [C: 032] Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [22:18:19] 10Continuous-Integration-Infrastructure, 10Fundraising Tech Backlog, 10Wikimedia-Fundraising-CiviCRM: civicrm-buildkit doesn't composer update our wikimedia projects correctly - https://phabricator.wikimedia.org/T104031#1406224 (10awight) 3NEW [22:19:08] (03Merged) 10jenkins-bot: Builder for running MW-Selenium integration tests [integration/config] - 10https://gerrit.wikimedia.org/r/219513 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [22:20:14] !log Reloading Zuul to deploy I7affe14e878d5c1fc4bcb4dfc7f2d1494cd795b7 [22:20:19] Logged the message, Master [22:22:31] (03PS3) 10Dduvall: Integration tests to be run by CI [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) [22:23:38] (03CR) 10Dduvall: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [22:31:18] (03PS1) 10Dduvall: Move mw-integration job to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/221302 [22:31:51] (03PS2) 10Dduvall: Move mw-selenium integration job to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/221302 [22:34:06] (03CR) 10Dduvall: [C: 032] Initialize browser when using `PageFactory#on` first [selenium] - 10https://gerrit.wikimedia.org/r/220625 (https://phabricator.wikimedia.org/T103746) (owner: 10Dduvall) [22:34:08] (03PS6) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [22:34:33] (03Merged) 10jenkins-bot: Initialize browser when using `PageFactory#on` first [selenium] - 10https://gerrit.wikimedia.org/r/220625 (https://phabricator.wikimedia.org/T103746) (owner: 10Dduvall) [22:35:11] (03CR) 10Dduvall: [C: 032] Move mw-selenium integration job to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/221302 (owner: 10Dduvall) [22:36:54] (03Merged) 10jenkins-bot: Move mw-selenium integration job to test and gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/221302 (owner: 10Dduvall) [22:39:12] !log Reloading Zuul to deploy I3deec5e5a7ce7eee75268d0546eafb3e4145fdc7 [22:39:17] Logged the message, Master [22:40:12] (03PS4) 10Dduvall: Integration tests to be run by CI [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) [22:49:24] 10Staging, 3releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1406310 (10greg) [22:49:30] 6Release-Engineering, 10Staging, 3releng-201415-Q3: [Quarterly Success Metric] Green nightly builds on the staging cluster (tracking) - https://phabricator.wikimedia.org/T88701#1406311 (10greg) [22:53:31] (03CR) 10Dduvall: [C: 032] Integration tests to be run by CI [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [22:55:32] (03Merged) 10jenkins-bot: Integration tests to be run by CI [selenium] - 10https://gerrit.wikimedia.org/r/220348 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall) [23:02:20] 10Deployment-Systems, 3releng-201415-Q4: Come up with an abstract deployment model that roughly addresses the needs of existing projects - https://phabricator.wikimedia.org/T97068#1406340 (10greg) [23:02:35] 10Deployment-Systems, 6Release-Engineering, 7Epic: EPIC: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1406343 (10greg) [23:10:43] on the integration slaves, is /mnt/jenkins-workspace/workspace shared or machine-specific? [23:14:22] (03PS1) 10Dduvall: Default local environments to use user factory [selenium] - 10https://gerrit.wikimedia.org/r/221308 [23:14:24] (03PS1) 10Dduvall: Release minor version 1.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/221309 [23:16:36] (03CR) 10Dduvall: [C: 032] Default local environments to use user factory [selenium] - 10https://gerrit.wikimedia.org/r/221308 (owner: 10Dduvall) [23:16:45] (03CR) 10Dduvall: [C: 032] Release minor version 1.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/221309 (owner: 10Dduvall) [23:17:05] (03Merged) 10jenkins-bot: Default local environments to use user factory [selenium] - 10https://gerrit.wikimedia.org/r/221308 (owner: 10Dduvall) [23:18:46] (03Merged) 10jenkins-bot: Release minor version 1.4.0 [selenium] - 10https://gerrit.wikimedia.org/r/221309 (owner: 10Dduvall) [23:23:33] (03PS1) 10Awight: Update vendor using composer rather than cloning the deployment repo [integration/config] - 10https://gerrit.wikimedia.org/r/221310 [23:23:55] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:24:26] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 20.00% of data above the critical threshold [0.0] [23:24:36] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:24:40] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:24:42] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:24:52] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL 20.00% of data above the critical threshold [0.0] [23:25:06] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 33.33% of data above the critical threshold [0.0] [23:25:12] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 44.44% of data above the critical threshold [0.0] [23:25:20] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL 22.22% of data above the critical threshold [0.0] [23:25:22] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL 22.22% of data above the critical threshold [0.0] [23:25:24] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 33.33% of data above the critical threshold [0.0] [23:25:52] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 40.00% of data above the critical threshold [0.0] [23:26:28] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL 40.00% of data above the critical threshold [0.0] [23:26:30] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 30.00% of data above the critical threshold [0.0] [23:26:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 40.00% of data above the critical threshold [0.0] [23:27:12] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 33.33% of data above the critical threshold [0.0] [23:27:37] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 40.00% of data above the critical threshold [0.0] [23:27:37] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 40.00% of data above the critical threshold [0.0] [23:28:53] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 50.00% of data above the critical threshold [0.0] [23:29:03] eek? [23:29:19] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 33.33% of data above the critical threshold [0.0] [23:29:39] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 60.00% of data above the critical threshold [0.0] [23:30:11] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 66.67% of data above the critical threshold [0.0] [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:55] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 60.00% of data above the critical threshold [0.0] [23:34:55] (03CR) 10Mattflaschen: "I don't have +2 in this repository, so there's no reason to add me unless it's relevant to a repository I work on or you need my specific " [integration/config] - 10https://gerrit.wikimedia.org/r/221175 (owner: 10Paladox) [23:35:18] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [23:36:33] RECOVERY - Puppet failure on deployment-urldownloader is OK Less than 1.00% above the threshold [0.0] [23:37:15] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [23:38:02] whew recoveries [23:39:37] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [23:40:11] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [23:41:33] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [23:42:35] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [23:42:37] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [23:44:27] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [23:44:53] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [23:45:07] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [23:45:11] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [23:45:20] RECOVERY - Puppet failure on deployment-elastic07 is OK Less than 1.00% above the threshold [0.0] [23:45:22] RECOVERY - Puppet failure on deployment-pdf01 is OK Less than 1.00% above the threshold [0.0] [23:46:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [23:46:58] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [23:48:18] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<44.44%) [23:48:51] marxarelli: :) [23:48:54] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [23:48:56] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [23:49:18] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [23:49:28] jdlrobson: come on Barry! [23:49:35] :) [23:49:36] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [23:49:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [23:49:42] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [23:49:53] jdlrobson: i think there's just one tweak to make in the JJB job [23:50:09] remove barry from the patch when you resubmit [23:50:13] and he'll have another go at it [23:50:26] he caught a bug in something that tgr was working on :) i was impressed [23:50:34] nice [23:50:52] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [23:55:02] Krinkle_: got the mw-selenium job working well generally, but it's still failing for mw extensions. looks like it's trying to cd to tests/browser which isn't there [23:55:08] https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/1/consoleFull [23:55:22] Krinkle_: is there an environment variable set for the mw ext directory? [23:57:09] marxarelli: can you merge https://gerrit.wikimedia.org/r/#/c/220566/7 ?