[00:01:48] (03PS1) 10Jforrester: Update V+2 pipeline e-mail for David Chan [integration/config] - 10https://gerrit.wikimedia.org/r/221316 [00:03:11] (03CR) 10Divec: [C: 031] "capital; splendid" [integration/config] - 10https://gerrit.wikimedia.org/r/221316 (owner: 10Jforrester) [00:04:58] Krinkle_: actually, nm. i think i see where i went wrong [00:10:03] (03CR) 10Legoktm: [C: 032] Update V+2 pipeline e-mail for David Chan [integration/config] - 10https://gerrit.wikimedia.org/r/221316 (owner: 10Jforrester) [00:10:10] Thanks, legoktm. [00:10:28] np [00:11:45] (03Merged) 10jenkins-bot: Update V+2 pipeline e-mail for David Chan [integration/config] - 10https://gerrit.wikimedia.org/r/221316 (owner: 10Jforrester) [00:13:10] (03PS1) 10Legoktm: Update David Chan's email in one more place too [integration/config] - 10https://gerrit.wikimedia.org/r/221318 [00:13:28] (03CR) 10Legoktm: [C: 032] Update David Chan's email in one more place too [integration/config] - 10https://gerrit.wikimedia.org/r/221318 (owner: 10Legoktm) [00:14:47] (03CR) 10Jforrester: [C: 031] Update David Chan's email in one more place too [integration/config] - 10https://gerrit.wikimedia.org/r/221318 (owner: 10Legoktm) [00:15:10] (03Merged) 10jenkins-bot: Update David Chan's email in one more place too [integration/config] - 10https://gerrit.wikimedia.org/r/221318 (owner: 10Legoktm) [00:15:16] legoktm: Whoops, sorry. [00:15:40] !log deploying https://gerrit.wikimedia.org/r/221316 & https://gerrit.wikimedia.org/r/221318 [00:15:45] Logged the message, Master [00:16:09] I missed it too [01:01:55] legoktm: Did you want me to +2-spree your skin npm commits? [01:02:23] legoktm: (And for that matter, your ImageMetrics commit.) [01:04:30] (03PS1) 10Legoktm: Convert MassMessage and GWToolset to use generic phpunit job [integration/config] - 10https://gerrit.wikimedia.org/r/221328 (https://phabricator.wikimedia.org/T96690) [01:06:30] James_F: uh, probably not now, my head is in other things right now [01:06:37] legoktm: No worries. [01:08:45] (03CR) 10Legoktm: [C: 032] Convert MassMessage and GWToolset to use generic phpunit job [integration/config] - 10https://gerrit.wikimedia.org/r/221328 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [01:10:57] (03Merged) 10jenkins-bot: Convert MassMessage and GWToolset to use generic phpunit job [integration/config] - 10https://gerrit.wikimedia.org/r/221328 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [01:13:50] !log deploying https://gerrit.wikimedia.org/r/221328 [01:13:55] Logged the message, Master [01:25:25] (03PS1) 10Legoktm: Use generic jobs for GuidedTour & ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221329 (https://phabricator.wikimedia.org/T96690) [01:26:10] (03CR) 10Legoktm: [C: 032] Use generic jobs for GuidedTour & ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221329 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [01:28:09] (03Merged) 10jenkins-bot: Use generic jobs for GuidedTour & ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221329 (https://phabricator.wikimedia.org/T96690) (owner: 10Legoktm) [01:28:24] !log deploying https://gerrit.wikimedia.org/r/221329 [01:28:29] Logged the message, Master [01:32:40] (03PS1) 10Legoktm: Use generic qunit job for GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/221330 [01:33:18] (03CR) 10Legoktm: [C: 032] Use generic qunit job for GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/221330 (owner: 10Legoktm) [01:34:36] legoktm: Want to do that now? [01:35:14] (03Merged) 10jenkins-bot: Use generic qunit job for GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/221330 (owner: 10Legoktm) [01:35:26] (Once pushed.) [01:36:45] !log deploying https://gerrit.wikimedia.org/r/221330 [01:36:51] Logged the message, Master [01:37:44] James_F: sure [01:40:10] (03PS1) 10Legoktm: Configure npm for skins and ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221331 [01:40:20] (03CR) 10Legoktm: [C: 032] Configure npm for skins and ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221331 (owner: 10Legoktm) [01:42:03] (03Merged) 10jenkins-bot: Configure npm for skins and ImageMetrics [integration/config] - 10https://gerrit.wikimedia.org/r/221331 (owner: 10Legoktm) [01:42:17] !log deploying https://gerrit.wikimedia.org/r/221331 [01:42:22] Logged the message, Master [01:42:40] James_F: done [01:45:09] legoktm: And done. [01:47:43] legoktm: Thanks! [01:48:02] yay [01:52:39] (03PS1) 10Legoktm: Make FlaggedRevs tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/221333 (https://phabricator.wikimedia.org/T63848) [01:52:44] (03PS1) 10Legoktm: Use generic phpunit job for FlaggedRevs [integration/config] - 10https://gerrit.wikimedia.org/r/221334 [01:52:58] (03CR) 10Legoktm: [C: 032] Make FlaggedRevs tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/221333 (https://phabricator.wikimedia.org/T63848) (owner: 10Legoktm) [01:53:22] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for FlaggedRevs [integration/config] - 10https://gerrit.wikimedia.org/r/221334 (owner: 10Legoktm) [01:54:44] (03Merged) 10jenkins-bot: Make FlaggedRevs tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/221333 (https://phabricator.wikimedia.org/T63848) (owner: 10Legoktm) [01:55:20] (03Merged) 10jenkins-bot: Use generic phpunit job for FlaggedRevs [integration/config] - 10https://gerrit.wikimedia.org/r/221334 (owner: 10Legoktm) [01:56:11] !log deploying https://gerrit.wikimedia.org/r/221333 & https://gerrit.wikimedia.org/r/221334 [01:56:17] Logged the message, Master [01:57:50] deployment-bastion is running out of space on /var [01:57:54] /var/log/account/pacct.0: 296M [01:57:54] /var/log/account/pacct: 241M [01:58:17] just delete those [01:58:19] they're useless [02:02:05] legoktm, why are they being created at all then? [02:02:38] dunno, I blame Yuvi [02:12:49] (03PS1) 10Legoktm: LifeWeb depends upon LifeWebCore [integration/config] - 10https://gerrit.wikimedia.org/r/221337 [02:13:09] (03CR) 10Legoktm: [C: 032] LifeWeb depends upon LifeWebCore [integration/config] - 10https://gerrit.wikimedia.org/r/221337 (owner: 10Legoktm) [02:15:06] (03Merged) 10jenkins-bot: LifeWeb depends upon LifeWebCore [integration/config] - 10https://gerrit.wikimedia.org/r/221337 (owner: 10Legoktm) [02:15:21] !log deploying https://gerrit.wikimedia.org/r/#/c/221337/ [02:15:26] Logged the message, Master [02:17:53] (03PS1) 10Krinkle: Remove non-voting jslint from MaintenanceShell, enable npm [integration/config] - 10https://gerrit.wikimedia.org/r/221338 [02:18:09] (03PS2) 10Krinkle: Remove non-voting jslint from MaintenanceShell, enable npm [integration/config] - 10https://gerrit.wikimedia.org/r/221338 [02:18:55] (03CR) 10Krinkle: [C: 032] Remove non-voting jslint from MaintenanceShell, enable npm [integration/config] - 10https://gerrit.wikimedia.org/r/221338 (owner: 10Krinkle) [02:20:39] (03Merged) 10jenkins-bot: Remove non-voting jslint from MaintenanceShell, enable npm [integration/config] - 10https://gerrit.wikimedia.org/r/221338 (owner: 10Krinkle) [02:21:51] (03PS1) 10Krinkle: Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 [02:22:18] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/221338 [02:22:24] Logged the message, Master [02:22:54] (03PS2) 10Krinkle: Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 [02:22:58] (03CR) 10Krinkle: [C: 032] Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 (owner: 10Krinkle) [02:24:31] (03CR) 10jenkins-bot: [V: 04-1] Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 (owner: 10Krinkle) [02:27:04] 10Beta-Cluster, 10Continuous-Integration-Config, 10Math: beta-recompile-math-texvc-eqiad job fails with "/usr/local/bin/scap-recompile: No such file or directory" - https://phabricator.wikimedia.org/T91191#1406611 (10Legoktm) So is texvc still required now that we have mathoid? I assume beta is using that no... [02:28:40] (03PS3) 10Krinkle: Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 [02:31:35] (03CR) 10Krinkle: [C: 032] Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 (owner: 10Krinkle) [02:32:46] (03PS1) 10Legoktm: Use generic phpunit jobs for Math & MathSearch extensions [integration/config] - 10https://gerrit.wikimedia.org/r/221343 [02:33:15] (03Merged) 10jenkins-bot: Enable phplint for TopTenPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/221342 (owner: 10Krinkle) [02:36:39] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/221342 [02:36:49] Logged the message, Master [02:38:29] (03PS1) 10Legoktm: Use generic phpunit job for NavigationTiming & ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/221344 [02:38:39] (03CR) 10Legoktm: [C: 032] Use generic phpunit jobs for Math & MathSearch extensions [integration/config] - 10https://gerrit.wikimedia.org/r/221343 (owner: 10Legoktm) [02:38:44] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for NavigationTiming & ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/221344 (owner: 10Legoktm) [02:40:58] (03Merged) 10jenkins-bot: Use generic phpunit jobs for Math & MathSearch extensions [integration/config] - 10https://gerrit.wikimedia.org/r/221343 (owner: 10Legoktm) [02:41:00] (03Merged) 10jenkins-bot: Use generic phpunit job for NavigationTiming & ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/221344 (owner: 10Legoktm) [02:42:15] !log deploying https://gerrit.wikimedia.org/r/221343 & https://gerrit.wikimedia.org/r/221344 [02:42:24] Logged the message, Master [02:47:55] aaand I'm done now [02:54:28] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Define JJB builder for running a subset of integration MW-Selenium tests - https://phabricator.wikimedia.org/T103039#1406620 (10dduvall) This is up and running for MW-Selenium itself—the project is dogfooding the JJB builder with it... [03:16:48] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406634 (10mmodell) 3NEW a:3mmodell [03:19:45] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406644 (10mmodell) [03:54:20] 10Continuous-Integration-Infrastructure, 10MonoBook, 10Vector: Set up phpunit structure tests for MediaWiki skin repositories - https://phabricator.wikimedia.org/T68926#1406679 (10Legoktm) [05:26:10] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 44.44% of data above the critical threshold [0.0] [05:46:10] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [06:10:48] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406745 (10mmodell) So there are a few features in the redesign-2015 branch that would be nice to have: 1. Projects can be added to calendar events 2. "Object policies" allow new ty... [06:10:59] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406748 (10mmodell) [06:38:17] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:43:10] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL 44.44% of data above the critical threshold [0.0] [07:13:12] RECOVERY - Puppet failure on deployment-logstash1 is OK Less than 1.00% above the threshold [0.0] [07:35:42] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406774 (10Qgil) [10:29:03] 6Release-Engineering, 6Phabricator, 5Release: Next Phabricator upgrade: 2015-07-01 - https://phabricator.wikimedia.org/T104047#1406891 (10Paladox) Yes that looks good haven't really spotted any bugs on phab-03 that would prevent us from upgrading to redesign branch. [10:46:49] (03PS7) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [14:40:35] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:44:55] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:46:05] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 55.56% of data above the critical threshold [0.0] [14:47:13] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 44.44% of data above the critical threshold [0.0] [14:47:55] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:48:39] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:48:56] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:49:54] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 20.00% of data above the critical threshold [0.0] [14:49:56] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:50:20] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:50:40] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:50:42] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:50:44] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:51:22] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 70.00% of data above the critical threshold [0.0] [14:51:40] PROBLEM - Puppet failure on deployment-upload is CRITICAL 50.00% of data above the critical threshold [0.0] [14:51:54] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 55.56% of data above the critical threshold [0.0] [14:52:30] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:52:31] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL 30.00% of data above the critical threshold [0.0] [14:52:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:53:06] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 44.44% of data above the critical threshold [0.0] [14:53:12] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:53:16] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:54:08] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:54:35] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 70.00% of data above the critical threshold [0.0] [14:55:08] PROBLEM - Puppet failure on deployment-test is CRITICAL 44.44% of data above the critical threshold [0.0] [14:55:32] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:55:32] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:55:58] lots of puppet failures due to "Connection reset by peer - SSL_connect" [14:56:16] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:56:16] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 44.44% of data above the critical threshold [0.0] [14:56:26] PROBLEM - Puppet failure on deployment-stream is CRITICAL 60.00% of data above the critical threshold [0.0] [14:57:06] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL 55.56% of data above the critical threshold [0.0] [14:57:32] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:58:38] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 40.00% of data above the critical threshold [0.0] [15:01:19] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL 66.67% of data above the critical threshold [0.0] [15:02:47] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 20.00% of data above the critical threshold [0.0] [15:05:27] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 50.00% of data above the critical threshold [0.0] [15:05:53] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL 40.00% of data above the critical threshold [0.0] [15:06:19] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL 50.00% of data above the critical threshold [0.0] [15:10:05] I hope that's not due to a button I pressed.. [15:15:54] It did do something that I thought was a bit weird earlier [15:20:07] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [15:20:29] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [15:21:13] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [15:21:14] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [15:21:20] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [15:22:32] RECOVERY - Puppet failure on deployment-urldownloader is OK Less than 1.00% above the threshold [0.0] [15:22:32] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [15:22:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [15:23:12] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [15:23:28] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:23:36] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [15:24:57] (I didn't press anything to fix it, though.) [15:24:58] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [15:24:58] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [15:25:16] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [15:25:20] I'm wondering if deployment-cache-bits01 should go away now we no longer have bits: [15:25:21] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: $$lvs::configuration::service_ips["bits"]["eqiad"] is :undef, not a hash or array at /etc/puppet/modules/role/manifests/cache/configuration.pp:12 on node deployment-cache-bits01.deployment-prep.eqiad.wmflabs [15:25:26] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [15:25:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [15:25:54] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [15:26:09] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [15:26:22] RECOVERY - Puppet failure on deployment-elastic07 is OK Less than 1.00% above the threshold [0.0] [15:26:26] RECOVERY - Puppet failure on deployment-pdf01 is OK Less than 1.00% above the threshold [0.0] [15:27:12] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [15:27:54] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [15:28:40] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [15:28:54] RECOVERY - Puppet failure on deployment-restbase01 is OK Less than 1.00% above the threshold [0.0] [15:29:10] RECOVERY - Puppet failure on deployment-logstash1 is OK Less than 1.00% above the threshold [0.0] [15:29:54] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [15:30:38] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [15:30:40] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [15:31:38] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [15:31:54] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [15:32:08] RECOVERY - Puppet failure on deployment-redis02 is OK Less than 1.00% above the threshold [0.0] [15:32:32] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [15:32:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [15:35:33] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [15:36:27] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [15:38:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [15:38:17] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [15:40:37] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [15:44:34] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [16:14:49] that error is the reason puppet also fails on cache-mobile03, cache-text02, cache-upload02, and parsoidcache02 btw [16:35:06] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [16:40:07] "sudo a2enmod authz_groupfile" fixed puppet on logstash2 [16:40:47] it has an "AuthGroupFile /dev/null" entry in /etc/apache2/sites-enabled/50-logstash-beta-wmflabs-org.conf, which apache didn't like to start with [16:44:35] 10Beta-Cluster, 10Traffic: Puppet failing on deployment-prep caches - https://phabricator.wikimedia.org/T104076#1407254 (10Krenair) 3NEW [16:49:17] 10Beta-Cluster: /var filling up on deployment-bastion - https://phabricator.wikimedia.org/T104077#1407261 (10Krenair) 3NEW [18:48:09] (03CR) 10Alex Monk: "Resolved T93728 ?" [integration/config] - 10https://gerrit.wikimedia.org/r/221337 (owner: 10Legoktm) [18:55:40] (03PS1) 10Alex Monk: LifeWeb depends on Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/221399 (https://phabricator.wikimedia.org/T104085) [19:01:16] (03CR) 10Legoktm: [C: 04-1] "Unfortunately the Wikibase installation process is more complicated than normal extensions (see jjb/wikidata.yaml) so this won't work..." [integration/config] - 10https://gerrit.wikimedia.org/r/221399 (https://phabricator.wikimedia.org/T104085) (owner: 10Alex Monk) [19:09:39] (03PS1) 10Legoktm: Create "composer-test" macro [integration/config] - 10https://gerrit.wikimedia.org/r/221400 [19:09:41] (03PS1) 10Legoktm: Take advantage of /usr/local/bin/composer symlink [integration/config] - 10https://gerrit.wikimedia.org/r/221401 [19:11:21] (03Abandoned) 10Alex Monk: LifeWeb depends on Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/221399 (https://phabricator.wikimedia.org/T104085) (owner: 10Alex Monk) [19:14:47] (03CR) 10Legoktm: [C: 032] "no-op" [integration/config] - 10https://gerrit.wikimedia.org/r/221400 (owner: 10Legoktm) [19:18:01] (03Merged) 10jenkins-bot: Create "composer-test" macro [integration/config] - 10https://gerrit.wikimedia.org/r/221400 (owner: 10Legoktm) [20:02:58] 10Beta-Cluster: /var filling up on deployment-bastion - https://phabricator.wikimedia.org/T104077#1407545 (10hashar) [20:03:01] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1407546 (10hashar) [21:18:26] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [22:25:50] 10Beta-Cluster, 7Graphite, 7Shinken: Delete specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T104091#1407700 (10Krenair) 3NEW [22:48:53] (03PS8) 10Paladox: Configure npm for Metrolook and update tests [integration/config] - 10https://gerrit.wikimedia.org/r/221175 [23:09:20] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<33.33%)