[02:17:30] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 20.00% of data above the critical threshold [0.0] [02:19:40] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:19:53] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:20:37] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 30.00% of data above the critical threshold [0.0] [02:20:38] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 30.00% of data above the critical threshold [0.0] [02:20:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 30.00% of data above the critical threshold [0.0] [02:21:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 50.00% of data above the critical threshold [0.0] [02:23:39] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 50.00% of data above the critical threshold [0.0] [02:24:07] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 33.33% of data above the critical threshold [0.0] [02:24:13] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 44.44% of data above the critical threshold [0.0] [02:24:27] PROBLEM - Puppet failure on deployment-stream is CRITICAL 20.00% of data above the critical threshold [0.0] [02:25:31] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 60.00% of data above the critical threshold [0.0] [02:25:37] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 70.00% of data above the critical threshold [0.0] [02:26:13] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 22.22% of data above the critical threshold [0.0] [02:26:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL 30.00% of data above the critical threshold [0.0] [02:26:58] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:27:26] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:28:10] PROBLEM - Puppet failure on deployment-test is CRITICAL 22.22% of data above the critical threshold [0.0] [02:28:22] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL 33.33% of data above the critical threshold [0.0] [02:28:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 20.00% of data above the critical threshold [0.0] [02:28:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:30:55] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 40.00% of data above the critical threshold [0.0] [02:31:42] PROBLEM - Puppet failure on deployment-salt is CRITICAL 22.22% of data above the critical threshold [0.0] [02:31:52] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL 30.00% of data above the critical threshold [0.0] [02:35:20] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL 70.00% of data above the critical threshold [0.0] [02:37:20] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 33.33% of data above the critical threshold [0.0] [02:37:28] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL 22.22% of data above the critical threshold [0.0] [02:38:40] PROBLEM - Puppet failure on deployment-upload is CRITICAL 60.00% of data above the critical threshold [0.0] [02:39:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 66.67% of data above the critical threshold [0.0] [02:39:32] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 60.00% of data above the critical threshold [0.0] [02:40:26] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 20.00% of data above the critical threshold [0.0] [02:40:28] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:40:40] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 40.00% of data above the critical threshold [0.0] [02:41:04] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 22.22% of data above the critical threshold [0.0] [02:41:06] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL 22.22% of data above the critical threshold [0.0] [02:44:07] PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL 33.33% of data above the critical threshold [0.0] [03:43:03] (03PS1) 10Alex Monk: No longer branch Extension:Oversight [tools/release] - 10https://gerrit.wikimedia.org/r/224298 (https://phabricator.wikimedia.org/T62373) [04:02:33] (03CR) 10Alex Monk: [C: 04-2] "Not ready yet." [tools/release] - 10https://gerrit.wikimedia.org/r/224298 (https://phabricator.wikimedia.org/T62373) (owner: 10Alex Monk) [05:36:09] Yippee, build fixed! [05:36:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #477: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/477/ [06:36:20] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:42:31] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<10.00%) [06:59:02] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 100.00% of data above the critical threshold [0.0] [07:07:28] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [08:11:47] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:21:04] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [10:22:18] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [10:23:40] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [10:24:08] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [10:24:16] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [10:25:26] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [10:25:28] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [10:25:42] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [10:26:06] RECOVERY - Puppet failure on deployment-redis02 is OK Less than 1.00% above the threshold [0.0] [10:27:29] RECOVERY - Puppet failure on deployment-urldownloader is OK Less than 1.00% above the threshold [0.0] [10:27:37] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [10:29:41] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [10:29:53] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [10:30:37] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [10:30:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [10:31:44] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [10:34:24] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [10:35:29] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [10:36:17] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [10:36:53] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [10:36:57] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [10:37:23] RECOVERY - Puppet failure on deployment-pdf01 is OK Less than 1.00% above the threshold [0.0] [10:38:19] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [10:38:57] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [10:39:15] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [10:40:33] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [10:40:39] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [10:40:57] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [10:41:49] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [10:43:10] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [10:43:36] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [10:43:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [10:44:08] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [10:44:32] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [10:45:22] RECOVERY - Puppet failure on deployment-elastic07 is OK Less than 1.00% above the threshold [0.0] [10:46:53] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [12:07:17] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:28:30] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [13:06:10] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #714: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/714/ [13:55:06] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:55:06] PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:55:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 11.11% of data above the critical threshold [0.0] [13:55:36] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:55:44] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 20.00% of data above the critical threshold [0.0] [13:55:52] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 30.00% of data above the critical threshold [0.0] [13:56:20] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:56:26] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 33.33% of data above the critical threshold [0.0] [13:56:28] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL 40.00% of data above the critical threshold [0.0] [13:56:36] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 30.00% of data above the critical threshold [0.0] [13:56:40] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 40.00% of data above the critical threshold [0.0] [13:57:04] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL 33.33% of data above the critical threshold [0.0] [13:57:44] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 40.00% of data above the critical threshold [0.0] [13:58:17] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 22.22% of data above the critical threshold [0.0] [13:58:29] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL 60.00% of data above the critical threshold [0.0] [13:58:35] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 60.00% of data above the critical threshold [0.0] [13:59:09] PROBLEM - Puppet failure on deployment-test is CRITICAL 66.67% of data above the critical threshold [0.0] [14:01:49] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:02:05] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:14:06] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [14:15:08] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [14:15:30] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [14:16:21] RECOVERY - Puppet failure on deployment-elastic07 is OK Less than 1.00% above the threshold [0.0] [14:18:23] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [14:22:05] RECOVERY - Puppet failure on deployment-redis02 is OK Less than 1.00% above the threshold [0.0] [14:22:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [14:25:10] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [14:25:20] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [14:26:26] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [14:26:34] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [14:26:40] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [14:28:28] RECOVERY - Puppet failure on deployment-urldownloader is OK Less than 1.00% above the threshold [0.0] [14:28:32] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [14:30:44] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [14:30:54] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [14:31:36] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [14:31:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [14:32:41] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [15:14:14] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:34:55] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1448267 (10Krinkle) [17:10:03] (03PS1) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) [17:25:16] PROBLEM - Puppet failure on nodepool-t105406 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:27:41] (03CR) 10Reedy: [WIP] Sync /srv/mediawiki-staging to co-masters (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [17:27:45] (03CR) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [17:32:00] (03CR) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [20:34:12] 10Deployment-Systems, 5Patch-For-Review: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server - https://phabricator.wikimedia.org/T104826#1448484 (10bd808) a:3bd808 [20:41:15] 10Deployment-Systems: [scap] Add support for a global mutex to keep multiple masters from clobbering each other - https://phabricator.wikimedia.org/T105195#1448486 (10bd808) See https://github.com/jplana/python-etcd/pull/114 for support in python-etcd for global locks. [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:59:41] (03PS1) 10Thcipriani: Add service deploy via scap [tools/scap] - 10https://gerrit.wikimedia.org/r/224374