[00:14:53] 10Phabricator (Upstream), 10Upstream: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051 (10greg) 05Open→03Declined Sorry, some things need to remove private. I agree with Andre and Evan. [00:26:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [04:48:34] Yippee, build fixed! [04:48:34] Project mediawiki-core-code-coverage-docker build #4256: 09FIXED in 1 hr 48 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4256/ [05:26:21] PROBLEM - Puppet staleness on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:41:53] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:57:34] RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0] [08:14:40] (03CR) 10Gabriel Birke: [C: 03+1] "Thank you! Our QUnit tests probably need some work to run on the command line. Can you give me some pointers?" [integration/config] - 10https://gerrit.wikimedia.org/r/510951 (owner: 10Kosta Harlan) [08:33:14] 10Release-Engineering-Team, 10Language-strategy, 10incubator.wikimedia.org, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Yupik) I've separated out the discussion about creating a new wikiproject for a language and being able to use that langua... [09:56:25] (03CR) 10Tobias Gritschacher: "@hashar @zfilipin" [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [10:21:52] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to analytics cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223697 (10Ladsgroup) [10:35:00] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10Tobi_WMDE_SW) >>! In T188742#5186473, @gerritbot wrote: > Change 460516 aba... [12:04:52] (03CR) 10Reedy: [C: 03+2] Add AntiSpoof to gate [integration/config] - 10https://gerrit.wikimedia.org/r/510967 (https://phabricator.wikimedia.org/T198653) (owner: 10Reedy) [12:06:51] (03Merged) 10jenkins-bot: Add AntiSpoof to gate [integration/config] - 10https://gerrit.wikimedia.org/r/510967 (https://phabricator.wikimedia.org/T198653) (owner: 10Reedy) [12:07:20] !log reload zuul to deploy https://gerrit.wikimedia.org/r/510967 [12:07:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:07:40] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10Reedy) [12:07:44] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Performance-Team (Radar), 10Wikimedia-production-error (Shared Build Failure): Add AntiSpoof to shared extension gate - https://phabricator.wikimedia.org/T198653 (10Reedy) 05Open→03Resolved a:03Reedy [12:16:18] I am not sure how we are going to fix want account. [12:16:36] His issue looks like the issue we experenced when we were running gerrit 2.13 [12:23:15] ohh [12:23:21] i love grep [12:23:26] case sensitive [12:23:35] found want in .//5b/3087241609b07e4de917025adfd030fb535f0a (his username is uppercase) [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:50:58] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10thcipriani) [14:13:56] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Fix Install Script For Some Versions of Ubuntu - https://phabricator.wikimedia.org/T223715 (10jeena) [14:17:26] Reedy you can let danny b and want know their accounts are fixed now! :) [14:26:04] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests, 10User-Urbanecm: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Urbanecm) 05Declined→03Open Reopening after in-person discussion with @greg. [14:42:33] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2019, 10User-zeljkofilipin: Investigate Postman for API testing - https://phabricator.wikimedia.org/T223627 (10zeljkofilipin) [14:44:24] Project beta-scap-eqiad build #249790: 04FAILURE in 0.85 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249790/ [14:48:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Wikimedia-Hackathon-2019, 10User-zeljkofilipin: Write the first Selenium test for a repository - https://phabricator.wikimedia.org/T223624 (10zeljkofilipin) [14:54:20] Project beta-scap-eqiad build #249791: 04STILL FAILING in 0.68 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249791/ [15:04:15] Yippee, build fixed! [15:04:15] Project beta-scap-eqiad build #249792: 09FIXED in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249792/ [15:05:26] very nice [15:17:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: CI runs same tests twice - https://phabricator.wikimedia.org/T223725 (10Zoranzoki21) [15:40:21] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: CI runs same tests twice - https://phabricator.wikimedia.org/T223725 (10Jdforrester-WMF) Pausing the test pipeline whilst gate-and-submit runs, and re-using the results from gate-and-submit, would be very complicated and not save much. [16:04:18] (03PS1) 10Umherirrender: [Newsletter] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511057 [16:49:45] (03PS1) 10Michael Große: Add sonar CodeHealth checks to Wikibase repos [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) [16:51:10] (03CR) 10Michael Große: "Is this a sane thing to do? If so, are there other wmde/wikidata repos to which we should add that as well?" [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [16:51:16] (03CR) 10Kosta Harlan: Add sonar CodeHealth checks to Wikibase repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/511066 (https://phabricator.wikimedia.org/T216630) (owner: 10Michael Große) [17:08:51] yay https://secure.phabricator.com/w/changelog/2019.17/ [17:15:34] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (<100.00%) [17:19:43] PROBLEM - Free space - all mounts on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [18:14:21] !log cherry-picking 511078 on puppetmaster [18:14:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:25] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests, 10User-Urbanecm: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Urbanecm) [20:03:50] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: (Service Check Timed Out) [20:03:58] PROBLEM - Puppet staleness on integration-cumin is CRITICAL: (Service Check Timed Out) [20:04:00] PROBLEM - Free space - all mounts on deployment-db05 is CRITICAL: (Service Check Timed Out) [20:04:00] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: (Service Check Timed Out) [20:04:01] PROBLEM - Puppet errors on integration-slave-docker-1034 is CRITICAL: (Service Check Timed Out) [20:04:01] PROBLEM - Puppet staleness on deployment-mx02 is CRITICAL: (Service Check Timed Out) [20:04:02] PROBLEM - Puppet staleness on integration-slave-docker-1052 is CRITICAL: (Service Check Timed Out) [20:04:04] PROBLEM - Puppet staleness on deployment-puppetdb02 is CRITICAL: (Service Check Timed Out) [20:04:35] PROBLEM - Puppet staleness on deployment-kafka-jumbo-2 is CRITICAL: (Service Check Timed Out) [20:04:38] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: (Service Check Timed Out) [20:04:38] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: (Service Check Timed Out) [20:04:38] PROBLEM - Free space - all mounts on integration-r-lang-01 is CRITICAL: (Service Check Timed Out) [20:04:38] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: (Service Check Timed Out) [20:04:48] PROBLEM - Puppet errors on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [20:04:48] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: (Service Check Timed Out) [20:04:49] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: (Service Check Timed Out) [20:04:50] PROBLEM - Free space - all mounts on deployment-ms-be05 is CRITICAL: (Service Check Timed Out) [20:04:52] PROBLEM - Puppet staleness on integration-slave-docker-1051 is CRITICAL: (Service Check Timed Out) [20:04:54] PROBLEM - Free space - all mounts on deployment-sca01 is CRITICAL: (Service Check Timed Out) [20:04:54] PROBLEM - Puppet staleness on deployment-jobrunner03 is CRITICAL: (Service Check Timed Out) [20:05:02] PROBLEM - Puppet staleness on integration-slave-docker-1055 is CRITICAL: (Service Check Timed Out) [20:05:02] PROBLEM - Puppet errors on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out) [20:05:05] PROBLEM - Puppet errors on deployment-db06 is CRITICAL: (Service Check Timed Out) [20:05:05] PROBLEM - Free space - all mounts on deployment-changeprop is CRITICAL: (Service Check Timed Out) [20:05:05] PROBLEM - Free space - all mounts on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out) [20:05:08] PROBLEM - Puppet errors on integration-slave-docker-1052 is CRITICAL: (Service Check Timed Out) [20:05:09] PROBLEM - Free space - all mounts on deployment-dumps-puppetmaster02 is CRITICAL: (Service Check Timed Out) [20:05:10] PROBLEM - Puppet staleness on deployment-sca02 is CRITICAL: (Service Check Timed Out) [20:06:21] PROBLEM - Puppet errors on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out) [20:06:21] PROBLEM - Puppet staleness on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out) [20:06:23] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: (Service Check Timed Out) [20:06:23] PROBLEM - Puppet errors on integration-slave-docker-1043 is CRITICAL: (Service Check Timed Out) [20:06:36] PROBLEM - Free space - all mounts on deployment-webperf11 is CRITICAL: (Service Check Timed Out) [20:06:41] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:43] PROBLEM - Puppet staleness on deployment-eventgate-1 is CRITICAL: (Service Check Timed Out) [20:06:43] PROBLEM - Free space - all mounts on integration-slave-docker-1052 is CRITICAL: (Service Check Timed Out) [20:07:12] PROBLEM - Free space - all mounts on deployment-aqs01 is CRITICAL: (Service Check Timed Out) [20:07:12] PROBLEM - Free space - all mounts on deployment-eventgate-1 is CRITICAL: (Service Check Timed Out) [20:07:14] PROBLEM - Free space - all mounts on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out) [20:07:14] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: (Service Check Timed Out) [20:07:14] PROBLEM - Puppet staleness on deployment-imagescaler02 is CRITICAL: (Service Check Timed Out) [20:07:24] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: (Service Check Timed Out) [20:07:31] PROBLEM - Puppet staleness on integration-slave-docker-1041 is CRITICAL: (Service Check Timed Out) [20:07:31] PROBLEM - Puppet errors on deployment-urldownloader02 is CRITICAL: (Service Check Timed Out) [20:07:32] PROBLEM - Free space - all mounts on integration-slave-docker-1034 is CRITICAL: (Service Check Timed Out) [20:07:34] PROBLEM - Free space - all mounts on deployment-cumin02 is CRITICAL: (Service Check Timed Out) [20:07:37] PROBLEM - Puppet staleness on deployment-memc06 is CRITICAL: (Service Check Timed Out) [20:07:38] PROBLEM - Free space - all mounts on deployment-memc04 is CRITICAL: (Service Check Timed Out) [20:07:43] PROBLEM - Free space - all mounts on deployment-imagescaler03 is CRITICAL: (Service Check Timed Out) [20:07:43] PROBLEM - Puppet errors on deployment-dumps-puppetmaster02 is CRITICAL: (Service Check Timed Out) [20:07:46] PROBLEM - Free space - all mounts on deployment-elastic06 is CRITICAL: (Service Check Timed Out) [20:07:46] PROBLEM - Puppet staleness on deployment-kafka-jumbo-1 is CRITICAL: (Service Check Timed Out) [20:07:48] PROBLEM - Puppet staleness on saucelabs-03 is CRITICAL: (Service Check Timed Out) [20:07:48] PROBLEM - Puppet staleness on deployment-ircd is CRITICAL: (Service Check Timed Out) [20:07:49] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: (Service Check Timed Out) [20:07:50] PROBLEM - Host Graphite Labs is DOWN: PING CRITICAL - Packet loss = 100% [20:07:51] PROBLEM - Puppet staleness on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out) [20:07:51] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: (Service Check Timed Out) [20:07:54] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: (Service Check Timed Out) [20:07:54] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: (Service Check Timed Out) [20:07:55] PROBLEM - Puppet staleness on deployment-deploy02 is CRITICAL: (Service Check Timed Out) [20:07:55] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: (Service Check Timed Out) [20:07:56] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: (Service Check Timed Out) [20:07:56] PROBLEM - Puppet staleness on deployment-sca04 is CRITICAL: (Service Check Timed Out) [20:07:57] PROBLEM - Puppet staleness on deployment-maps04 is CRITICAL: (Service Check Timed Out) [20:07:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:00] PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [20:08:00] PROBLEM - Free space - all mounts on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [20:08:08] (03CR) 10Awight: Context manager to time stuff (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/503125 (owner: 10Hashar) [20:09:15] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:40] PROBLEM - Puppet staleness on integration-slave-docker-1056 is CRITICAL: (Service Check Timed Out) [20:09:51] PROBLEM - Free space - all mounts on deployment-aqs03 is CRITICAL: (Service Check Timed Out) [20:09:52] PROBLEM - Puppet errors on integration-slave-docker-1050 is CRITICAL: (Service Check Timed Out) [20:09:59] PROBLEM - Puppet staleness on deployment-sentry01 is CRITICAL: (Service Check Timed Out) [20:10:03] PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: (Service Check Timed Out) [20:10:03] PROBLEM - Puppet staleness on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out) [20:10:11] PROBLEM - Free space - all mounts on deployment-pdfrender02 is CRITICAL: (Service Check Timed Out) [20:10:12] PROBLEM - Free space - all mounts on deployment-memc06 is CRITICAL: (Service Check Timed Out) [20:10:14] PROBLEM - Free space - all mounts on deployment-ms-be06 is CRITICAL: (Service Check Timed Out) [20:10:15] PROBLEM - Puppet staleness on deployment-elastic05 is CRITICAL: (Service Check Timed Out) [20:10:19] PROBLEM - Puppet staleness on deployment-chromium01 is CRITICAL: (Service Check Timed Out) [20:10:28] PROBLEM - Free space - all mounts on deployment-mcs01 is CRITICAL: (Service Check Timed Out) [20:10:28] PROBLEM - Puppet staleness on deployment-cache-text05 is CRITICAL: (Service Check Timed Out) [20:10:28] PROBLEM - Puppet staleness on deployment-pdfrender02 is CRITICAL: (Service Check Timed Out) [20:10:29] PROBLEM - Puppet errors on webperformance is CRITICAL: (Service Check Timed Out) [20:10:58] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out) [20:11:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [20:11:23] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: (Service Check Timed Out) [20:11:42] PROBLEM - Puppet errors on deployment-jobrunner03 is CRITICAL: (Service Check Timed Out) [20:12:11] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: (Service Check Timed Out) [20:12:12] PROBLEM - Free space - all mounts on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out) [20:12:21] PROBLEM - Host Graphite Labs is DOWN: PING CRITICAL - Packet loss = 100% [20:12:31] PROBLEM - Puppet errors on integration-slave-docker-1021 is CRITICAL: (Service Check Timed Out) [20:12:39] PROBLEM - Puppet staleness on integration-castor03 is CRITICAL: (Service Check Timed Out) [20:12:42] PROBLEM - Free space - all mounts on deployment-eventlog05 is CRITICAL: (Service Check Timed Out) [20:12:50] PROBLEM - Free space - all mounts on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out) [20:12:54] PROBLEM - Free space - all mounts on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out) [20:12:54] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: (Service Check Timed Out) [20:12:59] PROBLEM - Puppet errors on deployment-chromium02 is CRITICAL: (Service Check Timed Out) [20:13:00] PROBLEM - Puppet staleness on deployment-etcd-01 is CRITICAL: (Service Check Timed Out) [20:13:14] PROBLEM - Free space - all mounts on deployment-hadoop-test-1 is CRITICAL: (Service Check Timed Out) [20:13:15] PROBLEM - Free space - all mounts on integration-slave-docker-1043 is CRITICAL: (Service Check Timed Out) [20:13:17] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: (Service Check Timed Out) [20:13:24] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: (Service Check Timed Out) [20:13:27] PROBLEM - Free space - all mounts on deployment-restbase01 is CRITICAL: (Service Check Timed Out) [20:13:27] PROBLEM - Free space - all mounts on deployment-chromium01 is CRITICAL: (Service Check Timed Out) [20:13:32] PROBLEM - Puppet errors on deployment-maps05 is CRITICAL: (Service Check Timed Out) [20:13:39] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: (Service Check Timed Out) [20:13:44] PROBLEM - Free space - all mounts on deployment-sca04 is CRITICAL: (Service Check Timed Out) [20:13:44] PROBLEM - Free space - all mounts on deployment-memc05 is CRITICAL: (Service Check Timed Out) [20:13:45] PROBLEM - Puppet errors on deployment-db05 is CRITICAL: (Service Check Timed Out) [20:13:50] PROBLEM - Puppet errors on deployment-docker-mathoid01 is CRITICAL: (Service Check Timed Out) [20:13:57] PROBLEM - Free space - all mounts on integration-slave-docker-1053 is CRITICAL: (Service Check Timed Out) [20:14:41] PROBLEM - Puppet staleness on deployment-elastic06 is CRITICAL: (Service Check Timed Out) [20:14:42] PROBLEM - Free space - all mounts on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [20:14:59] PROBLEM - Puppet staleness on deployment-zookeeper02 is CRITICAL: (Service Check Timed Out) [20:15:04] PROBLEM - Puppet staleness on deployment-hadoop-test-3 is CRITICAL: (Service Check Timed Out) [20:15:07] PROBLEM - Free space - all mounts on deployment-poolcounter05 is CRITICAL: (Service Check Timed Out) [20:15:13] timed out? [20:15:22] Krenair maybe related to -operations? [20:16:10] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 47159 bytes in 0.786 second response time [20:16:28] RECOVERY - Free space - all mounts on deployment-webperf11 is OK: OK: All targets OK [20:16:33] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48270 bytes in 1.233 second response time [20:16:33] RECOVERY - Puppet staleness on deployment-eventgate-1 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:16:35] complete failure of all eqiad networking would explain it I guess [20:16:35] RECOVERY - Free space - all mounts on integration-slave-docker-1052 is OK: OK: All targets OK [20:16:35] RECOVERY - Puppet errors on deployment-jobrunner03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:16:40] not clear exactly what really happened yet [20:17:02] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:02] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [20:17:03] RECOVERY - Free space - all mounts on deployment-eventgate-1 is OK: OK: All targets OK [20:17:04] RECOVERY - Free space - all mounts on deployment-ms-fe03 is OK: OK: All targets OK [20:17:05] RECOVERY - Puppet staleness on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:05] RECOVERY - Free space - all mounts on deployment-aqs01 is OK: OK: All targets OK [20:17:06] RECOVERY - Free space - all mounts on deployment-kafka-main-1 is OK: OK: All targets OK [20:17:07] RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:10] RECOVERY - Host Graphite Labs is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [20:17:17] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:19] (03PS1) 10Umherirrender: [GlobalUsage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511097 [20:17:21] RECOVERY - Puppet errors on deployment-urldownloader02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:22] RECOVERY - Free space - all mounts on integration-slave-docker-1034 is OK: OK: All targets OK [20:17:22] RECOVERY - Puppet errors on integration-slave-docker-1021 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:22] RECOVERY - Puppet staleness on integration-slave-docker-1041 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:26] RECOVERY - Free space - all mounts on deployment-cumin02 is OK: OK: All targets OK [20:17:27] RECOVERY - Puppet staleness on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:31] RECOVERY - Free space - all mounts on deployment-memc04 is OK: OK: All targets OK [20:17:33] RECOVERY - Puppet errors on deployment-dumps-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:33] RECOVERY - Free space - all mounts on deployment-eventlog05 is OK: OK: deployment-prep.deployment-eventlog05.diskspace._var_lib_mysql.byte_percentfree (No valid datapoints found) [20:17:34] RECOVERY - Puppet staleness on integration-castor03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:36] RECOVERY - Free space - all mounts on deployment-imagescaler03 is OK: OK: All targets OK [20:17:38] RECOVERY - Puppet staleness on deployment-kafka-jumbo-1 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:41] RECOVERY - Free space - all mounts on deployment-elastic06 is OK: OK: deployment-prep.deployment-elastic06.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-elastic06.diskspace._var_log.byte_percentfree (No valid datapoints found) [20:17:42] RECOVERY - Free space - all mounts on deployment-acme-chief03 is OK: OK: All targets OK [20:17:43] RECOVERY - Puppet staleness on deployment-ircd is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:43] RECOVERY - Puppet staleness on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:44] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:45] RECOVERY - Host Graphite Labs is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [20:17:45] RECOVERY - Puppet staleness on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:17:48] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:49] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:17:50] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: All targets OK [20:19:10] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36627 bytes in 0.715 second response time [20:19:26] RECOVERY - Puppet staleness on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:28] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:30] RECOVERY - Free space - all mounts on integration-r-lang-01 is OK: OK: integration.integration-r-lang-01.diskspace._srv.byte_percentfree (No valid datapoints found) [20:19:30] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:30] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:31] RECOVERY - Puppet staleness on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:34] RECOVERY - Puppet staleness on integration-slave-docker-1056 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:39] RECOVERY - Puppet errors on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:40] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:41] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:43] RECOVERY - Free space - all mounts on deployment-ms-be05 is OK: OK: All targets OK [20:19:43] RECOVERY - Puppet staleness on integration-slave-docker-1051 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:44] RECOVERY - Free space - all mounts on deployment-aqs03 is OK: OK: All targets OK [20:19:45] RECOVERY - Puppet errors on integration-slave-docker-1050 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:47] RECOVERY - Puppet staleness on deployment-jobrunner03 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:47] RECOVERY - Free space - all mounts on deployment-sca01 is OK: OK: deployment-prep.deployment-sca01.diskspace._var.byte_percentfree (No valid datapoints found) deployment-prep.deployment-sca01.diskspace._srv.byte_percentfree (No valid datapoints found) deployment-prep.deployment-sca01.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-sca01.diskspace._var_log.byte_percentfree (No valid datapoints [20:19:50] RECOVERY - Puppet staleness on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:50] RECOVERY - Puppet staleness on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:52] RECOVERY - Puppet staleness on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:54] RECOVERY - Puppet staleness on integration-slave-docker-1055 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:55] RECOVERY - Puppet errors on deployment-db06 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:55] RECOVERY - Puppet staleness on deployment-hadoop-test-3 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:56] RECOVERY - Puppet staleness on deployment-acme-chief04 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:19:56] RECOVERY - Puppet errors on deployment-ms-fe03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:19:57] RECOVERY - Free space - all mounts on deployment-changeprop is OK: OK: All targets OK [20:20:00] RECOVERY - Free space - all mounts on deployment-dumps-puppetmaster02 is OK: OK: All targets OK [20:20:01] RECOVERY - Puppet staleness on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:20:01] RECOVERY - Free space - all mounts on deployment-docker-citoid01 is OK: OK: All targets OK [20:21:15] RECOVERY - Puppet errors on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:15] RECOVERY - Puppet staleness on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:21:15] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:16] RECOVERY - Puppet errors on integration-slave-docker-1043 is OK: OK: Less than 1.00% above the threshold [2.0] [20:21:16] RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:25:19] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [20:31:57] (03PS1) 10Umherirrender: [TranslationNotifications] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511101 [20:38:45] May 18 20:37:21 puppet-paladox puppet-master[21218]: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Failed to parse template gerrit/avatars_apache.erb: [20:38:46] huh [21:02:32] (03PS1) 10Umherirrender: [VipsScaler] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511107 [21:19:00] (03PS1) 10Umherirrender: [ContributionTracking] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/511111 [21:30:29] 10Continuous-Integration-Infrastructure, 10Quibble: Decouple Quibble planning and execution phases - https://phabricator.wikimedia.org/T223752 (10awight) [21:47:36] (03CR) 10Awight: "Very exciting!" (035 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [23:23:40] PROBLEM - Puppet staleness on webperformance is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0]