[00:05:47] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [00:07:37] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:42] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:43:33] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [01:15:54] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:18:17] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:24:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:43:02] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:44:00] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:52:38] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:55:54] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:21] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:13:01] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:19:00] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [02:37:38] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:52:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:12:46] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:17:15] Yippee, build fixed! [04:17:15] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #454: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/454/ [05:04:00] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:15:01] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:15:05] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:42:24] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:45:04] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:08:58] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:09:59] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:42:40] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:53:04] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:00:40] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:07:39] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:43] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:30] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:23:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:47:31] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:47:41] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:48:39] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:48:47] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:01:42] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:12:40] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:43:39] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:56:40] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:58] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:03:39] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:15:00] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [11:37:06] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [11:43:38] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:10] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:04:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:07:05] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:07:53] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn should ignore return in closure - https://phabricator.wikimedia.org/T170765#3441957 (10Tgr) [12:31:55] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:32:22] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:50:28] twentyafterfour but they have stopped allowing users contributing to phabricator's open source because here https://discourse.phabricator-community.org/t/issue-of-authentification-on-https-secure-phabricator-com/51/5 they say they have closed of allowing users to register accounts. IE it's invite only now. [12:52:23] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:52:45] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:59:45] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:01:53] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:07:44] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:38] Yippee, build fixed! [13:46:39] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #461: 09FIXED in 2 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/461/ [13:52:50] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:54:44] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:23:26] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:27:51] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:04] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:36:47] 10Scap, 10Services (watching), 10User-mobrovac: Scap config management: Jinja2 fills templates with Pythonic values - https://phabricator.wikimedia.org/T145510#3442060 (10mobrovac) [15:37:55] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:53] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:18] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:13:11] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:20:33] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:38:17] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:23] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:52:16] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:20:21] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:16] RECOVERY - Puppet errors on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:58] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:16:48] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:32:54] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:51] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:26] PROBLEM - Puppet staleness on deployment-eventlogging03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [21:16:22] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:18:44] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:47:08] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:51:26] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:57:05] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:40] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:42:21] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:58:45] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:07:40] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [23:22:22] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]