[03:41:58] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [0.0] [06:37:19] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:37:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL 50.00% of data above the critical threshold [0.0] [06:59:02] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 100.00% of data above the critical threshold [0.0] [07:02:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [09:28:08] 10Beta-Cluster, 6operations, 7Puppet: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#1426362 (10Nemo_bis) [09:28:10] 10Beta-Cluster, 6Release-Engineering, 7Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#1426361 (10Nemo_bis) [09:30:04] 10Beta-Cluster: Beta cluster bits should not cache static-master for three weeks - https://phabricator.wikimedia.org/T90983#1426365 (10Nemo_bis) [09:30:06] 10Beta-Cluster, 6Release-Engineering, 7Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#1426364 (10Nemo_bis) [09:36:30] 10Beta-Cluster, 6Release-Engineering, 7Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#1426368 (10Nemo_bis) IMHO issues such as T90983 are somehow inherent in the essence of Beta and it's not the only "hopeless" blocker of this bug. Perha... [09:39:34] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1426371 (10Krinkle) >>! In T99096#1354157, @dduvall wrote: >>>! In T99096#1337418, @BBlack wrote: >>... [10:45:57] 10Beta-Cluster, 10Pywikibot-OAuth: Investigate process for setting up an OAuth client on the Beta cluster - https://phabricator.wikimedia.org/T104764#1426391 (10jayvdb) 3NEW a:3VcamX [12:07:17] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:39:17] (03PS1) 10Eranroz: adding mediawiki/extensions/LanguageTool [integration/config] - 10https://gerrit.wikimedia.org/r/222760 [12:55:34] (03PS2) 10Amire80: Adding mediawiki/extensions/LanguageTool [integration/config] - 10https://gerrit.wikimedia.org/r/222760 (https://phabricator.wikimedia.org/T104760) (owner: 10Eranroz) [14:33:46] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #554: FAILURE in 7 min 45 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/554/ [16:19:47] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [22:58:22] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<30.00%) [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]