[02:34:47] Yippee, build fixed! [02:34:47] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #906: 09FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/906/ [03:41:51] (03PS1) 10Tim Landscheidt: Disable test operations-puppet-tox-pep8-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/282452 (https://phabricator.wikimedia.org/T114887) [04:21:35] Yippee, build fixed! [04:21:36] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #776: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/776/ [05:26:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #752: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/752/ [08:32:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #935: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/935/ [12:44:15] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [13:07:26] 10Continuous-Integration-Infrastructure, 6Operations, 13Patch-For-Review: Create a basic RSpec unit test for operations/puppet - https://phabricator.wikimedia.org/T78342#842870 (10Nicko) Sorry for being so dumb, but I had a hard time finding how to execute `bundle exec spec`, it was actually because the comm... [13:07:44] I'm not able to run the Flow or Echo phpunit tests anymore... can someone help? [14:35:27] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:15:24] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:51] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:04] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:52:30] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:52:30] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:52:30] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:34] PROBLEM - Puppet run on mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:42] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:52:42] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:52:42] PROBLEM - Puppet run on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:42] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:52:42] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:53:10] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:53:10] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:53:52] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:54:06] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:54:20] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:54:48] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:55:28] PROBLEM - Puppet run on deployment-parsoid05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:55:30] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:55:45] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:55:57] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:55:57] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:55:57] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:55:59] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:56:01] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:56:15] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:56:51] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:04:12] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:18] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:14] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:28] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:10] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:18] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:44] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to puppetmaster cert - https://phabricator.wikimedia.org/T131945#2192761 (10Krenair) 5Open>3Resolved a:3Krenair I just had to replace `server = deployment-puppetmaster.eqiad.wmflabs` in `/etc/puppet/puppet.conf` on each of the instances wit... [16:10:42] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:59] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:27] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:41] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:17] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:21:11] 10Beta-Cluster-Infrastructure: Finish removal of deployment-cache-mobile04 - https://phabricator.wikimedia.org/T130473#2192766 (10Krenair) 5Open>3Resolved a:3Krenair terminated in horizon, then I deleted the wikitech page [16:23:23] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:28:10] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:28] RECOVERY - Puppet run on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:00] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:00] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:54] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:34:51] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:35] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2192775 (10Krenair) A lot of these are not so easily explained however: deployment-ur... [16:38:29] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:19] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:42:48] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:43:04] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:44:06] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:45:04] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:45:48] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:46:26] PROBLEM - Puppet run on deployment-parsoid05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:46:26] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:46:44] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:47:02] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:48:39] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 7Puppet: Puppet failures on deployment-mx: can't find puppet://private/dkim/wikimedia.org-wiki-mail.key - https://phabricator.wikimedia.org/T87848#2192780 (10Krenair) 5Open>3Invalid puppet is no longer failing on that host [16:50:19] PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:52:31] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:53:59] RECOVERY - Puppet run on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:59] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:56:21] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:11] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [17:00:58] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:26] RECOVERY - Puppet run on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:02:54] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:14] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:06] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [17:07:22] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:04] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:47] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:24] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:04] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:14] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:55] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:34] RECOVERY - Puppet run on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:53:50] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:58:28] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #874: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/874/ [17:58:37] RECOVERY - Puppet run on mira is OK: OK: Less than 1.00% above the threshold [0.0] [18:49:45] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2192863 (10Krenair) I'm going to make separate tasks for the legitimate-seeming puppet... [18:51:53] 10Beta-Cluster-Infrastructure, 7Puppet, 7Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2192864 (10Krenair) [18:52:12] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-fluorine fails puppet '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists - https://phabricator.wikimedia.org/T117028#2192878 (10Krenair) [18:52:14] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2192877 (10Krenair) [18:52:17] 10Beta-Cluster-Infrastructure, 7Puppet, 7Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2192876 (10Krenair) [18:55:32] 10Beta-Cluster-Infrastructure, 7Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2192879 (10Krenair) [19:01:37] PROBLEM - Puppet run on deployment-memc03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:01:59] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [19:04:21] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [19:04:55] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2192909 (10Krenair) [19:07:12] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2192922 (10Krenair) [19:07:21] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2192909 (10Krenair) [19:07:37] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2192909 (10Krenair) [19:08:00] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2192922 (10Krenair) [19:09:22] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:11:58] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:52] 10Beta-Cluster-Infrastructure, 6Services, 7Puppet: deployment-(mathoid|sca0[12]|cxserver03) puppet failures "Error: Could not set home on user" due to user being in use by process - https://phabricator.wikimedia.org/T132265#2192956 (10Krenair) [19:41:35] RECOVERY - Puppet run on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:56:38] 10Beta-Cluster-Infrastructure, 3Scap3, 7Puppet: deployment-((sca|aqs)01|ores-web) fails due to scap3 errors - https://phabricator.wikimedia.org/T132267#2192991 (10Krenair) [20:17:33] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-puppetmaster puppet failures due to apache trying to start on same port as nginx - https://phabricator.wikimedia.org/T132269#2193022 (10Krenair) [20:20:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 7Jenkins: Use strict mode when running hhvm tests on jenkins - https://phabricator.wikimedia.org/T132270#2193039 (10Umherirrender) [20:24:15] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-puppetmaster puppet failures due to apache trying to start on same port as nginx - https://phabricator.wikimedia.org/T132269#2193052 (10Krenair) a:320after4 I believe that nginx loads because of role::aptly https://wikitech.wikimedia.org/w/index.php?title=No... [20:30:04] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-puppetmaster puppet failures due to apache trying to start on same port as nginx - https://phabricator.wikimedia.org/T132269#2193058 (10Krenair) a:520after4>3mmodell [20:42:26] Yippee, build fixed! [20:42:26] Project selenium-Echo » chrome,beta,Linux,,contintLabsSlave && UbuntuTrusty build #20: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,VERSION=,label=contintLabsSlave%20&&%20UbuntuTrusty/20/ [20:42:29] Yippee, build fixed! [20:42:29] Project selenium-Echo » firefox,beta,Linux,,contintLabsSlave && UbuntuTrusty build #20: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,VERSION=,label=contintLabsSlave%20&&%20UbuntuTrusty/20/ [20:45:52] Hi. There is an issue with the list of domains whitelisted on Commons Beta. It has been rewritten @ https://gerrit.wikimedia.org/r/#q,6528c0be,n,z. [20:46:31] https://gerrit.wikimedia.org/r/#/c/282495 takes care to fix that. [20:47:06] Dereckson: No, I'd suggest that is wrong [20:47:06] Reedy or someone else involved with beta cluster > could you merge that? [20:47:11] Oh? [20:47:24] Where should we append this setting instead? [20:47:25] -» » 'wgCopyUploadsDomains' => array( [20:47:25] 645 -» » » '+commonswiki' => array( 'upload.wikimedia.org' ) [20:47:25] 646 -» » ), [20:47:31] Shouldn't the + be on the wg? [20:48:12] Hmm, no, we can do both apparently [20:48:30] wmfLabsOverrideSettings only seems to handle "-" case [20:48:49] We've got multiple usages of + in IS-labs [20:50:10] okay, I'm rewriting the change [20:52:04] Reedy: so like this? https://gerrit.wikimedia.org/r/282495 (PS2) [20:52:40] Not sure on the 2 +'s [20:52:56] But I'd definitely say the fix should be something like that :) [20:54:07] Kelson is available to test it. If it works, it should print AND the list with all the prod whitelisted domains AND upload.wikimedia.org @ http://commons.wikimedia.beta.wmflabs.org/wiki/Special:GWToolset [20:54:19] reedy@deployment-tin:~$ mwscript eval.php commonswiki [20:54:19] > var_dump( $wgCopyUploadsDomains ); [20:54:19] array(3) { [20:54:19] [0]=> [20:54:19] string(20) "upload.wikimedia.org" [20:54:20] [1]=> [20:54:22] string(12) "*.flickr.com" [20:54:24] [2]=> [20:54:26] string(18) "*.staticflickr.com" [20:54:27] :/ [20:54:29] } [20:54:55] 101 on production [20:55:00] That's not testing your commit btw :P [20:55:05] oh [20:55:18] I was just checking how things were now [20:57:08] So the goal is to have (i) flickr/staticflickr (2) the array added in IS (3) upload.wikimedia.org. [20:57:22] (2) because GLAM people use Commons Beta for GWT training. [20:57:58] Doesn't look to fix it [20:58:06] and the PS1 method? [20:58:12] Hack! [20:58:48] * # Leading '+' means merging the array of value with the defaults [20:58:48] * '+beta' => array( NS_HELP => true ), [21:00:08] So, - before the setting, overrides all [21:01:13] Ah [21:01:19] so '+dbname' is merging with the default [21:01:23] ok [21:01:46] PS1 should definitely fix it [21:02:23] I rollback to PS1 or you want to try something else before? [21:02:39] But we don't seem to have an example of adding to production default IS [21:03:52] Do we really not have a better way? [21:03:53] :/ [21:05:56] Dereckson: really, we should fix this in wmfLabsOverrideSettings [21:06:22] Tricky. If we add the case to wmfLabsOverrideSettings, someone will want to replace all the + array in IS by a new + array in IS-labs. [21:08:37] Okay. So I offer: (1) we revert 6528c0be, so GLAM people can work again (2) we fix it in wmfLabsOverrideSettings (3) we merge my fix to restore upload.wikimedia.org. [21:20:53] Yippee, build fixed! [21:20:53] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #232: 09FIXED in 4 min 52 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/232/ [21:34:22] Dereckson: Yeah, I think we go with your PS1, and open a task to fix it up in the near future [21:39:24] Just reverted back to PS1 [21:54:40] k [22:05:15] Filled https://phabricator.wikimedia.org/T132274 for the fix. [23:37:03] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2193205 (10Krenair) {rOPUPa64cf8d6ff72d4326818f58410ae17e14bf498e4} {rOPUP47c7c2cdfa850394176ac9af53c7b51bbdf9bf80} {rOPUPf70cf107fa... [23:37:39] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2193206 (10Krenair) a:3Joe [23:41:06] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2193207 (10Krenair) a:3Joe {rOPUP3321bda2e501867e238f9152b68fb952801ace8b} [23:42:52] 10Beta-Cluster-Infrastructure: deployment-prep puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type" - https://phabricator.wikimedia.org/T131946#2193211 (10Krenair) Today I set up a tracking task at T132259 with blockers for each o... [23:47:44] 10Beta-Cluster-Infrastructure: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179#2193222 (10Krenair) 5Open>3Resolved a:3hashar