[00:11:06] 10MediaWiki-Releasing: keys.html has outdated styling - https://phabricator.wikimedia.org/T181018#3891803 (10Krinkle) [00:13:52] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:16:15] 10MediaWiki-Releasing, 10Patch-For-Review: keys.html has outdated styling - https://phabricator.wikimedia.org/T181018#3891831 (10Krinkle) >>! In T181018#3891818, @gerritbot wrote: > Change 403569 had a related patch set uploaded (by Krinkle; owner: Krinkle): > [operations/mediawiki-config@master] keys: Simplif... [00:30:49] 10Beta-Cluster-Infrastructure, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences: Deploy GlobalPreferences on beta cluster - https://phabricator.wikimedia.org/T184668#3891856 (10MaxSem) [00:31:52] 10Beta-Cluster-Infrastructure, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences: Deploy GlobalPreferences on beta cluster - https://phabricator.wikimedia.org/T184668#3891866 (10MaxSem) [00:32:08] 10Beta-Cluster-Infrastructure, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences: Deploy GlobalPreferences on beta cluster - https://phabricator.wikimedia.org/T184668#3891856 (10MaxSem) [00:35:04] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:39:59] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:40:23] AaronSchulz, are you working on the puppet repository on deployment-puppetmaster02? [00:42:58] Krenair: debugging a patch, yes [00:43:14] ah [00:53:05] PROBLEM - Puppet errors on deployment-eventlog02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:53:49] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:06:06] (03CR) 10Chad: [C: 032] release: Remove upload tar, dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/403561 (owner: 10Chad) [01:06:30] (03Merged) 10jenkins-bot: release: Remove upload tar, dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/403561 (owner: 10Chad) [01:06:53] (03CR) 10Chad: [C: 032] make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 (owner: 10Chad) [01:07:08] (03Abandoned) 10Chad: grumble grumble submodules [tools/release] - 10https://gerrit.wikimedia.org/r/403220 (owner: 10Chad) [01:07:27] (03Merged) 10jenkins-bot: make-wmf-branch: Combine skins/vendor to extension logic [tools/release] - 10https://gerrit.wikimedia.org/r/403148 (owner: 10Chad) [01:09:51] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:12:59] Krenair: I'm done. I reset it back to the reflog hash of that T184234 patch [01:13:00] T184234: Puppet broken on deployment-cache-text04 due to varnishkafka issues - https://phabricator.wikimedia.org/T184234 [01:19:50] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:20:00] RECOVERY - Puppet errors on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:02] RECOVERY - Puppet errors on deployment-eventlog02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:33:12] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap canary has a shifting baseline - https://phabricator.wikimedia.org/T183999#3891980 (10thcipriani) p:05Triage>03Normal a:03thcipriani [01:34:33] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%) [01:38:08] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [02:43:41] Project beta-scap-eqiad build #190250: 04FAILURE in 0.35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/190250/ [02:56:37] Yippee, build fixed! [02:56:38] Project beta-scap-eqiad build #190251: 09FIXED in 2 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/190251/ [06:53:06] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:58:04] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:07:56] RECOVERY - Puppet staleness on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:43:30] (03CR) 10Hashar: "@Paladox wrote:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [07:44:31] (03CR) 10Hashar: [C: 032] Add BlueSpiceFoundation dependency for BlueSpice extensions [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42) [07:44:39] (03CR) 10jerkins-bot: [V: 04-1] Add BlueSpiceFoundation dependency for BlueSpice extensions [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42) [09:19:01] (03CR) 10Hashar: [C: 032] Provide Java 8 for mwdumper testing [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [09:20:20] (03Merged) 10jenkins-bot: Provide Java 8 for mwdumper testing [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [09:21:32] (03CR) 10Hashar: [C: 032] "I have refreshed the job and rechecked the "major refactoring" change https://gerrit.wikimedia.org/r/#/c/309314/ . It passes now!" [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [09:24:29] (03CR) 10Phantom42: "Hm. I will try to fix this." [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42) [09:27:35] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3892411 (10Addshore) [09:27:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [09:27:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [10:00:22] (03Abandoned) 10Hashar: Update paramiko to 2.2 [integration/zuul] (upstream) - 10https://gerrit.wikimedia.org/r/383913 (https://phabricator.wikimedia.org/T171165) (owner: 10Paladox) [10:00:55] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Zuul: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165#3892520 (10hashar) p:05Triage>03Low [10:46:33] zeljkof: hi! could we setup a meeting in the coming days to see what left be done on https://gerrit.wikimedia.org/r/#/c/395872/ ? [10:46:58] with holydays I completely lost track of this one :/ [10:47:32] dcausse: sure! [10:47:46] sorry, I was sick for the last two weeks, so I am still catching up [10:48:09] I think what is left to do is for me to plug the patch into CI [10:48:35] dcausse: my google calendar is up to date, feel free to set up a meeting [10:48:58] zeljkof: sure I'll send an invite for next week, feel free to change the schedule [10:49:01] thanks! [11:01:39] (03PS2) 10Reedy: release: patchfile refactoring [tools/release] - 10https://gerrit.wikimedia.org/r/403562 (owner: 10Chad) [11:01:43] (03CR) 10Reedy: [C: 032] release: patchfile refactoring [tools/release] - 10https://gerrit.wikimedia.org/r/403562 (owner: 10Chad) [11:04:31] (03Merged) 10jenkins-bot: release: patchfile refactoring [tools/release] - 10https://gerrit.wikimedia.org/r/403562 (owner: 10Chad) [11:05:18] (03PS3) 10Hashar: Switch from precise to jessie in test-requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/403167 (https://phabricator.wikimedia.org/T162191) (owner: 10Paladox) [11:07:48] (03CR) 10Hashar: [C: 032] Switch from precise to jessie in test-requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/403167 (https://phabricator.wikimedia.org/T162191) (owner: 10Paladox) [11:09:29] (03Merged) 10jenkins-bot: Switch from precise to jessie in test-requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/403167 (https://phabricator.wikimedia.org/T162191) (owner: 10Paladox) [12:24:38] (03PS1) 10Hashar: Migrate mwgate-rake to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403630 [12:25:00] (03CR) 10Hashar: [C: 032] Migrate mwgate-rake to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403630 (owner: 10Hashar) [12:26:21] (03Merged) 10jenkins-bot: Migrate mwgate-rake to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403630 (owner: 10Hashar) [12:31:09] (03PS1) 10Hashar: Delete mwgate-rake-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/403631 [12:35:14] (03PS1) 10Hashar: Clean out rake-jessie legacy bits [integration/config] - 10https://gerrit.wikimedia.org/r/403632 [12:36:23] (03PS2) 10Hashar: Clean out rake-jessie legacy bits [integration/config] - 10https://gerrit.wikimedia.org/r/403632 [12:36:25] (03CR) 10Hashar: [C: 032] Delete mwgate-rake-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/403631 (owner: 10Hashar) [12:37:43] (03Merged) 10jenkins-bot: Delete mwgate-rake-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/403631 (owner: 10Hashar) [12:38:19] (03CR) 10Hashar: [C: 032] Clean out rake-jessie legacy bits [integration/config] - 10https://gerrit.wikimedia.org/r/403632 (owner: 10Hashar) [12:39:40] (03Merged) 10jenkins-bot: Clean out rake-jessie legacy bits [integration/config] - 10https://gerrit.wikimedia.org/r/403632 (owner: 10Hashar) [13:03:00] (03PS1) 10WMDE-Fisch: Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 [13:05:28] (03PS2) 10WMDE-Fisch: Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 (https://phabricator.wikimedia.org/T180246) [13:13:41] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3893158 (10Paladox) Ok, thanks. [13:19:35] (03PS1) 10Hashar: Migrate composer to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403642 [13:20:47] (03PS2) 10Hashar: Migrate composer to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403642 (https://phabricator.wikimedia.org/T177276) [13:24:45] (03CR) 10Zfilipin: [C: 031] Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 (https://phabricator.wikimedia.org/T180246) (owner: 10WMDE-Fisch) [13:31:36] (03PS1) 10Zfilipin: Run qunit-selenium job for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/403646 (https://phabricator.wikimedia.org/T171848) [13:34:42] (03PS1) 10Hashar: Migrate composer-package to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403647 (https://phabricator.wikimedia.org/T177276) [13:34:53] (03CR) 10Hashar: [C: 032] Migrate composer to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403642 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:36:46] (03Merged) 10jenkins-bot: Migrate composer to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403642 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:37:12] (03CR) 10Hashar: [C: 032] Migrate composer-package to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403647 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:39:39] (03Merged) 10jenkins-bot: Migrate composer-package to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403647 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:45:37] !log Migrate composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403647 [13:45:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:00:57] (03PS1) 10Hashar: Migrate composer-test to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403654 (https://phabricator.wikimedia.org/T177276) [14:01:13] (03CR) 10Hashar: [C: 032] Migrate composer-test to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403654 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:03:39] (03Merged) 10jenkins-bot: Migrate composer-test to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403654 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:05:11] !log Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403654 [14:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:11] (03PS1) 10Hashar: Use composer images from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403659 [14:46:23] (03CR) 10Hashar: [C: 032] Use composer images from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403659 (owner: 10Hashar) [14:47:44] (03Merged) 10jenkins-bot: Use composer images from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403659 (owner: 10Hashar) [14:48:37] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3893464 (10Addshore) [14:48:48] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [14:48:57] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [14:50:20] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [15:00:50] (03PS1) 10Hashar: Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403666 [15:00:52] (03PS1) 10Hashar: Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403667 [15:01:35] (03Abandoned) 10Hashar: Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403667 (owner: 10Hashar) [15:01:52] (03PS2) 10Hashar: Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403666 [15:02:02] (03CR) 10Hashar: [C: 032] Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403666 (owner: 10Hashar) [15:02:46] I'm assuming there's a MediaWiki train, but to be on the safe side, for Tech News purposes -- lack information about next week on https://wikitech.wikimedia.org/wiki/Deployments is merely because it hasn't been updated? [15:03:48] (03Merged) 10jenkins-bot: Rebuild using composer from docker-registry [integration/config] - 10https://gerrit.wikimedia.org/r/403666 (owner: 10Hashar) [15:05:42] JohanJ: there is indeed! [15:06:13] (as far as I understand) [15:06:31] (03PS1) 10Hashar: Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 [15:06:51] (03CR) 10Hashar: [C: 032] Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 (owner: 10Hashar) [15:08:06] (03PS2) 10Hashar: Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 [15:09:20] (03PS3) 10Hashar: Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 [15:09:31] (03CR) 10Hashar: [C: 032] Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 (owner: 10Hashar) [15:11:09] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [15:13:01] (03Merged) 10jenkins-bot: Rebuild ci-src-setup with composer from our Docker registry [integration/config] - 10https://gerrit.wikimedia.org/r/403672 (owner: 10Hashar) [15:15:51] (03PS1) 10Hashar: Drop cache busters from composer-hhvm/composer-php55 [integration/config] - 10https://gerrit.wikimedia.org/r/403676 [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:54:38] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#3893742 (10zeljkofilipin) [15:58:19] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#3893764 (10zeljkofilipin) [15:59:48] Anyone feel like debugging a freak case of php-cs failure? https://gerrit.wikimedia.org/r/#/c/403663/2 [16:00:09] php-cs insists that I remove a “use” statement which is in fact necessary. [16:02:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#3893742 (10zeljkofilipin) a:05zeljkofilipin>03None [16:02:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Continuous integration for mediawiki-vagrant - https://phabricator.wikimedia.org/T183456#3893783 (10zeljkofilipin) a:05zeljkofilipin>03None [16:05:03] Figured it out, this was caused by a silent rebase. [16:07:16] (03PS1) 10Zfilipin: Run qunit-selenium Jenkins job for ORES [integration/config] - 10https://gerrit.wikimedia.org/r/403688 (https://phabricator.wikimedia.org/T184451) [16:08:33] (03CR) 10Awight: [C: 031] "Right on, thanks for the help!" [integration/config] - 10https://gerrit.wikimedia.org/r/403688 (https://phabricator.wikimedia.org/T184451) (owner: 10Zfilipin) [16:08:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Continuous integration for mediawiki-vagrant - https://phabricator.wikimedia.org/T183456#3893811 (10zeljkofilipin) a:03zeljkofilipin [16:09:36] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3893825 (10zeljkofilipin) a:03zeljkofilipin [16:18:32] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [16:39:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Continuous integration for mediawiki-vagrant - https://phabricator.wikimedia.org/T183456#3893934 (10zeljkofilipin) >>! In T183456#3857197, @bd808 wrote: > Clean Puppet runs certainly do... [16:52:32] awight moved the disccusion here :). [17:12:36] awight https://gerrit.git.wmflabs.org/r/login/ looks better [17:12:37] :) [17:13:24] Woohoo! nice work paladox [17:13:28] :) [17:21:14] is there perhaps a networking difference between integration-slave-100[1-3] and 6,7? My search-mjolnir-tox-docker jobs all time out at 1-3, probably because it's trying to grab a jar from archiva.wikimedia.org, whereas 6 and 7 seem to pass [17:22:05] they also have a local cache of jars, which typically avoids contacting archiva, but a dep was updated and they have to pull the new jar [17:25:18] ebernhardson: file a task for that one, hasharAway is probably the best to answer [17:25:56] sure [17:41:02] paladox: I’m happy to give my lowly +1 when the patch is reworked with the latest, of course... [17:41:09] awight https://gerrit.wikimedia.org/r/#/c/402665/ [17:47:23] (03CR) 10Jdlrobson: [C: 031] "Can I +2 this or is there an "owner" who does +2ing for this repo?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [17:52:38] awight could you +1 again please :) [17:52:47] i added margin-top to center it to the middle [17:52:54] sure thing [17:53:05] needs a rebase, btw [17:54:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Scoring-platform-team (Current): What to do with deployment-sca03? - https://phabricator.wikimedia.org/T184501#3894240 (10Nuria) [17:55:01] awight thanks, this is also what it looks like if you enter incorrect details: https://phabricator.wikimedia.org/F12481452 [17:55:02] :) [18:35:32] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3894383 (10Aklapper) [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:29] PROBLEM - Free space - all mounts on deployment-kafka03 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<100.00%) [18:58:56] PROBLEM - Puppet staleness on deployment-kafka03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [19:38:37] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3894680 (10Catrope) Still not fixed in production. [19:50:01] 10Scap, 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3432960 (10Platonides) I don't think the file license would be compatible with scaps run by WMF employees. Could be used by those run by volunteers, though. [20:02:17] ebernhardson: yup task fill it please. I guess the job has a default timeout of 3 minutes and whenever a new dep is added it takes too long to download all the deps tree/compile etc [20:02:36] ebernhardson: the job template might support a {build_timeout} variable to let us raise it up [20:02:41] hasharAway: i filed a task, maybe in wrong tags: T184754 [20:02:44] T184754: many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754 [20:02:51] good [20:02:56] will look at it in a few [20:03:15] I am writing (literally) some invitation cards [20:03:16] hasharAway: i suspect thugh that the timeouts arn't from transfering too much data, its a couple tens of MB over the internal network from archiva.wikimedia.org to the slaves. Something else (firewalls?) seems to be happening [20:09:13] ebernhardson: if in doubt: blame the infrastructure: dns > switch > cables :] [20:09:28] 10Release-Engineering-Team (Kanban): Explain to TechComm (Daniel K) part of learnings from ORES post-mortem re arch reviews - https://phabricator.wikimedia.org/T182635#3894838 (10Jrbranaa) Sent off email to @daniel before the holidays with some observations and questions regarding the review queue process as it... [20:09:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894846 (10hashar) [20:09:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894385 (10hashar) a:03hashar [20:10:42] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3894849 (10mmodell) Rolling back group 1 to wmf.15 due to {T184749} [20:13:52] PROBLEM - Host integration-slave-jessie-1004 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [20:13:54] PROBLEM - Host deployment-netbox is DOWN: CRITICAL - Host Unreachable (10.68.19.203) [20:14:02] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3894862 (10mmodell) [20:14:16] PROBLEM - Host integration-slave-docker-1004 is DOWN: CRITICAL - Host Unreachable (10.68.16.233) [20:14:26] PROBLEM - Host integration-slave-jessie-1003 is DOWN: CRITICAL - Host Unreachable (10.68.17.164) [20:14:43] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10mmodell) [20:15:06] PROBLEM - Host deployment-redis05 is DOWN: CRITICAL - Host Unreachable (10.68.19.242) [20:15:12] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10mmodell) [20:15:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3894869 (10Addshore) [20:15:33] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [20:15:38] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [20:15:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10Addshore) [20:15:57] PROBLEM - Host integration-slave-docker-1005 is DOWN: CRITICAL - Host Unreachable (10.68.21.12) [20:16:04] PROBLEM - Host deployment-kafka-jumbo-1 is DOWN: CRITICAL - Host Unreachable (10.68.23.243) [20:16:06] PROBLEM - Host deployment-cpjobqueue is DOWN: CRITICAL - Host Unreachable (10.68.22.161) [20:16:08] hmmmm [20:17:33] PROBLEM - Host deployment-cassandra3-02 is DOWN: CRITICAL - Host Unreachable (10.68.21.237) [20:19:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894891 (10hashar) https://gerrit.wikimedia.org/r/#/c/403334/ introduces the python module `xgboost` which takes a while to compile. It depends... [20:19:46] (03PS1) 10Hashar: Bump timeout for search-mjolnir-tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/403746 (https://phabricator.wikimedia.org/T184754) [20:20:16] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894905 (10EBernhardson) Ahh, i was only thinking JVM dependencies and didn't remember the new python dep as well. Makes s... [20:20:19] ebernhardson: I guess 10 minutes will be fine? https://gerrit.wikimedia.org/r/#/c/403746/1/jjb/search.yaml :) [20:21:26] hashar: probably? Worth trying at least [20:21:30] deal [20:21:42] (03CR) 10Hashar: [C: 032] "Updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/403746 (https://phabricator.wikimedia.org/T184754) (owner: 10Hashar) [20:22:06] RECOVERY - Host deployment-redis05 is UP: PING OK - Packet loss = 0%, RTA = 394.85 ms [20:22:10] RECOVERY - Host deployment-kafka-jumbo-1 is UP: PING OK - Packet loss = 0%, RTA = 1.54 ms [20:22:26] RECOVERY - Host deployment-cassandra3-02 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [20:22:33] ebernhardson: the reason the run time is different is because the instance can run several build concurrently [20:22:49] and they are on labs so the instances CPU is shared with other random instances that happen to run on the same openstack node [20:22:52] hashar: makes sense, so sometimes they make it through the compile at 6 minutes which makes the machine pass for future jobs [20:23:02] so it is not necessarly randomy, but it is pretty undertermenistic [20:23:04] RECOVERY - Host deployment-netbox is UP: PING OK - Packet loss = 0%, RTA = 1.68 ms [20:23:04] RECOVERY - Host integration-slave-jessie-1003 is UP: PING OK - Packet loss = 0%, RTA = 8.75 ms [20:23:10] RECOVERY - Host deployment-cpjobqueue is UP: PING OK - Packet loss = 0%, RTA = 4.11 ms [20:23:12] yup [20:23:23] RECOVERY - Host integration-slave-docker-1004 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [20:23:24] the timeout is merely around to avoid death loop of some sorts [20:23:52] sure, don't want jobs accidently spinning the cpu for an hour because they got stuck in a loop [20:24:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894906 (10hashar) 05Open>03Resolved Once the change that introduces xdgboost is merged, the wheel will be saved in th... [20:24:41] RECOVERY - Host integration-slave-docker-1005 is UP: PING OK - Packet loss = 0%, RTA = 2.32 ms [20:24:41] I did a recheck and it is running on https://integration.wikimedia.org/ci/job/search-mjolnir-tox-docker/83/console [20:24:56] and maybe I can use a trick to get the cache populated [20:25:19] RECOVERY - Host integration-slave-jessie-1004 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [20:25:25] (03Merged) 10jenkins-bot: Bump timeout for search-mjolnir-tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/403746 (https://phabricator.wikimedia.org/T184754) (owner: 10Hashar) [20:27:41] hashar: that looks to have run on 1002 which previously passed, i've seen passes on 1002 and 1006 [20:28:32] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3894924 (10greg) https://lists.wikimedia.org/pipermail/wikitech-l/2018-January/089399.html [20:28:33] I guess, if another heavy CPU build happened to run at the same time as your build, that would cause the compilation of xgboost to be slower and reach the timeout [20:30:16] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [20:32:56] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [20:32:58] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [20:35:15] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [20:36:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754#3894943 (10hashar) I have also manually run the job with the parameter ZUUL_PIPELINE=postmerge which triggers a save of t... [20:37:57] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [20:54:26] PROBLEM - Host deployment-mx is DOWN: CRITICAL - Host Unreachable (10.68.17.78) [20:54:29] PROBLEM - Host deployment-memc05 is DOWN: CRITICAL - Host Unreachable (10.68.23.49) [20:54:33] PROBLEM - Host deployment-sca01 is DOWN: CRITICAL - Host Unreachable (10.68.20.183) [20:54:45] PROBLEM - Host deployment-sca03 is DOWN: CRITICAL - Host Unreachable (10.68.21.183) [20:56:07] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [20:56:29] weird ^ [20:56:50] 20:54:48 mutante | icinga says toolserver is down [20:58:15] greg-g: sounds like a labvirt went down [21:00:24] RECOVERY - Host deployment-mx is UP: PING OK - Packet loss = 0%, RTA = 83.03 ms [21:00:35] chasemp | greg-g: labvirt reboots happening -- see cloud-l notice [21:01:12] RECOVERY - Host deployment-redis01 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [21:01:38] RECOVERY - Host deployment-sca01 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [21:02:07] RECOVERY - Host deployment-memc05 is UP: PING OK - Packet loss = 0%, RTA = 3.83 ms [21:04:21] RECOVERY - Host deployment-sca03 is UP: PING OK - Packet loss = 0%, RTA = 1.43 ms [21:06:15] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%) [21:07:31] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [21:08:51] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:10:06] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [21:12:32] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:15:04] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:27:07] 10Continuous-Integration-Infrastructure, 10Fr-tech-archived-from-FY-14/15, 10Wikimedia-Fundraising-CiviCRM, 10Fundraising Sprint Grandmaster Flash: Mysterious failure to zuul-clone drupal repo - https://phabricator.wikimedia.org/T93707#3895096 (10DStrine) [21:27:28] 10Continuous-Integration-Infrastructure, 10Fr-tech-archived-from-FY-14/15, 10Wikimedia-Fundraising-CiviCRM, 10Fundraising Sprint Grandmaster Flash: Make Civi CI job run on civicrm, drupal, and vendor (DonationInterface and SmashPig) repos - https://phabricator.wikimedia.org/T91905#3895106 (10DStrine) [21:28:08] 10Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Fr-tech-archived-from-FY-14/15, and 9 others: Run CiviCRM testing scripts during CI - https://phabricator.wikimedia.org/T89896#3895126 (10DStrine) [21:28:11] 10Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Fr-tech-archived-from-FY-14/15, and 5 others: Configure Jenkins to run CiviCRM builds on Fundraising CI slave instance - https://phabricator.wikimedia.org/T89895#3895127 (10DStrine) [21:29:34] 10Blocked-on-RelEng, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Fr-tech-archived-from-FY-2014/15, and 8 others: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#3895162 (10DStrine) [21:35:55] 10Continuous-Integration-Config, 10Fr-tech-archived-from-FY-2014/15, 10Wikimedia-Fundraising-CiviCRM, 10Fundraising Sprint James Brown, 10Patch-For-Review: Disable CI job on CRM deployment branch - https://phabricator.wikimedia.org/T94586#3895222 (10DStrine) [21:36:36] 10RelEng-Archive-FY201718-Q1, 10Fr-tech-archived-from-FY-2014/15, 10MediaWiki-Vagrant, 10Wikimedia-Fundraising, and 3 others: Vagrant Fundraising role needs to be able to run a specific MediaWiki branch - https://phabricator.wikimedia.org/T78739#3895245 (10DStrine) [21:39:51] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10Fr-tech-archived-from-FY-2015/16, and 4 others: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#3895273 (10DStrine) [21:47:16] 10Continuous-Integration-Config, 10RelEng-Archive-FY201718-Q1, 10Fr-tech-archived-from-FY-2015/16, 10WorkType-Maintenance: wikimedia/fundraising/tools CI jobs are broken - https://phabricator.wikimedia.org/T117818#3895538 (10DStrine) [21:47:59] 10Continuous-Integration-Config, 10Fr-tech-archived-from-FY-2015/16, 10Fundraising Sprint William Shatner, 10Patch-For-Review, 10WorkType-Maintenance: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock o... - https://phabricator.wikimedia.org/T117062#3895561 [21:48:52] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:51:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3857222 (10Dzahn) I uploaded the files from https://people.wikimedia.org/~hashar/debs/node-tunnel-agent_0.... [22:00:44] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Dzahn) I uploaded the files from https://people.wikimedia.org/~hashar/debs/zuul_2.5.0-8-gcbc7f62-wmf6/ ``` reprepro -C main include jessie-wi... [22:11:39] 10Continuous-Integration-Config, 10RemexHtml, 10Patch-For-Review: Figure out how to speed up RemexHtml coverage runs - https://phabricator.wikimedia.org/T179055#3895718 (10Krinkle) [22:28:40] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:28:57] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:29:55] 10Scap, 10Operations, 10Packaging: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3895753 (10mmodell) p:05Triage>03High [22:31:05] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:31:26] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3895773 (10mmodell) [22:31:53] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3895753 (10mmodell) a:05akosiaris>03None [22:32:05] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:32:58] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:33:39] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:35:09] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:36:11] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:36:35] 10Release-Engineering-Team, 10MediaWiki-extensions-ORES, 10Scoring-platform-team: How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#3895786 (10awight) [22:36:55] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:37:56] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:38:02] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:39:50] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:40:48] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:43:03] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:43:31] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:43:59] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:45:33] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:46:05] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:46:55] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3895823 (10mmodell) cc: @fgiunchedi [22:47:16] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review, 10Release: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3895825 (10mmodell) [22:51:54] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:51:55] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:51:55] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:51:55] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:52:27] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:53:03] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:53:25] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:53:33] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:53:42] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:54:02] PROBLEM - Puppet errors on deployment-eventlog02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:57:27] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:57:29] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:59:51] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:03:44] 10Deployments: Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488#3895850 (10mmodell) p:05Normal>03Low [23:09:41] * apergos raises an eyebrow [23:26:16] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [23:28:03] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:09] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:43:47] 10Scap: cache_git_info (from e.g. scap sync-file) is way way too verbose - https://phabricator.wikimedia.org/T182643#3829796 (10greg) See {T184774}