[00:00:29] (03PS2) 10Krinkle: cover: Fix cover-extension breadcrumbs to work on different mount paths [integration/docroot] - 10https://gerrit.wikimedia.org/r/405986 [00:01:19] (03PS4) 10Krinkle: cover: Add sort=cover parameter to coverage report [integration/docroot] - 10https://gerrit.wikimedia.org/r/405996 [00:11:38] 10RelEng-Archive-FY201718-Q2, 10Release, 10Train Deployments: 1.31.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T180250#3921230 (10greg) [00:11:40] 10RelEng-Archive-FY201718-Q2, 10Release, 10Train Deployments: 1.31.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T180244#3921231 (10greg) [00:11:43] 10RelEng-Archive-FY201718-Q2, 10Release, 10Train Deployments: 1.31.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T178637#3921232 (10greg) [00:31:16] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:56:40] Krinkle: ok, I can test it later probably but feel free to +2 if you did? [01:08:17] K, test in production, got it [01:08:47] * greg-g looks sideways [02:48:39] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [03:54:02] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3921321 (10matmarex) [06:08:35] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:48:36] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:48:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:35:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [07:35:37] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<20.00%) [10:59:07] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review: Generate coverage report for Wikidata extensions - https://phabricator.wikimedia.org/T185211#3909461 (10Lucas_Werkmeister_WMDE) [10:59:47] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review: Generate coverage report for Wikidata extensions - https://phabricator.wikimedia.org/T185211#3909461 (10Lucas_Werkmeister_WMDE) [11:55:12] Project mwext-phpunit-coverage-publish build #247: 04FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/247/ [12:13:59] Yippee, build fixed! [12:14:00] Project mwext-phpunit-coverage-publish build #248: 09FIXED in 2 min 32 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/248/ [12:14:54] good morning :) is someone online who could look into a persistent post-merge build failure of an extension? [12:14:58] perhaps zeljkof? Amir1 told me you live in a European timezone ;) [12:15:01] the mwext-phpunit-coverage-publish job fails for the WikibaseQualityExternalValidation extension (example: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/247/console) [12:15:06] during database update, it can’t find the WikibaseQualityExternalValidationHooks::onCreateSchema function [12:15:09] could it be because the extension isn’t part of the wmf.* deployment branches? [12:15:50] I tried looking into the CI config but as far as I could tell the code should be there (it shows up in the Zuul cloner, at least) [13:20:19] (03CR) 10Phedenskog: Move Internet Explorer tests in WebPageTest to own test files [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:41:15] (03PS4) 10MacFan4000: Change linting types on ZppixBot repo [integration/config] - 10https://gerrit.wikimedia.org/r/405828 [15:52:51] 10MediaWiki-Codesniffer, 10MediaWiki-extensions-WikiLexicalData-or-OmegaWiki: Undefined index: scope_opener in IfElseStructureSniff.php - https://phabricator.wikimedia.org/T185642#3921853 (10Phantom42) [16:33:18] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:21] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10Patch-For-Review, and 3 others: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3921910 (10Ladsgroup) With the patches I made this now way more understandable now: https://integration.wiki... [16:47:12] 10Continuous-Integration-Config, 10Collaboration-Team-Triage, 10StructuredDiscussions, 10SQLite: Flow does not depend upon Echo in CI - https://phabricator.wikimedia.org/T185608#3921912 (10Aklapper) [16:59:50] 10Phabricator, 10Operations: Switch phabricator from using apache to nginx - https://phabricator.wikimedia.org/T185644#3921927 (10Paladox) [17:01:29] 10Gerrit, 10Operations: Switch gerrit from using apache to nginx - https://phabricator.wikimedia.org/T185645#3921946 (10Paladox) [17:08:25] 10Phabricator, 10Operations: Switch phabricator from using apache to nginx - https://phabricator.wikimedia.org/T185644#3921962 (10Paladox) [17:08:35] 10Gerrit, 10Operations: Switch gerrit from using apache to nginx - https://phabricator.wikimedia.org/T185645#3921963 (10Paladox) [17:28:47] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:30:48] 10Release-Engineering-Team (Kanban), 10Wiki-Setup (Close): Close chairwiki - https://phabricator.wikimedia.org/T184961#3901664 (10Schiste) Hey, thank you for the ping. Could you give me access to see what is in it please? Might even save you from moving content to boardwiki actually? [17:52:55] no_justification https://groups.google.com/forum/#!topic/repo-discuss/_BXh5RIkuuM :) [17:54:30] I like my bikesheds orange [17:55:17] no_justification lol [17:55:39] no_justification i also brought my own server https://gerrit.wikisaur.ml/r/ :) [18:02:05] (03CR) 10Krinkle: [C: 032] cover: Fix cover-extension breadcrumbs to work on different mount paths [integration/docroot] - 10https://gerrit.wikimedia.org/r/405986 (owner: 10Krinkle) [18:02:08] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922041 (10Paladox) [18:02:10] (03CR) 10Krinkle: [C: 032] cover: Add sort=cover parameter to coverage report [integration/docroot] - 10https://gerrit.wikimedia.org/r/405996 (owner: 10Krinkle) [18:02:14] (03CR) 10Krinkle: [C: 032] cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 (owner: 10Krinkle) [18:02:47] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922051 (10Paladox) 05Open>03Invalid Chad's dealing with it on irc. [18:02:49] (03Merged) 10jenkins-bot: cover: Fix cover-extension breadcrumbs to work on different mount paths [integration/docroot] - 10https://gerrit.wikimedia.org/r/405986 (owner: 10Krinkle) [18:02:51] (03Merged) 10jenkins-bot: cover: Add sort=cover parameter to coverage report [integration/docroot] - 10https://gerrit.wikimedia.org/r/405996 (owner: 10Krinkle) [18:02:53] (03CR) 10jenkins-bot: cover: Fix cover-extension breadcrumbs to work on different mount paths [integration/docroot] - 10https://gerrit.wikimedia.org/r/405986 (owner: 10Krinkle) [18:02:57] (03CR) 10jenkins-bot: cover: Add sort=cover parameter to coverage report [integration/docroot] - 10https://gerrit.wikimedia.org/r/405996 (owner: 10Krinkle) [18:03:13] legoktm: https://doc.wikimedia.org/cover/?sort=cov [18:10:58] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3922066 (10demon) 05Open>03Invalid That was done on purpose. That repo is for WMF deploys, and we should build the branches using composer (like you would from master). Cf: T172927. [18:14:25] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922041 (10Zoranzoki21) All is ok. For me in Serbia work without problems. [18:17:01] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922073 (10Paladox) Using ssh? [18:18:39] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922077 (10Zoranzoki21) >>! In T185649#3922073, @Paladox wrote: > Using ssh? With all avaiable options for cloning (anon https, noanon https and ssh) [18:19:09] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922078 (10Paladox) Hmm strange as it's affecting us and uk users over ssh. [18:19:47] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922080 (10demon) [18:20:46] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922085 (10Zoranzoki21) >>! In T185649#3922078, @Paladox wrote: > Hmm strange as it's affecting us and uk users over ssh. Note: I using internet from provider telenor. [18:24:05] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922041 (10Lucas_Werkmeister_WMDE) We were experiencing the same problem in the Wikimedia Germany office, but it seems to be working again now (since 18:17 UTC). [18:25:17] 10Gerrit, 10Operations: git cloning is not working with gerrit repos - https://phabricator.wikimedia.org/T185649#3922089 (10Zoranzoki21) I have not had a problem for a long time with this. [18:40:20] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:40:30] 10Gerrit, 10Security: Gerrit DoS - https://phabricator.wikimedia.org/T182756#3922158 (10demon) p:05Triage>03Unbreak! [18:40:58] 10Gerrit, 10Security: Gerrit DoS - https://phabricator.wikimedia.org/T182756#3922174 (10demon) This happened today, so went ahead and pushed a fix (hence the after-the-fact-UBN!) [18:43:27] 10Release-Engineering-Team (Kanban), 10Wiki-Setup (Close): Close chairwiki - https://phabricator.wikimedia.org/T184961#3922194 (10demon) Sure, gimme an e-mail address to send the temporary password to and I can create you an account. [18:44:36] So there was a swat window today, and there is another one tomorrow [18:44:41] should they be on the schedule this week? [18:45:06] No MW Train, but SWATs on Mon/Tue/Wed [18:45:11] but there is one on thursday ;] [18:45:21] 10Gerrit, 10Security: Gerrit DoS - https://phabricator.wikimedia.org/T182756#3833501 (10jcrespo) Separate question from Reliability side, would it be possible in the short future to serve gerrit from more than one machine at a time- is it a stateless service, aside from the database? [18:52:11] 10Gerrit, 10Security: Gerrit DoS - https://phabricator.wikimedia.org/T182756#3922202 (10demon) No, it's not stateless: * Git repositories on disk. We replicate, but it's not instantaneous enough for me to trust dual-writes. * There's a local-disk cache, plus in-memory caches. I'd love to combine the two to use... [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:58] 10Gerrit, 10Security: Gerrit DoS - https://phabricator.wikimedia.org/T182756#3922209 (10jcrespo) Thanks, FYI T176532 should be ready to start working on it (purchase proxies) after this quarter finishes, when all misc database services and the current eqiad proxies have been failed over to new servers and upgr... [18:58:19] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3922216 (10Njk) Thanks for the info. I will update this documentation then, since it sounds like there's some outdated info here. https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries [19:05:36] robh: define thursday [19:06:19] oh yeah i suppose its wednesday for here, nm... [19:06:19] robh: thursday UTC vs Pacific 4pm on Wed [19:06:23] :) [19:06:31] so yeah, i think the first swat was missed today due to the gerrit issue [19:06:48] 10Gerrit, 10Operations: Switch gerrit from using apache to nginx - https://phabricator.wikimedia.org/T185645#3922220 (10demon) 05Open>03declined Apache uses less than 100MB of memory on average, plus Gerrit's heap has plenty of space right now--we don't need the memory headroom. Also, I'd rather move it b... [19:08:51] Krinkle: woot! I thought the tabs were going to change? [19:09:40] robh: such is life, sadly [19:10:50] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3922229 (10Legoktm) 05Invalid>03Open We still need the REL branches though. I guess someone just needs to recreate them? [19:15:22] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:36:09] legoktm: They're slightly smaller than in the screenshot [19:36:13] As we preferred [19:36:39] Krinkle: I meant the "coverage home / mediawiki extensions" nav [19:36:45] legoktm: Hm.. [19:36:47] Indeed [19:36:57] https://gerrit.wikimedia.org/r/#/c/405995/ has a merge conflict [19:36:59] (03PS2) 10Krinkle: cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 [19:37:02] (03PS3) 10Legoktm: cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 (owner: 10Krinkle) [19:37:02] Missed one [19:37:04] Yeah [19:37:09] (03CR) 10Legoktm: [C: 032] cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 (owner: 10Krinkle) [19:37:11] Thanks [19:37:12] heh [19:37:13] :) [19:37:42] (03Merged) 10jenkins-bot: cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 (owner: 10Krinkle) [19:37:48] (03CR) 10jenkins-bot: cover: Convert coverage sub-nav from breadcrumbs to nav-tabs [integration/docroot] - 10https://gerrit.wikimedia.org/r/405995 (owner: 10Krinkle) [19:37:57] Yay, deployed now [19:38:00] https://doc.wikimedia.org/cover/?sort=cov [19:39:05] :D [19:41:20] (03PS3) 10Krinkle: Move Internet Explorer tests in WebPageTest to own test files [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [19:41:32] (03PS4) 10Krinkle: Move Internet Explorer tests in WebPageTest to own test files [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [19:44:09] (03CR) 10Krinkle: "Compiling and deploying.." [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [19:44:46] Krinkle: hmm, https://doc.wikimedia.org/cover/?sort=cov has a rounding bug, codesniffer shows 90% but has warning, Assert shows 90% with success [19:44:52] (03CR) 10Krinkle: [C: 032] Move Internet Explorer tests in WebPageTest to own test files [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [19:46:06] legoktm: He [19:46:24] 90.47% vs 93.24% [19:46:55] (03Merged) 10jenkins-bot: Move Internet Explorer tests in WebPageTest to own test files [integration/config] - 10https://gerrit.wikimedia.org/r/404925 (https://phabricator.wikimedia.org/T175240) (owner: 10Phedenskog) [19:47:19] legoktm: interesting. It does >= 90, not > 90. [19:47:28] So not sure why it would give 90.4 a warning [19:48:07] that's line coverage [19:48:17] we're calculating the PHPUnit way of percentage [19:48:43] And the PHPUnit way is not one of the three ways that PHPUnit outputs? https://doc.wikimedia.org/cover/mediawiki-tools-codesniffer/ [19:54:36] https://phabricator.wikimedia.org/diffusion/CIDR/browse/master/shared/CoveragePage.php;e5eb7bec388ab460fe5db2378f5a0dae43e0a728$152 [19:55:52] legoktm: Aha [19:56:12] legoktm: Well, in any event, passing round() to the getColorlevel should fix it [19:56:21] getLevelColor [19:56:33] yep [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:24:38] 10RelEng-Archive-FY201718-Q1, 10User-greg, 10Wikimedia-Incident: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#3922430 (10Krinkle) [21:27:10] Project mwext-phpunit-coverage-publish build #266: 04FAILURE in 2 min 23 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/266/ [21:27:30] Yippee, build fixed! [21:27:30] Project mwext-phpunit-coverage-publish build #267: 09FIXED in 19 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/267/ [22:36:56] PROBLEM - Host integration-slave-k8s-1002 is DOWN: CRITICAL - Host Unreachable (10.68.23.192) [22:43:22] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), and 4 others: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3922536 (10Ladsgroup) See T184451#3922533 [23:08:04] has cumin completely broken on deployment-prep or am I doing it wrong? [23:08:19] ----- OUTPUT of 'id' ----- [23:08:19] Permission denied (publickey). [23:08:19] ================ [23:08:20] PASS: | | 0% (0/72) [00:02 FAIL: |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100 [23:08:27] % (72/72) [00:02<00:00, 31.10hosts/s] [23:08:39] this is from `sudo cumin '*' id` [23:09:54] Krenair can you ssh into deployment-prep normaly? [23:10:07] yes [23:10:36] ok [23:11:05] remote auth.log shows: [23:11:07] Jan 24 23:10:11 deployment-apertium02 sshd[31761]: Set /proc/self/oom_score_adj to 0 [23:11:08] Jan 24 23:10:11 deployment-apertium02 sshd[31761]: Connection from 10.68.21.105 port 43522 on 10.68.22.254 port 22 [23:11:08] Jan 24 23:10:11 deployment-apertium02 sshd[31761]: Connection closed by 10.68.21.105 [preauth] [23:11:37] guess now I get to figure out how cumin works... [23:12:06] oh there's a keyholder [23:12:53] okay so i've armed that [23:13:28] that did the trick [23:14:50] well okay that was easy enough I guess [23:16:43] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3922582 (10demon) No, we shouldn't need them. If CI is using them for testing, then CI is incorrect. [23:17:13] 10Gerrit: mediawiki/vendor repo has lost all its REL branches - https://phabricator.wikimedia.org/T185618#3922583 (10demon) Fix CI instead of creating branches we don't want or need. [23:37:33] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<50.00%) [23:42:20] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:42:44] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:42:54] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:43:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:44:03] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:44:45] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:45:11] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:45:19] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:45:31] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:47:42] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:47:52] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:47:52] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:48:04] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:49:03] Hmm puppet failures ^^ [23:50:47] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:52:21] Someone should make one of these code stewardship reviews for EducationProgram [23:53:34] also probably DisableAccount [23:54:43] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:12] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:30] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:57:44] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:57:58] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:58:18] Project mwext-phpunit-coverage-publish build #272: 04FAILURE in 2 min 32 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/272/ [23:58:28] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:59:03] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0]