[00:09:51] Project beta-scap-eqiad build #207815: 04STILL FAILING in 6 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207815/ [00:20:00] Project beta-scap-eqiad build #207816: 04STILL FAILING in 6 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207816/ [00:30:02] Project beta-scap-eqiad build #207817: 04STILL FAILING in 6 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207817/ [00:31:18] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:39:59] Project beta-scap-eqiad build #207818: 04STILL FAILING in 6 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207818/ [00:49:43] Project beta-scap-eqiad build #207819: 04STILL FAILING in 6 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207819/ [00:59:47] Project beta-scap-eqiad build #207820: 04STILL FAILING in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207820/ [01:09:55] Project beta-scap-eqiad build #207821: 04STILL FAILING in 6 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207821/ [01:17:39] (03PS1) 10Legoktm: Spell workspace properly for mwext-phpunit-coverage-patch [integration/config] - 10https://gerrit.wikimedia.org/r/433303 (https://phabricator.wikimedia.org/T194206) [01:19:51] Project beta-scap-eqiad build #207822: 04STILL FAILING in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207822/ [01:21:43] Um [01:21:49] Is it out of space? [01:22:12] Cc legoktm or thcipriani or twentyafterfour ^^ [01:22:29] is what out of space? [01:22:53] deployment-mediawiki-07 [01:23:00] Scap is failing on that host [01:23:08] * legoktm checks [01:23:43] Thanks :) [01:23:52] doesn't look full [01:23:57] /dev/vda3 19G 16G 1.8G 91% / [01:23:57] /dev/mapper/vd-second--local--disk 60G 15G 42G 27% /srv [01:24:00] Ok hmm [01:24:09] Could it be the known_host thing? [01:24:11] 01:19:18 Job ['/usr/bin/scap', 'pull', '--no-update-l10n', 'deployment-mira.deployment-prep.eqiad.wmflabs', 'deployment-tin.deployment-prep.eqiad.wmflabs', 'deployment-tin.deployment-prep.eqiad.wmflabs'] called with an empty host list. [01:24:18] I'm not that familiar with scap [01:24:45] Ok [01:26:29] (03CR) 10Legoktm: [C: 032] Spell workspace properly for mwext-phpunit-coverage-patch [integration/config] - 10https://gerrit.wikimedia.org/r/433303 (https://phabricator.wikimedia.org/T194206) (owner: 10Legoktm) [01:28:11] (03Merged) 10jenkins-bot: Spell workspace properly for mwext-phpunit-coverage-patch [integration/config] - 10https://gerrit.wikimedia.org/r/433303 (https://phabricator.wikimedia.org/T194206) (owner: 10Legoktm) [01:30:20] Project beta-scap-eqiad build #207823: 04STILL FAILING in 6 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207823/ [01:39:43] Project beta-scap-eqiad build #207824: 04STILL FAILING in 5 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207824/ [01:49:53] Project beta-scap-eqiad build #207825: 04STILL FAILING in 6 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207825/ [01:59:59] Project beta-scap-eqiad build #207826: 04STILL FAILING in 6 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207826/ [02:03:45] (03PS1) 10Legoktm: Use mediawiki/phpunit-patch-coverage 0.0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/433305 [02:05:47] (03PS1) 10Legoktm: Add timed job for audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433306 [02:05:49] (03CR) 10Legoktm: [C: 032] Use mediawiki/phpunit-patch-coverage 0.0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/433305 (owner: 10Legoktm) [02:06:00] (03CR) 10Legoktm: [C: 032] Add timed job for audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433306 (owner: 10Legoktm) [02:07:32] (03Merged) 10jenkins-bot: Use mediawiki/phpunit-patch-coverage 0.0.8 [integration/config] - 10https://gerrit.wikimedia.org/r/433305 (owner: 10Legoktm) [02:08:03] (03Merged) 10jenkins-bot: Add timed job for audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433306 (owner: 10Legoktm) [02:16:53] Project beta-scap-eqiad build #207827: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207827/ [02:24:39] Project beta-scap-eqiad build #207828: 04STILL FAILING in 7 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207828/ [02:24:50] (03PS1) 10Legoktm: Whitelist plain wikimedia.org [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433307 [02:25:00] (03CR) 10Legoktm: [C: 032] Whitelist plain wikimedia.org [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433307 (owner: 10Legoktm) [02:29:07] (03CR) 10Legoktm: [V: 032 C: 032] Whitelist plain wikimedia.org [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433307 (owner: 10Legoktm) [02:29:57] (03PS1) 10Legoktm: Configure CI for integration/audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433308 [02:30:13] (03CR) 10Legoktm: [C: 032] Configure CI for integration/audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433308 (owner: 10Legoktm) [02:31:48] Project beta-scap-eqiad build #207829: 04STILL FAILING in 6 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207829/ [02:32:16] (03Merged) 10jenkins-bot: Configure CI for integration/audit-resources [integration/config] - 10https://gerrit.wikimedia.org/r/433308 (owner: 10Legoktm) [02:32:47] !log deployed https://gerrit.wikimedia.org/r/433308 [02:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:33:00] (03CR) 10Legoktm: [C: 032] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/433306 (owner: 10Legoktm) [02:33:23] (03CR) 10Legoktm: [V: 032 C: 032] "recheck" [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433307 (owner: 10Legoktm) [02:34:30] (03PS1) 10Legoktm: Force mypy to run on Python 3 [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433309 [02:35:55] (03CR) 10Legoktm: [C: 032] Force mypy to run on Python 3 [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433309 (owner: 10Legoktm) [02:36:31] (03Merged) 10jenkins-bot: Force mypy to run on Python 3 [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/433309 (owner: 10Legoktm) [02:39:42] Project beta-scap-eqiad build #207830: 04STILL FAILING in 6 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207830/ [02:49:41] Project beta-scap-eqiad build #207831: 04STILL FAILING in 6 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207831/ [02:59:36] Project beta-scap-eqiad build #207832: 04STILL FAILING in 5 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207832/ [03:09:46] Project beta-scap-eqiad build #207833: 04STILL FAILING in 6 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207833/ [03:19:43] Project beta-scap-eqiad build #207834: 04STILL FAILING in 6 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207834/ [03:29:58] Project beta-scap-eqiad build #207835: 04STILL FAILING in 6 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207835/ [03:39:36] Project beta-scap-eqiad build #207836: 04STILL FAILING in 5 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207836/ [03:49:34] Project beta-scap-eqiad build #207837: 04STILL FAILING in 5 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207837/ [03:59:32] Project beta-scap-eqiad build #207838: 04STILL FAILING in 5 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207838/ [04:09:36] Project beta-scap-eqiad build #207839: 04STILL FAILING in 5 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207839/ [04:19:36] Project beta-scap-eqiad build #207840: 04STILL FAILING in 5 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207840/ [04:29:46] Project beta-scap-eqiad build #207841: 04STILL FAILING in 6 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207841/ [04:39:36] Project beta-scap-eqiad build #207842: 04STILL FAILING in 5 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207842/ [04:49:55] Project beta-scap-eqiad build #207843: 04STILL FAILING in 6 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207843/ [04:59:51] Project beta-scap-eqiad build #207844: 04STILL FAILING in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207844/ [05:09:51] Project beta-scap-eqiad build #207845: 04STILL FAILING in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207845/ [05:20:00] Project beta-scap-eqiad build #207846: 04STILL FAILING in 6 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207846/ [05:26:33] Project beta-scap-eqiad build #207847: 04STILL FAILING in 6 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207847/ [05:33:33] Project beta-scap-eqiad build #207848: 04STILL FAILING in 6 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207848/ [05:40:23] Project beta-scap-eqiad build #207849: 04STILL FAILING in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207849/ [05:46:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [05:50:03] Project beta-scap-eqiad build #207850: 04STILL FAILING in 6 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207850/ [06:00:11] Project beta-scap-eqiad build #207851: 04STILL FAILING in 6 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207851/ [06:10:10] Project beta-scap-eqiad build #207852: 04STILL FAILING in 6 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207852/ [06:19:53] Project beta-scap-eqiad build #207853: 04STILL FAILING in 6 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207853/ [06:29:53] Yippee, build fixed! [06:29:54] Project beta-scap-eqiad build #207854: 09FIXED in 6 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207854/ [06:51:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:09:06] !log removed shadow mwdeploy users on deployment-mediawiki-07 [07:09:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:31:53] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4209500 (10hashar) [09:42:50] (03PS1) 10Giuseppe Lavagetto: cergen: update setuptools version for both python2 and python3 [integration/config] - 10https://gerrit.wikimedia.org/r/433342 [09:52:25] (03CR) 10Volans: "Question/doubt inline, LGTM otherwise." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/433342 (owner: 10Giuseppe Lavagetto) [09:55:53] 10Continuous-Integration-Config, 10Zuul: postmerge job for mediawiki/services/parsoid stuck in zuul for > 70 hours - https://phabricator.wikimedia.org/T194573#4209630 (10MarcoAurelio) Thanks. [10:34:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673#4209747 (10hashar) [10:34:44] _joe_: thcipriani: for cergen / python_requires issue ( T194673 ) We had the same issue on another repository a few days ago [10:34:46] T194673: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673 [10:35:03] <_joe_> hashar: yeah maybe it could be done in tox [10:35:08] gotta dig the exact root cause. One of the transient python module dependency relies on a more recent version of setuptools [10:35:09] <_joe_> btw lemme finish that pathc [10:35:30] <_joe_> hashar: can we unblock me in the meanwhile? [10:35:38] <_joe_> I'll add an entry to the changelog of cergen [10:35:42] nop [10:35:50] rebuild the docker container will not fix it [10:35:56] <_joe_> it will indeed [10:36:04] ah https://gerrit.wikimedia.org/r/#/c/428644/ [10:36:06] <_joe_> yes [10:36:09] <_joe_> sorry ahahah :) [10:36:15] <_joe_> yeah, rebuilding plus that :P [10:36:34] <_joe_> I meant plus https://gerrit.wikimedia.org/r/#/c/433342/ [10:37:26] <_joe_> I can go solve it through cergen [10:38:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673#4205206 (10hashar) The same issue happened on research/recommendation-api.git It is due to setuptools_scm 2.0.0+ which is not compatible with the... [10:39:01] _joe_: yeah and the setuptools bump will solve it :} for releng/tox-cergen that is fine [10:39:13] for the generic releng/tox image, I am not sure what else it is going to break eventually [10:39:21] <_joe_> hashar: yeah lemme fix it inside cergen for now [10:39:41] which would fix it for developers not having the latest setuptols [10:42:46] (03CR) 10Hashar: "That can be done by rebuilding the root image releng/tox and all the images that descend from it :-}" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/433342 (owner: 10Giuseppe Lavagetto) [10:43:07] _joe_: setuptools is installed via pip in the parent image releng/tox https://gerrit.wikimedia.org/r/#/c/433342/1/dockerfiles/tox-cergen/Dockerfile.template [10:43:20] so I guess we can rebuild that parent images and all the child ones. Potentially during the hackathon [10:43:28] <_joe_> hashar: so, I fixed cergen myself for now [10:43:32] I will be too busy this week to catchup / follow pu with the side effects [10:43:43] <_joe_> we need to fix the tox image, with time [10:43:45] <_joe_> I agree [10:43:51] <_joe_> instead of doing the hotfix [10:44:37] so workaround is to do the pinning whenever the issue is encountered [10:44:48] and during the hackathon (or after it) rebuild all the tox images [10:45:17] when it happened on April 24th for the other repo, I haven't investigated much more. I guess I assumed we were just using python-setuptools.deb [10:45:26] or missed the 'pip install setuptools' [10:46:31] thcipriani: ^^^ solved. setuptools is outdated in the CI container . That can be worked around with a pinning: setuptools_scm < 2.0.0 [10:58:28] 10Release-Engineering-Team (Kanban), 10Surveys: Survey for Beta Cluster use cases - https://phabricator.wikimedia.org/T194818#4209819 (10greg) p:05Triage>03High [11:00:06] 10Release-Engineering-Team (Kanban), 10Surveys: Survey for Beta Cluster use cases - https://phabricator.wikimedia.org/T194818#4209830 (10greg) a:03Jrbranaa [11:05:50] 10Release-Engineering-Team (Kanban), 10Surveys: Survey for Beta Cluster use cases - https://phabricator.wikimedia.org/T194818#4209874 (10Jrbranaa) FYI: the Etherpad also contains the copy for our invite email. Which probably also needs to be reviewed. [11:10:23] <_joe_> distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools_scm<2.0.0') [11:10:26] <_joe_> sigh [11:10:31] <_joe_> (╯°□°）╯︵ ┻━┻ [11:10:34] <_joe_> ok I give up [11:17:22] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (<11.11%) [12:16:34] (03PS1) 10Lucas Werkmeister (WMDE): Add Prssanna to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/433363 [12:25:06] (03PS2) 10Lucas Werkmeister (WMDE): Add Prssanna to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/433363 [12:27:19] (03CR) 10Jonas Kress (WMDE): [C: 031] Add Prssanna to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/433363 (owner: 10Lucas Werkmeister (WMDE)) [12:40:33] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:41:42] 10Phabricator, 10Operations, 10Traffic, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#4210035 (10Aklapper) >>! In T174342#3790202, @Mholloway wrote: > I've reached out to Partnerships about getting in touch with Maroc and INWI for IP range updates. @Mholloway:... [13:15:35] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:30:38] Project mwext-phpunit-coverage-publish build #4474: 04FAILURE in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4474/ [13:30:47] Project mwext-phpunit-coverage-publish build #4475: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4475/ [13:31:01] Project mwext-phpunit-coverage-publish build #4476: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4476/ [13:33:44] Yippee, build fixed! [13:33:44] Project mwext-phpunit-coverage-publish build #4477: 09FIXED in 2 min 28 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4477/ [14:26:59] PROBLEM - Puppet errors on deployment-chromium01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:35:45] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [14:53:17] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:54:34] 10Continuous-Integration-Infrastructure, 10GitHub-Mirrors: Travis CI on Wikimedia on Github - https://phabricator.wikimedia.org/T194772#4210269 (10Krinkle) 05Open>03Resolved a:03Krinkle The current use of Travis CI in Wikimedia repositories on GitHub does not require "installation" approval by the gh-org... [14:57:49] I'm about to stop nodepool (and basically the whole cloud network). With luck this will be quick [15:02:35] andrewbogott: I'm going to disable puppet all over [15:02:56] 'k [15:03:08] I already did shinken and labnodepool [15:03:44] ack [15:30:40] Yippee, build fixed! [15:30:41] Project beta-scap-eqiad build #207908: 09FIXED in 6 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/207908/ [15:55:24] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:55:26] PROBLEM - Puppet errors on webperformance is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:56:42] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (<100.00%) [15:57:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<33.33%) [15:57:55] PROBLEM - Puppet errors on integration-slave-docker-1011 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [15:58:09] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [15:58:37] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [15:59:13] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:25] RECOVERY - Puppet errors on webperformance is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:58] RECOVERY - Puppet errors on integration-slave-docker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:10] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:40] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:42:03] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<44.44%) [16:46:00] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:21:01] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:39] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (<11.11%) [17:49:02] (03CR) 1020after4: [C: 032] Run mobileapps-periodic-test hourly regardless of repo changes [integration/config] - 10https://gerrit.wikimedia.org/r/432310 (https://phabricator.wikimedia.org/T177896) (owner: 10Mholloway) [17:50:44] (03Merged) 10jenkins-bot: Run mobileapps-periodic-test hourly regardless of repo changes [integration/config] - 10https://gerrit.wikimedia.org/r/432310 (https://phabricator.wikimedia.org/T177896) (owner: 10Mholloway) [17:51:53] !log deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs T177896 [17:51:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:51:56] T177896: Create a CI task for MCS periodic tests - https://phabricator.wikimedia.org/T177896 [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:43:50] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:27:15] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [21:33:16] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<33.33%) [22:23:38] twentyafterfour: i am planning to merge this https://gerrit.wikimedia.org/r/#/c/433281/ it's becuase of https://phabricator.wikimedia.org/T194724 compiler result looks good http://puppet-compiler.wmflabs.org/11231/phab1001.eqiad.wmnet/ [22:23:51] it should make no difference for the services [22:24:04] but use newer puppet code for systemd [22:35:17] PROBLEM - Puppet errors on deployment-deploy1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:36:46] mutante: sure thing [22:36:53] ok :) [22:37:17] applying on phab2001 first [22:41:25] it passes for me [22:41:27] in puppet [22:43:09] thanks paladox. it also was no-op on phab2001. restarting ssh-phab works [22:43:18] phd is of course always failed there because DB [22:43:25] and aphlict only exists on the active server [22:43:29] :) [22:43:35] also running puppet on phab1001 [22:44:35] no-op in puppet.. that's it, i dont think we need to restart the active phd [22:44:43] puppet didnt touch any files [22:45:13] well, nice. this was the first "conversion" away from base::service_unit [22:45:18] now we can do more of them [22:45:48] -/8 [23:08:06] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:12:57] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0)