[05:24:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:50:38] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%) [06:53:59] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:04:35] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:11:47] (03PS1) 10Legoktm: Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 [07:11:49] (03PS1) 10Legoktm: Use mediawiki-phan-seccheck:0.2.1 image [integration/config] - 10https://gerrit.wikimedia.org/r/433533 [07:12:08] (03CR) 10Legoktm: [C: 032] Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 (owner: 10Legoktm) [07:13:28] (03CR) 10jerkins-bot: [V: 04-1] Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 (owner: 10Legoktm) [07:13:32] (03CR) 10jerkins-bot: [V: 04-1] Use mediawiki-phan-seccheck:0.2.1 image [integration/config] - 10https://gerrit.wikimedia.org/r/433533 (owner: 10Legoktm) [07:13:41] whaa [07:14:13] (03CR) 10jerkins-bot: [V: 04-1] Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 (owner: 10Legoktm) [07:14:33] (03PS2) 10Legoktm: Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 [07:14:35] (03PS2) 10Legoktm: Use mediawiki-phan-seccheck:0.2.1 image [integration/config] - 10https://gerrit.wikimedia.org/r/433533 [07:16:18] (03CR) 10Legoktm: [C: 032] Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 (owner: 10Legoktm) [07:17:53] (03Merged) 10jenkins-bot: Allow overriding seccheck version if an env var is set [integration/config] - 10https://gerrit.wikimedia.org/r/433532 (owner: 10Legoktm) [07:18:25] !log deploying mediawiki-phan-seccheck:0.2.1 image [07:18:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:26:22] (03CR) 10Legoktm: [C: 032] Use mediawiki-phan-seccheck:0.2.1 image [integration/config] - 10https://gerrit.wikimedia.org/r/433533 (owner: 10Legoktm) [07:28:01] (03Merged) 10jenkins-bot: Use mediawiki-phan-seccheck:0.2.1 image [integration/config] - 10https://gerrit.wikimedia.org/r/433533 (owner: 10Legoktm) [07:34:00] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:24:36] https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/3338/console [09:24:41] 09:23:21 rsync: failed to set times on "/cache/.": Operation not permitted (1) [09:24:41] 09:23:23 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2] [12:38:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:36:33] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:49:44] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:50:51] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:41] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:57:03] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [16:24:08] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:44] PROBLEM - SSH on integration-slave-docker-1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:19] PROBLEM - SSH on integration-slave-docker-1013 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:49] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:41] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [16:43:36] 10Release-Engineering-Team (Watching / External), 10Operations: rename naos to deploy2001 and reinstall with stretch - https://phabricator.wikimedia.org/T193916#4212223 (10Dzahn) [16:43:56] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [16:45:56] 10Release-Engineering-Team (Watching / External), 10Operations: rename naos to deploy2001 and reinstall with stretch - https://phabricator.wikimedia.org/T193916#4212226 (10Dzahn) merged in duplicate task. checked naos in racktables. it's still under warranty until 2019-03 [16:49:09] RECOVERY - SSH on integration-slave-docker-1013 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [16:55:39] 10Release-Engineering-Team (Watching / External), 10Operations: rename naos to deploy2001 and reinstall with stretch - https://phabricator.wikimedia.org/T193916#4212246 (10RobH) Please ensure when the rename is done, a sub-task for the on-site (@papaul) is created in #ops-codfw for him to update the hostname p... [17:20:33] RECOVERY - SSH on integration-slave-docker-1017 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [17:24:49] (03PS1) 10Legoktm: Set SECCHECK_VERSION parameter in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/433611 [17:31:50] (03CR) 10Legoktm: [C: 04-1] "This looks like it's going to end up duplicating all the parameters?? Hrmmm..." [integration/config] - 10https://gerrit.wikimedia.org/r/433611 (owner: 10Legoktm) [17:38:03] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<55.56%) [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:32:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4212446 (10mmodell) [19:32:29] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4212447 (10mmodell) [19:46:05] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4212475 (10mmodell) [19:47:22] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4092224 (10mmodell) [21:21:38] 10Gerrit, 10Cloud-Services, 10Pywikibot-core, 10Toolforge: Fresh clone of pywikibot from gerrit fails with error: RPC failed; result=56, HTTP code = 200 on Toollabs (NFS) - https://phabricator.wikimedia.org/T151351#2814978 (10Dvorapa) I can not reproduce this anymore. Is this still valid? [21:38:48] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/BlueSpiceEchoConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/433675 [21:38:50] (03CR) 10QChris: [V: 031 C: 032] Allow “Gerrit Managers” to import history [extensions/BlueSpiceEchoConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/433675 (owner: 10QChris) [21:41:38] (03PS1) 10QChris: Import done. Revoke import grants [extensions/BlueSpiceEchoConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/433676 [21:41:40] (03CR) 10QChris: [V: 031 C: 032] Import done. Revoke import grants [extensions/BlueSpiceEchoConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/433676 (owner: 10QChris) [21:57:09] anyone here an admin on meta.beta? [22:15:31] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:15:43] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:16:21] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:16:45] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:17:46] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:56:09] tzatziki: yes hello [22:56:17] legoktm: sorted! [22:56:18] thanks :)\ [23:25:36] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4212712 (10mmodell) [23:31:40] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (<44.44%) [23:31:59] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4212717 (10mmodell)