[00:00:25] * Krenair sighs [00:00:33] looks like I might need to rearm keyholder [00:02:49] !log armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase [00:02:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:04:37] RECOVERY - Puppet errors on deployment-deploy02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:09:16] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:34:13] PROBLEM - Free space - all mounts on integration-slave-docker-1026 is CRITICAL: CRITICAL: integration.integration-slave-docker-1026.diskspace.root.byte_percentfree (<22.22%) [00:35:03] ENOSPC [00:35:14] motely on integration-slave-docker-1026 (st [00:35:21] https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php70-docker/432/console [00:36:09] maintenance-disconnect-full-disks build 286 - integration-slave-jessie-1003: OFFLINE due to disk space [00:37:47] ow, that looks fancy [00:39:50] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:39:52] well, I'm not waiting 30min for whatever that awesome thing is to depool this one too [00:40:07] !log Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC) [00:40:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:44:17] RECOVERY - Puppet errors on deployment-deploy01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:47:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge - https://phabricator.wikimedia.org/T179963 (10Krinkle) [00:48:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble docker jobs fails due to disk full - https://phabricator.wikimedia.org/T202457 (10Krinkle) [00:54:12] RECOVERY - Free space - all mounts on integration-slave-docker-1026 is OK: OK: All targets OK [00:54:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble docker jobs fails due to disk full - https://phabricator.wikimedia.org/T202457 (10Krinkle) From PROBLEM - SSH on integration-slave-docker-1009 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:50] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:33:43] RECOVERY - SSH on integration-slave-docker-1009 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u5 (protocol 2.0) [01:36:14] maintenance-disconnect-full-disks build 287 - integration-slave-jessie-1003: OFFLINE due to disk space [01:40:59] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10Krinkle) [01:41:04] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10Krinkle) [01:43:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10Krinkle) @hashar Thanks for finding this. I'm hoping this can be prioritised given it is one of the factors... [01:51:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble docker jobs fails due to disk full - https://phabricator.wikimedia.org/T202457 (10Krinkle) [01:51:16] 10Continuous-Integration-Infrastructure (shipyard): Docker-based Jenkins jobs failing due to "No space left on device" - https://phabricator.wikimedia.org/T189616 (10Krinkle) [01:51:18] 10Continuous-Integration-Infrastructure: Error: file write error (No space left on device) - https://phabricator.wikimedia.org/T189361 (10Krinkle) [02:36:08] maintenance-disconnect-full-disks build 288 - integration-slave-jessie-1003: OFFLINE due to disk space [02:43:03] NICE!!!! [02:46:53] !log legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full [02:46:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:00:01] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [03:19:55] 10Beta-Cluster-Infrastructure, 10Core-Platform-Team: Deploy EU copyright stuff to beta cluster - https://phabricator.wikimedia.org/T203299 (10Legoktm) [03:27:11] (03PS1) 10Legoktm: make-wmf-branch: Add SkinPerPage [tools/release] - 10https://gerrit.wikimedia.org/r/456779 (https://phabricator.wikimedia.org/T203296) [05:00:19] (03PS1) 10Legoktm: seccheck for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/456782 [05:01:46] (03PS2) 10Legoktm: seccheck for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/456782 (https://phabricator.wikimedia.org/T202380) [05:01:58] (03CR) 10Legoktm: [C: 032] seccheck for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/456782 (https://phabricator.wikimedia.org/T202380) (owner: 10Legoktm) [05:04:39] (03Merged) 10jenkins-bot: seccheck for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/456782 (https://phabricator.wikimedia.org/T202380) (owner: 10Legoktm) [05:05:14] !log deployed https://gerrit.wikimedia.org/r/456782 [05:05:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:11:11] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [05:21:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10ArielGlenn) This is waiting for the next SRE meeting for discussion. [05:22:24] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [06:47:15] (03PS1) 10Legoktm: seccheck for Translate [integration/config] - 10https://gerrit.wikimedia.org/r/456812 (https://phabricator.wikimedia.org/T202379) [06:50:11] (03CR) 10Legoktm: [C: 032] seccheck for Translate [integration/config] - 10https://gerrit.wikimedia.org/r/456812 (https://phabricator.wikimedia.org/T202379) (owner: 10Legoktm) [06:51:44] (03Merged) 10jenkins-bot: seccheck for Translate [integration/config] - 10https://gerrit.wikimedia.org/r/456812 (https://phabricator.wikimedia.org/T202379) (owner: 10Legoktm) [06:53:06] !log deployed https://gerrit.wikimedia.org/r/456812 [06:53:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:18:25] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [08:15:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T191065 (10Umherirrender) [11:17:31] 10Project-Admins: Create phabricator project for Extension:Blackout - https://phabricator.wikimedia.org/T203227 (10Aklapper) 05Open>03stalled Setting to stalled per discussion in {T198890} [11:56:31] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Umherirrender) [11:57:08] (03PS1) 10Umherirrender: preg_quote should not be used with 1 argument [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/456896 (https://phabricator.wikimedia.org/T203312) [12:48:39] 10MediaWiki-Codesniffer, 10Patch-For-Review: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Umherirrender) The expaned sniff does not detect the function at location where callables are allowed: ``` array_map( 'preg_quote', array_keys( $config[ApiB... [16:52:10] 10Phabricator: Reduce or eliminate bracketed labels to shorten phabricator emails subjects - https://phabricator.wikimedia.org/T650 (10Aklapper) 05Open>03declined Users who'd like to not see stuff like `[Changed Project Column]` in the subject line of mail notifications can disable "Vary Subjects" under http... [20:25:11] PROBLEM - Free space - all mounts on integration-slave-docker-1026 is CRITICAL: CRITICAL: integration.integration-slave-docker-1026.diskspace.root.byte_percentfree (<11.11%) [20:35:11] RECOVERY - Free space - all mounts on integration-slave-docker-1026 is OK: OK: All targets OK [22:06:20] (03PS1) 10Legoktm: seccheck for EducationProgram [integration/config] - 10https://gerrit.wikimedia.org/r/457068 (https://phabricator.wikimedia.org/T202376) [22:06:54] (03CR) 10Legoktm: [C: 032] seccheck for EducationProgram [integration/config] - 10https://gerrit.wikimedia.org/r/457068 (https://phabricator.wikimedia.org/T202376) (owner: 10Legoktm) [22:08:33] (03Merged) 10jenkins-bot: seccheck for EducationProgram [integration/config] - 10https://gerrit.wikimedia.org/r/457068 (https://phabricator.wikimedia.org/T202376) (owner: 10Legoktm) [22:08:48] !log deploying https://gerrit.wikimedia.org/r/457068 [22:08:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:20:57] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [22:57:48] (03PS1) 10Legoktm: Have quibble jobs mount /tmp as tmpfs [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) [23:29:04] (03PS2) 10Legoktm: Have quibble jobs mount /tmp as tmpfs [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) [23:46:21] (03PS3) 10Legoktm: Have all docker jobs use tmpfs for /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) [23:47:20] (03CR) 10jerkins-bot: [V: 04-1] Have all docker jobs use tmpfs for /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) (owner: 10Legoktm) [23:49:37] (03PS4) 10Legoktm: Have all docker jobs use tmpfs for /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) [23:52:23] touches 274 jobs wonderful [23:52:59] !log deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs [23:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL