[00:06:41] !log transferred /home from deployment-cumin to deployment-cumin02 and shut down old one [00:06:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:06:50] !log transferred /home from deployment-cache-upload04 to deployment-cache-upload05 and shut down old one [00:06:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:07:26] PROBLEM - Host deployment-cumin is DOWN: CRITICAL - Host Unreachable (172.16.5.1) [00:08:21] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [00:10:38] PROBLEM - Host deployment-cache-upload04 is DOWN: CRITICAL - Host Unreachable (172.16.5.14) [01:26:17] (03CR) 10Krinkle: [C: 04-1] "Needs conflict resolution, but could land after that. From the task, seems like this is pushed back for later. Marking -1 for now to clear" [integration/config] - 10https://gerrit.wikimedia.org/r/496559 (https://phabricator.wikimedia.org/T218345) (owner: 10Kosta Harlan) [01:31:31] (03CR) 10Krinkle: "Phabricator FR30799386" [integration/config] - 10https://gerrit.wikimedia.org/r/497582 (owner: 10Kosta Harlan) [01:31:34] (03Abandoned) 10Krinkle: Add Force Radical to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/497582 (owner: 10Kosta Harlan) [01:43:57] (03CR) 10Krinkle: [WIP] Add Parsoid to docker image and run for Selenium tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/497222 (https://phabricator.wikimedia.org/T218534) (owner: 10Kosta Harlan) [01:53:32] PROBLEM - Free space - all mounts on integration-slave-docker-1051 is CRITICAL: CRITICAL: integration.integration-slave-docker-1051.diskspace._srv.byte_percentfree (<10.00%) [02:03:32] RECOVERY - Free space - all mounts on integration-slave-docker-1051 is OK: OK: All targets OK [06:23:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:48:54] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:52:21] 10Gerrit, 10LDAP: Problem with login Gerrit - https://phabricator.wikimedia.org/T220867 (10Aklapper) Nothing secret here hence making this task public. [09:52:43] 10Gerrit, 10LDAP: Cannot log into Gerrit: "Cannot assign user name to account; name already in use." - https://phabricator.wikimedia.org/T220867 (10Aklapper) [10:21:04] 10Project-Admins: A way to track when a ticket was created with a pre-filled template - https://phabricator.wikimedia.org/T220568 (10Aklapper) "URL hacking" seems to be "[passing URL parameters](https://www.mediawiki.org/wiki/Phabricator/Help#Passing_certain_values_in_the_task_creation_URL)". Also note that [For... [14:10:14] maintenance-disconnect-full-disks build 63370 integration-slave-docker-1050 (/srv: 100%): OFFLINE due to disk space [14:13:05] PROBLEM - Free space - all mounts on integration-slave-docker-1043 is CRITICAL: CRITICAL: integration.integration-slave-docker-1043.diskspace._srv.byte_percentfree (<11.11%) [14:13:56] PROBLEM - Free space - all mounts on integration-slave-docker-1050 is CRITICAL: CRITICAL: integration.integration-slave-docker-1050.diskspace._srv.byte_percentfree (<22.22%) [14:23:07] RECOVERY - Free space - all mounts on integration-slave-docker-1043 is OK: OK: All targets OK [14:23:54] RECOVERY - Free space - all mounts on integration-slave-docker-1050 is OK: OK: All targets OK [14:35:13] maintenance-disconnect-full-disks build 63375 integration-slave-docker-1050: OFFLINE due to disk space [15:00:12] maintenance-disconnect-full-disks build 63380 integration-slave-docker-1050: OFFLINE due to disk space [15:25:12] maintenance-disconnect-full-disks build 63385 integration-slave-docker-1050: OFFLINE due to disk space [15:50:12] maintenance-disconnect-full-disks build 63390 integration-slave-docker-1050: OFFLINE due to disk space [16:15:12] maintenance-disconnect-full-disks build 63395 integration-slave-docker-1050: OFFLINE due to disk space [16:40:13] maintenance-disconnect-full-disks build 63400 integration-slave-docker-1050: OFFLINE due to disk space [17:05:13] maintenance-disconnect-full-disks build 63405 integration-slave-docker-1050: OFFLINE due to disk space [17:30:13] maintenance-disconnect-full-disks build 63410 integration-slave-docker-1050: OFFLINE due to disk space [17:41:28] (03PS2) 10Krinkle: Use php72 instead of php70 (5) - for mw postgres job [integration/config] - 10https://gerrit.wikimedia.org/r/502807 [17:42:22] (03CR) 10Krinkle: [C: 03+2] Use php72 instead of php70 (5) - for mw postgres job [integration/config] - 10https://gerrit.wikimedia.org/r/502807 (owner: 10Krinkle) [17:44:15] (03Merged) 10jenkins-bot: Use php72 instead of php70 (5) - for mw postgres job [integration/config] - 10https://gerrit.wikimedia.org/r/502807 (owner: 10Krinkle) [17:44:50] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502807 (postgres php72) [17:44:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:45:19] (03PS1) 10Krinkle: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 [17:47:29] Php 72 finally? [17:51:36] (03CR) 10Krinkle: [C: 03+1] "Deployed, testing a build now." [integration/config] - 10https://gerrit.wikimedia.org/r/503662 (owner: 10Krinkle) [17:52:56] Zppix: these are the low priority jobs where the php version doesn't matter much. [17:53:04] the important ones have switched long ago [17:53:12] Ah [17:53:29] Project mediawiki-core-code-coverage-docker build #4185: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4185/ [17:53:52] and, it doesn't seem to work [17:54:11] Krinkle: sounds like normal [17:54:29] nah, the file for computing the coverage number is part of the php70 package. [17:54:37] need to move that over to php72 as well [17:55:07] I wouldnt think that php 72 wouldnt be compatiable with php70 files [17:55:15] maintenance-disconnect-full-disks build 63415 integration-slave-docker-1050: OFFLINE due to disk space [17:55:44] (03CR) 10Krinkle: [C: 04-1] "The mediawiki-coverage.sh script is part of the php70 dockerfile directory, so that needs to move over and be deployed first." [integration/config] - 10https://gerrit.wikimedia.org/r/503662 (owner: 10Krinkle) [17:57:04] it is compatible, developers have been running it locally for a while already. It's just a matter of configuring Jenkins the same way. The script isn't there, it's missing, it's not broken, it's not there :) [17:59:33] Ah [17:59:41] I misunderstood [18:00:14] maintenance-disconnect-full-disks build 63416 integration-slave-docker-1037 (/srv: 96%): OFFLINE due to disk space [18:02:30] PROBLEM - Free space - all mounts on integration-slave-docker-1037 is CRITICAL: CRITICAL: integration.integration-slave-docker-1037.diskspace._srv.byte_percentfree (<22.22%) [18:05:51] (03PS2) 10Krinkle: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 [18:05:53] (03PS1) 10Krinkle: dockerfiles: Add mediawiki-coverage script to quibble-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503664 [18:07:26] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Add mediawiki-coverage script to quibble-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503664 (owner: 10Krinkle) [18:07:55] (03CR) 10jerkins-bot: [V: 04-1] jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 (owner: 10Krinkle) [18:08:29] (03PS2) 10Krinkle: dockerfiles: Add mediawiki-coverage script to quibble-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503664 [18:08:31] (03PS3) 10Krinkle: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 [18:12:32] RECOVERY - Free space - all mounts on integration-slave-docker-1037 is OK: OK: All targets OK [18:14:45] (03CR) 10Krinkle: [C: 03+2] "Builds without issue locally. Deploying..." [integration/config] - 10https://gerrit.wikimedia.org/r/503664 (owner: 10Krinkle) [18:16:10] (03Merged) 10jenkins-bot: dockerfiles: Add mediawiki-coverage script to quibble-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503664 (owner: 10Krinkle) [18:16:42] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503664 [18:16:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:14] maintenance-disconnect-full-disks build 63420 integration-slave-docker-1037: OFFLINE due to disk space [18:20:15] maintenance-disconnect-full-disks build 63420 integration-slave-docker-1050: OFFLINE due to disk space [18:45:12] maintenance-disconnect-full-disks build 63425 integration-slave-docker-1037: OFFLINE due to disk space [18:45:12] maintenance-disconnect-full-disks build 63425 integration-slave-docker-1050: OFFLINE due to disk space [18:52:25] !log "Your JENKINS_HOME (/var/lib/jenkins) is almost full. " [18:52:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:10:13] maintenance-disconnect-full-disks build 63430 integration-slave-docker-1037: OFFLINE due to disk space [19:10:13] maintenance-disconnect-full-disks build 63430 integration-slave-docker-1050: OFFLINE due to disk space [19:35:15] maintenance-disconnect-full-disks build 63435 integration-slave-docker-1037: OFFLINE due to disk space [19:35:15] maintenance-disconnect-full-disks build 63435 integration-slave-docker-1050: OFFLINE due to disk space [20:00:14] maintenance-disconnect-full-disks build 63440 integration-slave-docker-1037: OFFLINE due to disk space [20:00:14] maintenance-disconnect-full-disks build 63440 integration-slave-docker-1050: OFFLINE due to disk space [20:25:12] maintenance-disconnect-full-disks build 63445 integration-slave-docker-1037: OFFLINE due to disk space [20:25:13] maintenance-disconnect-full-disks build 63445 integration-slave-docker-1050: OFFLINE due to disk space [20:33:37] Project mediawiki-core-code-coverage-docker build #4186: 04FAILURE in 1 hr 43 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4186/ [20:50:13] maintenance-disconnect-full-disks build 63450 integration-slave-docker-1037: OFFLINE due to disk space [20:50:13] maintenance-disconnect-full-disks build 63450 integration-slave-docker-1050: OFFLINE due to disk space [20:55:00] (03PS4) 10Krinkle: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 [20:55:02] (03PS1) 10Krinkle: dockerfiles: Add clover-edit.py to quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503669 [20:55:52] (03PS2) 10Krinkle: dockerfiles: Add clover-edit.py to quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503669 [20:55:54] (03PS5) 10Krinkle: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 [20:56:01] (03CR) 10Krinkle: [C: 03+2] dockerfiles: Add clover-edit.py to quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503669 (owner: 10Krinkle) [20:57:33] (03Merged) 10jenkins-bot: dockerfiles: Add clover-edit.py to quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/503669 (owner: 10Krinkle) [21:00:08] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/503669 [21:00:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:56] !log Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. [21:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:31] !log Deleting a bunch of job config+history from Jenkins for jobs that no longer exist in JJB/Zuul. T91410 [21:05:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:34] T91410: Automatically delete old jobs not (or no longer) managed by JJB - https://phabricator.wikimedia.org/T91410 [21:15:12] maintenance-disconnect-full-disks build 63455 integration-slave-docker-1037: OFFLINE due to disk space [21:15:13] maintenance-disconnect-full-disks build 63455 integration-slave-docker-1050: OFFLINE due to disk space [21:29:26] PROBLEM - Free space - all mounts on integration-slave-docker-1040 is CRITICAL: CRITICAL: integration.integration-slave-docker-1040.diskspace._srv.byte_percentfree (<11.11%) [21:39:26] RECOVERY - Free space - all mounts on integration-slave-docker-1040 is OK: OK: All targets OK [21:40:13] maintenance-disconnect-full-disks build 63460 integration-slave-docker-1037: OFFLINE due to disk space [21:40:13] maintenance-disconnect-full-disks build 63460 integration-slave-docker-1050: OFFLINE due to disk space [22:05:12] maintenance-disconnect-full-disks build 63465 integration-slave-docker-1037: OFFLINE due to disk space [22:05:13] maintenance-disconnect-full-disks build 63465 integration-slave-docker-1050: OFFLINE due to disk space [22:30:12] maintenance-disconnect-full-disks build 63470 integration-slave-docker-1037: OFFLINE due to disk space [22:30:13] maintenance-disconnect-full-disks build 63470 integration-slave-docker-1050: OFFLINE due to disk space [22:55:12] maintenance-disconnect-full-disks build 63475 integration-slave-docker-1037: OFFLINE due to disk space [22:55:13] maintenance-disconnect-full-disks build 63475 integration-slave-docker-1050: OFFLINE due to disk space [23:20:13] maintenance-disconnect-full-disks build 63480 integration-slave-docker-1037: OFFLINE due to disk space [23:20:13] maintenance-disconnect-full-disks build 63480 integration-slave-docker-1050: OFFLINE due to disk space [23:33:45] (03CR) 10Krinkle: [C: 03+2] "Deployed, tested and succeeded!" [integration/config] - 10https://gerrit.wikimedia.org/r/503662 (owner: 10Krinkle) [23:36:48] (03Merged) 10jenkins-bot: jjb: Switch mediawiki-core-coverage from php70 to php72 (6) [integration/config] - 10https://gerrit.wikimedia.org/r/503662 (owner: 10Krinkle) [23:45:16] maintenance-disconnect-full-disks build 63485 integration-slave-docker-1037: OFFLINE due to disk space [23:45:16] maintenance-disconnect-full-disks build 63485 integration-slave-docker-1050: OFFLINE due to disk space [23:54:01] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair)