[00:05:00] Project beta-update-databases-eqiad build #28974: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28974/ [00:25:08] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10Niedzielski) Thank you so much @zeljkofilipin!! {icon thumbs-up} {icon thumbs-up} {icon thumbs-up} [01:05:00] Project beta-update-databases-eqiad build #28975: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28975/ [02:05:00] Project beta-update-databases-eqiad build #28976: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28976/ [02:38:25] > 00:00:01.602 rsync: failed to set times on "/cache/.": Operation not permitted (1) [02:38:33] Looks like castor-load is failing on all mysql and postgres builds [02:38:49] Since about 12h, maybe longer [02:39:03] Falls back gracefully to installing each time, but might be contributing to slowdown [03:06:13] Project beta-update-databases-eqiad build #28977: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28977/ [03:44:01] (03CR) 10Legoktm: [C: 032] Use stronger password in quibble related browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/461195 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [03:44:31] (03Merged) 10jenkins-bot: Use stronger password in quibble related browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/461195 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [03:45:49] (03CR) 10jenkins-bot: Use stronger password in quibble related browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/461195 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [03:46:38] !log tagged 0.0.27 of quibble (ec8d280de69cda798d95183a4e3f480b45bc9eba) [03:46:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:48:33] (03CR) 10Legoktm: [C: 04-1] "I tagged 0.0.27. Can you change the version in all the changelogs to 0.0.27?" [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [03:51:44] (03CR) 10Legoktm: [C: 032] Make MediaWiki PHP 7.1 tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/466811 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [03:52:42] !log deployed https://gerrit.wikimedia.org/r/466811 [03:52:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:57:53] (03Merged) 10jenkins-bot: Make MediaWiki PHP 7.1 tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/466811 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [03:58:20] wikibugs is laaaaaagggggggggging [04:05:00] Project beta-update-databases-eqiad build #28978: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28978/ [04:31:52] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [04:40:48] !log deployed https://gerrit.wikimedia.org/r/c/integration/config/+/466812 [04:40:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:42:18] (03CR) 10Legoktm: [C: 032] GoogleAuthenticator: disable browser tests (noselenium) [integration/config] - 10https://gerrit.wikimedia.org/r/466812 (owner: 10Legoktm) [04:42:22] (03Merged) 10jenkins-bot: GoogleAuthenticator: disable browser tests (noselenium) [integration/config] - 10https://gerrit.wikimedia.org/r/466812 (owner: 10Legoktm) [05:00:59] (03PS1) 10Legoktm: Switch VisualEditor/VisualEditor to npm6 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/466821 [05:05:00] Project beta-update-databases-eqiad build #28979: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28979/ [05:09:55] Yippee, build fixed! [05:09:55] Project mediawiki-core-code-coverage-docker build #3816: 09FIXED in 2 hr 9 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3816/ [05:11:55] (03CR) 10Legoktm: [C: 032] Switch VisualEditor/VisualEditor to npm6 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/466821 (owner: 10Legoktm) [05:14:07] (03Merged) 10jenkins-bot: Switch VisualEditor/VisualEditor to npm6 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/466821 (owner: 10Legoktm) [05:14:40] !log deployed https://gerrit.wikimedia.org/r/466821 [05:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:05:00] Project beta-update-databases-eqiad build #28980: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28980/ [07:05:00] Project beta-update-databases-eqiad build #28981: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28981/ [08:05:00] Project beta-update-databases-eqiad build #28982: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28982/ [09:05:00] Project beta-update-databases-eqiad build #28983: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28983/ [09:10:19] 10MediaWiki-Releasing, 10Growth-Team, 10Notifications, 10MW-1.32-release: Bundle Echo extension with MW 1.32 - https://phabricator.wikimedia.org/T191738 (10MGChecker) [10:05:00] Project beta-update-databases-eqiad build #28984: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28984/ [10:07:13] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10QuickSurveys, 10Readers-Web-Backlog (Tracking): Stability issues with beta cluster caused selenium-QuickSurveys to fail - https://phabricator.wikimedia.org/T206776 (10phuedx) 05Open>03Resolved a:03phuedx >>! In T206776#4659356,... [10:44:06] 10Phabricator-Sprint-Extension: Wikimedia Phabricator sprint extension uses removed ManiphestEditStatusCapability - https://phabricator.wikimedia.org/T206538 (10urzds) [10:44:10] 10Phabricator-Sprint-Extension: Unhandled Exception ("PhutilMissingSymbolException") - https://phabricator.wikimedia.org/T206489 (10urzds) [11:05:00] Project beta-update-databases-eqiad build #28985: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28985/ [11:48:28] !log ores:ba4ea40 is going beta [11:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:01:22] (03PS4) 10Reedy: Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) [12:05:00] Project beta-update-databases-eqiad build #28986: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28986/ [13:05:00] Project beta-update-databases-eqiad build #28987: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28987/ [13:22:41] !log running foreachwiki update.pph [13:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:22:47] nnot create an anonymous actor for it. Run maintenance/cleanupUsersWithNoId.php to fix this situation. [13:22:49] ffs [13:23:06] greg-g: I think I've found what's wrong with update.php via CI [13:23:08] User name "Lhagiang" is usable, cannot create an anonymous actor for it. Run maintenance/cleanupUsersWithNoId.php to fix this situation. [13:23:23] It's spewing loads of those [13:26:07] at least on simplewiki [13:52:24] 10Beta-Cluster-Infrastructure: beta simplewiki in a bad state due to production tables being imported - https://phabricator.wikimedia.org/T206859 (10Reedy) [14:23:50] 10Beta-Cluster-Infrastructure: beta simplewiki in a bad state due to production tables being imported - https://phabricator.wikimedia.org/T206859 (10Anomie) This should be what we need to do. Review would be good. ``` lang=sql CREATE TEMPORARY TABLE temp AS SELECT rev_id FROM revision LEFT JOIN user ON(rev_user... [15:26:15] !log killing 51 long-running containers (> 1 hour) on docker integration nodes (T198517) [15:26:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:26:19] T198517: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 [15:26:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog (Tracking): selenium-daily-beta-Popups CI job failing - https://phabricator.wikimedia.org/T206640 (10zeljkofilipin) a:03zeljkofilipin [15:26:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: selenium-daily-beta-Popups CI job failing - https://phabricator.wikimedia.org/T206640 (10zeljkofilipin) [15:30:15] !log deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 320 affected jobs [15:30:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:35:22] (03CR) 10Dduvall: Ensure Docker containers are properly killed following builds (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [15:43:01] (03CR) 10Dduvall: [C: 032] Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [15:43:27] (03CR) 10Dduvall: [C: 032] "Deployed for all 321 affected jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [15:47:30] (03Merged) 10jenkins-bot: Provide common docker-run builder [integration/config] - 10https://gerrit.wikimedia.org/r/465668 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [15:55:13] (03PS3) 10Dduvall: Ensure Docker containers are properly killed following builds [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) [15:55:27] (03CR) 10Dduvall: Ensure Docker containers are properly killed following builds (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [16:05:00] Project beta-update-databases-eqiad build #28988: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28988/ [16:05:13] !log deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 1 job mediawiki-quibble-vendor-mysql-php70-docker (T198517) [16:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:05:17] T198517: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 [16:05:52] (03CR) 10Dduvall: [C: 031] "Deployed for mediawiki-quibble-vendor-mysql-php70-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [16:25:45] !log deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 321 affected jobs (T198517) [16:25:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:25:48] T198517: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 [16:28:24] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:55] (03CR) 10Dduvall: [C: 032] "Deployed for all affected jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [16:31:41] !log taking integration-slave-docker-1021 offline (host is unresponsive) [16:31:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:36:26] (03Merged) 10jenkins-bot: Ensure Docker containers are properly killed following builds [integration/config] - 10https://gerrit.wikimedia.org/r/465671 (https://phabricator.wikimedia.org/T198517) (owner: 10Dduvall) [16:42:57] !log killing 9 long-running containers (> 1 hour) on docker integration nodes (T198517) [16:43:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:43:01] T198517: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 [16:50:15] addshore: legoktm what's happening with: https://phabricator.wikimedia.org/T205958? just noticed it's failing a lot of stuff? [16:50:53] thcipriani: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/465578/ should fix it [16:50:58] but the build / jenkins job timed out [16:51:07] which is another thing that has been happening for the past few days D: [16:51:27] oh, i see above some long running containers got killed? perhaps thats why things have been going slowly? [16:51:45] that's what we think [16:51:54] try a recheck there [16:52:10] done [16:52:19] lets see what the result is in 30 more mins :P [16:52:21] marxarelli: just deployed a fix (hopefully) for all the left over containers [16:52:24] * addshore currently has his head in https://phabricator.wikimedia.org/T206743#4661716 [16:53:15] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [16:53:28] the theory is: everytime a job fails and zuul reques the patches behind it, the current running containers weren't getting shutdown [16:53:53] interesting [16:53:58] so the more failures, the more tests running per box, the slower those tests run, the more timeout failures, the more contianers [16:54:15] and so on [17:02:00] 10Continuous-Integration-Config: CI console output is messy for wmf-quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T206867 (10Daimona) [17:05:00] Project beta-update-databases-eqiad build #28989: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28989/ [17:15:56] !log killing 7 long-running containers (> 1 hour) on docker integration nodes (T198517) [17:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:16:02] T198517: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 [17:18:29] !log bringing integration-slave-docker-1021 back online [17:18:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:26:23] addshore: tada: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/465578/ [17:52:01] thcipriani: I just gave it a self merging +2 [17:52:10] evil me, but should fix the problem [17:52:39] I seem to remember filing a ticket ages about about containers staying around if builds timed out [17:53:04] yeah, I don't think this is the first time we tried to fix it :P [17:53:20] thcipriani: https://phabricator.wikimedia.org/T176747 [17:54:02] links to the one we just closed today: https://phabricator.wikimedia.org/T198517 [17:54:16] or wait, haven't close :) [17:54:42] but hopefully just deployed the fix for a little bit ago [18:05:00] Project beta-update-databases-eqiad build #28990: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28990/ [18:10:10] thcipriani, addshore: i wanted to wait until all old containers finished or were killed off before closing the task, but it looks like we've arrived :) [18:25:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 (10dduvall) 05Open>03Resolved Two `integration/config` patches were deployed to fix the iss... [18:54:24] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Define a code health metrics reporting approach/strategy - https://phabricator.wikimedia.org/T205143 (10Jrbranaa) a:03Jrbranaa [19:05:00] Project beta-update-databases-eqiad build #28991: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28991/ [19:16:01] Project beta-scap-eqiad build #224650: 04FAILURE in 0.98 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224650/ [19:16:10] ^ me [19:18:29] (03PS5) 10Legoktm: Bump Quibble images to 0.0.27 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [19:18:40] (03PS6) 10Legoktm: Bump Quibble images to 0.0.27 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [19:20:03] (03CR) 10Legoktm: [C: 032] "I rewrote the changelog entries to use the git-shortlog format Hashar was using." [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [19:21:43] (03Merged) 10jenkins-bot: Bump Quibble images to 0.0.27 [integration/config] - 10https://gerrit.wikimedia.org/r/465606 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [19:22:05] (03PS1) 10Legoktm: docker: Bump Quibble images to 0.0.27 [integration/config] - 10https://gerrit.wikimedia.org/r/466950 [19:23:29] !log rebuilding docker images for https://gerrit.wikimedia.org/r/465606 [19:23:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:37:55] Yippee, build fixed! [19:37:55] Project beta-scap-eqiad build #224651: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/224651/ [19:51:50] PROBLEM - Disk space on contint1001 is CRITICAL: DISK CRITICAL - free space: / 2510 MB (5% inode=67%) [19:53:23] thcipriani ^^ [20:05:00] Project beta-update-databases-eqiad build #28992: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28992/ [20:08:58] !log contint1001 - apt-get clean - frees 1GB to bring it _just_ under the Icinga warning threshold :p [20:09:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:09:16] eh.. NOT 1GB .. 1% .. darn it [20:11:24] !log contint1001 - gzip zuul.log.2018-* in /var/log/ to prevent running out of disk [20:11:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:14:14] !log contint1001 - gzip merge-debug.log.2018-* debug.log.2018-* etc.. all in /var/log/zuul/ there seems to be no compression (but max age is 1 month) [20:14:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:14:40] RECOVERY - Disk space on contint1001 is OK: DISK OK [20:15:54] paladox: 81% ^ [20:16:01] ah thanks :) [20:16:27] we should make it use gzip by itself i guess [20:16:51] yeh [20:17:01] how big were the logs mutante ? [20:18:19] when we got the alert /var/log/ was 8.5G now it's only 2.8G [20:18:37] oh [20:18:50] /var/log/zuul is now 655M left [20:19:29] / is 46G 35G of it are still used. which is 81% [20:19:39] alert started at 95% [20:22:13] :) [20:25:17] 10Project-Admins: Create phab project for certcentral - https://phabricator.wikimedia.org/T206877 (10Krenair) [20:28:12] legoktm: RE: job perf, I noticed yesterday that castor load doesn't appear to be working. [20:28:31] I don't know enough about it to know when it worked or where the issue might lie.. [20:29:01] It might also have to do with the recent directory structure/permissions change, marxarelli [20:32:22] !log deploy01 /usr/local/bin/mwscript update.php --wiki=ruwiki --quick [20:32:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:27] !log deploy01 /usr/local/bin/mwscript update.php --wiki=simplewiki --quick [20:32:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:33:28] Krinkle: um [20:33:51] Krinkle: update.php on simplewiki is spammy [20:34:07] https://phabricator.wikimedia.org/T206859 [20:34:26] https://phabricator.wikimedia.org/P7671 [20:46:07] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Smalyshev) [20:55:33] Krinkle: where are you seeing the failure? [20:56:18] marxarelli: all quibble flavours, top of the output [20:56:22] https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/4402/consoleFull [20:56:30] > rsync: failed to set times on "/cache/.": Operation not permitted (1) [21:05:00] Project beta-update-databases-eqiad build #28993: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28993/ [21:09:52] (03PS2) 10Legoktm: Run phan for UserMerge [integration/config] - 10https://gerrit.wikimedia.org/r/465049 (owner: 10Umherirrender) [21:09:58] (03CR) 10Legoktm: [C: 032] Run phan for UserMerge [integration/config] - 10https://gerrit.wikimedia.org/r/465049 (owner: 10Umherirrender) [21:12:17] (03Merged) 10jenkins-bot: Run phan for UserMerge [integration/config] - 10https://gerrit.wikimedia.org/r/465049 (owner: 10Umherirrender) [21:13:29] !log deployed https://gerrit.wikimedia.org/r/465049 [21:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:13:50] Krinkle: hmm. i'm not sure if that's a problem. the console of a currently running build for quibble-vendor-mysql-hhvm-docker shows the same error, yet the cache directory is populated (with older, previously archived files) [21:14:29] it may just fail to set the ctime/mtime on /cache itself, since it's root owned within the container [21:14:34] (just a theory) [21:25:46] relatedly (ci perf), i'm hoping we'll see overall improvements in build durations now that zombie containers aren't marauding through ci (see https://phabricator.wikimedia.org/T198517#4662026) [21:32:56] (03PS1) 10Legoktm: Add quibble-stretch-php72 image [integration/config] - 10https://gerrit.wikimedia.org/r/466991 (https://phabricator.wikimedia.org/T204884) [22:05:00] Project beta-update-databases-eqiad build #28994: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28994/ [22:11:08] (03PS5) 10Umherirrender: Add sniff to detect __METHOD__ in closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/463656 (https://phabricator.wikimedia.org/T205800) [22:22:07] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Ubuntu Trusty Deprecation): cloudvps: deployment-prep project trusty deprecation - https://phabricator.wikimedia.org/T204500 (10bd808) [22:38:27] Reedy: wow, there's alike 20 copies of update.php simplewiki running on deploy01 [22:38:40] lol [22:38:42] helpful... [22:40:46] Yippee, build fixed! [22:40:47] Project beta-update-databases-eqiad build #28995: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28995/ [22:41:25] awesome. the build passes if its subprocess was kill'ed [22:42:35] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Ubuntu Trusty Deprecation): cloudvps: deployment-prep project trusty deprecation - https://phabricator.wikimedia.org/T204500 (10bd808) [23:06:21] lol