[00:08:43] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Use a more complex password for WikiAdmin in selenium tests - https://phabricator.wikimedia.org/T188520#4138076 (10Krinkle) 05Open>03Resolved a:03Krinkle
[00:10:01] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4138084 (10Krinkle)
[00:17:28] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Operations, 10Scoring-platform-team: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842#4118429 (10Dzahn) Which deployment server is this about? Production or deployment-prep or another?
[00:38:08] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4138142 (10Jdforrester-WMF)
[00:38:11] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Run maintenance/updateCredits.php on master branch and upload changes to CREDITS for review. - https://phabricator.wikimedia.org/T191090#4138140 (10Jdforrester-WMF) 05Open>03Resolved
[00:42:13] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Operations, 10Patch-For-Review: Scap stalled at sync-masters, ok: 1, left: 1 - https://phabricator.wikimedia.org/T191029#4138152 (10Dzahn) 05Open>03Resolved a:03Dzahn
[00:46:59] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Operations, 10Scoring-platform-team: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842#4138160 (10awight) Sorry--this is about production.
[01:06:30] <wikibugs>	 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.31-release: Upgrade patches for tarball releases don't apply cleanly to tarball installation - https://phabricator.wikimedia.org/T73379#748261 (10MaxSem) Are patches even useful these days when download speed is much less of a problem than 20 years ago?
[01:15:17] <wikibugs>	 10Phabricator, 10Wikimedia-Hackathon-2018: Build a bot that pushes Phabricator updates to Google Chat - https://phabricator.wikimedia.org/T189313#4138215 (10Aklapper) >>! In reply to T189313#4137191 by @dbarratt: I'd see a small difference here. In an ideal world, [[ https://meta.wikimedia.org/wiki/Wikimedia_F...
[01:38:03] <legoktm>	 Platonides: Niharika: I didn't check it off because I just hadn't reviewed it yet. And it depends on Echo being bundled, so I was prioritizing the review of Echo. :)
[01:41:08] <wikibugs>	 (03CR) 10Legoktm: [C: 031] "Chad, now that 1.31 is branched, do we/you need to add the submodule manually?" [tools/release] - 10https://gerrit.wikimedia.org/r/427209 (owner: 10Cicalese)
[02:34:00] <wikibugs>	 10Continuous-Integration-Config, 10PHP 7.0 support, 10Patch-For-Review: PHP7 jobs fail on wmf.24 branch - https://phabricator.wikimedia.org/T189319#4138297 (10Krinkle) 05Open>03Resolved a:03Krinkle
[02:43:42] <wmf-insecte>	 Project beta-scap-eqiad build #204261: 04FAILURE in 0.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204261/
[02:58:57] <wmf-insecte>	 Yippee, build fixed!
[02:58:57] <wmf-insecte>	 Project beta-scap-eqiad build #204262: 09FIXED in 5 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204262/
[03:18:25] <wikibugs>	 (03PS1) 10Legoktm: nodepool: Install mod_php7, not php5 [integration/config] - 10https://gerrit.wikimedia.org/r/427309 (https://phabricator.wikimedia.org/T190548)
[03:19:39] <wikibugs>	 (03CR) 10Legoktm: [C: 032] "I hope this works" [integration/config] - 10https://gerrit.wikimedia.org/r/427309 (https://phabricator.wikimedia.org/T190548) (owner: 10Legoktm)
[03:21:03] <wikibugs>	 (03Merged) 10jenkins-bot: nodepool: Install mod_php7, not php5 [integration/config] - 10https://gerrit.wikimedia.org/r/427309 (https://phabricator.wikimedia.org/T190548) (owner: 10Legoktm)
[03:22:37] <legoktm>	 !log manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
[03:22:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[03:27:15] <wikibugs>	 (03CR) 10Chad: "I have to do them manually anyway right now (soon to be fixed), but yes" [tools/release] - 10https://gerrit.wikimedia.org/r/427209 (owner: 10Cicalese)
[03:43:06] <legoktm>	 > 03:42:40 X-MediaWiki-PHP: 5.6.33-0+deb8u1
[03:43:07] <legoktm>	 grrr
[03:45:19] <legoktm>	 jenkins@ci-jessie-wikimedia-1034457:~$ dpkg -l | grep mod-php
[03:45:19] <legoktm>	 ii  libapache2-mod-php5                    5.6.33+dfsg-0+deb8u1                                amd64        server-side, HTML-embedded scripting language (Apache 2 module)
[03:45:21] <legoktm>	 that's wrong
[04:20:50] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Update mediawiki-core-qunit-selenium-jessie/mediawiki-extensions-qunit-jessie jobs from PHP5 to PHP7/HHVM - https://phabricator.wikimedia.org/T190548#4076541 (10Legoktm) My patch doesn't seem to have wor...
[04:26:04] <legoktm>	 !log try once more to refresh nodepool snapshot
[04:26:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[04:33:13] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Update mediawiki-core-qunit-selenium-jessie/mediawiki-extensions-qunit-jessie jobs from PHP5 to PHP7/HHVM - https://phabricator.wikimedia.org/T190548#4138316 (10Legoktm) ``` 2018-04-18 04:29:39,669 INFO...
[04:35:29] <Krinkle>	 legoktm: Hm.. could you fix the topic SAL link? Seems ur1.ca is dead. http://bit.ly/2Ha4IFQ also points to RelEng SAL.
[04:35:39] <Krinkle>	 https://bit.ly/2Ha4IFQ * https
[04:39:12] <wikibugs>	 10Phabricator, 10Wikimedia-Hackathon-2018: Build a bot that pushes Phabricator updates to Google Chat - https://phabricator.wikimedia.org/T189313#4138317 (10dbarratt) > As an organization, we strive to use open source tools over proprietary ones, although we use proprietary or closed tools (such as software, o...
[04:45:27] <James_F>	 legoktm: Does the "include contint::php" line pull in mod_php5?
[04:47:29] <James_F>	 We also "require_package('php5-xhprof')".
[04:48:00] <James_F>	 Which feels a little suspect.
[04:50:05] <legoktm>	 James_F: some of the debug logs don't fit in my screen's scrollback but yes, something is pulling in mod_php, but it's not a puppet thing.
[04:50:32] <James_F>	 Always a good start.
[04:50:42] <James_F>	 Hmm. Base image from the OS?
[04:52:09] <legoktm>	 !log live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
[04:52:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[04:53:21] <legoktm>	 (if this does work I'll be shocked)
[04:54:28] <James_F>	 Shocked and unable to work out how to make puppet do what we want? ;-(
[04:55:59] <legoktm>	 shocked that I got it right on the first try :p
[04:56:02] <legoktm>	 spoiler alert: I didn't
[04:57:56] * James_F grins.
[04:59:20] <legoktm>	 so my understanding is that it's taking the old image, and running puppet over it
[04:59:26] <legoktm>	 so we have to uninstall mod_php5 first
[04:59:43] <legoktm>	 $ git diff
[04:59:44] <legoktm>	 +# We really really don't want php5
[04:59:44] <legoktm>	 +package { [
[04:59:44] <legoktm>	 +    'libapache2-mod-php5',
[04:59:44] <legoktm>	 +    ]:
[04:59:44] <legoktm>	 +    ensure => absent,
[04:59:44] <James_F>	 Right.
[04:59:46] <legoktm>	 +}
[05:00:01] <James_F>	 Foolish question, but there's no way we could do this on docker?
[05:01:18] <James_F>	 Given that we're meant to be getting rid of nodepool, all this effort to update the to-be-replaced system seems a little sad.
[05:03:31] <legoktm>	 hashar didn't think it was ready yet
[05:03:36] <legoktm>	 he wanted a few more weeks
[05:04:01] <James_F>	 He knows more than I do. :-)
[05:24:20] <wikibugs>	 (03PS1) 10Legoktm: Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316
[05:24:49] <wikibugs>	 (03CR) 10Legoktm: [C: 04-1] "Doesn't work" [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[05:24:50] <legoktm>	 Krinkle: ^ do you have suggestions on where my non-existent puppet skills are going wrong?
[05:25:33] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[05:38:42] <wikibugs>	 (03PS2) 10Legoktm: Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316
[05:39:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[05:42:41] <wikibugs>	 (03PS3) 10Legoktm: Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316
[05:59:19] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 031] Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[05:59:25] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[06:00:52] <wikibugs>	 (03Merged) 10jenkins-bot: Get rid of mod_php5 before installing mod_php7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/427316 (owner: 10Legoktm)
[06:24:06] <legoktm>	 James_F: boom, done.
[06:24:28] <legoktm>	 well, kinda
[06:25:40] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Update mediawiki-core-qunit-selenium-jessie/mediawiki-extensions-qunit-jessie jobs from PHP5 to PHP7/HHVM - https://phabricator.wikimedia.org/T190548#4138426 (10Legoktm) OK, after https://gerrit.wikimedi...
[06:44:24] <wikibugs>	 (03PS2) 10Legoktm: Move qunit jobs from Zend PHP 5.5 to PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/426115 (https://phabricator.wikimedia.org/T190548) (owner: 10Jforrester)
[06:52:44] <wikibugs>	 (03PS3) 10Legoktm: Move qunit jobs from Zend PHP 5.5 to PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/426115 (https://phabricator.wikimedia.org/T190548) (owner: 10Jforrester)
[07:15:46] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Move qunit jobs from Zend PHP 5.5 to PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/426115 (https://phabricator.wikimedia.org/T190548) (owner: 10Jforrester)
[07:17:15] <wikibugs>	 (03Merged) 10jenkins-bot: Move qunit jobs from Zend PHP 5.5 to PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/426115 (https://phabricator.wikimedia.org/T190548) (owner: 10Jforrester)
[07:17:29] <legoktm>	 !log deploying https://gerrit.wikimedia.org/r/426115
[07:17:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[07:21:13] <legoktm>	 Krinkle: I don't think I have perms to change the topic
[07:44:55] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support: Make Wikimedia CI run PHP in either PHP 7.0+ or HHVM - https://phabricator.wikimedia.org/T190547#4138497 (10Legoktm)
[07:44:58] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Update mediawiki-core-qunit-selenium-jessie/mediawiki-extensions-qunit-jessie jobs from PHP5 to PHP7/HHVM - https://phabricator.wikimedia.org/T190548#4138495 (10Legoktm) 05Open>03Resolved a:03Legoktm
[07:45:59] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support: Make Wikimedia CI run PHP in either PHP 7.0+ or HHVM - https://phabricator.wikimedia.org/T190547#4076530 (10Legoktm) So I think the last thing to do is blacklist php55 jobs from running against master and REL1_31 branches?
[07:52:45] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4138509 (10Legoktm) License review: I talked with @lfaraone...
[07:55:55] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138512 (10hashar)
[07:56:25] <wikibugs>	 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.31 - https://phabricator.wikimedia.org/T191736#4138522 (10Legoktm)
[07:58:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138524 (10hashar)
[07:59:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138512 (10hashar)
[08:05:08] <wikibugs>	 (03PS1) 10Hashar: Add REL1_31 to quibble-integration job [integration/config] - 10https://gerrit.wikimedia.org/r/427324 (https://phabricator.wikimedia.org/T190879)
[08:06:59] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Prepare CI for REL1_31 - https://phabricator.wikimedia.org/T190879#4138533 (10hashar) 05Open>03Resolved a:03hashar I don't think there is anything that has to be done bes...
[08:07:12] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add REL1_31 to quibble-integration job [integration/config] - 10https://gerrit.wikimedia.org/r/427324 (https://phabricator.wikimedia.org/T190879) (owner: 10Hashar)
[08:08:40] <wikibugs>	 (03Merged) 10jenkins-bot: Add REL1_31 to quibble-integration job [integration/config] - 10https://gerrit.wikimedia.org/r/427324 (https://phabricator.wikimedia.org/T190879) (owner: 10Hashar)
[08:18:16] <wikibugs>	 (03PS1) 10Hashar: Allow selection of builds for quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/427326 (https://phabricator.wikimedia.org/T192432)
[08:18:52] <wikibugs>	 (03PS2) 10Hashar: Allow selection of builds for quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/427326 (https://phabricator.wikimedia.org/T192372)
[08:21:14] <wikibugs>	 (03CR) 10Hashar: [C: 032] "Eg: https://phabricator.wikimedia.org/F17070981" [integration/config] - 10https://gerrit.wikimedia.org/r/427326 (https://phabricator.wikimedia.org/T192372) (owner: 10Hashar)
[08:22:39] <wikibugs>	 (03Merged) 10jenkins-bot: Allow selection of builds for quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/427326 (https://phabricator.wikimedia.org/T192372) (owner: 10Hashar)
[08:43:51] <wikibugs>	 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Jenkins daily builds broken (config parser) - https://phabricator.wikimedia.org/T192367#4138562 (10zeljkofilipin) a:03zeljkofilipin
[08:54:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138579 (10hashar) Looking at the core file. Self notes: ``` docker run -it --user root -v "$(pwd)":/workspace --entrypoint /bin/bash dock...
[08:55:59] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138583 (10hashar) There is the garbage collector mentioned at some point. We had to disable it for php5.5 due to constant segfaults ( T14...
[09:19:28] <wikibugs>	 (03CR) 10Chad: [C: 032] Bundle CategoryTree extension [tools/release] - 10https://gerrit.wikimedia.org/r/427209 (owner: 10Cicalese)
[09:19:58] <wikibugs>	 (03PS1) 10Hashar: docker: disable garbage collector on Zend php5.5 [integration/config] - 10https://gerrit.wikimedia.org/r/427330 (https://phabricator.wikimedia.org/T142158)
[09:20:00] <wikibugs>	 (03Merged) 10jenkins-bot: Bundle CategoryTree extension [tools/release] - 10https://gerrit.wikimedia.org/r/427209 (owner: 10Cicalese)
[09:23:36] <wikibugs>	 (03CR) 10Hashar: [C: 032] docker: disable garbage collector on Zend php5.5 [integration/config] - 10https://gerrit.wikimedia.org/r/427330 (https://phabricator.wikimedia.org/T142158) (owner: 10Hashar)
[09:24:47] <wikibugs>	 (03PS1) 10Hashar: Update qiubble php55 job to disable zend gc [integration/config] - 10https://gerrit.wikimedia.org/r/427332 (https://phabricator.wikimedia.org/T192432)
[09:24:50] <wikibugs>	 (03Merged) 10jenkins-bot: docker: disable garbage collector on Zend php5.5 [integration/config] - 10https://gerrit.wikimedia.org/r/427330 (https://phabricator.wikimedia.org/T142158) (owner: 10Hashar)
[09:28:08] <wikibugs>	 (03PS2) 10Hashar: Update quibble php55 job to disable zend gc [integration/config] - 10https://gerrit.wikimedia.org/r/427332 (https://phabricator.wikimedia.org/T192432)
[09:30:35] <wikibugs>	 (03CR) 10Hashar: "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/427332 (https://phabricator.wikimedia.org/T192432) (owner: 10Hashar)
[09:38:18] <wikibugs>	 (03CR) 10Hashar: [C: 032] "https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php55-docker/ passed!" [integration/config] - 10https://gerrit.wikimedia.org/r/427332 (https://phabricator.wikimedia.org/T192432) (owner: 10Hashar)
[09:39:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432#4138709 (10hashar) 05Open>03Resolved a:03hashar php55 with mysql ended up segfaulting due to the Zend garbage collector. A known iss...
[09:39:42] <wikibugs>	 (03Merged) 10jenkins-bot: Update quibble php55 job to disable zend gc [integration/config] - 10https://gerrit.wikimedia.org/r/427332 (https://phabricator.wikimedia.org/T192432) (owner: 10Hashar)
[09:40:26] <hashar>	 !log Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/  T192432
[09:40:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:40:28] <stashbot>	 T192432: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432
[09:41:07] <hashar>	 !log Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
[09:41:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:13:02] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T183969#4138761 (10Aklapper)
[10:29:19] <wikibugs>	 10Project-Admins: Create a new component project: MediaWiki-Live-preview - https://phabricator.wikimedia.org/T192074#4126664 (10Aklapper) See https://www.mediawiki.org/wiki/Phabricator/Project_management/Tracking_tasks regarding T41272 if someone decides to go ahead.
[10:32:49] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865#4138795 (10Aklapper) (If I edit a regular, existing task, is there any way to get that "Due Date" field displayed?)
[11:37:54] <wikibugs>	 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Failed loading configuration file ... wdio.conf.jenkins.js - https://phabricator.wikimedia.org/T192367#4136251 (10zeljkofilipin)
[11:41:51] <wikibugs>	 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Failed loading configuration file ... wdio.conf.jenkins.js - https://phabricator.wikimedia.org/T192367#4138951 (10zeljkofilipin)
[11:46:06] <wikibugs>	 (03PS1) 10Zfilipin: Selenium: Jenkins configuration file no longer needed [integration/config] - 10https://gerrit.wikimedia.org/r/427362 (https://phabricator.wikimedia.org/T192367)
[12:03:10] <wikibugs>	 (03PS2) 10Zfilipin: Selenium: Jenkins configuration file no longer needed [integration/config] - 10https://gerrit.wikimedia.org/r/427362 (https://phabricator.wikimedia.org/T192367)
[12:20:05] <wikibugs>	 (03PS2) 10Addshore: Enable voting ext-mw-selenium-node-composer-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/424277
[12:20:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable voting ext-mw-selenium-node-composer-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/424277 (owner: 10Addshore)
[12:21:57] <addshore>	 hashar: hows quibble oging?
[12:21:59] <addshore>	 *going
[12:24:33] <wikibugs>	 (03CR) 10Zfilipin: "Deployed. CirrusSearch already ran and it's green. Popups and RelatedArticles are waiting for free vms, but they should be fine too." [integration/config] - 10https://gerrit.wikimedia.org/r/427362 (https://phabricator.wikimedia.org/T192367) (owner: 10Zfilipin)
[12:27:33] <hashar>	 addshore: https://integration.wikimedia.org/ci/job/quibble-integration/ :D
[12:27:48] <hashar>	 addshore: php7 builds are broken because I need an old version of libtidy
[12:28:15] <addshore>	 hehe
[12:28:21] <hashar>	 addshore: there are a bunch of tests being broken under sqlite due to database id being different or some sort order madness  ( https://phabricator.wikimedia.org/T191035 )
[12:28:22] <addshore>	 I can't even tell what that is showing me :P
[12:28:30] <hashar>	 so on https://integration.wikimedia.org/ci/job/quibble-integration/
[12:28:53] <hashar>	 that runs quibble against several parameters:  mediawiki branch, database backend, php flavor,  packages source (composer or vendor.git)
[12:29:11] <hashar>	 the job builds the cartesian product of all combinations then trigger a shit ton of jobs 
[12:29:24] <hashar>	 the table is merely a summary of all the builds and how they behave
[12:29:33] <addshore>	 cooll
[12:29:34] <hashar>	 so master + mysql is all green
[12:29:40] <hashar>	 master + sqlite is all red  :D
[12:29:58] <hashar>	 REL1_30 + php7 are all red (due to libtidy)
[12:30:14] <addshore>	 where is postgres?
[12:30:16] <addshore>	 ;)
[12:42:41] <hashar>	 addshore: E_LATER
[12:42:45] <hashar>	 and I will skip sqlite as well
[12:43:00] <hashar>	 I think tomorrow I will add quibble to mediawiki/core and vendor and replace the old jobs
[12:43:05] <hashar>	 will test that tonight
[12:43:09] <hashar>	 at least for master branches
[12:43:19] <hashar>	 postgre should be a breeze to add
[12:43:39] <hashar>	 just need to figure out how to start postgre in the background with the appropriate settings, how to create the database and grant rights
[12:43:46] <hashar>	 then figure out the parameters to pass to install.php
[12:54:19] <addshore>	 :D
[13:15:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster Obama page often responds with 503 - https://phabricator.wikimedia.org/T188913#4139141 (10Niedzielski) Here's the latest report:  ``` Request from 73.252.38.252 via deployment-cache-text04 deployment-cache-text04, Varnish XID 254949111 Error: 503, Backe...
[13:18:50] <addshore>	 !log added WMDE-Fisch to deployment-prep
[13:18:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:23:34] <addshore>	 !log added Thiemo Kreuz (WMDE) to deployment-prep
[13:23:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:53:39] <wikibugs>	 (03CR) 10Zfilipin: "Popups is also green. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/427362 (https://phabricator.wikimedia.org/T192367) (owner: 10Zfilipin)
[14:03:15] <wikibugs>	 (03CR) 10Zfilipin: "RelatedArticles is also green" [integration/config] - 10https://gerrit.wikimedia.org/r/427362 (https://phabricator.wikimedia.org/T192367) (owner: 10Zfilipin)
[14:20:30] <Reedy>	 14:11:43 INFO:zuul.Cloner:Creating repo mediawiki/core from cache /srv/git/mediawiki/core.git
[14:20:30] <Reedy>	 14:11:44 DEBUG:git.cmd:AutoInterrupt wait stderr: "fatal: destination path '/src' already exists and is not an empty directory.\n"
[14:35:59] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4139454 (10CCicalese_WMF)
[14:36:02] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CategoryTree, 10MW-1.31-release, 10Patch-For-Review: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4139452 (10CCicalese_WMF) 05Open>03Resolved a:03CCicalese_WMF
[14:36:23] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-CategoryTree, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4115103 (10CCicalese_WMF)
[14:36:34] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#3697225 (10CCicalese_WMF)
[14:40:41] <wikibugs>	 10Beta-Cluster-Infrastructure: Spam/Spambot registration on deployment.beta is out of hand - https://phabricator.wikimedia.org/T187046#4139491 (10Aklapper) Seems to still happen: https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&limit=500&days=30&urlversion=2
[14:42:05] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Analytics-Tech-community-metrics, 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4139496 (10Aklapper)
[14:54:56] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4139561 (10Anomie) T72541#1805584 and subsequent comments de...
[15:01:30] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, 10Patch-For-Review, 10User-zeljkofilipin: Mocha tests for Echo notifications - https://phabricator.wikimedia.org/T177412#4139566 (10zeljkofilipin)
[15:41:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations: "Obama" page on Beta Cluster often responds with 503 - https://phabricator.wikimedia.org/T188913#4139674 (10Krinkle)
[16:00:47] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Analytics-Tech-community-metrics, 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4086695 (10Legoktm) https://phabricator.wikimedia.org/source/tool-ci/browse/...
[16:19:49] <wikibugs>	 (03PS1) 10Legoktm: Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547)
[16:19:51] <wikibugs>	 (03PS1) 10Legoktm: Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547)
[16:19:53] <wikibugs>	 (03PS1) 10Legoktm: Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547)
[16:21:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:21:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:21:27] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:24:48] <wikibugs>	 (03PS2) 10Legoktm: Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547)
[16:24:50] <wikibugs>	 (03PS2) 10Legoktm: Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547)
[16:24:52] <wikibugs>	 (03PS2) 10Legoktm: Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547)
[16:26:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:26:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:26:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:28:41] <wikibugs>	 (03PS3) 10Legoktm: Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547)
[16:28:43] <wikibugs>	 (03PS3) 10Legoktm: Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547)
[16:28:45] <wikibugs>	 (03PS3) 10Legoktm: Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547)
[16:32:30] <wikibugs>	 (03CR) 10Krinkle: [C: 031] Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[16:33:11] <wmf-insecte>	 Project beta-scap-eqiad build #204337: 04FAILURE in 31 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204337/
[16:34:12] <wmf-insecte>	 Project beta-scap-eqiad build #204338: 04STILL FAILING in 0.26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204338/
[16:34:51] <Reedy>	 16:33:10 File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 500
[16:34:51] <Reedy>	 16:33:10 }, env={'PHP': 'hhvm -vEval.PerfPidMap=true')
[16:34:51] <Reedy>	 16:33:10 ^
[16:34:51] <Reedy>	 16:33:10 SyntaxError: invalid syntax
[16:34:54] <Reedy>	 Did someone break scap?
[16:35:39] <wikibugs>	 (03PS1) 10Krinkle: composer.json: Remove reference to non-existant lint.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/427431
[16:36:53] <wikibugs>	 (03CR) 10Krinkle: [C: 032] composer.json: Remove reference to non-existant lint.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/427431 (owner: 10Krinkle)
[16:37:51] <wikibugs>	 (03Merged) 10jenkins-bot: composer.json: Remove reference to non-existant lint.php [integration/jenkins] - 10https://gerrit.wikimedia.org/r/427431 (owner: 10Krinkle)
[16:38:14] <hashar>	 legoktm: Krinkle: addshore: I will most probably add quibble to mediawiki/vendor mediawiki/core tonight
[16:38:25] <addshore>	 woo
[16:38:27] <hashar>	 depending on the amount of craziness that involves with the branch filters
[16:38:46] <hashar>	 so either I got confidence and deploy it after the train then announce to wikitech
[16:38:55] <hashar>	 else I send a patch and add everyone as reviewers :D
[16:39:17] <legoktm>	 hashar: as voting?
[16:40:38] <wikibugs>	 10Scap: `scap update-interwiki-cache` broken - https://phabricator.wikimedia.org/T192469#4139903 (10Reedy)
[16:42:36] <Krinkle>	 Maybe give it a day as non-voting on all jobs
[16:42:43] <Krinkle>	 to weed out any false negatives just in case
[16:43:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Scap: Scap syntax error on beta - https://phabricator.wikimedia.org/T192470#4139924 (10Reedy)
[16:44:22] <hashar>	 legoktm: na I guess non  voting at first
[16:44:38] <hashar>	 though I have confidence it will work just fine
[16:44:38] <greg-g>	 Reedy: tyler just undid his live-hack
[16:44:54] <greg-g>	 Reedy: debugging https://phabricator.wikimedia.org/T191921
[16:45:08] <Reedy>	 gawwwd thcipriani l2python
[16:51:56] <wikibugs>	 10Scap: `scap update-interwiki-cache` broken - https://phabricator.wikimedia.org/T192469#4139959 (10Reedy)
[16:52:00] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4139962 (10Reedy)
[16:53:21] <DowagerCountess>	 legoktm: global rename seems broken at beta :|
[16:53:40] <legoktm>	 broken in what sense?
[16:53:51] <DowagerCountess>	 https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress <-- this one
[16:54:04] <DowagerCountess>	 I cannot find the log entry for the first one
[16:54:10] <DowagerCountess>	 and the second one is not moving
[16:54:28] <legoktm>	 that's bizzare
[16:54:35] <DowagerCountess>	 ++ the user is indefed on en so we should revoke the rights on beta
[16:54:59] <DowagerCountess>	 (and maybe trout people granting privs there w/o oversight)
[16:58:12] <DowagerCountess>	 ok found the log: https://login.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/Artix_Krieger_2
[16:58:23] <DowagerCountess>	 14 March
[16:58:28] <DowagerCountess>	 so stuck for 4 days
[16:58:30] <legoktm>	 DowagerCountess: could you file a bug? I'm about to go on vacation actually
[16:58:41] <DowagerCountess>	 legoktm: sure, enjoy your relax
[17:02:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4139995 (10MarcoAurelio)
[17:02:09] <DowagerCountess>	 ^^ done
[17:02:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4140008 (10MarcoAurelio)
[17:06:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4140018 (10MarcoAurelio) ``` maurelio@deployment-tin:~$ mwscript showJobs.php --wiki=wikidatawiki [Wed Apr 18 17...
[17:19:59] <wikibugs>	 10Scap: `scap update-interwiki-cache` broken - https://phabricator.wikimedia.org/T192469#4140079 (10Reedy) 05duplicate>03Open Wrong one
[17:20:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Scap: Scap syntax error on beta - https://phabricator.wikimedia.org/T192470#4140083 (10Reedy)
[17:20:16] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4140085 (10Reedy)
[17:20:23] <wikibugs>	 10Beta-Cluster-Infrastructure: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140086 (10MarcoAurelio)
[17:20:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-log-errors: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140100 (10MarcoAurelio) p:05Triage>03Unbreak!
[17:21:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4139995 (10MarcoAurelio)
[17:21:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-log-errors: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140104 (10MarcoAurelio)
[17:25:19] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:25:21] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:25:24] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:26:18] <wikibugs>	 10Beta-Cluster-Infrastructure: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140122 (10EddieGP)
[17:26:37] <wikibugs>	 (03Merged) 10jenkins-bot: Create mediawiki-core-php70lint job [integration/config] - 10https://gerrit.wikimedia.org/r/427421 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:26:49] <wikibugs>	 10Beta-Cluster-Infrastructure: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140125 (10MarcoAurelio)
[17:26:56] <wikibugs>	 (03Merged) 10jenkins-bot: Add mwgate-php70lint [integration/config] - 10https://gerrit.wikimedia.org/r/427422 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:26:59] <wikibugs>	 (03Merged) 10jenkins-bot: Don't trigger PHP 5.5 jobs on REL1_31 [integration/config] - 10https://gerrit.wikimedia.org/r/427423 (https://phabricator.wikimedia.org/T190547) (owner: 10Legoktm)
[17:31:20] <eddiegp>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find class role::mediawiki::imagescaler for deployment-mediawiki-09.deployment-prep.eqiad.wmflabs on node deployment-mediawiki-09.deployment-prep.eqiad.wmflabs
[17:31:58] <eddiegp>	 Could any deployment-prep project admin remove the mediawiki::imagescaler role from the deployment-mediawiki-* prefix puppet in horizon please?
[17:32:12] <eddiegp>	 It was deleted in puppet.git but is still applied to those hosts.
[17:32:54] <wmf-insecte>	 Yippee, build fixed!
[17:32:54] <wmf-insecte>	 Project beta-scap-eqiad build #204339: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204339/
[17:33:13] <wikibugs>	 (03PS1) 10Legoktm: Drop override for mediawiki-extensions-php55-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/427452
[17:33:29] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Drop override for mediawiki-extensions-php55-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/427452 (owner: 10Legoktm)
[17:34:11] <legoktm>	 !log deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
[17:34:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:34:45] <wikibugs>	 (03Merged) 10jenkins-bot: Drop override for mediawiki-extensions-php55-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/427452 (owner: 10Legoktm)
[17:34:46] <wmf-insecte>	 Project beta-scap-eqiad build #204340: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204340/
[17:35:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4140174 (10MarcoAurelio) Some data can be found at: https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*...
[17:36:46] <DowagerCountess>	 eddiegp: wrt the global rename bug at beta I've found on kibana this: Could not connect to server "/var/run/nutcracker/redis_eqiad.sock"
[17:36:55] <DowagerCountess>	 though not sure if it is related or not
[17:37:02] <DowagerCountess>	 it's hard for me to search on logstash
[17:39:12] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#4140190 (10Legoktm)
[17:39:16] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support, 10Patch-For-Review: Make Wikimedia CI run PHP in either PHP 7.0+ or HHVM - https://phabricator.wikimedia.org/T190547#4140186 (10Legoktm) 05Open>03Resolved a:03Legoktm Should be all good now!
[17:42:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Samtar → There'sNoTime - https://phabricator.wikimedia.org/T192476#4140199 (10MarcoAurelio)
[17:46:21] <wmf-insecte>	 Yippee, build fixed!
[17:46:22] <wmf-insecte>	 Project beta-scap-eqiad build #204341: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204341/
[17:46:42] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikidata, and 2 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4140213 (10greg)
[17:47:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Samtar → There'sNoTime - https://phabricator.wikimedia.org/T192476#4140218 (10MarcoAurelio) https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2018.04.18/mediawiki?id=AWLZN...
[17:48:51] <eddiegp>	 hauskatze: Hmm, I really don't know anything about global renames.
[17:49:36] <hauskatze>	 legoktm: it seems some sort of nutcracker/redis stuff
[17:49:42] <hauskatze>	 I don't know what that means
[17:49:50] <hauskatze>	 + some object cache stuff as well
[17:49:53] <hauskatze>	 what a mess
[17:53:53] <awight>	 marlier: hey, so the virtualenv work happened under task T181071
[17:53:53] <stashbot>	 T181071: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071
[17:54:15] <marlier>	 Nice, I'll take a look at it.
[17:54:40] <awight>	 marlier: Most of the action was just tangential and non-specific support in scap, for running scripts at various stages of the deployment workflow.
[17:54:48] <marlier>	 Makes sense.
[17:55:06] <marlier>	 I'd actually be curious to find out more about your general deployment strategy re: virtualenvs
[17:55:18] <marlier>	 Having to package python things as .deb in order to use them is the worst.
[17:55:42] <awight>	 The key source stuff is quite simple: https://github.com/wikimedia/mediawiki-services-ores-deploy/blob/master/scap/scripts/fetch_check.sh
[17:56:03] <awight>	 oh.  yeah we tried the packaging thing and it was like 2 full-time volunteer positions.
[17:56:32] <awight>	 virtualenv is a crappy workaround, actually...  The long-term plan is to use docker.
[17:57:01] <awight>	 Here's the scap glue to run our script, https://github.com/wikimedia/mediawiki-services-ores-deploy/blob/master/scap/checks.yaml#L2-L6
[17:57:25] <wikibugs>	 (03CR) 1020after4: [C: 032] [Wikibase] Temporarily remove flaky mwext-mw-selenium-composer-jessie from test [integration/config] - 10https://gerrit.wikimedia.org/r/425929 (https://phabricator.wikimedia.org/T189762) (owner: 10Jforrester)
[17:58:19] <awight>	 marlier: Ah, one more idiosyncracy, we have to build and commit python wheels in order to accomplish all this.  https://phabricator.wikimedia.org/source/ores-deploy-wheels/
[17:58:39] <wikibugs>	 (03Merged) 10jenkins-bot: [Wikibase] Temporarily remove flaky mwext-mw-selenium-composer-jessie from test [integration/config] - 10https://gerrit.wikimedia.org/r/425929 (https://phabricator.wikimedia.org/T189762) (owner: 10Jforrester)
[17:58:42] <marlier>	 Yeah, I was just looking at that :-)
[17:58:56] <marlier>	 awight: thanks much, this is really helpful
[17:59:02] <awight>	 The code to build wheels: https://github.com/wiki-ai/ores-wmflabs-deploy/blob/master/Makefile
[17:59:20] <awight>	 Ping us any time, here or in #wikimedia-ai
[17:59:46] <awight>	 We're probably stuck with this strategy for the next 6-12 months, so any refinements are welcomed.
[18:00:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10mmodell) Ok I +2'd the change to integration config. Someone with more know...
[18:02:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471#4140257 (10MarcoAurelio) Fixed while running:   ``` maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/m...
[18:03:13] <hauskatze>	 !log maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - T192471
[18:03:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:03:15] <stashbot>	 T192471: Please unblock stuck global rename Artix Kreiger 2 to Artix Krieger 2 - https://phabricator.wikimedia.org/T192471
[18:05:52] <twentyafterfour>	 SMalyshev re: T189811 - I can add a password to the jenkins credential store but need to have that communicated to me in a secure way. Should we create a private phabricator task or ....exchange pgp keys?
[18:05:52] <stashbot>	 T189811: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811
[18:09:00] <awight>	 marlier: last thing, I just wanted to be sure that you got the Makefile link above.
[18:09:12] <marlier>	 awight: Yep!
[18:10:01] <marlier>	 I'll need to look through this a bit to make sure that I understand what's actually happening
[18:10:05] <marlier>	 But I think it all makes sense.
[18:10:15] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikidata, and 2 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4054206 (10mmodell) I can add a password to the jenkins credential store but need to have that communicated to m...
[18:11:39] <marlier>	 awight: I'm tempted to suggest a "common" wheels repo for things that are more general purpose (eg, I want to get an aiokafka wheel ready, which is something that others would probably also want to use).  At a certain point we're just rebuilding pip, though :-)
[18:20:00] <awight>	 marlier: That's a great idea, and is consistent with the way we handle other packages, i.e. the production apt mirror.  Looks like pip supports 3rd-party repos supporting PEP 503.
[18:21:13] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4140315 (10CCicalese_WMF)
[18:21:42] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4115121 (10CCicalese_WMF)
[18:23:20] <marlier>	 awight: Yeah -- last company, we just ran an internal pip repo, and then configured /etc/pip.conf (I think that's the file name, anyway).  The repo layout is super simple: https://packaging.python.org/guides/hosting-your-own-index/
[18:25:11] <awight>	 \o/
[18:31:42] <wikibugs>	 10Release-Engineering-Team, 10Scoring-platform-team, 10Performance: Investigate PEP 503 repo for production deployment of python wheels - https://phabricator.wikimedia.org/T192478#4140331 (10awight)
[18:41:41] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikidata, and 2 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4140359 (10Smalyshev) @mmodell If you send me an email with your public key (smalyshev@wikimedia) I think this s...
[18:49:09] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikimedia-log-errors (Jenkins Failure): selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10Krinkle)
[19:02:50] <Reedy>	 Where is mediawiki-phpunit-hhvm-composer-jessie actually defined?
[19:03:21] <paladox>	 Is mediawiki-phpunit-hhvm-composer that a nodepool job?
[19:04:18] <Reedy>	 I don't think so
[19:04:29] <Reedy>	 Or maybe it is
[19:04:41] <Reedy>	 I guess it is
[19:05:29] <thcipriani>	 https://github.com/wikimedia/integration-config/blob/master/jjb/mediawiki.yaml#L148-L179
[19:05:51] <Reedy>	 yay, wildcards :)
[19:06:08] <Reedy>	 cheers
[19:06:10] <thcipriani>	 makes everything in that repo impossible to grep for :)
[19:07:01] <wikibugs>	 (03Draft2) 10Reedy: Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469
[19:07:05] <Reedy>	 Krinkle: ^
[19:07:49] <hashar>	 Reedy: the jobs are shared between branches!
[19:08:20] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[19:09:24] <Krinkle>	 Reedy: That job doesn't exist yet.
[19:09:27] <Krinkle>	 The jjb config needs to be updated.
[19:09:38] <Krinkle>	 This is the zuul config, which connects the gerrit repos, zuul pipelines with jenkins jobs
[19:10:19] <Krinkle>	 Reedy: Down on line 277 in mediawiki.yaml, add php70 to the expansion
[19:10:41] <Krinkle>	 Might as well remove the hhvm one, given it's only used in one place, and not used after this
[19:11:55] <wikibugs>	 (03PS3) 10Reedy: Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469
[19:16:11] <Krinkle>	 Yay, it passes
[19:16:28] <Krinkle>	 I didn't realise it validates against current would-be compilation
[19:16:29] <Krinkle>	 that's nice
[19:16:43] <Krinkle>	 Reedy: Wanna do the honours of generating/pushing the new job?
[19:16:56] <Reedy>	 I don't think I have a deployment setup for it
[19:17:29] <Krinkle>	 it requires a copy of integration/jenkins-job-builder, python, and a ldap/wmf login for Jenkins
[19:17:56] <Krinkle>	 https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder
[19:17:59] <Reedy>	 I think addshore wrote a docker thing :P
[19:18:06] <Krinkle>	 It takes a bit, but not too hard.
[19:18:16] <Krinkle>	 Might be useful for future reference if you've got it.
[19:18:23] <Krinkle>	 Right now it's only a handful of people
[19:18:50] <Reedy>	 I usually use the sudo method
[19:19:14] <Krinkle>	 Once it's all set up including the .ini file
[19:19:18] <Krinkle>	 then it's just `jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/`
[19:19:19] <Reedy>	 Let's have another look
[19:19:24] <Krinkle>	 verify the php70 file exists in output/
[19:19:39] <Krinkle>	 and then run the 'update' command with the job name as parameter
[19:20:01] <Krinkle>	 `jenkins-jobs --conf etc/jenkins_jobs.ini update config/  "mediawiki*php70-composer-jessie"`
[19:20:03] <Krinkle>	 Or some such
[19:20:50] * Krinkle Reedy Let me know if you need anything. I've had it fail on me many times. Especially the install step. For some reason my combination of macOs, homebrew and pythons, is a recipe for screwing up any reasonable pip install.
[19:21:01] <Reedy>	 I tend to do this stuff on *nix
[19:21:05] <Krinkle>	 I actually use virtualenv at the moment, but forgot how I got it to work.
[19:21:59] <Reedy>	 almost should be a vagrant role for it ;)
[19:23:24] <bd808>	 Reedy: there should be a jenkins job for it ;)
[19:23:29] <Reedy>	 Touche
[19:23:33] <hashar>	 Reedy: Krinkle: that mediawiki-phpunit-php70-composer-jessie  it is going to unneeded 
[19:23:39] <hashar>	 erger
[19:23:43] <hashar>	 my english is crap sorry
[19:23:46] <Reedy>	 merd
[19:24:23] <Krinkle>	 hashar: Oh?
[19:24:37] <Krinkle>	 hashar: Do we have another job that uses composer install (instead of vendor/)
[19:25:18] <wikibugs>	 (03CR) 10Hashar: Run mediawiki-phpunit-composer-jessie on php7 not hhvm (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[19:25:47] <Reedy>	 Do we care that we don't run composer install with hhvm on other branches?
[19:25:54] <wikibugs>	 (03CR) 10Krinkle: Run mediawiki-phpunit-composer-jessie on php7 not hhvm (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[19:26:26] <wikibugs>	 (03PS1) 10Hashar: Use quibble on mediawiki/core@master [integration/config] - 10https://gerrit.wikimedia.org/r/427472
[19:26:39] <hashar>	 Reedy: Krinkle:  https://gerrit.wikimedia.org/r/427472 Use quibble on mediawiki/core@master        
[19:26:45] <wikibugs>	 (03CR) 10Krinkle: Run mediawiki-phpunit-composer-jessie on php7 not hhvm (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[19:26:51] <hashar>	 that is probably going to be the replacement
[19:27:21] <Krinkle>	 Let's keep quibble out of this for the moment. Priority is first to fix php5 errors on REL1_31/master.
[19:27:37] <Krinkle>	 Right now, the hhvm job is breaking that because unlike the other hhvm job, the composer-test triggers a bug in composer.json
[19:27:39] <Krinkle>	 that we cannot fix.
[19:27:46] <hashar>	 :(
[19:27:54] <Krinkle>	 And that job is not about php versions, it is about composer-install, so that ony needs 1 version, and can be php7 instead of hhvm, right?
[19:28:18] <wikibugs>	 (03CR) 10Reedy: Run mediawiki-phpunit-composer-jessie on php7 not hhvm (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[19:28:20] <Krinkle>	 We will still have hhvm job on all branches, but not the one that uses composer-install.
[19:28:39] <Reedy>	 Considering we don't run composer install in production (ie on hhvm)...
[19:29:30] <Krinkle>	 We previously also don't have composer-install variations for php55 and php70. And we don't need every combination of (composer, vendor)(php55,77,hh)(mysql,sqlite,pgsql).
[19:29:38] <Krinkle>	 This test is about composer and can have just one php and db version.
[19:33:17] <Reedy>	 ~/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini test config/ -o output/
[19:33:18] <Reedy>	 -bash: jenkins-jobs: command not found
[19:35:56] <Krinkle>	 Reedy: Might not be in PATH, or need ../jenkins-job-builder/venv/bin/jenkins-jobs
[19:36:00] <Krinkle>	 depending on how/where it was installed
[19:37:19] <Reedy>	 I've no idea where the venv is
[19:38:24] <Reedy>	 no venv folder in the jjb git clone
[19:40:29] <Krinkle>	 Reedy: Yes, there's a step 2 after the clone.
[19:40:30] <Krinkle>	 pip install
[19:40:35] <Reedy>	 I ran that
[19:40:45] <Krinkle>	 Which one :)
[19:40:51] <Reedy>	 reedy@ubuntu64-web-esxi:~/jenkins-job-builder$ pip install -e .
[19:40:51] <Reedy>	 Obtaining file:///home/reedy/jenkins-job-builder
[19:40:51] <Reedy>	 Collecting six>=1.9.0 (from jenkins-job-builder==0.0.1.dev2371)
[19:41:04] <Reedy>	 etc
[19:41:18] <Krinkle>	 OK. Does it say where it created bin/ ?
[19:41:28] <Reedy>	 No
[19:41:29] <Reedy>	 Installing collected packages: six, PyYAML, pbr, stevedore, multi-key-dict, python-jenkins, monotonic, fasteners, jenkins-job-builder
[19:41:29] <Reedy>	   Running setup.py develop for jenkins-job-builder
[19:41:29] <Reedy>	 Successfully installed PyYAML-3.12 fasteners-0.14.1 jenkins-job-builder monotonic-1.4 multi-key-dict-2.0.3 pbr-4.0.2 python-jenkins-0.4.16 six-1.11.0 stevedore-1.28.0
[19:42:08] <Reedy>	 https://phabricator.wikimedia.org/P7011
[19:44:09] <Krinkle>	 Reedy: It'll be in one of the directires output by `python -m site`
[19:44:22] <Krinkle>	 Which one one is the right one, that one'll need to be in PATH and/or specify directly as command path
[19:44:50] <Krinkle>	 It's unfortunate that pip both 1) doesn't  tell you where it went and 2) doesn't  check if that is missing from PATH
[19:45:02] <Reedy>	 ~/.local/bin/jenkins-jobs
[19:45:04] <Reedy>	 helpful
[19:45:36] <Reedy>	 ta
[19:46:27] * Reedy waits for it to run
[19:47:57] <Reedy>	 INFO:jenkins_jobs.builder:Number of jobs generated:  299
[19:58:39] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4140587 (10thcipriani) p:05Triage>03High From scap today: `19:37:56 Finished l10n-update (duration: 39m 12s)` 30-40 minutes beta-scap-eqiad runs: h...
[19:59:57] <Reedy>	 Krinkle: Do we need to change that other line?
[20:00:22] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T183969#4140589 (10Aklapper) https://phabricator.wikimedia.org/T192488 might also be a candidate?: Anchors in resulting URLs don't work after performing a section edit
[20:00:34] <Krinkle>	 Reedy: Other line?
[20:00:58] <Krinkle>	 the skip-if
[20:00:59] <Krinkle>	 yeah
[20:01:00] <Reedy>	 Your comment on 795
[20:01:25] <Krinkle>	 Depends
[20:01:34] <Krinkle>	 Does php70 pass on 1.29+?
[20:01:42] <Krinkle>	 I was somewhat surprised to see the current regex
[20:01:53] <Krinkle>	 But I guess it doesn't pass cleanly
[20:01:57] <Krinkle>	 so we'll just not have this job on those branches
[20:01:59] <Reedy>	 We've backported numerous fixes
[20:02:15] <Krinkle>	 If we enable it by changing that branch pattern, it'll apply to all php7 jobs
[20:02:16] <Krinkle>	 including this one
[20:02:18] <Krinkle>	 so I'd say include for now
[20:02:35] <Krinkle>	 g2g, meeting
[20:03:47] <Reedy>	 That .24 skip can go I guess
[20:06:51] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865#4140621 (10mmodell) @aklapper: Unfortunately, no it only works for newly created tasks that use the 'deadline' form to set the subtype of the task.  I could add the field to all tasks but...
[20:08:09] <wikibugs>	 (03PS2) 10Hashar: Use quibble on mediawiki/core@master [integration/config] - 10https://gerrit.wikimedia.org/r/427472
[20:08:11] <wikibugs>	 (03PS1) 10Hashar: Tox tests should use python2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/427479
[20:08:39] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T183969#4140622 (10thcipriani) >>! In T183969#4140589, @Aklapper wrote: > https://phabricator.wikimedia.org/T192488 might also be a candidate?: Anchors in resulting URLs...
[20:08:53] <hashar>	 Krinkle: Reedy: yup php7.0 should pass on 1.29
[20:09:07] <hashar>	 though parser tests require an old libtidy version
[20:09:52] <andrewbogott>	 hashar: did you wind up using labtestservices2003 for anything?  I have some things to clean up if you aren't using it.
[20:10:04] <wikibugs>	 (03CR) 10Hashar: [C: 032] Tox tests should use python2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/427479 (owner: 10Hashar)
[20:10:14] <hashar>	 andrewbogott: nop :(( been busy with too many things
[20:10:22] <andrewbogott>	 great, I'll reset it
[20:10:25] <hashar>	 andrewbogott: so you can clean it up of anything youd id for me I guess! 
[20:11:14] <mdholloway>	 !log deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to fb161d7
[20:11:16] <wikibugs>	 (03Merged) 10jenkins-bot: Tox tests should use python2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/427479 (owner: 10Hashar)
[20:11:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:11:28] <Krinkle>	 hashar: Cool. Once that is confirmed we can loosen the pattern in general. 
[20:11:41] <Krinkle>	 Reedy: Did you push the job to Jenkins?
[20:11:44] <hashar>	 but what are you trying to do ?
[20:11:56] <Reedy>	 Krinkle: No... Because I'm not 100% sure what you want me to change to what
[20:12:11] <hashar>	 Krinkle: Reedy what is the problem you are currently facing ?
[20:12:13] <Krinkle>	 hashar: I was just curious why we do skip-if on php70 jobs for REL1_27-30. 
[20:12:20] <Reedy>	 hashar: We said before :P
[20:12:34] <Reedy>	 We can't run composer install on hhvm on REL1_31 after bumping required PHP from 5.5 to 7.0
[20:12:38] <Krinkle>	 Reedy: You currnetly have the new job name in your output/ dir, yes?
[20:12:39] <Reedy>	 Because HHVM isn't in php7 mode
[20:12:47] <Reedy>	 No, I've not pulled anything
[20:12:58] <hashar>	 so for REL1_31, you want to drop hhvm and php5 right?
[20:13:01] <Krinkle>	 Right, the config/ dir needs to be a symlink to int/config.git chekcout with the patch applied
[20:13:04] <Reedy>	 moment
[20:13:08] <Krinkle>	 hashar: No
[20:13:09] <Reedy>	 I've done the symlink
[20:13:25] <Reedy>	 hashar: only for composer. we want to s/hhvm/php70/
[20:14:22] <Krinkle>	 hashar: This has nothing to do with hhvm/php itself. Tests and install work on php5/7/hhvm. The problem is installing with 'composer install'. That step fails on HHVM (from 1 job only, because we only do this in one job, the rest uses vendor). It fails because composer.json is verified by composer and will soon contain 'require: php: 7.0' which is false on hhvm.
[20:14:28] <Krinkle>	 It doesn't matter and isn't important.
[20:14:50] <Krinkle>	 We still want to test without vendor for correctness, but that version of the job can use php70 instead.
[20:15:50] <Reedy>	 it's created mediawiki-phpunit-php70-composer-jessie
[20:16:13] <Reedy>	 case &quot;php70&quot; in
[20:16:42] <Reedy>	 The diff between the two files looks sane Krinkle
[20:16:52] <Reedy>	 hhvm swapped for php70
[20:17:14] <Krinkle>	 Yeah, now push it to jenkins by using 'update' and pass the job name (following the example of the wiki page)
[20:17:22] <Krinkle>	 Assuming you have the ini file set up with API key
[20:17:34] <Reedy>	 I did
[20:17:38] <Reedy>	 I did follow the instructions
[20:17:45] <Reedy>	 As much as you seem to think I didn't ;)
[20:18:49] <hashar>	 Reedy: Krinkle dont we have a switch in composer to ignore the platform requirements?
[20:18:54] <hashar>	 I think I encountered that recently
[20:19:04] <Krinkle>	 Yes, it exists.
[20:19:11] <hashar>	       --ignore-platform-reqs     Ignore platform requirements (php & ext- packages).
[20:19:17] <hashar>	 I think of that one
[20:19:30] <Krinkle>	 But given we only have 1 version of this job, and php70+mysql will be supported the longest (and the job is not about php or sql), makes sense to use something less volatile than php5 or hhvm.
[20:19:33] <Krinkle>	 So it can stay this way longer.
[20:20:00] <Reedy>	 INFO:jenkins_jobs.builder:Number of jobs generated:  1
[20:20:00] <Reedy>	 INFO:jenkins_jobs.builder:Creating jenkins job mediawiki-phpunit-php70-composer-jessie
[20:20:00] <Reedy>	 INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 1
[20:20:17] <Krinkle>	 Cool
[20:20:17] <Krinkle>	 https://integration.wikimedia.org/ci/job/mediawiki-phpunit-php70-composer-jessie/
[20:20:52] <Reedy>	 Try and re-merge james' patch?
[20:21:10] <hashar>	 for what it is worth, those jobs are going to disappear anyway  (the  mediawiki-phpunit-{phpflavor}-composer-{image}  and mediawiki-phpunit-{phpflavor}-{image}  )
[20:22:00] <Reedy>	 Oh, need to do the zuul part too?
[20:22:24] <Krinkle>	 Reedy: Yeah, update that oneline, merge in git, and then fab deploy_zuul
[20:22:36] <Reedy>	 I've not got fab setup :P
[20:22:42] <Krinkle>	 Reedy: It's just ssh + one command
[20:22:44] <Krinkle>	 you can do it without fab :)
[20:22:53] <Krinkle>	 well, two
[20:23:20] <Reedy>	 I'm still not 100% sure what line you want me to change to what :P
[20:23:30] <Reedy>	 The regex as I posted?
[20:23:34] <Krinkle>	 Yeah, LGTM
[20:24:01] <wikibugs>	 (03PS4) 10Reedy: Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469
[20:24:04] <wikibugs>	 (03CR) 10Reedy: [C: 032] Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[20:25:36] <wikibugs>	 (03Merged) 10jenkins-bot: Run mediawiki-phpunit-composer-jessie on php7 not hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/427469 (owner: 10Reedy)
[20:29:02] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4140674 (10mmodell)
[20:29:45] <Reedy>	 !log reloaded zuul for 427469
[20:29:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:30:01] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4128785 (10mmodell)
[20:30:09] <paladox>	 Reedy wondering could you merge https://gerrit.wikimedia.org/r/#/c/427533/ please?
[20:30:35] <wikibugs>	 (03PS1) 10Hashar: quibble doesn't need a DISPLAY [integration/config] - 10https://gerrit.wikimedia.org/r/427534
[20:30:49] <paladox>	 thanks Reedy :)
[20:30:50] <wikibugs>	 (03CR) 10Hashar: [C: 032] quibble doesn't need a DISPLAY [integration/config] - 10https://gerrit.wikimedia.org/r/427534 (owner: 10Hashar)
[20:31:43] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4140683 (10mmodell) @fgiunchedi Can you upload the package? the tag is at rMSCA004f7635ff44
[20:32:02] <wikibugs>	 (03Merged) 10jenkins-bot: quibble doesn't need a DISPLAY [integration/config] - 10https://gerrit.wikimedia.org/r/427534 (owner: 10Hashar)
[20:32:07] <hashar>	 !log reloaded zuul for https://gerrit.wikimedia.org/r/427534
[20:32:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:32:13] <wmf-insecte>	 Project mwext-phpunit-coverage-publish build #3561: 04FAILURE in 7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3561/
[20:32:37] <Reedy>	 Krinkle: https://integration.wikimedia.org/ci/job/mediawiki-phpunit-php70-composer-jessie/2/console LGTM
[20:33:02] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4140686 (10mmodell) a:05mmodell>03None
[20:33:18] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4128785 (10mmodell) p:05Triage>03Normal
[20:33:27] <Krinkle>	 Reedy: nice
[20:34:27] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4128785 (10mmodell)
[20:34:43] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4128785 (10mmodell) a:03fgiunchedi
[20:35:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for access to the beta cluster - https://phabricator.wikimedia.org/T190755#4140694 (10mpopov) 05Open>03Resolved a:03mpopov Thanks for the recommendation! I've added `eventlogging@127.0.0.1` as a user with the same password.
[20:35:58] <Reedy>	 mediawiki-phpunit-php70-composer-jessie SUCCESS in 4m 04s
[20:36:02] <Reedy>	 James_F: ^^
[20:36:13] <wmf-insecte>	 Yippee, build fixed!
[20:36:14] <wmf-insecte>	 Project mwext-phpunit-coverage-publish build #3562: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3562/
[20:36:24] <James_F>	 Gosh.
[20:36:31] <James_F>	 It's actually happening.
[20:36:39] <Reedy>	 I wonder how quick it was on HHVM
[20:36:53] <Reedy>	 mediawiki-phpunit-hhvm-composer-jessie SUCCESS in 7m 00s
[20:36:54] <Reedy>	 Haha
[20:36:55] <Reedy>	 Nice
[20:37:08] <Reedy>	 PHP7 is 3/7ths quicker than HHVM ;)
[20:37:16] <James_F>	 Unsurprising.
[20:38:56] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Release: Deploy Scap 3.8.0 to production - https://phabricator.wikimedia.org/T192124#4140706 (10mmodell)
[20:39:47] <wikibugs>	 (03PS3) 10Hashar: Use quibble on mediawiki/core@master [integration/config] - 10https://gerrit.wikimedia.org/r/427472
[20:44:36] <Reedy>	 https://www.mediawiki.org/wiki/Compatibility#PHP
[20:44:40] <Reedy>	 This might be fun
[20:48:45] <paladox>	 hmm i get
[20:48:49] <paladox>	 ErrorException from line 347 of /srv/mediawiki/w/includes/context/RequestContext.php: PHP Warning: Recursion detected in RequestContext::getLanguage with php7.2
[20:53:52] <wikibugs>	 (03CR) 10Hashar: "I will deploy it Thursday on my morning then write some doc and send an announce to wikitech-l." [integration/config] - 10https://gerrit.wikimedia.org/r/427472 (owner: 10Hashar)
[20:56:48] <paladox>	 no_justification should this https://github.com/wikimedia/mediawiki-vendor be branched for REL1_31?
[20:57:44] <Reedy>	 Yes. He was going to do it earlier
[20:57:56] <paladox>	 oh
[20:58:35] <no_justification>	 I forgot
[20:58:48] <no_justification>	 And I'm afk now
[20:59:21] <Reedy>	 Easy to do
[20:59:49] <Hauskatze>	 tgr|away: really away?
[21:00:05] <Hauskatze>	 [9a5e71edaef5850eecbeff76] /rpc/RunJobs.php?wiki=enwiki&type=enqueue&maxtime=60&maxmem=300M   ErrorException from line 238 of /srv/mediawiki/php-master/includes/libs/redis/RedisConnectionPool.php: PHP Warning: Failed connecting to redis server at unix:///var/run/nutcracker/redis_eqiad.sock: No such file or directory
[21:00:19] <Hauskatze>	 breaking stuff, etc.
[21:00:33] <Reedy>	 $ git push origin REL1_31
[21:00:33] <Reedy>	 Total 0 (delta 0), reused 0 (delta 0)
[21:00:33] <Reedy>	 remote: Processing changes: refs: 1, done    
[21:00:33] <Reedy>	 To ssh://gerrit.wikimedia.org:29418/mediawiki/vendor.git
[21:00:33] <Reedy>	  * [new branch]      REL1_31 -> REL1_31
[21:00:40] <Reedy>	 no_justification: paladox done
[21:00:47] <paladox>	 Reedy thanks :)
[21:00:58] <paladox>	 Reedy i guess php7.2 is not supported on REL1_30?
[21:01:05] <paladox>	 it's broken the ui for me
[21:01:19] <Reedy>	 Honestly no idea
[21:15:36] <Hauskatze>	 !log Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
[21:15:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:16:05] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4140820 (10CCicalese_WMF) @Legoktm Is this one ready to go, or do you still need to check the licensing re: Debian?
[21:16:20] <Hauskatze>	 !log (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
[21:16:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:16:37] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-ReplaceText, 10MW-1.31-release: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4140842 (10CCicalese_WMF)
[21:17:43] <no_justification>	 Now it needs to be a submodule in core on rel1_31 :p
[21:18:01] <James_F>	 no_justification: Have fun.
[21:19:21] <no_justification>	 That remains to be determined :p
[21:22:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140892 (10MarcoAurelio)
[21:22:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140086 (10MarcoAurelio) @hoo and @Rxy suggests this may be nutcracker refusing to stay up and running. Granted, w...
[21:23:30] <eddiegp>	 Could any deployment-prep project admin remove the mediawiki::imagescaler role from the deployment-mediawiki-* prefix puppet in horizon please? It was deleted in puppet.git but is still applied to those hosts.
[21:23:34] * eddiegp repeats myself
[21:24:01] <eddiegp>	 If not, I'll just create a task, go figure out who removed it from puppet.git, and assign it to them :P
[21:24:18] <Reedy>	 Probably easier said than done
[21:24:45] <eddiegp>	 Which part?
[21:25:01] <Reedy>	 puppet via horizon
[21:25:05] * Reedy waits for the spinner
[21:25:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Notice: Undefined index: channels in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 340 - https://phabricator.wikimedia.org/T192473#4140918 (10hoo) ``` hoo@deployment-jobrunner03:~$ sudo tail /var/log/nutcracker/nutcracker.log [2018-04-18 21:18:5...
[21:26:52] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-extensions-ReplaceText, 10MW-1.31-release: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4140924 (10Yaron_Koren) Oh, I just looked at this set of tasks now, and unfortunately, it turns out that Replace Text doesn't support SQLite - s...
[21:27:19] <Hauskatze>	 eddiegp: so finally that global rename stuff is the redis/nutcracker thingie
[21:27:46] <eddiegp>	 Hauskatze: Yeah, thought so.
[21:28:25] <eddiegp>	 I actually had a look previously, but puppet being broken on the redis hosts for weeks didn't exactly help me understanding what the problem is.
[21:28:26] <Hauskatze>	 now the question is how to fix that
[21:28:35] <Hauskatze>	 ja, genau
[21:30:14] * eddiegp goes filing the 'puppet broken on beta mediawiki instances' task now
[21:31:26] <Reedy>	 eddiegp: role::mediawiki::imagescaler removed from deployment-mediawiki-*
[21:31:47] <eddiegp>	 Reedy: Thanks!
[21:32:48] <eddiegp>	 Yay, another error!
[21:32:59] <Hauskatze>	 deployment-fluorine02 is getting short of space
[21:33:11] <eddiegp>	 The following packages have unmet dependencies:
[21:33:13] <eddiegp>	  hhvm-dbg : Depends: hhvm (= 3.18.5+dfsg-1+wmf5+deb9u1) but 3.18.5+dfsg-1+wmf7+deb9u1 is to be installed
[21:33:14] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4140938 (10thcipriani) I managed to grab a better perf report by using `PHP='hhvm -vEval.PerfPidMap=true'`. Then I waited for beta-scap-eqiad to start...
[21:35:12] <Hauskatze>	 what about taking down deployment-jobrunner02 as well? it's down on shinken
[21:35:32] <eddiegp>	 Hauskatze: It's already down
[21:35:53] <Hauskatze>	 remove from shinken then I mean
[21:36:49] <eddiegp>	 j.oe shutdown mediawiki0[456] and jobrunner02 earlier this week and will delete them by the end of this week if no reason pops up to start them again
[21:37:13] <eddiegp>	 They're replaced by mediawiki-0[79] and jobrunner03, respectively.
[21:38:19] <Hauskatze>	 this day so far, 7,732,854 errors on d-jobrunner03
[21:38:50] <eddiegp>	 There's no such thing as "remove from shinken". If an instance exists, shinken will check whether it's up. Shutting it down and leave it in that state for a few days to be able to restart it if needed is a good thing. We'll get rid of the shinken alerts when the instance is deleted.
[21:40:51] <eddiegp>	 Which j.oe said he'll do by the end of this week.
[21:44:12] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: redis/nutcracker down on deployment-prep - https://phabricator.wikimedia.org/T192473#4140958 (10MarcoAurelio)
[21:50:10] <paladox>	 looks like the admin page in polygerrit is being redesgned
[21:51:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: redis/nutcracker down on deployment-prep - https://phabricator.wikimedia.org/T192473#4140979 (10MarcoAurelio)
[21:51:22] <wikibugs>	 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Please unblock stuck global rename Samtar → There'sNoTime - https://phabricator.wikimedia.org/T192476#4140978 (10MarcoAurelio)
[21:52:56] <eddiegp>	 Hmm, puppet works on mediawiki09. Broken on mediawiki07 because it can't update hhvm, which is hold on 07 but not 09.
[21:59:51] <Hauskatze>	 what about using runJobs.php to unblock pending job queue tasks?
[22:01:56] <wikibugs>	 10Beta-Cluster-Infrastructure: Puppet errors on deployment-mediawiki07 - https://phabricator.wikimedia.org/T192507#4141033 (10EddieGP)
[22:10:10] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T183969#4141050 (10Aklapper) Garr. I misinterpreted https://tools.wmflabs.org/versions/ . Sorry for the noise!
[22:12:22] <paladox>	 should mediawiki be bumped to 1.32 on master?
[22:17:07] <eddiegp>	 Hauskatze: renaming is it's own job type, isn't it?
[22:17:29] <Hauskatze>	 eddiegp: well, it has two jobs: CentralAuthRename and LocalRenameJob iirc
[22:18:17] <eddiegp>	 Hauskatze: Yeah, the point is that you can make runJobs.php only run specific job types. I thing you're fine to go ahead if you don't run plain 'runJobs.php' but limit it to those two
[22:18:19] <Hauskatze>	 I /could/ run the script to unblock the rename as more than 3 hours have passed and thus it won't retry
[22:19:31] <Hauskatze>	 maybe with 'enqueue' job would be enough
[22:19:58] <Hauskatze>	 mwscript runJobs.php --wiki=deploymentwiki --type=enqueue ?
[22:20:03] <eddiegp>	 Lemme see how many jobs there are in the queue.
[22:20:33] <Hauskatze>	 showJobs returns a lot of crap
[22:20:40] <Hauskatze>	 but be my guest :)
[22:22:03] <Hauskatze>	 maurelio@deployment-tin:~$ mwscript showJobs.php --wiki=deploymentwiki
[22:22:04] <Hauskatze>	 [Wed Apr 18 22:21:36 2018] [hphp] [15909:7f79a7f13200:0:000001] []
[22:22:04] <Hauskatze>	 Notice: Undefined index: channels in /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php on line 340
[22:22:09] <eddiegp>	 Yeah, just saw that.
[22:22:20] <Hauskatze>	 this is just frustrating
[22:22:26] <eddiegp>	 But at the bottom it says 0
[22:22:35] <Hauskatze>	 yep
[22:22:36] <eddiegp>	 Probably the error is that it can't even read the job queue
[22:22:57] <eddiegp>	 So you can run showJobs/runJobs with whatever you want, it just won't do anything.
[22:22:58] <Hauskatze>	 with redis/nutcracker down it probably can't, if I read the docs correctly
[22:23:16] <eddiegp>	 We've got docs on that?
[22:23:22] <eddiegp>	 Do you have a link for me to read?
[22:23:37] <Hauskatze>	 it worked with centralauth/maintenance/fixStuckGlobalRename BUT it messes the unification dates
[22:23:47] <Hauskatze>	 so I'd prefer if we fix the job queue first
[22:24:02] <Hauskatze>	 https://wikitech.wikimedia.org/wiki/Nutcracker
[22:24:06] <Hauskatze>	 eddiegp: ^
[22:24:49] <eddiegp>	 So we have a wikitech page with that name, not documentation :D
[22:25:12] <eddiegp>	 (where "documentation" is "something that helps understanding roughly how the service works and where to look if it fails")
[22:25:28] <Hauskatze>	 do we have something like that anywhere? ;-)
[22:27:43] <eddiegp>	 I think it's improving. At least from what I see at devrel (new developers) and WMCS (toolforge) it's a high priority and getting better day by day.
[22:28:00] <Hauskatze>	 yep
[22:28:08] <eddiegp>	 Which doesn't necessarily mean that's the case everywhere though
[22:28:41] <Hauskatze>	 well, it's late here, hopefully somebody would take a look at the Phab task and fix whatever conf file is needed
[22:33:22] <paladox>	 oh wow
[22:33:23] <paladox>	 https://imgur.com/a/J0K3jnZ
[22:33:27] <paladox>	 that looks really nice
[22:52:22] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10PHP 7.0 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7 - https://phabricator.wikimedia.org/T144962#4141149 (10Krinkle)
[22:53:15] <eddiegp>	 !log root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref T178457
[22:53:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:53:18] <stashbot>	 T178457: nutcracker fails to start due to lack of /var/run/nutcracker (ex: deployment-videoscaler01 has memcached failures) - https://phabricator.wikimedia.org/T178457
[22:53:33] <eddiegp>	 !log started nutcracker on deployment-jobrunner03
[22:53:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:57:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: nutcracker fails to start due to lack of /var/run/nutcracker (ex: deployment-videoscaler01 has memcached failures) - https://phabricator.wikimedia.org/T178457#3692957 (10EddieGP) Saw the same on deployment...
[23:00:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: redis/nutcracker down on deployment-prep - https://phabricator.wikimedia.org/T192473#4140086 (10EddieGP) nutcracker now starts on deployment-jobrunner03, see my comment on T178457. The error in the task description still happens though.
[23:59:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Product-Analytics, 10Puppet: redis/nutcracker down on deployment-prep - https://phabricator.wikimedia.org/T192473#4141453 (10EddieGP) - @aaron merged 8ad186728 yesterday at 23:51 UTC. - [[https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10836/consol...