[00:10:39] 10Gerrit: Update gerrit's login ui - https://phabricator.wikimedia.org/T184778#3895960 (10Paladox) [00:11:54] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Update gerrit's login ui - https://phabricator.wikimedia.org/T184778#3895974 (10Paladox) [00:12:14] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Update gerrit's login ui - https://phabricator.wikimedia.org/T184778#3895960 (10Paladox) Needs someone from releng team to review before it is considered to be merged. [00:31:18] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:40:32] Heyhey, if anyone's around, I'm confused about this failure ... ? https://integration.wikimedia.org/ci/job/mwgate-npm-node-6-docker/13263/console [00:40:38] "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?." [00:41:18] I just ran grunt locally and I actually seem to have some eslint errors that I'm fixing now, but jenkins didn't show those errors, only the above one about docker [01:04:34] hrm, well that's not good. docker refuses to start on that box, dunno way. I'm going to take it offline and reboot. [01:07:49] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<33.33%) [01:16:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Docker won't start on integration-slave-docker-1005 - https://phabricator.wikimedia.org/T184781#3896043 (10thcipriani) [01:18:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Docker won't start on integration-slave-docker-1005 - https://phabricator.wikimedia.org/T184781#3896054 (10thcipriani) I disconnected this machine in jenkins for now. [01:21:22] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-ORES, 10Scoring-platform-team: How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#3896057 (10Legoktm) > If what I'm looking for doesn't exist, I'd be happy to split the maintenance testing... [01:22:50] RECOVERY - Free space - all mounts on integration-slave-docker-1002 is OK: OK: All targets OK [01:43:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Docker won't start on integration-slave-docker-1005 - https://phabricator.wikimedia.org/T184781#3896076 (10thcipriani) I think I found the problem: ``` thcipriani@integration-slave-docker-1005:~$ cat /etc/modprobe.d/blacklist-wmf_ov... [01:48:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Docker won't start on integration-slave-docker-1005 - https://phabricator.wikimedia.org/T184781#3896077 (10thcipriani) 05Open>03Resolved a:03thcipriani Added ``` profile::base::overlayfs: true ``` to horizon project puppet fo... [03:59:07] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [07:14:08] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:38:07] thanks for that end of day debugging, thcipriani|afk [09:28:55] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Update gerrit's login ui - https://phabricator.wikimedia.org/T184778#3896241 (10Aklapper) For better understanding, is this some 'default/upstream' update or rather a customization? [09:48:58] (03PS1) 10Hashar: Convert npm, npm-test to docker-pkg (2) [integration/config] - 10https://gerrit.wikimedia.org/r/403896 (https://phabricator.wikimedia.org/T177276) [09:57:57] (03PS2) 10Hashar: Convert npm, npm-test to docker-pkg (2) [integration/config] - 10https://gerrit.wikimedia.org/r/403896 (https://phabricator.wikimedia.org/T177276) [10:08:21] (03CR) 10Hashar: [C: 032] Convert npm, npm-test to docker-pkg (2) [integration/config] - 10https://gerrit.wikimedia.org/r/403896 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [10:09:42] (03Merged) 10jenkins-bot: Convert npm, npm-test to docker-pkg (2) [integration/config] - 10https://gerrit.wikimedia.org/r/403896 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [10:21:28] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-ORES, 10Scoring-platform-team: How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#3895786 (10Ladsgroup) Count me in if you want help, [10:30:07] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3896360 (10hashar) [10:30:09] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: npm 1.4.21 can't use a http proxy - https://phabricator.wikimedia.org/T183569#3896358 (10hashar) 05Open>03Resolved Awesome. That fixed the issue I had locally as well as when runni... [10:31:12] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3896364 (10hashar) a:05Joe>03hashar **Status update** Containers left... [10:46:41] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Update gerrit's login ui - https://phabricator.wikimedia.org/T184778#3896374 (10Paladox) This is customisation (this is based on markusguenther @ typo3 work). I did suggest on his change that this should be upstreamed as it looks really good. [10:58:28] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3896402 (10Aklapper) [11:10:57] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Zuul: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165#3896425 (10Paladox) @hashar couldn’t we backport the one from stretch to Jessie if it’s the newer one and works? [11:16:33] (03PS1) 10Ladsgroup: Remove BetaFeatures from ORES extension dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/403904 (https://phabricator.wikimedia.org/T184554) [11:37:40] (03PS1) 10Hashar: Migrate npm-test images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403907 (https://phabricator.wikimedia.org/T177276) [11:39:46] (03CR) 10Hashar: [C: 032] Migrate npm-test images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403907 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [11:41:06] (03Merged) 10jenkins-bot: Migrate npm-test images to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403907 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:18:49] (03PS1) 10Hashar: Migrate npm stretch containers to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403921 (https://phabricator.wikimedia.org/T177276) [13:27:19] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review, 10Release: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3896652 (10mmodell) 05Open>03Resolved a:03mmodell Thanks @fgiunchedi [13:27:33] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review, 10Release: SCAP: Upload debian package version 3.7.5-1 - https://phabricator.wikimedia.org/T184774#3896655 (10mmodell) a:05mmodell>03fgiunchedi [13:30:29] (03CR) 10Hashar: [C: 032] Migrate npm stretch containers to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403921 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [13:33:00] (03Merged) 10jenkins-bot: Migrate npm stretch containers to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403921 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:06:22] (03PS3) 10Zfilipin: Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 (https://phabricator.wikimedia.org/T180246) (owner: 10WMDE-Fisch) [14:09:02] (03CR) 10Zfilipin: [C: 032] Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 (https://phabricator.wikimedia.org/T180246) (owner: 10WMDE-Fisch) [14:10:15] (03Merged) 10jenkins-bot: Replace ruby browser tests with nodejs tests [integration/config] - 10https://gerrit.wikimedia.org/r/403639 (https://phabricator.wikimedia.org/T180246) (owner: 10WMDE-Fisch) [14:11:53] (03PS1) 10Hashar: Migrate mediawiki-phpcs to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403923 (https://phabricator.wikimedia.org/T177276) [14:11:57] !log Reloading Zuul to deploy 3816837e6d70ee7eb276c9b00034a5d2012a8afd [14:12:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:12:13] (03CR) 10Hashar: [C: 032] Migrate mediawiki-phpcs to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403923 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:13:25] (03Merged) 10jenkins-bot: Migrate mediawiki-phpcs to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403923 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:29:10] (03PS1) 10Hashar: Update ci-src-setup-simple example [integration/config] - 10https://gerrit.wikimedia.org/r/403924 [14:39:32] (03PS1) 10Hashar: Migrate mediawiki-phan to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403925 (https://phabricator.wikimedia.org/T177276) [14:41:18] (03CR) 10Legoktm: [C: 032] "I think given the disagreement over this sniff and the active work by some against it, we should probably remove it for now. (But that doe" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [14:42:09] (03Merged) 10jenkins-bot: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [14:42:17] (03CR) 10Hashar: [C: 032] Migrate mediawiki-phan to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403925 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:42:34] (03CR) 10Legoktm: [C: 032] Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [14:42:40] (03CR) 10jerkins-bot: [V: 04-1] Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [14:43:15] (03CR) 10jenkins-bot: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [14:43:34] (03Merged) 10jenkins-bot: Migrate mediawiki-phan to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/403925 (https://phabricator.wikimedia.org/T177276) (owner: 10Hashar) [14:52:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3896838 (10hashar) 05Open>03Resolved I have migrated all the remaining... [14:53:14] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:57:57] (03PS2) 10Umherirrender: Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 [14:58:21] (03CR) 10Umherirrender: "Patch Set 2: Fixed tests (conflicts with Ic2aa6c8f005d635d23a5c324cca48ab8fb4ad594)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [15:00:22] 10Release-Engineering-Team, 10DBA, 10Operations, 10cloud-services-team: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3896851 (10jcrespo) [15:02:03] 10Release-Engineering-Team, 10DBA, 10Operations, 10cloud-services-team: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3896861 (10jcrespo) Only adding #releng and #wmcs in case they can think of a reason not to move them (groupX reasons) or not to put labswiki there. [15:03:28] hashar: yay docker-pkg [15:14:15] 10MediaWiki-Codesniffer, 10Patch-For-Review: Remove FunctionComment.MissingParamComment from default rules - https://phabricator.wikimedia.org/T184650#3896877 (10Umherirrender) 05Open>03Resolved p:05Triage>03Normal a:03Reedy [15:22:09] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3896900 (10zeljkofilipin) [15:31:31] addshore: yeah eventually!!!!!!!!!!! :) [15:31:46] addshore: next week I am going to propose a patch to get rid of the ci-src-setup images [15:31:50] stay tuned :] [15:31:53] okaY! [15:43:25] y'all have so many of these "CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[scap]" [15:43:37] Is someone working on that, or is there a ticket? (I'm searching phab but can only find similar closed things) [16:20:54] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current): How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#3896986 (10awight) a:03awight [16:29:50] there's a scap ticket someplace yeah [16:32:20] (03PS1) 10Umherirrender: Detect variadic arguments in function comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403940 (https://phabricator.wikimedia.org/T175504) [16:36:01] (03CR) 10Jdlrobson: "Thanks Lego for your response :) Glad we've all been able to find some middle ground (given the active work against it) in this discussion" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [16:37:36] I cannot find it again though [16:41:50] (03CR) 10Umherirrender: "I have choosed the both common cases from core and extensions for the tests and adjust the sniff to pass that test." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403940 (https://phabricator.wikimedia.org/T175504) (owner: 10Umherirrender) [16:43:34] oh good. [16:44:02] there was a ticket for the scap thing, but it was fixed and now there's a new problem :) [16:45:29] I guess T184628 would be the relevant task [16:45:30] T184628: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628 [16:47:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897053 (10thcipriani) 05Resolved>03Open happening again but maybe with a different cause. Are beta packages being built from the... [16:47:30] 10MediaWiki-Codesniffer, 10Patch-For-Review: Remove FunctionComment.MissingParamComment from default rules - https://phabricator.wikimedia.org/T184650#3897055 (10Umherirrender) Or should the task keep open for further discussion, if the sniff is needed or should be removed at all? Feel free to open. [16:47:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897057 (10thcipriani) a:05thcipriani>03None [16:52:44] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897095 (10thcipriani) For whatever reason beta debs got built from the release branch? https://integration.wikimedia.org/ci/job/phab... [17:12:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897171 (10mmodell) The branch for which packages are built is controlled by the Autoclose setting in the repo - [[ https://phabricat... [17:16:12] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3897181 (10demon) >>! In T119908#3889210, @HappyDog wrote: > What is the timeline on the new UI? Has... [17:18:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897200 (10thcipriani) p:05High>03Normal Changing priority because this is currently "fixed" because I pushed a new commit onto s... [17:18:58] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3890457 (10thcipriani) [17:19:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897204 (10thcipriani) [17:19:49] 10Continuous-Integration-Config, 10VisualEditor: CI npm job for VisualEditor repo fails as can't find the npm-browser-test docker image - https://phabricator.wikimedia.org/T184810#3897210 (10Jdforrester-WMF) [17:20:33] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:41] thcipriani: etc, got a patch up to fix the issue that mediawiki had (the UBN) [17:20:55] not sure what releng want to do with the train though [17:21:21] twentyafterfour: ^^ [17:21:26] greg-g: ^^ [17:22:34] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897233 (10thcipriani) Looking at the `phabricator-jessie-commits` job on T184118#3897095 I had to refresh my memory on what was happening. # Current process As I understand it 1. A commit... [17:22:46] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897234 (10thcipriani) p:05Triage>03Normal [17:22:48] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:04] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:18] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:00] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:12] awww that's nice [17:25:51] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:05] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:12] yay [17:27:28] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:45] addshore: twentyafterfour looks like we're on option 2 of https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#What_happens_next? [17:28:42] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:46] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:29:04] RECOVERY - Puppet errors on deployment-eventlog02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:29] greg-g: it doesn't meet any of those criteria ;) so no more train this week? :) [17:29:48] oh wait, group2, not group1... [17:30:17] sweet, so monday it is! [17:30:22] "If the new wmf.XX version wasn't deployed to group2 (all wikipedias) on Thursday due to blockers" which is true, so we'll try on Monday [17:30:24] yup [17:30:53] In that case lets make sure we swat or deploy the fix today though, otherwise group0 are still just broken? [17:31:09] although, actually, none of group0 have legacy encoding.... so, thats a non issue [17:31:21] in which case, nothing to do till monday :) [17:31:25] ah, sure, but they aren't broken in that none of the wikis have that setting, right? [17:31:38] indeed! [17:31:44] * greg-g remembers things! [17:32:08] turns out the bug is easily reproducible on beta, but no tests or manual testing spotted it [17:32:21] again, on beta, only reproducable on the 5 or si sites that have the setting [17:32:31] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:42] * greg-g nods [17:32:51] right, back to breaking other things [17:32:55] addshore: I saw some tests being added to that task, is it covered now at least? [17:33:06] greg-g: yes :) [17:33:17] sweet, break things the same way only once :) [17:33:25] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:31] I'm going to aim to write a bunch of basic selenium tests for core, i didn't realise there are only 2..... [17:33:35] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:55] addshore: I will buy you 5 drinks of your choice in Barcelona if you do. [17:34:01] hahahahaa [17:34:15] i'll hold you to that [17:34:26] how many are we defining as "a bunch" here? [17:34:33] 30? :P [17:34:40] HAH! we will see [17:34:51] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:05] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:16] 10MediaWiki-Releasing: Cleanup https://www.mediawiki.org/keys/keys.html and related - https://phabricator.wikimedia.org/T181017#3897280 (10Krinkle) [17:36:21] 10MediaWiki-Releasing: keys.html has outdated styling - https://phabricator.wikimedia.org/T181018#3897278 (10Krinkle) 05Open>03Resolved [17:37:08] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:24] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897281 (10thcipriani) For reference, this is why this is happening: * The generic job `phabricator-jessie-commit` is triggered for... [17:38:40] RECOVERY - Puppet errors on deployment-ores01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:58] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:53] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:59] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:59] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:40] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:10] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:50] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:54] addshore: oh, is that patch fix merged into master? [17:46:10] is it this one? [mediawiki/core@master] RevisionStore, fix loadSlotContent with no $blobFlags [17:46:12] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:39] looks like it [17:46:45] greg-g: yup [17:46:54] i put a cherry pick up for .16 already too [17:46:54] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:21] probably would have had it all done last night / just after the train slot, but i had to sleep ;) [17:48:31] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:49:52] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:23] <3. [17:50:30] :D [17:51:08] I didn't go back and check how many edits were made on the 2 wiktionaries between the deploy and rollback (24 hours) but I don't imagine it was too many. [17:51:43] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897320 (10thcipriani) ## Things I like * being able to run master in beta * cutting production releases from a branch that isn't as in-flux as the master branch ## Things I'd like to change... [17:51:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3897321 (10greg) >>! In T184749#3897310, @greg wrote: > Status update: The fix is merged into master (with a backport to wmf.16) and thus on the Beta Cluster for... [17:54:06] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897327 (10mmodell) We still have to address the problem with deploying scap to beta: commits to master are disruptive to other developers who use beta to test their deployments. They need a st... [18:03:46] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3890457 (10ArielGlenn) And that's why my puppetmaster manifests go to a local puppetmaster instead of the main deployment-prep one. Having everyone spin up their own puppetmaster isn't the ans... [18:03:53] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897376 (10mmodell) 1. Removal of the {icon folder} /debian in master. ** I think it makes the most sense to have the ci job merge **{icon code-fork} master** {icon arrow-right} **{icon code-f... [18:05:23] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897377 (10mmodell) @ArielGlenn: indeed, I'm open to clever suggestions ;) [18:08:12] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897387 (10mmodell) Can we just use docker-compose to define a whole scap micro-cluster and then run that in CI? [18:08:14] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897388 (10demon) >>! In T184628#3897327, @mmodell wrote: > We still have to address the problem with deploying scap to beta: commits to master are disruptive to other developers who use beta t... [18:09:00] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897390 (10mmodell) @demon: there are a whole class of bugs that we can't test until code hits master, currently.... [18:11:25] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897395 (10demon) We don't disagree: but my point is that if we're merging stuff to master that is risky, we (the one doing the merge) should be prepared to either fix things or roll back quick... [18:11:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3897396 (10mmodell) Aside: It would be really swell if puppet wouldn't freak out and break in this situation. [18:13:14] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897401 (10mmodell) @demon: That complicates development (and raises my stress level) while failing to fully address the issues with CI not actually catching bugs and release branch versions co... [18:15:33] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897412 (10mmodell) Also, the "master must always be runnable" idea sounds great but it transfers responsibility from CI to a manual process that developers are responsible for doing properly a... [18:15:54] (03CR) 10Legoktm: [C: 032] Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [18:18:46] (03Merged) 10jenkins-bot: Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [18:19:33] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897442 (10demon) I think developers **should** be responsible for the code they merge and not rely on CI to catch everything. This doesn't preclude good CI, but I don't think "be ready to reve... [18:21:43] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897464 (10mmodell) That's entirely beside the point, however. The problem is currently the route to finding out if the code is broken is through commits to master which also break other people... [18:22:24] (03CR) 10jenkins-bot: Remove direction from @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/402871 (owner: 10Umherirrender) [18:24:22] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3897469 (10mmodell) Maintaining a local VM for testing is reasonable, however, it's a non-trivial amount of work and IMO it would be more efficient to have a shared testing environment that we... [18:29:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3897497 (10mmodell) [18:29:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10mmodell) [18:44:03] wtf... [18:44:50] when I click on "AdManager.php" in the files section on https://gerrit.wikimedia.org/r/#/c/392021/8 ... I get "Code Review - Error\nServer Unavailable\n0" [18:45:11] is it just me? [18:48:31] same [18:54:27] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:12:42] Project mwext-phpunit-coverage-publish build #99: 04FAILURE in 1 hr 52 min: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/99/ [20:40:13] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:03:42] (03PS2) 10Chad: releases: make patch_dir configurable (and only configurable) [tools/release] - 10https://gerrit.wikimedia.org/r/403563 [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:08:57] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:16:20] (03CR) 10Umherirrender: "Superseeded by I2778bf0b12156bc05694495f46165d3ac75f4db1" [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42) [21:18:40] (03CR) 10Chad: [C: 032] releases: make patch_dir configurable (and only configurable) [tools/release] - 10https://gerrit.wikimedia.org/r/403563 (owner: 10Chad) [21:19:24] (03Merged) 10jenkins-bot: releases: make patch_dir configurable (and only configurable) [tools/release] - 10https://gerrit.wikimedia.org/r/403563 (owner: 10Chad) [21:20:06] (03PS2) 10Chad: releases: Unify root_dir/build_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403564 [21:20:06] (03PS2) 10Chad: releases: use os.path.join() in a few places [tools/release] - 10https://gerrit.wikimedia.org/r/403565 [21:20:08] (03PS2) 10Chad: release: move get_git() to function, fix up export_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403566 [21:20:13] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:22:34] (03CR) 10Chad: [C: 032] releases: Unify root_dir/build_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403564 (owner: 10Chad) [21:23:17] (03Merged) 10jenkins-bot: releases: Unify root_dir/build_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403564 (owner: 10Chad) [21:25:17] (03CR) 10Chad: [C: 032] releases: use os.path.join() in a few places [tools/release] - 10https://gerrit.wikimedia.org/r/403565 (owner: 10Chad) [21:25:54] (03Merged) 10jenkins-bot: releases: use os.path.join() in a few places [tools/release] - 10https://gerrit.wikimedia.org/r/403565 (owner: 10Chad) [21:26:11] (03CR) 10Chad: [C: 032] release: move get_git() to function, fix up export_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403566 (owner: 10Chad) [21:29:04] Yippee, build fixed! [21:29:06] Project mwext-phpunit-coverage-publish build #100: 09FIXED in 6 min 52 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/100/ [21:30:07] (03Merged) 10jenkins-bot: release: move get_git() to function, fix up export_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403566 (owner: 10Chad) [21:31:00] (03PS1) 10Chad: makerelease: Fix up return on all code paths, pylint complains [tools/release] - 10https://gerrit.wikimedia.org/r/404044 [21:31:02] (03CR) 10Chad: [C: 032] makerelease: Fix up return on all code paths, pylint complains [tools/release] - 10https://gerrit.wikimedia.org/r/404044 (owner: 10Chad) [21:31:45] (03Merged) 10jenkins-bot: makerelease: Fix up return on all code paths, pylint complains [tools/release] - 10https://gerrit.wikimedia.org/r/404044 (owner: 10Chad) [21:38:02] hashar: https://gerrit.wikimedia.org/r/#/c/369663/2/mediawiki/conf.d/00_dev_settings.php [21:39:16] Krinkle: ho yeah I had that comment has an uncommited hack locally [21:39:22] so I probably blindly pushed it [21:39:43] I think the idea was to warn about logging to a file matching mw-*.log [21:39:55] since probably Jenkins or something else has a publisher to assert those are all empty [21:40:06] but maybe that has never been implemented [21:40:12] 10MediaWiki-Codesniffer, 10Patch-For-Review: Figure out how to properly document variadic arguments - https://phabricator.wikimedia.org/T175504#3897851 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [21:40:17] if it is misleading, please drop the comment [21:48:02] hashar: Yeah, it's currently inaccurate. We don't use mw-* as a pattern, the assert matches individual file names. And that is important, because we have mw-debug-* which would match, which is not supposed to be empty. [21:48:24] So making it true, would break things :) [21:48:52] (03PS1) 10Hashar: Revert "Add a note about mw debug file name" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/404047 [21:48:53] :;D [21:48:57] (03PS2) 10Hashar: Revert "Add a note about mw debug file name" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/404047 [21:48:58] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [21:57:48] Krinkle: I guess you can +2 the revert patch https://gerrit.wikimedia.org/r/#/c/404047/ :D [21:58:18] (03CR) 10Krinkle: [C: 032] "Thanks :)" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/404047 (owner: 10Hashar) [21:58:25] ;D [21:58:38] I am out!!! have a good week-end! [21:59:09] (03Merged) 10jenkins-bot: Revert "Add a note about mw debug file name" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/404047 (owner: 10Hashar) [21:59:38] 10Continuous-Integration-Config, 10VisualEditor: CI npm job for VisualEditor repo fails as can't find the npm-browser-test docker image - https://phabricator.wikimedia.org/T184810#3897210 (10hashar) ARHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH [22:00:47] (03PS1) 10Chad: Remove pointless have_i18n variable, add some missing docstrings [tools/release] - 10https://gerrit.wikimedia.org/r/404050 [22:00:49] (03CR) 10Chad: [C: 032] Remove pointless have_i18n variable, add some missing docstrings [tools/release] - 10https://gerrit.wikimedia.org/r/404050 (owner: 10Chad) [22:01:17] (03Merged) 10jenkins-bot: Remove pointless have_i18n variable, add some missing docstrings [tools/release] - 10https://gerrit.wikimedia.org/r/404050 (owner: 10Chad) [22:01:37] (03PS2) 10Chad: make-wmf-branch: Remove patching support [tools/release] - 10https://gerrit.wikimedia.org/r/403154 [22:01:40] (03CR) 10Chad: [C: 032] make-wmf-branch: Remove patching support [tools/release] - 10https://gerrit.wikimedia.org/r/403154 (owner: 10Chad) [22:02:31] (03Merged) 10jenkins-bot: make-wmf-branch: Remove patching support [tools/release] - 10https://gerrit.wikimedia.org/r/403154 (owner: 10Chad) [22:02:35] (03PS1) 10Hashar: Fix npm-browser-test docker tag [integration/config] - 10https://gerrit.wikimedia.org/r/404051 (https://phabricator.wikimedia.org/T177276) [22:02:47] 10Continuous-Integration-Config, 10VisualEditor, 10Patch-For-Review: CI npm job for VisualEditor repo fails as can't find the npm-browser-test docker image - https://phabricator.wikimedia.org/T184810#3897903 (10hashar) The Docker tag should not have a `v` prefix. I did the mistake in 54d031342c21922a5d2a4707... [22:04:24] (03PS1) 10Chad: Remove --list-bundled option, pointless [tools/release] - 10https://gerrit.wikimedia.org/r/404052 [22:04:26] (03CR) 10Chad: [C: 032] Remove --list-bundled option, pointless [tools/release] - 10https://gerrit.wikimedia.org/r/404052 (owner: 10Chad) [22:05:03] (03Merged) 10jenkins-bot: Remove --list-bundled option, pointless [tools/release] - 10https://gerrit.wikimedia.org/r/404052 (owner: 10Chad) [22:05:42] 10Continuous-Integration-Config, 10VisualEditor, 10Patch-For-Review, 10User-Ryasmeen: CI npm job for VisualEditor repo fails as can't find the npm-browser-test docker image - https://phabricator.wikimedia.org/T184810#3897912 (10hashar) 05Open>03Resolved a:03hashar I should better test my deployments.... [22:36:41] 10MediaWiki-Releasing, 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins as submodules of core - https://phabricator.wikimedia.org/T137564#3897969 (10demon) [22:36:50] 10MediaWiki-Releasing, 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins as submodules of core - https://phabricator.wikimedia.org/T137564#2371986 (10demon) 05Open>03Resolved This is done. [22:38:42] 10MediaWiki-Releasing, 10MediaWiki-Vendor: 1.27 tarball: Unnecessary library "ruflin/elastica 2.3.1" requirement - https://phabricator.wikimedia.org/T156637#3897972 (10demon) 05Open>03Resolved a:03demon This will be fixed for all future releases by removing mw/vendor from branches and fixing makerelease.py. [22:39:02] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday): Fix make-deploy-notes to use proper API formats - https://phabricator.wikimedia.org/T140268#3897975 (10demon) 05Open>03declined Nobody cares. [22:46:15] 10MediaWiki-Releasing, 10MW-1.29-release: MW 1.29 missing extensions/.gitignore - https://phabricator.wikimedia.org/T181741#3897985 (10demon) 05Open>03Resolved a:03demon Was fixed in rMWc5c7140017e8d4dc4538f5a909028de4c11a7102 [22:48:52] 10MediaWiki-Releasing: make-release.py should generate gitinfo.json files for extensions/skins so VCS information shows up in Special:Version - https://phabricator.wikimedia.org/T154352#3898011 (10demon) p:05Triage>03Low [22:48:55] 10MediaWiki-Releasing, 10Security-Extensions: Backport security fixes to stable + LTS extension branches - https://phabricator.wikimedia.org/T108734#3898012 (10demon) p:05Triage>03Low [22:48:58] 10MediaWiki-Releasing, 10Release-Engineering-Team (Watching / External), 10Parsoid, 10Service-Architecture, 10Services (watching): Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#3898013 (10demon) p:05Normal>03Low [22:49:42] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Security-Team: Streamline/automate MW tarball security release process - https://phabricator.wikimedia.org/T156445#3898015 (10demon) p:05Triage>03High [22:49:44] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974#3898016 (10demon) p:05Normal>03High [22:50:43] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.27-release-notes, 10Patch-For-Review: mediawiki/vendor is bloated for tarball releases - https://phabricator.wikimedia.org/T172927#3898017 (10demon) 05Open>03Resolved a:03demon Per T156637 : This will be fixed for all future releases by removing mw/ve... [22:53:55] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: Make make-release patch structure like we do for production security patches et al - https://phabricator.wikimedia.org/T180526#3898044 (10demon) 05Open>03Resolved a:03demon Bam: 4dfc9b648c37237ef582ade197392984c9e48cf7 [22:55:30] 10MediaWiki-Releasing: Cleanup https://www.mediawiki.org/keys/keys.html and related - https://phabricator.wikimedia.org/T181017#3776610 (10demon) What else do we want to do here? The subtask isn't really related imho [22:57:07] 10MediaWiki-Releasing, 10MediaWiki-Stakeholders-Group, 10MediaWiki-extensions-General, 10Developer-Wishlist (2017): Improve LTS support of extensions - https://phabricator.wikimedia.org/T156640#3898059 (10demon) [22:57:10] 10MediaWiki-Releasing, 10Security-Extensions: Backport security fixes to stable + LTS extension branches - https://phabricator.wikimedia.org/T108734#3898062 (10demon) [22:57:38] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: MediaWiki release patch files should be based off of the previous tarball - https://phabricator.wikimedia.org/T181116#3898065 (10demon) p:05Triage>03Normal [22:57:41] 10MediaWiki-Releasing: Cleanup https://www.mediawiki.org/keys/keys.html and related - https://phabricator.wikimedia.org/T181017#3898066 (10demon) p:05Triage>03Normal [22:57:45] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: make-release vendor diffing broken for REL1_27/REL1_28 - https://phabricator.wikimedia.org/T180532#3898067 (10demon) p:05Triage>03Normal [22:57:47] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: Make make-release not need to clone MW for every branch - https://phabricator.wikimedia.org/T180522#3898068 (10demon) p:05Triage>03Normal [22:57:50] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday): Removal of EOL release branches broke SemanticMediaWiki CI - https://phabricator.wikimedia.org/T179754#3898069 (10demon) p:05Triage>03Normal [22:58:12] 10MediaWiki-Releasing, 10MediaWiki-Installer: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#3898070 (10demon) p:05Triage>03Normal [22:58:15] 10MediaWiki-Releasing, 10MW-1.27-release, 10MW-1.27-release-notes, 10NewPHP: Make MediaWiki 1.27 (LTS) compatible with PHP 7.1 - https://phabricator.wikimedia.org/T174262#3898071 (10demon) p:05Triage>03Normal [22:58:18] 10MediaWiki-Releasing, 10Analytics: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772#3898072 (10demon) p:05Triage>03Normal [23:12:29] 10Continuous-Integration-Config, 10VisualEditor, 10Patch-For-Review, 10User-Ryasmeen: CI npm job for VisualEditor repo fails as can't find the npm-browser-test docker image - https://phabricator.wikimedia.org/T184810#3898101 (10Jdforrester-WMF) Thank you! :-) [23:26:15] 10Release-Engineering-Team (Kanban): Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3898130 (10demon) So Tuesdays we have an hour break in between Service deploys and the train. Wednesdays and Thursdays there's also an hour break--but with SWAT... [23:37:00] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974#3898156 (10Mglaser) Just an aside, as it does not test the patches: there is a rudimentary test of the tarball, checking proper extraction and correct version information. It can be found in mediawiki/t... [23:38:57] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974#3898163 (10Mglaser) Oh, and it does test the installer on a basic level [23:57:46] no_justification hi, would you like me to backport a change that allows you to disable private changes through config? [23:58:01] It's in master, but if you want i can backport into 2.15. [23:58:11] I'm not even thinking about 2.15 [23:58:18] no_justification i mean for later :). [23:58:42] It's far enough later that I have zero opinions [23:58:43] :p [23:58:49] ok [23:59:23] 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3898218 (10greg)