[00:37:50] 10MediaWiki-Codesniffer: MediaWiki.ExtraCharacters.CharacterBeforePHPOpeningTag.Found broken on hhvm-fatal-error.php - https://phabricator.wikimedia.org/T171234#3458930 (10Legoktm) 05Open>03declined On master the error comes from Generic.PHP.CharacterBeforePHPOpeningTag.Found, but the thing is, this is exact... [00:56:51] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins/vendor as submodules of core - https://phabricator.wikimedia.org/T137564#3458947 (10Joergi123) [00:56:53] 10Release-Engineering-Team, 10MW-1.29-release, 10Patch-For-Review, 10Release: MediaWiki 1.29 tarball comes with the wrong extensions - and misses some - https://phabricator.wikimedia.org/T171197#3458946 (10Joergi123) [01:18:08] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn doesn't like static.php's wmfStaticRespond - https://phabricator.wikimedia.org/T171254#3459022 (10Reedy) [01:31:22] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn doesn't like static.php's wmfStaticRespond - https://phabricator.wikimedia.org/T171254#3459022 (10Legoktm) Dupe of {T170765}? [01:32:17] Looks to be [01:32:47] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn doesn't like static.php's wmfStaticRespond - https://phabricator.wikimedia.org/T171254#3459057 (10Reedy) [01:32:50] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn should ignore return in closure - https://phabricator.wikimedia.org/T170765#3459060 (10Reedy) [01:50:35] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn should ignore return in closure - https://phabricator.wikimedia.org/T170765#3441957 (10Krinkle) @Reedy I believe the sniff is meant to enforce `@return` if the author already decided to document the method at least to some degree. This i... [01:53:35] (03PS2) 10Krinkle: Revert "Add set-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/351701 (https://phabricator.wikimedia.org/T157750) (owner: 10Hashar) [01:53:38] (03PS3) 10Krinkle: Revert "Add set-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/351701 (https://phabricator.wikimedia.org/T157750) (owner: 10Hashar) [01:55:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: mw-tools-codesniffer-mwcore-testrun test fails with php is not hhvm - https://phabricator.wikimedia.org/T157750#3015241 (10Krinkle) @hashar It seems... [01:55:13] (03CR) 10Krinkle: [C: 031] Revert "Add set-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/351701 (https://phabricator.wikimedia.org/T157750) (owner: 10Hashar) [01:56:30] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn should ignore return in closure - https://phabricator.wikimedia.org/T170765#3459116 (10Legoktm) Regardless of whether there's already a documentation comment or not, the sniff will check that if there is a `return` token, there should al... [02:01:59] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.MissingReturn should ignore return in closure - https://phabricator.wikimedia.org/T170765#3459120 (10Reedy) ``` FILE: /home/reedy/git/operations/mediawiki-config/w/static.php ---------------------------------------------------------------------- FOUND... [08:02:10] hashar: hey, if I want to move https://github.com/wmde/Wikiba.se to gerrit, where is should live? [08:02:28] it's not an extension, so obviously not mediawiki/extensions/wikiba.se [08:02:41] not in tools or labs [08:04:10] phab card: https://phabricator.wikimedia.org/T171160 [08:55:53] Amir1: ask wikidata folks ? Whatever you want really [08:56:43] Amir1: there is already a /wikibase project, so maybe // [08:56:52] Amir1: there is already a /wikibase project, so maybe /wikibase/www or whatever :] [09:04:14] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:19:06] Amir1: also, potentially we could have a jenkins job to generate the site for us automagically when a change is merged [09:19:23] and upload the resulting build to a git repo like wikibase/wwwbuild or something like that [09:39:12] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:39:17] hashar: that'd be great [09:39:38] hashar: what do you think of wikibase/wikiba.se ? [09:48:55] Amir1: sure :) [09:49:25] great, let me bring this up in the daily and then let's make the repo [09:49:33] (it happens in ten minutes) [09:49:42] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Phabricator task graphs need a limit on how big they draw - https://phabricator.wikimedia.org/T140326#3459713 (10Rical) [09:51:10] Amir1: sounds good [09:51:25] Amir1: and I guess one can then find out a way to have the site generated via a script run by jenkins [09:51:27] pushed to a git repo [09:51:31] and we could deploy from that [09:51:37] (that is merely a suggestion) [09:52:12] hashar: we can't deploy it unless it's in WMF infra (right now people do it manually using ftp, ewww) [09:52:39] I'm working on moving it to misc cluster and once it's there, yeah we should do it [09:52:48] neat :] [09:53:05] and I guess then someone from releng can assist in using scap to deploy it [09:55:31] yeah, I mentioned scap in the email that I sent to Ops-l yesterday [10:10:41] hashar: wikiba.se it is :) [10:20:49] 10Release-Engineering-Team (Watching / External), 10Operations, 10Puppet: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066#3459834 (10fgiunchedi) p:05Triage>03Normal [10:39:56] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631#3272220 (10fgiunchedi) >>! In T165631#3272935, @demon wrote: > We can't move them behind LVS. Unlike Phabricator, which uses a separate hostname for the SSH service, Gerrit exp... [10:40:04] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631#3459874 (10fgiunchedi) p:05Triage>03Normal [10:46:47] !log Gerrit: created wikibase/wikibase.se repo for Amir1 / T171160 [10:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:46:51] T171160: Move wikiba.se repository from github to gerrit - https://phabricator.wikimedia.org/T171160 [10:47:00] Thanks! [10:47:16] I need push rights for a moment to push the whole history [10:47:25] Amir1: it inherits from the wikibase/ repo [10:47:27] hashar you also have to create the diffusion repo :) [10:47:29] its rights are https://gerrit.wikimedia.org/r/#/admin/projects/wikibase,access [10:47:40] bsically its owned by people in the wikidata and wikibase groups [10:47:51] That's great [10:47:53] paladox: isn't that auto generated? [10:47:57] nope [10:48:02] it's all manual [10:48:08] bah wikibase group is empty :D [10:48:33] also need to create the github mirror :) [10:48:52] hashar: all are in wikidata [10:49:35] Amir1: want me to copy the github repo to gerrit ? [10:49:38] with all references? [10:49:43] (ie including pull requests etc) [10:50:02] yeah, I think we did it once for PropertySuggester extension if you remember :) [10:50:19] well depends [10:50:25] we can import solely the master branch [10:51:25] that would be enough for now [10:55:49] you could import it into diffusion then mirror to gerrit then switch it around so gerrit mirrors to diffusion :) [10:56:39] nope, I don't think wikidata team can work with differential (yet) [10:56:57] let's stick with gerrit until extensions and medawiki moved to differential [10:57:02] Amir1: https://phabricator.wikimedia.org/T171160#3459967 :D [10:57:35] hashar: fantastic, thanks! [10:58:07] then I guess add some CI glue for composer :D [10:58:28] will follow up later. I have to bring some books back to the library [10:59:52] Amir1: bah composer ends up cloning pretty much everything :( [10:59:59] "minimum-stability": "dev" [10:59:59] is evil! [11:00:33] hashar: yeah, I added that because sculpin wasn't working without it [11:00:35] :D [11:01:31] requires dflydev/embedded-composer ^1.0@dev [11:01:32] pff [11:02:37] Amir1: https://github.com/dflydev/dflydev-embedded-composer/commit/c9ca20fd3ccfbfb7bfcc3c65c33191f458c8a3a7 :]] [11:03:04] it is all @dev :( [11:03:56] yeah, let's have dev composer because the stable doesn't work or we are so cool we can't work with stable things [11:04:16] seems like adding to composer.json: "dflydev/embedded-composer": "^1.0@dev" [11:04:16] fix it [11:04:33] and maybe that can be pinned to a specific commit (not sure whether that is possible with composer) [11:06:03] Amir1 i mean diffusion [11:06:07] it's the repo browser [11:06:12] it replaced gitblit [11:06:19] differential is different to diffusion [11:15:12] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631#3460065 (10fgiunchedi) Also in a master/slave configuration are the ssh host keys exposed by gerrit the same on both machines? Only slightly related to lvs but it just occurred... [11:26:34] hasharAway: We made a mistake, it should be wikibase/wikiba.se, not wikibase/wikibase.se [11:43:05] Amir1: argh [11:47:19] Amir1: should be good now [11:50:13] hashar: Thanks. Some patches are coming for review :D [11:53:03] hashar: https://gerrit.wikimedia.org/r/366841 [11:55:28] hashar: https://gerrit.wikimedia.org/r/366843 :D [11:57:27] It got +2'ed but it doesn't let me submit it [11:57:29] strange [12:17:20] hashar: I can't submit this patch, do you know why? [12:17:21] https://gerrit.wikimedia.org/r/#/c/366841/ [12:20:46] Amir1 is there any errors? [12:21:06] paladox: nope, the "submit" is not just there [12:21:13] oh i see [12:21:13] why [12:21:28] integration is the project group owner [12:21:29] https://gerrit.wikimedia.org/r/#/admin/projects/wikibase/wikiba.se,access [12:21:44] and it inherits [12:21:46] from wikibase [12:21:49] with group being [12:21:50] https://gerrit.wikimedia.org/r/#/admin/groups/32,members [12:21:53] which your not in [12:22:43] Amir1 ^^ [12:23:06] paladox: it should have wikidata group as owner too [12:23:17] also I can +2 (code review and verified) [12:23:19] oh [12:23:20] i see [12:24:26] Amir1 https://gerrit.wikimedia.org/r/#/c/366844/ [12:25:34] well, I have no idea what the patch is and don't have the rights to merge it :/ hashar maybe :) [12:28:59] ok [12:46:03] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10HHVM, and 4 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#3460233 (10hoo) I just wanted to note that this still is a problem and apparen... [13:05:13] Amir1: sorry I am back [13:05:31] no worries [13:06:53] ahh [13:07:10] so the wikibase group is owned by "wikidata" group [13:07:17] not sure whether that grant submit right [13:08:04] Amir1: can you try again ? [13:08:50] sure [13:09:09] hashar, https://gerrit-review.googlesource.com/#/c/114891/ :) [13:09:20] * paladox is introducing translations in polygerrit. [13:09:47] hashar: I can submit now [13:09:49] thanks [13:09:55] hashar: does it have a github mirror? [13:11:57] Amir1: I have granted submit right to the wikidata group :D [13:12:36] hashar: strange, I remember we submitted several times in wikibase extension when jenkins was very broken [13:13:12] (03CR) 10Hashar: Run WebdriverIO tests in CI for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:14:19] Amir1: that is under mediawiki/extensions and thus it is a different set of access lists :] [13:14:27] hashar: also I have this patch too which waits for your input :) https://gerrit.wikimedia.org/r/#/c/366843/ [13:14:52] and then we should start basic jenkins so it can at least does the merge [13:15:11] Amir1: I guess just update the commit message of that patch? [13:15:17] I am looking at the CI part [13:15:38] it is going to be installed on a Ganeti instance right? I guess jessie + php 5.6 [13:16:19] hashar: yeah probably [13:16:35] faidon said it will be in krypton or brome [13:16:40] *bromine [13:24:39] (03PS5) 10Zfilipin: Run WebdriverIO tests in CI for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) [13:25:46] (03CR) 10Zfilipin: Run WebdriverIO tests in CI for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:32:10] (03CR) 10Zfilipin: "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:32:25] (03CR) 10Zfilipin: "Testing:" [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:33:39] (03CR) 10Zfilipin: "Reverting because of:" [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:35:05] (03PS1) 10Hashar: wikiba.se [integration/config] - 10https://gerrit.wikimedia.org/r/366847 (https://phabricator.wikimedia.org/T171160) [13:35:11] Amir1: and CI on its way :] [13:35:29] sorry I am multitasking with other errands [13:35:59] hashar: no worries, Thank you! [13:36:53] (03CR) 10Hashar: [C: 031] "Lets try it!" [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:37:36] (03CR) 10Hashar: [C: 032] wikiba.se [integration/config] - 10https://gerrit.wikimedia.org/r/366847 (https://phabricator.wikimedia.org/T171160) (owner: 10Hashar) [13:37:56] hashar: Also, I think github mirror would be great so we can travis on it too (just to make some people haapy) [13:38:08] Amir1: yeah I think we bulk create them from time to time [13:38:23] okay, just tell me once it's there [13:38:36] (03Merged) 10jenkins-bot: wikiba.se [integration/config] - 10https://gerrit.wikimedia.org/r/366847 (https://phabricator.wikimedia.org/T171160) (owner: 10Hashar) [13:40:27] [2017-07-21 13:40:09,472] [3e3a4bf0] Created remote repository: gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/wikibase/wikiba.se.git [13:40:27] [2017-07-21 13:40:09,472] [3e3a4bf0] Missing repository created; retry replication to git@github.com:wikimedia/wikibase-wikiba.se [13:40:30] Amir1: maybe it auto create them [13:41:14] https://github.com/wikimedia?utf8=✓&q=wikiba.se&type=&language= [13:41:28] ah fuck I messed up my ci patch [13:42:56] (03PS1) 10Hashar: Fix wikiba.se repo name in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/366848 (https://phabricator.wikimedia.org/T171160) [13:43:20] (03CR) 10Hashar: [C: 032] Fix wikiba.se repo name in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/366848 (https://phabricator.wikimedia.org/T171160) (owner: 10Hashar) [13:46:31] 10Release-Engineering-Team, 10MW-1.29-release, 10Patch-For-Review, 10Release: MediaWiki 1.29 tarball comes with the wrong extensions - and misses some - https://phabricator.wikimedia.org/T171197#3460337 (10MacFan4000) we must've user the make-wok-branch-climate branch by mistake, which seems very likely co... [13:47:18] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #466: 04FAILURE in 3 min 17 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/466/ [13:49:42] (03Merged) 10jenkins-bot: Fix wikiba.se repo name in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/366848 (https://phabricator.wikimedia.org/T171160) (owner: 10Hashar) [13:50:28] (03PS6) 10Zfilipin: Run WebdriverIO tests in CI for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) [13:51:16] Amir1: some job is running composer for the repo https://integration.wikimedia.org/ci/job/wikiba.se-composer-php56-jessie/1/console [13:51:32] nice [13:53:03] hmm, test is not defined [13:53:12] I need to fix that in another patch [13:53:39] Amir1: I am going to force merge that one [13:53:40] probably add phpcs [13:56:02] !log Created github mirror repo https://github.com/wikimedia/wikibase-wikiba.se T171160 [13:56:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:56:06] T171160: Move wikiba.se repository from github to gerrit - https://phabricator.wikimedia.org/T171160 [13:56:16] (03CR) 10Zfilipin: "Core is green:" [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:57:26] (03CR) 10Hashar: [C: 032] Run WebdriverIO tests in CI for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [13:58:40] Awesome [13:59:14] Amir1: so I guess you can add a "test" script with the usual stuff [13:59:28] "jakub-onderka/php-parallel-lint": "0.9.2", [13:59:28] "jakub-onderka/php-console-highlighter": "0.3.2", [13:59:30] hashar: I'm on it but strangely it can't connect to packagist [13:59:32] and a dummy [13:59:35] to do compoer update [13:59:42] "test": [ "parallel-lint . --exclude vendor" ], [13:59:57] and a follow up to add phpcs [14:00:11] I am not sure what is the best to deploy the site [14:00:16] maybe it is as easy as just doing git pull [14:00:20] since it is static [14:00:25] (03Merged) 10jenkins-bot: Run WebdriverIO tests in CI for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/366531 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:00:29] or we can go with scap, so one deploy it just like the other apps [14:01:22] "The "https://packagist.org/packages.json" file could not be downloaded: failed to open stream: Connection refused" [14:03:26] bah [14:03:49] I had a proxy set in mine [14:03:52] that was the reason [14:03:57] let me fix it [14:04:37] yes [14:04:40] it works now [14:05:51] hashar: also we need to add "phpunit" to tests [14:05:57] not sure one job or two jobs [14:06:18] (fortunately, phpunit is already there) [14:07:40] all in the same job :D [14:07:47] nice :D [14:10:30] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.9)), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3460442 (10zeljkofilipin) 05Open>03Resol... [14:10:32] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3460445 (10zeljkofilipin) [14:12:44] !log added novaadmin to deployment-prep as a regular user. That lets MediaWiki OpenStack API list the instances T171280 [14:12:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:12:48] T171280: wikitech api list=novainstances not returning list of instances - https://phabricator.wikimedia.org/T171280 [14:13:10] godog: list=novainstances works now for deployment-prep. I have added novaadmin as a member of the project [14:13:20] godog: i am not sure what kind of mess it can do though [14:14:28] hashar: I don't know either :( [14:14:57] godog: andrew would know maybe [14:15:05] then it is already a member of other labs project so it is probably sane [14:15:10] yeah I asked [14:15:16] maybe novaadmin got removed from some project because that clutters the list of members [14:18:11] so that looks more or less fixed [14:24:31] hashar: https://gerrit.wikimedia.org/r/366854 [14:27:19] Amir1: phpunit: not found :D [14:27:25] yup [14:27:27] :P [14:27:53] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10JavaScript, 10User-zeljkofilipin: Update documentation for mediawiki-core-qunit-selenium-jessie Jenkins job - https://phabricator.wikimedia.org/T164002#3460509 (10zeljkofilipin) a:03zeljkofilipin [14:29:57] hashar: it passes now [14:31:20] Amir1: +1 will let someone else to +2 it [14:31:33] Sure [14:31:34] Thanks! [14:44:16] (03Abandoned) 10Hashar: Revert "Add set-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/351701 (https://phabricator.wikimedia.org/T157750) (owner: 10Hashar) [14:44:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: mw-tools-codesniffer-mwcore-testrun test fails with php is not hhvm - https://phabricator.wikimedia.org/T157750#3460602 (10hashar) Definitely yes. Lo... [14:55:10] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.9)), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3460640 (10zeljkofilipin) [14:55:12] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3460641 (10zeljkofilipin) [14:55:13] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10JavaScript, 10User-zeljkofilipin: Update documentation for mediawiki-core-qunit-selenium-jessie Jenkins job - https://phabricator.wikimedia.org/T164002#3460638 (10zeljkofilipin) 05Open>03Resolved https://www.mediawiki.org/w/index.ph... [14:55:47] !log Jenkins: upgraded Android Emulator plugin with https://gerrit.wikimedia.org/r/#/c/366253/ && https://gerrit.wikimedia.org/r/#/c/366484/ - T150623 [14:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:55:53] T150623: Upgrade CI emulator to API 25 - https://phabricator.wikimedia.org/T150623 [15:04:00] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10Patch-For-Review, 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3460664 (10zeljkofilipin) 05stalled>03Open a:05Jdlrobson>03zeljkofilipin [15:04:18] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10Patch-For-Review, 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3460668 (10zeljkofilipin) I will try to refactor the test so they use page object pattern. [15:28:05] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631#3460756 (10demon) I was under the impression we couldn't do port-based LVS to the same domain. But I'm gladly willing to be wrong ☺️ And yes, same host key for the ssh daemon. [15:37:24] (03CR) 10Hashar: "Deployed on Jenkins and https://integration.wikimedia.org/ci/job/apps-android-wikipedia-periodic-test/ seems to work fine." [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [15:49:00] (03CR) 10Hashar: "In EmulatorConfig.java:" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [16:02:58] hashar: what did you do to add novaadmin back? I wanted to do the same for tools since prometheus there is still broken cc andrewbogott [16:03:15] godog: I want to understand what happened before I clean up the issue [16:04:44] fair enough, any leads so far? [16:05:46] godog: yes, maybe [16:06:13] Bryan removed rights for a different service user at that time. I'm assuming some kind of uid overlap that resulted in rights being purged for novaadmin instead [16:06:20] but I haven't found the actual bug yet [16:08:12] ack! thanks for the update [16:13:42] godog: I have added it via the Wikitech web interface [16:14:16] godog: https://wikitech.wikimedia.org/wiki/Special:NovaProject Projects: [deployment-prep] then link [Add member] -> [novaadmin] [16:14:46] most probably the user got removed by project admins [16:19:19] godog: does it cause any serious problems if it stays broken for another few hours? [16:19:47] hasharDinner: it was removed from 50 projects all at once! https://phabricator.wikimedia.org/T171313 [16:20:01] the good news is I'm pretty sure it was a bug and not a security breach [16:48:21] andrewbogott: ah nice finding :) [17:17:54] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10Patch-For-Review, 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3461233 (10Jdlrobson) Note merging this will be blocked on T170880 but it looks like the Sele... [17:21:01] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10Patch-For-Review, 10User-zeljkofilipin: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3461256 (10Jdlrobson) (To be clear I'm fine with using it here now the code has been written,... [17:21:49] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.9)), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3461258 (10Jdlrobson) Good job! :) [17:23:39] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3461260 (10Dzahn) https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=16&hoststatustypes=3&servicepr... [17:28:03] 10Release-Engineering-Team (Kanban), 10Release: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3461285 (10greg) a:03mmodell [17:28:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3461287 (10greg) a:03mmodell [17:29:43] 10Release-Engineering-Team (Kanban), 10Release: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3461290 (10greg) a:03thcipriani [17:30:09] 10Release-Engineering-Team (Kanban), 10Release: 1.30.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T170633#3461293 (10greg) a:03thcipriani [18:15:38] Project beta-scap-eqiad build #165025: 04FAILURE in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/165025/ [18:17:20] 10MediaWiki-Codesniffer: Add Sniff to forbid PHP 7+ type hints - https://phabricator.wikimedia.org/T171335#3461440 (10Legoktm) [18:24:16] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3461462 (10RobH) So this had an issue when the new role assigned an IP to it that was in use in iridium. So we've put it... [18:25:55] Yippee, build fixed! [18:25:55] Project beta-scap-eqiad build #165026: 09FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/165026/ [18:50:34] twentyafterfour: now that there is a new phab1001 and iridium is not going to be reinstalled.. [18:50:48] we can move to phab1001 and make it a jessie upgrade and kill iridium [18:50:57] without having to do codfw and be blocked by that change [18:51:49] fwiw.. re: the LVS stuff [18:52:09] mutante: ok [18:52:15] all we'd have to do is move the role and while doing that manually fix the IP for vcs [18:52:16] was just talking to paladox about that [18:52:19] same here [18:52:22] that made me join :) [18:52:27] :) [18:52:41] so rob just re-installed phab1001 after the disks broke on it [18:52:44] before we even used it [18:52:45] heh [18:52:45] ok, that sounds like a good idea [18:52:58] and then things broke because the role was applied on iridium and phab1001 [18:53:02] trying to add the vcs IP [18:53:14] so like half an hour ago we set it to "role:spare" [18:53:22] but just for that reason and to coordinate the change later [18:53:28] ok [18:54:04] can we have both iridium and phab1001 set up load balanced and then just remove iridium once it's tested working? [18:54:29] that would make for a nice clean switch without an outage, hopefully [18:54:51] then we are back in traffic-team land [18:55:13] i was thinking it's a good thing that we can un-couple the jessie upgrade from any cluster setup [18:55:13] hmm [18:55:30] yeah but how do we actually cut over from iridium to phab1001? [18:55:43] just take over iridium's ip addresses? [18:56:04] just one of them [18:56:10] the secondary one for git-ssh [18:56:16] ok [18:56:18] eh, sorry, vcs [18:56:54] it's already called phab1001-vcs.eqiad.wmnet [18:56:58] while it's on iridium [18:57:04] in anticipation of that reinstall [18:57:06] and the web interface? [18:57:08] that then didnt happen [18:57:15] is it on the same IP? [18:57:20] and caused the conflict with the real/new phab1001 [18:57:27] * twentyafterfour checks which IP apache is using [18:57:49] hmm, tcp6 0 0 :::80 [18:58:32] 10.64.32.150:80 [18:59:25] phab1001-vcs.eqiad.wmnet has address 10.64.32.186 [18:59:36] so web uses a different IP from vcs [18:59:48] if that matters [19:00:08] though I don't see why we couldn't use the vcs IP for web as well [19:01:49] twentyafterfour what if we rsynced the data to phab1001 then shut apache on iridium then switch the ip from going to iridium and instead redirect to phab1001 would that work? It would envolve some downtime but not alot. Reason why i suggest this is because waiting for traffic the load balance thing will probaly be a long while. As it has already been two months since your patch for git-ssh codfw was uploaded and no reply from traffic. [19:01:51] yea, it's possible, but does it matter? [19:01:55] 75 iridium: [19:01:55] 76 backends: [19:01:55] 77 eqiad: 'iridium.eqiad.wmnet' [19:02:04] that's from the caching config [19:02:18] it would just be changing the backend name there [19:02:19] for web [19:02:42] first thing i would rename the director, "iridium" is a bad name for the director itself, which will have multiple servers [19:02:51] should be "phabricator" or whatnot [19:02:55] mutante: I don't know if it matters, just bringing it up in case it does [19:03:35] so far i think it doesn't matter which of the 2 we use for web [19:03:45] I just don't know all the consequences for changing some of this stuff in puppet, and it's not documented very well (that I know of, anyway) [19:03:57] mutante: probably doesn't matter, indeed [19:04:07] oh, here's the thing how to minimize downtime and test on it before switching [19:04:30] we can give it a "phab-new" name, like we did before [19:04:45] the actual switch can be instant and rollback instant as well [19:04:47] ^^ +1 :) [19:04:54] well.. minus running puppet on cache boxes [19:04:59] cool [19:05:01] we did that for gerrit when we moved from old server to the new one last year [19:05:13] another thing i am not sure about [19:05:23] that wmfusercontent domain [19:05:29] do we have to check that too [19:05:31] looks [19:05:46] it should be served the same way as the main phab [19:05:58] it's even using the same virtualhost in apache I think [19:06:09] phabricator does the domain virtualization in php land [19:06:39] so just need to switch it to the same IP at the same time as we switch phabricator.wikimedia.org [19:06:53] er, not sure how to test it ahead of time though [19:07:04] ok, that doesnt need a DNS change [19:07:13] it just points it to misc::cache [19:07:18] ok [19:07:34] so we just change the cache backend for that one and we're done [19:07:48] 219 phab.wmfusercontent.org: [19:07:48] 220 director: 'iridium' [19:07:54] it's also that "iridium" director [19:08:04] so no problem [19:08:07] yes [19:08:25] should we rename the director first, just to eliminate confusion later? [19:08:33] that's what i was thinking too [19:08:38] i'll make a change for that [19:08:41] cool [19:08:44] a rename without any changes [19:09:16] :) [19:10:42] Yippee, build fixed! [19:10:43] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #18: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/18/ [19:16:30] a switch would have to be: [19:16:53] start downtime, remove IP from iridium, add puppet role back on phab1001, run puppet, get IP on new host [19:17:16] switch varnish config, end downtime [19:18:12] also, https://phabricator-new.wikimedia.org/ [19:18:23] we already did the DNS part [19:18:37] to have a temp name for the new one [19:19:06] Yippee, build fixed! [19:19:06] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #18: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/18/ [19:21:00] mutante: cool [19:21:21] so we could point phabricator-new to the new host and test it out... should be pretty simple then [19:21:39] yes, i think we pointed it to phab2001, but now we can just use it for 1001 [19:21:52] https://gerrit.wikimedia.org/r/#/c/366893/ [19:22:38] i am not deleting the old director in same change because something tells me there might be race conditions if i remove it at the same time while other cache::misc servers still use it [19:22:58] puppet has to run on like.. i think 11 servers are cache::misc currently [19:23:14] but i usually run it on all via cumin [19:29:30] (03PS2) 10Hashar: [Test] Stop forcing QEMU 1 use with the "-engine classic" flag [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [19:43:26] 10Release-Engineering-Team, 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461703 (10thcipriani) [19:48:34] 10Release-Engineering-Team, 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461712 (10Dzahn) I'm wondering if it can just live next to the existing pwstore repo that ops uses. It's just a repo and everything is encrypted with GPG and releng probably alr... [19:49:54] 10Release-Engineering-Team, 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461714 (10Dzahn) actually, it may even make sense to do it all in a single pwstore repo and just add a new group for releng. [19:55:52] (03CR) 10Hashar: "PS 1..2 https://gerrit.wikimedia.org/r/#/c/366484/1..2/src/main/java/hudson/plugins/android_emulator/EmulatorConfig.java" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [19:58:14] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3461727 (10RobH) [19:58:36] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3215445 (10RobH) a:05RobH>03Dzahn back to @dzahn for service implementation [19:58:47] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3461743 (10RobH) [20:00:22] (03CR) 10Hashar: "deployed" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [20:08:41] 10Release-Engineering-Team, 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461689 (10greg) Yeah, whatever works, we didn't want to over-presume on Ops' part :) [20:08:59] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461782 (10greg) [20:09:34] (03CR) 10Hashar: "Good news the warning:" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [20:16:38] (03PS1) 10Hashar: android: drop -no-audio [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) [20:17:23] (03PS2) 10Hashar: android: drop -no-audio [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) [20:18:04] (03CR) 10Mholloway: [C: 031] android: drop -no-audio [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) (owner: 10Hashar) [20:18:44] (03CR) 10Hashar: "That fixed the build with android-24 at https://integration.wikimedia.org/ci/job/apps-android-wikipedia-periodic-test/1770/consoleFull" [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) (owner: 10Hashar) [20:19:56] RainbowSprinkles https://gerrit.wikimedia.org/r/#/c/366910/ :) [20:44:46] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461689 (10demon) >>! In T171342#3461714, @Dzahn wrote: > actually, it may even make sense to do it all in a single pwstore repo and just add a new group fo... [20:46:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461844 (10Dzahn) Should be feasible since we already added separate access group for dc-ops before (T158285). [20:53:11] (03CR) 10Hashar: [C: 04-1] "The job uses QEMU2 which no more support -no-audio. Based on https://stackoverflow.com/questions/40272884/running-android-emulator-with-n" [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) (owner: 10Hashar) [20:55:05] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461689 (10MoritzMuehlenhoff) There's really no need for a separate VM, a pwstore is just a git repo with a few megabytes of data :-) I suggest you simply... [21:14:56] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461959 (10demon) >>! In T171342#3461910, @MoritzMuehlenhoff wrote: > There's really no need for a separate VM, a pwstore is just a git repo with a few mega... [21:17:03] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461960 (10Dzahn) The other deployers would still have to decrypt the encrypted files to actually see content though, so it's not really sharing secrets wit... [21:29:03] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3461968 (10demon) >>! In T171342#3461960, @Dzahn wrote: > The other deployers would still have to decrypt the encrypted files to actually see content though... [21:30:37] (03PS3) 10Hashar: android: drop -no-audio [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) [21:31:37] (03CR) 10Hashar: "On QEMU2, one has to use: QEMU_AUDIO_DRV=none" [integration/config] - 10https://gerrit.wikimedia.org/r/366908 (https://phabricator.wikimedia.org/T150623) (owner: 10Hashar) [21:32:40] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:33:22] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:33:56] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:04] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:15] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:54] (03PS1) 10Hashar: Bump Android periodic test to android-25 [integration/config] - 10https://gerrit.wikimedia.org/r/366965 (https://phabricator.wikimedia.org/T150623) [21:35:13] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:17] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:23] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:42] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:48] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:48] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:00] (03CR) 10Hashar: [C: 031] "With QEMU2 -no-audio is no more supported. One has to QEMU_AUDIO_DRV=none" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366484 (owner: 10Mholloway) [21:36:47] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:37:22] (03CR) 10Hashar: [C: 031] "This looks good to me. The periodic job is already hacked to use android-25 / google_apis/x86" [integration/jenkinsci/android-emulator-plugin] - 10https://gerrit.wikimedia.org/r/366253 (https://phabricator.wikimedia.org/T150623) (owner: 10Mholloway) [21:37:23] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:38:11] RainbowSprinkles https://gerrit-review.googlesource.com/#/c/114862/ heh about time they begin supporting inline editing in polygerrit heh :) [21:43:11] Hello, quick question - can scap3 deploy a service from a specific branch of the deploy repo other then master? [21:44:53] (03PS1) 10Chad: Remove signoff from security patch process. [tools/release] - 10https://gerrit.wikimedia.org/r/366969 [21:44:55] (03CR) 10Chad: [C: 032] Remove signoff from security patch process. [tools/release] - 10https://gerrit.wikimedia.org/r/366969 (owner: 10Chad) [21:45:55] (03Merged) 10jenkins-bot: Remove signoff from security patch process. [tools/release] - 10https://gerrit.wikimedia.org/r/366969 (owner: 10Chad) [21:48:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikipedia-Android-App-Backlog, 10Jenkins, and 2 others: Upgrade CI emulator to API 25 - https://phabricator.wikimedia.org/T150623#2791328 (10hashar) **status update** The [[ https://integration.wikimedia.org/ci/job/apps-androi... [21:49:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:50:27] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Wikipedia-Android-App-Backlog, 10Spike, 10Technical-Debt: Investigate how to improve Android CI performance and stability - https://phabricator.wikimedia.org/T158014#3023430 (10hashar) 05declined>03Resolved So that is partly resolved than... [21:58:38] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3462054 (10thcipriani) 05Open>03declined >>! In T171342#3461960, @Dzahn wrote: > The other deployers would still have to decrypt the encrypted files to... [22:06:13] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3462088 (10Dzahn) >>! In T171342#3462054, @thcipriani wrote: > An extra layer of security in the case of storing passwords is prudent and not unreasonable.... [22:07:50] 10Release-Engineering-Team (Watching / External), 10Operations, 10vm-requests: VM request for RelEng pwstore - https://phabricator.wikimedia.org/T171342#3462091 (10Dzahn) sorry, you said they are still encrypted. so still using pwstore but with phab repo. gotcha then.! [22:15:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Zuul: recheck is ignored if there are also inline comments - https://phabricator.wikimedia.org/T171352#3462120 (10hashar) p:05Triage>03Normal The detection is based on a regular expression in integration/config zuul/layout.yaml:... [22:24:30] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:50:57] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins/vendor as submodules of core - https://phabricator.wikimedia.org/T137564#3462181 (10MacFan4000) Created patchsets for REL1_27 and REL1_28 [23:09:15] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins/vendor as submodules of core - https://phabricator.wikimedia.org/T137564#3462222 (10demon) I don't want to backport this behavior. It's inconsistent, and the tags will b...