[00:01:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:43:17] Hmm, something weird is happening with makedeploynotes.py [00:43:34] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/527624 is not listed at https://www.mediawiki.org/wiki/MediaWiki_1.34/wmf.21#Echo , and it's also not listed when I run the script locally [00:44:21] The old bash script (make-deploy-notes) does list it though [00:57:57] Filed as T231956 [00:57:58] T231956: makedeploynotes.py omits some changes for no apparent reason - https://phabricator.wikimedia.org/T231956 [01:01:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [01:16:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:49:16] (03CR) 10Krinkle: "Right, so if the purpose isn't to do a full gate-like check. Presumably all we need is the ones that are php-related and not already in 't" [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester) [03:42:23] 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10Dzahn) Thank you @sbassett [04:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:01:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:10:00] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909), 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10mmodell) [05:10:05] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) [05:10:08] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability: Improve error reporting / integration between Kibana and Phabricator - https://phabricator.wikimedia.org/T185155 (10mmodell) [05:10:10] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201909), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Create an interface for the local-charts ecosystem - https://phabricator.wikimedia.org/T224939 (10mmodell) [05:11:31] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) @fgiunchedi: This is ready to go whenever you have some time to look it over. [05:14:28] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10User-MModell: Test out the Phabricator 'ferrit' search engine. - https://phabricator.wikimedia.org/T230787 (10mmodell) p:05Triage→03Normal [05:15:26] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) [05:15:44] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909), 10User-MModell: Test out the Phabricator 'ferrit' search engine. - https://phabricator.wikimedia.org/T230787 (10mmodell) [06:09:02] (03CR) 10Giuseppe Lavagetto: Fix restart_php (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857) (owner: 10Giuseppe Lavagetto) [06:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:28:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10hashar) >>! In T220746#5462601, @RhinosF1 wrote: >>>! In T220746#5462448, @gerri... [06:38:33] 10Gerrit, 10Cleanup, 10Services, 10service-runner: Gerrit repository mediawiki/services/service-runner appears abandoned - https://phabricator.wikimedia.org/T231953 (10hashar) mediawiki/services/service-template-node is similar. Both have been created in February 2015 apparently with an import from Github.... [06:38:49] 10Gerrit, 10Cleanup, 10Services, 10service-runner: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 (10hashar) [06:39:10] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, and 2 others: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 (10hashar) [06:39:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, and 2 others: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 (10hashar) p:05Triage→03L... [06:39:38] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, and 2 others: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 (10mobrovac) Yup, these are... [06:50:46] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:56:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) @thcipriani should we create a new package/release? [07:15:36] (03CR) 10Hashar: "The test and php pipeline should be disjoint and gate-and-submit should be a superset of both. An example:" [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester) [07:51:52] 10Continuous-Integration-Config, 10phan: Phan job should use composer instead of vendor for release branches ( undeclared class \Wikimedia\Equivset\Equivset ) - https://phabricator.wikimedia.org/T231966 (10hashar) [07:53:00] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, and 2 others: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10hashar) Related: {T231966} . The Phan job uses mediawiki/vendor.git which does not have al... [07:53:50] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10AbuseFilter, and 3 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560... [07:54:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10phan: Phan job should use composer instead of vendor for release branches ( undeclared class \Wikimedia\Equivset\Equivset ) - https://phabricator.wikimedia.org/T231966 (10hashar) [08:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [09:08:29] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10Ladsgroup) >>! In T231862#5462556, @hashar wrote: > The change was for Wikibase and it get added again in EXT_DEPENDENCIES. For exa... [09:17:34] 10Phabricator: Some users can't add the fr-tech private task form to their task selection menu. - https://phabricator.wikimedia.org/T229628 (10Aklapper) >>! In T229628#5461442, @DStrine wrote: > I have seen this happen when @Ppena shared her screen with me. Can someone get on a call with her and check it out? @... [09:19:24] (03PS1) 10Hashar: zuul: test templates gates are superset of tests [integration/config] - 10https://gerrit.wikimedia.org/r/534393 [09:21:00] (03CR) 10jerkins-bot: [V: 04-1] zuul: test templates gates are superset of tests [integration/config] - 10https://gerrit.wikimedia.org/r/534393 (owner: 10Hashar) [09:21:58] (03PS2) 10Hashar: zuul: test templates gates are superset of tests [integration/config] - 10https://gerrit.wikimedia.org/r/534393 [09:23:14] (03Abandoned) 10Hashar: Attempt to test Quibble job filters [integration/config] - 10https://gerrit.wikimedia.org/r/500065 (owner: 10Hashar) [09:41:36] (03PS1) 10Hashar: jjb: convert phan jobs to job templates [integration/config] - 10https://gerrit.wikimedia.org/r/534395 (https://phabricator.wikimedia.org/T231966) [10:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [11:20:34] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Maps (Kartotherian): CI service-pipeline-test-and-publish job assumes blubber config has a single production image - https://phabricator.wikimedia.org/T231928 (10Mathew.onipe) Not sure but seems... [11:32:15] (03PS1) 10Hashar: jjb: move mediawiki-i18n-check-docker def up [integration/config] - 10https://gerrit.wikimedia.org/r/534420 [11:44:32] (03PS1) 10Kosta Harlan: [java8-sonar-scanner] Bump sonar-scanner-cli to version 4.0 [integration/config] - 10https://gerrit.wikimedia.org/r/534422 [11:44:34] (03PS1) 10Kosta Harlan: jjb: [*-codehealth]: Use java8-sonar-scanner 0.4.12 [integration/config] - 10https://gerrit.wikimedia.org/r/534423 [11:48:03] 10Continuous-Integration-Config, 10Code-Health-Metrics: SonarScanner: Persist plugin cache between builds - https://phabricator.wikimedia.org/T231983 (10kostajh) [11:49:09] 10Continuous-Integration-Config, 10Code-Health-Metrics: SonarScanner: Persist plugin cache between builds - https://phabricator.wikimedia.org/T231983 (10kostajh) @hashar @thcipriani if you have ideas for how to accomplish this, please let me know. [11:56:20] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10awight) >>! In T231862#5464240, @Ladsgroup wrote: >>>! In T231862#5462556, @hashar wrote: >> The change was for Wikibase and it get... [12:03:03] (03PS1) 10Hashar: zuul: disable phan on extensions release branches [integration/config] - 10https://gerrit.wikimedia.org/r/534426 (https://phabricator.wikimedia.org/T226945) [12:03:42] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, and 2 others: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10hashar) https://gerrit.wikimedia.org/r/#/c/integration/config/+/534426 disables the phan j... [12:06:54] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10Daimona) [12:08:56] (03PS2) 10Hashar: zuul: disable phan on extensions release branches [integration/config] - 10https://gerrit.wikimedia.org/r/534426 (https://phabricator.wikimedia.org/T226945) [12:11:06] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:16:05] (03CR) 10Hashar: [C: 03+2] zuul: disable phan on extensions release branches [integration/config] - 10https://gerrit.wikimedia.org/r/534426 (https://phabricator.wikimedia.org/T226945) (owner: 10Hashar) [12:16:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10phan: Phan job should use composer instead of vendor for release branches ( undeclared class \Wikimedia\Equivset\Equivset ) - https://phabricator.wikimedia.org/T231966... [12:19:50] (03Merged) 10jenkins-bot: zuul: disable phan on extensions release branches [integration/config] - 10https://gerrit.wikimedia.org/r/534426 (https://phabricator.wikimedia.org/T226945) (owner: 10Hashar) [12:20:37] !log Dropped Phan jobs for extensions from release branches T226945 [12:20:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:20:40] T226945: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 [12:40:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10hashar) REL1_33... [12:56:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10hashar) [12:58:35] (03CR) 10Hashar: [C: 03+2] [java8-sonar-scanner] Bump sonar-scanner-cli to version 4.0 [integration/config] - 10https://gerrit.wikimedia.org/r/534422 (owner: 10Kosta Harlan) [13:02:18] (03Merged) 10jenkins-bot: [java8-sonar-scanner] Bump sonar-scanner-cli to version 4.0 [integration/config] - 10https://gerrit.wikimedia.org/r/534422 (owner: 10Kosta Harlan) [13:04:30] (03CR) 10Hashar: "docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.4.12 has been build." [integration/config] - 10https://gerrit.wikimedia.org/r/534423 (owner: 10Kosta Harlan) [13:05:39] (03PS1) 10Hashar: fab: also pull wikimedia-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/534439 [13:18:41] (03CR) 10Zfilipin: [C: 03+1] jjb: [*-codehealth]: Use java8-sonar-scanner 0.4.12 [integration/config] - 10https://gerrit.wikimedia.org/r/534423 (owner: 10Kosta Harlan) [13:47:58] 10Phabricator: Convert task "Issues affecting translatewiki.net" to a tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) [13:49:58] 10Phabricator, 10translatewiki.net: Convert task "Issues affecting translatewiki.net" to a tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) [13:50:28] 10Phabricator, 10translatewiki.net: Convert task "Issues affecting translatewiki.net" to a tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) To be clear, I'm looking for advice what's the best way to track these issues and help to implement that. [14:06:39] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Aklapper) [14:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:20:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10WDoranWMF) @hash... [14:28:13] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Aklapper) > Currently there are hundreds of subtasks of this task. This makes it very hard to browse them and see what's open. Indeed. Workaround: https:/... [14:39:19] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) [14:41:04] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10zeljkofilipin) [14:44:53] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) I would like to keep the work we translatewiki.net folks do separate from the issues we notice in "upstream" code while operating a MediaWiki... [14:52:08] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10zeljkofilipin) a:05zeljkofilipin→03None Status: Initial patch is in gerrit ([[ http... [14:55:49] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10Ruby, 10User-zeljkofilipin: Mark mediawiki_api and mediawiki_selenium Ruby gems as deprecated - https://phabricator.wikimedia.org/T228160 (10zeljkofilipin) [14:56:07] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin, 10good first bug: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [14:56:26] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [14:56:56] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10fgiunchedi) @mmodell ok! I hope to be able to take a look this week [15:05:01] (03PS2) 10Thcipriani: fab: also pull wikimedia-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/534439 (owner: 10Hashar) [15:05:09] (03CR) 10Thcipriani: [C: 03+2] fab: also pull wikimedia-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/534439 (owner: 10Hashar) [15:09:42] (03Merged) 10jenkins-bot: fab: also pull wikimedia-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/534439 (owner: 10Hashar) [15:15:30] (03PS1) 10Ladsgroup: Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) [15:16:57] (03CR) 10jerkins-bot: [V: 04-1] Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [15:24:52] (03PS2) 10Ladsgroup: Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) [15:25:31] (03CR) 10jerkins-bot: [V: 04-1] Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [15:30:25] (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester) [15:34:04] (03PS3) 10Ladsgroup: Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) [15:45:22] (03CR) 10Jforrester: "Oops. Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/534439 (owner: 10Hashar) [15:49:22] (03CR) 10Jforrester: [C: 03+2] "Nice." [integration/config] - 10https://gerrit.wikimedia.org/r/534393 (owner: 10Hashar) [15:53:41] (03Merged) 10jenkins-bot: zuul: test templates gates are superset of tests [integration/config] - 10https://gerrit.wikimedia.org/r/534393 (owner: 10Hashar) [16:00:27] (03CR) 10Krinkle: [C: 03+1] Disable output buffering in running php [integration/quibble] - 10https://gerrit.wikimedia.org/r/534468 (https://phabricator.wikimedia.org/T219694) (owner: 10Ladsgroup) [16:07:48] <_joe_> thcipriani: around? should we do a scap release? [16:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:20:40] (03PS9) 10Krinkle: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 [16:20:44] (03PS1) 10Hashar: CentralNotice: Depend on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/534481 [16:20:55] (03CR) 10Hashar: [C: 03+2] CentralNotice: Depend on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/534481 (owner: 10Hashar) [16:24:39] (03Merged) 10jenkins-bot: CentralNotice: Depend on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/534481 (owner: 10Hashar) [16:25:37] !log Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/534481/ [16:25:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:25:44] (03CR) 10AndyRussG: "@Hashar Merci bien!!!!" [integration/config] - 10https://gerrit.wikimedia.org/r/534481 (owner: 10Hashar) [16:27:49] _joe_: I'm in meetings for the next 1.5 hours :( [16:28:05] I can cut a release after that [16:28:18] <_joe_> thcipriani: well if you could prepare the release, I can build the deb and upload it tomorrow morning my time :) [16:28:24] (03PS10) 10Krinkle: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 [16:28:49] _joe_: I will do that, I'll ping you on the ongoing task when I've got the release prepped. [16:28:57] <_joe_> <3 [16:28:59] (03CR) 10Krinkle: "The diff is now consistently: test/test-wmf: s/hhvm/php72/ and php: s/php72/hhvm/." [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle) [16:29:07] <_joe_> in the meantime, I fixed the script we were going to use [16:29:34] hashar: Can you sanity check https://gerrit.wikimedia.org/r/#/c/integration/config/+/533973/ ? [16:29:36] <_joe_> it now causes an unnoticeable error bump even when ran aggressively across the clusters [16:29:47] <_joe_> and no perf penalty either [16:33:56] (03CR) 10jerkins-bot: [V: 04-1] Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle) [16:34:31] (03Abandoned) 10Jforrester: layout: Ensure all php: job sets test all five flavours [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester) [16:35:25] (03PS11) 10Krinkle: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 [16:37:23] James_F: Sorry for bypassing. I do think it's worth balancing the extra CI load vs maintenance cost (and don't have an opinion on that myself). I just wanted to switch to php72 first, as its own change. [16:38:07] Yeah, it's fine, was working on the same change myself, now I don't have to land it. ;-) [16:38:15] Did you want to deploy now? [16:38:32] Yeah [16:40:54] (03CR) 10Jforrester: [C: 03+2] Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle) [16:47:29] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) Awesome, thank you @fgiunchedi! [16:47:46] (03Merged) 10jenkins-bot: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle) [16:49:15] !log Zuul: Replacing basic quibble HHVM test with PHP72 https://gerrit.wikimedia.org/r/533973 [16:49:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:09] James_F: ahhh that is the change I forgot to rebase today sorry :-\ [16:50:23] No need to be sorry. :-) [16:50:32] and yeah +1 :] [16:51:26] 10Release-Engineering-Team (Deployment services), 10Security-Team, 10Wikimedia-Site-requests, 10Wikimedia-extension-review-queue: Deploy WebAuthn to Wikimedia Wikis - https://phabricator.wikimedia.org/T227242 (10Reedy) [16:54:50] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) [16:57:17] (03PS5) 10Jforrester: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 [16:57:19] (03PS3) 10Jforrester: layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228 [16:57:21] (03PS5) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 [17:00:20] back in a couple hours for the train [17:02:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10Daimona) [17:06:50] (03CR) 10Jforrester: [C: 03+2] jjb: convert phan jobs to job templates [integration/config] - 10https://gerrit.wikimedia.org/r/534395 (https://phabricator.wikimedia.org/T231966) (owner: 10Hashar) [17:06:53] (03CR) 10Jforrester: [C: 03+2] jjb: move mediawiki-i18n-check-docker def up [integration/config] - 10https://gerrit.wikimedia.org/r/534420 (owner: 10Hashar) [17:13:51] (03Merged) 10jenkins-bot: jjb: convert phan jobs to job templates [integration/config] - 10https://gerrit.wikimedia.org/r/534395 (https://phabricator.wikimedia.org/T231966) (owner: 10Hashar) [17:13:56] (03Merged) 10jenkins-bot: jjb: move mediawiki-i18n-check-docker def up [integration/config] - 10https://gerrit.wikimedia.org/r/534420 (owner: 10Hashar) [17:32:26] (03CR) 10Krinkle: "the test and mwcore changes are really nice :) - I'm not 100% sure about the test-1.31 changes etc. It seems beneficial to, for at long as" [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester) [17:35:10] (03CR) 10Jforrester: "> Patch Set 5:" [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester) [17:50:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) >>! In T225730#5450672, @Simetri... [17:56:12] 10Release-Engineering-Team-TODO (201909), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) [18:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:38:45] (03CR) 10Jforrester: [C: 03+2] layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester) [18:40:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10WDoranWMF) [18:40:57] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:41:21] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Jdforrester-WMF) [18:41:48] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Jdforrester-WMF) [18:42:10] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Branch REL1_34 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T232024 (10Jdforrester-WMF) [18:42:45] 10MediaWiki-Releasing, 10Release-Engineering-Team-TODO, 10MW-1.34-notes, 10MW-1.34-release: Configure CI to support REL1_34 - https://phabricator.wikimedia.org/T232025 (10Jdforrester-WMF) [18:43:06] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Announce 1.34 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T232026 (10Jdforrester-WMF) [18:43:19] (03Merged) 10jenkins-bot: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester) [18:43:29] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34.0-rc.0 - https://phabricator.wikimedia.org/T232027 (10Jdforrester-WMF) [18:43:52] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Prune /vendor for REL1_34 - https://phabricator.wikimedia.org/T232028 (10Jdforrester-WMF) [18:45:47] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Announce 1.34 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T232026 (10Jdforrester-WMF) The plan is that 1.34.0-wmf.25, cut and shipped to production on **30 September 2019**, will b... [18:46:45] 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) [18:46:58] 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) [18:47:01] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Branch REL1_34 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T232024 (10Jdforrester-WMF) [18:47:39] (03PS4) 10Jforrester: layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228 (https://phabricator.wikimedia.org/T232025) [18:48:01] (03CR) 10jerkins-bot: [V: 04-1] layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228 (https://phabricator.wikimedia.org/T232025) (owner: 10Jforrester) [18:48:30] !log Zuul: Move quibble and mwgate basic test jobs from php70 to php72 [18:48:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:04] (03PS5) 10Jforrester: layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228 (https://phabricator.wikimedia.org/T232025) [19:01:27] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:28:03] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Release, 10Train Deployments: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [19:28:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, and 2 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [19:31:18] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Note that the al... [19:31:25] (03PS6) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 [19:32:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [19:32:23] (03PS1) 10Brennen Bearnes: restbase: bump image to 0.0.1-2 for image rebuild [releng/dev-images] - 10https://gerrit.wikimedia.org/r/534516 [19:52:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [19:58:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) p:05Triage→03... [20:01:31] (03PS7) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 [20:01:33] (03PS1) 10Jforrester: [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T176370) [20:01:35] (03PS1) 10Jforrester: [DNM] layout: Drop HHVM testing from all quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/534521 (https://phabricator.wikimedia.org/T192166) [20:01:37] (03PS1) 10Jforrester: layout: Collapse -nohhvm jobs into their base as we've dropped that [integration/config] - 10https://gerrit.wikimedia.org/r/534522 [20:01:39] (03PS1) 10Jforrester: layout: Drop all HHVM testing except for Fundraising [integration/config] - 10https://gerrit.wikimedia.org/r/534523 [20:02:03] Those were very cathartic to write, even if they won't land soon. [20:04:13] (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop HHVM testing from all quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/534521 (https://phabricator.wikimedia.org/T192166) (owner: 10Jforrester) [20:04:15] (03CR) 10jerkins-bot: [V: 04-1] layout: Collapse -nohhvm jobs into their base as we've dropped that [integration/config] - 10https://gerrit.wikimedia.org/r/534522 (owner: 10Jforrester) [20:04:17] (03CR) 10jerkins-bot: [V: 04-1] layout: Drop all HHVM testing except for Fundraising [integration/config] - 10https://gerrit.wikimedia.org/r/534523 (owner: 10Jforrester) [20:05:40] James_F: when are we branching 1.34 ? [20:07:07] (I remember adding pictures to https://www.mediawiki.org/wiki/MediaWiki_1.18#What's_new? :D ) [20:09:43] I am off *wave* [20:09:44] ;) [20:11:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:38:10] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5463992, @Joe wrote: > @thcipriani should we create a... [20:40:51] antoine's not here, but in our team etherpad: * Oct 14 - 1.35.0-wmf.1 [20:42:18] :O the date of the proposed election or rather was :P [20:44:39] Also sukkot. [20:45:16] Also, no, we'd branch REL1_34 with 1.34.0wmf.25, so on 30 September. [20:45:21] Hence my flurry of tasks. :-) [20:47:29] (Key one is T232026 which I might be able to lure CPT into doing this week.) [20:47:30] T232026: Announce 1.34 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T232026 [20:50:32] James_F heh, that's why they changed it to the 15 which also failed :P [21:12:25] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Ladsgroup) [21:26:46] 10Release-Engineering-Team-TODO (201909), 10Wikidata, 10Wikidata-Termbox, 10serviceops, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Ladsgroup) The link to logs: https://logstash.wikimedia.org/... [21:31:05] 10MediaWiki-Releasing, 10Release-Engineering-Team-TODO (201909), 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Branch REL1_34 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T232024 (10Jdforrester-WMF) 05Open→03Stalled To be done on 2019-09-30 alongside the wmf.2... [21:31:08] 10Release-Engineering-Team-TODO (201909), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) [21:31:13] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Jdforrester-WMF) [22:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [22:13:12] James_F: Looks like jsduck/doxygen stopped running on tags [22:13:12] https://doc.wikimedia.org/mediawiki-core/ [22:13:24] stopped at 1.30.0 and non-zero times for REL1_32 [22:13:30] Eurgh, fun. [22:13:42] REL1_33 as well. [22:13:46] So I guess the branch trigger is working [22:13:48] but tag stopped [22:13:54] Do we have tags for 1.31.0+? [22:14:05] sure [22:14:06] Yes. [22:14:26] https://github.com/wikimedia/integration-config/commit/8f9b3528176eca6c60dee0bb8b2c65a3cd21ba2a [22:14:27] Aha [22:14:38] It was changed from postmerge/publish to an hourly cron [22:14:42] which watches branches only [22:14:44] Oh. [22:15:44] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: MW core doxygen / jsduck build doesn't run for tags, just branches - https://phabricator.wikimedia.org/T232055 (10Jdforrester-WMF) [22:15:50] runs take about 9 min currently [22:15:54] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9614/ [22:16:11] I'm doing some optimisations for that right now as part of clean up and prep work to make Markdown files easier to show in Doxygen [22:16:18] already possible today but ugly and thus nobody does it [22:16:24] Neat [22:16:33] maybe if it runs faster after that we can go back to debounced postmerge [22:16:40] Can we run it "debounced" in post-merge somehow nicely? [22:16:53] Well, kind of. I cheated when I said that [22:16:56] there is no debounce feature [22:16:59] Previously it was non-concurrent post-merge. [22:17:06] So 20 merges means 20 runs. [22:17:08] yeah, low-prio queue that gets backlogged [22:17:09] Which is a bit sucky. [22:17:19] We do that already for codehealth and jsuck on all core commits though [22:17:22] And that queue already gets quite backlogged. [22:17:24] jsduck* [22:17:31] not intentional misspelling [22:17:37] Yeah, we can combine the two jobs though [22:17:44] mediawiki-core-publish-docker (jsduck+doxygen) [22:17:51] or some such [22:18:07] and yeah, there is an open task somewhere for it to be debounced [22:18:13] I probably filed that in '15 or '13 sometime [22:18:20] So, recently? ;-) [22:24:30] Hey releng folks: is there any way a gerrit admin could kill a patch set as if it never existed? [22:26:41] they can delete changes. [22:27:04] thcipriani ^ [22:28:41] sbassett: I do have a delete changes button, yes [22:28:56] It'll still be in Phab somewhere. [22:28:59] it deletes the full change (not just a patchset)note [22:29:14] * note: it deletes the full change (not just a patchset) [23:02:08] James_F: even on an unmerged WIP patch? Diffusion picks those up? [23:02:39] yeh [23:02:45] sbassett gerrit is all NoteDB :) [23:03:02] so diffusion pulls in all refs [23:03:43] Interesting. Thanks - chatting thcipriani right now. [23:04:24] even gerrit's private change allows diffusion to pull it in. which i reported upstream. [23:04:59] Yeah, essentially Phabricator security model is terrible and gerrit security model is terrible and yoked together like we have them they're extra-terrible. [23:07:49] Cool. Well, I made an extra-terrible test data mistake, so I guess as long as everything is strategically-aligned. [23:09:10] I'm not sure I can find that repo in phab... [23:12:07] thcipriani: I wouldn't be surprised if it was never configured there. It's really just for random, internal secteam stuff. [23:19:14] You can set repos as private though [23:20:24] sbassett ^ [23:23:41] James_F: lol, two versions of the same patch are behind each other in the gate [23:24:09] Don't we have code to stop that? [23:24:14] https://usercontent.irccloud-cdn.com/file/wCJ3HSzD/capture.png [23:24:23] * James_F nods. [23:24:37] so it waits for the whole thing, only to realise it can't merge non-current PS, [23:24:41] so its starts again [23:33:06] I thought we had a reaper that killed un-mergeable builds from gate-and-submit? [23:34:23] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10serviceops, 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) [23:35:37] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10serviceops, 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) p:05Triage→03High [23:52:41] 10Phabricator, 10Documentation: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10mmodell) a:03mmodell [23:53:02] thcipriani i think upstream are planning to EOL gerrit 2.15 in nov [23:53:03] https://www.gerritcodereview.com/2019-09-03-esc-minutes.html#minutes [23:54:05] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Documentation, 10User-MModell: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10mmodell) p:05Triage→03Normal I'll take a stab at writing some documentation on wiki in the near future. [23:55:19] 10Phabricator, 10Release-Engineering-Team-TODO, 10serviceops, 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) [23:55:25] 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, 10User-MModell: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10mmodell) [23:58:13] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) [23:58:40] 10Phabricator, 10Release-Engineering-Team-TODO (201908), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell)