[00:01:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[01:01:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[01:16:03] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[02:11:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[03:09:49] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%)
[04:11:01] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[06:11:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[06:54:47] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:18:47] <dcausse>	 hi, is there a way for a particular build to inspect mediawiki generated logs of a selenium build? in my case for this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/533956
[07:51:34] <onimisionipe>	 James_F: hello! can you take a look at: https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/530836 when you have a min
[08:11:04] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[08:16:30] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Remove wmf pipelines from non mediawiki repos [integration/config] - 10https://gerrit.wikimedia.org/r/533905 (owner: 10Hashar)
[08:18:09] <wikibugs>	 (03Merged) 10jenkins-bot: Remove wmf pipelines from non mediawiki repos [integration/config] - 10https://gerrit.wikimedia.org/r/533905 (owner: 10Hashar)
[08:19:31] <dcausse>	 hashar: o/
[08:19:50] <hashar>	 dcausse: guten tag
[08:20:02] <wikibugs>	 (03PS1) 10Mathew.onipe: layout: [kartotherian] enable pipeline on gate [integration/config] - 10https://gerrit.wikimedia.org/r/534033 (https://phabricator.wikimedia.org/T223275)
[08:20:17] <dcausse>	 do you know how to access mw generated logs of a selenium build?
[08:20:49] <dcausse>	 context is this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/533956
[08:21:30] <dcausse>	 Amir says: "According to logs, this patch doesn't disable CirrusSearch"
[08:21:48] <dcausse>	 how can I see these logs? :)
[08:24:05] <hashar>	 dcausse: the build should have LocalSettings.php attached as well as files such as  mw-debug-*.log.gz
[08:24:28] <hashar>	 the selenium tests hit the web front end and MediaWiki should have been configured to write debug logs to mw-debug-www.log.gz
[08:26:00] <dcausse>	 is this downloadable from jenkins?
[08:26:30] <dcausse>	 damn I see it now
[08:26:38] <dcausse>	 hashar: sorry for the noise and thanks!
[08:26:39] <hashar>	 oh
[08:30:45] <hashar>	 dcausse: I don't even know whether that file is still loaded
[08:31:05] <dcausse>	 hashar: the file updated in the patch?
[08:31:11] <hashar>	 yeah
[08:31:17] <hashar>	 but maybe that is loaded from Wikibase itself
[08:31:33] <dcausse>	 if it's loaded PS2 should fix the issue
[08:32:22] <hashar>	 !log reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533909/ "zuul: add pipelines for MediaWiki releases"
[08:32:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:32:29] <hashar>	 err
[08:32:39] <hashar>	 !log err
[08:32:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:32:49] <hashar>	 !log reloading zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/533905/ "Remove wmf pipelines from non mediawiki repos"
[08:32:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:35:54] <dcausse>	 indeed my fix does not work meaning that file is perhaps no longer loaded...
[08:41:32] <hashar>	 dcausse: another way is to vary config based on   isset( $wgWikimediaJenkinsCI ) && $wgWikimediaJenkinsCI == true
[08:41:38] <hashar>	 that variable is only set by Quibble / CI
[08:41:55] <hashar>	 and this way disable the deferred updates  maybe
[08:42:58] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "That breaks changes send to REL branches since there are no jobs added" [integration/config] - 10https://gerrit.wikimedia.org/r/533909 (owner: 10Hashar)
[08:45:02] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: add pipelines for MediaWiki releases [integration/config] - 10https://gerrit.wikimedia.org/r/533909 (owner: 10Hashar)
[08:47:15] <dcausse>	 hashar: but I'm not sure which config file I need to update
[08:52:49] <dcausse>	 ah the suggested way is to set defaults in the code that match what we want to test
[08:54:10] <dcausse>	 but here we want to depend on CirrusSearch for running its phpunit tests but we don't want CirrusSearch loaded for selenium tests
[09:09:58] <wikibugs>	 (03PS2) 10Hashar: Apply REL based pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/533916 (https://phabricator.wikimedia.org/T189560)
[09:10:34] <wikibugs>	 (03PS3) 10Hashar: Apply REL based pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/533916 (https://phabricator.wikimedia.org/T189560)
[09:11:30] <hashar>	 dcausse: sorry can't really follow up, I have to break CI this morning :-\\\
[09:12:09] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Lets break the CI 🌎" [integration/config] - 10https://gerrit.wikimedia.org/r/533916 (https://phabricator.wikimedia.org/T189560) (owner: 10Hashar)
[09:13:42] <dcausse>	 np
[09:15:48] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10AbuseFilter, and 3 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560...
[09:16:00] <wikibugs>	 (03Merged) 10jenkins-bot: Apply REL based pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/533916 (https://phabricator.wikimedia.org/T189560) (owner: 10Hashar)
[09:16:18] <hashar>	 !log Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
[09:16:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:17:23] <hashar>	 !log Reloaded Zuul for  https://gerrit.wikimedia.org/r/533916 "Apply REL based pipelines"
[09:17:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:25:54] <wikibugs>	 (03PS1) 10Hashar: Skip PHP 7.0, 7.1 and 7.3 on Wikibase repositories [integration/config] - 10https://gerrit.wikimedia.org/r/534117
[09:28:23] <wikibugs>	 (03CR) 10Hashar: "Added as reviewers a few persons from WMDE since I am not really sure whom to ask exactly." [integration/config] - 10https://gerrit.wikimedia.org/r/534117 (owner: 10Hashar)
[09:44:40] <wikibugs>	 (03CR) 10Pablo Grass (WMDE): "The reasoning makes sense to me as a concession made to operational needs - of course it somewhat takes the C out of CI for these versions" [integration/config] - 10https://gerrit.wikimedia.org/r/534117 (owner: 10Hashar)
[09:53:45] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Pending discussions with WMDE, there might be 3rd party users." [integration/config] - 10https://gerrit.wikimedia.org/r/534117 (owner: 10Hashar)
[09:55:48] <wikibugs>	 10Continuous-Integration-Config, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10Ladsgroup)
[10:03:00] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10hashar) Sounds like a bug in #quibble
[10:11:03] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[10:14:46] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10Lucas_Werkmeister_WMDE) Since the logs will be gone eventually, here’s two screenshots for posterity:  {F30209845,layout=inline} {F30209844,layout=inline}
[10:19:39] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10AbuseFilter, and 3 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560...
[10:27:42] <wikibugs>	 10Continuous-Integration-Config, 10Documentation: doc.wikimedia.org contains empties on end of page - https://phabricator.wikimedia.org/T231421 (10hashar) p:05Triage→03Low Seems fine with Firefox, but on Chromium eventually there is some extra space at the bottom.  I don't think we should bother with that...
[10:30:21] <wikibugs>	 (03PS1) 10Hashar: Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394)
[10:30:38] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10VisualEditor, 10Patch-For-Review: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10hashar)
[10:31:01] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[10:31:23] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10VisualEditor, 10Patch-For-Review: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10hashar) Because supposedly it is only for mediawiki core, extensions a...
[10:32:03] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394) (owner: 10Hashar)
[10:52:02] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10awight) Wikibase appears twice in this list: > 00:02:15.523 DEBUG:quibble.cmd:Browser tests using DISPLAY=Xvfb, for projects mediawiki/extensions/Wikibas...
[10:56:11] <wikibugs>	 (03PS9) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862)
[10:56:54] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:18:04] <wikibugs>	 (03PS1) 10Awight: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862)
[11:18:49] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:19:01] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10awight) a:03awight Ready for review.
[11:30:06] <wikibugs>	 (03PS1) 10Awight: Fix file permissions in CI [integration/quibble] - 10https://gerrit.wikimedia.org/r/534137
[11:31:40] <wikibugs>	 (03PS10) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862)
[11:31:44] <wikibugs>	 (03PS2) 10Awight: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862)
[11:32:54] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:32:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:36:00] <wikibugs>	 (03Abandoned) 10Awight: Fix file permissions in CI [integration/quibble] - 10https://gerrit.wikimedia.org/r/534137 (owner: 10Awight)
[11:37:01] <wikibugs>	 (03PS11) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862)
[11:37:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:43:41] <wikibugs>	 (03PS2) 10Hashar: Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394)
[11:44:12] <wikibugs>	 (03CR) 10Awight: "I'm confused, log_dir should be defaulting to `/workspace/log` but instead is `/workspace` ." [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:44:36] <hashar>	 zeljkof: so what is up with the train ? ;]
[11:45:04] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394) (owner: 10Hashar)
[11:45:44] <wikibugs>	 (03PS3) 10Hashar: Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394)
[11:46:54] <wikibugs>	 (03CR) 10Awight: "It must be related to mocking os.environ, but still not clear what's broken." [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[11:48:35] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-TwnMainPage, 10Language-Team (Language-2019-July-September): Stop running HHVM tests for TwnMainPage - https://phabricator.wikimedia.org/T231289 (10hashar) 05Open→03Resolved
[11:58:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10MobileFrontend: Javascript test failures for release-quibble-composer-mysql-hhvm-docker - https://phabricator.wikimedia.org/T230454 (10hashar) That might have been due to...
[11:59:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10MobileFrontend: Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10hashar)
[11:59:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10MobileFrontend: Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10hashar) p:05Triage→03Normal
[12:03:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908), 10Technical-Debt: Cleanup CI puppet manifests - https://phabricator.wikimedia.org/T225735 (10hashar) 05Open→03Resolved I guess it is good enough for now, the bulk of th...
[12:11:04] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[12:13:34] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10hashar) So there have been no blockers since Friday.  Monday was a US holiday...
[12:27:49] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10hashar) 05Open→03Resolved
[12:33:26] <icinga-wm>	 PROBLEM - SSH access on cobalt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit
[12:36:28] <icinga-wm>	 RECOVERY - SSH access on cobalt is OK: SSH OK - GerritCodeReview_2.15.14-16-g855b179b5f (SSHD-CORE-1.6.0) (protocol 2.0) https://wikitech.wikimedia.org/wiki/Gerrit
[12:40:45] <Reedy>	 13:30:45 INFO:mw.maintenance.install:php maintenance/install.php --scriptpath= --dbtype=mysql --dbname=wikidb --dbuser=wikiuser --dbpass=secret --dbserver=/workspace/db/quibble-mysql-0arwgn48/socket --with-extensions --pass=testwikijenkinspass TestWiki WikiAdmin
[12:40:46] <Reedy>	 13:30:46 
[12:40:46] <Reedy>	 13:30:46 Fatal error: Undefined interface: Psr\Container\ContainerInterface in /workspace/src/includes/libs/services/ServiceContainer.php on line 49
[12:40:51] <Reedy>	 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/69202/console
[12:41:05] <Reedy>	 hashar: ^ I know you're busy with thet rain... but dunno if that has anything to do with your refactoring?
[12:41:18] <hashar>	 it is sunny here!
[12:41:23] <Reedy>	 *the train :P
[12:41:30] <hashar>	 oh
[12:41:46] <hashar>	 na unrelated to my refactoring from this morning
[12:41:52] <Reedy>	 That error sounds like psr/container from mw core's composer.json isn't being loaded
[12:42:21] <Reedy>	 It's definitey not the patch at fault https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OATHAuth/+/534146/
[12:42:32] <hashar>	 the change I made this morning is stop triggering the jobs using vendor.git on release branches
[12:42:43] <hashar>	 maybe that is an issue from mediawiki/core ? 
[12:43:18] <hashar>	 and that happens on maintenance/install.php so at the very start
[12:43:40] <Reedy>	 psr/container has been in for 3 months...
[12:44:35] <hashar>	 yeah then hmm
[12:44:57] <hashar>	 it can be whatever other reason, such as the extension.json being loaded before the installer had a chance to fully load the autoloader
[12:45:06] <hashar>	 I don't know really
[12:45:30] <hashar>	 but that should be bisectable 
[12:46:31] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General: Extension quibble-vendor-mysql-hhvm-docker failing with Undefined interface: Psr\Container\ContainerInterface - https://phabricator.wikimedia.org/T231869 (10Reedy)
[12:46:43] <hashar>	 Reedy: you can try   with mediawiki/core  + OATHAuth cloned in ./extensions
[12:46:52] <hashar>	 then install using   maintenance/cli.php --with-extensions
[12:46:57] <hashar>	 that should reproduce it
[12:47:46] <hashar>	 also maybe  check php would get a stacktrace on the fatal error
[12:49:37] <Reedy>	 [13:48:58] <wikibugs> (Merged) jenkins-bot: Modify -—with-extensions to throw extension dependency errors [core] - https://gerrit.wikimedia.org/r/533239 (https://phabricator.wikimedia.org/T225512) (owner: Clarakosi)
[12:49:42] <Reedy>	 Anything to do with this you reckon?
[12:49:59] <Reedy>	 Hmm. Except that merged after the test
[12:51:19] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-General: Extension quibble-vendor-mysql-hhvm-docker failing with Undefined interface: Psr\Container\ContainerInterface - https://phabricator.wikimedia.org/T231869 (10Reedy) 05Open→03Stalled Might've been transient...  Recheck on https://gerrit.wikimedia.org/r/#/...
[12:51:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 3 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10mobrovac) 05Ope...
[12:56:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 3 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Reedy) >>! In T2...
[13:04:44] <wmf-insecte>	 Project beta-code-update-eqiad build #262142: 04FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262142/
[13:06:39] <Lucas_WMDE>	 Gerrit feels pretty slow right now…
[13:06:46] <Lucas_WMDE>	 (not sure if -releng is the right channel for that, apologies if it isn’t)
[13:07:14] <aharoni>	 Lucas_WMDE that's the thing I came here to complain about :)
[13:07:27] <hashar>	 Lucas_WMDE: aharoni yeah it is slow and i have no clue why :-^
[13:07:32] <hashar>	 beside that I am cutting the wmf branches
[13:08:26] <aharoni>	 hashar also https://gerrit.wikimedia.org/r/534148 fails for an unclear reason. It's not supposed to cause any changes.
[13:10:23] <hashar>	 aharoni: fill a task pleas
[13:10:24] <hashar>	 e
[13:10:24] <hashar>	 :)
[13:10:45] <hashar>	 Reedy: no idea really :]  But your issue really smell like an issue in core
[13:11:03] <hashar>	 for Gerrit hmm nothing show up really
[13:12:24] <Lucas_WMDE>	 getting some proxy errors on Gerrit as well
[13:12:38] <Lucas_WMDE>	 presumably that’s the same slowness problem, just occasionally crossing a threshold where the proxy gives up waiting for the backend
[13:14:40] <hashar>	 Lucas_WMDE: exactly
[13:14:46] <wmf-insecte>	 Project beta-code-update-eqiad build #262143: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262143/
[13:14:49] <hashar>	 Gerrit http server stops responding for whatever reason
[13:14:53] <Lucas_WMDE>	 GC time is exploding on https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody
[13:15:00] <hashar>	 then apache proxy in front of it eventually reaches whatever magic timeout
[13:15:17] <Lucas_WMDE>	 aharoni: are you creating the task or should I do it?
[13:16:23] <hashar>	 aharoni: fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/UniversalLanguageSelector/': The requested URL returned error: 502
[13:16:30] <hashar>	 aharoni: so yeah that is gerrit not replying
[13:16:37] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani)
[13:17:17] <aharoni>	 Lucas_WMDE creating :)
[13:17:24] <hashar>	 aharoni: created already ^^^ :D
[13:17:35] <aharoni>	 Oh :)
[13:17:46] <aharoni>	 thanks
[13:17:56] <Lucas_WMDE>	 ah damn I did it as well now ^^
[13:18:18] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10Lucas_Werkmeister_WMDE)
[13:19:12] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10hashar) That is definitely started with the branch cut.  Can't we raise the HEAP and or throw some more RAM at the machine?
[13:19:18] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani) Also relevant is the GC timing chart from the train window. The branch cut for train today began at 12:32UTC: https://grafana.wikimedia.org/d/Bw2mQ3i...
[13:19:19] <hashar>	 to be fair, thcipriani should have breakfast now :D
[13:23:21] <Lucas_WMDE>	 probably not related, but the errors count also looks suspicious https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?from=1567430563330&to=1567516963330
[13:23:33] <Lucas_WMDE>	 oscillates between 0 and 0.4 every half hour, like clockwork
[13:23:58] <Lucas_WMDE>	 oh, nevermind, that’s due to the aggregation window
[13:24:09] <Lucas_WMDE>	 if I change that to 5m it looks more like spikes on the full hour
[13:24:29] <wmf-insecte>	 Yippee, build fixed!
[13:24:29] <wmf-insecte>	 Project beta-code-update-eqiad build #262144: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/262144/
[13:25:23] <hashar>	 branch has been cut so Gerrit should be fine again
[13:26:38] <hashar>	 Lucas_WMDE: should be good now
[13:26:40] <hashar>	 aharoni: you can recheck
[13:28:39] <Lucas_WMDE>	 thanks
[13:33:00] <paladox>	 https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=httpMeanTimes
[13:33:05] <paladox>	 confirms the slowness
[14:05:05] <James_F>	 onimisionipe: Hey, is the kartotherian patch ready to land?
[14:06:03] <onimisionipe>	 James_F: almost :) It requires your reviews and the releng team
[14:06:20] <James_F>	 I've just C+1'ed; I could merge, if you want?
[14:09:28] <Reedy>	 14:37:45 Could not find the registration file for the extension "OATHAuth"
[14:09:28] <Reedy>	 14:37:45 [b3e1c0851ed01e6211c7ea27] [no req]   Wikimedia\Services\ServiceDisabledException from line 423 of /workspace/src/includes/libs/services/ServiceContainer.php: Service disabled: DBLoadBalancerFactory
[14:09:36] <James_F>	 Oh dear.
[14:09:39] <James_F>	 In prod?
[14:10:01] <Reedy>	 well, CI..
[14:10:06] <Reedy>	 https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/10457/console
[14:10:36] <James_F>	 Oh, just someone trying to run a unit test with services?
[14:10:57] <Reedy>	 It wasn't broken earlier today
[14:10:58] <James_F>	 There was an issue with the unit test runner that tried but failed to isolate the run.
[14:11:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[14:11:22] <James_F>	 If the fix for that landed in the last few hours, it'd have silently broken some unit tests.
[14:11:44] <James_F>	 Oh, the WebAuthn unmerged mega-patch?
[14:11:46] <James_F>	 Yeah, sorry. :-(
[14:12:16] <James_F>	 Oh, no, this is the other change, asserting that extensions are loaded in CI?
[14:12:25] <James_F>	 Surely WebAuthn depends on OATHAuth in CI already?
[14:12:27] * James_F checks.
[14:12:32] <Reedy>	 The first line would look like that
[14:13:06] <James_F>	     'WebAuthn': ['OATHAuth'],
[14:13:22] <onimisionipe>	 James_F: not yet please. I need to make one last change
[14:13:35] <James_F>	 onimisionipe: Sure.
[14:13:54] <James_F>	 Reedy: Yeah, d9eec3c9124d87fd44e6917d5b1512b78352afb3 just landed.
[14:14:06] <James_F>	 `Modify -—with-extensions to throw extension dependency errors`
[14:14:30] <James_F>	 (Which you commented on, Reedy. ;-))
[14:14:33] <Reedy>	 I know
[14:15:05] <James_F>	 00:00:19.216 INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/skins/Vector, mediawiki/vendor, mediawiki/extensions/WebAuthn
[14:15:08] <James_F>	 That seems wrong.
[14:15:18] <Reedy>	 looks fishy indeed
[14:15:25] <James_F>	 It should also be pulling OATHAuth.
[14:16:30] <James_F>	 Yeah, lots and lots of failures.
[14:17:56] <wikibugs>	 (03PS1) 10Jforrester: layout: Drop EducationProgram from dependencies, archived repo [integration/config] - 10https://gerrit.wikimedia.org/r/534164
[14:17:57] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Drop EducationProgram from dependencies, archived repo [integration/config] - 10https://gerrit.wikimedia.org/r/534164 (owner: 10Jforrester)
[14:17:59] <James_F>	 Fixing the EducationProgram one right away, at least
[14:18:24] <James_F>	 hashar: Do you have any ideas why CI is not injecting the right dependencies?
[14:18:38] <hashar>	 check whether EXT_DEPENDENCIES is properly set?
[14:18:45] <hashar>	 that is available in the build parameters
[14:19:00] <hashar>	 alternative, look at the start of the quibble console output, it should dump the list of projects to clone
[14:19:35] <James_F>	 So certainly some EXT_DEPENDENCIES values are getting set.
[14:19:39] <hashar>	 good
[14:19:53] <hashar>	 and quibble supposedly recognizes that variable (though it emits a warning about it)
[14:19:58] <hashar>	 and would then clone each of those repos
[14:20:09] <James_F>	 E.g. https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php72-docker/3497/parameters/ -> EventLogging; Translate; UniversalLanguageSelector; cldr
[14:20:13] <James_F>	 Which is right.
[14:20:32] <hashar>	 then given that the extension is valid / detected by   install.php,  it would eventually be injected in  LocalSettings.php as a wfLoadExtension( 'Foobar' );
[14:20:43] <James_F>	 Yeah.
[14:20:44] <hashar>	 and that LocalSettings.php should be attached to the jenkins build page
[14:21:09] <James_F>	 For https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/10457/parameters/ EXT_DEPENDENCIES is blank.
[14:21:12] <James_F>	 Which is wrong.
[14:21:31] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Drop EducationProgram from dependencies, archived repo [integration/config] - 10https://gerrit.wikimedia.org/r/534164 (owner: 10Jforrester)
[14:21:44] <James_F>	 !log Zuul: Drop EducationProgram from dependencies
[14:21:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:23:06] <James_F>	 Aha.
[14:23:15] <James_F>	 On a recheck, https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/17743/parameters/ has the right EXT_DEPENDENCIES set.
[14:23:36] <James_F>	 Did we reload Zuul/gearman/whatever in a way that some builds were set up wrongly?
[14:23:49] <James_F>	 Most confusing.
[14:23:50] <wikibugs>	 (03PS1) 10Lucas Werkmeister (WMDE): Remove EducationProgram from parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/534168 (https://phabricator.wikimedia.org/T214457)
[14:24:27] <wikibugs>	 (03CR) 10Jforrester: "Already done and deployed. See I9893adb308." [integration/config] - 10https://gerrit.wikimedia.org/r/534168 (https://phabricator.wikimedia.org/T214457) (owner: 10Lucas Werkmeister (WMDE))
[14:24:49] <wikibugs>	 (03Abandoned) 10Lucas Werkmeister (WMDE): Remove EducationProgram from parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/534168 (https://phabricator.wikimedia.org/T214457) (owner: 10Lucas Werkmeister (WMDE))
[14:24:50] <hashar>	 James_F: or it is not applied to the mwext-php72-phan-docker job ? :\
[14:24:59] <hashar>	 well
[14:25:02] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): "aah you’re too fast for me :D" [integration/config] - 10https://gerrit.wikimedia.org/r/534168 (https://phabricator.wikimedia.org/T214457) (owner: 10Lucas Werkmeister (WMDE))
[14:25:04] <hashar>	 probably that should not applied anyway
[14:25:04] <James_F>	 hashar: But it worked the second time.
[14:25:09] <hashar>	 different job
[14:25:29] <wikibugs>	 (03CR) 10Pablo Grass (WMDE): "Duplicate of I9893adb308514def3879bf72a0c0f6c0ef1a1748" [integration/config] - 10https://gerrit.wikimedia.org/r/534168 (https://phabricator.wikimedia.org/T214457) (owner: 10Lucas Werkmeister (WMDE))
[14:25:30] <hashar>	 anyway, I think the zuul debug log has all the info related to which parameters are injected
[14:25:32] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) p:05Triage→03Unbreak! a:03Jdforrester-WMF
[14:25:36] <hashar>	 that might have some more details
[14:25:52] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) This theoretically is now fixed. Checking results of a few more runs...
[14:26:46] <hashar>	 meanwhile, I have merged/deployed the new zuul pipelines for release branches
[14:26:51] <James_F>	 I saw.
[14:26:53] <James_F>	 Nice work.
[14:27:10] <wikibugs>	 10Phabricator: Some users can't add the fr-tech private task form to their task selection menu. - https://phabricator.wikimedia.org/T229628 (10DStrine) 05Declined→03Open
[14:27:11] <hashar>	 and I most probably broke a few things here and there :-\
[14:27:17] <James_F>	 Sure.
[14:27:19] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Pablo-WMDE) Thanks people!
[14:27:59] <wikibugs>	 10Phabricator: Some users can't add the fr-tech private task form to their task selection menu. - https://phabricator.wikimedia.org/T229628 (10DStrine) I have seen this happen when @Ppena shared her screen with me. Can someone get on a call with her and check it out?
[14:28:23] <James_F>	 So for the non-suffixed pipelines (test-prio, patch-performance, experimental, coverage, codehealth, php, postmerge, post, publish) do we "just" run all branches in those pipelines? I guess most of them are master-only?
[14:28:43] <hashar>	 50minutes for the scap sync ;-\\
[14:28:47] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) Apologies for the late reply, I was really busy with {T220745} 🚂  There's just 3 accounts at https://app.saucelabs.com/team-management: - jenki...
[14:28:48] <wikibugs>	 (03CR) 10Alaa Sarhan: [C: 03+1] "I think not running browser tests on all supported versions of PHP is quite desirable, as long as we keep running unit/integration tests o" [integration/config] - 10https://gerrit.wikimedia.org/r/534117 (owner: 10Hashar)
[14:29:00] <James_F>	 Ouch, really?
[14:29:11] <hashar>	 yeah well
[14:29:18] <hashar>	 it syncs the masters two or three times
[14:29:24] <hashar>	 check canaries a couple time for no real reason
[14:29:37] <hashar>	 and then the usual network transfer being long as well as cdb rebuild ;]
[14:29:43] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394) (owner: 10Hashar)
[14:29:45] <hashar>	 anyway I have let it run in the background so ..
[14:29:58] <James_F>	 It's still a bunch longer than normal.
[14:30:02] <James_F>	 Ah.
[14:30:23] <James_F>	 Anyway, now that the UBN disaster is over I'm going to get some breakfast. ;-)
[14:31:25] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) @Niedzielski which data center are you logging in to? US West 1, EU Central 1, Headless US-East? I don't know that it matters, just checking.
[14:33:17] <wikibugs>	 (03Merged) 10jenkins-bot: Add wmf pipelines to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/534130 (https://phabricator.wikimedia.org/T231394) (owner: 10Hashar)
[14:34:34] <James_F>	 !log Zuul: And wmf/ branches for VisualEditor core.
[14:34:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:37:42] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF)
[14:42:56] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani)
[14:42:59] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10hashar) 05Open→03Resolved The actual error was: ` Could not find the registration file for the extensio...
[14:43:40] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909): Upgrade Gerrit to 2.15.15 - https://phabricator.wikimedia.org/T229110 (10thcipriani)
[14:53:17] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10WDoranWMF)
[14:58:07] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201908): [Cloud VPS alert] Puppet failure on integration-agent-puppet-docker-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T231447 (10thcipriani) 05Open→03Resolved a:03thcipriani I saw this email as...
[14:58:28] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Documentation: Document how to deploy a new Quibble version to CI - https://phabricator.wikimedia.org/T231251 (10thcipriani)
[14:58:52] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10WDoranWMF)
[14:59:42] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Graphoid, 10Release, 10Train Deployments: 1.34.0-wmf.21 cause Graphoid service check to fail due to 403 from mediawiki.org - https://phabricator.wikimedia.org/T231894 (10hashar)
[15:00:34] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Graphoid, 10Release, 10Train Deployments: 1.34.0-wmf.21 cause Graphoid service check to fail due to 403 from mediawiki.org - https://phabricator.wikimedia.org/T231894 (10hashar) p:05Triage→03Unbreak! Train blockers are UBN
[15:00:43] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10VisualEditor, and 2 others: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Confirme...
[15:01:00] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10VisualEditor, and 2 others: No CI for wmf branches of VE-core - https://phabricator.wikimedia.org/T231394 (10Jdforrester-WMF) a:05Jdforrester-WMF→03hashar
[15:01:09] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10hashar) Eventually that hasn't reached out group0 due to mediawiki.org / graphoid having some breaking interaction T231894
[15:12:15] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Graphs, 10Core Platform Team Workboards (Clinic Duty Team), 10Release, 10Train Deployments: 1.34.0-wmf.21 cause Graphoid service check to fail due to 403 from mediawiki.org - https://phabricator.wikimedia.org/T231894 (10mobrovac)
[15:20:00] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) Seems to not be fixed for phan tasks? https://integration.wikimedia.org/ci/job/mwext-php72...
[15:20:39] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team, 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10hashar) I would guess the easiest is to add some kind of entry point in...
[15:20:56] <wikibugs>	 10Phabricator: Some users can't add the fr-tech private task form to their task selection menu. - https://phabricator.wikimedia.org/T229628 (10mmodell) My suggestion, if searching for `wmf-fr` or `-fr` still doesn't work, then try this:  1. Go to [[ /favorites/menu/configure/custom/ | favorites editor ]] 2. Clic...
[15:20:58] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10hashar)
[15:22:26] <wikibugs>	 (03PS1) 10Jforrester: layout: [MachineVision] Add Wikibase and WikibaseMediaInfo as phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/534175
[15:22:46] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [MachineVision] Add Wikibase and WikibaseMediaInfo as phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/534175 (owner: 10Jforrester)
[15:23:03] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10CPT Initiatives (API Integration Tests), 10Core Platform Team Workboards (Purple): Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10daniel) >>! In T230340#5461716, @hashar wrote: >...
[15:30:17] <wikibugs>	 (03PS1) 10Mholloway: MachineVision: Depend on WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/534176
[15:30:35] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [MachineVision] Add Wikibase and WikibaseMediaInfo as phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/534175 (owner: 10Jforrester)
[15:36:07] <wikibugs>	 (03PS1) 10Jforrester: layout: [mediawiki/services/kartotherian] Enable the publish pipeline now this works [integration/config] - 10https://gerrit.wikimedia.org/r/534178 (https://phabricator.wikimedia.org/T223275)
[15:36:09] <wikibugs>	 (03PS1) 10Jforrester: layout: [mediawiki/services/kartotherian] Drop the kartotherian-node10-browser-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/534179 (https://phabricator.wikimedia.org/T216826)
[15:36:11] <wikibugs>	 (03PS1) 10Jforrester: layout: Archive the kartotherian and tilerator deploy repos now they use the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/534180 (https://phabricator.wikimedia.org/T216826)
[15:36:13] <wikibugs>	 (03PS1) 10Jforrester: jjb: Drop kartotherian-deploy-npm-node-6-docker and kartotherian-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/534181 (https://phabricator.wikimedia.org/T216826)
[15:36:15] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: Drop node10-kartotherian and npm-test-maps-service [integration/config] - 10https://gerrit.wikimedia.org/r/534182 (https://phabricator.wikimedia.org/T216826)
[15:36:47] <James_F>	 !log Zuul: Add phan dependencies for MachineVision
[15:36:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:38:09] <wikibugs>	 (03CR) 10Jforrester: "Do" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/534176 (owner: 10Mholloway)
[15:39:52] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/services/kartotherian] Enable the publish pipeline now this works [integration/config] - 10https://gerrit.wikimedia.org/r/534178 (https://phabricator.wikimedia.org/T223275) (owner: 10Jforrester)
[15:44:12] <wikibugs>	 10Continuous-Integration-Config, 10Code-Health-Metrics, 10Patch-For-Review: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990 (10Jrbranaa) Added Popups and Mobile Frontend extensions to pipeline.
[15:55:27] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [mediawiki/services/kartotherian] Enable the publish pipeline now this works [integration/config] - 10https://gerrit.wikimedia.org/r/534178 (https://phabricator.wikimedia.org/T223275) (owner: 10Jforrester)
[15:57:24] <James_F>	 !log Zuul: Enable the publish pipeline for mediawiki/services/kartotherian  T223275
[15:57:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:57:27] <stashbot>	 T223275: Create blubberfile for deploying kartotherian into docker environment. - https://phabricator.wikimedia.org/T223275
[15:57:32] <wikibugs>	 (03Abandoned) 10Mholloway: MachineVision: Depend on WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/534176 (owner: 10Mholloway)
[16:00:14] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] "Not ready yet; waiting for the service to migrate." [integration/config] - 10https://gerrit.wikimedia.org/r/534179 (https://phabricator.wikimedia.org/T216826) (owner: 10Jforrester)
[16:01:24] <wikibugs>	 (03Abandoned) 10Jforrester: layout: [kartotherian] enable pipeline on gate [integration/config] - 10https://gerrit.wikimedia.org/r/534033 (https://phabricator.wikimedia.org/T223275) (owner: 10Mathew.onipe)
[16:04:02] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: Fix restart_php [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857)
[16:04:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend: Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10thcipriani)
[16:04:47] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) 05Resolved→03Open Or wmf/ branches: https://integration.wikimedia.org/ci/job/quibble-v...
[16:05:12] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) The recent changes to CI for the new pipelines are likely at fault, but too complicated to...
[16:06:04] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Fix restart_php [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857) (owner: 10Giuseppe Lavagetto)
[16:06:10] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Update PipelineLib Base Image to use WMF-registry image - https://phabricator.wikimedia.org/T230426 (10thcipriani) a:03thcipriani
[16:08:00] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10observability, 10serviceops-radar: Gerrit: Alert when # of active threads > some threshold - https://phabricator.wikimedia.org/T230138 (10thcipriani) 05Open→03Invalid >>! In T230138#5404114, @CDan...
[16:08:02] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani)
[16:08:42] <paladox>	 https://bugs.chromium.org/p/gerrit/issues/detail?id=11418 heh
[16:09:02] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Performance-Team (Radar): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10thcipriani)
[16:09:49] <wikibugs>	 (03PS3) 10Giuseppe Lavagetto: Fix restart_php [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857)
[16:11:03] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[16:11:39] <Reedy>	 James_F: Is there more stuff to investigate to fix the phan extension dependancies?
[16:14:05] <James_F>	 Reedy: I've spent an hour reading Antoine's code changes to work out what broke.
[16:14:11] <Reedy>	 :(
[16:14:35] <James_F>	 It's possible that it was already totally broken but we didn't notice because of a bug in the code he replaced.
[16:15:01] <James_F>	 It seems like the dependency injection is doing what it's configured to do some of the time.
[16:15:03] <James_F>	 But not others.
[16:16:07] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani)
[16:16:48] <wikibugs>	 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10thcipriani)
[16:17:02] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 2 others: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10thcipriani)
[16:17:26] <wikibugs>	 10Phabricator, 10Release-Engineering-Team-TODO, 10Security-Team, 10User-MModell: Create a security issue task type with additional attributes - https://phabricator.wikimedia.org/T204160 (10thcipriani)
[16:18:20] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Security-Team, 10User-MModell: Publish the source for phabricator-antivandalism - https://phabricator.wikimedia.org/T202080 (10thcipriani)
[16:18:39] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910 (10thcipriani)
[16:19:25] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts, 10Patch-For-Review: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10thcipriani)
[16:19:41] <wikibugs>	 10Release-Engineering-Team-TODO, 10Release Pipeline, 10CPT Initiatives (Extension Management (TEC13)), 10MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), 10Patch-For-Review: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 (...
[16:20:03] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10thcipriani)
[16:21:14] <wikibugs>	 10Deployments, 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10User-MModell: Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488 (10thcipriani)
[16:28:40] <hashar>	 o/
[16:28:42] <hashar>	 liw: zeljkof I am there!
[16:29:28] <hashar>	 so what is happening?
[16:29:44] <James_F>	 Zuul isn't injecting dependencies correctly.
[16:29:51] <James_F>	 Or, at least, differently.
[16:29:55] <liw>	 hashar, people on the team channel know more.. people like James_F 
[16:30:07] <James_F>	 wmf/ branches don't have more than half the dependencies.
[16:30:10] <James_F>	 E.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/17750/parameters/
[16:30:22] <hashar>	 ah so same issue you had with EducationProgram earlier today?
[16:31:01] <James_F>	 And phan jobs seem to be getting the "right" dependencies (i.e., they're getting what's in config), but it looks like previously they were getting both the phan-only dependencies and the general dependencies, so now they're breaking.
[16:31:02] <James_F>	 Yes.
[16:31:15] <hashar>	 hmm
[16:31:20] <hashar>	 that build had EXT_DEPENDENCIES set
[16:31:27] <James_F>	 Yes, but it's far too short.
[16:31:40] <James_F>	 It's missing all the other production extensions.
[16:32:11] <hashar>	 for the Graph extension no that looks correct
[16:32:22] <wikibugs>	 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10sbassett) Discussed with the #security-team this morning - we're fine making this task public as long as no sensitive issues from the related protected ta...
[16:32:25] <wikibugs>	 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10sbassett)
[16:32:34] <James_F>	 But for wmf/ don't we have all production extensions and skins as dependencies?
[16:32:51] <hashar>	 not really
[16:32:56] <James_F>	 Certainly, the job is looking for extensions we don't have in CI normally, like PageViewInfo
[16:33:05] <hashar>	 since bunch of them are not compatible with each other
[16:33:14] <James_F>	 OK, so where are the "Could not find the registration file" errors coming from?
[16:33:21] <hashar>	 so we only have a subset of them being tested together. That is the gated_extension list in zuul/parameter_functions.py
[16:33:29] <James_F>	 Yeah, that's for master.
[16:33:31] <hashar>	 and the wmf-quibble.* jobs 
[16:33:34] <James_F>	 I thought we did somehting special for wmf/
[16:33:35] <hashar>	 and wmf/
[16:34:10] <hashar>	 and in parallel, we also run the regular job with the dependencies defined in the crazy list in zuul/parameters_functsion.py
[16:34:21] <hashar>	 for that Graph extension, the dependencies are correct 
[16:34:25] <hashar>	 BUT
[16:34:38] <hashar>	 somehow something in mediawiki/core seems to now resolve the dependencies in extension.json
[16:34:44] <hashar>	 and enforce them to be fulfilled
[16:34:47] <hashar>	 which really is non sense
[16:34:57] <hashar>	 since we know that they are far from complete
[16:34:58] <James_F>	 Oh, yes, that's a new patch Daniel just merged.
[16:35:10] <hashar>	 the only reference we have for deps is in zuul/parameters_functions.py 
[16:35:10] <James_F>	 Is that the issue?
[16:35:14] <hashar>	 so yeah
[16:35:19] <hashar>	 what ever got changed got to be reverted
[16:35:31] <James_F>	 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/533239
[16:35:57] <hashar>	 ah yeah
[16:36:35] * James_F paints go-faster stripes on CI.
[16:36:47] <Krinkle>	 No quibble release recently. https://github.com/wikimedia/integration-config/commits/master
[16:36:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend: Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10Jdlrobson) Is this test running on minerva? Is the...
[16:36:58] <Krinkle>	 So I guess it's with the env variables or job config.
[16:37:06] <James_F>	 Krinkle: ^^
[16:37:08] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10Jdlrobson)
[16:37:13] <liw>	 James_F, adding "Must be red" to the new CI requirements list...
[16:37:23] * James_F grins at liw.
[16:37:48] <James_F>	 It'd be nice to have a way to do expedited-merge on a regular master patch (like this one), bumping everything out of the way.
[16:38:06] <James_F>	 But that's quite disruptive, so I'm not sure we'd want people to have access to that feature except in emergencies.
[16:38:09] <hashar>	 so essentially, got revert https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/533239/
[16:38:18] <James_F>	 Yes, https://gerrit.wikimedia.org/r/c/mediawiki/core/+/534195
[16:38:27] <hashar>	 ther eis a feature to promote a change ahaed of the gate-and-submit queue
[16:38:48] <hashar>	 contint1001$ zuul promote --pipeline gate-and-submit changeid,ps
[16:38:52] <hashar>	 should still work
[16:38:52] <James_F>	 Oh, interesting.
[16:39:00] <Krinkle>	 how come it passed CI though?
[16:39:00] <hashar>	 that move the given list of changes to the top of the queue
[16:39:05] <hashar>	 which reshuffle the queue
[16:39:12] <James_F>	 Krinkle: Because it works fine for things in the gate?
[16:39:16] <hashar>	 rebuild a queue of changes with the one promoted first
[16:39:19] <hashar>	 and retrigger the jobs
[16:39:28] <Krinkle>	 k
[16:39:43] <hashar>	 Krinkle: that change has no test? ;D
[16:40:16] <James_F>	 hashar: `zuul promote --pipeline gate-and-submit --changes 534195,1` failed with "Exception: Unable to find shared change queue for 534195,1"
[16:40:19] <hashar>	 I am not sure how the wmf-quibble* jobs managed to pass
[16:40:42] <James_F>	 As I said, I imagine that it happens to be true for all gated extensions.
[16:40:49] <hashar>	 James_F: because the given change is not in the gate-and-submit pipeline
[16:40:58] <James_F>	 hashar: But it is.
[16:41:15] <James_F>	 Or, at least, https://integration.wikimedia.org/zuul/ thinks it is.
[16:41:28] <Krinkle>	 > ERROR:zuul.Repo:Unable to initialize repo for https://gerrit.wikimedia.org/r/npm-test
[16:41:36] <Krinkle>	 hashar: that is also a new error on every build. not sure since when
[16:41:56] <James_F>	 Huh, never seen that before.
[16:42:07] <James_F>	 Krinkle: In quibble runs/
[16:42:15] <hashar>	 James_F: maybe it is without the patchset number :-\
[16:42:30] <James_F>	 hashar: Then I get "IndexError: list index out of range" instead.
[16:42:39] * James_F shrugs.
[16:42:48] <James_F>	 It's already being processed.
[16:42:52] <James_F>	 I'll just wait.
[16:43:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10mobrovac) 05Res...
[16:43:47] <hashar>	 grr
[16:44:57] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Zuul: zuul promote broken: - https://phabricator.wikimedia.org/T231913 (10hashar)
[16:45:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Zuul: zuul promote broken: Exception: Unable to find shared change queue for 534195,1 - https://phabricator.wikimedia.org/T231913 (10hashar)
[16:46:34] <liw>	 "grr" is often how I feel about CI, outside of WMF :)
[16:47:14] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Zuul: zuul promote broken: Exception: Unable to find shared change queue for 534195,1 - https://phabricator.wikimedia.org/T231913 (10hashar)
[16:47:24] <hashar>	 so yeah zuul promote is broken bah
[16:47:33] <hashar>	 liw: you are kind ahah
[16:49:06] <hashar>	 James_F: meanwhile having the cli installer to break is definitely something we want ;]
[16:49:25] <hashar>	 since right now it just silently ignores faulty extensions and keep processing
[16:49:38] <hashar>	 but I guess I can only step back into that once I am done with the train
[16:51:12] <James_F>	 Yeah, maybe re-do this patch but have it warn rather than error in CI?
[16:51:13] * James_F shrugs.
[16:52:30] <hashar>	 I cant remember what was the behavior previously
[16:52:43] <hashar>	 or maybe because multiple extensions now define dependencies
[16:52:49] <James_F>	 Possibly?
[16:53:02] <hashar>	 hmm
[16:53:04] <hashar>	 or no
[16:53:15] <hashar>	 so CI injects Graph + the dependencies listed via EXT_DEPENDENCIES
[16:53:24] <hashar>	 so the installer.php runs against a list of cloned repos
[16:53:34] <hashar>	 mediawiki installer process all the extension.json
[16:53:44] <hashar>	 creates a large list of dependencies
[16:54:07] <hashar>	 and fail because some (well actually a lot) repositories are defined in extension.json files   but are not cloned
[16:54:28] <hashar>	 so yeah that mediawiki/core would need to be synced with bunch of changes in CI and in the extension.json
[16:54:43] <James_F>	 Oh, so the problem is two fold: (1) we have a bunch of dependencies in extension.json that we don't feed in in CI, and (2) we have lots of implicit dependencies that we don't feed in in CI for phan?
[16:54:46] <hashar>	 so it was probably fine somehow
[16:54:55] <hashar>	 yeah there is phan already
[16:55:09] <James_F>	 Which is why sometimes patches would pass for everything except Phan.
[16:55:10] <hashar>	 which magically added another layer of centrally managed dependencies just when I was about to phase them out
[16:55:16] <James_F>	 :-(
[16:55:21] <hashar>	 eventually the world became a spaghetti soup
[16:55:34] <hashar>	 I have stepped upon a blocking mediawiki change and wen twith other adventures
[16:55:46] <James_F>	 For self-serve CI we need to make these items configured in-repo.
[16:55:53] <hashar>	 and ... nothing advanced which further reinforce my impostor syndrom
[16:55:53] <James_F>	 Which I guess means reading extension.json?
[16:56:08] <hashar>	 so yeah the rough .plan was roughly to stop injectin gEXT_DEPENDENCIES
[16:56:12] * James_F nods.
[16:56:20] <hashar>	 have Quibble to process extension.json and recursively clone the dependencies
[16:56:26] <hashar>	 the once complete
[16:56:29] <hashar>	 run the installer which would fail
[16:56:33] <hashar>	 if one is missing
[16:56:37] <James_F>	 Why did we not do that?
[16:56:42] <hashar>	 so the mediawiki/core patch is actually correct
[16:56:49] <James_F>	 Blocked on something happening in core?
[16:56:53] <hashar>	 but it can not be merged until we have validated the dependencies
[16:57:05] <hashar>	 and adjusted CI to drop EXT_DEPENDENCIES and switch to parsing extension.json files
[16:57:09] <James_F>	 Oh, right, you'd need to read extension.json which needs code in core to do that/
[16:57:16] <hashar>	 chicken and eggs I guess ;-\
[16:57:19] <James_F>	 Bootstrapping problem.
[16:57:19] <hashar>	 oh
[16:57:21] <hashar>	 no
[16:57:31] <hashar>	 I made Quibble to read the extension.json and do the recursive clone
[16:57:36] <James_F>	 Oh.
[16:57:39] <James_F>	 Then…?
[16:57:51] <hashar>	 though maybe the clone logic should be migrated from Quibble to Mediawiki (and thus ported from python to php hehe)
[16:58:19] <hashar>	 I should write all of that on whatever task I had filled before summer
[16:58:37] <James_F>	 Yes.
[17:00:27] * hashar writes on T225512
[17:00:59] * hashar blames himself
[17:02:03] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Graphs, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), and 3 others: 1.34.0-wmf.21 cause Graphoid service check to fail due to 403 from mediawiki.org - https://phabricator.wikimedia.org/T231894 (10Jdforrest...
[17:02:05] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10Jdforrester-WMF)
[17:02:58] <wikibugs>	 10Release-Engineering-Team-TODO (201908), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10Etonkovidova) >>! In T206358#5443280, @Niedzielski wrote: > @zeljkofilipin, I'm getting "Your trial period has ended. Upgrade now..." messages again :[ Can yo...
[17:04:47] <odder>	 So CI is indeed failing, was just about to ask
[17:08:30] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Performance-Team, and 3 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF)
[17:09:57] <wikibugs>	 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10Urbanecm) >>! In T229620#5462150, @sbassett wrote: > Discussed with the #security-team this morning - we're fine making this task public as long as no sen...
[17:12:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10hashar) So the m...
[17:12:57] <hashar>	 Jdlrobson: wrote a summary at https://phabricator.wikimedia.org/T225512#5462372
[17:13:20] <hashar>	 err wrong person
[17:13:29] <hashar>	 James_F:  wrote a summary at https://phabricator.wikimedia.org/T225512#5462372
[17:13:32] <hashar>	 and I am off for dinner etc
[17:13:35] <hashar>	 will do train again tonight
[17:28:20] <greg-g>	 James_F: is CI unblocked?
[17:28:33] <wikibugs>	 (03Abandoned) 10Thcipriani: php-fpm: sudo_check_call takes a string [tools/scap] - 10https://gerrit.wikimedia.org/r/532828 (https://phabricator.wikimedia.org/T224857) (owner: 10Thcipriani)
[17:28:45] <wikibugs>	 (03Abandoned) 10Thcipriani: php-fpm: restart as mwdeploy as root [tools/scap] - 10https://gerrit.wikimedia.org/r/532829 (owner: 10Thcipriani)
[17:28:47] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] Fix restart_php (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857) (owner: 10Giuseppe Lavagetto)
[17:30:58] <wikibugs>	 (03Merged) 10jenkins-bot: Fix restart_php [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857) (owner: 10Giuseppe Lavagetto)
[17:32:05] <wikibugs>	 10MediaWiki-Codesniffer, 10phan: PHPCS: Alllow long lines for phan annotations - https://phabricator.wikimedia.org/T231922 (10Daimona)
[17:32:07] <wikibugs>	 (03CR) 10jenkins-bot: Fix restart_php [tools/scap] - 10https://gerrit.wikimedia.org/r/533782 (https://phabricator.wikimedia.org/T224857) (owner: 10Giuseppe Lavagetto)
[17:35:26] <wikibugs>	 10MediaWiki-Codesniffer, 10phan: PHPCS: Allow long lines for phan annotations - https://phabricator.wikimedia.org/T231922 (10Nikerabbit)
[17:41:01] <wikibugs>	 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10sbassett) @Urbanecm - Task IDs and even protected Phabricator project //names// should be fine, as those aren't published or accessible for unprivileged P...
[17:46:35] <wikibugs>	 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): All CI failing with “Service disabled: DBLoadBalancerFactory” - https://phabricator.wikimedia.org/T231876 (10Jdforrester-WMF) 05Open→03Resolved OK, provisionally re-declaring as fixed. If an...
[17:49:52] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Graphs, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.34-notes (1.34.0-wmf.21; 2019-09-03), and 3 others: 1.34.0-wmf.21 cause Graphoid service check to fail due to 403 from mediawiki.org - https://phabricator.wikimedia.org/T231894 (10hashar)...
[17:55:23] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 353 bytes in 0.049 second response time
[17:56:13] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10greg)
[17:57:04] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review, 10ci-test-error: Selenium tests for Wikibase are being ran twice - https://phabricator.wikimedia.org/T231862 (10hashar) The change was for Wikibase and it get added again in EXT_DEPENDENCIES. For example Wikibase depends on WikibaseLexeme whic...
[18:04:59] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T220746 (10RhinosF1) >>! In T220746#5462448, @gerritbot wrote: > Change 534203 **merged** b...
[18:07:14] <wikibugs>	 (03PS1) 10Mholloway: [MachineVision] Add ULS as a Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/534211
[18:09:28] <wikibugs>	 (03CR) 10Jforrester: "This should be an implicit dependency of Wikibase, but happy to add explicitly." [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[18:11:01] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[18:12:00] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [MachineVision] Add ULS as a Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[18:13:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 4 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10daniel) Perhaps...
[18:13:56] <wikibugs>	 (03CR) 10Mholloway: "Oh... wait a minute, this is still failing with "Service disabled: DBLoadBalancerFactory."  I'd only seen some output immediately above ab" [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[18:17:03] <subbu>	 so beta cluster config in the mediawiki-config repo uses the "labs" identifier, right?
[18:17:27] <subbu>	 just checking labs is not something else before i start poking at it for enabling parsoid/php there.
[18:18:17] <subbu>	 answer seems to be yes. "    case 'labs': // wmflabs beta cluster         "
[18:18:19] * subbu continues
[18:27:09] <wikibugs>	 (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[18:27:17] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10hashar) >>! In...
[18:31:06] <wikibugs>	 (03CR) 10Mholloway: "> OK, have done a recheck, if that comes back I'll abandon." [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[18:36:24] <wikibugs>	 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10tramm) >>! In T209354#5458965, @Aklapper wrote: > Not sure for which exact reasons @Krinkle renamed `#Need-volunteer` to `#patch-welcome`. I think I'd prefer to stick to the first phrase.  >  > This would a...
[18:46:01] <wikibugs>	 (03PS12) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862)
[18:46:03] <wikibugs>	 (03PS3) 10Awight: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862)
[18:46:05] <wikibugs>	 (03PS1) 10Awight: Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219
[18:51:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[18:52:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10Jdlrobson) Look...
[18:52:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[18:52:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219 (owner: 10Awight)
[18:53:05] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Maps (Kartotherian): CI service-pipeline-test-and-publish job assumes blubber config has a single production image - https://phabricator.wikimedia.org/T231928 (10Jdforrester-WMF)
[18:55:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10thcipriani)
[18:58:18] <wikibugs>	 (03PS2) 10Awight: Fix default logdir [integration/quibble] - 10https://gerrit.wikimedia.org/r/534219
[19:01:14] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5462415, @gerritbot wrote: > Change 533782 **merged**...
[19:01:18] <wikibugs>	 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Krinkle) >>! In T209354#5458965, @Aklapper wrote: > Not sure for which exact reasons @Krinkle renamed `#Need-volunteer` to `#patch-welcome`. I think I'd prefer to stick to the first phrase.  I renamed it be...
[19:12:44] <wikibugs>	 (03PS2) 10Jforrester: layout: [CirrusSearch] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533889 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[19:13:33] <wikibugs>	 (03PS13) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862)
[19:13:35] <wikibugs>	 (03PS4) 10Awight: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862)
[19:14:12] <wikibugs>	 (03PS3) 10Jforrester: layout: [ORES] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533866 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[19:14:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[19:14:38] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) (owner: 10Awight)
[19:14:52] <wikibugs>	 (03Abandoned) 10Jforrester: [MachineVision] Add ULS as a Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/534211 (owner: 10Mholloway)
[19:22:19] <wikibugs>	 (03PS5) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[19:22:35] <wikibugs>	 (03CR) 10Jforrester: "PS5: Manual rebase." [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[19:26:32] <wikibugs>	 (03PS1) 10Jforrester: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223
[19:29:49] <greg-g>	 subbu: re labs/beta, yeah, "naming is hard" :)
[19:34:08] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[19:36:34] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester)
[19:36:53] <subbu>	 greg-g, ok :)
[19:56:36] <wikibugs>	 (03PS6) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[19:57:26] <wikibugs>	 (03CR) 10Krinkle: Switch prio test/test-wmf job from hhvm to php72 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[20:11:03] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[20:17:09] <wikibugs>	 (03PS7) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[20:17:11] <wikibugs>	 (03PS2) 10Jforrester: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223
[20:17:13] <wikibugs>	 (03PS1) 10Jforrester: layout: Drop duplicate pipeline definitions in extension-quibble-composer-nohhvm [integration/config] - 10https://gerrit.wikimedia.org/r/534226
[20:17:15] <wikibugs>	 (03PS1) 10Jforrester: layout: [mediawiki/php/luasandbox] Drop cooment about the long-dead check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/534227
[20:17:17] <wikibugs>	 (03PS1) 10Jforrester: layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228
[20:17:19] <wikibugs>	 (03PS1) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229
[20:17:37] <wikibugs>	 (03CR) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[20:18:26] <wikibugs>	 (03CR) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[20:19:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (owner: 10Jforrester)
[20:19:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester)
[20:19:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228 (owner: 10Jforrester)
[20:20:27] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "> FAILURE No layout difference" [integration/config] - 10https://gerrit.wikimedia.org/r/534226 (owner: 10Jforrester)
[20:20:34] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/php/luasandbox] Drop cooment about the long-dead check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/534227 (owner: 10Jforrester)
[20:27:22] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Drop duplicate pipeline definitions in extension-quibble-composer-nohhvm [integration/config] - 10https://gerrit.wikimedia.org/r/534226 (owner: 10Jforrester)
[20:27:25] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [mediawiki/php/luasandbox] Drop cooment about the long-dead check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/534227 (owner: 10Jforrester)
[20:28:43] <wikibugs>	 (03PS8) 10Jforrester: Switch prio test/test-wmf job from hhvm to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[20:28:45] <wikibugs>	 (03PS1) 10Jforrester: layout: Suspend trigger-mediawiki-pipeline-dev for now [integration/config] - 10https://gerrit.wikimedia.org/r/534231 (https://phabricator.wikimedia.org/T218360)
[20:28:47] <wikibugs>	 (03PS1) 10Jforrester: layout: Ensure all php: job sets test all five flavours [integration/config] - 10https://gerrit.wikimedia.org/r/534232
[20:28:58] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Suspend trigger-mediawiki-pipeline-dev for now [integration/config] - 10https://gerrit.wikimedia.org/r/534231 (https://phabricator.wikimedia.org/T218360) (owner: 10Jforrester)
[20:29:32] <wikibugs>	 (03CR) 10Jforrester: "Given that `check php` is very rarely used, it seems simpler to keep this as all-five." [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester)
[20:32:42] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Suspend trigger-mediawiki-pipeline-dev for now [integration/config] - 10https://gerrit.wikimedia.org/r/534231 (https://phabricator.wikimedia.org/T218360) (owner: 10Jforrester)
[20:47:24] <James_F>	 !log Zuul: Suspend trigger-mediawiki-pipeline-dev for now T218360
[20:47:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:47:28] <stashbot>	 T218360: Define a .pipeline/blubber.yaml for mediawiki/core - https://phabricator.wikimedia.org/T218360
[20:53:12] <wikibugs>	 (03PS3) 10Jforrester: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223
[20:54:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223 (owner: 10Jforrester)
[20:56:44] <wikibugs>	 (03CR) 10Hashar: "I have done too much copy pasting ;-]  Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/534226 (owner: 10Jforrester)
[20:56:57] <wikibugs>	 (03PS4) 10Jforrester: layout: Change default php flavour for everything from php70 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/534223
[20:57:23] <wikibugs>	 (03Abandoned) 10Jforrester: [WIP] layout: Skip PHP70 and PHP71 jobs for MW master [integration/config] - 10https://gerrit.wikimedia.org/r/516838 (https://phabricator.wikimedia.org/T225628) (owner: 10Jforrester)
[20:57:28] <wikibugs>	 (03Abandoned) 10Jforrester: layout: Replace negative branch matches of quibble jobs with positive ones [integration/config] - 10https://gerrit.wikimedia.org/r/518088 (owner: 10Jforrester)
[20:59:44] <wikibugs>	 (03Abandoned) 10Jforrester: tests: Expand tests to explicitly cover each branch we care about [integration/config] - 10https://gerrit.wikimedia.org/r/518179 (owner: 10Jforrester)
[21:00:17] <wikibugs>	 (03Abandoned) 10Jforrester: layout: Use positive branch matches for quibble, and explicitly list matched jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533546 (owner: 10Jforrester)
[21:01:04] <wikibugs>	 (03Abandoned) 10Jforrester: layout: [chromium-render] Enable pipeline tests [integration/config] - 10https://gerrit.wikimedia.org/r/520317 (https://phabricator.wikimedia.org/T225680) (owner: 10Jforrester)
[21:03:49] <wikibugs>	 (03PS2) 10Jforrester: layout: Define pipelines for REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534228
[21:03:51] <wikibugs>	 (03CR) 10Krinkle: "it doesn't need to rerun the main ones from 'recheck'/'test' though, that seems wasteful. The idea is usually that we optimise 'test' by e" [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester)
[21:04:34] <wikibugs>	 (03CR) 10Krinkle: Switch prio test/test-wmf job from hhvm to php72 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/533973 (owner: 10Krinkle)
[21:16:16] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add custom policy to access denied message - https://phabricator.wikimedia.org/T231734 (10epriestley) See also T211498.  In the use case described in T211498, the viewer has permission to view (and possibly edit) the task, so there's obviously no (security/policy) reason t...
[21:20:17] <wikibugs>	 (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/534232 (owner: 10Jforrester)
[21:30:31] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [CirrusSearch] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533889 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[21:30:40] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [ORES] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533866 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[21:34:49] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [CirrusSearch] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533889 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[21:35:14] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [ORES] Switch from coverage to codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/533866 (https://phabricator.wikimedia.org/T224990) (owner: 10Kosta Harlan)
[21:59:38] <James_F>	 !log Zuul: Move ORES and CirrusSearch to codehealth T224990
[21:59:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:59:41] <stashbot>	 T224990: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990
[22:07:49] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add custom policy to access denied message - https://phabricator.wikimedia.org/T231734 (10Urbanecm) >>! In T231734#5463224, @epriestley wrote: > Are there specific cases where knowing policy details changes how you interact with objects? My expectation is that no matter wh...
[22:11:02] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[22:14:38] <wikibugs>	 10Release-Engineering-Team-TODO, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work), 10Wiki-Setup (Delete / Redirect): Close and delete TestCommons from production by the end of December 2019 - https://phabricator.wikimedia.org/T213295 (10Jdforrester-WMF)
[22:17:00] <wikibugs>	 10Release-Engineering-Team-TODO, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work), 10Wiki-Setup (Delete / Redirect): Close and delete TestCommons from production by the end of December 2019 - https://phabricator.wikimedia.org/T213295 (10greg) >>! In T213295#5463390, @Abit wrote: > An...
[22:22:16] <wikibugs>	 (03PS2) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229
[22:24:37] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (owner: 10Jforrester)
[22:40:24] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add custom policy to access denied message - https://phabricator.wikimedia.org/T231734 (10epriestley) > You kinda answered your question. I can ask "someone", but who is that someone?  My expectation is that you usually learn about a task in some context which you can use...
[22:42:29] <wikibugs>	 (03PS3) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229
[22:46:50] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229 (owner: 10Jforrester)
[22:49:18] <wikibugs>	 (03PS4) 10Jforrester: [DNM] layout: Drop php70 and php71 from quibble jobs for master and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/534229
[22:55:45] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%)
[23:36:10] <wikibugs>	 10Gerrit, 10service-runner: Gerrit repository mediawiki/services/service-runner appears abandoned - https://phabricator.wikimedia.org/T231953 (10colewhite)