[00:05:49] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10RazeSoldier) Abo... [00:47:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Krinkle) 05Reso... [00:47:46] 10Continuous-Integration-Config: A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Krinkle) [00:47:50] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Krinkle) [00:48:31] 10Continuous-Integration-Config: A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Krinkle) The issue with extension dependencies in Phab jobs is: T225512. The service error is: T229601. [00:49:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Krinkle) 05duplicate→03Open Actually, let's track the CI issue separately here. [00:50:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 5 others: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 (10Krinkle) 05Open... [00:50:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Krinkle) p:05Triage→03Unbreak! [00:51:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): A dependency error in CI tests - https://phabricator.wikimedia.org/T232413 (10Krinkle) Seems to be caused by {T225512} >>! In T225512#5476516, @gerritbot wrote:... [01:12:21] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Krinkle) @Jdforrester-WMF Do you want to try incorporat... [02:29:24] (03CR) 10Krinkle: Civi: bump php version for extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518742 (owner: 10Ejegg) [06:51:00] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Documentation, 10User-MModell: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10greg) [06:51:29] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work), 10Wiki-Setup (Delete / Redirect): Close and delete TestCommons from production by the end of December 2019 - https://phabricator.wikimedia.org/T213295... [07:38:57] 10Project-Admins: tag phab task once it is deployed - https://phabricator.wikimedia.org/T228960 (10Aklapper) a:03KartikMistry Temporarily reassigning to @KartikMistry; feel free to remove assignee again (via the {nav name=Add Action... > Assign / Claim} dropdown) once answered - thanks! [07:39:29] 10Project-Admins: Create generic "Android" tag for tagging issues somehow related to the Android mobile OS - https://phabricator.wikimedia.org/T181306 (10Aklapper) @ashley: ping [07:43:30] 10Project-Admins, 10Community-Tech: Create a Phabricator project tag for "Commons deletion notification bot" - https://phabricator.wikimedia.org/T229759 (10Aklapper) Could I get input from #Community-Tech here? Thanks! [07:51:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing the extensio... - https://phabricator.wikimedia.org/T232413 [07:53:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing the extensio... - https://phabricator.wikimedia.org/T232413 [08:01:28] (03PS1) 10Hashar: phan: stop installing MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/535507 (https://phabricator.wikimedia.org/T232413) [08:03:05] !log Updated Phan jobs for T232413 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/535507 phan: stop installing MediaWiki [08:03:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:03:08] T232413: mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing the extension "MinervaNeue": Could not find the registration file for the extension "MobileFrontend" - https://phabricator.wikimedia.org/T232413 [08:03:15] * hashar crosses fingers [08:06:14] seems to work [08:27:49] yay, `--skip-install` has found another use case :) [08:28:07] hashar: Hi, I have a minor question about quibble. `setup_environment` has logic to override WORKSPACE with the commandline --workspace, but no CI jobs ever pass that parameter to quibble. Is this intentional, or can I fix? [08:28:32] And, should we be expecting os.environ['WORKSPACE'] to be correct in docker containers? [08:35:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while i... - https://phabricator.wikimedia.org/T232413 [08:35:21] ah looks higher priority. No rush! [08:35:56] kostajh: yeah!!! And i actually looked at switching to just use zuul-cloner to clone the git repositories [08:36:06] kostajh: but we also need to install composer and have the composer merge plugin etc [08:36:13] kostajh: so yeah --skip-install definitely has a huge case :] [08:36:32] awight: yeah hmm it is a bit messy [08:36:56] awight: the log directory / workspace directory are determined via some faulty logic [08:37:10] so that if you run it directly on your machine I think it defaults to: [08:37:15] workspace: ./workspace/ [08:37:19] log: ./workspace/log [08:37:44] but when being run under Docker (as detected by the presence of a /.dockerenv file, the logic is slightly changed [08:37:49] so that workspace: /workspace [08:37:52] and log: /log [08:37:53] I think [08:38:00] hashar: thanks! I'll mop soon but can work around for the moment... [08:38:09] the WORKSPACE environment variable is set by Jenkins itself [08:38:32] but that point to the path on the jenkins instance. Something like /srv/jenkins-workspace/workspace/name_of_the_job there [08:38:39] which is not a valid path in the container [08:38:57] so somewhere there is something to ignore WORKSPACE when using Docker [08:39:16] I also might have changed my mind half way as to where the log should be [08:39:59] I would like to review the few Quibble changes that are pending, but I get some CI migration stuff to conduct [08:40:05] and .. the mediawiki train :-\ [08:40:33] aah got it. Yeah my quibble patches are in shambles, anything V-1 isn't worth reviewing. [08:40:56] Is the LOG_DIR env variable something we should use? [08:40:58] (03CR) 10Hashar: [C: 03+2] "That apparently fixed it properly :]" [integration/config] - 10https://gerrit.wikimedia.org/r/535507 (https://phabricator.wikimedia.org/T232413) (owner: 10Hashar) [08:41:45] awight: the old scripts used MW_LOG_DIR / LOG_DIR [08:42:14] and both are referred to by some mediawiki/core test framework: [08:42:22] includes/DevelopmentSettings.php:$logDir = getenv( 'MW_LOG_DIR' ); [08:42:22] tests/selenium/wdio.conf.js: logPath = process.env.LOG_DIR || path.join( __dirname, '/log' ); [08:42:27] so [08:42:29] quibble --log [08:42:33] sets both environment variables [08:42:47] so that phpunit and wdio.conf.js write their stuff to the specified log dir [08:42:57] ty, that helps me [08:43:02] when running Quibble locally (without docker), that conveniently defaults to ./workspace/log/ [08:43:20] but when running in a Docker container, the workspace and log directories are different volume mounts [08:43:24] eg /workspace and /log [08:43:54] (I am not entirely sure, maybe we just mount ./src ) [08:44:02] twist [08:44:23] twist: if you run quibble --help , it should shows the default value for each of --workspace and --logdir [08:44:40] and the default should change depending on the existence of /.dockerenv [08:44:45] (03Merged) 10jenkins-bot: phan: stop installing MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/535507 (https://phabricator.wikimedia.org/T232413) (owner: 10Hashar) [08:45:00] tldr: that is messy and inconvenient :-\ [08:45:02] This only came up because I got confused when writing some tests. Quibble's behavior when running as the test framework is different than when quibble itself is under test. [08:45:41] which difference? [08:45:58] https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/520240/ [08:46:24] I see what's happening now, just trying to finesse in a minimally egregious way. [08:46:42] also the tests on CI run in a Docker container [08:46:42] 10Release-Engineering-Team-TODO (201909), 10Browser-Tests: selenium-daily-beta-MediaWiki possibly broken by password changes - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin) a:03zeljkofilipin [08:46:51] so some code might end up considering they are under docker [08:46:52] 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-MediaWiki possibly broken by password changes - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin) [08:46:54] well they are [08:47:21] I think I had to mock the quibble.is_in_docker() calls just for that [08:47:35] 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-MediaWiki possibly broken by password changes - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin) p:05Triage→03Normal [08:47:48] so that tests pass both on CI (which are executed under Docker) and on a local install (I dont use a docker container but my machine env) [08:48:46] :) that's simple, I'll borrow that approach. [08:51:13] (03PS14) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (https://phabricator.wikimedia.org/T231862) [08:52:19] (03PS5) 10Awight: Deduplicate projects [integration/quibble] - 10https://gerrit.wikimedia.org/r/534134 (https://phabricator.wikimedia.org/T231862) [09:33:00] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10LucasWerkmeister) [09:33:28] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) @Krinkle: Thanks for taking the time to explain. That reasoning makes sense; I agree that it sounded submissive (though `#need-volunteer` also confuses me as every task requires someone who volunt... [09:38:10] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10Aklapper) @AndyRussG: Has this happened again? [09:40:29] 10Beta-Cluster-Infrastructure, 10User-Rxy: Interface admin rights request for meta.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T232341 (10Rxy) 05Open→03Resolved a:03Rxy ✓ **[Done](https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:Redirect/logid/26821)** [11:52:13] 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10zeljkofilipin) a:05jbond→03Jrbranaa The list is public. @Jrbranaa looks like you have to make it private: https://lists.wikim... [11:54:33] 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10zeljkofilipin) p:05Triage→03Normal [11:57:21] !log Gerrit: deleting mediawiki/services/service-template-node and mediawiki/services/service-runner (unused) # T231953 [11:57:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:57:24] T231953: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 [11:58:17] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, and 3 others: Gerrit repositories mediawiki/services/service-runner and mediawiki/services/service-template-node appear abandoned - https://phabricator.wikimedia.org/T231953 (10hashar) 05Open→03Resol... [11:59:06] 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10jbond) @zeljkofilipin I have [now] set the subscription model to `Require approval`. I leave it to the admins to change the oth... [12:24:16] 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10zeljkofilipin) 05Open→03Resolved Thanks! I do remember now that that's how other lists work, there's just one password. [12:25:56] 10Release-Engineering-Team-TODO (201909), 10Operations, 10Wikimedia-Mailing-lists: Please create private "testeng" team mailing list - https://phabricator.wikimedia.org/T232178 (10zeljkofilipin) 05Resolved→03Open Oops, looks like I've resolved this to quickly. List archives are still public: https://lis... [12:32:02] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10Jenkins: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar) The zuul layout validation has logging set at DEBUG level explicitly. Lame a... [12:35:14] (03PS1) 10Hashar: Filter out debug spam when validating zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) [12:35:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Jenkins, 10Patch-For-Review: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar) [12:39:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Jenkins, 10Patch-For-Review: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar)... [12:39:30] (03CR) 10Hashar: "That should make the job integration-config-zuul-layout-validate-docker faster (it triggers on +2)" [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) (owner: 10Hashar) [12:45:56] Project beta-code-update-eqiad build #263148: 04FAILURE in 2 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263148/ [12:50:31] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [13:04:47] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10fgiunchedi) Happened again today apparently, gerrit's been down/unresponsive and its monitoring data has been unavailable since ~12.43 [13:09:04] Project mediawiki-core-doxygen-docker build #9748: 04FAILURE in 5 min 0 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9748/ [13:12:40] Project beta-code-update-eqiad build #263149: 04STILL FAILING in 19 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263149/ [13:14:10] Project beta-code-update-eqiad build #263150: 04STILL FAILING in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263150/ [13:15:33] Project beta-code-update-eqiad build #263151: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263151/ [13:15:33] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26247 bytes in 0.064 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [13:17:03] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani) Restarted Gerrit. {F30290022} 42 second(!!!) GC pause at 12:47:28 means the service likely wouldn't have recovered. [13:21:02] (03PS1) 10Daimona Eaytoy: build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:21:53] (03PS2) 10Daimona Eaytoy: build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:22:19] (03CR) 10jerkins-bot: [V: 04-1] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:22:29] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10hashar) On cobalt from `lsof -p $(pidof java)` (F30290036), I see a LOT of pack files opened: ` COMMAND PID USER FD TYPE DEVICE SIZE/OFF... [13:24:24] Yippee, build fixed! [13:24:25] Project beta-code-update-eqiad build #263152: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/263152/ [13:26:28] (03PS1) 10Hashar: Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) [13:27:09] (03Abandoned) 10Hashar: Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535571 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:27:20] (03CR) 10jerkins-bot: [V: 04-1] Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:29:02] (03PS3) 10Daimona Eaytoy: [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:29:42] (03CR) 10jerkins-bot: [V: 04-1] [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:36:33] (03PS2) 10Hashar: Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) [13:37:18] (03PS4) 10Daimona Eaytoy: [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:37:45] (03CR) 10jerkins-bot: [V: 04-1] [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:39:29] 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Release-Tools, 10Patch-For-Review: PHP CodeSniffer on mediawiki/tools/release warns when running with php 7.3 - https://phabricator.wikimedia.org/T232473 (10hashar) a:03hashar Doing the upgrade one version after the other so we get small changes that ar... [13:40:55] (03PS5) 10Daimona Eaytoy: [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:41:38] (03CR) 10jerkins-bot: [V: 04-1] [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:43:17] 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Release-Tools, 10Patch-For-Review: PHP CodeSniffer on mediawiki/tools/release warns when running with php 7.3 - https://phabricator.wikimedia.org/T232473 (10Daimona) >>! In T232473#5478988, @hashar wrote: > Doing the upgrade one version after the other so... [13:44:23] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [13:44:40] (03CR) 10Daimona Eaytoy: Upgrade mediawiki-codesniffer to 0.8.0 (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:44:42] (03CR) 10Daimona Eaytoy: [C: 03+2] Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:45:17] (03Merged) 10jenkins-bot: Upgrade mediawiki-codesniffer to 0.8.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:48:23] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [13:48:54] (03PS6) 10Daimona Eaytoy: [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:49:28] (03CR) 10jerkins-bot: [V: 04-1] [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:50:05] (03PS7) 10Daimona Eaytoy: [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) [13:50:50] (03CR) 10jerkins-bot: [V: 04-1] [WIP] build: Update dev dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [13:52:28] (03PS1) 10Hashar: Upgrade mediawiki-codesniffer to 19.1.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535595 (https://phabricator.wikimedia.org/T232473) [13:53:10] (03CR) 10jerkins-bot: [V: 04-1] Upgrade mediawiki-codesniffer to 19.1.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535595 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:53:42] (03CR) 10Daimona Eaytoy: [C: 04-1] Upgrade mediawiki-codesniffer to 19.1.0 (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/535595 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [13:55:40] (03PS1) 10Daimona Eaytoy: Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 [13:56:20] (03CR) 10jerkins-bot: [V: 04-1] Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 (owner: 10Daimona Eaytoy) [13:56:31] 10Phabricator: Not able to edit description of T218184! - https://phabricator.wikimedia.org/T232478 (10KartikMistry) [13:57:27] (03PS2) 10Daimona Eaytoy: Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 [13:57:52] (03CR) 10jerkins-bot: [V: 04-1] Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 (owner: 10Daimona Eaytoy) [14:00:04] 10Release-Engineering-Team-TODO (201909), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Aklapper) Regarding https://wikimedia.biterg.io (see https://www.mediawiki.org/wiki/Community_metrics for docs) and Gerrit only, not Github: * Submitters of patches: We h... [14:07:32] (03PS3) 10Daimona Eaytoy: Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 [14:07:49] (03PS2) 10Hashar: Upgrade mediawiki-codesniffer to 19.1.0 [tools/release] - 10https://gerrit.wikimedia.org/r/535595 (https://phabricator.wikimedia.org/T232473) [14:08:57] (03CR) 10jerkins-bot: [V: 04-1] Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 (owner: 10Daimona Eaytoy) [14:09:50] (03Abandoned) 10Daimona Eaytoy: Upgrade PHPUnit [tools/release] - 10https://gerrit.wikimedia.org/r/535597 (owner: 10Daimona Eaytoy) [14:11:46] (03CR) 10Daimona Eaytoy: [C: 04-1] Upgrade mediawiki-codesniffer to 19.1.0 (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/535595 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [14:12:57] Yippee, build fixed! [14:12:57] Project mediawiki-core-doxygen-docker build #9749: 09FIXED in 8 min 53 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/9749/ [14:17:29] 10Phabricator, 10Code-Review-Workgroup: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10mmodell) [14:22:35] (03CR) 10Hashar: "Thank you for the quick review :]" (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [14:25:12] (03CR) 10Daimona Eaytoy: [C: 03+2] Upgrade mediawiki-codesniffer to 0.8.0 (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/535590 (https://phabricator.wikimedia.org/T232473) (owner: 10Hashar) [14:41:14] hashar: "Unfortunately, CI does not magically clone the requirements yet, so you would have to also add them to the CI config" <-- is there a phab task tracking this? [14:42:51] kostajh: the magic clone? [14:42:58] Ja [14:43:14] 15aa235243490328baa26167d9d4c1eb735eb85d in quibble [14:43:20] which points to T193824 [14:43:22] T193824: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824 [14:43:34] which is probably the wrong task [14:43:36] so I guess hmm [14:43:37] no :-\ [14:43:50] I should create a task yeah [14:44:09] 10Release-Engineering-Team-TODO (201909), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) I've copied the above into https://docs.google.com/document/d/128RYZ1EWuePEKXLcpotfh5E5jqPLKPlvhQfQzpqszdA/edit?usp=sharing for ease of commenting/sugges... [14:44:22] oh [14:44:33] So I can use --resolve-requires now [14:44:34] and I guess I will as soon as I start working on the migration [14:44:35] cause [14:44:36] hmm [14:44:47] I don't have a good plan yet, I just did a bunch of additions to quibble [14:44:52] had them releasd / deployed [14:44:55] but: there is no doc [14:44:58] it is barely tested [14:45:00] hashar: context is https://www.mediawiki.org/wiki/Topic:V72cbl5cw6fhx5aw [14:45:02] I don't know how to migrate yet [14:45:09] there will be earthquake in the process [14:45:13] In this patch I stopped cloning all dependencies, only the extension https://gerrit.wikimedia.org/r/c/integration/config/+/531600 [14:45:19] and I would like to find a way to do the transition smoothly [14:45:29] But e.g. CirrusSearch depends on Elastica, and so we need to clone that repo too for codehealth jobs [14:46:03] yu [14:46:05] p [14:46:18] So I think I can just pass --resolve-requires for that mwext-codehealth-patch job [14:46:36] I am unsure really [14:46:39] but in theory yeah [14:46:47] if the requirement is expressed in extension.json [14:47:49] I'll test it out [14:52:31] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani) >>! In T231872#5478912, @hashar wrote: > On cobalt from `lsof -p $(pidof java)` (F30290036), I see a LOT of pack files opened: It's pretty typical t... [14:53:06] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani) p:05Triage→03Normal [14:57:47] 10Phabricator: Update phabricator to use Authentication Only consumer - https://phabricator.wikimedia.org/T99750 (10mmodell) p:05Normal→03High a:03mmodell Not sure why this fell through the cracks but we (or rather, I) should do this.... [15:11:03] PROBLEM - Host contint1001.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [15:12:08] what [15:16:45] RECOVERY - Host contint1001.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.14 ms [15:28:46] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Update phabricator to use Authentication Only consumer - https://phabricator.wikimedia.org/T99750 (10greg) [15:30:09] Hmm, resolve requires didn't seem to work [15:39:46] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10Reedy) [15:48:00] hey, I have a really strange problem with ServiceWirings and our deployment process: https://phabricator.wikimedia.org/T231486 [15:48:51] something like that fails only during deployment (syncing Minervaneueu/includes/ServiceWirings.php) file. We modified some services twice, and it failed two times [15:49:43] we got some pretty big spikes, and after couple minutes the error went away. Is it possible to have some cache invalidation issues? [15:55:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10dcausse) [16:13:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:27:23] 10Phabricator: Not able to edit description of T218184! - https://phabricator.wikimedia.org/T232478 (10Aklapper) 05Open→03Stalled Quoting `@Tgr` from T108449#1839934: > `ERR_SPDY_PROTOCOL_ERROR` is a very generic error message that basically means something went wrong with the communication with the web serv... [16:29:35] 10Phabricator: Not able to edit description of T218184: ERR_SPDY_PROTOCOL_ERROR - https://phabricator.wikimedia.org/T232478 (10Aklapper) [16:45:51] (03CR) 10Jforrester: [C: 03+1] Filter out debug spam when validating zuul layout [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) (owner: 10Hashar) [16:47:11] (03CR) 10Hashar: "Tested it locally by adding a non existent pipeline. Eg:" [integration/config] - 10https://gerrit.wikimedia.org/r/535566 (https://phabricator.wikimedia.org/T232287) (owner: 10Hashar) [16:48:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing the extensio... - https://phabricator.wikimedia.org/T232413 [17:01:49] 10Project-Admins: Wiki Club West project - https://phabricator.wikimedia.org/T231416 (10Aklapper) @JarrahTree: Please provide a project tag description. [17:05:40] 10Project-Admins: Wiki Club West project - https://phabricator.wikimedia.org/T231416 (10Reedy) [17:24:14] 10Phabricator, 10Release-Engineering-Team-TODO (201908), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) Phatality is now deployed in beta, as can be seen: https://logstash-beta.wmflabs.org/app/kibana#/dashboard/Fatal... [17:25:33] (03CR) 1020after4: "The latest changeset has been deployed in beta and it seems to be working correctly." [releng/phatality] - 10https://gerrit.wikimedia.org/r/531047 (https://phabricator.wikimedia.org/T230752) (owner: 1020after4) [17:47:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5477481, @Krinkle wrote... [18:02:55] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.67% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:13:20] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:37:42] 10Deployments, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-LocalisationUpdate, and 2 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10WDoranWMF) [18:38:04] (03CR) 10Umherirrender: "I would replace it to have only one annotation." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/535241 (https://phabricator.wikimedia.org/T232256) (owner: 10Daimona Eaytoy) [18:45:46] (03PS1) 10Kosta Harlan: Revert "jjb: [mwext-codehealth*] Only clone ZUUL_PROJECT" [integration/config] - 10https://gerrit.wikimedia.org/r/535668 [18:47:29] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10TechCom-RFC, and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10WDoranWMF) [18:47:47] (03PS2) 10Kosta Harlan: Revert "jjb: [mwext-codehealth*] Only clone ZUUL_PROJECT" [integration/config] - 10https://gerrit.wikimedia.org/r/535668 [19:02:46] 10Phabricator: Lost 2F auth for Amitie_10g - https://phabricator.wikimedia.org/T232518 (10Amitie_10g) [19:06:08] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10greg) [19:12:25] Hey folks. I'm trying to rewrite the last couple of commits to a gerrit repo because some files ended up outside of LFS. [19:12:30] I am getting "remote: error: commit 2d0f95b: email address acraze@wikimedia.org is not registered in your account, and you lack 'forge committer' permission." [19:12:38] Any chance y'all can help me work around that? [19:13:40] Relevant repo: /research/ores/wheels [19:34:08] 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10JTannerWMF) >>! In T230831#5433325, @Aklapper wrote: > Alright, my understanding is that we go for subprojects. Correct me if I misunder... [19:38:22] 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10mmodell) a:05JTannerWMF→03Aklapper I can also help with the server side of this but Andre asked to assign to him so I'm doing so now. [20:01:03] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:02:04] Project mwcore-phpunit-coverage-master build #165: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/165/ [20:13:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:59:21] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.15% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:23:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), 10MW-1.32-release, and 3 others: MediaWiki web installer do not show extension when their dependency is... - https://phabricator.wikimedia.org/T220514 [21:29:16] 10Beta-Cluster-Infrastructure, 10Parsoid-PHP, 10RESTBase, 10Core Platform Team Workboards (Purple): Make the parsoid server on the beta cluster a mediawiki app server - https://phabricator.wikimedia.org/T232538 (10ssastry) p:05Triage→03High [21:31:39] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:40:23] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [21:43:43] (03PS1) 10Jforrester: jjb: Drop operations-mw-config-php70-composer-lint-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535714 [21:43:45] (03PS1) 10Jforrester: jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 [21:43:47] (03PS1) 10Jforrester: layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716 [21:43:49] (03PS1) 10Jforrester: layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717 [21:43:51] (03PS1) 10Jforrester: jjb: Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535718 [21:43:57] (03CR) 10Jforrester: [C: 04-1] layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717 (owner: 10Jforrester) [21:52:55] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki-config] Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535716 (owner: 10Jforrester) [21:54:11] (03CR) 10jerkins-bot: [V: 04-1] jjb: Add operations-mw-config-hhvm-composer-lint-docker and operations-mw-config-php72-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535715 (owner: 10Jforrester) [21:54:13] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki-config] Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535717 (owner: 10Jforrester) [21:57:32] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop operations-mw-config-hhvm-composer-test-docker [integration/config] - 10https://gerrit.wikimedia.org/r/535718 (owner: 10Jforrester) [22:10:20] (03PS1) 10Paladox: Modify access rules [research/ores/wheels] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/535721 [22:12:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10CirrusSearch, and 3 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10awight) Removing #move-files-to-commons because we think {T223... [22:12:21] 10Beta-Cluster-Infrastructure, 10ContentTranslation, 10Wikimedia-Site-requests, 10User-Luke081515: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157 (10Amitie_10g) Hi, I also want to test the ContentTranslation tool, specially due the bug affecting blocked users from translating. [22:13:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [23:21:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10dduvall) I'm still working on a thorough summary of this PoC setup, but here's a quickstart on how users might interact with it... [23:59:22] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Health, 10Epic, and 2 others: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294 (10Jdlrobson) MobileFrontend now publishes JS code coverage reports: https://doc.wikimedia.org/Mo...