[00:01:34] (03PS4) 10Dduvall: Further optimize initial checkout [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/661446 [00:01:36] (03PS1) 10Dduvall: wip [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664956 [00:01:41] (03PS1) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [00:01:58] (03Abandoned) 10Dduvall: wip [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664956 (owner: 10Dduvall) [00:04:25] (03PS2) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [00:06:09] !log gerrit: added abstract-wikipedia to members for extension-WikiLambda, mediawiki-services-function-schemata [00:06:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:18:32] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10brennen) Thanks both for the detailed responses. I'll cherry-pick a couple of things here and then respond in more detail. > I suspect not in all honestly... [01:00:12] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10Legoktm) (saw this scroll by on IRC) >>! In T274461#6839754, @brennen wrote: >> I suspect not in all honestly however i personally think we should encourage... [01:22:26] (03Abandoned) 10Eileen: Switch fundraising tech setup to use recurse_submodules [integration/config] - 10https://gerrit.wikimedia.org/r/661829 (https://phabricator.wikimedia.org/T273822) (owner: 10Eileen) [01:50:06] !log Kill stuck beta-scap-eqiad job and start a new one to sync beta [01:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:59:52] Urbanecm: FWIW, you don't need to start another [01:59:59] Just kill enough jobs and it'll start again itself [02:00:07] oh, good to know :) [02:00:34] At worst, you kill all the jobs... it won't be too long until the next spawns [02:00:50] i killed just one, was enough this time [02:00:57] and when it's usually been broken a few days before anyone notices.... a few more minutes doesn't usually matter too much :D [02:01:06] true [02:01:28] i didn't know it autostarts -- through it'll probably start with the nearest +2 anyway [02:02:10] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/333447/console [02:02:13] >01:53:30 Started by timer [02:02:47] I think it runs every 10 minutes [02:02:54] and then when code update is done, that'll trigger a scap job [02:03:09] i see [03:49:32] (03PS1) 10Dduvall: Allow inline blubberfile and in stage scope [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [03:50:11] (03CR) 10jerkins-bot: [V: 04-1] Allow inline blubberfile and in stage scope [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 (owner: 10Dduvall) [03:57:43] (03PS2) 10Dduvall: Allow inline blubberfile and in stage scope [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [03:58:36] (03PS3) 10Dduvall: Allow per-stage inline or path blubberfile [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [03:59:35] (03PS4) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [04:08:33] (03CR) 10Jeena Huneidi: [C: 03+1] Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [05:05:35] (03PS4) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [05:05:37] (03PS3) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [05:05:38] (03PS5) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [05:06:45] (03CR) 10jerkins-bot: [V: 04-1] Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 (owner: 10Dduvall) [05:10:46] (03CR) 10Dduvall: "Sorry for the churn but I realized that unconditionally reading all command output into a context variable might really bloat console logg" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [05:13:36] (03PS6) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [05:27:22] (03PS5) 10Dduvall: Export container process output from run step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 [05:27:24] (03PS4) 10Dduvall: Support list comprehensions as configuration values [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 [05:27:26] (03PS7) 10Dduvall: Allow per-stage blubberfile as path or inline config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 [08:26:40] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab, 10Patch-For-Review: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10akosiaris) [08:28:16] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab, 10Patch-For-Review: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10akosiaris) [08:34:16] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab, 10Patch-For-Review: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10akosiaris) [08:54:44] Yippee, build fixed! [08:54:44] Project mwcore-phpunit-coverage-master build #1224: 09FIXED in 5 hr 54 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1224/ [08:57:39] Yippee, build fixed! [08:57:39] Project beta-scap-eqiad build #338408: 09FIXED in 3 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/338408/ [11:50:19] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10akosiaris) >>! In T273521#6812274, @JMeybohm wrote: > I think we could make kubernetes... [13:36:21] (03CR) 10Hashar: [V: 03+2 C: 03+2] "Definitely yes. I guess we never had to fix it cause cherry picks to vendor.git are very rare." [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618808 (https://phabricator.wikimedia.org/T259832) (owner: 10Legoktm) [14:04:21] 10Beta-Cluster-Infrastructure, 10Dumps-Generation: mediawiki scripts fail on new buster image in deployment-prep - https://phabricator.wikimedia.org/T273089 (10ArielGlenn) I suppose this can be resolved by running /usr/sbin/update-ca-certificates on the instance; maybe this should be added to the docs for crea... [15:01:40] hashar: Hello! I'm reviewing https://gerrit.wikimedia.org/r/c/integration/config/+/651791 [15:01:57] Since then, gearman was added, which does not use mvnwrapper [15:02:17] do you prefer if I add it there? Or if I add the option of not using it? [15:02:52] I have a slight preference for standardizing on mvnwrapper everywhere, but for forks from external projects, that might be more messy than we need [15:07:50] gehel: yeah that is possibly messy for gearman-java, then that code has been unmaintained since 2011 or so [15:08:02] so we can add the mvnwrapper there to standardize our ci [15:08:18] it is not like I plan to do any further commit on that lib though [16:41:58] (03PS1) 10Ahmon Dancy: Fix mapStrings sample code [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/665122 [16:46:06] (03CR) 10Dduvall: [C: 03+2] Fix mapStrings sample code [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/665122 (owner: 10Ahmon Dancy) [16:48:30] (03Merged) 10jenkins-bot: Fix mapStrings sample code [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/665122 (owner: 10Ahmon Dancy) [16:54:03] (03Abandoned) 10Krinkle: Switch action to rebase if necessary [libs/Minify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/664659 (owner: 10DannyS712) [16:58:53] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Lucas_Werkmeister_WMDE) a:03Lucas_Werkmeister_WMDE [17:06:20] (03CR) 10Ahmon Dancy: Export container process output from run step (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [17:12:09] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, and 2 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Lucas_Werkmeister_WMDE) >>! In T222200#6818652, @noarave wrote: > Task inspection... [17:27:34] (03PS1) 10Jbond: tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 [17:30:08] (03CR) 10jerkins-bot: [V: 04-1] tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (owner: 10Jbond) [17:43:49] (03PS2) 10Jbond: tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 [17:45:28] (03CR) 10jerkins-bot: [V: 04-1] tox-ci: Improve abbility to run localy [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (owner: 10Jbond) [17:51:11] (03CR) 10Jforrester: [C: 04-2] "This is premature." [tools/release] - 10https://gerrit.wikimedia.org/r/663892 (https://phabricator.wikimedia.org/T237809) (owner: 10Prod) [17:52:59] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Upgrade Jenkins Gearman plugin from a forked repo - https://phabricator.wikimedia.org/T271683 (10hashar) I have done... [17:54:43] (03CR) 10Jbond: "Seems ci is faililng because `rsync: command not found` possible chicken egg issue" [integration/config] - 10https://gerrit.wikimedia.org/r/665133 (owner: 10Jbond) [17:58:35] 10phan-taint-check-plugin: Hardcode htmlspecialchars_decode as removing ESCAPED taint - https://phabricator.wikimedia.org/T275146 (10Daimona) [18:00:45] (03CR) 10Ahmon Dancy: Support list comprehensions as configuration values (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664957 (owner: 10Dduvall) [18:04:48] (03CR) 10Ahmon Dancy: [C: 04-1] "docs" (035 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/664992 (owner: 10Dduvall) [18:04:56] 10phan, 10MediaWiki-Core-Testing, 10Patch-For-Review: Audit using a library to bypass 'final' keyword in PHPUnit - https://phabricator.wikimedia.org/T231419 (10Krinkle) [18:09:06] (03PS1) 10Ostrzyciel: Add MediaUploader extension [integration/config] - 10https://gerrit.wikimedia.org/r/665141 [18:20:24] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dancy) >>! In T273521#6840723, @akosiaris wrote: > One extra thing to always keep in m... [18:40:56] (03CR) 10DannyS712: "> There is nothing about this repo or the people maintaining it that warrants a custom policy on this trivial aspect of merges. I believe " [libs/Minify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/664659 (owner: 10DannyS712) [19:13:53] (03CR) 10Dduvall: [C: 04-1] "Running into some CPS related `NotSerializableException` errors with this and I think it might be related to the way Lazy is working: stor" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/663905 (owner: 10Dduvall) [19:23:12] Project mediawiki-core-doxygen-docker build #22743: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22743/ [19:32:35] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) >>! In T273521#6841527, @dancy wrote: >>>! In T273521#6840723, @akosiaris wro... [19:49:37] (03CR) 10Legoktm: [C: 03+1] Add MediaUploader extension [integration/config] - 10https://gerrit.wikimedia.org/r/665141 (owner: 10Ostrzyciel) [20:23:03] Yippee, build fixed! [20:23:03] Project mediawiki-core-doxygen-docker build #22744: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22744/ [20:31:13] (03PS3) 10DannyS712: SpecificAssertionsSniff: fix order of assertStringContainsString parameters [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661247 (https://phabricator.wikimedia.org/T273624) [21:00:05] Project mwcore-phpunit-coverage-master build #1225: 04FAILURE in 6 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1225/ [21:03:58] 10Phabricator: Users CCed in private tasks should be able to access them - https://phabricator.wikimedia.org/T518 (10Aklapper) The behavior is not going to change, however soon Phabricator will visualize when a subscriber cannot access the object: https://secure.phabricator.com/rPa4cb2bb7724757a8f9dd4d68fde3bda2... [21:20:40] (03CR) 10Prod: "> Patch Set 1: Code-Review-2" [tools/release] - 10https://gerrit.wikimedia.org/r/663892 (https://phabricator.wikimedia.org/T237809) (owner: 10Prod) [21:45:15] (03PS1) 10Umherirrender: internal: Avoid extra local var before adding sniff error or warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 [21:47:46] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T271345 (10dduvall) [21:51:51] (03CR) 10DannyS712: [C: 03+1] internal: Avoid extra local var before adding sniff error or warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/665198 (owner: 10Umherirrender) [22:16:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) [22:16:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) Given the lack of input on the site name, I'm going to leave that... [22:17:06] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10Patch-For-Review, and 3 others: [Regression] some beta cluster wikis using official logos - https://phabricator.wikimedia.org/T274210 (10Jdlrobson) [22:23:22] Project mediawiki-core-doxygen-docker build #22746: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22746/ [22:34:30] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab, 10Patch-For-Review: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10wkandek) [23:22:50] Yippee, build fixed! [23:22:51] Project mediawiki-core-doxygen-docker build #22747: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/22747/ [23:32:17] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:53:08] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1