[00:02:08] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:27:53] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:37:23] PROBLEM - Work requests waiting in Zuul Gearman server on contint1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [00:47:53] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [01:02:23] RECOVERY - Work requests waiting in Zuul Gearman server on contint1002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [05:31:21] 10Phabricator: Restrict maximum file size for Phabricator file uploads - https://phabricator.wikimedia.org/T151544#10163258 (10Iniquity) >>! In T151544#10162948, @matmarex wrote: > Try recording the screencasts as WEBM files encoded with VP9. We've allowed WEBM files on Phabricator in T215360 (in 2020), and... [07:44:44] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T373642#10163350 (10jnuche) 05Open→03Resolved [08:00:55] (03PS1) 10Kosta Harlan: zuul: Re-enable parallel tests on TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) [08:05:27] kostajh: +2 ): [08:05:29] :) [08:05:33] (03CR) 10Hashar: [C:03+2] zuul: Re-enable parallel tests on TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) (owner: 10Kosta Harlan) [08:05:51] (03PS2) 10Hashar: zuul: Re-enable parallel tests on TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) (owner: 10Kosta Harlan) [08:06:09] (03CR) 10Hashar: [C:03+2] zuul: Re-enable parallel tests on TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) (owner: 10Kosta Harlan) [08:07:21] (03PS1) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) [08:07:23] (03Merged) 10jenkins-bot: zuul: Re-enable parallel tests on TwoColConflict [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) (owner: 10Kosta Harlan) [08:07:35] (03PS2) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) [08:09:20] (03CR) 10CI reject: [V:04-1] zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [08:09:33] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1074362 (https://phabricator.wikimedia.org/T369146) (owner: 10Kosta Harlan) [08:13:17] (03CR) 10Arthur taylor: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [08:18:59] (03PS3) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) [08:19:08] (03CR) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [08:19:38] hashar: thanks! [08:20:09] (03CR) 10CI reject: [V:04-1] zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [08:20:29] (03PS4) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) [08:21:23] (03PS5) 10Kosta Harlan: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) [08:58:27] OMG Phatality stacktrace output is now a table with each call linking to its source code line? That is darrn neat! <3 [09:28:32] andre: clearly we should migrate Wikipedia to Phiwiki [09:28:44] it is nice feat [13:00:45] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Switch Jenkins instances from Java 11 to Java 17 - https://phabricator.wikimedia.org/T359795#10163987 (10hashar) 05In progress→03Resolved All hosts have been switched to Java 17. [13:02:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: releases1003 file system over 90% full - https://phabricator.wikimedia.org/T368239#10163991 (10hashar) [13:27:56] * jhathaway gently pokes hashar to deploy bullseye operations-puppet [13:30:49] jhathaway: Has the image not been built yet? [13:31:21] Reedy: not sure, hashar merged in the patch yesterday [13:32:05] the `deploy_docker` operation seems to be a noop as far as CI is concerned [13:32:19] 06Project-Admins: Create project tag for Community Growth DE&L team - https://phabricator.wikimedia.org/T373957#10164069 (10Aklapper) >>! In T373957#10123205, @Arinaigu wrote: > Why is it considered a personal non-WMF account? From the two account pages for my SUL accounts [Arinaigu](https://meta.wikimedia.o... [13:32:23] he asked me to poke him this morning, perhaps he already built and deployed it? [13:32:42] Possible, but it's not in use (yet) [13:32:43] jjb/operations-puppet.yaml: image: docker-registry.wikimedia.org/releng/operations-puppet:0.9.2-s1 [13:32:43] etc [13:33:04] I can fix that [13:34:28] (03PS1) 10Reedy: jjb: Switch operations-puppet to 0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 [13:35:24] https://docker-registry.wikimedia.org/releng/operations-puppet/tags/ shows 0.9.3... [13:37:29] (03CR) 10Reedy: [C:03+2] jjb: Switch operations-puppet to 0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [13:38:00] 0.9.3 is the bullseye version [13:38:41] (03CR) 10Reedy: jjb: Switch operations-puppet to 0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [13:38:46] You're wanting to run both in parallel? [13:49:27] I don't think so, just the bullseye image [13:52:13] (03PS2) 10Reedy: jjb: Switch operations-puppet to 0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 [13:53:24] (03CR) 10CI reject: [V:04-1] jjb: Switch operations-puppet to 0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [13:54:22] (03PS3) 10Reedy: Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 [13:55:51] (03CR) 10JHathaway: [C:03+1] Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [14:04:33] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T373642#10164112 (10thcipriani) Stats: - [[https://www.mediawiki.org/wiki/MediaWiki_1.43/wmf.23|274 Changes]] ▁█▄▄▂ (📉 -18% vs last week) -- in 62 repos... [14:23:00] (03CR) 10Subramanya Sastry: [C:04-1] Visual diff: Remove noise from known low priority issues (032 comments) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [14:37:32] (03CR) 10Pmiazga: [C:03+1] zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [14:50:38] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07Developer Productivity, 10MediaWiki-Platform-Team (Radar): Create a daily job running core + extension PHPUnit tests serially - https://phabricator.wikimedia.org/T372618#10164215 (10pmiazga) Where this job could log in when it fails? Maybe it could push... [15:16:36] Thanks Reedy, +1 the patch are we waiting for hashar's review? [15:35:59] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Jenkins: thirdparty/ci apt component is configured multiple times - https://phabricator.wikimedia.org/T375278 (10hashar) 03NEW [15:36:29] jhathaway: which one ?:) [15:36:49] hashar: https://gerrit.wikimedia.org/r/1074426 [15:37:44] oh I forgot about that one [15:37:51] lets do out it on Monday morning [15:38:14] I'd like to avoid breaking operations/puppet.git :) [15:39:14] cause if something is broken, I am not sure how fast it can be reverted [15:42:52] oh [15:43:12] dancy: hi do you know how to deploy/rollback a jenkins job update? [15:43:26] jhathaway has a patch to witch the operations/puppet test job to a new container version [15:43:45] and if it breaks today, it would be nice to have someone around to rollback (given I am soon out to prepare dinner / kids / week-end etc) [15:49:03] I made the patch, I can deploy it :P [15:57:27] thanks [16:06:56] Reedy: then go for it, but I will not be able to rollback :/ [16:07:05] I guess I am bit paranoid [16:13:19] (03CR) 10Hashar: "Lets roll that on our Monday morning! :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:16:54] I am off [16:35:10] hashar: yes, I know how to deploy jenkins jobs. [16:35:22] :) [16:35:36] dancy: may you roll https://gerrit.wikimedia.org/r/1074426 for jhathaway ? [16:35:52] Sure. jhathaway. Still around? [16:36:09] yup! [16:36:12] that upgrades the CI container for operations/puppet from Buster to Bullseye [16:36:32] and I am afraid that will end up blocking a SRE at some point in my friday evening :D [16:37:17] Reasonable. [16:38:03] jhathaway: And you have a way to confirm the result once deployed? [16:38:51] I can submit some test gerrit patches [16:39:11] ok.. starting [16:39:19] (03CR) 10Ahmon Dancy: [C:03+2] zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:40:21] 🤞 [16:40:38] (03Merged) 10jenkins-bot: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:42:06] thanks ! [16:42:10] oh no [16:42:23] ? [16:42:24] I pointed out the wrong patch [16:42:26] damn copy pasting [16:42:54] https://gerrit.wikimedia.org/r/c/integration/config/+/1074426 [16:42:56] is the one for jhathaway [16:43:01] sorry :-\ [16:43:03] ah ok. [16:43:08] hm.. I wonder if Piotr is around [16:43:19] Or kostah [16:43:23] we would deploy that one on Monday with kosta [16:43:25] let me revert it [16:43:31] okl [16:43:38] switching to https://gerrit.wikimedia.org/r/c/integration/config/+/1074426 [16:43:38] (03PS1) 10Hashar: Revert "zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches" [integration/config] - 10https://gerrit.wikimedia.org/r/1074479 [16:43:57] (03CR) 10Hashar: [C:03+2] "I have pointed the wrong patch for deployment, this one will land on Monday." [integration/config] - 10https://gerrit.wikimedia.org/r/1074479 (owner: 10Hashar) [16:44:24] (03CR) 10Ahmon Dancy: [C:03+2] "Accidental merge." [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:44:31] sorry [16:45:06] (03CR) 10Ahmon Dancy: [C:03+2] Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [16:45:37] (03Merged) 10jenkins-bot: Revert "zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches" [integration/config] - 10https://gerrit.wikimedia.org/r/1074479 (owner: 10Hashar) [16:45:42] (03PS1) 10Hashar: zuul: Enable PHPUnit parallel tests on all extensions, exclude REL1* branches [integration/config] - 10https://gerrit.wikimedia.org/r/1074480 (https://phabricator.wikimedia.org/T50217) [16:46:14] (03CR) 10Hashar: "I gave the wrong link to Ahmon!" [integration/config] - 10https://gerrit.wikimedia.org/r/1074364 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:46:32] (03CR) 10CI reject: [V:04-1] Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [16:46:59] (03CR) 10Hashar: [C:04-2] "-2 until Monday :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1074480 (https://phabricator.wikimedia.org/T50217) (owner: 10Hashar) [16:48:28] I am off, I am late for diner [16:48:36] Have a good weekend! [16:48:38] the CI job failed cause the job has to be deployed with ./jjb-update [16:48:40] :) [16:48:45] nod.. working on it [16:52:08] (03CR) 10Ahmon Dancy: Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [16:52:12] (03CR) 10Ahmon Dancy: [C:03+2] Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [16:53:28] (03Merged) 10jenkins-bot: Switch operations-puppet to bullseye/0.9.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1074426 (owner: 10Reedy) [16:54:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1074426 [16:54:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:54:41] jhathaway: Ready for testing [16:54:50] on it [16:57:33] dancy: yeah, I agree is Monday much safer [16:57:42] 👍🏾 [17:14:00] dancy: looks generally okay, thanks [17:14:07] Excellent. [17:15:30] (03open) 10dancy: Set up /workspace/venvs/scap [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/90 [17:15:34] (03update) 10dancy: Set up /workspace/venvs/scap [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/90 [17:16:39] (03update) 10dancy: Set up /workspace/venvs/scap [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/90 [18:20:02] 06Release-Engineering-Team, 10Scap, 10Phatality: Outstanding issues with scap phatality deployment - https://phabricator.wikimedia.org/T375294 (10dancy) 03NEW [18:23:32] 06Release-Engineering-Team, 10Scap, 10Phatality: Outstanding issues with scap phatality deployment - https://phabricator.wikimedia.org/T375294#10164822 (10dancy) [18:25:08] (03PS4) 10Jdlrobson: Visual diff: Remove noise from known low priority issues [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 [18:27:31] 10Scap, 10Phatality: scap phatality deployment problem - https://phabricator.wikimedia.org/T374880#10164825 (10dancy) >>! In T374880#10153611, @dancy wrote: > Considering this resolved after https://gerrit.wikimedia.org/r/1073286. However I do have some remaining suspicions about the rollback behavior sho... [18:28:50] (03PS5) 10Jdlrobson: Visual diff: Remove noise from known low priority issues [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 [18:31:43] (03PS6) 10Jdlrobson: Visual diff: Remove noise from known low priority issues [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 [19:09:47] (03CR) 10Jdlrobson: Visual diff: Remove noise from known low priority issues (032 comments) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 (owner: 10Jdlrobson) [19:15:15] 10Phabricator-Bot-Requests, 10Incident Tooling: Create phabricator bot user for 'corto' - https://phabricator.wikimedia.org/T355755#10164907 (10Eevans) p:05Triage→03Medium [19:16:02] 10Phabricator-Bot-Requests, 10Incident Tooling: Create phabricator bot user for 'corto' - https://phabricator.wikimedia.org/T355755#10164911 (10Eevans) 05Open→03Resolved a:03fgiunchedi Near as I can tell, this issue is resolved. [19:22:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: releases1003 file system over 90% full - https://phabricator.wikimedia.org/T368239#10164921 (10Dzahn) a:03Dzahn [19:22:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services: releases1003 file system over 90% full - https://phabricator.wikimedia.org/T368239#10164922 (10Dzahn) I will be out for a week but get back to this after. [19:29:02] (03PS1) 10Subramanya Sastry: MFE: Get rid of the hacky lead para hoisting code [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074527 [19:33:04] (03open) 10dancy: Makefile: Don't pass --quiet when building the verify-deps image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/452 [19:33:08] (03update) 10dancy: Makefile: Don't pass --quiet when building the verify-deps image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/452 [19:34:22] (03update) 10dancy: Makefile: Don't pass --quiet when building the verify-deps image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/452 [19:34:26] (03update) 10dancy: Makefile: Don't pass --quiet when building the verify-deps image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/452 [19:34:32] (03update) 10dancy: Makefile: Don't pass --quiet when building the verify-deps image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/452 [19:40:05] (03PS1) 10Umherirrender: Zuul: [mediawiki/extensions/GrowthExperiments] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1074533 [19:45:49] (03PS1) 10JHathaway: operations-puppet: re-bundle to grab the updates in the Gemfile for Puppet 7 [integration/config] - 10https://gerrit.wikimedia.org/r/1074534 (https://phabricator.wikimedia.org/T330490) [19:48:19] (03CR) 10Subramanya Sastry: Visual diff: Remove noise from known low priority issues (032 comments) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 (owner: 10Jdlrobson) [19:48:40] (03CR) 10Subramanya Sastry: [C:03+2] MFE: Get rid of the hacky lead para hoisting code [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074527 (owner: 10Subramanya Sastry) [19:49:15] (03Merged) 10jenkins-bot: MFE: Get rid of the hacky lead para hoisting code [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074527 (owner: 10Subramanya Sastry) [21:38:19] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [21:43:48] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [21:44:03] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [21:46:58] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [22:09:53] (03PS7) 10Jdlrobson: Visual diff: Remove noise from known low priority issues [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284 [22:15:03] (03PS8) 10Jdlrobson: Visual diff: Remove noise from known low priority issues [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1074284