[00:21:31] (03CR) 10Ottomata: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1307228 (https://phabricator.wikimedia.org/T430967) (owner: 10Ottomata) [00:22:00] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 06Product Safety and Integrity, 07ci-test-error: /usr/sbin/sendmail: not found - https://phabricator.wikimedia.org/T431228#12097663 (10Reedy) [00:33:40] Yippee, build fixed! [00:33:40] Project mediawiki-core-doxygen build #21440: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21440/ [01:10:12] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12097720 (10Dzahn) Thank you @hashar for the nice words and the migration today! :) There were some follow... [01:35:39] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-Newsletter] Add Newsletter job [integration/config] - 10https://gerrit.wikimedia.org/r/1308246 (https://phabricator.wikimedia.org/T427010) [01:37:56] (03CR) 10Vaughn Walters: "This is ready for review" [integration/config] - 10https://gerrit.wikimedia.org/r/1308246 (https://phabricator.wikimedia.org/T427010) (owner: 10Vaughn Walters) [02:01:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431448#12097772 (10Dzahn) [02:01:41] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T431456#12097774 (10Dzahn) β†’14Duplicate dup:03T431448 [03:48:48] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Phabricator Bot Request: Test Kitchen Experiments Status Reporter - https://phabricator.wikimedia.org/T429377#12097858 (10cjming) tysm! works great [03:50:33] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Documentation: [[wikitech:Password and 2FA reset]] should cover GitLab MFA resets - https://phabricator.wikimedia.org/T431415#12097860 (10brennen) Well that's handy. [07:38:53] (03open) 10addshore: Fix composer question in CI (no interaction) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/659 [08:06:16] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 06Product Safety and Integrity, 07ci-test-error: /usr/sbin/sendmail: not found - https://phabricator.wikimedia.org/T431228#12098257 (10mszwarc) a:03mszwarc [08:13:34] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta Wikidata uses production Commons - https://phabricator.wikimedia.org/T431507#12098288 (10Lucas_Werkmeister_WMDE) Every Wikibase uses production Commons, because the data type is `commonsMedia`, not local media. (I’m guessing it tries to serve a local thumbnail be... [08:20:10] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12098309 (10Viktoria_Hillerud_WMSE) Hi! I'm working on the Wikispeech project (WMSE) and would like to be added to acl*Project-Admins so I can create sprint mi... [08:26:01] 10Phabricator Translations Extension, 06translatewiki.net: Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12098332 (10Nikerabbit) It is working as intended. Phabricator contains unsafe looking html in the translations. I've cleaned up the ol... [08:43:26] 10Phabricator Translations Extension, 06translatewiki.net, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12098443 (10Nikerabbit) p:05Triageβ†’03High a:03Nikerabbit [08:44:13] 10Phabricator Translations Extension, 06translatewiki.net, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12098447 (10Nikerabbit) [09:02:18] 10Phabricator Translations Extension, 06translatewiki.net, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12098531 (10Aklapper) Uh, thanks. Obviously this isn't on my radar. :-/ [09:06:29] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12098572 (10Aklapper) @Viktoria_Hillerud_WMSE Hi, I've added you. [09:07:25] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12098574 (10Viktoria_Hillerud_WMSE) >>! In T706#12098572, @Aklapper wrote: > @Viktoria_Hillerud_WMSE Hi, I've added you. Thank you! [09:36:10] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 07ci-test-error, 07Essential-Work, and 2 others: /usr/sbin/sendmail: not found - https://phabricator.wikimedia.org/T431228#12098765 (10mszwarc) [09:47:06] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add PDFCreator [integration/config] - 10https://gerrit.wikimedia.org/r/1308584 [09:47:42] 06Release-Engineering-Team, 103D, 06ServiceOps new: Evaluate shared Chromium service (WebSocket) for 3d2png and chromium-render - https://phabricator.wikimedia.org/T424872#12098830 (10MoritzMuehlenhoff) Just to confirm that this requirement is important: > we'd need CI checks validating both services still w... [10:52:50] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12099207 (10Pintoch) I imagine some current members will want to remove themselves from this new subproject after it is created and might want to add themselves b... [10:54:46] 10GitLab, 06Diffusion-Repository-Administrators, 10Projects-Cleanup: Archive GitLab "api-platform/catalog-to-wiki" repo and "api-platform" namespace - https://phabricator.wikimedia.org/T431560 (10Krinkle) 03NEW [10:58:41] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12099263 (10Pintoch) Let me ping them explicitly here: Hello @Spinster and @Eugene233, As part of the process of onboarding @Pxxlhxslxn and @DaxServer as maintai... [10:58:50] 10Phabricator Translations Extension, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12099264 (10Nikerabbit) [11:20:20] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta Wikidata uses production Commons - https://phabricator.wikimedia.org/T431507#12099334 (10AlexisJazz) Well, I managed to do my testing.. by making sure I only used filenames that exist on production. It would be more logical if betawikidata used betacommons, but... [12:07:35] 10Gerrit, 06collaboration-services, 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12099607 (10hashar) [12:15:03] (03CR) 10Hashar: [C:03+1] jjb: [catalyst-daily-Math] Add Math job [integration/config] - 10https://gerrit.wikimedia.org/r/1307868 (https://phabricator.wikimedia.org/T427008) (owner: 10Vaughn Walters) [12:15:15] (03CR) 10Hashar: [C:03+1] jjb: [catalyst-daily-Newsletter] Add Newsletter job [integration/config] - 10https://gerrit.wikimedia.org/r/1308246 (https://phabricator.wikimedia.org/T427010) (owner: 10Vaughn Walters) [12:19:02] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add PDFCreator [integration/config] - 10https://gerrit.wikimedia.org/r/1308584 (owner: 10Hslater) [12:19:25] (03CR) 10Hashar: [C:03+2] "I have added `VueJsPlus`Β as well since it is a requirement of `PDFCreator`." [integration/config] - 10https://gerrit.wikimedia.org/r/1308584 (owner: 10Hslater) [12:21:29] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add PDFCreator [integration/config] - 10https://gerrit.wikimedia.org/r/1308584 (owner: 10Hslater) [12:24:17] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ContentTransfer] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1308136 (owner: 10Hslater) [12:24:54] (03CR) 10Hashar: [C:03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1308584 (owner: 10Hslater) [12:26:13] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ContentTransfer] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1308136 (owner: 10Hslater) [12:27:03] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1307868 (https://phabricator.wikimedia.org/T427008) (owner: 10Vaughn Walters) [12:27:19] !log deployment-prep: on deployment server: clearing old branches for mediawiki/extensions and mediawiki/skins # T428864 [12:27:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:27:21] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [12:27:30] (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily-Newsletter] Add Newsletter job [integration/config] - 10https://gerrit.wikimedia.org/r/1308246 (https://phabricator.wikimedia.org/T427010) (owner: 10Vaughn Walters) [12:27:56] (03CR) 10Hashar: [C:03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1308136 (owner: 10Hslater) [12:30:24] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-Math] Add Math job [integration/config] - 10https://gerrit.wikimedia.org/r/1307868 (https://phabricator.wikimedia.org/T427008) (owner: 10Vaughn Walters) [12:30:37] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-Newsletter] Add Newsletter job [integration/config] - 10https://gerrit.wikimedia.org/r/1308246 (https://phabricator.wikimedia.org/T427010) (owner: 10Vaughn Walters) [12:45:23] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add EnhancedUpload [integration/config] - 10https://gerrit.wikimedia.org/r/1308629 [12:48:39] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add EnhancedUpload [integration/config] - 10https://gerrit.wikimedia.org/r/1308629 (owner: 10Hslater) [12:58:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add EnhancedUpload [integration/config] - 10https://gerrit.wikimedia.org/r/1308629 (owner: 10Hslater) [13:03:55] (03CR) 10Hashar: [C:03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1308629 (owner: 10Hslater) [13:04:44] !log deployment-prep: git repack on /srv/mediawiki-staging/php-master [13:04:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:15:33] (03CR) 10Hashar: [C:03+2] "Sorry this is NOW deployed :]" [integration/config] - 10https://gerrit.wikimedia.org/r/1308629 (owner: 10Hslater) [13:23:07] fatal: remote error: want 43365e726aaa355a696d3a8e639425db764f82c3 not valid [13:23:07] fatal: internal server error [13:23:07] Errors during submodule fetch: [13:23:07] BlueSpiceChecklist [13:27:10] Reedy: where is that? [13:27:19] locally [13:27:57] from the superproject mediawiki/extensions.git, I assume it is a race condition between the module being updated and the commit making it in the repo [13:28:30] hmm [13:30:57] I see that commit in the repo on the server [13:31:05] with commit message: `phpcs` [13:31:07] nothing else [13:31:21] so it is probably a direct push to the repo and Gerrit would not have updated the super project [13:31:38] 3cdbdc1 (HEAD -> master) phpcs [13:31:38] 05ad3b7 phpcs [13:31:38] 47a58ba eslint [13:31:39] a6c97fe Localisation updates from https://translatewiki.net. [13:37:52] who knows really :\ [13:38:20] * 3cdbdc1 - (HEAD -> master) phpcs (2 hours ago) [13:38:20] | * 43365e7 - phpcs (2 hours ago) [13:38:20] |/ [13:38:20] * 05ad3b7 - phpcs (2 hours ago) [13:38:31] Reedy: I think they pushed that 43365e7 commit to master [13:38:40] then force pushed another one (3cdbdc1) [13:38:46] leaving 43365e7 as a loose object [13:39:06] and maybe it is falsely advertised due to some caching [13:42:49] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 07ci-test-error, 07Essential-Work, 06Product Safety and Integrity (Sprint Venus Flytrap (Jun 29 - Jul 17)): /usr/sbin/sendmail: not found - https://phabricator.wikimedia.org/T431228#12100123 (10mszwarc) 05Openβ†’03Resolved [13:43:48] $ git fsck --unreachable --no-reflogs [13:43:48] unreachable commit 43365e726aaa355a696d3a8e639425db764f82c3 [13:43:48] [13:44:56] why don't people do things properly :P [13:53:50] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceNSFileRepoConnector] Add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1308645 [13:54:16] Reedy: my theory is your local repo head is at 43365e7 [13:54:38] your fetch advertise that as your head for master, the server can't find it in any head and bails out [13:54:47] (fringe theory, I don't really know how ref advertisements work) [13:54:55] I just re-ran the command [13:54:59] maybe reset to an older rev [13:56:07] the re-run did [13:56:08] From ssh://gerrit.wikimedia.org:29418/mediawiki/extensions [13:56:08] 016c2575f4a..b5cc630c1de master -> origin/master [13:57:18] yeah that is the superproject though [14:10:39] o/ CI is failing with error: `BuildKit is enabled but the buildx component is missing or broken` [14:10:40] here is the detailed log: https://integration.wikimedia.org/ci/job/inference-services-pipeline-pre-commit-check/733/console [14:15:34] kevinbazira: it is probably an aftermath of having migrated Jenkins to new hosts sorry. May you file it as a task please? [14:15:46] I'll look into it else refer to dduvall (he is on US west coast) [14:26:45] great! here is the phab task: https://phabricator.wikimedia.org/T431582 [14:28:39] awesome :) [14:28:42] meanwhile I found the issue! [14:29:46] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12100450 (10hashar) [14:31:59] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12100462 (10hashar) It is an aftermath of us having moved the Jenkins hosts... [14:36:23] !log contint1003, contint2003: manually installed `docker-buildx` Debian package to validate https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308659 # T431582 [14:36:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:36:26] T431582: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582 [14:36:56] I am confused [14:37:13] kevinbazira: so it failed earlier but then got fixed? I am reading https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1308631 [14:37:37] or I am on the wrong change [14:38:01] yep, seems fixed after running: `recheck` [14:38:39] I don't understand how that got fixed hmm [14:39:02] OHH yeah [14:39:19] the job is roaming that is why [14:44:18] kevinbazira: I think that will fix it, there is another issue but I will speak with Dan about it [14:44:32] thank you for the report! [14:44:56] great! as always, thank you for the swift fix <3 [14:45:14] my pleasure, thank you for flying with Release Engineering [14:46:32] ✈️ [14:48:03] those long stacks of patches in CI :( [15:08:12] 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12100740 (10JJMC89) [15:16:16] * Lucas_WMDE looks at zuul [15:16:18] jesus christ [15:16:36] I don’t think I’ve ever seen postmerge *this* backloged (Elapsed: 7Β hr 15Β min) [15:21:07] yeaahhhh [15:33:23] oof. [15:40:13] is this a case of needing a jenkins restart? [15:46:15] I don't think so [15:46:18] stuff is processing [15:46:33] those stacks of campaignevents changes aren't helpfu [15:46:49] a stack of ~10 makes everything slow [15:50:03] gerrit says there's a chain of 17 patches [15:55:43] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12100962 (10Dzahn) a:03Dzahn [16:16:47] 10Beta-Cluster-Infrastructure, 10Thumbor: commons.wikimedia.beta: There was a problem while contacting the thumbnailing service: [Errno -2] No address found - https://phabricator.wikimedia.org/T350165#12101041 (10bd808) β†’14Duplicate dup:03T365116 [16:16:49] 10Beta-Cluster-Infrastructure, 10Thumbor: Image thumbnail generation not working in Beta Cluster - https://phabricator.wikimedia.org/T365116#12101043 (10bd808) [16:17:39] 10Beta-Cluster-Infrastructure, 10UploadWizard, 06Quality-and-Test-Engineering-Team (Test Infrastructure): 404 error for SVG image thumbnail in UploadWizard on Beta Cluster - https://phabricator.wikimedia.org/T265580#12101054 (10bd808) β†’14Duplicate dup:03T365116 [16:17:40] 10Beta-Cluster-Infrastructure, 10Thumbor: Image thumbnail generation not working in Beta Cluster - https://phabricator.wikimedia.org/T365116#12101056 (10bd808) [16:49:53] we've discussed zuul state in releng meeting, will be trying the steps under https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions shortly and failing that probably do a restart. [16:57:28] fatal: remote error: want 90c8cf29cfdf5430adc845173aae40bf3e97170a not valid [16:57:29] fatal: internal server error [16:57:29] Errors during submodule fetch: [16:57:29] BlueSpiceExtendedSearch [16:57:31] WHY DO THEY [17:02:00] where's that? [17:02:17] locally when trying to upate from the extensions meta repo [17:02:24] ah [17:02:31] gross [17:02:39] it was doing it earlier for another repo [17:02:43] antoine poked around [17:02:48] ~3-4 hours ago [17:03:13] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily] Run core tests first [integration/config] - 10https://gerrit.wikimedia.org/r/1308686 (https://phabricator.wikimedia.org/T429829) [17:06:55] (03CR) 10Vaughn Walters: "This is ready for review" [integration/config] - 10https://gerrit.wikimedia.org/r/1308686 (https://phabricator.wikimedia.org/T429829) (owner: 10Vaughn Walters) [17:09:30] well, looks like things are trending towards healthy based on https://grafana.wikimedia.org/d/d4bf3e68-9396-4173-a59c-59e398172ba0/zuul-prometheus?orgId=1&from=now-7d&to=now&timezone=utc - we'll give it another 20-30 [17:09:48] ^ cc: hashar, we dropped off the call [17:09:59] noticed :) [17:10:15] you did the read-only magic trick or is that the queue just recovered? [17:10:52] the CampaignEvents series of patch was made by Claudio who is based in europe so I guess he will not rebase it any further today [17:11:00] I have left him a direct message via Slack [17:11:14] what is left is to add link to our homepage of runbooks [17:11:39] it _seems_ like the queue is recovering. [17:11:57] i'll update the runbook page. [17:12:12] merger:merge 100 2 2 [17:12:19] down from ~240 earlier [17:12:29] i don't really know what a healthy number is here. [17:12:48] it might be possible to inspect the jobs somehow, but I don't think the gearman protocol has a command to inspect the details of jobs [17:13:06] the first is the queue, there are 100 requests to run that function merger:merge [17:13:18] 10Beta-Cluster-Infrastructure: "DBQueryError: Table 'testwiki.cx_translations' doesn't exist" when creating a page on testwiki.beta - https://phabricator.wikimedia.org/T335210#12101272 (10bd808) 05Openβ†’03Invalid Fixed by the vague changes of time: https://test.wikipedia.beta.wmcloud.org/wiki/This_won%27t... [17:13:19] the 2nd would be the number of worker currently executing that function [17:13:26] ah, makes sense [17:13:32] 88 now [17:13:47] and the last entry the number of workers that knows how to run that function (eg: the worker registered to the server yes I `CAN_DO` `merger:merge` function) [17:14:46] the zuul-merger processes stream their debug output in /var/log/zuul/merger-debug.log [17:15:04] so one can: ssh contint1002.wikimedia.org tail -F /var/log/zuul/merger-debug.log [17:15:10] ::nod:: [17:15:12] (aand same with contint2002.wikimedia.org) [17:15:37] the first thing the zuul-merger does is reset the state of the repositories branches compared to what is in Gerrit [17:15:57] it does it by doing a foreach $branch; do ; done; [17:16:19] newer zuul is smarter iirc it does an appropriate fetch from Gerrit and git takes care of updating the branch [17:16:32] on mediawiki/core it takes a while [17:17:12] and operations/puppet.git takes 10/15 seconds to update iirc [17:17:43] (that repo has a bunch of branches sandbox/$user/xx) [17:25:51] Newer Zuul can't come soon enough! ;-) [17:28:04] so merger:merge is down to 0. still a ton of stuff in postmerge? [17:29:43] brennen: so merger:merge at 0 means there are no speculative states to compute (merge commits of proposed patches against the branch/patches ahead in the queue) [17:30:03] the changes in the `postmerge` section at https://integration.wikimedia.org/zuul/ [17:30:27] corresponds to the `postmerge`Β pipeline defined in Zuul, which is entirely unrelated to the merger:merge function [17:30:50] yeah [17:30:50] `postmerge` reacts after a change has been merged, typically to publish reports / doc [17:31:12] the changes there are all waiting for `mwext-codehealth-master-non-voting (non voting)` [17:31:21] (03PS2) 10Jforrester: jjb: [catalyst-daily-*] Run core tests first [integration/config] - 10https://gerrit.wikimedia.org/r/1308686 (https://phabricator.wikimedia.org/T429829) (owner: 10Vaughn Walters) [17:31:26] (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily-*] Run core tests first [integration/config] - 10https://gerrit.wikimedia.org/r/1308686 (https://phabricator.wikimedia.org/T429829) (owner: 10Vaughn Walters) [17:31:34] which has a mutex, only ONE build can happen at any time (why, I don't know) [17:32:09] that jobs rerun tests using quibble and generate a report to SonarQube. [17:32:35] the `postmerge` pipeline has a low precedence (other precedences areΒ `high` and `normal` [the default]) [17:33:06] so those will take their time, but should clear _eventually_ [17:33:11] so as long as there are change/jobs in other pipelines (eg test / gate-and-submit), the jobs waiting in postmerge do not get processed and build up [17:33:15] correct [17:33:17] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-*] Run core tests first [integration/config] - 10https://gerrit.wikimedia.org/r/1308686 (https://phabricator.wikimedia.org/T429829) (owner: 10Vaughn Walters) [17:33:41] and given they have elapsed "9 hours", we had a disrupted CI for most of the day (due to the chain of changes to CampaignEvents) [17:34:28] maybe that "mwext-codehealth-master-non-voting" could be allowed to run more than one build at the time [17:37:53] zuul/layout.yaml [17:37:54] - name: mwext-phpunit-coverage-publish [17:37:54] mutex: mwext-phpunit-coverage-publish [17:38:09] err wrong job :b [17:39:13] and the reason for the mutex is descirbed in the commit message https://gerrit.wikimedia.org/r/c/integration/config/+/540805 [17:39:28] which was https://phabricator.wikimedia.org/T234259 [17:39:30] for updating the docs, how many waiting jobs is too many for merger:merge? > 20? > 100? [17:39:43] no clue [17:40:00] it is more about the rate it is processing them versus the estimated time to resolve that queue [17:40:33] you could have 500 of them for smaller repos with a little number of branches and fast merge commits [17:40:38] which would be processed quickly [17:41:17] https://grafana.wikimedia.org/d/ad656c66-d8b5-4b09-a54b-61e7df71fb17/zuul-gearman-prometheus has an example that happened 20 hours ago [17:42:22] with up to 650 waiting at 00:30 which got finished at 3:00 so 2h30 minutes for 650 [17:42:35] https://grafana.wikimedia.org/d/ad656c66-d8b5-4b09-a54b-61e7df71fb17/zuul-3a-3a-gearman-prometheus?orgId=1&from=2026-07-07T20:33:20.054Z&to=2026-07-08T03:24:45.769Z&timezone=utc&viewPanel=panel-10 [17:42:48] there is an horizontal line set at 500 (it used to emit an alert here) [17:42:53] but that was set quite arbitrarily [17:43:21] so maybe 300 [17:43:58] and if we had a script to mark a repo read-only, emit the status of the queue periodically then move it back to write-only, I guess that will ease things [17:44:34] maybe as a script in integration/config (to ssh to both contint* machines and issues the sudo command that puts things readonly) [17:44:42] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 10Maps (Kartographer): Console error on all pages in beta-meta: ConfigException: $wgKartographerNearby requires GeoData and CirrusSearch extensions - https://phabricator.wikimedia.org/T331528#12101420 (10bd808) @Daimona Is this still reproducible? If... [17:44:45] then busy loop till that goes down [17:47:18] 10Beta-Cluster-Infrastructure, 10CXServer: CXServer doesn't work on Beta Cluster - https://phabricator.wikimedia.org/T323417#12101430 (10bd808) [17:48:49] https://www.mediawiki.org/w/index.php?title=Continuous_integration%2FZuul&diff=8496753&oldid=8098515 [17:50:35] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491#12101452 (10bd808) 05Openβ†’03Invalid No hits for `cirrusSearchCheckerJob` in r... [17:51:08] 10Beta-Cluster-Infrastructure, 10CXServer: CXServer doesn't work on Beta Cluster - https://phabricator.wikimedia.org/T323417#12101460 (10bd808) [17:51:12] 10Beta-Cluster-Infrastructure, 10ContentTranslation: Content security policy error when using ContentTranslation on the beta cluster - https://phabricator.wikimedia.org/T312092#12101463 (10bd808) β†’14Duplicate dup:03T323417 [18:01:35] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 10Maps (Kartographer): Console error on all pages in beta-meta: ConfigException: $wgKartographerNearby requires GeoData and CirrusSearch extensions - https://phabricator.wikimedia.org/T331528#12101483 (10Daimona) 05Openβ†’03Resolved a:03Daimona... [18:02:05] brennen: thank you! :] [18:02:08] I am off for dinner [18:02:13] enjoy! [19:07:07] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12101745 (10Dzahn) Happy to take care of merging that... [19:18:18] From ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/BlueSpiceExtendedSearch [19:18:18] + c6f25428...2cc9dea3 master -> origin/master (forced update) [19:18:19] jfc [19:18:22] WILL THEY STOP [19:21:25] replacing registerEmailPrivateKey in Gerrit config - this invalidates pending/outstanding email validation links for gerrit users - but does not affect active accounts or already verified email addresses [19:21:33] !log gerrit - replacing registerEmailPrivateKey in Gerrit config - this invalidates pending/outstanding email validation links for gerrit users - but does not affect active accounts or already verified email addresses [19:21:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:46:33] Reedy: have we yelled at them lately about this [20:06:39] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12102001 (10brennen) 05Openβ†’03In progress [20:07:31] 10Continuous-Integration-Config, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Introduce yamlllint to the integration/config repository - https://phabricator.wikimedia.org/T429576#12102011 (10brennen) 05Openβ†’03In progress p:05Triageβ†’03Low [20:08:18] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for TehKittyCat - https://phabricator.wikimedia.org/T428836#12102014 (10brennen) 05Openβ†’03Stalled p:05Triageβ†’03Low [20:38:14] (03open) 10dancy: kubernetes: Add deployment scopes and check supervision rules [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1218 (https://phabricator.wikimedia.org/T428971) [20:46:43] (03update) 10dancy: kubernetes: Add deployment scopes and check supervision rules [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1218 (https://phabricator.wikimedia.org/T428971) [20:50:06] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T430829#12102162 (10Reedy) [20:58:52] !log patchdemo: deployed https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/367 (T427964) [20:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:58:56] T427964: patchdemo: Move "delete" action into a Slim action, template, etc. - https://phabricator.wikimedia.org/T427964 [20:59:33] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 13Patch-For-Review: patchdemo: Move "delete" action into a Slim action, template, etc. - https://phabricator.wikimedia.org/T427964#12102192 (10brennen) 05Openβ†’03Resolved [21:34:43] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.31.1 released - https://phabricator.wikimedia.org/T429988#12102279 (10dancy) 05In progressβ†’03Resolved [22:05:08] (03update) 10addshore: Fix composer question in CI (no interaction) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/659 [22:06:09] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611#12102364 (10bd808) 05Openβ†’03Declined `lang=irc [22:04] < bd808> I am cleaning the backlog. Is this 8 year old "somebody should do something" worth keeping alive until someone gets so bored they work on i... [22:38:57] 06Release-Engineering-Team, 10Scap: scap triggers a FutureWarning on a deployment - https://phabricator.wikimedia.org/T431635 (10Urbanecm_WMF) 03NEW [23:25:03] (03update) 10swfrench: kubernetes: Add deployment scopes and check supervision rules [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1218 (https://phabricator.wikimedia.org/T428971) (owner: 10dancy) [23:28:45] (03open) 10brennen: Draft: Logstash: use https scheme for logstash_search_url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1219 (https://phabricator.wikimedia.org/T431635) [23:29:21] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: scap triggers a FutureWarning on a deployment - https://phabricator.wikimedia.org/T431635#12102563 (10brennen) 05Openβ†’03In progress [23:29:51] 06Release-Engineering-Team (Doing 😎), 10Scap, 13Patch-For-Review: scap triggers a FutureWarning on a deployment - https://phabricator.wikimedia.org/T431635#12102565 (10brennen) p:05Triageβ†’03Medium [23:54:09] (03update) 10brennen: Draft: Logstash: use https scheme for logstash_search_url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1219 (https://phabricator.wikimedia.org/T431635)