[00:32:22] (03open) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 [00:33:22] (03update) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 [00:38:44] (03update) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 [00:38:45] (03update) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 [00:38:47] (03update) 10dduvall: apt: Support `signed-by` field in apt configuration [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/124 [07:22:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch13 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:32:28] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch12 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:37:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch12 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:03:02] (03Abandoned) 10Slyngshede: New Docker image, dotnet version 8 [integration/config] - 10https://gerrit.wikimedia.org/r/1149413 (https://phabricator.wikimedia.org/T395036) (owner: 10Slyngshede) [08:29:35] (03CR) 10Hashar: "I will poke them on Slack, thank you!" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1151696 (https://phabricator.wikimedia.org/T395190) (owner: 10Hashar) [09:43:00] !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 57273 and 57274 [09:43:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:43:24] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search (2025.05.24 - 2025.06.13): Updating weighed tags via EventBus in beta does not work - https://phabricator.wikimedia.org/T395425#10870396 (10Gehel) p:05Triage→03High [09:49:01] (03PS1) 10Lucas Werkmeister (WMDE): jjb: rsync with --no-times [integration/config] - 10https://gerrit.wikimedia.org/r/1152248 (https://phabricator.wikimedia.org/T188488) [09:50:16] (03CR) 10CI reject: [V:04-1] jjb: rsync with --no-times [integration/config] - 10https://gerrit.wikimedia.org/r/1152248 (https://phabricator.wikimedia.org/T188488) (owner: 10Lucas Werkmeister (WMDE)) [09:54:09] (03PS2) 10Lucas Werkmeister (WMDE): jjb: rsync with --no-times [integration/config] - 10https://gerrit.wikimedia.org/r/1152248 (https://phabricator.wikimedia.org/T188488) [10:05:48] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search (2025.05.24 - 2025.06.13): Updating weighed tags via EventBus in beta does not work - https://phabricator.wikimedia.org/T395425#10870510 (10dcausse) 05Open→03Resolved a:03Urbanecm_WMF The EventBus approach is not supposed to work on... [11:20:36] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T395665 (10Atom.oil.2) 03NEW [11:36:35] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Atom.oil.2 - https://phabricator.wikimedia.org/T395665#10870680 (10Bunnypranav) [11:53:22] 06Release-Engineering-Team, 10Metrics Platform, 07ci-test-error: MetricsPlatform dependency is causing CI to fail on older (yet supported) MediaWiki release branches (REL1_39, REL1_42) - https://phabricator.wikimedia.org/T395494#10870708 (10phuedx) >>! In T395494#10865213, @Jdforrester-WMF wrote: > Probably... [12:00:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Atom.oil.2 - https://phabricator.wikimedia.org/T395665#10870755 (10Aklapper) That repository is hosted in Gerrit though and not in Gitlab? Or do I misunderstand your plans? [12:04:47] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Atom.oil.2 - https://phabricator.wikimedia.org/T395665#10870759 (10Atom.oil.2) As far as I understand Gerrit is old and it's now moved to Gitlab - I believe it's this repo - https://gitlab.wikimedia.org/repos/wmde... [12:07:03] 06Release-Engineering-Team, 10Metrics Platform, 07ci-test-error: MetricsPlatform dependency is causing CI to fail on older (yet supported) MediaWiki release branches (REL1_39, REL1_42) - https://phabricator.wikimedia.org/T395494#10870762 (10phuedx) I also considered adding a dependency on EventStreamConfigs... [12:30:10] (03approved) 10jnuche: Always include an old mediawiki version in container images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/826 (https://phabricator.wikimedia.org/T395514) (owner: 10dancy) [12:57:40] 06Project-Admins: Create a new SLO Phabricator tag - https://phabricator.wikimedia.org/T395537#10870870 (10elukey) My aim is to have a single Phabricator tag where to collect everything SLO related, from tooling to specific requests. In the future it may be used by the SLO working group, to figure out what to do... [12:58:49] (03PS1) 10Phuedx: Revert "Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1152260 [12:59:03] (03PS2) 10Phuedx: Revert "Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1152260 (https://phabricator.wikimedia.org/T395494) [13:29:56] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10870908 (10Gopavasanth) Hi, Could you please consider adding me to the acl `*Project-Admins` group? Reason: As part of the Indic MediaWiki Developers User Gr... [13:34:30] (03CR) 10Hashar: [C:03+2] Revert "Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1152260 (https://phabricator.wikimedia.org/T395494) (owner: 10Phuedx) [13:35:50] (03Merged) 10jenkins-bot: Revert "Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1152260 (https://phabricator.wikimedia.org/T395494) (owner: 10Phuedx) [13:39:33] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1152260 (https://phabricator.wikimedia.org/T395494) (owner: 10Phuedx) [14:03:33] 10WikimediaDebug, 10noc.wikimedia.org, 06serviceops, 10MediaWiki-Platform-Team (Radar): Enable WikimediaDebug support for noc.wikimedia.org - https://phabricator.wikimedia.org/T395682 (10Krinkle) 03NEW [14:49:58] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871195 (10dancy) The puppet failure has chan... [15:21:56] dancy my ears are burning on that deployment-prep opensearch ticket! I think I can help with the curator stuff if you like [15:22:13] That would be great! [15:22:26] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526#10871343 (10Scott_French) Ah, thank you, Moritz! I had no idea this was actually supported in a later version of reprepro.... [15:22:46] Cool, will take a look [15:22:56] Thanks! [15:23:23] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871358 (10bking) 05Open→03In progress a:... [15:30:32] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Atom.oil.2 - https://phabricator.wikimedia.org/T395665#10871379 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T395665#10870759, @Atom.oil.2 wrote: > As far as I understand Gerrit is old and it's... [15:44:35] 10WikimediaDebug, 10noc.wikimedia.org, 06serviceops, 10MediaWiki-Platform-Team (Radar): Enable WikimediaDebug support for noc.wikimedia.org - https://phabricator.wikimedia.org/T395682#10871429 (10Pppery) [15:44:51] 10WikimediaDebug, 10noc.wikimedia.org, 06serviceops, 10MediaWiki-Platform-Team (Radar): Enable WikimediaDebug support for noc.wikimedia.org - https://phabricator.wikimedia.org/T395682#10871430 (10Pppery) [15:52:14] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871457 (10bking) I'm taking a look at this n... [16:01:06] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526#10871472 (10MoritzMuehlenhoff) >>! In T392526#10871343, @Scott_French wrote: > I'm also unsure how to resolve the difference... [16:01:39] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871475 (10bking) Interestingly, I can reprod... [16:15:24] 10Continuous-Integration-Config, 07Developer Productivity: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10871502 (10Krinkle) [16:15:25] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review: Castor rsync causes: rsync: failed to set times on "/cache/.": Operation not permitted (1) - https://phabricator.wikimedia.org/T188488#10871503 (10Krinkle) [16:28:05] 10WikimediaDebug, 10MediaWiki-Platform-Team (Radar): WikimediaBackend ignores the last change to "backend" dropdown - https://phabricator.wikimedia.org/T395190#10871567 (10dancy) https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/versions/ shows that version 3.1.0 is available. I receiv... [16:37:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch12 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:39:50] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871623 (10bking) 05In progress→03Reso... [16:47:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch12 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:52:28] RESOLVED: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cirrussearch12 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:54:44] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10871657 (10dancy) Thanks @bking! There is... [17:24:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10871773 (10Arnoldokoth) Thank you @KFrancis [17:26:19] 10Gerrit: Expanding commit in Gerrit automatically changes status of review to reviewed - https://phabricator.wikimedia.org/T395700 (10Adithyak1997) 03NEW [19:40:46] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Atom.oil.2 - https://phabricator.wikimedia.org/T395665#10872345 (10Aklapper) Sorry for the confusion and my question earlier, I obviously didn't check thoroughly. [19:58:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deleteme on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:58:29] 10Gerrit: Expanding commit in Gerrit automatically changes status of review to reviewed - https://phabricator.wikimedia.org/T395700#10872390 (10Umherirrender) That is a helpful feature when clicking throw a patch set. You can change it in the preferences: https://gerrit.wikimedia.org/r/settings/#DiffPreferences... [20:07:52] 10Beta-Cluster-Infrastructure, 07Epic: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709 (10Krinkle) 03NEW [20:07:57] Looks like my home is blocked again :) [20:08:13] 10Beta-Cluster-Infrastructure: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709#10872417 (10Krinkle) [20:17:18] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#10872453 (10Krinkle) Still seen. https://g... [21:06:42] Krinkle: since you saw the beta cluster block screen recently, can you confirm that it no longer tells folks to contact noc@? [21:07:12] T393404 [21:07:12] T393404: Beta cluster IP block page should not point to noc@wikimedia.org - https://phabricator.wikimedia.org/T393404 [21:08:01] Indeed, it does not. https://usercontent.irccloud-cdn.com/file/8xo1VFOv/Screenshot%202025-05-30%20at%2022.07.38.png [21:08:18] excellent [21:08:43] I'll get the hole opened up here soon hopefully [21:10:09] 10Beta-Cluster-Infrastructure: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709#10872578 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:10:24] 10Gerrit: Expanding commit in Gerrit automatically changes status of review to reviewed - https://phabricator.wikimedia.org/T395700#10872583 (10Aklapper) 05Open→03Invalid [21:20:54] !log Poked hole in blocked_nets for 188.214.8.0/21 (T395709) [21:20:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:56] T395709: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709 [21:20:56] 10Beta-Cluster-Infrastructure: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709#10872598 (10bd808) `lang=shell-session bd808@deployment-cache-upload08:~$ sudo -i puppet agent -tv ... Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/File[/etc/varnish/blocked-nets.inc.v... [21:21:36] Krinkle: hopefully it works for you now [21:22:04] that was a different range than we opened the last time [21:34:12] 10Beta-Cluster-Infrastructure: Unblock IPs for Beta Cluster: Hyperoptic UK - https://phabricator.wikimedia.org/T395709#10872670 (10bd808) 05In progress→03Resolved I did a follow up change to the hiera data to sort all of the blocked_nets networks using `sort -V`. That diff is at https://gerrit.wikimedia.... [21:37:44] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Beta cluster IP block page should not point to noc@wikimedia.org - https://phabricator.wikimedia.org/T393404#10872680 (10bd808) With the cherry-pick in place and the hiera customization the block screen looks something like this now: {F60922191,size=full} [21:48:57] Something is up with zuul and Jenkins... [21:49:26] zuul thinks it has a zillion things waiting for Jenkins to run them and Jenkins thinks it has nothing to do [21:49:48] thcipriani: ^ help? [21:54:44] It looks like two giant patch chains were pushed to gerrit around the same time and now zuul is super sad. [21:57:22] oh boy [21:57:38] so that's zuul merger backlog [21:58:15] https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions [21:58:48] merger:merge 2392 2 2 [21:58:56] 2392 merge jobs and two worker [21:58:57] s [21:59:12] yeah... [21:59:28] * bd808 trouts cscott and ollie [21:59:40] what repo are these for? [21:59:53] parsoid? [21:59:59] mediawiki/services/parsoid and mediawiki/extensions/Wikibase [22:00:21] those were the big chains [22:01:25] I was fumbling about and just realized that `gearadmin` and `gearman` are different commands. CLI UI fail. [22:03:22] well. [22:03:31] I killed all the parsoid and wikibase merger jobs [22:03:40] but now they're coming in for core [22:04:05] and the queue is just getting longer [22:04:11] yeah. it looks like cscott has a big core chain too [22:05:32] I see the bug referenced in the docs is a WONTFIX because it needs a newer zuul. So maybe in a month or so... [22:06:24] alright, killed it all [22:07:06] queue is dropping [22:09:53] jenkins is actually doing work again now too, so yay [22:11:34] https://grafana.wikimedia.org/goto/JtlI5BfNg?orgId=1 that was a heck of a queue depth [22:13:16] left angry comments rather than doing work: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/1152139 [22:13:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deleteme on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:14:16] csc guess I'll dm to ensure he does this...some other time [22:16:16] !log killed 1000s of zuul merger jobs via https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions for parsoid, wikibase, and core [22:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:17:17] been a while since that documentation has come in handy