[00:00:54] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:54] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:24] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:11:24] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [05:20:45] 06Project-Admins: Consider archiving #notice, #developer-notice, #sysadmin-notice - https://phabricator.wikimedia.org/T372431#10098611 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks everyone. Done. [07:05:32] 06Release-Engineering-Team, 13Patch-For-Review: Train: support standard NETRC in automatic-branch-cut job - https://phabricator.wikimedia.org/T373441#10098721 (10hashar) [08:08:44] 06Release-Engineering-Team: Automatic branch cut should skip already cut branch - https://phabricator.wikimedia.org/T373501 (10hashar) 03NEW [08:11:59] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T366965#10098786 (10hashar) >>! In T366965#10095227, @hashar wrote: > I did the push to explicitly fast forward the wmf branch to master... [08:13:55] hashar: did you manage to release gearman? Do you need help? [08:18:18] 10Phabricator maintenance bot, 10SectionTranslation, 10LPL Essential (LPL Essential 2024 Jul-Sep), 13Patch-For-Review: Complete enablement Section Translation in new wikis and make the process less manual for the future - https://phabricator.wikimedia.org/T371420#10098808 (10Pginer-WMF) Meanwhile Wikipedia... [08:57:56] gehel: nop but I made some progress yesterday ;) [08:58:12] the issue was actually in upstream CI infrastructure which has a mirror / caching system [08:58:23] which is unable to fetch the gearman-java library from archiva for whatever reason [08:58:36] but if I opt out from their caching system, it works. So that is an issue on their side [08:58:52] or I need to find the configuration that allows their CI to fetch from our Archiva and bypass their cache [08:58:57] I ll ask them :-] [08:59:19] so I have little need for access to sonatype OSS thing right now [08:59:30] but would probably need at some point if I need to adjust code for Java 17 [08:59:32] I'll let you know [08:59:42] good! let me know! [09:30:15] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T366965#10099068 (10hashar) I have filed two errors related to authentication: * {T373504} * {T373507} I haven't made train blockers be... [10:04:23] 10Continuous-Integration-Infrastructure, 06Data-Platform-SRE, 13Patch-For-Review, 07Upstream: Archiva release repository yields 404 for release repository breaking builds - https://phabricator.wikimedia.org/T373352#10099146 (10hashar) I have filed the proxy issue #upstream with https://github.com/jenkins-i... [10:10:11] [ERROR] Run 1: Unable to make protected native java.lang.Object java.lang.Object.clone() throws java.lang.CloneNotSupportedException accessible: module java.base does not "opens java.lang" to unnamed module @45c8abff [10:10:14] java is not happy :] [10:41:59] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/ParserMigration] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1067936 [10:57:33] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10099294 (10Jgiannelos) I think after the latest... [11:00:57] 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470#10099304 (10zeljkofilipin) [11:10:46] I have managed to migrate out of powermock (which is no more maintained since 2020, had not tags despite fixes merged) [11:10:53] all of that in exactly an hour since I found the above error [11:11:02] by powermock [11:11:04] bye [11:14:40] we shall see how it behaves ( https://github.com/jenkinsci/gearman-plugin/runs/29361618936 ) [11:14:47] * hashar lunches [11:52:45] <3 [12:33:54] 10Phabricator maintenance bot, 10SectionTranslation, 10LPL Essential (LPL Essential 2024 Jul-Sep), 13Patch-For-Review: Complete enablement Section Translation in new wikis and make the process less manual for the future - https://phabricator.wikimedia.org/T371420#10099502 (10KartikMistry) >>! In T371420#10... [13:21:29] 10Continuous-Integration-Infrastructure, 06Data-Platform-SRE, 13Patch-For-Review, 07Upstream: Archiva release repository yields 404 for release repository breaking builds - https://phabricator.wikimedia.org/T373352#10099630 (10hashar) From upstream: > > For a new / unknown external repository, the maven c... [13:39:08] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] "See I23e1634." [integration/config] - 10https://gerrit.wikimedia.org/r/1067936 (owner: 10Fomafix) [14:25:32] 06Release-Engineering-Team, 10docker-pkg, 06serviceops: Attach opencontainers image metadata to docker images - https://phabricator.wikimedia.org/T345070#10099843 (10elukey) Created T371549 to move the `Maintainer` field of the production-images repo's control files to team-specific (where possible). Ideally... [15:13:52] ah powermock is banned https://github.com/jenkinsci/plugin-pom/pull/847 [15:13:53] :D [15:13:57] it is obsolete lib anyway [15:22:15] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10100043 (10Eevans) 05Open→03Resolved a:... [15:26:35] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify Jenkins Gearman plugin works with Java 17 - https://phabricator.wikimedia.org/T373351#10100081 (10hashar) #upstream https://github.com/jenkinsci/gearman-plugin/pull/18 [15:26:58] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10100082 (10JayCano) I was added to this group, but it seems like I'm not part of the corresponding acl and I cannot create sprints or modify any existing proj... [15:27:06] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify Jenkins Gearman plugin works with Java 17 - https://phabricator.wikimedia.org/T373351#10100083 (10hashar) a:03hashar [15:30:14] https://github.com/jenkinsci/gearman-plugin/pull/18/commits/1b8b344d9dbd4cabb2011f592f6e4264a087b029 [15:30:25] which is really three lines changed [16:03:41] (03PS2) 10Matěj Suchánek: Make Phan load AbuseFilter for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 [16:11:11] (03CR) 10Hashar: [C:03+2] Make Phan load AbuseFilter for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [16:12:53] (03Merged) 10jenkins-bot: Make Phan load AbuseFilter for Thanks [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [16:14:33] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [16:15:41] (03CR) 10Hashar: [C:03+2] "Note that Phan does not fail on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Thanks/+/1056582 so apparently it does not need Abus" [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [16:23:51] (03CR) 10Matěj Suchánek: "The Phan errors had been suppressed. Now it will fail, then I will remove the suppressions." [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [16:41:54] 06Release-Engineering-Team, 06serviceops: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10100427 (10dancy) 05Resolved→03Open p:05Unbreak!→03Medium [16:52:00] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team (25Q1 (Jul–Sep)): Port routing on deployment-docker-wikifunctions01 port routing (?) seems broken, making Beta Cluster Wikifunctions orchestrator unable to talk to its evaluator - https://phabricator.wikimedia.org/T373464#10100500 (10Jdforrester-WMF) p:... [16:53:30] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team (25Q1 (Jul–Sep)): Port routing on deployment-docker-wikifunctions01 port routing (?) seems broken, making Beta Cluster Wikifunctions orchestrator unable to talk to its evaluator - https://phabricator.wikimedia.org/T373464#10100506 (10Jdforrester-WMF) [16:54:04] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team (25Q1 (Jul–Sep)): Port routing on deployment-docker-wikifunctions01 port routing (?) seems broken, making Beta Cluster Wikifunctions orchestrator unable to talk to its evaluator - https://phabricator.wikimedia.org/T373464#10100534 (10Jdforrester-WMF) [17:15:36] (03CR) 10Hashar: [C:03+2] "Ahh. That explains it. Thank you for your explanation!" [integration/config] - 10https://gerrit.wikimedia.org/r/1068035 (owner: 10Matěj Suchánek) [17:16:44] (03open) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:16:47] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:18:16] of course I have yet another issue, this time with `mvn spotbugs:check` which does not honor me having flagged a false positive @SuppressFBWarnings("DCN_NULLPOINTER_EXCEPTION") [17:18:16] bah [17:18:56] 06Release-Engineering-Team, 06serviceops: scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10100604 (10thcipriani) [17:18:58] https://github.com/spotbugs/spotbugs-maven-plugin/issues/558 :D [17:19:00] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T366965#10100605 (10thcipriani) [17:19:08] 06Release-Engineering-Team, 06serviceops: Investigate scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10100606 (10thcipriani) [17:19:34] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:20:52] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:21:05] 06Release-Engineering-Team, 06serviceops: Investigate scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10100611 (10thcipriani) Question for investigation: - What were the steps leading to this state? -- How did you check out the code for this... [17:28:27] I forgot to recompile ... [17:28:37] so `mvn clean compile spotbugs:check` [17:29:00] and surely my PR passes all checks now https://github.com/jenkinsci/gearman-plugin/pull/18 [17:29:04] after well.. two days :/ [17:29:09] java 17 \o/ [17:29:13] have a good day [17:29:25] (03open) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] (main-I88381bfb3ca8a5892131f99c432e0b5689e8db39) - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:29:25] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:29:25] (03update) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] (main-I88381bfb3ca8a5892131f99c432e0b5689e8db39) - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:29:29] (03update) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] (main-I88381bfb3ca8a5892131f99c432e0b5689e8db39) - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:29:30] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:30:00] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Investigate docker-hub-mirror PVC in Terminating state - https://phabricator.wikimedia.org/T373545 (10thcipriani) 03NEW [17:30:46] (03update) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] (main-I88381bfb3ca8a5892131f99c432e0b5689e8db39) - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:32:18] (03approved) 10dduvall: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 (owner: 10dancy) [17:33:19] (03update) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:33:38] (03merge) 10dancy: New docker_hub_mirror_pvc_size setting, separate for prod/staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/408 [17:33:40] (03update) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:34:50] (03approved) 10dduvall: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 (owner: 10dancy) [17:37:28] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Alerts for disk space issues in GitLab's Digital Ocean cluster - https://phabricator.wikimedia.org/T373548 (10thcipriani) 03NEW [17:41:24] (03merge) 10dancy: README.md: Revise PVC resize text to mention docker-hub-mirror [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/409 [17:46:29] (03open) 10dduvall: production: Increase docker-hub-mirror PVC size to 50G [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/410 [17:46:30] (03update) 10dduvall: production: Increase docker-hub-mirror PVC size to 50G [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/410 [17:47:03] (03approved) 10dancy: production: Increase docker-hub-mirror PVC size to 50G [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/410 (owner: 10dduvall) [17:47:18] (03merge) 10dduvall: production: Increase docker-hub-mirror PVC size to 50G [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/410 [19:41:45] (03approved) 10dduvall: Revise use of deployment history [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/418 (owner: 10dancy) [19:48:18] (03merge) 10dancy: Revise use of deployment history [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/418 [20:14:48] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10101144 (10thcipriani) >>! In T706#10100082, @JayCano wrote: > I was added to this group, but it seems like I'm not part of the corresponding acl and I cannot... [20:35:09] (03update) 10sandeeps: cli.py: add scap version to fatal error meessages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 (https://phabricator.wikimedia.org/T311080) [20:37:31] (03open) 10dancy: history.py: Add return type info to some methods [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/420 [20:37:34] (03update) 10dancy: history.py: Add return type info to some methods [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/420 [20:39:14] (03merge) 10dancy: history.py: Add return type info to some methods [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/420 [20:48:47] (03update) 10sandeeps: cli.py: add scap version to fatal error meessages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 (https://phabricator.wikimedia.org/T311080) [20:54:29] (03open) 10dancy: history.py: A couple more type declarations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/421 [20:54:31] (03update) 10dancy: history.py: A couple more type declarations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/421 [20:54:37] (03open) 10dancy: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422 [20:54:40] (03update) 10dancy: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422 [20:55:27] (03update) 10dancy: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422 [20:55:38] (03merge) 10dancy: history.py: A couple more type declarations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/421 [20:56:33] (03update) 10dancy: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422 [22:02:28] (03approved) 10dduvall: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422 (owner: 10dancy) [22:27:26] (03merge) 10dancy: history: Drop 'synced' flag. Just use 'completed' [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/422