[00:00:39] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:39] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:16:10] (03open) 10catrope: releases: Bump Codex to 1.11.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/32 [00:26:33] (03merge) 10lwatson: releases: Bump Codex to 1.11.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/32 (owner: 10catrope) [07:26:10] 10Beta-Cluster-Infrastructure, 10Math: Make MathML default rendering in Labs - https://phabricator.wikimedia.org/T371254#10080172 (10Physikerwelt) [07:27:47] 10Beta-Cluster-Infrastructure, 10Math: Make MathML default rendering in Labs - https://phabricator.wikimedia.org/T371254#10080173 (10Physikerwelt) 05Open→03Resolved a:03Physikerwelt [08:56:55] 10Phabricator, 06Release-Engineering-Team, 06DBA: "Index for table 'fact_intdatapoint' is corrupt" rendering a specific Phabricator project report chart - https://phabricator.wikimedia.org/T372996 (10Aklapper) 03NEW [09:02:16] 10Phabricator, 06Release-Engineering-Team, 06DBA: "Index for table 'fact_intdatapoint' is corrupt" rendering a specific Phabricator project report chart - https://phabricator.wikimedia.org/T372996#10080335 (10ABran-WMF) p:05Triage→03Medium [09:04:26] 10Beta-Cluster-Infrastructure: Suggestion: Give stewards userrights-interwiki on metawiki beta cluster and remove the global userrights, userrights-interwiki - https://phabricator.wikimedia.org/T188874#10080345 (10Leaderboard) 05Open→03Declined I think this is stale and needs to be declined. Beta-Cluster... [09:25:32] 10Phabricator, 06Release-Engineering-Team, 06DBA: "Index for table 'fact_intdatapoint' is corrupt" rendering a specific Phabricator project report chart - https://phabricator.wikimedia.org/T372996#10080393 (10ABran-WMF) 05Open→03Resolved a:03ABran-WMF index has been rebuilt: `set session sql_log_bi... [09:26:08] 10Phabricator, 06Release-Engineering-Team, 06DBA: "Index for table 'fact_intdatapoint' is corrupt" rendering a specific Phabricator project report chart - https://phabricator.wikimedia.org/T372996#10080401 (10Aklapper) Thank you! <3 [09:56:51] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 2 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10080652 (10Clement_Goubert) This is most likely caused by envoy terminating before mediaw... [13:27:04] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T366965#10081280 (10matmarex) [13:53:35] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06Security-Team, 10SRE-Access-Requests: Offboard Guergana Tzatchkova (WMDE) and Frederik Ring from WMF systems - https://phabricator.wikimedia.org/T372767#10081384 (10jhathaway) sure will do [14:15:39] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10081530 (10joanna_borun) p:05Triage→03High [14:16:17] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10081531 (10joanna_borun) p:05Triage→03Medium [14:17:21] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10081536 (10Jelto) After reviewing the `DENYLIST` and the nftables logs, we noticed that so... [14:31:19] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#10081637 (10joanna_borun) [14:31:27] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10081638 (10joanna_borun) [14:45:33] 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10081721 (10Andrew) releng people, still have interest in this? I still do :) [15:06:14] 10Continuous-Integration-Config, 10phan-taint-check-plugin: mw-tools-phan-demos-publish fails with: "configure: error: cannot run C compiled programs." - https://phabricator.wikimedia.org/T372887#10081808 (10Daimona) The job isn't run often, so all we know is that it completed successfully on July 3 (logs have... [15:23:10] 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikidata, 10Wikimedia-GitHub, 10wmde-wikidata-tech: Archive WMDE analytics Gerrit repositories - https://phabricator.wikimedia.org/T357697#10081886 (10karapayneWMDE) [15:27:10] 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10081899 (10bd808) {T372498} is semi-related work in that it is attempting to create a gitops system for working with Magnum... [15:33:07] !log Attempting to add `role::deployment_server::kubernetes` role to deployment-deploy04.deployment-prep [15:33:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:44:18] 10Release-Engineering-Team (Priority Backlog 📥), 10dev-images, 10MediaWiki-Docker, 07ARM support: Create arm64 image variants of releng/dev-images used by MediaWiki-Docker - https://phabricator.wikimedia.org/T272500#10081984 (10brennen) [15:48:01] 06Gerrit-Privilege-Requests: Request REMOVAL of membership in mediawiki/extensions/Lingo group for Foxtrott - https://phabricator.wikimedia.org/T372133#10081997 (10Foxtrott) >>! In T372133#10053669, @matmarex wrote: > Do you mean the notifications that you signed up for here? https://www.mediawiki.org/wiki/Git/R... [15:56:51] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 07Documentation: Enable parse_ci_job_timestamps in gitlab-settings and document use of FF_TIMESTAMPS in GitLab CI - https://phabricator.wikimedia.org/T367765#10082016 (10bd808) [15:57:08] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 07Documentation: Enable parse_ci_job_timestamps in gitlab-settings and document use of FF_TIMESTAMPS in GitLab CI - https://phabricator.wikimedia.org/T367765#10082013 (10bd808) "Feature flag parse_ci_job_timestamps... [15:57:12] 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10082021 (10dduvall) >>! In T353356#10081721, @Andrew wrote: > releng people, still have interest in this? I still do :) In... [15:57:34] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 07Documentation: Document use of FF_TIMESTAMPS in GitLab CI - https://phabricator.wikimedia.org/T367765#10082023 (10brennen) [16:04:12] 10GitLab (CI & Job Runners), 13Patch-For-Review: Images from registry.cloud.releng.team should be usable by the "wmcs" runners - https://phabricator.wikimedia.org/T372848#10082046 (10bd808) 05Open→03Resolved https://gitlab.wikimedia.org/bd808/deployment-prep-opentofu/-/jobs/348883: ` 2024-08-21T16:01:2... [16:16:46] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 3 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10082100 (10Lydia_Pintscher) [16:16:58] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 4 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10082102 (10Lydia_Pintscher) [16:35:48] remember where to check the version of MINA (sshd) used by gerrit? [16:36:03] it's not considered a plugin and not gerrit itself [16:36:33] grepping through /srv/deployment/gerrit but hmmmm [16:37:05] commands via ssh -p 29418 show me all the plugin versions but not this [16:39:36] 06Release-Engineering-Team: Try to get the role::deployment_server::kubernetes role working in deployment-deploy04.deployment-prep - https://phabricator.wikimedia.org/T373040 (10dancy) 03NEW [16:41:10] @seen paladox :) [16:42:45] ah, found it. telnet localhost 29418 :) [16:45:44] mutante: you need paladox? [16:47:03] RhinosF1: not anymore for this question. in general, sure :) [16:47:27] mutante: I can ping them on discord if you need them in future [16:47:54] RhinosF1: thanks, not needed for today, but good to know for the future [16:53:00] Project beta-code-update-eqiad build #509700: 04FAILURE in 48 ms: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/509700/ [16:53:54] !log Upgraded scap to 4.99.0 in beta [16:53:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:54:48] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: RESTBase deployment fails with unhandled error when there are empty checks defined - https://phabricator.wikimedia.org/T372921#10082318 (10dancy) I worked around the scap installation problem in beta. @Eevans Please try your deployment again! [16:54:59] urandom: Give your deployment a try again please! [16:57:11] (03open) 10dancy: release-scripts/update-scap-in-beta: Update deploy hostname [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/411 [16:57:15] (03update) 10dancy: release-scripts/update-scap-in-beta: Update deploy hostname [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/411 [16:58:58] (03merge) 10dancy: release-scripts/update-scap-in-beta: Update deploy hostname [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/411 [17:05:38] Yippee, build fixed! [17:05:38] Project beta-code-update-eqiad build #509701: 09FIXED in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/509701/ [17:27:34] dancy: \o/ [17:27:38] thank you! [17:44:38] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#10082504 (10bd808... [17:49:38] thcipriani, brennen: releases-jenkins is whining that it is on a soon to be EOL Java version. I didn't find any obvious Phab tasks about a server rebuild there, but thought y'all might know if there is one hiding somewhere I missed. Alternately, who should I tag on a task to get that on folks radar? [17:51:34] related: releases-jenkins seems to be very under documented on wikitech and mediawiki.org. I mostly found https://wikitech.wikimedia.org/wiki/Jenkins#Releases_Jenkins and some mentions of the name/URL in various project notes docs. [17:52:21] bd808: i'm not _aware_ of a task, but i will poke around a bit [17:52:35] bd808: yep, upgrade to java 17 is a known one, jnuche recently made a proposed update plan, so it's in our heads. Let me see if I can find anything on phab... [17:52:57] brennen: shame on you for not memorizing all phab tasks. ;) [17:53:20] we can't all be a.ndre [17:53:25] we just went through this for gerrit, so may be clustered together somewhere [17:53:40] https://phabricator.wikimedia.org/T359795 [17:53:56] releases-jenkins is a weird corner of things in general [17:54:56] just in the "this is important, but like probably only a handful of people even remember this exists" kind of way. [17:55:11] yeah [17:55:21] it's origins date to the long ago, but we only recently began using it for its original intended purpose [17:56:16] group -1 is maybe going to keep that intended purpose streak rolling. [17:56:20] b.rennen: You can be whoever you want, pinky promise! [17:56:52] foiled on the attempted non-mention yet again [17:57:56] lurkers gonna lurk ;) [17:58:27] "My regexes bring all the boys to the yard" [18:00:41] that just made me hear Lumpy Space Princess say "Oh my glob!" in my head. -- https://www.youtube.com/watch?v=DSqNLA1pWxk [18:01:19] awwww <3 [18:02:29] please tag collab. we should do this for releases hosts [18:02:55] also just now checking java version for .. hypothetical gerrit on bookworm [18:03:07] and the answer is you get jdk 17 by default [18:03:35] 10Continuous-Integration-Infrastructure, 07Jenkins: Switch Jenkins instances from Java 11 to Java 17 - https://phabricator.wikimedia.org/T359795#10082563 (10bd808) With {T334517} done I guess this just leaves the https://releases-jenkins.wikimedia.org hosts (`/^releases[12]003\.(codfw|eqiad)\./`) to upgrade to... [18:04:27] mutante: I added a note to T359795, but I can make a dedicated task if that would be easier to keep track of. [18:04:27] T359795: Switch Jenkins instances from Java 11 to Java 17 - https://phabricator.wikimedia.org/T359795 [18:04:52] bd808: I added the team tag, we can take it from there. it will now show up on Monday meeting [18:05:19] excellent, and thanks [18:05:27] the machines are already on bullseye, fwiw [18:05:37] but the java version setting is probably separate [18:07:14] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Switch Jenkins instances from Java 11 to Java 17 - https://phabricator.wikimedia.org/T359795#10082564 (10Dzahn) [18:07:30] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10082561 (10Eevans) Ok, status update: With th... [18:07:31] yea, it's specifically set to java 11 in Hiera [18:07:54] while on bullseye.. preparing patch because code review itself is probably easiest venue to discuss it [18:08:49] afair when we did the bullseye upgrade it was purposefully separated from the java upgrade [18:18:02] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: RESTBase deployment fails with unhandled error when there are empty checks defined - https://phabricator.wikimedia.org/T372921#10082618 (10dancy) 05Open→03Resolved [18:18:09] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: RESTBase deployment fails with unhandled error when there are empty checks defined - https://phabricator.wikimedia.org/T372921#10082619 (10dancy) Fixed in scap 4.99.0. [18:48:17] 10Beta-Cluster-Infrastructure: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051#10082686 (10dancy) [19:11:12] 10GitLab, 10Release-Engineering-Team (Seen): Enable Extensions Marketplace on GitLab Web IDE - https://phabricator.wikimedia.org/T372314#10082738 (10thcipriani) p:05Triage→03Low [19:16:58] 10Beta-Cluster-Infrastructure: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051#10082747 (10dancy) I fired `deployment-deploy03` back up to see how it was set up. Looks like it has a `/srv/deployment/repo` symlink pointing to `/srv/packages/public/`. This link does n... [19:32:25] !log Removed `role::aptly::client` from `deployment-prep` project Puppet (T373051) [19:32:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:32:27] T373051: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051 [19:46:46] 10Beta-Cluster-Infrastructure: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051#10082836 (10thcipriani) In case it's helpful. Previously, there was a CI job in jenkins that built the deb for scap and deployed. See [[https://github.com/wikimedia/integration-config/blob/... [19:47:25] 10Beta-Cluster-Infrastructure: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051#10082842 (10dancy) Since `deployment-deploy04` is no longer a source for deb packages in deployment-prep, I removed the `role::aptly::client` class from deployment-prep project puppet. Exi... [19:48:49] 10Beta-Cluster-Infrastructure: apt-get update fails in fresh deployment-prep VM - https://phabricator.wikimedia.org/T373051#10082844 (10dancy) 05Open→03Resolved p:05Triage→03Medium [19:55:32] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:01:35] PROBLEM - Work requests waiting in Zuul Gearman server on contint1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [20:10:32] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:10:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint1002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [21:24:09] (03update) 10sandeeps: cli.py: add scap version to fatal error meessages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 [21:26:47] (03update) 10sandeeps: cli.py: add scap version to fatal error meessages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 [23:22:28] (03open) 10dduvall: image: Remove inline credentials from remote context URLs [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/104 [23:22:29] (03update) 10dduvall: image: Remove inline credentials from remote context URLs [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/104 [23:22:38] (03update) 10dduvall: image: Remove inline credentials from remote context URLs [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/104 [23:22:41] (03update) 10dduvall: image: Remove inline credentials from remote context URLs [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/104