[00:03:02] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#10256905 (10Dzahn) ` [gerrit2003:~] $ id gerrit uid=925(gerrit) gid=925(gerrit) groups=925(gerrit) [gerrit2003:~] $ id gerrit2 uid=498(gerrit2)... [00:21:38] PROBLEM - Host gerrit2003 is DOWN: PING CRITICAL - Packet loss = 100% [00:23:25] that's a reboot [00:23:52] RECOVERY - Host gerrit2003 is UP: PING OK - Packet loss = 0%, RTA = 30.42 ms [00:36:34] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:37:55] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#10256965 (10Dzahn) There are many more because -xdev doesn't only exclude /proc but also /srv since that's a different filesystem too. Regardl... [00:40:54] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#10256985 (10Dzahn) This isn't all there is to it though, puppet creates plenty of things in the old path. WIP. ` du -hs /var/lib/gerrit2/re... [00:42:41] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#10256986 (10Dzahn) [00:42:59] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#10256988 (10Dzahn) [00:46:34] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [01:48:48] Project beta-scap-sync-world build #177633: 04FAILURE in 2 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/177633/ [01:50:10] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Wikimedia-production-error: 02:48:48 [15 hits] PHP Notice: Undefined variable: namespace - https://phabricator.wikimedia.org/T378046 (10Reedy) 03NEW [01:52:22] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Wikimedia-production-error: Intermittant "PHP Notice: Undefined variable: namespace" during beta-scap-sync-world - https://phabricator.wikimedia.org/T378046#10257171 (10Reedy) [01:57:20] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10257203 (10Reedy) [01:58:54] Yippee, build fixed! [01:58:54] Project beta-scap-sync-world build #177634: 09FIXED in 2 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/177634/ [02:00:54] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Wikimedia-production-error: Intermittant "PHP Notice: Undefined variable: namespace" during beta-scap-sync-world - https://phabricator.wikimedia.org/T378046#10257216 (10Reedy) [02:01:50] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Wikimedia-production-error: Intermittant "PHP Notice: Undefined variable: namespace" during beta-scap-sync-world - https://phabricator.wikimedia.org/T378046#10257220 (10Reedy) [02:02:58] Yay, another train blocker [02:05:18] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Wikimedia-production-error: Intermittant "PHP Notice: Undefined variable: namespace" during beta-scap-sync-world - https://phabricator.wikimedia.org/T378046#10257224 (10Reedy) [02:09:49] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10257234 (10Reedy) I couldn't see {T378048} on production logstash with a quick look, but maybe that's mostly due to the wikis it's being run on.. [02:12:05] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T375660#10257259 (10Reedy) [02:12:07] Or not, next week train blocker if it's not fixed :P [02:12:08] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10257260 (10Reedy) [02:21:35] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:31:35] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:25:43] Yippee, build fixed! [03:25:44] Project mwcore-phpunit-coverage-master build #3924: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3924/ [07:04:18] (03open) 10jelto: gitlab-runner: bump image version to alpine-17.3.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/414 (https://phabricator.wikimedia.org/T378016) [08:45:21] 06Project-Admins: Convert Codex Special:Block to milestone of Multiblocks - https://phabricator.wikimedia.org/T378066 (10KSiebert) 03NEW [08:49:21] (03approved) 10jnuche: gitlab-runner: bump image version to alpine-17.3.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/414 (https://phabricator.wikimedia.org/T378016) (owner: 10jelto) [09:33:35] dcausse: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/93745/2/maintenance/fuzz-tester.php :) [09:44:26] 06Project-Admins: Convert Codex Special:Block to milestone of Multiblocks - https://phabricator.wikimedia.org/T378066#10258087 (10Peachey88) @KSiebert As per the comment on the page, That script no longer exists and can't be used. Phabricator (Phorge) doesn't support this, The best process at the moment is: Re... [11:26:04] 06Project-Admins: Convert Codex Special:Block to milestone of Multiblocks - https://phabricator.wikimedia.org/T378066#10258289 (10KSiebert) Thanks that worked [12:06:29] (03PS1) 10Dreamy Jazz: [mediawiki/extensions/WikimediaEvents] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082749 (https://phabricator.wikimedia.org/T375508) [13:12:52] (03CR) 10Hashar: [C:03+2] [mediawiki/extensions/WikimediaEvents] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082749 (https://phabricator.wikimedia.org/T375508) (owner: 10Dreamy Jazz) [13:14:32] (03Merged) 10jenkins-bot: [mediawiki/extensions/WikimediaEvents] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082749 (https://phabricator.wikimedia.org/T375508) (owner: 10Dreamy Jazz) [13:34:25] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen), 10observability, 10Wikimedia-Logstash: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733#10258741 (10hashar) 05Open→03Declined [13:35:10] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1082749 (https://phabricator.wikimedia.org/T375508) (owner: 10Dreamy Jazz) [13:35:39] (03CR) 10Hashar: [C:03+2] Archive mediawiki/tools/cli [integration/config] - 10https://gerrit.wikimedia.org/r/1082519 (https://phabricator.wikimedia.org/T288502) (owner: 10Hashar) [13:36:53] (03Merged) 10jenkins-bot: Archive mediawiki/tools/cli [integration/config] - 10https://gerrit.wikimedia.org/r/1082519 (https://phabricator.wikimedia.org/T288502) (owner: 10Hashar) [13:44:47] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen), 07Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995#10258753 (10hashar) See also {T352319} for which I went to patch the postbuildscript. The `InterruptedException`... [13:51:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10258763 (10hashar) 05O... [14:20:29] 10Continuous-Integration-Config, 07Test-Coverage: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma - https://phabricator.wikimedia.org/T116808#10258883 (10hashar) 05Open→03Declined [14:27:30] (03PS1) 10Tchanders: Zuul: [mediawiki/extensions/CheckUser] Add GlobalPreferences dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082797 (https://phabricator.wikimedia.org/T377831) [14:28:04] 10Continuous-Integration-Infrastructure, 07Code-Health, 07JavaScript, 07Test-Coverage: Generate JavaScript code coverage reports for extensions - https://phabricator.wikimedia.org/T184657#10258917 (10hashar) 05Open→03Declined We did it for PHPUnit tests as reflected on https://doc.wikimedia.org/cov... [14:29:29] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 07Test-Coverage: [Task] generate patch code coverage on gerrit patch-set upload for wikibase.git - https://phabricator.wikimedia.org/T88435#10258948 (10hashar) →14Duplicate dup:03T74318 [14:37:24] (03CR) 10Máté Szabó: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add GlobalPreferences dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082797 (https://phabricator.wikimedia.org/T377831) (owner: 10Tchanders) [14:37:31] 10Continuous-Integration-Infrastructure, 10[DEPRECATED] wdwb-tech, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 07Test-Coverage: generate Wikibase.git code coverage on Jenkins - https://phabricator.wikimedia.org/T88434#10259008 (10hashar) 05Stalled→03Declined There are multiple tasks ab... [14:38:25] 10Release-Engineering-Team (Seen): Create QA Health scoreboard - https://phabricator.wikimedia.org/T108768#10259032 (10hashar) 05Open→03Resolved a:03hashar This was part of a program to improve testing practice at the foundation ten years ago. My guess is that it is largely covered by https://doc.wikim... [14:40:57] 10Continuous-Integration-Infrastructure, 10CoverMe, 10Wikidata, 10wmde-wikidata-tech, and 2 others: Generate coverage report for Wikidata extensions - https://phabricator.wikimedia.org/T185211#10258987 (10hashar) [14:41:04] 10Release-Engineering-Team (Seen), 07Code-Health, 07Epic, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#10259050 (10hashar) 05Open→03Resolved a:03hashar I am claiming this one to have resol... [14:41:19] 10Release-Engineering-Team (Seen), 07Code-Health, 07Epic, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#10259056 (10hashar) a:05hashar→03None [14:41:44] 10Continuous-Integration-Infrastructure, 10CoverMe, 10Wikidata, 10wmde-wikidata-tech, and 2 others: Generate coverage report for Wikidata extensions - https://phabricator.wikimedia.org/T185211#10258990 (10hashar) 05Open→03Resolved a:03hashar Wikibase itself is a different beast. I have declined t... [15:03:30] 10Continuous-Integration-Config, 10Automoderator, 10MediaWiki-extensions-CommunityConfiguration, 06Moderator-Tools-Team, 10Growth-Team (Current Sprint): CI: Make CommunityConfiguration a dependency of Automoderator - https://phabricator.wikimedia.org/T378092 (10Michael) 03NEW [15:06:29] (03PS2) 10Michael Große: Automoderator: Make CommunityConfiguration a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082263 (https://phabricator.wikimedia.org/T378092) [15:12:40] (03CR) 10Hashar: [C:03+2] Automoderator: Make CommunityConfiguration a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082263 (https://phabricator.wikimedia.org/T378092) (owner: 10Michael Große) [15:13:55] (03Merged) 10jenkins-bot: Automoderator: Make CommunityConfiguration a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082263 (https://phabricator.wikimedia.org/T378092) (owner: 10Michael Große) [15:16:32] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1082263 (https://phabricator.wikimedia.org/T378092) (owner: 10Michael Große) [15:16:59] (03CR) 10Michael Große: "Thank you! 🙏" [integration/config] - 10https://gerrit.wikimedia.org/r/1082263 (https://phabricator.wikimedia.org/T378092) (owner: 10Michael Große) [15:31:50] 10Continuous-Integration-Infrastructure, 07Jenkins: ContextMenu endpoint missing in Jenkins - https://phabricator.wikimedia.org/T284754#10259377 (10hashar) 05Open→03Declined I suspect it was an incompatibility issue between the JUnit format generated by PHPUnit and the JUnit / XUnit Jenkins plugin havi... [15:33:52] 10Continuous-Integration-Config, 10Automoderator, 10MediaWiki-extensions-CommunityConfiguration, 06Moderator-Tools-Team, 10Growth-Team (Current Sprint): CI: Make CommunityConfiguration a dependency of Automoderator - https://phabricator.wikimedia.org/T378092#10259391 (10Michael) 05Open→03Resolved... [15:50:11] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921#10259533 (10Aklapper) In combination with T362401 the setup seems to create issues for some folks: https://w... [15:50:26] just for you, dancy: https://arstechnica.com/ai/2024/10/anthropic-publicly-releases-ai-tool-that-can-take-over-the-users-mouse-cursor/ [15:52:32] Sign me up! [15:53:23] lol [15:55:43] 10Phabricator-Bot-Requests: Request bot account for @Stang - https://phabricator.wikimedia.org/T311902#10259539 (10Stang) 05Open→03Declined Withdraw my request, I don't think it's still needed here. [16:10:28] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10259596 (10dancy) [16:21:38] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10259623 (10dancy) Train is blocked at group1 due to T378108. [16:25:42] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10259649 (10dancy) >>! In T375659#10259623, @dancy wrote: > Train is blocked at group1 due to T378108. After receiving for in... [16:25:57] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10259650 (10dancy) [16:34:08] (03merge) 10dancy: gitlab-runner: bump image version to alpine-17.3.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/414 (https://phabricator.wikimedia.org/T378016) (owner: 10jelto) [16:40:26] 10Phabricator: Paragraph mysteriously vanished - https://phabricator.wikimedia.org/T378110#10259712 (10Pppery) [17:21:02] 10Deployments, 06Release-Engineering-Team, 06serviceops: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10259903 (10Scott_French) One additional data point in terms of whether `Recreate` is something we want to keep long term for mw-d... [17:53:30] 10Phabricator: Paragraph mysteriously vanished - https://phabricator.wikimedia.org/T378110#10260043 (10Aklapper) →14Duplicate dup:03T326398 [17:54:17] 10Phabricator (Upstream), 07Upstream: Writing {{#something:}} into a Phab comment makes text disappear - https://phabricator.wikimedia.org/T326398#10260045 (10Aklapper) [18:57:08] 10MediaWiki-Releasing, 05MW-1.43-release, 05Release: Release MediaWiki 1.43.0 - https://phabricator.wikimedia.org/T372295#10260339 (10cscott) [19:42:57] (03PS1) 10Scardenasmolinar: AutoModerator: Add DiscussionTools as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1082840 (https://phabricator.wikimedia.org/T372476) [20:09:15] (03update) 10dancy: Spiderpig Web UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471 [20:09:22] (03update) 10dancy: Spiderpig Web UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471 [20:12:40] (03update) 10dancy: Spiderpig Web UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471 [20:13:14] (03update) 10dancy: Spiderpig Web UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471 [20:18:14] (03merge) 10dancy: Spiderpig Web UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471 [22:13:42] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#10261087 (10Dzahn) This also needs to be fixed. It's the parameters of the rsync command for LFS syncing. ` Notice: /Stage[main]/Gerrit/File[... [22:57:14] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10261199 (10dancy) [22:58:41] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T375659#10261205 (10thcipriani) 05Open→03Resolved