[07:44:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [08:11:53] 10Diffusion, 10Phabricator, 06collaboration-services: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11218257 (10A_smart_kitten) > (What will update our GitHub mirror going forward?) I might be wrong, but I believe that most of our GitHub mirrors (I believe including `wiki... [08:19:57] 10GitLab (Infrastructure), 06Infrastructure-Foundations, 10SRE-tools: CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706 (10ABran-WMF) 03NEW [08:48:12] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: APT error when installing Jenkins package in releases instances - https://phabricator.wikimedia.org/T405352#11218374 (10jnuche) Well, that was unbelievably unlucky :sweat_smile: Thanks for resolving the situat... [09:12:15] 10Phabricator: Clean up invalid Dashboard chart panels in database - https://phabricator.wikimedia.org/T405710 (10Aklapper) 03NEW p:05Triageβ†’03Low [09:12:39] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Clean up invalid Dashboard chart panels in database - https://phabricator.wikimedia.org/T405710#11218455 (10Aklapper) [09:31:23] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11218496 (10Aklapper) Hi, there are Issues at https://github.com/cis-india/Wikisource-Reader/issues - do you plan to switch off Issues in GitHub in favor of using Wikimedia Phabricator? [09:55:43] 10Continuous-Integration-Config, 06Growth-Team, 10Metrics Platform, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11218587 (10phuedx) @thcipriani and I met to discuss this and we finished agreeing that perhaps mutual dependence in CI is the correct... [10:02:41] 10Continuous-Integration-Config, 06Growth-Team, 10Metrics Platform, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11218615 (10phuedx) [10:07:42] 10Continuous-Integration-Config, 06Experimentation Lab: Configure MetricsPlatform to depend on GrowthExperiments in CI - https://phabricator.wikimedia.org/T405715#11218630 (10phuedx) [10:17:38] 10Continuous-Integration-Config, 06Growth-Team, 10Metrics Platform, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11218678 (10phuedx) [10:40:18] 10Continuous-Integration-Config, 06Experimentation Lab (Experiment Platform Sprint 13), 07OKR-Work: Configure MetricsPlatform to depend on GrowthExperiments in CI - https://phabricator.wikimedia.org/T405715#11218758 (10phuedx) p:05Triageβ†’03High [10:40:26] 10Continuous-Integration-Config, 06Experimentation Lab (Experiment Platform Sprint 13), 07OKR-Work: Configure MetricsPlatform to depend on GrowthExperiments in CI - https://phabricator.wikimedia.org/T405715#11218760 (10phuedx) [10:56:35] 06Project-Admins: Create project tag for Product Safety and Integrity - https://phabricator.wikimedia.org/T405722 (10OKryva-WMF) 03NEW [11:08:03] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Consider adding a "Joined Projects" panel to default frontpage dashboard - https://phabricator.wikimedia.org/T405725 (10Aklapper) 03NEW p:05Triageβ†’03Low [11:14:25] 06Project-Admins: Create project tag for Product Safety and Integrity - https://phabricator.wikimedia.org/T405722#11218898 (10kostajh) https://phabricator.wikimedia.org/project/view/8218/ [11:16:13] (03open) 10taavi: monitoring: Add team tag for Toolforge/PAWS dependency bump tasks [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/92 [11:16:17] (03update) 10taavi: monitoring: Add team tag for Toolforge/PAWS dependency bump tasks [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/92 [11:55:38] GitLab needs a short maintenance break in 1 hour (13:00 UTC) [12:05:45] 06Project-Admins: Create project tag for Product Safety and Integrity - https://phabricator.wikimedia.org/T405722#11219008 (10Aklapper) 05Openβ†’03Resolved a:03kostajh Thanks Kosta! Standard text: In case you need to edit the project or project workboard itself at some point and lack permissions, please... [12:09:39] (03approved) 10jforrester: monitoring: Add team tag for Toolforge/PAWS dependency bump tasks [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/92 (owner: 10taavi) [12:09:41] (03merge) 10jforrester: monitoring: Add team tag for Toolforge/PAWS dependency bump tasks [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/92 (owner: 10taavi) [12:52:30] 10Phabricator: Request adding fnegri to members of acl*phabricator - https://phabricator.wikimedia.org/T405732 (10fnegri) 03NEW [13:05:59] GitLab maintenance finished [13:08:29] (03open) 10jelto: gitlab-runner: bump image version to alpine-v18.2.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/507 (https://phabricator.wikimedia.org/T405699) [13:24:53] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Graph] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1184874 (https://phabricator.wikimedia.org/T405550) [13:24:55] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Graph] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1184874 (https://phabricator.wikimedia.org/T405550) (owner: 10Jforrester) [13:26:31] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Graph] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1184874 (https://phabricator.wikimedia.org/T405550) (owner: 10Jforrester) [13:27:04] !log Zuul: [mediawiki/extensions/Graph] Archive, for T405550 [13:27:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:27:08] T405550: Archive the Graph extension - https://phabricator.wikimedia.org/T405550 [13:28:30] !log jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/Graph/ # T405550 [13:28:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:34:41] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1189831 (owner: 10Hslater) [13:36:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1189831 (owner: 10Hslater) [13:37:31] !log Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add dependency on PDFCreator [13:37:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:51:33] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request adding fnegri to members of acl*phabricator - https://phabricator.wikimedia.org/T405732#11219542 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper @fnegri: Here you go :) [13:58:59] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11219711 (10awight) p:05Triageβ†’03High We're okay waiting for the next train to fix the immedia... [14:05:33] (03PS1) 10CDanis: puppet: use Lua 5.3 for CI tests, for utf8 library [integration/config] - 10https://gerrit.wikimedia.org/r/1191701 (https://phabricator.wikimedia.org/T401383) [14:06:21] hi, hoping for a speedy review on ^ please, we broke some parts of Puppet CI without realizing it πŸ‘‰πŸ₯ΊπŸ‘ˆ [14:11:02] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request adding fnegri to members of acl*phabricator - https://phabricator.wikimedia.org/T405732#11219920 (10fnegri) Thanks <3 [14:15:00] (03CR) 10Ssingh: [C:03+1] "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1191701 (https://phabricator.wikimedia.org/T401383) (owner: 10CDanis) [14:16:32] hi folks. anyone around for a quick review and merge on https://gerrit.wikimedia.org/r/c/integration/config/+/1191701 ? it is blocking Puppet CI (and while it has a specific niche case on the invocation, you never know when you need it) [14:30:35] (03CR) 10Reedy: [C:03+2] puppet: use Lua 5.3 for CI tests, for utf8 library [integration/config] - 10https://gerrit.wikimedia.org/r/1191701 (https://phabricator.wikimedia.org/T401383) (owner: 10CDanis) [14:32:13] (03Merged) 10jenkins-bot: puppet: use Lua 5.3 for CI tests, for utf8 library [integration/config] - 10https://gerrit.wikimedia.org/r/1191701 (https://phabricator.wikimedia.org/T401383) (owner: 10CDanis) [14:32:20] Reedy: ah, you beat me. sukhe Reedy we'll need to bump the changelog, too [14:32:35] thanks for taking care of it Reedy <3 [14:32:50] thanks thcipriani! [14:32:56] Reedy: I've got a patch ready, 1 sec [14:34:40] (03PS1) 10Thcipriani: puppet: use Lua 5.3 for CI tests, changelog bump [integration/config] - 10https://gerrit.wikimedia.org/r/1191705 [14:34:40] (03PS1) 10Thcipriani: puppet: bump image version for Lua 5.3 utf8 [integration/config] - 10https://gerrit.wikimedia.org/r/1191706 [14:36:15] ahhh I always forget that [14:36:28] (03CR) 10Thcipriani: [C:03+2] puppet: use Lua 5.3 for CI tests, changelog bump [integration/config] - 10https://gerrit.wikimedia.org/r/1191705 (owner: 10Thcipriani) [14:36:35] thanks all! [14:37:18] a good chance for me to remember how to use dch [14:38:17] heh [14:38:27] (03Merged) 10jenkins-bot: puppet: use Lua 5.3 for CI tests, changelog bump [integration/config] - 10https://gerrit.wikimedia.org/r/1191705 (owner: 10Thcipriani) [14:38:35] thcipriani: Want me to deploy it? [14:38:40] well, build the images [14:38:49] obviously they need referencing too [14:39:03] yep, I made a followup for the reference in the job [14:39:25] Reedy: if you can trigger the rebuild, that'd be great <3 [14:39:59] log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1191701 https://gerrit.wikimedia.org/r/1191705 [14:40:46] (happy to do it as well if you're busy, but the wall of console output does trigger a visceral reaction in my lizard brain) [14:40:55] 🦎 [14:43:32] (03merge) 10dancy: gitlab-runner: bump image version to alpine-v18.2.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/507 (https://phabricator.wikimedia.org/T405699) (owner: 10jelto) [14:57:58] Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.9.8 [14:58:12] (03CR) 10Reedy: [C:03+2] puppet: bump image version for Lua 5.3 utf8 [integration/config] - 10https://gerrit.wikimedia.org/r/1191706 (owner: 10Thcipriani) [14:59:41] (03Merged) 10jenkins-bot: puppet: bump image version for Lua 5.3 utf8 [integration/config] - 10https://gerrit.wikimedia.org/r/1191706 (owner: 10Thcipriani) [14:59:49] <3 [15:00:08] thanks a lot folks! [15:03:44] sukhe: That... should be fully deployed now [15:09:10] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11220237 (10dancy) The deploy server does have the desired commit in the staging directory. ` dan... [15:09:44] thanks Reedy <3 [15:10:27] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11220243 (10dancy) [15:17:00] !log Upgraded gitlab-cloud-runners (staging and produ) gitlab-runner to v18.2.2 (T405699) [15:17:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:16] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: APT error when installing Jenkins package in releases instances - https://phabricator.wikimedia.org/T405352#11220288 (10Dzahn) a:03Dzahn [15:18:57] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11220302 (10dancy) I have also verified that the mediawiki container image used in production does... [15:33:08] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11220367 (10dancy) I think the commit reference on https://test.wikipedia.org/wiki/Special:Version... [15:38:26] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11220381 (10Saiphani02) We will primarily use Phabricator but will allow new people to create issues on GitHub. (We will transfer them to Phabricator if they are valid tasks) Is that okay? [15:43:57] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11220401 (10Dzahn) I recommend to avoid having 2 competing places for tickets and use only Phabricator. [15:56:29] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: APT error when installing Jenkins package in releases instances - https://phabricator.wikimedia.org/T405352#11220440 (10Dzahn) 05Openβ†’03In progress p:05Triageβ†’03Medium we are getting back to this on Mo... [16:02:49] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220474 (... [16:11:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11220496 (10Dzahn) If I am strict about the wording of this ticket then it's resolved. executor i... [16:24:46] hmm seems things have stalled on https://integration.wikimedia.org/zuul/ [16:24:51] at least specifically https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/13778/console [16:25:12] Which last wrote at 16:47:46 (uk time I think that's in?) [16:30:28] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220568 (... [16:32:25] (03open) 10dancy: git.py: Fix info() public commit selection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004 (https://phabricator.wikimedia.org/T405738) [16:32:27] (03update) 10dancy: git.py: Fix info() public commit selection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004 (https://phabricator.wikimedia.org/T405738) [16:35:07] paladox: Looks like https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/13778/ finished [16:35:29] yeh as soon as I wrote that and I looked back it started to move lol [16:36:01] (03update) 10dancy: git.py: Fix info() public commit selection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004 (https://phabricator.wikimedia.org/T405738) [16:56:12] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220641 (... [17:11:25] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220739 (... [17:13:16] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220745 (... [17:13:39] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11220751 (10Aklapper) Having two issue trackers is usually a source of confusion (where does the actual planning take place), fragmented conversations, and time wasted on duplicating already existing ticket... [17:17:54] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11220757 (10Dzahn) Strong support for the latest comment. That is exactly it. And in my experience the "we will transfer" part will quickly become annoying busy work with a lot of room for errors. Which the... [18:16:37] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220920 (... [18:19:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [19:00:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:00:35] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T405788 (10wmcs-alerts) 03NEW [19:03:55] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11221083 (... [19:04:44] 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11221084 (... [19:30:05] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul scheduler on zuul1001 - https://phabricator.wikimedia.org/T405118#11221129 (10Dzahn) [19:30:36] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Test new zuul test VMs - https://phabricator.wikimedia.org/T405120#11221130 (10Dzahn) [19:41:38] (03update) 10asmartkitten: Delete `bin/move_project` [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/101 (https://phabricator.wikimedia.org/T342275) [21:58:14] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T396381#11221489 (10thcipriani) 05Openβ†’03Resolved [23:30:08] (03approved) 10thcipriani: git.py: Fix info() public commit selection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004 (https://phabricator.wikimedia.org/T405738) (owner: 10dancy) [23:30:20] (03merge) 10thcipriani: git.py: Fix info() public commit selection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004 (https://phabricator.wikimedia.org/T405738) (owner: 10dancy)