[00:01:54] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:07:09] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:42:49] 10Continuous-Integration-Config, 10MobileFrontend, 07Developer Productivity, 10Web-Team-Backlog (Needs Prioritization (Tech)): Export Popups and MobileFrontend dist files as CI artifacts - https://phabricator.wikimedia.org/T352240#10173905 (10Jdlrobson) 05Open→03Stalled Should wait on outcome of T315929 [00:50:21] 10Continuous-Integration-Config, 10MobileFrontend, 07Developer Productivity: Export Popups and MobileFrontend dist files as CI artifacts - https://phabricator.wikimedia.org/T352240#10173937 (10Jdlrobson) [01:21:18] spam on https://phabricator.wikimedia.org/T175877 [07:40:12] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera, 13Patch-For-Review: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10174241 (10hashar) 05Open→03In progress [08:05:09] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.09.06 - 2024.09.27): Create a Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#10174337 (10Gehel... [08:05:21] (03CR) 10Hashar: [C:03+2] zuul: show change queues window value [integration/docroot] - 10https://gerrit.wikimedia.org/r/1071197 (owner: 10Hashar) [08:05:51] (03Merged) 10jenkins-bot: zuul: show change queues window value [integration/docroot] - 10https://gerrit.wikimedia.org/r/1071197 (owner: 10Hashar) [08:07:04] 10Release-Engineering-Team (Radar), 06Data-Engineering, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#10174364 (10Gehel) [08:07:35] 10Release-Engineering-Team (Radar), 06Data-Engineering, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#10174369 (10Gehel) →14Duplicate dup:03T367405 [08:08:04] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10174389 (10jnuche) a:05jnuche→03akosiaris [08:08:54] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Disambiguator] Add Scribunto as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075301 (https://phabricator.wikimedia.org/T71441) (owner: 10SD0001) [08:09:18] 10Scap, 10MW-on-K8s, 06serviceops: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10174383 (10jnuche) 05Open→03Resolved a:03jnuche Thank you @akosiaris! [08:10:17] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Disambiguator] Add Scribunto as a phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075301 (https://phabricator.wikimedia.org/T71441) (owner: 10SD0001) [08:18:42] (03CR) 10Hashar: [C:04-1] "That is used when testing `mediawiki/services/parsoid` which has a couple dependencies:" [integration/config] - 10https://gerrit.wikimedia.org/r/1075299 (owner: 10Reedy) [08:29:28] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Wikibase] Add SpamBlacklist as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075234 (owner: 10Dima koushha) [08:30:41] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Add SpamBlacklist as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075234 (owner: 10Dima koushha) [08:31:20] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1075234 (owner: 10Dima koushha) [09:11:54] (03CR) 10Hashar: "There are a few bash tricks." [integration/config] - 10https://gerrit.wikimedia.org/r/1074573 (https://phabricator.wikimedia.org/T282893) (owner: 10Thcipriani) [09:37:30] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10174702 (10hashar) The `ircservserv-wm` is no more configured, I just had to kick it from the channel. [09:37:33] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10174716 (10hashar) [09:59:11] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10174763 (10hashar) I have tried to set the redirect but that fails: ` lang=irc SET #wikimedia-quibbl... [12:15:56] (03PS1) 10Bartosz Dziewoński: Add dependencies from WikimediaMessages on MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 [12:16:45] (03CR) 10Bartosz Dziewoński: "I hope that this will resolve the failures on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/+/1075535" [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [12:18:37] (03PS4) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) [12:38:52] (03CR) 10CI reject: [V:04-1] DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) (owner: 10Kosta Harlan) [12:49:53] (03CR) 10Hashar: [C:04-1] "I don't think WikimediaMessages should depend upon anything." [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [12:53:25] (03CR) 10Jforrester: "> I have checked mediawiki-config and the wikifunctions wiki does not load MobileFrontend, so I guess it will have an issue whenever Wikim" [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [13:21:35] (03open) 10jnuche: Draft: WIP [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/455 [13:54:37] (03CR) 10Bartosz Dziewoński: "> I don't think WikimediaMessages should depend upon anything." [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [13:59:26] (03CR) 10Bartosz Dziewoński: "If you really want me to, then I could suppress the Phan warnings and skip the failing unit tests, and then we can pretend that MobileFron" [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [14:30:36] I filed T361044 about WikimediaMessages... [14:30:37] T361044: WikimediaMessages becoming a dumping ground for "common" code - https://phabricator.wikimedia.org/T361044 [14:51:18] Reedy: "becoming"? good one [14:51:40] :P [14:51:51] Yeah, at least 7 years of it... [15:06:23] 06Release-Engineering-Team, 10Scap, 10Phatality: Outstanding issues with scap phatality deployment - https://phabricator.wikimedia.org/T375294#10176060 (10thcipriani) [15:12:10] (03PS5) 10Kosta Harlan: DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) [15:30:33] (03CR) 10CI reject: [V:04-1] DNM: Check to reproduce parallel PHP test failures [integration/quibble] - 10https://gerrit.wikimedia.org/r/1075298 (https://phabricator.wikimedia.org/T375483) (owner: 10Kosta Harlan) [15:57:49] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.09.06 - 2024.09.27): Create a Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#10176280 (10Ottoma... [16:00:28] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10observability: Explore monitoring for the GitLab runners k8s cluster - https://phabricator.wikimedia.org/T363919#10176288 (10thcipriani) >>! In T363919#10142129, @lmata wrote: > Hi @thcipriani, I've checked with the team, and we'... [16:02:55] (03merge) 10dancy: Establish a table for storing jobrunner status [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/454 [16:02:58] (03update) 10dancy: WIP: HTTP API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/451 [16:41:27] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10176444 (10brennen) I get a `You are not authorized to perform this operation.` Have not really investigated... [16:45:13] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10176450 (10stwalkerster) @brennen Were you opped in #wikimedia-releng when you tried? You probably need to be... [16:46:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10176480 (10brennen) > @brennen Were you opped in #wikimedia-releng when you tried? You probably need to be op... [16:56:39] (03PS2) 10Michael Große: gatedextensions: Add dependency extension for CommunityConfiguration [integration/config] - 10https://gerrit.wikimedia.org/r/1075575 (https://phabricator.wikimedia.org/T373114) [17:16:30] thcipriani: brennen: I have confirmed the channel is gone. Thank you very much! [17:16:51] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:16:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901#10176646 (10hashar) 05In progress→03Resolved Tyler Brennen and I jumped in a video call and tried sett... [17:21:51] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:26:26] hashar: nice :) [17:35:09] 10Phabricator: unable to zoom images on mobile - https://phabricator.wikimedia.org/T375671 (10Novem_Linguae) 03NEW [17:36:12] 10Phabricator: unable to zoom images on mobile - https://phabricator.wikimedia.org/T375671#10176882 (10Novem_Linguae) [19:26:09] (03open) 10dancy: initialize-mediawiki-staging: Prepare /srv/mediawiki symlink [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/91 (https://phabricator.wikimedia.org/T329857) [19:26:11] (03update) 10dancy: initialize-mediawiki-staging: Prepare /srv/mediawiki symlink [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/91 (https://phabricator.wikimedia.org/T329857) [19:27:54] (03merge) 10dancy: initialize-mediawiki-staging: Prepare /srv/mediawiki symlink [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/91 (https://phabricator.wikimedia.org/T329857) [19:28:10] (03open) 10dancy: php-base: Include /etc/php/7.4/cli/php.ini, from prod deploy server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/92 [19:28:13] (03update) 10dancy: php-base: Include /etc/php/7.4/cli/php.ini, from prod deploy server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/92 [19:28:36] (03update) 10dancy: php-base: Include /etc/php/7.4/cli/php.ini, from prod deploy server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/92 [19:33:36] (03approved) 10dduvall: php-base: Include /etc/php/7.4/cli/php.ini, from prod deploy server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/92 (owner: 10dancy) [19:34:05] (03merge) 10dancy: php-base: Include /etc/php/7.4/cli/php.ini, from prod deploy server [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/92 [19:37:21] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [19:39:19] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [19:42:22] (03update) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [19:54:11] (03approved) 10dancy: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) (owner: 10dduvall) [20:12:46] (03PS1) 10Subramanya Sastry: MFE: Hide footer to suppress visualdiff noise [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075632 [20:13:41] (03CR) 10Subramanya Sastry: "Committing experimental tweaks that are already applied and tested on the server." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075632 (owner: 10Subramanya Sastry) [20:13:54] (03CR) 10Subramanya Sastry: [C:03+2] MFE: Hide footer to suppress visualdiff noise [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075632 (owner: 10Subramanya Sastry) [20:14:28] (03Merged) 10jenkins-bot: MFE: Hide footer to suppress visualdiff noise [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075632 (owner: 10Subramanya Sastry) [20:14:52] (03PS6) 10Subramanya Sastry: MFE: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [20:15:04] (03CR) 10Subramanya Sastry: [C:03+2] "We've tested these changes on parsing-qa-02 already ... get this in." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [20:15:37] (03Merged) 10jenkins-bot: MFE: Workaround various other known issues and force loading of images [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1075089 (https://phabricator.wikimedia.org/T375248) (owner: 10Jdlrobson) [20:54:50] (03merge) 10dduvall: mwscript: Containerize MediaWiki script execution [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/450 (https://phabricator.wikimedia.org/T369115) [20:58:54] hi folks, it's in the calendar, but I wanted to flag it ahead of time here too: we're planning to switch the active deployment server over to deploy2002 tomorrow (Thursday) starting some time a bit after 15:00 UTC. [20:59:20] I'll raise it again in -operations tomorrow well before starting work [21:52:48] swfrench-wmf: thanks for flagging doesn't look like that will conflict with any deploys, good luck with switchover :) [21:55:16] thcipriani: ack, and thank you :) [21:55:54] (relayed to train conductor and backup for the week as well) [21:56:52] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10177849 (10bd808) 05In progress→03Resolved I am going to close this tech spike task as completed. @rook should probably start a new... [21:59:24] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T373643#10177858 (10Jdlrobson) [22:00:04] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T373643#10177861 (10Jdlrobson) Adding a potential train blocker, but waiting for @ovasileva to confirm. [22:17:13] 10Beta-Cluster-Infrastructure, 10Add-Link, 10Growth-Team (FY2024-25 Q1 Sprint 6): refreshLinkRecommendation script fails in Beta cluster with FileNotFoundError - https://phabricator.wikimedia.org/T370792#10177905 (10Etonkovidova) 05Open→03Resolved [23:21:58] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T373643#10178026 (10Jdlrobson) Hi @brennen we'd like https://gerrit.wikimedia.org/r/1075652 to go out before rolling out group2. We'll try to coordinate with... [23:28:58] (03PS1) 10Jforrester: jjb: Drop wdio-selenium-daily-wikifunctions, broken and unlikely to be fixed [integration/config] - 10https://gerrit.wikimedia.org/r/1075653 [23:40:28] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaMessages] Add MobileFrontend as a test and phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:40:33] (03CR) 10Jforrester: "WFM." [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:41:38] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/WikimediaMessages] Add MobileFrontend as a test and phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:42:38] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaMessages] +MobileFrontend test & phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:44:33] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikimediaMessages] +MobileFrontend test & phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:45:59] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaMessages] +MobileFrontend test & phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1075543 (owner: 10Bartosz Dziewoński) [23:46:06] !log Zuul: [mediawiki/extensions/WikimediaMessages] +MobileFrontend test & phan dep [23:46:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL