[00:09:31] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11121973 (10thcipriani) >>! In T393487#11121892, @bd808 wrote: > HAProxy is failing to start be... [00:10:08] where do you even see useful logs for any of this? [00:10:44] haproxy's journalctl really just reports if it's running or not [00:10:54] thcipriani: the blocking happens on Varnish still. You should be able to see the header there [00:11:08] bd808: shoo :) [00:11:20] and thanks [00:12:46] the change is functionally that haproxy now sets the x-provenance header and the varnish acts based on it's contents. [00:12:51] * bd808 is not here [00:13:11] yeah, I was curious if I could see what that lua call was doing [01:16:10] 10Beta-Cluster-Infrastructure: Wiki markup - https://phabricator.wikimedia.org/T402996 (10W11228) 03NEW [01:31:03] Project mediawiki-core-doxygen build #13087: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13087/ [02:09:35] 06Project-Admins: Create project tag for T402996 - https://phabricator.wikimedia.org/T402998 (10W11228) 03NEW [02:14:50] 06Project-Admins: Create project tag for T402996 - https://phabricator.wikimedia.org/T402998#11122086 (10W11228) a:05wiki-helenatxuβ†’03W11228 [02:25:46] 14Phabricator (2025-08-19), 10Wikimedia-Phabricator-Extensions: Update Phabricator translations to 2024.35 release - https://phabricator.wikimedia.org/T399604#11122115 (10Pppery) 05Openβ†’03Resolved [02:31:25] Yippee, build fixed! [02:31:26] Project mediawiki-core-doxygen build #13088: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13088/ [08:25:02] (03CR) 10Hashar: [C:03+2] "Ouppps πŸ˜„" [integration/config] - 10https://gerrit.wikimedia.org/r/1182237 (https://phabricator.wikimedia.org/T398292) (owner: 10Samwilson) [08:26:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/TemplateData] Fix typo in 'Wikibase' [integration/config] - 10https://gerrit.wikimedia.org/r/1182237 (https://phabricator.wikimedia.org/T398292) (owner: 10Samwilson) [08:38:09] (03PS3) 10Hashar: tox: lint with Python3, Zuul with Python 2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1182196 [08:43:21] (03PS4) 10Hashar: tox: detect and lint python scripts without .py [integration/config] - 10https://gerrit.wikimedia.org/r/1182200 [08:52:51] (03CR) 10Hashar: [C:03+2] tox: lint with Python3, Zuul with Python 2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1182196 (owner: 10Hashar) [08:53:25] (03CR) 10Hashar: [C:03+2] "I have fixed the grammar in the commit message (and I really need a plugin to do that in vim)." [integration/config] - 10https://gerrit.wikimedia.org/r/1182196 (owner: 10Hashar) [08:54:44] (03Merged) 10jenkins-bot: tox: lint with Python3, Zuul with Python 2.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1182196 (owner: 10Hashar) [08:56:54] (03PS5) 10Hashar: tox: detect and lint python scripts without .py [integration/config] - 10https://gerrit.wikimedia.org/r/1182200 [09:03:14] (03CR) 10Hashar: "That is a bit magic, but at least I have documented it in the commit message :] The negative pathspec `:!*.py` comes from Tyler." [integration/config] - 10https://gerrit.wikimedia.org/r/1182200 (owner: 10Hashar) [09:06:08] (03PS3) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [09:07:42] (03CR) 10CI reject: [V:04-1] utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [11:55:43] 10Phabricator (2025-08-26), 10Release-Engineering-Team (Doing 😎): Allow admins to rename personal Herald rules - https://phabricator.wikimedia.org/T386703#11123318 (10Aklapper) NOTE: As Personal Rules are personal, the UI shows "Send an email to rule author" as action. In "Edit Rule" mode that action says... [11:56:00] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Review and rename our existing Herald rules - https://phabricator.wikimedia.org/T402934#11123319 (10Aklapper) NOTE: As Personal Rules are personal, the UI shows "Send an email to rule author" as action. In "Edit Rule" mode that action says "Send m... [12:20:31] 10Beta-Cluster-Infrastructure: Request to be the member of the Cloud VPS project for the beta wikis - https://phabricator.wikimedia.org/T403048 (10hueitan) 03NEW [12:31:19] 10Beta-Cluster-Infrastructure: Request to be the member of the Cloud VPS project for the beta wikis - https://phabricator.wikimedia.org/T403048#11123554 (10Urbanecm_WMF) 05Openβ†’03Resolved a:03Urbanecm_WMF Done. Related conversation on IRC (`#wikimedia-cloud`): `lang=irc 13:56 hellow this is... [12:57:04] (03CR) 10Jforrester: Script to sync dependencies with extension registry (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1182115 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [13:10:50] (03PS4) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [13:11:28] (03CR) 10Krinkle: "Done. Switched to a new docker-updates env instead of re-using jjb." [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [13:11:56] (03PS5) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [13:13:29] Krinkle: looks good :] [13:13:55] (03PS6) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [13:14:18] ok, I'm done fiddling with it [13:18:59] (03CR) 10Hashar: [C:03+1] "Excellent and we can then adjust archive-repo to use the same system." [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [13:34:30] (03CR) 10Jforrester: [C:03+1] utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [13:34:43] (03PS1) 10Hashar: utils: add utils/archive-repo wrapper [integration/config] - 10https://gerrit.wikimedia.org/r/1182569 [13:35:45] (03CR) 10Hashar: "This is to align archive-repo with the wrapper you have introduced for docker-updates. This way we can invoke commands consistently:" [integration/config] - 10https://gerrit.wikimedia.org/r/1182569 (owner: 10Hashar) [13:36:54] (03CR) 10Hashar: [C:03+2] utils: add description to zuul-mw-jobs-runner [integration/config] - 10https://gerrit.wikimedia.org/r/1182036 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [13:38:25] (03Merged) 10jenkins-bot: utils: add description to zuul-mw-jobs-runner [integration/config] - 10https://gerrit.wikimedia.org/r/1182036 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [14:10:28] !log deployment-prep: Update cpjobqueue config to apply https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1182501 [14:10:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:21:22] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11124205 (10colewhite) Happened again today, but there was only a filter on `level:ERROR`.... [15:01:05] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10Fundraising analytics stack: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11124542 (10IAckerman-WMF) a:05JMandoβ†’03Jgreen [15:45:06] 10Continuous-Integration-Config, 10MediaWiki-extensions-Wikibase-Client, 10Quibble: EEXIST: file already exists on npm install in Selenium job - https://phabricator.wikimedia.org/T300018#11124917 (10Dreamy_Jazz) Maybe happened again?: https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/36065... [15:49:00] (03CR) 10Hashar: [C:03+2] Zuul, jjb: rename mwskin phan jobs to mwext [integration/config] - 10https://gerrit.wikimedia.org/r/1180572 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [15:50:35] (03Merged) 10jenkins-bot: Zuul, jjb: rename mwskin phan jobs to mwext [integration/config] - 10https://gerrit.wikimedia.org/r/1180572 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [15:51:04] twist [15:51:17] !log deployment-prep: Update cpjobqueue config to apply https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1182583 [15:51:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:51:29] SRE is now blocking traffic that lacks user-agent https://phabricator.wikimedia.org/T400119 :/ [15:52:39] hashar: yeah, it blocked one of our CIs too [15:52:52] https://phabricator.wikimedia.org/T402801 [16:00:22] my life is such a waste of time really [16:00:24] ... [16:06:27] 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089 (10hashar) 03NEW [16:06:31] 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125101 (10hashar) p:05Triageβ†’03Unbreak! [16:08:27] 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125110 (10hashar) [16:19:01] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125167 (10ssingh) [16:20:40] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125188 (10ssingh) We are looking into this, thanks for flagging. [16:24:35] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125199 (10ssingh) @hashar: As the message indicates, it would be preferable if we can set a user-agent as a start, even for an internal traffic requirement like this. Is... [16:33:04] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125236 (10ssingh) Reading into `./jjb-update` and `https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master`, it seems like this is being run... [16:34:27] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125243 (10hashar) The code base is legacy and I am not sure how to monkey patch it ([[https://opendev.org/jjb/python-jenkins/src/branch/master/jenki... [16:35:38] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125272 (10CDanis) As one option for a workaround, running `tunnelencabulator -s` after applying the above patchset will allow you to bypass this tem... [16:52:13] (03PS1) 10Hashar: Revert "Zuul, jjb: rename mwskin phan jobs to mwext" [integration/config] - 10https://gerrit.wikimedia.org/r/1182617 (https://phabricator.wikimedia.org/T402398) [16:52:23] (03CR) 10Hashar: [C:03+2] Revert "Zuul, jjb: rename mwskin phan jobs to mwext" [integration/config] - 10https://gerrit.wikimedia.org/r/1182617 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [16:54:07] (03Merged) 10jenkins-bot: Revert "Zuul, jjb: rename mwskin phan jobs to mwext" [integration/config] - 10https://gerrit.wikimedia.org/r/1182617 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [16:54:09] (03PS1) 10Hashar: Zuul, jjb: rename mwskin phan jobs to mwext [integration/config] - 10https://gerrit.wikimedia.org/r/1182618 (https://phabricator.wikimedia.org/T402398) [16:54:35] (03CR) 10Hashar: [C:03+2] "Sent again as https://gerrit.wikimedia.org/r/c/integration/config/+/1182618" [integration/config] - 10https://gerrit.wikimedia.org/r/1180572 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [16:59:00] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125364 (10ssingh) >>! In T403089#11125243, @hashar wrote: > The code base is legacy and I am not sure how to monkey patch it ([[https://opendev.org/... [17:11:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [17:17:44] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T402950#11125458 (10bd808) @Krinkle's patches are not rebasing cleanly on origin/production: * https://gerrit.wikimedia... [17:18:41] Krinkle: your varnish cherry-picks need rebasing. T402950 [17:18:42] T402950: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T402950 [17:21:18] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125467 (10hashar) I have filed that as an unbreak now cause that prevented me from deploying updates to jobs and I wasn't aware we would start block... [17:21:33] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125468 (10hashar) a:03hashar [17:22:20] James_F: we can't update Jenkins jobs anymore cause JJB does not set an user-agent and it is now blocked by our edge cache ( https://phabricator.wikimedia.org/T403089 ) [17:22:27] I'll figure out a solution tomorrow [17:22:30] Argh. [17:22:32] Whoops! [17:22:57] hashar: there is an easy workaround on task :) [17:23:09] * James_F tries that out. [17:23:28] and one potential solution, even, I think [17:26:16] THEY BROKE OUR KINGDOM OF LEGACY AND OBSOLETE STUFF! [17:26:24] anyway [17:26:25] I hate tox. [17:26:43] anyway adding an user-agent to python-jenkins / jjb is certainly over due [17:26:44] Apparently setenv doesn't do what I want it to. Or at least, not the way I did it. [17:26:48] ah [17:26:57] where do i start [17:27:12] TedX Β« how to learn to love and hate tox Β» [17:27:23] if you have tox v4 installed: good luck [17:27:26] I gave up on that [17:27:32] (03PS1) 10Jforrester: [WIP] Fix jenkins-jobs script to set a user agent [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) [17:27:43] ^^ Doesn't work, but maybe it's going in the right direction. Or not. [17:33:11] oh sorry James_F -- it's not an environment variable, it needs a small fork or monkey-patch of the library [17:34:48] (03CR) 10Hashar: [C:04-1] [WIP] Fix jenkins-jobs script to set a user agent (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) (owner: 10Jforrester) [17:35:00] James_F: yeah that is the wrong right direction that is good [17:35:18] so irionically as I was looking at what the hell is that `DEFAULT_HEADERS` for [17:35:27] it is not an environment variable that is read [17:35:35] you would have in the code some `os.environ.get()` [17:35:54] so I went to grep that in python-jenkins and surely it has ONE instance of reading an environment variable which is [17:35:58] [17:36:06] [17:36:12] [17:36:14] [17:36:15] JENKINS_API_EXTRA_HEADERS [17:36:42] which is doing exact WHAT THE NAME IMPLIES: [17:36:49] inject extra headers to api calls [17:36:50] so [17:37:24] setenv = JENKINS_API_EXTRA_HEADERS = User-Agent:wikimedia-jjb-update/1.0 [17:37:40] if that works, FIXED [17:37:42] and the task can be closed [17:37:45] I am off for dinner [17:37:48] James_F: congrats! [17:37:48] :) [17:42:48] (03PS2) 10Jforrester: Fix jenkins-jobs script to set a user agent [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) [17:42:49] Well, it works… [17:46:53] 10Beta-Cluster-Infrastructure: Update wikitech runbook on Beta Cluster traffic blocking/unblocking for x-provenance header - https://phabricator.wikimedia.org/T403103 (10bd808) 03NEW [17:47:23] 10Beta-Cluster-Infrastructure, 07Documentation: Update wikitech runbook on Beta Cluster traffic blocking/unblocking for x-provenance header - https://phabricator.wikimedia.org/T403103#11125629 (10bd808) p:05Triageβ†’03Medium [17:48:10] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T402950#11125631 (10Krinkle) 05Openβ†’03Resolved a:03Krinkle Removed one, rebased the other. [17:49:23] thanks for the Puppet fix Krinkle :) [17:52:28] (03CR) 10Krinkle: [C:03+1] "Works on my machine :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) (owner: 10Jforrester) [17:52:39] (03CR) 10Jforrester: [C:03+2] Fix jenkins-jobs script to set a user agent [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) (owner: 10Jforrester) [17:53:15] 10Beta-Cluster-Infrastructure: Remove need for manually applied MaxMind data hacks on Beta Cluster cache servers - https://phabricator.wikimedia.org/T403105 (10bd808) 03NEW [17:54:06] (03Merged) 10jenkins-bot: Fix jenkins-jobs script to set a user agent [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) (owner: 10Jforrester) [17:54:34] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125681 (10Jdforrester-WMF) 05Openβ†’03Resolved Should now be Resolved. [17:56:05] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125686 (10ssingh) >>! In T403089#11125467, @hashar wrote: > I have filed that as an unbreak now cause that prevented me from deploying updates t... [17:56:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [17:56:47] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T403107 (10wmcs-alerts) 03NEW [17:57:56] 10Beta-Cluster-Infrastructure: Remove need for manually applied MaxMind data hacks on Beta Cluster cache servers - https://phabricator.wikimedia.org/T403105#11125697 (10bd808) p:05Triageβ†’03Medium The hack that is currently in place on deployment-cache-text08 and deployment-cache-upload08 should keep working.... [18:02:48] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T402950#11125737 (10bd808) [18:02:49] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T403107#11125739 (10bd808) β†’14Duplicate dup:03T402950 [18:06:02] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep - https://phabricator.wikimedia.org/T402985#11125754 (10bd808) β†’14Duplicate dup:03T402919 [18:06:03] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep - https://phabricator.wikimedia.org/T402919#11125752 (10bd808) [18:09:30] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep - https://phabricator.wikimedia.org/T402919#11125764 (10bd808) `counterexample $ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving p... [18:13:46] hashar: Thanks to you and cdanis for working out what to fix. [18:18:05] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-urldownloader04 on project deployment-prep - https://phabricator.wikimedia.org/T402919#11125828 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/7fdfb72c7b762c001c6f215c1ea8229324e4f6c1%5E%21/#F0 ` dif... [18:28:04] !log Unblock 177.102.0.0/15 (T403059) [18:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:35:19] !log Unblock 47.196.0.0/14 (T403075) [18:35:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:37:08] bd808: just saw your task, is deployment-urldownloader04 bookworm or bullseye? [18:39:14] mszabo: las-release says bookworm [18:40:29] *lsb_release (typing is hard apparently) [18:40:55] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/url_downloader.yaml [18:41:17] that doesnt look good unless we have nginx-extras packages ourselves for bw [18:42:26] setting `profile::nginx::variant: 'extras'` fixed the Puppet run. Let me make sure nginx is running now [18:43:14] I think it is all running now? Β―\_(ツ)_/Β― [18:43:16] that was my bad, I didn't know about deployment-urldownloader04, should have included it under Hosts: [18:44:18] A thing you might not know/remember is that the "role" hiera settings do not apply in Beta Cluster (long story about Puppet lookup weirdness) [18:46:41] Yeah I did not know that at all sorry [18:48:46] I guess part of the confusion is that the nginx::variant setting was added a few months ago, but no actual extras features (lua) were utilized until my patch [19:24:10] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Unparsed newline in message - https://phabricator.wikimedia.org/T378535#11126194 (10Aklapper) a:03Aklapper https://gitlab.wikimedia.org/repos/phabricator/extensions/-/commit/3803cfdc87a0903c3ab61acc7... [20:00:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Traffic: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11126321 (10hashar) The script used to deploy CI job uses python-jenkins. We found out it reads the environment variable `JENKINS_API_EXTRA_HEADERS` which, as the name... [20:00:48] (03CR) 10Hashar: Fix jenkins-jobs script to set a user agent (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1182623 (https://phabricator.wikimedia.org/T403089) (owner: 10Jforrester) [21:00:33] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Investigate WMCS Magnum for GitLab runners - https://phabricator.wikimedia.org/T403125 (10thcipriani) 03NEW [21:09:57] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Investigate WMCS Magnum for GitLab runners - https://phabricator.wikimedia.org/T403125#11126522 (10bd808) Tofu seems to be the nicest way to deploy a Magnum cluster. The https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisionin...