[01:34:58] (03open) 10legoktm: Updates (April 2026) [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/1 [01:54:12] (03merge) 10legoktm: Updates (April 2026) [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/1 [02:45:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:50:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:05:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [03:22:37] 10Tool-global-search: Global search fails with HTTP 500 - https://phabricator.wikimedia.org/T286388#11788997 (101234qwer1234qwer4) It seems that this particular query is working now, so barring further examples of this, I suppose the task can be closed. [03:35:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [04:53:04] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789022 (10DavidBrooks) Can the new permission be defaulted to true for... [05:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:56:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:01:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:16:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [09:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:10:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:33:32] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789248 (10Urbanecm_WMF) This issue should now be fixable on the bot ow... [09:35:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:37:38] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789250 (10Aka) I can confirm that it works again after the permission... [09:40:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:48:41] (03PS1) 10Majavah: eslint: Remove comments from .eslintrc [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268204 [09:50:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:02:33] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789257 (10Novem_Linguae) 05Open→03Resolved a:03Urbanecm_WMF... [10:34:30] 10Tool-campwiz-nxt: Request to unlock GitLab account - https://phabricator.wikimedia.org/T422364#11789297 (10Amitkhatri89) [11:47:13] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370 (10Novem_Linguae) 03NEW [11:48:24] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11789471 (10Novem_Linguae) @Curb_Safe_Charmer, you gave me maintainer on GitHub and on Toolforge a year or two ago. Thank you for that. Are you OK with me self-merging patches on this repo? [11:50:30] 10Tool-refill, 07good first task: reFill UI still shows links to former maintainer’s user pages and GitHub account - https://phabricator.wikimedia.org/T421478#11789475 (10Novem_Linguae) I think we need to do {T422370} before even a small patch like editing text in the front end can be accepted. I have some tim... [11:53:12] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11789477 (10Novem_Linguae) [11:53:50] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11789478 (10Novem_Linguae) [12:02:04] 10Tool-refill, 07good first task: reFill UI still shows links to former maintainer’s user pages and GitHub account - https://phabricator.wikimedia.org/T421478#11789481 (10Curb_Safe_Charmer) >>! In T421478#11789475, @Novem_Linguae wrote: > I think we need to do {T422370} before even a small patch like editing t... [12:07:36] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11789492 (10Novem_Linguae) >>! In T422370#11789470, @Novem_Linguae wrote: > @Curb_Safe_Charmer, you gave me maintainer on GitHub and on Toolforge a year or two ago. Thank you for that. > > Are you OK with me self-m... [12:26:31] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/66 [12:26:31] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/38 [12:28:22] 10Data-Services, 06tools-infrastructure-team, 10Datasets-General-or-Unknown, 06Traffic: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11789526 (10Vgutierrez) [12:34:12] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/66 (owner: 10l10n-bot) [12:34:21] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/66 (owner: 10l10n-bot) [12:34:53] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/38 (owner: 10l10n-bot) [12:34:58] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/38 (owner: 10l10n-bot) [12:53:46] 10Tool-refill: icons are missing - https://phabricator.wikimedia.org/T422373 (10Novem_Linguae) 03NEW [12:56:22] 10Tool-refill: arrow button doesn't work - https://phabricator.wikimedia.org/T422374 (10Novem_Linguae) 03NEW [12:56:27] 10Tool-echo-chamber: Echo Chamber API error: You don't have permission - https://phabricator.wikimedia.org/T422318#11789599 (10Prototyperspective) Please don't delete any notifications (5 years or older or any other) before this tool has been functional for at least a while! The tool was linked in some within-w... [13:09:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s3 backend clouddb1023.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [13:11:17] 10Tool-refill: icons are missing - https://phabricator.wikimedia.org/T422373#11789625 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae https://github.com/CurbSafeCharmer/refill/commit/2921331e6ec77b9cae7ac01915517eab4cbd8f54 [13:14:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s3 backend clouddb1023.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [13:52:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:53:36] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:54:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:54:55] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:55:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:56:13] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:58:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:59:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:20:19] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T422384 (10magnusmanske) 03NEW [14:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:45:52] (03CR) 10Alien4444: [C:03+2] eslint: Remove comments from .eslintrc [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268204 (owner: 10Majavah) [14:46:36] (03Merged) 10jenkins-bot: eslint: Remove comments from .eslintrc [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268204 (owner: 10Majavah) [14:58:17] 06cloud-services-team, 10Quarry: [bug] Discuss still links to the mediawiki talk page when the documentation page has been moved to metawiki - https://phabricator.wikimedia.org/T422386 (10Prototyperspective) 03NEW [15:03:39] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#11789957 (10Novem_Linguae) I deleted it again today. It was at 13GB. So it grew by 13GB in 1 year. [15:04:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [15:08:11] 10Tool-refill: Dependabot PRs need updating - https://phabricator.wikimedia.org/T352367#11789980 (10Novem_Linguae) I'll probably end up turning off Dependabot. It just spams the PR queue. For my notes, there appear to be dependency lists in the following spots: * backend/Pipfile * backend/requirements.txt * we... [15:10:23] 10Tool-refill: arrow button doesn't work - https://phabricator.wikimedia.org/T422374#11789985 (10Novem_Linguae) This is fixed by modifying web/config.development.js to be... ` module.exports = { publicPath: '/', api: 'https://refill-api.toolforge.org' } ` ... and then restarting the webservice (`npm run de... [15:15:18] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790026 (10Novem_Linguae) If I'm reading the code correctly, we only need to deploy like 4 files to https://refill.toolforge.org/ for the front end. The 4 files created by the `npm run build` command: {F75160189}... [15:20:07] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11790091 (10DavidBrooks) I realize this task is closed, but I still... [15:27:00] 10Tool-refill, 07good first task: reFill UI still shows links to former maintainer’s user pages and GitHub account - https://phabricator.wikimedia.org/T421478#11790113 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae This is already fixed in the master branch. Just needs to be deployed. [15:34:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [15:39:47] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790170 (10Alachuckthebuck) Is git even installed on the tool? That seems like what happens when you scp files to the tool from a local machine but not clearing things. Or it’s log files. Can’t tell without looking... [15:51:02] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790189 (10Novem_Linguae) Good thinking. I found two git directories. Also, people `git clone`ing the repo instead of surgically adding the dist files. Also, three "versions" of the website: citoid, stable, and t... [15:53:59] (03merge) 10vriaa: fix: use banner ID instead of template ID when generating CSS selectors [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/50 [16:03:15] (03merge) 10vriaa: feat: add confirmation modal before deleting a banner [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/48 (https://phabricator.wikimedia.org/T420958) [16:05:18] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790220 (10Novem_Linguae) [16:21:09] 10Tool-refill: arrow button doesn't work - https://phabricator.wikimedia.org/T422374#11790277 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae https://github.com/CurbSafeCharmer/refill/commit/81dcf7d49370d72fb7207e0cb9fe8500d30cdf6a [16:26:59] 06cloud-services-team, 10Quarry, 13Patch-For-Review: [bug] Discuss still links to the mediawiki talk page when the documentation page has been moved to metawiki - https://phabricator.wikimedia.org/T422386#11790285 (10Aklapper) https://github.com/toolforge/quarry/pull/97 [16:29:52] (03update) 10vriaa: feat: add confirmation dialog when selecting a template with existing content [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/46 (https://phabricator.wikimedia.org/T420955) [16:30:29] (03merge) 10vriaa: feat: add confirmation dialog when selecting a template with existing content [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/46 (https://phabricator.wikimedia.org/T420955) [16:37:13] 10Tool-centralnotice-banner-editor: Add confirmation modal when deleting a banner - https://phabricator.wikimedia.org/T420958#11790343 (10Oyelola_Victoria) 05Open→03Resolved a:03Oyelola_Victoria [16:37:35] 10Tool-centralnotice-banner-editor: Add confirmation modal when selecting a new template while editing - https://phabricator.wikimedia.org/T420955#11790345 (10Oyelola_Victoria) 05Open→03Resolved a:03Oyelola_Victoria [16:40:32] 10Tool-refill: Preview/Save button doesn't work - https://phabricator.wikimedia.org/T422399 (10Novem_Linguae) 03NEW [16:41:19] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790375 (10Novem_Linguae) [16:42:44] (03update) 10vriaa: feat: auto-strip protocol from image URL input [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/47 (https://phabricator.wikimedia.org/T421070) [16:42:54] (03merge) 10vriaa: feat: auto-strip protocol from image URL input [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/47 (https://phabricator.wikimedia.org/T421070) [16:43:28] 10Tool-centralnotice-banner-editor: Automatically strip protocol from URLs - https://phabricator.wikimedia.org/T421070#11790397 (10Oyelola_Victoria) 05Open→03Resolved a:03Oyelola_Victoria [16:44:01] 10Tool-centralnotice-banner-editor: Allow bold to be applied to individual words or phrases within a text element - https://phabricator.wikimedia.org/T420943#11790401 (10Oyelola_Victoria) 05Open→03In progress p:05Triage→03High a:03Oyelola_Victoria [16:44:50] 10Tool-centralnotice-banner-editor: Implement fixed CSS feature for default uneditable styles in banners - https://phabricator.wikimedia.org/T420950#11790405 (10Oyelola_Victoria) 05Open→03In progress a:03Oyelola_Victoria [17:21:15] 10Data-Services, 06tools-infrastructure-team, 10Datasets-General-or-Unknown, 06Traffic: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11790589 (10xcollazo) CC @BTullis [17:23:37] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11790592 (10Alachuckthebuck) Maybe making somthing like https://github.com/chuckthebuck/bucksaltbot2/blob/main/scripts/toolforge-deploy-new-version.sh Could help make this a nonissue for future devs. [17:32:07] 10Tool-humaniki-2: Setup pre-commit hooks - https://phabricator.wikimedia.org/T422408 (10Danya) 03NEW [17:32:11] 10Tool-humaniki-2: Setup pre-commit hooks - https://phabricator.wikimedia.org/T422408#11790624 (10Danya) p:05Triage→03High [17:43:47] 10Tool-extjsonuploader: extjsonuploader update-and-upload job fails with "Error when saving : Invalid CSRF token." - https://phabricator.wikimedia.org/T421307#11790658 (10matmarex) Based on the timing, I will chalk this up to T421168. Please re-open if it is still happening. [17:44:15] 10Tool-extjsonuploader: extjsonuploader update-and-upload job fails with "Error when saving : Invalid CSRF token." - https://phabricator.wikimedia.org/T421307#11790667 (10matmarex) →14Duplicate dup:03T421168 [17:45:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:00:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:38:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:01:26] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11790940 (10matmarex) @DavidBrooks Indeed, "bot passwords" is a bit of a misnomer, and som... [19:37:20] (03update) 10danyya: Draft: Migrate to SQLite [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/3 [19:49:01] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11791037 (10Soda) @Novem_Linguae I got nerd-sniped onto this. The project is basically an [off-the-shelf PHP 8.2 container](https://k8s-status.toolforge.org/namespaces/tool-refill/pods/refill-5b9f7858c7-5xwl6/) that... [19:52:55] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11791039 (10Soda) (oh wait, y'all already figured this out, a few messages ago :) [20:00:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:10:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:33:04] 10Tool-echo-chamber: Echo Chamber API error: You don't have permission - https://phabricator.wikimedia.org/T422318#11791148 (10Urbanecm_WMF) This now needs to be fixed on the tool's side (expanding the Echo Chamber OAuth consumer's scope to include "read my notifications" should be sufficient). This can be done... [21:05:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:35:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:39:59] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add PTR record for azwikimedia (mail.wikimedia.az) - https://phabricator.wikimedia.org/T421025#11791414 (10Andrew) 05Open→03Resolved I think I've fixed both (!) things that were wiping out your ptr record. Please re-open if it vanishes again! [21:59:07] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11791442 (10bd808) >>! In T422224#11787862, @Hawkeye7 wrote: > Permission problem? If that question is about `[step-export] 2026-04-03T00:02:55.903753996Z 2026/04/03 00:02:55 warning:... [22:15:41] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11791514 (10bd808) > `2026-04-03T00:03:52Z [autoreport3-2wsls] [job] : line 1: heroku_output/AutoReport: No such file or directory` https://gitlab.wikimedia.org/toolforge-repos/milhis... [22:23:54] 10Tool-refill: figure out how to deploy the front end - https://phabricator.wikimedia.org/T422370#11791546 (10Novem_Linguae) @Alachuckthebuck, [[ https://discord.com/channels/221049808784326656/1123258346045198366 | join us on Discord if you'd like, in the refill channel ]]. This is getting complex enough that S... [22:36:25] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#11791571 (10bd808) I think this happened in {T380127} [22:42:22] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#11791590 (10bd808) {T422384} is reporting the bug I filed as {T394466} which was merged into this task but not actually fixed before the [[https://... [22:44:23] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T422384#11791595 (10bd808) It looks to me like the failure is: ` [step-build] 2026-04-06T14:04:59.855343648Z -----> Fetching .debs for php [step-build] 2026-04-06T14:04:59.987474659Z ERROR: failed to build: exit... [22:45:02] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails due to fagiani/apt and builder stack "24" mismatch - https://phabricator.wikimedia.org/T422384#11791608 (10bd808) [22:49:37] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11791612 (10Hawkeye7) @dcaro: What change do I need to make to the procfile to enable it to run the compiled binary? This always worked before and is currently working on the runs tha... [22:54:13] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11791626 (10bd808) >>! In T422224#11791612, @Hawkeye7 wrote: > This always worked before and is currently working on the runs that have not been rebuilt - there must have been a recent... [22:55:00] 06cloud-services-team, 10Toolforge: Running dotnet job fails on Toolforge because "24" builder stack changed the compiled binary output path - https://phabricator.wikimedia.org/T422224#11791629 (10bd808) [22:56:51] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#11791649 (10bd808) [22:56:52] 06cloud-services-team, 10Toolforge: Running dotnet job fails on Toolforge because "24" builder stack changed the compiled binary output path - https://phabricator.wikimedia.org/T422224#11791648 (10bd808) [22:56:53] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails due to fagiani/apt and builder stack "24" mismatch - https://phabricator.wikimedia.org/T422384#11791647 (10bd808) [22:56:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11791646 (10bd808) [23:04:17] FIRING: JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:08:44] (03close) 10raymond-ndibe: Draft: test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/187 [23:11:02] 10Tool-refill: Dependabot PRs need updating - https://phabricator.wikimedia.org/T352367#11791679 (10Novem_Linguae) @Curb_Safe_Charmer, any chance I could get some additional access to the Settings tab of the GitHub repo? I would like to turn off dependabot and then close the old dependabot PRs. [23:19:17] RESOLVED: JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:19:30] (03close) 10raymond-ndibe: toolforge-weld: add build_deb.sh [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/23 [23:20:42] (03close) 10raymond-ndibe: [maintain-harbor] persist log [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/42 (https://phabricator.wikimedia.org/T383081) [23:30:09] 10Tool-refill: in the GitHub repo, reorganize the /backend/ files so that the directory is named /refill-api.toolforge.org/, and so that all subfolders match what is deployed - https://phabricator.wikimedia.org/T422436 (10Novem_Linguae) 03NEW [23:30:39] 10Tool-refill: in the GitHub repo, reorganize the /backend/ files so that the directory is named /refill-api.toolforge.org/, and so that all subfolders match what is deployed - https://phabricator.wikimedia.org/T422436#11791715 (10Novem_Linguae) [23:37:23] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#11791727 (10Novem_Linguae) uwsgi.log in refill-api is also concerning. It's at 0.5 GB. I've deleted it. It appears to be a log of every page ever processed by the reFill back end. Interesting... [23:39:41] 10Tool-refill: in the GitHub repo, reorganize the /backend/ files so that the directory is named /refill-api.toolforge.org/, and so that all subfolders match what is deployed - https://phabricator.wikimedia.org/T422436#11791730 (10Novem_Linguae) [23:42:20] 06cloud-services-team, 10decommission-hardware: decommission cloudcephmon2004-dev - https://phabricator.wikimedia.org/T422437 (10Andrew) 03NEW [23:42:33] 06cloud-services-team, 10decommission-hardware: decommission cloudcephmon2004-dev - https://phabricator.wikimedia.org/T422437#11791748 (10Andrew) [23:42:35] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: cloudcephmon2007-dev service implementation - https://phabricator.wikimedia.org/T420282#11791747 (10Andrew) [23:52:01] 10Tool-refill: figure out how to deploy the back end - https://phabricator.wikimedia.org/T422439 (10Novem_Linguae) 03NEW [23:56:28] 10Tool-refill: figure out if we're still using internationalization/localization - https://phabricator.wikimedia.org/T422440 (10Novem_Linguae) 03NEW