[00:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:09:30] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [02:26:44] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [02:32:18] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:33:48] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [02:57:42] (03open) 10andrew: mangum templates: only include eqiad1 templates in eqiad1 &c [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/335 [02:59:56] (03update) 10andrew: mangum templates: only include eqiad1 templates in eqiad1 &c [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/335 [03:11:59] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [03:16:56] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:54:06] 10Toolforge, 06tools-platform-team: Map Documentation Resources to Questions Asked During Workshop - https://phabricator.wikimedia.org/T432677 (10komla) 03NEW [04:01:44] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [04:02:27] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [04:03:36] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [04:27:57] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [04:44:00] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [04:44:31] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [04:48:49] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [04:49:05] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:11:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:37:08] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12139692 (10Marostegui) [05:49:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:39:48] FIRING: PuppetFailure: Puppet has failed on clouddb1030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:54:15] (03update) 10dcaro: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [08:01:25] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687 (10dcaro) 03NEW [08:01:42] (03update) 10dcaro: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [08:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:20:50] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12140044 (10Jacob_WMDE) @taavi The AAAA record has now been created [08:33:13] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12140089 (10Marostegui) clouddb1030:3311 is now replicating. [08:46:56] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12140136 (10dcaro) [08:54:49] RESOLVED: PuppetFailure: Puppet has failed on clouddb1030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:12:11] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:16:26] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12140218 (10CWilliams-WMF) [09:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:30:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:57:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:22:13] (03update) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [10:33:41] (03update) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:35:00] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] `toolforge jobs load` uses PATCH for one-off jobs too - https://phabricator.wikimedia.org/T432592#12140722 (10Danysan1) I'm trying to use [[ https://gitlab.wikimedia.org/toolforge-repos/wikidata-geo-dumps/-/blob/d346bf5de72f808b116ab2218639c0b5559d08ae/to... [10:38:03] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12140741 (10taavi) @Jacob_WMDE this is what I see: `lang=shell-session taavi@runko:~ $ host explorer.wikiba.se explorer.wikiba.se has address 185.1... [10:41:23] (03update) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:42:13] (03update) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:47:11] (03update) 10dcaro: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [10:47:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12140778 (10dcaro) 05Stalled→03In progress [10:53:39] (03update) 10tlepage: Draft: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:54:22] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:57:33] 10Cloud-VPS, 06tools-infrastructure-team: Instance still in planning after three days - https://phabricator.wikimedia.org/T432712 (10Danysan1) 03NEW [10:59:12] 10Cloud-VPS, 06tools-infrastructure-team: Instance still in planning after three days - https://phabricator.wikimedia.org/T432712#12140886 (10Danysan1) [10:59:13] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS osmit-estratti: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429722#12140887 (10Danysan1) [11:00:34] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12140892 (10taavi) [11:00:57] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [11:01:54] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12140894 (10fnegri) Merged the patch, now attempting to deploy the demo server (https://toolhub-demo.wmcloud.org/) following https://wikite... [11:09:44] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12140944 (10fnegri) Got a bunch of exceptions during `make pull restart tail`: ` web-1 | 2026-07-21T10:59:35Z [none] elasticsearch WAR... [11:11:15] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12140945 (10cmooney) [11:40:19] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: nova.exception.GroupAffinityViolation while live-migrating tools-redis-6 - https://phabricator.wikimedia.org/T431282#12141066 (10fgiunchedi) Following up on this, the reason seems to be use of `anti-affinity` vs `soft-anti-affini... [11:46:18] 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#12141084 (10elukey) pki-db-1 and pki-root-1 replaced the last buster VMs. I already configured everything and made sure that the new pki-root... [11:53:14] 10Cloud-VPS, 06tools-infrastructure-team: Instance still in planning after three days - https://phabricator.wikimedia.org/T432712#12141100 (10fgiunchedi) I noticed the -trixie instance has hostname `osmit-estratti` and found this in the logs ` /var/log/nova/nova-conductor.log.4.gz:2026-07-17 21:38:33.193 3648... [12:01:23] (03open) 10filippo: server: default to HostToContainer propagated mounts [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/51 (https://phabricator.wikimedia.org/T432325) [12:05:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:06:10] 10Cloud-VPS, 06tools-infrastructure-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#12141120 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This has happened, I'm assuming by virtue of trixie upgrade? Optimistically resolving ` root@... [12:09:29] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12141134 (10MGoncalves-WMF) [12:10:56] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:11:14] FIRING: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:15:56] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:16:14] RESOLVED: [5x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:20:56] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:21:11] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:24:18] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] `toolforge jobs load` uses PATCH for one-off jobs too - https://phabricator.wikimedia.org/T432592#12141172 (10DamianZaremba) >>! In T432592#12140722, @Danysan1 wrote: That is exactly the bug this task covers. You can execute a `toolforge jobs run` passin... [12:25:56] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:29:08] (03open) 10waldir: First draft of the project [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/1 [12:29:35] (03merge) 10jforrester: First draft of the project [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/1 (owner: 10waldir) [12:35:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:38:03] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141236 (10cmooney) [12:40:34] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:40:37] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:42:08] (03update) 10tlepage: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:47:59] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12141325 (10jsn.sherman) looking good, but recommend deferring deletion of the old vm until we... [12:50:48] 10Cloud-VPS (Debian Bullseye Deprecation), 10LibUp: Replace libup-web02 (bullseye) - https://phabricator.wikimedia.org/T432728 (10taavi) 03NEW [12:51:17] 10Cloud-VPS (Debian Bullseye Deprecation), 10LibUp: Replace libup-web02 (bullseye) - https://phabricator.wikimedia.org/T432728#12141349 (10taavi) a:03taavi [12:53:46] (03merge) 10tlepage: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:57:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.549-20260721125400-935dde18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1344 [12:57:31] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.549-20260721125400-935dde18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1344 [12:59:22] (03update) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [13:04:59] (03update) 10dcaro: server: default to HostToContainer propagated mounts [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/51 (https://phabricator.wikimedia.org/T432325) (owner: 10filippo) [13:08:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:12:03] (03approved) 10arian-ar: Set last_login on first OAuth user creation [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/12 (owner: 10ebenezerrao) [13:12:11] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:12:13] (03merge) 10arian-ar: Set last_login on first OAuth user creation [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/12 (owner: 10ebenezerrao) [13:13:32] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12141398 (10Arian_Ar) 05Open→03Resolved @EbenezerRao Merged, Thanks for your contribution! [13:13:43] 10Tool-sentinel: Feed poller must not use long-lived MediaWiki Redis cache - https://phabricator.wikimedia.org/T431888#12141401 (10Arian_Ar) 05In progress→03Resolved [13:21:30] (03update) 10arian-ar: Set Secure flag on session cookie [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/13 (https://phabricator.wikimedia.org/T431897) (owner: 10priyankar22) [13:21:32] (03update) 10arian-ar: Set Secure flag on session cookie [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/13 (https://phabricator.wikimedia.org/T431897) (owner: 10priyankar22) [13:21:37] (03update) 10arian-ar: Set Secure flag on session cookie [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/13 (https://phabricator.wikimedia.org/T431897) (owner: 10priyankar22) [13:22:17] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in a 10-minute in-memory TTL cache [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [13:32:22] (03open) 10waldir: Simplify the prototype [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/2 [13:33:15] (03merge) 10jforrester: Simplify the prototype [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/2 (owner: 10waldir) [13:33:33] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141478 (10cmooney) [13:37:57] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141501 (10ops-monitoring-bot) Draining ganeti2032.codfw.wmnet of running VMs [13:40:27] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:46:31] (03open) 10waldir: Use the Wikimedia Toolforge mirror of CDNjs [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/3 [13:46:55] (03update) 10waldir: Use the Wikimedia Toolforge mirror of CDNjs [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/3 [13:47:14] (03merge) 10tlepage: jobs-api: bump to 0.0.549-20260721125400-935dde18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1344 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:47:14] (03update) 10tlepage: jobs-api: bump to 0.0.549-20260721125400-935dde18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1344 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:56:37] (03merge) 10jforrester: Use the Wikimedia Toolforge mirror of CDNjs [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/3 (owner: 10waldir) [13:57:50] (03open) 10jforrester: Footer: Add a link, licence, and a full-stop [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/4 [13:57:59] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141588 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=da979790-60cc... [13:58:28] (03merge) 10waldir: Footer: Add a link, licence, and a full-stop [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/4 (owner: 10jforrester) [14:00:03] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141591 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e7152add-70e3... [14:10:09] (03open) 10jforrester: Add French exonym/demonym coverage choropleth from Wikidata [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/5 [14:11:18] (03open) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 (https://phabricator.wikimedia.org/T429164) [14:20:12] (03update) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 (https://phabricator.wikimedia.org/T429164) [14:23:49] (03update) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [14:24:36] (03update) 10jforrester: Add French exonym/demonym coverage choropleth from Wikidata [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/5 [14:25:55] (03update) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [14:29:20] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141737 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=988f9bff-f6d2... [14:31:05] 06tools-infrastructure-team, 06Data-Platform-SRE, 13Patch-For-Review: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12141743 (10fgiunchedi) [14:33:29] (03merge) 10waldir: Add French exonym/demonym coverage choropleth from Wikidata [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/5 (owner: 10jforrester) [14:33:42] (03update) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 (https://phabricator.wikimedia.org/T429164) [14:34:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:36:43] (03update) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 (https://phabricator.wikimedia.org/T429164) [14:38:24] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:38:51] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:39:19] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12141790 (10cmooney) Switch is back up after upgrade and things look good at first glance. [14:41:18] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873#12141798 (10fgiunchedi) 05Open→03Resolved This is done -- there might be codfw noise from the... [14:41:19] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Incident Severity 3, 07Wikimedia-Incident: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12141801 (10fgiunchedi) [14:45:55] (03open) 10jforrester: Add legend status filter and Wikidata edit link [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/6 [14:48:14] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:49:21] (03approved) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [14:49:45] (03merge) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [14:50:25] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol2006-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:52:11] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:52:11] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:53:08] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.551-20260721144956-8fc863bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1345 (https://phabricator.wikimedia.org/T403735) [14:53:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.551-20260721144956-8fc863bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1345 (https://phabricator.wikimedia.org/T403735) [14:53:26] RESOLVED: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:55:25] FIRING: [10x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:58:41] (03PS1) 10Brouberol: define the airflow_dumps client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1313212 [15:00:32] (03CR) 10Brouberol: [C:03+2] define the airflow_dumps client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1313212 (owner: 10Brouberol) [15:00:50] (03CR) 10Brouberol: [V:03+2 C:03+2] define the airflow_dumps client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1313212 (owner: 10Brouberol) [15:01:54] (03open) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/337 (https://phabricator.wikimedia.org/T429164) [15:05:34] (03merge) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/337 (https://phabricator.wikimedia.org/T429164) [15:06:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:06:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:07:29] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:10:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:10:49] (03open) 10waldir: Swap to MapLibre and use globe view [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/7 [15:11:05] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:13:44] (03approved) 10jforrester: Swap to MapLibre and use globe view [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/7 (owner: 10waldir) [15:13:47] (03merge) 10jforrester: Swap to MapLibre and use globe view [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/7 (owner: 10waldir) [15:14:27] (03approved) 10waldir: Add legend status filter and Wikidata edit link [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/6 (owner: 10jforrester) [15:18:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:19:15] (03close) 10andrew: Rearrange datacenter separation of magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/336 (https://phabricator.wikimedia.org/T429164) [15:19:20] (03close) 10andrew: mangum templates: only include eqiad1 templates in eqiad1 &c [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/335 [15:19:53] (03open) 10andrew: Magnum templates: correct node_image [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/338 [15:20:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:20:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:20:12] (03merge) 10andrew: Magnum templates: correct node_image [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/338 [15:20:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:21:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:22:39] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:22:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:23:58] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:26:52] (03open) 10andrew: magnum templates: correct codfw1dev fixed_subnet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/339 [15:26:56] (03merge) 10andrew: magnum templates: correct codfw1dev fixed_subnet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/339 [15:27:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:27:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:27:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:28:25] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:32:48] (03open) 10jforrester: Footer: state the ODbL explicitly [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/8 [15:33:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:33:32] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745 (10fgiunchedi) 03NEW [15:33:38] (03update) 10jforrester: Add a Wikidata edit link to the country panel; split out JS/CSS, and lint [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/8 [15:34:02] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12142110 (10fgiunchedi) [15:34:48] (03update) 10dcaro: server: default to HostToContainer propagated mounts [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/51 (https://phabricator.wikimedia.org/T432325) (owner: 10filippo) [15:35:30] (03approved) 10dcaro: jobs-api: bump to 0.0.551-20260721144956-8fc863bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1345 (https://phabricator.wikimedia.org/T403735) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:35:35] (03merge) 10dcaro: jobs-api: bump to 0.0.551-20260721144956-8fc863bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1345 (https://phabricator.wikimedia.org/T403735) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:36:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] use `launcher` also for health-check script commands - https://phabricator.wikimedia.org/T403735#12142123 (10dcaro) a:03dcaro [15:36:48] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] use `launcher` also for health-check script commands - https://phabricator.wikimedia.org/T403735#12142125 (10dcaro) 05Open→03Resolved [15:38:52] (03merge) 10waldir: Add a Wikidata edit link to the country panel; split out JS/CSS, and lint [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/8 (owner: 10jforrester) [15:39:46] (03unapproved) 10waldir: Add legend status filter and Wikidata edit link [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/6 (owner: 10jforrester) [15:40:51] (03open) 10jforrester: Filter the map by clicking a legend status [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/9 [15:40:55] (03close) 10jforrester: Add legend status filter and Wikidata edit link [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/6 [15:41:30] 06tools-platform-team, 10Toolhub: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12142136 (10bd808) >>! In T425303#12140944, @fnegri wrote: > @bd808 if it all looks good to you, I'll proceed with deployment to staging and production. I spot... [15:42:54] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12142149 (10fnegri) Most of these are on newly-setup hosts, I expect some of these might clear after a reboot. /cc @Marostegui Anyway it's... [15:50:10] 10Tools, 06Commons: locator-tool 401 "Error" - https://phabricator.wikimedia.org/T429192#12142165 (10simon04) 05Open→03Resolved a:03simon04 Upstream issue: https://github.com/simon04/locator-tool/issues/92 Upstream fix: https://github.com/simon04/locator-tool/commit/9bfd43859ff9ad582a67f9d8b11e63e34... [15:53:28] (03merge) 10waldir: Filter the map by clicking a legend status [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/9 (owner: 10jforrester) [16:14:16] 06tools-infrastructure-team, 06Data-Platform-SRE, 13Patch-For-Review: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12142333 (10fgiunchedi) [16:16:59] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] use `launcher` also for health-check script commands - https://phabricator.wikimedia.org/T403735#12142360 (10DamianZaremba) Before; ` tools.cluebotng@tools-bastion-15:~$ kubectl get pod bot-57f4859f94-4rdc8 -o jso... [16:18:26] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12142379 (10jeena) [16:21:50] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12142406 (10taavi) Neither the linked task in Reason nor the parent task really explain to me what you are planning to use these resources for or how you ended... [16:21:52] (03update) 10dcaro: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [16:23:13] (03approved) 10dcaro: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) (owner: 10raymond-ndibe) [16:23:47] 10Toolforge, 06tools-platform-team: Map Documentation Resources to Questions Asked During Workshop - https://phabricator.wikimedia.org/T432677#12142416 (10komla) 05Open→03Resolved [16:46:36] 10Tool-sentinel, 06RoadToWiki: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12142558 (10EbenezerRao) [16:52:10] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [16:52:18] 10Cloud-VPS, 06tools-infrastructure-team: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12142613 (10Andrew) I'm interested in this comment: https://github.com/rook/rook/discussions/15403#discussioncomment-14701720 It mentions "Mor... [16:52:25] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [16:54:50] (03open) 10jberkel: Workaround for missing ISO3 code [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/10 [16:56:52] (03open) 10tburmeister: Edit index.md to replace RapiDoc link that goes to a dead/hacked page [toolforge-repos/api-spec-reader] - 10https://gitlab.wikimedia.org/toolforge-repos/api-spec-reader/-/merge_requests/3 [16:58:15] 10Toolforge, 06tools-platform-team: [jobs-api] crashes if k8s object does not match expected format (un-guarded list access) - https://phabricator.wikimedia.org/T432586#12142656 (10DamianZaremba) T403735 fixed the literal cause of this, but I'll leave it open as the pending MR handles the more general case of... [17:00:53] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12142672 (10HCoplin-WMF) 05Open→03Stalled a:03pmiazga [17:01:29] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12142679 (10HCoplin-WMF) a:05pmiazga→03None [17:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:02:59] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12142681 (10HCoplin-WMF) [17:04:53] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12142689 (10HCoplin-WMF) [17:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:12:40] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12142724 (10Andrew) This just happened again for gitlab-1002.devtools.eqiad1.wikimedia.cloud first sign of 'network unreachable' is at 2026-07-21T08:12:50.345128+00:00 [17:19:52] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12142782 (10Nemoralis) Also, if this is not possible, I can simply [[ https://docs.mailcow.email/post_installation/firststeps-disable_ipv6/ | disable IPv6 on... [17:25:45] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [17:27:33] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12142803 (10cmooney) 05Open→03Resolved All seems good, hosts have been repooled. [17:33:47] (03PS2) 10Majavah: build: Updating stylelint-config-wikimedia to 0.19.3 [labs/striker] - 10https://gerrit.wikimedia.org/r/1312234 (owner: 10Libraryupgrader) [17:33:48] (03PS1) 10Majavah: build: Pin a slightly older Tox version [labs/striker] - 10https://gerrit.wikimedia.org/r/1313270 [17:37:22] (03CR) 10Majavah: [C:03+2] build: Pin a slightly older Tox version [labs/striker] - 10https://gerrit.wikimedia.org/r/1313270 (owner: 10Majavah) [17:40:37] (03Merged) 10jenkins-bot: build: Pin a slightly older Tox version [labs/striker] - 10https://gerrit.wikimedia.org/r/1313270 (owner: 10Majavah) [18:09:31] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12142942 (10bd808) [18:09:32] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add PTR record for azwikimedia (mail.wikimedia.az) - https://phabricator.wikimedia.org/T421025#12142943 (10bd808) [18:10:11] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12142956 (10bd808) [18:24:58] 10Tool-wikiloves: Add Wiki Loves Ramadan to Wikiloves Statistics Tool - https://phabricator.wikimedia.org/T427971#12142984 (10Jony) @JeanFred @Danilo and @Lokal_Profil I have prepared a Pull Request to resolve this task. It adds the UI routes in app.py and whitelists the scopes in functions.py so the tool can pu... [18:27:10] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12142986 (10Andrew) dhclient was not running, no evidence of a crash or other things in syslog. Starting it caused a full recovery on the host. [18:55:25] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:07:51] 10Tool-sentinel: Remove redundant local import json in mediawiki.py - https://phabricator.wikimedia.org/T431907#12143075 (10EbenezerRao) a:05Arian_Ar→03EbenezerRao Hi! I'd like to work on this task. [19:10:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:15:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:44:23] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12143194 (10jeena) >>! In T432617#12142406, @taavi wrote: > Neither the linked task in Reason nor the parent task really explain to me what you are planning to... [19:46:38] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12143212 (10jeena) [20:11:40] 10Cloud-VPS, 06tools-infrastructure-team: Install and run dhcpd on VMs - https://phabricator.wikimedia.org/T432782 (10Andrew) 03NEW [20:22:43] 10Cloud-VPS, 06tools-infrastructure-team: Install and run dhcpd on VMs - https://phabricator.wikimedia.org/T432782#12143384 (10Andrew) Very much of the above is wrong. bookworm and trixie /do/ have systemd-networkd installed. So probably the parent task would've healed on its own in another 12 hours or so. [20:27:08] 10VPS-project-Wikistats: Add bolwiki to wikistats - https://phabricator.wikimedia.org/T429956#12143396 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ('bol','Bole','bòo pìkkà',8)... [21:15:55] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12143509 (10Andrew) This may be recurring on logging-logstash-04. Taavi (and now I) suspect that the issue is with systemd-networkd being unable to reach the dhcp server when renewing leases.... [21:20:43] (03open) 10kengkong: Functioning JS App [toolforge-repos/leximap] - 10https://gitlab.wikimedia.org/toolforge-repos/leximap/-/merge_requests/2 [21:20:54] (03approved) 10kengkong: Functioning JS App [toolforge-repos/leximap] - 10https://gitlab.wikimedia.org/toolforge-repos/leximap/-/merge_requests/2 [21:22:16] (03merge) 10kengkong: Functioning JS App [toolforge-repos/leximap] - 10https://gitlab.wikimedia.org/toolforge-repos/leximap/-/merge_requests/2 [21:26:59] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12143523 (10Andrew) This is a close fit for what we're seeing: https://github.com/systemd/systemd/issues/25441 (h/t to @taavi who found that) [21:33:48] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS osmit-estratti: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429722#12143538 (10Danysan1) I created the instance [[ https://horizon.wikimedia.org/project/instances/433d7b6d-65d0-411b-a81d-db75963a6a6e/ | osmi... [21:49:33] 10Cloud-VPS, 06tools-infrastructure-team: Instance still in planning after three days - https://phabricator.wikimedia.org/T432712#12143597 (10Danysan1) 05Open→03Resolved a:03Danysan1 Another instance with the name osmit-estratti exists in the same project. I had created that instance with the intenti... [22:10:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:15:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:46:14] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12143761 (10TLessa-WMF) Adding extra context here, shared by Rita (who supported the in... [22:55:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:36:18] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12143905 (10Kengkong1) Hi everyone! I hope you are all doing well. I really wi...