[00:09:39] (03approved) 10raymond-ndibe: functional-tests - builds-api - wrap command [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1327 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [00:10:12] (03merge) 10raymond-ndibe: functional-tests - builds-api - wrap command [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1327 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [00:10:23] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:10:37] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [00:18:11] (03PS1) 10Raymond Ndibe: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) [00:21:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [00:21:54] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:21:59] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:28:55] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:28:58] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:29:16] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [00:29:37] (03approved) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:42:18] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [00:45:09] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:45:14] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:45:23] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:45:33] (03merge) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [00:45:40] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [00:49:15] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.540-20260714004554-38eaaabf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1335 (https://phabricator.wikimedia.org/T431274) [00:49:24] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.540-20260714004554-38eaaabf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1335 (https://phabricator.wikimedia.org/T431274) [00:53:29] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [01:04:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:06:13] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [01:11:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12117634 (10Raymond_Ndibe) 05Open→03Resolved [01:18:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:22:34] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [01:23:25] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [01:24:22] (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) [01:25:31] (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) [01:26:01] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.540-20260714004554-38eaaabf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1335 (https://phabricator.wikimedia.org/T431274) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:26:07] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.540-20260714004554-38eaaabf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1335 (https://phabricator.wikimedia.org/T431274) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:27:30] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898 https://phabricator.wikimedia.org/T431410) [01:28:04] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [01:30:50] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T428898) [01:31:11] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [01:51:14] 10Tool-global-search: Global-search-tool should not display Wikinews - https://phabricator.wikimedia.org/T432063#12117677 (10Bugreporter) We can instead hide all closed wikis by default (but still allow users to display them). [02:07:57] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:45:02] (03CR) 10MusikAnimal: "I haven't done a full test of against my local wiki yet. I'll try to take care of that tomorrow." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [03:10:09] (03close) 10buxtang: fix: handle expected 404 in `deployment list` instead of erroring [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/83 [03:39:18] (03CR) 10MusikAnimal: [C:03+2] Repository: 503 on overload and lock waits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309874 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [03:40:01] (03Merged) 10jenkins-bot: Repository: 503 on overload and lock waits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309874 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [05:45:20] 10Tool-global-search: Global-search-tool should not display Wikinews - https://phabricator.wikimedia.org/T432063#12117845 (10Wargo) >>! In T432063#12117677, @Bugreporter wrote: > We can instead hide all closed wikis by default (but still allow users to display them). Yes. I think most searches are things to fix... [06:07:57] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:08:33] RESOLVED: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:40:44] 10Cloud-VPS, 06tools-infrastructure-team: Consider per-rack availability-zones in nova - https://phabricator.wikimedia.org/T432093 (10fgiunchedi) 03NEW [07:45:06] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12118080 (10fgiunchedi) I forgot to attach a task number, however: 07:44 -stashbot:#wikimedia-cloud- Logg... [08:11:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:11:18] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_toolsbeta_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:16:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDow [08:16:18] RESOLVED: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_toolsbeta_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:18:03] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#12118138 (10AGhirelli-WMF) 05In... [08:18:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Define a process for OAD resource synchronization - https://phabricator.wikimedia.org/T422927#12118141 (10AGhirelli-WMF) 05In progress→03Resolved [08:19:44] 10Toolforge, 06tools-platform-team: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12118143 (10DamianZaremba) Working as expected; ` tools.cluebotng@tools-bastion-15:~$ kubectl get pod -l app.kubernetes.io/name=bot -o... [08:21:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12118144 (10aputhin) a:03Raymond_Ndibe [08:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:41:18] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29318761866 (https://github.com/cluebotng/component-configs/commits/04593c4734db7ed505ba34446adacc22176c18ed) [08:41:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:45:57] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097 (10DamianZaremba) 03NEW [08:47:50] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12118252 (10DamianZaremba) Manually fixed with ` tools.cluebotng@tools-bastion-15:~$ toolforge jobs delete bot tools.cluebotng@tools-ba... [08:54:06] 10Toolforge, 06tools-platform-team: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12118259 (10DamianZaremba) Signals are not working as expected; ` tools.cluebotng@tools-bastion-15:~$ kubectl logs -f bot-74d9c5f44f-v... [08:55:20] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12118281 (10DamianZaremba) [09:00:35] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118300 (10CWilliams-WMF) [09:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:05:17] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12118309 (10fgiunchedi) As of now `dumps_use_nfs_lb: true` is deployed to toolsbeta, unfortunately for k8s... [09:05:27] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118310 (10CWilliams-WMF) [09:12:58] FIRING: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [09:17:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12118352 (10ops-monitoring-bot) Rebooting db2159.codfw.wmnet [09:20:33] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12118354 (10ops-monitoring-bot) Completed depooling of db2159 by cwilliams@cumin1003: Rebooting db21... [09:23:13] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12118364 (10fnegri) > Some of it came from testing out a patch on top of the one for this ticket: h... [09:31:48] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12118378 (10ops-monitoring-bot) Reboot of db2159.codfw.wmnet completed [09:31:55] 10Tool-wiki-gender-stats: Use a temporary favicon - https://phabricator.wikimedia.org/T432099 (10Danya) 03NEW [09:32:15] 10Tool-wiki-gender-stats: Use a temporary favicon - https://phabricator.wikimedia.org/T432099#12118392 (10Danya) [09:32:37] 10Tool-wiki-gender-stats: Use a temporary favicon - https://phabricator.wikimedia.org/T432099#12118394 (10Danya) p:05Triage→03Low [09:32:41] 10Tool-wiki-gender-stats: Use a temporary favicon - https://phabricator.wikimedia.org/T432099#12118396 (10Danya) 05Open→03In progress [09:33:13] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12118400 (10Danya) [09:33:14] 10Tool-wiki-gender-stats: Use a temporary favicon - https://phabricator.wikimedia.org/T432099#12118399 (10Danya) [09:34:05] (03open) 10danyya: Draft: Resolve T432099 "Use a temporary favicon" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/77 [09:35:14] (03update) 10danyya: Draft: Resolve T432099 "Use a temporary favicon" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/77 [09:44:46] (03update) 10danyya: Draft: Resolve T432099 "Use a temporary favicon" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/77 [09:45:01] (03update) 10danyya: Resolve T432099 "Use a temporary favicon" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/77 [09:46:02] (03merge) 10danyya: Resolve T432099 "Use a temporary favicon" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/77 [09:46:23] 10Tool-wiki-gender-stats, 13Patch-For-Review: Use a temporary favicon - https://phabricator.wikimedia.org/T432099#12118413 (10Danya) 05In progress→03Resolved [09:54:11] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118443 (10aputhin) [10:00:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12118463 (10CWilliams-WMF) > No reason, I based the multiinstance_reboot cookbook on the upgrade one... [10:03:01] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118478 (10aputhin) [10:07:50] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118486 (10TLepage-WMF) [10:07:57] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:24:35] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-cumin' [10:27:15] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118555 (10fnegri) [10:28:45] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118560 (10fnegri) [10:30:46] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-cumin' [10:35:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-cumin-2 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [10:45:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-cumin-2 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:08:19] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118704 (10cmooney) [11:08:58] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118705 (10taavi) [11:11:22] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-cumin-1 [11:11:45] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-cumin-1 [11:13:52] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 [11:14:16] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 [11:16:37] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12118738 (10cmooney) [11:16:56] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-4 [11:17:49] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-4 [11:18:09] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-5 [11:19:01] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-5 [11:19:20] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-6 [11:20:13] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-6 [11:23:12] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12118763 (10aputhin) [11:31:03] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Update Toolforge Cumin nodes off of Bullseye - https://phabricator.wikimedia.org/T401817#12118786 (10taavi) 05Open→03Resolved a:03taavi [11:31:26] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 [11:32:24] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 [11:34:15] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 [11:35:11] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 [11:35:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:38:46] (03merge) 10taavi: tools: Move Redis DNS record to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/118 (https://phabricator.wikimedia.org/T360704) [11:43:55] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud [11:43:58] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud [11:45:36] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118824 (10cmooney) [11:45:43] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud [11:45:51] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:46:00] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud [11:48:59] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118846 (10BTullis) I have fully drained dse-k8s-worker2003 and depooled cirrussearch2110. Good to go from our side. [11:49:13] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12118853 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro Looking [11:50:51] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:57:33] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431682) [11:57:36] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431682) [11:57:40] T431682: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682 [11:58:11] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048' (T431682) [12:02:44] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12118951 (10DamianZaremba) FWIW that job was re-created (https://phabricator.wikimedia.org/T430578#12118143) earlier today prior to the... [12:09:49] (03open) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [12:12:28] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048' (T431682) [12:12:36] T431682: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682 [12:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:18:15] (03update) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [12:19:28] (03PS1) 10Neriah: Update Codex from v2.5.1 to v2.6.0 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1310563 [12:19:47] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 [12:20:43] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 [12:21:18] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Upgrade Toolforge Redis cluster to Bookworm or newer - https://phabricator.wikimedia.org/T360704#12119033 (10taavi) 05Open→03Resolved [12:22:11] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-harbor' [12:24:08] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12119042 (10fgiunchedi) @VRiley-WMF I thought we could start with one host, cloudvirt1048 and test-run the whole procedure there. On... [12:26:44] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12119061 (10MGoncalves-WMF) [12:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:28:23] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-harbor' [12:37:32] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:39:18] 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: Upgrade Toolsbeta Harbor instance to Trixie to match tools - https://phabricator.wikimedia.org/T414475#12119123 (10taavi) a:03taavi [12:41:19] 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: Upgrade Toolsbeta Harbor instance to Trixie to match tools - https://phabricator.wikimedia.org/T414475#12119129 (10taavi) 05Open→03Resolved `toolsbeta-harbor-2` seems to be working as expected. [12:42:32] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:51:25] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12119157 (10Marostegui) [12:55:38] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119177 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8b130fc6-8cf0-455b-b0d3-cfde4861384c) set by cmooney@cumi... [12:57:32] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119202 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bdeaaae5-a950-451d-a6e0-365a469fd764) set by cmooney@cumi... [13:00:03] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 13Patch-For-Review: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12119205 (10taavi) 05Open→03Resolved Optimistically closing [13:01:56] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12119212 (10aputhin) p:05Triage→03High [13:02:30] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12119225 (10aputhin) p:05Triage→03Medium [13:06:55] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12119254 (10jijiki) a:05jijiki→03None [13:08:12] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12119267 (10taavi) [13:12:58] FIRING: [2x] JobsEmailerNoEmails: No emails sent in the last five hours - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [13:17:48] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12119323 (10taavi) ` Jul 14 13:10:07 cloudinfra-idp-2 tomcat10[19128]: 2026-07-14 13:10:07,020 INFO [org.apereo.cas.web.flow.DefaultDelega... [13:26:11] 06tools-platform-team: [jobs-emailer] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115 (10dcaro) 03NEW [13:27:43] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12119371 (10Marostegui) [13:29:17] (03update) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [13:35:31] (03update) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [13:35:34] 10Toolforge, 06tools-platform-team: [jobs-emailer] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12119411 (10taavi) [13:40:10] 10Toolforge, 06tools-platform-team: [jobs-emailer] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12119458 (10dcaro) @fgiunchedi on irc, probably not related but just in case: ` re: T432115 JFYI earlier this morning I deployed dumps-nfs lb to toolsbeta and did a worker roll-reboot... [13:42:43] 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119 (10atsuko) 03NEW [13:43:13] 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119#12119478 (10atsuko) [13:43:17] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12119481 (10dcaro) [13:48:43] FIRING: TooManyCloudnetsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudnetsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudnetsDown [13:48:48] FIRING: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [13:48:52] FIRING: TooManyCloudcontrolsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudcontrolsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudcontrolsDown [13:49:00] FIRING: TooManyCloudgwsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudgwsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudgwsDown [13:49:37] FIRING: [2x] ProbeDown: Service wan.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:50:30] FIRING: [5x] JobUnavailable: Reduced availability for job jmx_zookeeper in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:51:37] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119568 (10cmooney) Upgrade is now complete, all looks healthy. Thanks everyone for their assistance. [13:53:43] RESOLVED: TooManyCloudcontrolsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudcontrolsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudcontrolsDown [13:53:48] RESOLVED: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [13:53:52] RESOLVED: TooManyCloudgwsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudgwsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudgwsDown [13:53:56] RESOLVED: TooManyCloudnetsDown: No availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudnetsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudnetsDown [13:54:37] RESOLVED: [8x] ProbeDown: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:55:21] RESOLVED: [8x] JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:56:04] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12119596 (10fgiunchedi) I'm going through https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/rac... [13:58:23] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12119607 (10VRiley-WMF) Thanks @fgiunchedi! I will be running through this today. I just got back from vacation and catching up on a... [14:03:46] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12119651 (10fgiunchedi) >>! In T431682#12119607, @VRiley-WMF wrote: > Thanks @fgiunchedi! I will be running through this today. I ju... [14:06:58] (03open) 10filippo: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) [14:07:57] FIRING: CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:11:38] (03update) 10filippo: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) [14:12:18] (03update) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [14:12:22] (03update) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [14:20:16] (03open) 10dcaro: core: when something changed in storage, update runtime always [repos/cloud/toolforge/jobs-api] (handle_when_runtime_does_not_exist) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/352 [14:22:08] (03CR) 10Alien4444: Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [14:30:59] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudweb.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' [14:32:04] (03update) 10dcaro: core: when something changed in storage, update runtime always [repos/cloud/toolforge/jobs-api] (handle_when_runtime_does_not_exist) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/352 [14:36:08] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudweb.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' [14:36:40] (03update) 10dcaro: core: when something changed in storage, update runtime always [repos/cloud/toolforge/jobs-api] (handle_when_runtime_does_not_exist) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/352 [14:38:16] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12119897 (10dcaro) Found the fix, will test and deploy. [14:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:49:16] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12119936 (10dcaro) The job that generates emails in toolsbeta seems stuck: ` toolsbeta.sample-complex-app@toolsbeta-bastion-7:~$ toolforge jobs show generate-emails +-------... [14:50:34] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12119941 (10dcaro) Yep, something failing with mounts: ` apiVersion: v1... [14:55:46] (03approved) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [14:55:54] (03merge) 10dcaro: core: don't blow up if the runtime job does not exist [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/351 [14:56:01] (03update) 10dcaro: core: when something changed in storage, update runtime always [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/352 [14:59:29] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.541-20260714145624-f6ab46a5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T432097) [14:59:37] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.541-20260714145624-f6ab46a5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T432097) [15:09:32] (03CR) 10BryanDavis: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [15:16:16] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [15:16:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [15:17:36] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [15:18:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [15:18:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [15:25:28] RESOLVED: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [15:35:54] 10Tool-wiki-gender-stats: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12120244 (10Danya) 05In progress→03Open [15:36:42] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12120246 (10Raymond_Ndibe) [15:38:01] 10Tool-wiki-gender-stats: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#12120258 (10Danya) [15:38:31] 10Tool-wiki-gender-stats: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#12120260 (10Danya) 05Open→03In progress [15:38:37] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12120262 (10Danya) [15:38:38] 10Tool-wiki-gender-stats: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#12120261 (10Danya) [15:38:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:39:10] (03open) 10danyya: Draft: Resolve T427446 "Store labels in pinia stores" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/78 [15:42:15] (03merge) 10dcaro: core: when something changed in storage, update runtime always [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/352 [15:45:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.542-20260714154228-0565bc9c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T428230 https://phabricator.wikimedia.org/T432097) [15:45:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.542-20260714154228-0565bc9c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T428230 https://phabricator.wikimedia.org/T432097) [15:49:19] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [15:49:21] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:50:03] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [15:50:43] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Define a process for OAD resource synchronization - https://phabricator.wikimedia.org/T422927#12120325 (10OWresch-WMF) [15:50:46] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 05MW-1.47-notes (1.47.0-wmf.12; 2026-07-21), 07OKR-Work: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#12120323 (10OWresch-WMF) [15:50:56] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#12120329 (10OWresch-WMF) [15:51:03] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07): Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#12120333 (10OWresch-WMF) [15:51:11] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#12120337 (10OWresch-WMF) [15:52:15] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120342 (10Arnoldokoth) Thanks @Dzahn I got this from @brennen https://w... [15:54:49] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12120351 (10komla) Reworking this to expose aggregate counts through Prometheus. The main question I have is about dedu... [15:59:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:02:06] 10Toolforge, 06tools-platform-team: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12120386 (10aputhin) 05Open→03In progress [16:05:03] 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12120395 (10aputhin) 05In progress→03Resolved [16:06:07] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120397 (10Arnoldokoth) I also dumped the the database from `phabricator-b... [16:06:23] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120399 (10Dzahn) cool! congrats getting it to work! [16:08:50] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12120417 (10dcaro) We were able to free ~1T of space by deleting some data in locked accounts [16:11:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:12:12] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12120434 (10aputhin) 05Open→03In progress [16:13:35] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Refactor before webservice support - https://phabricator.wikimedia.org/T359804#12120443 (10aputhin) 05Stalled→03In progress [16:14:21] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12120451 (10aputhin) a:03dcaro [16:14:24] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12120452 (10dcaro) p:05Triage→03Medium [16:16:35] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12120473 (10BLiviero-WMF) Can you say more about database quotas, what do you anticipate the growth to be over time, to what maximum limit? [16:17:12] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [16:17:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [16:17:43] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add webservice support - https://phabricator.wikimedia.org/T362077#12120481 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:17:51] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add webservice support - https://phabricator.wikimedia.org/T362077#12120483 (10Raymond_Ndibe) 05Open→03In progress [16:18:14] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [16:18:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [16:22:14] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Make webservice-cli use jobs-api - https://phabricator.wikimedia.org/T431421#12120509 (10Raymond_Ndibe) 05Open→03In progress [16:26:04] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12120524 (10dcaro) @Count_Count Do you have a sample code I can use to test? (I'll figure out on my own but might h... [16:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:31:51] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [16:32:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:33:01] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:40:36] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [16:44:12] (03PS1) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [16:45:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:48:56] (03PS2) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [16:54:35] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12120656 (10fnegri) @CWilliams-WMF I pushed a new version of the patch with the `/srv` check and the... [17:00:51] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12120666 (10dcaro) >>! In T423549#12120350, @komla wrote: > Reworking this to expose aggregate counts through Prometheus... [17:04:29] (03CR) 10MusikAnimal: Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [17:05:22] (03update) 10danyya: Draft: Resolve T427446 "Store labels in pinia stores" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/78 [17:06:39] (03update) 10danyya: Resolve T427446 "Store labels in pinia stores" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/78 [17:06:44] (03merge) 10danyya: Resolve T427446 "Store labels in pinia stores" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/78 [17:07:06] 10Tool-wiki-gender-stats, 13Patch-For-Review: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#12120682 (10Danya) 05In progress→03Resolved [17:07:53] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150 (10MusikAnimal) 03NEW [17:08:00] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12120695 (10MusikAnimal) [17:08:17] 10Tool-wiki-gender-stats: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#12120698 (10Danya) p:05Medium→03High [17:08:34] 10Tool-wiki-gender-stats: Admin page - https://phabricator.wikimedia.org/T425293#12120700 (10Danya) p:05High→03Medium [17:09:31] (03approved) 10dcaro: jobs-api: bump to 0.0.542-20260714154228-0565bc9c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T428230 https://phabricator.wikimedia.org/T432097) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:09:39] (03merge) 10dcaro: jobs-api: bump to 0.0.542-20260714154228-0565bc9c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1336 (https://phabricator.wikimedia.org/T428230 https://phabricator.wikimedia.org/T432097) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:10:12] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12120704 (10dcaro) Oh, that bug link was wrong, there's no MR for this yet [17:10:32] 10Tool-wiki-gender-stats: Admin page - https://phabricator.wikimedia.org/T425293#12120718 (10Danya) [17:11:23] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12120723 (10Danya) [17:11:24] 10Tool-wiki-gender-stats: Fix Content-Security-Policy console error - https://phabricator.wikimedia.org/T427559#12120722 (10Danya) [17:11:36] 10Tool-wiki-gender-stats: Fix Content-Security-Policy console error - https://phabricator.wikimedia.org/T427559#12120725 (10Danya) 05Open→03In progress [17:12:47] (03open) 10danyya: Draft: Resolve T427559 "Fix Content-Security-Policy console error" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/79 [17:16:07] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12120786 (10jsn.sherman) [17:19:03] (03open) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [17:22:35] 10Cloud-VPS (Quota-requests), 10WMIT-Infrastructure: Quota increase request for project osmit - https://phabricator.wikimedia.org/T432153 (10Danysan1) 03NEW [17:22:39] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12120852 (10CWilliams-WMF) @fnegri thanks! Yes, I was just looking at the code, so I will give it a... [17:23:37] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS osmit-estratti: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429722#12120864 (10Danysan1) [17:23:38] 10Cloud-VPS (Quota-requests), 10WMIT-Infrastructure: Quota increase request for project osmit - https://phabricator.wikimedia.org/T432153#12120863 (10Danysan1) [17:24:08] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12120872 (10dcaro) This is sorted out, I'll create a task for the pods getting stuck in waiting for 5h+ and see how we surface that issue. [17:24:24] 10Toolforge, 06tools-platform-team: [jobs-emailer, toolsbeta] No emails sent in the last hour - https://phabricator.wikimedia.org/T432115#12120873 (10dcaro) 05In progress→03Resolved [17:28:36] 06tools-platform-team: [jobs-api] surface as error when a tool fails to start due to mount issues - https://phabricator.wikimedia.org/T432154 (10dcaro) 03NEW [17:30:05] 10Toolforge, 06tools-platform-team: [jobs-api] surface as error when a tool fails to start due to mount issues - https://phabricator.wikimedia.org/T432154#12120898 (10dcaro) [17:30:54] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12120908 (10dcaro) LGTM [17:31:54] 10Toolforge, 06tools-platform-team: [o11y] Start collecting uptime metrics for toolforge - https://phabricator.wikimedia.org/T412349#12120916 (10dcaro) [17:32:13] 10Toolforge, 06tools-platform-team: [o11y]: User Stats Baseline (Outreach Channels, Accounts, Activity) - https://phabricator.wikimedia.org/T422783#12120918 (10dcaro) [17:32:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS osmit-umap2: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429721#12120919 (10Danysan1) p:05Triage→03Low @McCio made a backup of the persistent data of this (currently broken and offline) uMap instance and... [17:32:25] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#12120924 (10dcaro) [17:33:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12120929 (10dcaro) [17:33:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [webservice-runner] Add pre-flight validity checks to the `webservice-runner` script in webservice images - https://phabricator.wikimedia.org/T431563#12120930 (10dcaro) [17:34:01] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12120946 (10dcaro) [17:34:04] 10Toolforge, 06tools-platform-team: [docs] How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12120949 (10dcaro) [17:41:26] (03update) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [17:43:32] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12120985 (10dcaro) 05In progress→03Resolved @DamianZaremba This is deployed now, I'll close but feel free to reopen if you see... [17:44:04] (03update) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [17:47:04] (03update) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [17:50:29] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12121017 (10jsn.sherman) The deploy commit is merged and CI is green: the production branch ca... [18:07:57] FIRING: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:14:27] (03update) 10danyya: Draft: Resolve T427559 "Fix Content-Security-Policy console error" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/79 [18:14:45] (03update) 10danyya: Resolve T427559 "Fix Content-Security-Policy console error" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/79 [18:15:46] (03merge) 10danyya: Resolve T427559 "Fix Content-Security-Policy console error" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/79 [18:16:14] 10Tool-wiki-gender-stats, 13Patch-For-Review: Fix Content-Security-Policy console error - https://phabricator.wikimedia.org/T427559#12121082 (10Danya) 05In progress→03Resolved [18:17:47] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12121084 (10BLiviero-WMF) Also noting this ticket to let you know that this request has my team's attention now (we manage Cloud VPS) and will be looking at this more closely. Up u... [18:18:21] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12121086 (10bd808) I'm not sure if this is relevant to defining the measurement, but my normal workflow is to remain logged in... [18:20:32] 10Tool-wiki-gender-stats: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#12121090 (10Danya) [18:20:45] 10Tool-wiki-gender-stats: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337#12121092 (10Danya) [18:20:46] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12121091 (10Count_Count) >>! In T428230#12120524, @dcaro wrote: > @Count_Count Do you have a sample code I can use... [18:21:06] 10Tool-wiki-gender-stats: Localize table texts - https://phabricator.wikimedia.org/T423787#12121097 (10Danya) [18:21:39] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12121100 (10Gnoeee) [18:21:40] 10Tool-wiki-gender-stats: Update contrib guide to clarify position on AI-generated code - https://phabricator.wikimedia.org/T426758#12121101 (10Danya) [18:21:41] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12121102 (10Danya) [18:23:12] (03open) 10danyya: Draft: Resolve T426758 "Update contrib guide to clarify position on AI-generated code" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/80 [18:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:32:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:35:53] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161 (10Danya) 03NEW [18:35:57] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121166 (10Danya) p:05Triage→03Medium [18:36:04] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121167 (10Danya) [18:36:07] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121169 (10Danya) p:05Medium→03High [18:36:22] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121173 (10Danya) 05Open→03In progress [18:36:30] 10Tool-wiki-gender-stats, 13Patch-For-Review: Update contrib guide to clarify position on AI-generated code - https://phabricator.wikimedia.org/T426758#12121175 (10Danya) 05Open→03In progress [18:37:08] 10Tool-wiki-gender-stats: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121177 (10Danya) [18:37:10] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12121178 (10Danya) [18:37:19] (03open) 10danyya: Draft: Resolve T432161 "404 error for some pages" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/81 [18:39:31] (03update) 10danyya: Draft: Resolve T432161 "404 error for some pages" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/81 [18:39:32] (03update) 10danyya: Resolve T432161 "404 error for some pages" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/81 [18:40:12] (03merge) 10danyya: Resolve T432161 "404 error for some pages" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/81 [18:40:33] 10Tool-wiki-gender-stats, 13Patch-For-Review: 404 error for some pages - https://phabricator.wikimedia.org/T432161#12121183 (10Danya) 05In progress→03Resolved a:03Danya [18:44:07] 10Tool-wikinewsie: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162 (10Pharos) 03NEW [19:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:48:57] (03update) 10danyya: Draft: Resolve T426758 "Update contrib guide to clarify position on AI-generated code" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/80 [19:50:13] (03update) 10danyya: Resolve T426758 "Update contrib guide to clarify position on AI-generated code" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/80 [19:50:21] (03merge) 10danyya: Resolve T426758 "Update contrib guide to clarify position on AI-generated code" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/80 [19:50:44] 10Tool-wiki-gender-stats, 13Patch-For-Review: Update contrib guide to clarify position on AI-generated code - https://phabricator.wikimedia.org/T426758#12121378 (10Danya) 05In progress→03Resolved [19:59:44] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12121407 (10Bodhisattwa) [20:10:51] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12121462 (10DamianZaremba) Thanks @dcaro, I'll push some releases tomorrow (eu) late morning/afternoon and let you know if anything... [20:17:41] 10Toolforge, 06tools-platform-team: [jobs-api] surface as error when a tool fails to start due to mount issues - https://phabricator.wikimedia.org/T432154#12121477 (10DamianZaremba) >>! In T432115#12120872, @dcaro wrote: > This is sorted out, I'll create a task for the pods getting stuck in waiting for 5h+ and... [20:30:08] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12121512 (10Pa2chant.bis) 05Invalid→03Open Isn't it wikimedia wich manage Toolforge resource limits ? [20:34:16] (03PS1) 10Bking: Add secret for wikidata_platform thanos-swift [labs/private] - 10https://gerrit.wikimedia.org/r/1310652 (https://phabricator.wikimedia.org/T432167) [20:36:22] (03PS2) 10Bking: Add secret for wikidata_platform thanos-swift [labs/private] - 10https://gerrit.wikimedia.org/r/1310652 (https://phabricator.wikimedia.org/T432167) [20:36:50] (03CR) 10Bking: [C:03+2] Add secret for wikidata_platform thanos-swift [labs/private] - 10https://gerrit.wikimedia.org/r/1310652 (https://phabricator.wikimedia.org/T432167) (owner: 10Bking) [20:37:00] (03CR) 10Bking: [V:03+2 C:03+2] Add secret for wikidata_platform thanos-swift [labs/private] - 10https://gerrit.wikimedia.org/r/1310652 (https://phabricator.wikimedia.org/T432167) (owner: 10Bking) [20:47:07] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12121564 (10Pa2chant.bis) NB : here is the massage when I try to connect : upstream connect error or disconnect/reset before headers. reset reason: connection termination [22:07:57] FIRING: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:19:49] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12121943 (10Framawiki) Hey @Pathoschild , do you know what performance limits could be reached by the tool? Perhaps there may be entries in the logs that might indicate, for example, that it get k... [23:51:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown