[00:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:48:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:07:57] FIRING: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:20:44] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12122394 (10komla) @bd808 We want to get a sense of maintainers' activity levels, and we're using SSH authentication as a heu... [05:34:01] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12122396 (10komla) > I have a question though, what is it that you want to measure? Please see my response to Bryan's specifi... [06:06:28] (03update) 10countcount: Mirror per-wiki block logs into ToolsDB and serve them as detail=blockhistory [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:07:57] FIRING: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:08:23] (03update) 10countcount: Mirror per-wiki block logs into ToolsDB and serve them as detail=blockhistory [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:34:59] 10Tool-refill: Watchlist labels are reset for article when ReFill is used on it - https://phabricator.wikimedia.org/T432197 (10StefenTower) 03NEW [06:36:51] 10Toolforge, 06tools-platform-team: [docs] How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12122470 (10komla) [06:37:02] 10Toolforge, 06tools-platform-team: [docs] How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12122471 (10komla) 05In progress→03Resolved [06:38:15] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12122473 (10fgiunchedi) A side effect of the toolsbeta deployment from yesterday was {T432115} i.e. pods g... [06:43:25] (03update) 10countcount: Mirror per-wiki block logs into ToolsDB and serve them as detail=blockhistory [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:43:41] (03update) 10countcount: Mirror per-wiki block logs into ToolsDB and serve them as detail=blockhistory [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [07:15:11] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#12122532 (10Pankhudi_Singh) Hi! I'm new to Wikimedia and would like to work on this task. Could someone confirm... [07:21:05] (03update) 10raymond-ndibe: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 (owner: 10dcaro) [07:22:19] (03update) 10raymond-ndibe: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 (owner: 10dcaro) [08:00:18] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12122623 (10DamianZaremba) >>! In T432097#12121462, @DamianZaremba wrote: > Thanks @dcaro, I'll push some releases tomorrow (eu) la... [08:00:29] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29399091032 (https://github.com/cluebotng/component-configs/commits/5880b5b641021f572e9dd6a3d76443db4a607114) [08:00:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:06:53] (03update) 10raymond-ndibe: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (owner: 10buxtang) [08:38:24] (03open) 10filippo: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) [08:42:37] 10Toolforge, 06tools-platform-team: [jobs-api] did not update the runtime, [components-api] did not actually get the image deployed - https://phabricator.wikimedia.org/T432097#12122745 (10dcaro) >>! In T432097#12122623, @DamianZaremba wrote: >>>! In T432097#12121462, @DamianZaremba wrote: >> Thanks @dcaro,... [08:43:59] (03update) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [08:44:41] (03update) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [08:47:48] (03merge) 10dcaro: core.restart_job: don't fail if the job only exists in storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/353 [08:50:15] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12122777 (10dcaro) >> I have a question though, what is it that you want to measure? > Please see my response to Bryan's speci... [08:50:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.543-20260715084800-17aed7da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1337 (https://phabricator.wikimedia.org/T432097) [08:51:04] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.543-20260715084800-17aed7da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1337 (https://phabricator.wikimedia.org/T432097) [08:51:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:54:53] (03open) 10filippo: playbooks: add /mnt/nfs/dumps [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/329 (https://phabricator.wikimedia.org/T411248) [08:58:24] (03update) 10filippo: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) [08:58:33] (03open) 10filippo: Update tests with /mnt/nfs/dumps directory [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/98 (https://phabricator.wikimedia.org/T411248) [09:02:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:05:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.962% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:08:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:13:47] (03update) 10dcaro: playbooks: add /mnt/nfs/dumps [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/329 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:15:08] (03approved) 10dcaro: playbooks: add /mnt/nfs/dumps [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/329 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:15:12] (03update) 10dcaro: playbooks: add /mnt/nfs/dumps [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/329 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:17:11] (03update) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 (owner: 10renovatebot) [09:18:47] (03approved) 10dcaro: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:20:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:25:27] (03approved) 10dcaro: jobs-api: bump to 0.0.543-20260715084800-17aed7da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1337 (https://phabricator.wikimedia.org/T432097) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:25:32] (03merge) 10dcaro: jobs-api: bump to 0.0.543-20260715084800-17aed7da [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1337 (https://phabricator.wikimedia.org/T432097) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:29:39] (03unapproved) 10dcaro: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:38:22] (03approved) 10dcaro: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:41:54] 06tools-infrastructure-team, 06Data-Platform-SRE: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212 (10fgiunchedi) 03NEW [09:42:50] (03update) 10fnegri: Update tests with /mnt/nfs/dumps directory [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/98 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:42:57] (03approved) 10fnegri: Update tests with /mnt/nfs/dumps directory [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/98 (https://phabricator.wikimedia.org/T411248) (owner: 10filippo) [09:46:05] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12123086 (10Odinaldo) >>! In T416590#12120473, @BLiviero-WMF wrote: > Can you say more about database quotas, what do you anticipate the growth to be over time, to what maximum limit... [09:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:07:57] FIRING: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:16:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [10:16:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [10:47:44] (03open) 10countcount: Bump spin to 0.9.9 in the lockfile to clear the yanked-crate audit warning [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/47 [10:51:38] (03merge) 10countcount: Bump spin to 0.9.9 in the lockfile to clear the yanked-crate audit warning [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/47 [10:58:18] (03update) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 (owner: 10renovatebot) [11:05:29] 10Tool-wiki-gender-stats: Typo in about page in Esperanto - https://phabricator.wikimedia.org/T432223 (10Danya) 03NEW [11:05:31] 10Tool-wiki-gender-stats: Typo in about page in Esperanto - https://phabricator.wikimedia.org/T432223#12123446 (10Danya) p:05Triage→03High [11:05:39] 10Tool-wiki-gender-stats: Typo in about page in Esperanto - https://phabricator.wikimedia.org/T432223#12123448 (10Danya) a:03Danya [11:06:16] 10Tool-wiki-gender-stats: Typo in about page in Esperanto - https://phabricator.wikimedia.org/T432223#12123454 (10Danya) [11:06:17] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12123455 (10Danya) [11:08:54] 10Tool-wiki-gender-stats: Integrate June 2026 data from dumps - https://phabricator.wikimedia.org/T432224 (10Danya) 03NEW [11:09:06] 10Tool-wiki-gender-stats: Integrate June 2026 data from dumps - https://phabricator.wikimedia.org/T432224#12123471 (10Danya) [11:09:08] 10Tool-wiki-gender-stats: Integrate June 2026 data from dumps - https://phabricator.wikimedia.org/T432224#12123473 (10Danya) p:05Triage→03High [11:11:44] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12123484 (10Danya) [11:11:45] 10Tool-wiki-gender-stats: Integrate June 2026 data from dumps - https://phabricator.wikimedia.org/T432224#12123483 (10Danya) [11:11:59] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12123486 (10Danya) [11:12:00] 10Tool-wiki-gender-stats: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#12123485 (10Danya) [11:15:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [11:24:16] 10Toolforge, 06tools-platform-team, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#12123551 (10aputhin) Hey @Anushka10patel thank you for your contribution! @Pankhudi_Singh the task as a whole has been worked on to som... [11:28:02] 10Tool-wiki-gender-stats: Typos in Esperanto translations - https://phabricator.wikimedia.org/T432223#12123568 (10Danya) [11:38:29] (03update) 10countcount: Mirror per-wiki block logs into ToolsDB and serve them as detail=blockhistory [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [11:39:11] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [11:40:25] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228 (10KBach) 03NEW [11:40:42] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12123645 (10KBach) p:05Triage→03Low [11:48:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [11:52:05] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12123705 (10BTullis) [12:26:10] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [12:32:35] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [12:33:19] 10Tool-wiki-gender-stats: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#12123910 (10Danya) a:05Danya→03None [12:33:26] 10Tool-wiki-gender-stats: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337#12123912 (10Danya) a:05Danya→03None [12:33:56] 10Tool-wiki-gender-stats: Add button to save page state as GET params - https://phabricator.wikimedia.org/T422330#12123913 (10Danya) a:05Danya→03None [12:33:58] 10Tool-wiki-gender-stats: Localize table texts - https://phabricator.wikimedia.org/T423787#12123914 (10Danya) a:05Danya→03None [12:34:03] 10Tool-wiki-gender-stats: Translate language names - https://phabricator.wikimedia.org/T426057#12123915 (10Danya) a:05Danya→03None [12:34:10] 10Tool-wiki-gender-stats: Vertically align view links on home page - https://phabricator.wikimedia.org/T427213#12123916 (10Danya) a:05Danya→03None [12:34:57] 10Tool-wiki-gender-stats: Notify maintainers when snapshot capture fails - https://phabricator.wikimedia.org/T424279#12123919 (10Danya) [12:35:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.924% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:37:39] 10Tool-wiki-gender-stats: [To be refined] Create an FAQ page - https://phabricator.wikimedia.org/T425482#12123937 (10Danya) [12:39:32] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12123943 (10AGhirelli-WMF) a:03AGhirelli-WMF [12:39:35] 10Tool-wiki-gender-stats: Save page state as GET params - https://phabricator.wikimedia.org/T422330#12123944 (10Danya) [12:39:44] 10Tool-wiki-gender-stats: Save page state as GET params - https://phabricator.wikimedia.org/T422330#12123963 (10Danya) [12:40:08] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12123968 (10AGhirelli-WMF) [12:40:21] 10Tool-wiki-gender-stats: Add a Privacy Statement - https://phabricator.wikimedia.org/T426262#12123970 (10Danya) [12:40:59] 10Tool-wiki-gender-stats: Add a Privacy Statement - https://phabricator.wikimedia.org/T426262#12123971 (10Danya) [12:44:55] 10Tool-wiki-gender-stats: Vertically align view links on home page - https://phabricator.wikimedia.org/T427213#12123999 (10Danya) [12:49:45] 10Tool-wiki-gender-stats: Translate language names in genders-by-language page - https://phabricator.wikimedia.org/T426057#12124023 (10Danya) a:03Danya [12:52:42] RESOLVED: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:55:56] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12124052 (10Danya) [12:58:15] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12124060 (10Danya) [13:02:08] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12124086 (10cmooney) 05Open→03Resolved [13:02:56] 10Tool-wiki-gender-stats: Chart filter states reset when dataset changes - https://phabricator.wikimedia.org/T432240 (10Danya) 03NEW [13:03:02] 10Tool-wiki-gender-stats: Chart filter states reset when dataset changes - https://phabricator.wikimedia.org/T432240#12124102 (10Danya) p:05Triage→03Medium [13:03:19] 10Tool-wiki-gender-stats: Chart filter states reset when dataset changes - https://phabricator.wikimedia.org/T432240#12124106 (10Danya) [13:04:30] 10Tool-wiki-gender-stats: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#12124110 (10Danya) 05Open→03Invalid Ticket replaced by a proper bug ticket: T432240. Closing this one. [13:07:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431429) [13:07:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T431429) [13:07:42] T431429: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429 [13:08:15] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12124136 (10Andrew) Thanks! I have put this host back in service and we'll keep an eye on it. [13:08:33] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12124137 (10Andrew) a:05VRiley-WMF→03Andrew [13:09:05] 10Tool-wikinewsie: README for Wikinewsie - https://phabricator.wikimedia.org/T432241 (10ZhaoFJx) 03NEW [13:09:07] 10Tool-wikinewsie: README for Wikinewsie - https://phabricator.wikimedia.org/T432241#12124157 (10ZhaoFJx) 05Open→03In progress [13:09:09] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12124154 (10Andrew) Thank you @jsn.sherman [13:11:59] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [13:13:00] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [13:15:18] 10Tool-wiki-gender-stats: Link to Wikidata elements in country and profession stats tables - https://phabricator.wikimedia.org/T425089#12124183 (10Danya) [13:16:12] 10Tool-wiki-gender-stats: Link to Wikidata elements in country and profession stats tables - https://phabricator.wikimedia.org/T425089#12124187 (10Danya) a:05Danya→03None [13:18:28] 10Tool-wiki-gender-stats: Link to wikis and Wikidata elements in languages, countries, and professions tables - https://phabricator.wikimedia.org/T425089#12124190 (10Danya) [13:19:15] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12124195 (10Danya) [13:20:36] 10Tool-wiki-gender-stats: Integrate June 2026 data from dumps - https://phabricator.wikimedia.org/T432224#12124198 (10Danya) [13:20:39] 10Tool-wiki-gender-stats: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#12124199 (10Danya) [13:20:45] 10Tool-wiki-gender-stats: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#12124200 (10Danya) [13:20:55] 10Tool-wiki-gender-stats: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12124201 (10Danya) [13:22:55] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12124203 (10fnegri) 05Resolved→03Open Reopening as the transaction history length is again over 50mil, and it's been high in the past 2 weeks: {F93... [13:23:01] 10Tool-wiki-gender-stats: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#12124208 (10Danya) [13:24:00] 10Tool-wiki-gender-stats: Try to split the compiled JS bundle - https://phabricator.wikimedia.org/T428248#12124210 (10Danya) [13:24:20] 10Tool-wiki-gender-stats: Show a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12124218 (10Danya) [13:27:45] 10Tool-wiki-gender-stats: Localize table texts - https://phabricator.wikimedia.org/T423787#12124226 (10Danya) [13:28:23] 10Tool-wiki-gender-stats: Localize table texts - https://phabricator.wikimedia.org/T423787#12124228 (10Danya) [13:28:56] 10Tool-wiki-gender-stats: Localize table labels - https://phabricator.wikimedia.org/T423787#12124233 (10Danya) [13:33:20] 10Tool-wiki-gender-stats: Integrate May 2026 data from Danÿa’s local copy - https://phabricator.wikimedia.org/T431933#12124259 (10Danya) [13:33:51] (03open) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [13:34:08] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12124263 (10Danya) [13:34:09] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12124262 (10Danya) [13:34:51] 10Tool-wiki-gender-stats: Website unable to access database while dumps ingester is running - https://phabricator.wikimedia.org/T432052#12124267 (10Danya) [13:38:20] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), and 2 others: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST... - https://phabricator.wikimedia.org/T428153#12124284 [13:39:45] 10Tool-wiki-gender-stats: Remove contribution restrictions notice in CONTRIBUTING.md - https://phabricator.wikimedia.org/T432243 (10Danya) 03NEW [13:39:53] 10Tool-wiki-gender-stats: Remove contribution restrictions notice in CONTRIBUTING.md - https://phabricator.wikimedia.org/T432243#12124314 (10Danya) p:05Triage→03Medium [13:40:14] 10Tool-wiki-gender-stats: Remove contribution restrictions notice in CONTRIBUTING.md - https://phabricator.wikimedia.org/T432243#12124324 (10Danya) [13:41:03] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12124329 (10Danya) [13:41:04] 10Tool-wiki-gender-stats: Remove contribution restrictions notice in CONTRIBUTING.md - https://phabricator.wikimedia.org/T432243#12124328 (10Danya) [14:07:07] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12124437 (10fnegri) A graph of disk space usage over the past 4 weeks: {F93798619} [14:20:51] 10Tool-leximap: Creating the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246 (10Gnoeee) 03NEW [14:21:30] 10Tool-leximap: Creating the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246#12124515 (10Bodhisattwa) [14:21:35] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12124516 (10Bodhisattwa) [14:24:28] 10Tool-leximap: Creating the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246#12124533 (10Bodhisattwa) [14:24:52] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: Creating the backend for leximap - https://phabricator.wikimedia.org/T432248 (10Gnoeee) 03NEW [14:26:33] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: Create the frontend for leximap - https://phabricator.wikimedia.org/T432250 (10Gnoeee) 03NEW [14:26:41] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124569 (10brouberol) [14:27:17] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12124571 (10hnowlan) [14:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:28:38] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12124577 (10hnowlan) alert2002 *shouldn't* cause many problems as all clients are aware of both alert hosts. However, th... [14:31:08] 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, 06Infrastructure-Foundations, and 6 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119#12124591 (10hnowlan) [14:32:08] 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119#12124605 (10hnowlan) Is there anything observability can help with on testing this? [14:33:07] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: Created the backend for leximap - https://phabricator.wikimedia.org/T432248#12124616 (10Gnoeee) [14:33:11] 10Tool-leximap: Created the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246#12124621 (10Gnoeee) [14:34:58] (03PS4) 10Krinkle: frontend: Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [14:35:35] (03PS5) 10Krinkle: frontend: Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [14:38:24] (03CR) 10Krinkle: [C:03+2] frontend: Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [14:40:20] (03Merged) 10jenkins-bot: frontend: Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [14:46:41] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12124681 (10hnowlan) [14:49:47] 10PAWS, 06tools-platform-team: Limit NFS file / folder size and automatically clean up large files - https://phabricator.wikimedia.org/T432251 (10aputhin) 03NEW [14:49:51] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124698 (10bking) [14:50:27] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124703 (10brouberol) [14:54:02] 10PAWS, 06tools-platform-team: Create guardrails against PAWS misuse - https://phabricator.wikimedia.org/T432255 (10aputhin) 03NEW [14:57:14] 10VPS-project-Codesearch, 07Developer Productivity: Codesearch should have a non-test filter - https://phabricator.wikimedia.org/T197802#12124782 (10Krinkle) 05Open→03Resolved a:03simon04 Deployed. [14:57:31] 10VPS-project-Codesearch, 07Developer Productivity: Codesearch should have a non-test filter - https://phabricator.wikimedia.org/T197802#12124785 (10Krinkle) [15:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:08:28] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124837 (10klausman) [15:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:20:24] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29427447334 (https://github.com/cluebotng/component-configs/commits/0d2bc5e6c57ba81a40584a85fe4ea3868875999f) [15:20:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:26:35] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12124881 (10Arnoldokoth) There is one issue though. Avatars are not loading... [15:27:36] 06cloud-services-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12124893 (10Andrew) Typically I would restart nfs clients after an nfs migration to make sure they're mounting the new server properly. So the most straightfo... [15:28:03] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [15:34:49] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29428509974 (https://github.com/cluebotng/component-configs/commits/842854f79b07dc4ec739505a99c8cb11009f8f4f) [15:34:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:35:53] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [15:35:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:38:45] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [15:43:16] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12124913 (10Gnoeee) [15:43:47] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12124915 (10fgiunchedi) Thank you @Odinaldo for reaching out and the useful information. I don't think 60GB is problematic, I'm looking forward to the size estimation for the full da... [15:50:38] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12124951 (10bd808) >>! In T423549#12122394, @komla wrote: > It's intentionally a proxy for activity, not a direct measure of s... [15:50:43] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [15:52:11] (03update) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [15:57:34] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12124983 (10Arnoldokoth) Actually, the dump might have caused file integri... [15:59:14] 10Toolforge, 06tools-platform-team: [docs] How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12124987 (10komla) [16:01:22] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [16:01:24] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29430449675 (https://github.com/cluebotng/component-configs/commits/ecf4f749293c44e72e74858bef8193eb29695c9a) [16:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:03:06] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12125027 (10aputhin) 05In progress→03Resolved [16:03:25] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12125030 (10aputhin) 05Open→03In progress [16:10:59] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [16:14:19] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [16:14:27] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [16:27:28] (03open) 10don-vip: Remove dead uploadedMedia/uploadedMetadata tracking from updateMedia [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/47 [16:33:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12125208 (10komla) > It does not, the question I ask is about the aggregation, not about what's considered as active user (se... [16:33:31] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12125211 (10LSobanski) I confirmed with RelEng that gerrit-bullseye can be safely deleted. [16:43:01] (03merge) 10don-vip: Remove dead uploadedMedia/uploadedMetadata tracking from updateMedia [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/47 [16:47:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12125257 (10komla) >>! In T423549#12124951, @bd808 wrote: >>>! In T423549#12122394, @komla wrote: >> It's intentionally a prox... [16:51:22] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [16:56:53] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29434331433 (https://github.com/cluebotng/component-configs/commits/37dd33ec5b56f991e912ec44ce96a536735cb1a0) [16:56:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:05:41] 06tools-infrastructure-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12125362 (10BLiviero-WMF) [17:07:44] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29435050584 (https://github.com/cluebotng/component-configs/commits/912a8ea89e62cddf63a20263572b5c94900ef06a) [17:07:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:14:50] 10Cloud-VPS (Quota-requests), 10WMIT-Infrastructure: Quota increase request for project osmit - https://phabricator.wikimedia.org/T432153#12125412 (10LorenzoStucchi) +1 it is needed to update a deprecated version of a machine [17:15:34] 10Toolforge, 06tools-platform-team: Request for Toolforge account access for puspaaamm-bot - https://phabricator.wikimedia.org/T432268#12125417 (10Reedy) 05Open→03Stalled Have you followed https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart ? This shouldn't need a task... [17:22:34] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12125446 (10Dzahn) [17:22:58] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12125450 (10Dzahn) Thanks! Done. The gerrit instance has been deleted. [17:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:31:11] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29436632270 (https://github.com/cluebotng/component-configs/commits/ff37e7e6d3e126ba24d186be144aeae3d2048954) [17:31:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:54:31] (03update) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [18:11:12] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276 (10dcaro) 03NEW [18:21:03] 10Tool-wikinewsie: "About" link for Wikinewsie - https://phabricator.wikimedia.org/T431955#12125722 (10ZhaoFJx) a:03ZhaoFJx [18:21:10] 10Tool-wikinewsie: "About" link for Wikinewsie - https://phabricator.wikimedia.org/T431955#12125724 (10ZhaoFJx) 05Open→03In progress [18:22:20] (03update) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [18:55:06] 10Striker, 06tools-platform-team, 10Toolhub: "GO TO TOOL" targeting toolsadmin.wikimedia.org pages fails because of remote cross-origin-opener-policy header value - https://phabricator.wikimedia.org/T432262#12125844 (10bd808) The issue that is happening here will not be confined only to `target="_blank"` lin... [19:14:13] 10Striker, 06tools-platform-team, 10Toolhub: "GO TO TOOL" targeting toolsadmin.wikimedia.org pages fails because of remote cross-origin-opener-policy header value - https://phabricator.wikimedia.org/T432262#12125900 (10bd808) >>! In T432262#12125843, @DamianZaremba wrote: > I think this could be fixed with r... [19:38:16] (03PS1) 10MusikAnimal: 3.25.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1311108 [19:39:05] (03CR) 10MusikAnimal: [C:03+2] 3.25.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1311108 (owner: 10MusikAnimal) [19:39:42] (03Merged) 10jenkins-bot: 3.25.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1311108 (owner: 10MusikAnimal) [19:58:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:58:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:03:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:03:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:13:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:18:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:23:37] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:24:17] 10VPS-project-Wikistats: Rank page inaccessible (500 Error) - https://phabricator.wikimedia.org/T431462#12126037 (10Dzahn) ` PHP Fatal error: Uncaught Error: Undefined constant "prefix".. ` ` script '/var/www/wikistats/detailw.php' not found ` [20:27:46] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:29:35] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:43:39] 10Tool-wikinewsie: README for Wikinewsie - https://phabricator.wikimedia.org/T432241#12126073 (10ZhaoFJx) 05In progress→03Resolved [20:44:51] 10VPS-project-Wikistats: Rank page inaccessible (500 Error) - https://phabricator.wikimedia.org/T431462#12126074 (10Dzahn) 05Open→03Resolved Thanks for reporting! The issue has been fixed by adding single quotes around unquoted key names in $row arrays. PHP became stricter about this and moved from warn... [20:53:03] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:53:23] 10Toolforge, 06tools-platform-team: Request for Toolforge account access for puspaaamm-bot - https://phabricator.wikimedia.org/T432268#12126100 (10bd808) 05Stalled→03Invalid >>! In T432268#12125417, @Reedy wrote: > Have you followed https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart ? This s... [20:56:42] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:59:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:00:58] (03update) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [21:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:25:36] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297 (10Danya) 03NEW [21:25:49] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297#12126195 (10Danya) [21:25:54] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297#12126197 (10Danya) p:05Triage→03High [21:27:58] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12126210 (10Danya) [21:27:59] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297#12126209 (10Danya) [21:28:38] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297#12126212 (10Danya) [21:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:19:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:12:39] 06tools-infrastructure-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12126420 (10Krinkle) @andrew It runs on the Apache host as well. The backup and push tasks are for the app hosts. The pull task is for the apache host.