[00:15:40] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [00:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:18:42] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:10:19] (03open) 10samwilson: Add message for WsExportException [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/11 (https://phabricator.wikimedia.org/T430381) [02:12:02] (03update) 10samwilson: Add message for WsExportException [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/11 (https://phabricator.wikimedia.org/T430381) [02:36:08] (03merge) 10samwilson: Add message for WsExportException [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/11 (https://phabricator.wikimedia.org/T430381) [02:38:20] (03merge) 10samwilson: Update GitHub references and remove version number from footer [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/10 (https://phabricator.wikimedia.org/T429475) [04:16:05] 06tools-infrastructure-team: Network unavailable on a few VMs. - https://phabricator.wikimedia.org/T432426 (10Andrew) 03NEW [04:23:50] 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12131233 (10Bugreporter2) [07:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:02:33] (03merge) 10countcount: Drop range_end from the IP range dimensions, key on (range_start, prefix_len) [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/50 [08:15:16] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [08:15:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [08:26:51] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12131542 (10elukey) Hey folks! IIUC this task is to track both Bookworm's and Trixie's updates, but I don't s... [09:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:13:01] (03open) 10countcount: Skew-aware two-branch IP range probe: fat/exact-length stabbing plus a sparse-length fetch branch [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/51 [09:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:28:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:46:13] (03update) 10countcount: Skew-aware two-branch IP range probe: fat/exact-length stabbing plus a sparse-length fetch branch [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/51 [09:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:54:32] (03merge) 10countcount: Skew-aware two-branch IP range probe: fat/exact-length stabbing plus a sparse-length fetch branch [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/51 [09:54:48] (03open) 10countcount: Retry failed block-history backfill attempts from the last committed chunk [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/52 [09:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:01:53] (03update) 10countcount: Retry failed block-history backfill attempts from the last committed chunk [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/52 [10:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:05:35] (03merge) 10countcount: Retry failed block-history backfill attempts from the last committed chunk [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/52 [10:10:11] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131781 (10ops-monitoring-bot) Rebooting db1176.eqiad.wmnet [10:11:50] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131795 (10ops-monitoring-bot) Rebooting db1176.eqiad.wmnet [10:15:58] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [10:16:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [10:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:28:42] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131831 (10ops-monitoring-bot) Rebooting db-test1001.eqiad.wmnet [10:33:36] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131850 (10ops-monitoring-bot) Reboot of db-test1001.eqiad.wmnet completed [10:33:53] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131852 (10ops-monitoring-bot) Rebooting db-test1002.eqiad.wmnet [10:35:44] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131866 (10ops-monitoring-bot) Reboot of db-test1002.eqiad.wmnet completed [10:36:49] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131867 (10ops-monitoring-bot) Rebooting db-test1003.eqiad.wmnet [10:39:15] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131868 (10ops-monitoring-bot) Reboot of db-test1003.eqiad.wmnet completed [10:41:12] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131882 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [10:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:48:29] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131912 (10ops-monitoring-bot) Reboot of db-test2001.codfw.wmnet completed [10:48:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131913 (10ops-monitoring-bot) Rebooting db-test2002.codfw.wmnet [10:51:24] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12131919 (10ops-monitoring-bot) Reboot of db-test2002.codfw.wmnet completed [11:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:18:42] 10Toolforge, 06tools-platform-team: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450 (10AryasarhanBot) 03NEW [11:23:16] (03open) 10countcount: Return the block-history existence flags by default [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/53 [11:24:06] 10Toolforge, 06tools-platform-team: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450#12132013 (10AryasarhanBot) Request tool account: football-kit-sync [11:24:33] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12132016 (10ops-monitoring-bot) Rebooting db2230.codfw.wmnet [11:26:34] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12132017 (10jsn.sherman) [11:29:22] (03merge) 10countcount: Return the block-history existence flags by default [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/53 [11:32:05] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12132032 (10ops-monitoring-bot) Reboot of db2230.codfw.wmnet completed [11:34:06] (03open) 10countcount: Include the actual blocked range as target in /ipinfo block-history events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/54 [11:37:20] (03PS1) 10Klausman: hiera/deployment_server: Add pseudo-secrtes for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) [11:37:54] (03PS2) 10Klausman: hiera/deployment_server: Add pseudo-secrets for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) [11:38:11] (03merge) 10countcount: Include the actual blocked range as target in /ipinfo block-history events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/54 [11:38:15] (03CR) 10Klausman: [V:03+2 C:03+2] hiera/deployment_server: Add pseudo-secrets for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) (owner: 10Klausman) [11:50:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12132094 (10CWilliams-WMF) Testing passed: - aborted when pointed at the primary - asked when all-bu... [11:55:50] (03open) 10countcount: Design revisions: date-only timestamps, exact durations, section anchors, action filter [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/1 [11:57:21] (03merge) 10countcount: Design revisions: date-only timestamps, exact durations, section anchors, action filter [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/1 [11:59:12] (03open) 10countcount: Add Claude Code dev-server launch config [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/2 [12:15:33] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [12:15:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [12:17:16] (03open) 10countcount: Bound /ipinfo probe statements with a byte-budgeted wiki chunker [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/55 [12:32:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:34:29] (03open) 10countcount: Close replica connections abandoned mid-command instead of returning them to the pool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/56 [12:37:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:51:38] (03update) 10countcount: Close replica connections abandoned mid-command instead of returning them to the pool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/56 [12:51:39] (03update) 10countcount: Bound /ipinfo probe statements with a byte-budgeted wiki chunker [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/55 [12:51:54] (03update) 10countcount: Bound /ipinfo probe statements with a byte-budgeted wiki chunker [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/55 [12:55:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12132239 (10fnegri) 05In progress→03Resolved @CWilliams-WMF thank you very much for testing... [13:03:56] (03update) 10andrew: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [13:13:00] (03merge) 10countcount: Bound /ipinfo probe statements with a byte-budgeted wiki chunker [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/55 [13:13:19] (03update) 10countcount: Close replica connections abandoned mid-command instead of returning them to the pool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/56 [13:15:38] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [13:15:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [13:16:30] (03update) 10andrew: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [13:17:09] (03merge) 10countcount: Close replica connections abandoned mid-command instead of returning them to the pool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/56 [13:17:51] (03close) 10andrew: explicitly export PYTHONPATH in install script [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/10 (owner: 10jelto) [13:19:04] (03close) 10andrew: secgroups: open ipv6 access to codfw1dev ntp server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/261 (https://phabricator.wikimedia.org/T408387) [13:21:39] (03open) 10countcount: Per-event blocked targets for IP block history plus block-table design revisions [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/3 [13:23:08] (03merge) 10countcount: Per-event blocked targets for IP block history plus block-table design revisions [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/3 [13:23:30] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Rip out barbican puppet code - https://phabricator.wikimedia.org/T432349#12132316 (10Andrew) [13:23:32] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Learn about Barbican - https://phabricator.wikimedia.org/T263680#12132315 (10Andrew) [13:26:54] (03update) 10countcount: Add Claude Code dev-server launch config [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/2 [13:27:00] (03update) 10countcount: Add Claude Code dev-server launch config [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/2 [13:28:13] (03merge) 10countcount: Add Claude Code dev-server launch config [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/2 [13:33:50] (03open) 10countcount: Tidy flags and duration formatting in the block tables [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/4 [13:35:16] (03merge) 10countcount: Tidy flags and duration formatting in the block tables [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/4 [13:41:48] 10Toolforge, 06tools-platform-team: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450#12132401 (10taavi) 05Open→03Invalid See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart for instructions on getting access [13:48:42] 10Cloud-VPS (Project-requests): Request creation of vps - https://phabricator.wikimedia.org/T431996#12132425 (10taavi) 05Open→03Declined Closing due to no response to Andre's comment. [13:51:28] (03open) 10countcount: Hyperlink wikilinks and bare URLs in block reasons [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/5 [13:53:04] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12132449 (10taavi) What is the full double disk space needed for? Cinder volumes can be moved between VMs, one at a time, which should save the need for full... [13:54:54] 10Cloud-VPS (Quota-requests), 10WMIT-Infrastructure: Quota increase request for project osmit - https://phabricator.wikimedia.org/T432153#12132452 (10fnegri) +1 [13:55:16] 10Cloud-VPS (Quota-requests): Quota increase request for project kafka-infrastructure - https://phabricator.wikimedia.org/T431590#12132454 (10fnegri) +1 [13:56:39] (03merge) 10countcount: Hyperlink wikilinks and bare URLs in block reasons [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/5 [13:58:38] !log taavi@cloudcumin1001 osmit START - Cookbook wmcs.openstack.quota_increase by 2 cores, 4096 ram [13:58:45] !log taavi@cloudcumin1001 osmit END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 2 cores, 4096 ram [13:58:57] !log taavi@cloudcumin1001 kafka-infrastructure START - Cookbook wmcs.openstack.quota_increase by 10 cores, 8 instances, 10240 ram [13:59:04] !log taavi@cloudcumin1001 kafka-infrastructure END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 10 cores, 8 instances, 10240 ram [13:59:21] 10Cloud-VPS (Quota-requests), 10WMIT-Infrastructure: Quota increase request for project osmit - https://phabricator.wikimedia.org/T432153#12132460 (10taavi) 05Open→03Resolved a:03taavi This is complete; sorry for the delay. [13:59:24] 10Cloud-VPS (Quota-requests): Quota increase request for project kafka-infrastructure - https://phabricator.wikimedia.org/T431590#12132466 (10taavi) 05Open→03Resolved a:03taavi This is complete; sorry for the delay. [14:00:35] 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933#12132469 (10Andrew) I think I'm behind on something. When you say... > should not be hardcoded in puppet anymore Are you talking about the profile::openstack::base::nova::compute::id v... [14:14:38] (03update) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [14:15:26] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [14:15:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [14:16:59] (03merge) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [14:17:02] (03update) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [14:19:34] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.29-20260717141712-9ea8d3f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1342 [14:19:51] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:19:56] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [14:20:28] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [14:26:18] (03open) 10hnowlan: Add API endpoint [toolforge-repos/template-radar] - 10https://gitlab.wikimedia.org/toolforge-repos/template-radar/-/merge_requests/1 [14:29:15] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [14:33:55] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [14:38:57] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12132584 (10Nihar_Chakravarti) @Aklapper , @Pppery I've tested this personnely on docker . I am wondering why isn't it merged! If there's anything I cou... [14:43:17] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [14:48:39] (03merge) 10tlepage: logs-api: bump to 0.0.29-20260717141712-9ea8d3f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1342 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:13:00] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12132676 (10fnegri) 05Resolved→03In progress Reopening as I found a bug (see patch above). [15:33:23] 10Tool-wiki-gender-stats: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#12132754 (10Danya) 05Open→03In progress [15:34:11] (03open) 10danyya: Draft: Resolve T427219 "Add a filter bar to chart labels toggles" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/85 [15:35:32] (03merge) 10countcount: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 (owner: 10renovatebot) [15:43:01] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12132777 (10Danya) a:03Danya [15:43:15] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12132781 (10Danya) p:05Medium→03High [15:47:05] 10Tool-itwiki, 06Commons, 03Wikimania-Hackathon-2026: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#12132788 (10valerio.bozzolan) [15:47:43] 10Tool-itwiki, 06Commons, 03Wikimania-Hackathon-2026: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#12132789 (10valerio.bozzolan) Again, it sounds like something for an hackathon. [16:07:35] 10Tool-wiki-gender-stats: Commons categorized among Wikipedias - https://phabricator.wikimedia.org/T432474 (10Danya) 03NEW [16:07:36] 10Tool-wiki-gender-stats: Commons categorized among Wikipedias - https://phabricator.wikimedia.org/T432474#12132866 (10Danya) p:05Triage→03Medium [16:08:13] 10Tool-wiki-gender-stats: Commons categorized as a Wikipedia language code - https://phabricator.wikimedia.org/T432474#12132882 (10Danya) a:03Danya [16:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [16:17:35] 10Tool-wiki-gender-stats: Commons and other wikis categorized as a Wikipedia language code - https://phabricator.wikimedia.org/T432474#12132912 (10Danya) [16:18:27] 10Tool-wiki-gender-stats: Commons and other wikis categorized as a Wikipedia language code - https://phabricator.wikimedia.org/T432474#12132916 (10Danya) p:05Medium→03High [16:18:49] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12132917 (10Danya) [16:21:06] (03update) 10danyya: Draft: Resolve T427219 "Add a filter bar to chart labels toggles" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/85 [16:22:03] (03update) 10danyya: Resolve T427219 "Add a filter bar to chart labels toggles" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/85 [16:22:25] (03merge) 10danyya: Resolve T427219 "Add a filter bar to chart labels toggles" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/85 [16:23:02] 10Tool-wiki-gender-stats, 13Patch-For-Review: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#12132921 (10Danya) 05In progress→03Resolved [16:23:10] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12132923 (10Danya) [16:23:11] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12132924 (10Danya) [16:23:20] 10Tool-wiki-gender-stats: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12132926 (10Danya) 05Open→03In progress [16:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:25:15] (03open) 10countcount: Show block reasons in full instead of clamping to three lines [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/6 [16:31:03] (03open) 10danyya: Draft: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [16:35:58] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12132952 (10fnegri) 05In progress→03Resolved This went down on its own, and the history length is back to around 20 mil (which is still high, bu... [16:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:05:47] (03update) 10danyya: Draft: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [17:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:29:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:36:27] (03update) 10danyya: Draft: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [18:47:36] (03update) 10danyya: Draft: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [18:48:11] (03update) 10danyya: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [18:48:52] (03merge) 10danyya: Resolve T422337 "Add a text field to filter gender tables" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/86 [18:49:37] 10Tool-wiki-gender-stats, 13Patch-For-Review: Add a text field to filter gender tables - https://phabricator.wikimedia.org/T422337#12133245 (10Danya) 05In progress→03Resolved [18:50:40] 10Tool-wiki-gender-stats: Show a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12133249 (10Danya) 05Open→03In progress [19:34:32] (03open) 10danyya: Draft: Resolve T427412 "Show a notice banner whenever a website maintenance is planned" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/87 [19:54:42] 10Tool-leximap: Create the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246#12133434 (10Gnoeee) [19:57:07] 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12133443 (10Andrew) There are lots of reasons for a host to become unreachable, but falling off the network is one of them. This dashboard is a good place to start: https://prometheus-alerts.wmcloud.org/?q=... [19:57:21] 10Tool-wiki-gender-stats: Create a script to delete an entire snapshot’s data - https://phabricator.wikimedia.org/T432490 (10Danya) 03NEW [19:57:29] 10Tool-wiki-gender-stats: Create a script to delete an entire snapshot’s data - https://phabricator.wikimedia.org/T432490#12133455 (10Danya) [19:57:34] 10Tool-wiki-gender-stats: Create a script to delete an entire snapshot’s data - https://phabricator.wikimedia.org/T432490#12133457 (10Danya) p:05Triage→03Low [19:57:51] 10Tool-wiki-gender-stats: Create a script to delete an entire snapshot’s data from the db - https://phabricator.wikimedia.org/T432490#12133459 (10Danya) [20:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:17:04] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Rip out barbican puppet code - https://phabricator.wikimedia.org/T432349#12133494 (10Andrew) 05Open→03Resolved [20:17:15] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Learn about Barbican - https://phabricator.wikimedia.org/T263680#12133496 (10Andrew) 05Stalled→03Resolved a:03Andrew [20:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:48:15] (03update) 10danyya: Draft: Resolve T427412 "Show a notice banner whenever a website maintenance is planned" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/87 [21:54:21] (03update) 10danyya: Draft: Resolve T427412 "Show a notice banner whenever a website maintenance is planned" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/87 [22:31:03] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12133698 (10jsn.sherman) [22:31:49] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12133700 (10jsn.sherman) [23:14:26] (03update) 10renovatebot: Update pnpm to v11.14.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [23:14:29] (03open) 10renovatebot: Update pnpm to v11.14.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [23:15:25] (03update) 10renovatebot: Update pnpm to v11.14.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [23:15:28] (03open) 10renovatebot: Update pnpm to v11.14.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [23:16:25] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [23:16:26] (03update) 10renovatebot: Update pnpm to v11.14.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [23:16:42] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [23:17:09] (03CR) 10Jsn.sherman: [C:03+1] tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 (owner: 10MusikAnimal)