[00:14:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [00:14:07] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [00:16:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [00:16:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [00:16:14] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [00:53:23] (03PS3) 10Jsn.sherman: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) [00:53:23] (03PS3) 10Jsn.sherman: Repository: remove dead code, add test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) [00:56:24] 10wikitech.wikimedia.org, 10MediaWiki-User-management, 07Wikimedia-database-issue (Bad data): Some system users have invalid 'actor' database rows - https://phabricator.wikimedia.org/T326431#12113099 (10Krinkle) [00:56:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:57:55] (03CR) 10Jsn.sherman: "> that is inaccurate. The s1, ..., sN do not as of the previous patch have any signification beyond being the identifiers in config/packag" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) (owner: 10Jsn.sherman) [00:58:16] 10Tool-glam-tool-hospital: Other / I am not sure - Bug - VIAF no longer pastes in author's name - https://phabricator.wikimedia.org/T431958 (10GLAM-Tool-Hospital-Bot) 03NEW p:05Triage→03Medium [01:01:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:14:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [01:14:38] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [01:28:52] 10Tool-glam-tool-hospital: VIAF no longer pastes in author's name - https://phabricator.wikimedia.org/T431958#12113147 (10Pppery) [01:29:31] 10Tool-glam-tool-hospital: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12113148 (10Pppery) [01:30:38] 10Tool-glam-tool-hospital: VIAF no longer pastes in author's name - https://phabricator.wikimedia.org/T431958#12113151 (10Aklapper) 05Open→03Invalid Hi, thanks for reporting this. The maintainer of the software uses https://github.com/arthurpsmith/author-disambiguator/issues for bug reports; please repor... [01:32:08] (03CR) 10Jsn.sherman: "I realized that I did work on top of this without actually voting on it; I think 3rd party support makes sense for this in general, and th" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [01:32:21] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [01:36:23] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [02:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:15:37] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [02:15:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [02:16:52] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [02:17:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [02:17:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [02:41:52] (03close) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8 [03:45:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:50:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:56:43] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962 (10Pharos) 03NEW [03:57:38] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962#12113244 (10Pharos) [04:15:00] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [04:15:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [04:15:31] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [04:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [05:02:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:07:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:08:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:13:11] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:43:29] (03CR) 10Thiemo Kreuz (WMDE): Clarify placeholder: regexp not wrapped in `/` (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [06:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:17:18] (03CR) 10Simon04: Clarify placeholder: regexp not wrapped in `/` (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [06:20:43] (03PS2) 10Simon04: Clarify placeholder: regexp not wrapped in /…/ [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) [06:21:06] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Clarify placeholder: regexp not wrapped in /…/ [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [06:34:46] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:39:46] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:46:42] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12113382 (10EbenezerRao) a:05Arian_Ar→03EbenezerRao Hi! I'd like to work on this task, but I couldn't find the corresponding repository. Could you please link the repository (or provide the clone URL) that... [07:00:15] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12113401 (10Jelto) [07:23:03] 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967 (10RhinosF1) 03NEW [07:36:00] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12113462 (10Marostegui) >>! In T420203#12090685, @CWilliams-WMF wrote: > Confirmation was indeed req... [07:43:40] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12113495 (10Arian_Ar) @EbenezerRao The tool is linked in the tool profile page in Phab, but here you go: https://gitlab.wikimedia.org/toolforge-repos/sentinel [07:45:55] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12113499 (10Marostegui) Starting to pool/test clouddb1028 ` [07:45:16] marostegui@cumin1003:~$ sudo confctl select nam... [08:13:21] 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12113592 (10dcaro) The logs of me doing requests that triggered that comment: ` root@tools-k8s-haproxy-8:~# grep k8s-ingress-http/tools-k8s-gateway-2 /var/log/haproxy/haproxy... [08:25:25] (03CR) 10Alien4444: Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [08:26:58] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12113651 (10Marostegui) [08:28:13] 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12113654 (10dcaro) Overall, the rate is a bit higher than the other two gateways: ` root@tools-k8s-haproxy-8:~# for i in 1 2 3; do errors=$(grep k8s-ingress-http/tools-k8s-gat... [08:32:21] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12113669 (10Marostegui) [08:34:09] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12113672 (10Marostegui) clouddb1028 has been fully pooled with normal weight. [08:50:06] 06tools-platform-team: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12113755 (10dcaro) [08:50:10] 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12113754 (10dcaro) [08:52:55] 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12113760 (10dcaro) Note that this might not be related to the incident, but it correlates. [09:19:31] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12113837 (10EbenezerRao) Thanks a lot @Arian_Ar [09:37:20] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [09:44:13] 06tools-platform-team, 13Patch-For-Review: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12113872 (10dcaro) a:05dcaro→03taavi [09:58:48] FIRING: PuppetFailure: Puppet has failed on clouddb1029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:08:22] (03update) 10tlepage: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 (owner: 10dcaro) [10:08:32] (03approved) 10tlepage: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 (owner: 10dcaro) [10:08:34] (03update) 10tlepage: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 (owner: 10dcaro) [10:30:05] (03open) 10ebenezerrao: Set last_login on first OAuth user creation [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/12 [10:34:58] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12113997 (10EbenezerRao) Submitted a fix for review. Merge Request: [[ https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/12 ]] This change sets `last_login` during the initial OAuth user... [10:47:25] (03open) 10damian: functional-tests - builds-api - wrap command [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1327 (https://phabricator.wikimedia.org/T430578) [10:51:11] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 13Patch-For-Review: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12114072 (10taavi) [10:53:13] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Create phab tag/form/workflow/something for volunteer priv requests - https://phabricator.wikimedia.org/T212012#12114085 (10taavi) 05Open→03Resolved https://wikitech.wikimedia.org/wiki/Help:Access_policies#Access_Types do... [10:53:35] (03update) 10damian: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [10:57:21] (03approved) 10filippo: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) (owner: 10taavi) [10:57:49] (03approved) 10filippo: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) (owner: 10taavi) [10:58:03] (03approved) 10filippo: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) (owner: 10taavi) [10:58:19] (03approved) 10filippo: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) (owner: 10taavi) [10:59:52] (03merge) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [10:59:52] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [10:59:54] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:02:14] (03merge) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:02:22] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:08:02] RESOLVED: [2x] PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-haproxy-7 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:08:49] RESOLVED: PuppetFailure: Puppet has failed on clouddb1029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:09:02] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:09:02] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:09:09] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:09:15] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:10:55] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:10:58] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:11:02] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12114144 (10DamianZaremba) >>! In T430578#12076220, @aputhin wrote: > @DamianZaremba thank... [11:12:31] (03merge) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:12:37] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:13:17] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12114149 (10hnowlan) graphite2004 doesn't need depooling. centrallog will have some gaps in logs on-disk but that's unavoidable... [11:31:16] (03update) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [11:31:18] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:31:23] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:31:27] (03open) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [11:32:39] (03update) 10taavi: toolsbeta: Provision a single Redis instance at first [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/112 [11:32:40] (03update) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [11:32:43] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [11:32:46] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:32:55] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:33:04] (03open) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [11:33:05] (03open) 10taavi: toolsbeta: Provision a single Redis instance at first [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/112 [11:33:06] (03update) 10taavi: toolsbeta: Provision a single Redis instance at first [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/112 [11:33:07] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [11:33:08] (03update) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [11:34:36] (03merge) 10taavi: toolsbeta: Provision a single Redis instance at first [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/112 [11:34:44] (03update) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [11:38:30] 10Cloud-VPS, 06tools-infrastructure-team: Delegate lists.wmcloud.org domain to be able to add DNS DKIM records - https://phabricator.wikimedia.org/T278358#12114264 (10taavi) 05Open→03Resolved The domain is delegated, so we don't have much left to do here. It is possible to add a RSA DKIM key via Design... [11:42:17] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud [11:43:49] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud [11:47:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-redis-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:51:27] (03approved) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [11:51:33] (03merge) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [11:51:59] (03update) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [11:52:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-redis-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [12:01:19] (03merge) 10taavi: shared: Ignore changes to port [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/111 [12:01:26] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [12:02:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.219-20260713115149-2ec00f53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1328 (https://phabricator.wikimedia.org/T402568) [12:02:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.219-20260713115149-2ec00f53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1328 (https://phabricator.wikimedia.org/T402568) [12:04:06] (03update) 10taavi: service: Provision security groups at port creation time [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/114 [12:04:08] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [12:04:09] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [12:04:25] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [12:04:26] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [12:04:26] (03open) 10taavi: service: Provision security groups at port creation time [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/114 [12:04:30] (03update) 10taavi: service: Provision security groups at port creation time [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/114 [12:07:17] (03update) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [12:15:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [12:18:51] (03merge) 10taavi: service: Provision security groups at port creation time [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/114 [12:18:54] (03update) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [12:19:23] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12114418 (10jsn.sherman) [12:19:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [12:21:19] 10Cloud-VPS (Project-requests): Request creation of vps - https://phabricator.wikimedia.org/T431996 (10Chaosnb) 03NEW [12:21:20] (03update) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [12:21:36] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [12:26:02] (03merge) 10taavi: Revert "toolsbeta: Provision a single Redis instance at first" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/113 [12:26:08] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [12:27:01] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [12:28:34] (03approved) 10dcaro: builds-api: bump to 0.0.219-20260713115149-2ec00f53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1328 (https://phabricator.wikimedia.org/T402568) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:40] (03merge) 10dcaro: builds-api: bump to 0.0.219-20260713115149-2ec00f53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1328 (https://phabricator.wikimedia.org/T402568) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:52] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud [12:33:28] (03approved) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [12:33:41] (03merge) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [12:34:05] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud [12:34:26] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud [12:35:19] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud [12:35:21] !log taavi@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud [12:35:40] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud [12:37:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.203-20260713123357-f4cbf418 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1329 (https://phabricator.wikimedia.org/T402568) [12:37:07] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.203-20260713123357-f4cbf418 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1329 (https://phabricator.wikimedia.org/T402568) [12:37:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [12:37:58] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud [12:38:15] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:38:16] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud [12:38:53] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:38:56] (03update) 10tlepage: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:41:09] (03merge) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [12:41:45] 10PAWS, 06tools-platform-team: [o11y] prometheus is not fetching jupyterhub metrics anymore - https://phabricator.wikimedia.org/T431999 (10dcaro) 03NEW [12:42:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [12:42:46] (03update) 10tlepage: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [12:53:57] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [12:59:21] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:02:01] (03approved) 10dcaro: components-api: bump to 0.0.203-20260713123357-f4cbf418 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1329 (https://phabricator.wikimedia.org/T402568) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:02:07] (03merge) 10dcaro: components-api: bump to 0.0.203-20260713123357-f4cbf418 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1329 (https://phabricator.wikimedia.org/T402568) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:07:56] (03approved) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [13:08:03] (03merge) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [13:08:08] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [13:08:11] (03update) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [13:11:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.535-20260713130816-9ae8e516 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1330 (https://phabricator.wikimedia.org/T430071) [13:11:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.535-20260713130816-9ae8e516 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1330 (https://phabricator.wikimedia.org/T430071) [13:13:25] (03open) 10raymond-ndibe: draft: container exec experiment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/349 [13:18:12] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api,beta] Image should only be build once when re-used in components - https://phabricator.wikimedia.org/T401851#12114766 (10dcaro) [13:18:20] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] Queue builds when the build queue is full - https://phabricator.wikimedia.org/T402568#12114762 (10dcaro) 05In progress→03Resolved A minimal version of this has been implemented, I'll close this task and we can d... [13:28:22] (03approved) 10dcaro: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 (owner: 10tlepage) [13:29:41] (03update) 10taavi: tools: Add tools-redis-9 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/116 (https://phabricator.wikimedia.org/T360704) [13:29:41] (03update) 10taavi: tools: Add tools-redis-10 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/117 (https://phabricator.wikimedia.org/T360704) [13:29:41] (03update) 10taavi: tools: Add tools-redis-8 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/115 (https://phabricator.wikimedia.org/T360704) [13:29:43] (03update) 10taavi: tools: Move Redis DNS record to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/118 (https://phabricator.wikimedia.org/T360704) [13:30:09] (03open) 10taavi: tools: Add tools-redis-8 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/115 (https://phabricator.wikimedia.org/T360704) [13:30:11] (03open) 10taavi: tools: Add tools-redis-10 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/117 (https://phabricator.wikimedia.org/T360704) [13:30:12] (03open) 10taavi: tools: Move Redis DNS record to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/118 (https://phabricator.wikimedia.org/T360704) [13:30:12] (03open) 10taavi: tools: Add tools-redis-9 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/116 (https://phabricator.wikimedia.org/T360704) [13:30:45] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [13:31:46] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:32:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12114848 (10Marostegui) clouddb1029 is now replicating s5 s8 and x3 [13:36:05] 10Cloud-VPS (Project-requests): Request creation of vps - https://phabricator.wikimedia.org/T431996#12114863 (10Aklapper) a:05C-Brennan-Poole→03None Hi, > Project Name: wikidata That name is much too vague. > - ily No such developer account seems to exist. > The service processes Wikidata dumps and prov... [13:42:14] (03merge) 10taavi: tools: Add tools-redis-8 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/115 (https://phabricator.wikimedia.org/T360704) [13:42:17] (03update) 10taavi: tools: Add tools-redis-9 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/116 (https://phabricator.wikimedia.org/T360704) [13:44:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:49:17] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud [13:50:40] 10Cloud-VPS, 06tools-infrastructure-team: Get rid of neutronclient deprecation warnings - https://phabricator.wikimedia.org/T432012 (10fgiunchedi) 03NEW [13:50:47] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud [13:51:59] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:53:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12114971 (10ssingh) >>! In T430651#12109242, @cmooney wrote: > @fabfur have you any insight into... [13:56:20] 10PAWS, 06tools-platform-team: Limit paws storage - https://phabricator.wikimedia.org/T327936#12114981 (10Alien333) [13:56:51] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db03 [13:57:37] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db03 [13:57:58] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db04 [13:58:44] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db04 [14:02:47] (03update) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [14:05:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:06:57] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.535-20260713130816-9ae8e516 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1330 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:06:59] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.535-20260713130816-9ae8e516 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1330 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:07:42] (03merge) 10dcaro: jobs-api: bump to 0.0.535-20260713130816-9ae8e516 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1330 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:08:20] (03update) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [14:08:21] (03approved) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [14:08:29] (03update) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [14:09:09] (03merge) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [14:09:15] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [14:11:37] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 98.6%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:12:49] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.537-20260713140924-06e5d09a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1331 (https://phabricator.wikimedia.org/T430550) [14:12:53] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.537-20260713140924-06e5d09a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1331 (https://phabricator.wikimedia.org/T430550) [14:13:23] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:17:02] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12115033 (10aputhin) a:03DamianZaremba [14:25:15] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:26:41] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:32:03] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29258212925 (https://github.com/cluebotng/component-configs/commits/2d491fa13ec950460297fa6cfae6801b75d17cf4) [14:32:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:34:26] (03merge) 10taavi: tools: Add tools-redis-9 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/116 (https://phabricator.wikimedia.org/T360704) [14:34:30] (03update) 10taavi: tools: Add tools-redis-10 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/117 (https://phabricator.wikimedia.org/T360704) [14:38:33] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:40:24] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115161 (10cmooney) [14:40:32] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:42:10] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud [14:42:45] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12115164 (10Marostegui) [14:43:50] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud [14:46:49] (03merge) 10taavi: tools: Add tools-redis-10 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/117 (https://phabricator.wikimedia.org/T360704) [14:46:51] (03update) 10taavi: tools: Move Redis DNS record to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/118 (https://phabricator.wikimedia.org/T360704) [14:47:50] 10Toolforge, 06tools-platform-team: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12115193 (10TLepage-WMF) a:03TLepage-WMF [14:55:21] (03merge) 10tlepage: jobs-api: bump to 0.0.537-20260713140924-06e5d09a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1331 (https://phabricator.wikimedia.org/T430550) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:05:52] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team, 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#12115283 (10fnegri) [15:06:05] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#12115285 (10fnegri) [15:06:19] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: memcache is a SPOF for designate/tooz coordination - https://phabricator.wikimedia.org/T422646#12115289 (10fnegri) [15:06:24] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#12115291 (10fnegri) [15:06:27] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089#12115293 (10fnegri) [15:06:33] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Move Cloud VPS control plane alerting to alertmanager - https://phabricator.wikimedia.org/T345294#12115296 (10fnegri) [15:06:39] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones - https://phabricator.wikimedia.org/T313030#12115298 (10fnegri) [15:06:48] 10Tool-wikinewsie: New Wikipedia articles should be their own event type on Wikinewsie - https://phabricator.wikimedia.org/T431879#12115299 (10Pharos) A potential third type of Wikinewsie article would be X-in-year, like Deaths in 2026 or Films of 2026, or such-and-such fire reason or typhoon season, where it's... [15:07:47] 10Cloud Services Proposals, 10Data-Services, 06tools-platform-team, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12115312 (10fnegri) [15:08:52] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12115314 (10fnegri) [15:08:55] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12115317 (10fnegri) [15:09:18] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#12115334 (10fnegri) [15:13:45] 10Data-Services, 06tools-platform-team: [wikireplicas] Gather usage stats - https://phabricator.wikimedia.org/T381587#12115384 (10fnegri) [15:14:24] 10Cloud Services Proposals, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332#12115386 (... [15:14:48] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 2 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12115387 (10fnegri) [15:14:54] 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933#12115388 (10fnegri) [15:20:20] 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12115415 (10fnegri) [15:20:43] 10Toolforge, 06tools-platform-team, 07Documentation: Create a "my first React app" tutorial for Toolforge - https://phabricator.wikimedia.org/T231950#12115416 (10fnegri) [15:20:51] 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph: Permanently set 'noout' for cloudceph - https://phabricator.wikimedia.org/T419877#12115418 (10fnegri) [15:21:01] 10Toolforge, 06tools-platform-team: [clis] standardize the package names - https://phabricator.wikimedia.org/T399080#12115419 (10fnegri) [15:21:12] 10Cloud-VPS, 06tools-infrastructure-team: Controlled cloudsw down tests for F4 - https://phabricator.wikimedia.org/T419658#12115421 (10fnegri) [15:21:22] 10Cloud-VPS, 06tools-infrastructure-team: Deprecated live_migration_uri nova-compute config option - https://phabricator.wikimedia.org/T430129#12115432 (10fnegri) [15:21:31] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12115433 (10fnegri) [15:21:50] 10Cloud-VPS, 06tools-infrastructure-team: oslo.messaging does not failover to the next rabbit host on traffic blackhole situations - https://phabricator.wikimedia.org/T422820#12115435 (10fnegri) [15:22:04] 10Cloud-VPS, 06tools-infrastructure-team: Controlled cloudsw down tests for E4 - https://phabricator.wikimedia.org/T419657#12115436 (10fnegri) [15:22:25] 10Cloud-VPS, 06tools-infrastructure-team: grafana.wmcloud.org unavailable - failed db migration - https://phabricator.wikimedia.org/T418831#12115438 (10fnegri) [15:22:36] 10Cloud-VPS, 06tools-infrastructure-team: Controlled cloudsw down tests for D5 - https://phabricator.wikimedia.org/T419656#12115439 (10fnegri) [15:22:44] 10Cloud-VPS, 06tools-infrastructure-team: Controlled cloudsw down tests for D5 - https://phabricator.wikimedia.org/T419656#12115440 (10fnegri) [15:23:17] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313#12115442 (10fnegri) [15:23:18] 10Cloud-VPS, 06tools-infrastructure-team: Carry out controlled network switch down tests in cloud - https://phabricator.wikimedia.org/T417393#12115454 (10fnegri) [15:23:35] 10Cloud-VPS, 06tools-infrastructure-team: Add monitoring for Metadata service - https://phabricator.wikimedia.org/T395797#12115455 (10fnegri) [15:23:52] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#12115456 (10fgiunchedi) 05Open→03Invalid During the last reboots of clouddumps hosts we have not observed... [15:23:53] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12115459 (10fgiunchedi) [15:23:58] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#12115460 (10fnegri) [15:24:16] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12115461 (10fnegri) [15:24:47] 10Cloud-VPS, 06tools-infrastructure-team: Increase signal of Tofuinfratest alerts - https://phabricator.wikimedia.org/T413185#12115462 (10fnegri) [15:24:54] (03CR) 10Jsn.sherman: [C:03+1] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [15:25:04] 10Toolforge, 06tools-platform-team: Start collecting uptime metrics for toolforge - https://phabricator.wikimedia.org/T412349#12115464 (10fnegri) [15:25:13] 10Cloud-VPS, 06tools-infrastructure-team: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#12115465 (10fnegri) [15:25:28] 10Cloud-VPS, 06tools-infrastructure-team: Fix pacct rotation properly everywhere - https://phabricator.wikimedia.org/T410410#12115467 (10fnegri) [15:25:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#12115468 (10fnegri) [15:25:51] 10Toolforge, 06tools-infrastructure-team: Improve detection of failing ssh to toolforge bastions - https://phabricator.wikimedia.org/T404054#12115469 (10fnegri) [15:26:02] 10Cloud-VPS, 06tools-infrastructure-team: Add fqdn input to instance-related wmcs cookbooks - https://phabricator.wikimedia.org/T404052#12115470 (10fnegri) [15:26:42] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12115474 (10fnegri) [15:27:13] 10Data-Services, 06tools-infrastructure-team: maintain-dbusers: Use cloud-private to talk to NFS servers instead of proxies - https://phabricator.wikimedia.org/T392794#12115477 (10fnegri) [15:27:52] 10Horizon, 10Striker, 06tools-infrastructure-team, 10Prod-Kubernetes, and 2 others: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#12115482 (10fnegri) [15:28:03] 10Cloud-VPS, 06tools-infrastructure-team: [cloudceph] Improve downtime when a switch goes down - https://phabricator.wikimedia.org/T375204#12115483 (10fnegri) [15:28:12] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add webservice support - https://phabricator.wikimedia.org/T362077#12115484 (10fnegri) [15:28:51] 10Toolforge, 06tools-platform-team, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolforge.infra] run and monitor our own sample tools - https://phabricator.wikimedia.org/T357977#12115497 (10fnegri) [15:28:55] 10Cloud-VPS, 06tools-infrastructure-team, 06SRE-OnFire, 13Patch-For-Review, 07Sustainability (Incident Followup): Add external meta-monitoring for metricsinfra - https://phabricator.wikimedia.org/T288053#12115498 (10fnegri) [15:29:04] 10Horizon, 06tools-infrastructure-team: CloudVPS: VMs created with non-allowed characters in the hostname fail to be autosigned by puppet - https://phabricator.wikimedia.org/T192059#12115500 (10fnegri) [15:29:18] 10Cloud-VPS, 06tools-infrastructure-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#12115501 (10fnegri) [15:29:29] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#12115504 (10fnegri) [15:46:01] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12115582 (10CWilliams-WMF) [15:50:37] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [15:52:04] (03update) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [15:53:58] (03update) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [16:19:14] (03approved) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [16:19:19] (03merge) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [16:23:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.538-20260713161930-18e20e8c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1332 (https://phabricator.wikimedia.org/T430071) [16:23:28] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.538-20260713161930-18e20e8c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1332 (https://phabricator.wikimedia.org/T430071) [16:23:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:27:02] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#12115880 (10taavi) 05Open→03Resolved [16:27:29] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [16:36:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:36:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:49:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:51:38] (03CR) 10MusikAnimal: [C:03+2] session: stop locking the session row [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309879 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [16:53:45] (03Merged) 10jenkins-bot: session: stop locking the session row [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309879 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [16:55:06] (03approved) 10dcaro: jobs-api: bump to 0.0.538-20260713161930-18e20e8c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1332 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:55:15] (03merge) 10dcaro: jobs-api: bump to 0.0.538-20260713161930-18e20e8c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1332 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:10:13] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [17:10:41] (03open) 10raymond-ndibe: jobs-api: remove enable_storage config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1333 (https://phabricator.wikimedia.org/T359650) [17:11:17] (03update) 10raymond-ndibe: jobs-api: remove enable_storage config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1333 (https://phabricator.wikimedia.org/T359650 https://phabricator.wikimedia.org/T430071) [17:12:42] (03update) 10raymond-ndibe: jobs-api: remove enable_storage config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1333 (https://phabricator.wikimedia.org/T359650 https://phabricator.wikimedia.org/T430071) [17:12:42] (03close) 10raymond-ndibe: jobs-api: remove enable_storage config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1333 (https://phabricator.wikimedia.org/T359650 https://phabricator.wikimedia.org/T430071) [17:15:07] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [17:17:16] (03open) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [17:17:43] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [17:32:02] RESOLVED: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.55%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [17:36:52] 10Tool-wiki-gender-stats: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12116230 (10Danya) 05Open→03In progress [17:36:58] 10Tool-wiki-gender-stats: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#12116234 (10Danya) 05Open→03In progress [17:37:47] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12116240 (10Aklapper) [18:04:11] (03PS1) 10MusikAnimal: DefaultController: allow logging in as any user via APP_LOGGED_IN_USER [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310143 [18:07:57] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:08:40] (03CR) 10MusikAnimal: [C:03+2] DefaultController: allow logging in as any user via APP_LOGGED_IN_USER [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310143 (owner: 10MusikAnimal) [18:09:30] (03Merged) 10jenkins-bot: DefaultController: allow logging in as any user via APP_LOGGED_IN_USER [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310143 (owner: 10MusikAnimal) [18:09:54] (03PS4) 10Alien4444: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) [18:23:10] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [18:24:25] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [18:26:57] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [18:26:59] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052 (10Danya) 03NEW [18:27:01] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12116450 (10Danya) p:05Triage→03High [18:27:23] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12116451 (10Danya) a:03Danya [18:27:32] 10Tool-wiki-gender-stats: Integrate May 2026 data from local copy - https://phabricator.wikimedia.org/T431933#12116454 (10Danya) [18:27:44] (03CR) 10MusikAnimal: "Just two tiny issues, and some questions:" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [18:30:27] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [18:30:45] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12116464 (10Danya) [18:31:00] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12116466 (10Danya) This may be fixed by switching to ToolsDB. [18:32:14] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [18:32:25] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [18:32:38] (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) [18:33:12] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] (set_port_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [18:33:40] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053 (10Danya) 03NEW [18:33:41] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12116481 (10Danya) p:05Triage→03High [18:34:32] 10Tool-wiki-gender-stats: Fix Content-Security-Policy console error - https://phabricator.wikimedia.org/T427559#12116484 (10Danya) [18:46:33] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' [18:53:53] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' [18:58:22] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12116562 (10BTullis) I have cordoned `dse-k8s-worker2003` ` root@deploy2003:~# kubectl cordon dse-k8s-worker2003.codfw.wmnet nod... [19:01:07] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud [19:01:13] 10Toolforge, 06tools-platform-team, 10GitLab (Account Approval): developer account password resets fail (old email address?) - https://phabricator.wikimedia.org/T432056 (10LuisVilla) 03NEW [19:05:12] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud [20:20:44] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12116957 (10Arnoldokoth) The `scap deploy` was successful after jumping thr... [20:29:17] (03open) 10danyya: Draft: Resolve T422344 "Header items overflow on mobile screens" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/74 [20:30:16] (03update) 10danyya: Draft: Resolve T422344 "Header items overflow on mobile screens" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/74 [20:32:50] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [20:32:51] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12116994 (10Danya) [20:33:00] 10Tool-wiki-gender-stats: Website unable to access database when dumps ingester is running - https://phabricator.wikimedia.org/T432052#12116995 (10Danya) [20:33:17] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927#12116997 (10Danya) [20:33:48] (03update) 10danyya: Resolve T422344 "Header items overflow on mobile screens" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/74 [20:34:02] (03merge) 10danyya: Resolve T422344 "Header items overflow on mobile screens" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/74 [20:34:39] 10Tool-wiki-gender-stats, 13Patch-For-Review: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#12117004 (10Danya) 05In progress→03Resolved [20:36:49] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927#12117008 (10Danya) [20:38:06] 10Tool-global-search: Global-search-tool should not display Wikinews - https://phabricator.wikimedia.org/T432063 (10neriah) 03NEW [20:40:10] 10Tool-wiki-gender-stats: Fix translations - https://phabricator.wikimedia.org/T427211#12117023 (10Danya) [20:40:11] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12117024 (10Danya) [20:48:06] (03open) 10danyya: Draft: Resolve T427211 "Fix translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/75 [20:49:00] (03update) 10danyya: Draft: Resolve T427211 "Fix translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/75 [20:49:07] (03update) 10danyya: Resolve T427211 "Fix translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/75 [20:53:27] (03update) 10raymond-ndibe: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [20:53:29] (03approved) 10raymond-ndibe: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [21:02:33] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12117087 (10Dzahn) @Arnoldokoth I looked at this and wanted to fix the data... [21:02:47] 10Tool-phab-ban, 10Toolforge, 06tools-platform-team, 10Phabricator: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12117089 (10bd808) [21:14:20] 10Tool-wiki-gender-stats, 13Patch-For-Review: Fix translations - https://phabricator.wikimedia.org/T427211#12117111 (10Danya) 05Open→03In progress [21:14:22] (03merge) 10danyya: Resolve T427211 "Fix translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/75 [21:14:44] 10Tool-wiki-gender-stats, 13Patch-For-Review: Fix translations - https://phabricator.wikimedia.org/T427211#12117115 (10Danya) 05In progress→03Resolved [21:15:15] 10Tool-wiki-gender-stats: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937#12117122 (10Danya) 05Open→03In progress [21:15:53] (03open) 10danyya: Draft: Resolve T431937 "Translate card descriptions in About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/76 [21:17:12] (03update) 10raymond-ndibe: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [21:22:12] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [21:25:55] (03update) 10danyya: Draft: Resolve T431937 "Translate card descriptions in About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/76 [21:26:04] (03update) 10danyya: Resolve T431937 "Translate card descriptions in About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/76 [21:26:57] (03merge) 10danyya: Resolve T431937 "Translate card descriptions in About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/76 [21:30:32] 10Tool-wiki-gender-stats, 13Patch-For-Review: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937#12117155 (10Danya) 05In progress→03Resolved [21:37:56] (03merge) 10raymond-ndibe: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [21:41:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) [21:41:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.539-20260713213805-22bde880 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334 (https://phabricator.wikimedia.org/T430578) [21:43:14] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [21:47:04] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [21:47:24] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [21:50:44] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [21:55:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [21:59:18] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [22:02:52] 10Tool-global-search: Global-search-tool should not display Wikinews - https://phabricator.wikimedia.org/T432063#12117223 (10Ladsgroup) [22:03:50] 10Tool-global-search: Global-search-tool should not display Wikinews - https://phabricator.wikimedia.org/T432063#12117230 (10Ladsgroup) It's not really a subtask: https://www.mediawiki.org/wiki/Phabricator/Help#Parent_tasks_and_subtasks [22:07:57] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:14:01] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [22:14:23] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [22:15:17] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [22:15:17] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [22:16:13] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [22:16:15] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [22:28:47] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [23:14:54] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [23:15:23] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [23:21:56] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [23:39:31] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078 (10Raymond_Ndibe) 03NEW [23:40:00] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [23:52:59] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [23:53:08] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [23:54:10] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_publish) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [23:56:41] (03update) 10raymond-ndibe: functional-tests - builds-api - wrap command [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1327 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [23:57:54] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410)