[00:28:39] FIRING: [5x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:33:39] RESOLVED: [5x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:38:34] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12569103 (10Amdrel) `ImageAnnotateTool` is new, yes. `SVGTranslate` and `ChartWizard` are as well. [02:19:20] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44001624548 (https://github.com/cluebotng/component-configs/commits/ffcbef963735db1f49a7c496c1092fc3615ed9c8) [02:42:31] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12569134 (10JackNunn) @Yaron_Koren - thanks, and to answer your question directly: **ye... [02:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:26:17] (03PS1) 10MusikAnimal: PageInfo: add noprint class to gadget HTML [labs/xtools] - 10https://gerrit.wikimedia.org/r/1370231 [03:30:17] (03open) 10sonali-7: working db functions for adding links [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/37 [03:30:21] (03CR) 10MusikAnimal: [C:03+2] PageInfo: add noprint class to gadget HTML [labs/xtools] - 10https://gerrit.wikimedia.org/r/1370231 (owner: 10MusikAnimal) [03:30:35] (03close) 10sonali-7: Draft: working database functions for adding links [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/36 [03:30:51] (03Merged) 10jenkins-bot: PageInfo: add noprint class to gadget HTML [labs/xtools] - 10https://gerrit.wikimedia.org/r/1370231 (owner: 10MusikAnimal) [03:32:08] (03update) 10zhaofjx: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 (owner: 10chimaifeanyi29) [03:32:10] (03update) 10zhaofjx: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 (owner: 10chimaifeanyi29) [03:32:38] (03update) 10zhaofjx: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 (owner: 10chimaifeanyi29) [03:32:46] (03update) 10zhaofjx: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 (owner: 10chimaifeanyi29) [03:32:53] (03update) 10zhaofjx: T438640: link recent changes to article card instead of Wikidata [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/24 (owner: 10chimaifeanyi29) [03:38:36] 10Tool-wikinewsie: Deployment Failure after Wikinewsie Localization Update - https://phabricator.wikimedia.org/T445172 (10ZhaoFJx) 03NEW [03:38:57] 10Tool-wikinewsie: Deployment Failure after Wikinewsie Localization Update - https://phabricator.wikimedia.org/T445172#12569207 (10ZhaoFJx) p:05Triage→03Unbreak! [03:40:15] 10Tool-wikinewsie, 13Patch-For-Review: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12569209 (10ZhaoFJx) p:05Triage→03High [03:40:20] 10Tool-wikinewsie, 13Patch-For-Review: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12569211 (10ZhaoFJx) 05Open→03In progress [03:40:27] 10Tool-wikinewsie, 13Patch-For-Review: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12569212 (10ZhaoFJx) [03:40:44] (03update) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:40:44] (03update) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:40:56] (03update) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:40:56] (03approved) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:41:08] (03update) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:42:16] (03merge) 10zhaofjx: T444223: Show 3d and 7d time window buttons on the Front Page [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/35 (owner: 10chimaifeanyi29) [03:42:57] 10Tool-wikinewsie, 13Patch-For-Review: Extend time windows to 7d or longer on Wikinewsie Front Page - https://phabricator.wikimedia.org/T444223#12569216 (10ZhaoFJx) 05In progress→03Resolved [04:47:39] (03open) 10pharos: Revert "T444223: Show 3d and 7d time window buttons on the Front Page" [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/38 [04:47:57] (03merge) 10pharos: Revert "T444223: Show 3d and 7d time window buttons on the Front Page" [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/38 [05:54:20] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12569311 (10Yaron_Koren) Alright, great - and I agree that it would be good to have thi... [06:30:40] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12569353 (10Doc_James) There are actually two Image Annotate Tools. This is the one we are refering to https://commons.wikimedia.org/wiki/Commons:ImageAnnotateT... [06:47:07] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12569378 (10Marostegui) [06:48:39] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12569391 (10Marostegui) [06:48:46] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12569394 (10Marostegui) [07:19:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:07:00] (03open) 10r4356th: Remove stray div and span tags to fix stripped-tag errors [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/60 (https://phabricator.wikimedia.org/T442319) [08:13:36] (03update) 10r4356th: Remove stray div and span tags to fix stripped-tag errors [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/60 (https://phabricator.wikimedia.org/T442319) [08:18:12] (03merge) 10r4356th: Remove stray div and span tags to fix stripped-tag errors [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/60 (https://phabricator.wikimedia.org/T442319) [08:19:18] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Remove old Developer account association from Phabricator account - https://phabricator.wikimedia.org/T445179 (10MareikeHeuerWMDE) 03NEW [08:19:20] 10Tool-delintbot: Fix stripped-tag errors - https://phabricator.wikimedia.org/T442319#12569499 (10Redmin) 05Open→03Resolved a:03Redmin [08:19:48] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Remove old Developer account association from Phabricator account - https://phabricator.wikimedia.org/T445179#12569506 (10MareikeHeuerWMDE) a:03MareikeHeuerWMDE [08:31:33] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations: Consider reducing memory from cloudbackup200X hosts - https://phabricator.wikimedia.org/T444903#12569514 (10fgiunchedi) I'm +1 on reducing memory on these hosts. On cloudbackup2 there are 64GB sticks and cloudbackup1, cloudweb2002-dev ha... [08:54:26] 10Tool-itwiki: Harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12569608 (10valerio.bozzolan) [09:00:08] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Cleanup broken oidentd implementation from cloudnet hosts - https://phabricator.wikimedia.org/T445113#12569626 (10taavi) 05Open→03Resolved [09:06:06] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: BackupCoverageDrop: Only 50 distinct volume objects are backed up on cloudbackup2004:9100 (codfw), below 90% of the level 24h ago. Some backups may have been lost. - https://phabricator.wikimedia.org/T443945#12569688 (10fgiunchedi) 05Open→0... [09:12:37] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12569704 (10GiovanniPen) Carusi ma secondo me è ridicolo che una roba su toolforge non a... [09:17:01] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations: The Cloud intermediate CA is going to expire on Dec 12th - https://phabricator.wikimedia.org/T442012#12569776 (10fgiunchedi) @LSobanski this is good to go on WMCS' end, i.e. good to proceed to renew the intermediate . what is going to be... [09:53:12] (03CR) 10Lokal Profil: [C:03+2] api: Rename Language to RegionNames [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362179 (https://phabricator.wikimedia.org/T443562) (owner: 10Lokal Profil) [09:55:25] (03Merged) 10jenkins-bot: api: Rename Language to RegionNames [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362179 (https://phabricator.wikimedia.org/T443562) (owner: 10Lokal Profil) [10:13:53] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44062399857 (https://github.com/cluebotng/component-configs/commits/473c7f59574d0bc974be930751284a0085ee0ac3) [10:14:35] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44062399818 (https://github.com/cluebotng/component-configs/commits/473c7f59574d0bc974be930751284a0085ee0ac3) [10:15:20] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44062399939 (https://github.com/cluebotng/component-configs/commits/473c7f59574d0bc974be930751284a0085ee0ac3) [10:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:08:06] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12570215 (10valerio.bozzolan) (@GiovanniPen has been informed about this task as was act... [11:10:38] 10Tool-itwiki: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12570232 (10valerio.bozzolan) p:05Low→03High a:03valerio.bozzolan Raise priority as at least 1 user reported that maybe the tool stopped working. https://it.wikipedia.org/wiki/Di... [11:16:26] (03PS1) 10Lokal Profil: api: Log a failed countries cache write instead of warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370463 (https://phabricator.wikimedia.org/T445190) [11:16:50] (03PS1) 10Lokal Profil: api: List each country's languages in action=countries JSON [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370464 (https://phabricator.wikimedia.org/T445191) [11:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:29:49] 10Horizon, 10Striker, 06tools-infrastructure-team: striker and openstack-dashboard crashloop on cloudweb1003 reboot - https://phabricator.wikimedia.org/T445201 (10fgiunchedi) 03NEW [11:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:26:19] 06tools-platform-team, 10Toolhub: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206 (10fnegri) 03NEW [12:28:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-test-nfs on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:29:33] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12570585 (10fnegri) I have added @TLepage-WMF and @Raymond_Ndibe to the Gerrit group, and opened https://gerrit.wikimedia.org/r/c/operations/puppet/+/1370485 t... [12:29:40] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12570586 (10fnegri) [12:30:29] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12570588 (10fnegri) 05Open→03In progress [12:33:43] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12570616 (10Raymond_Ndibe) I think I already have this access @fnegri , not sure I can remember how that was configured, but yeaa I've deployed toolhub things... [12:35:55] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12570640 (10taavi) >>! In T445206#12570584, @fnegri wrote: > The Puppet patch needs approval by @thcipriani or @dancy as they are listed as the "approvers" for... [12:46:16] 10Horizon, 10Striker, 06tools-infrastructure-team: striker and openstack-dashboard crashloop on cloudweb1003 reboot - https://phabricator.wikimedia.org/T445201#12570710 (10taavi) a:03taavi This is a conflict with a nginx install manually configured to serve a php2shell-era error page. I'll reimage the box... [12:46:17] 10Horizon, 10Striker, 06tools-infrastructure-team: striker and openstack-dashboard crashloop on cloudweb1003 reboot - https://phabricator.wikimedia.org/T445201#12570712 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by taavi@cumin1004 for host cloudweb1003.wikimedia.org with OS trixie [12:47:46] (03PS1) 10Lokal Profil: build: Drop the removed common/ directory from the ESLint excludes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370498 [12:48:35] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44086001203 (https://github.com/cluebotng/component-configs/commits/2aba31b86c0dfb5cb00fc7a6033e50956c5af3af) [13:08:15] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12570818 (10Samwilson) > a new parameter called "all values from wikidata" That'd be u... [13:10:55] (03CR) 10Lokal Profil: [C:03+2] build: Drop the removed common/ directory from the ESLint excludes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370498 (owner: 10Lokal Profil) [13:11:46] (03update) 10raymond-ndibe: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) (owner: 10damian) [13:13:14] (03Merged) 10jenkins-bot: build: Drop the removed common/ directory from the ESLint excludes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370498 (owner: 10Lokal Profil) [13:28:49] 10Horizon, 10Striker, 06tools-infrastructure-team: striker and openstack-dashboard crashloop on cloudweb1003 reboot - https://phabricator.wikimedia.org/T445201#12570889 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by taavi@cumin1004 for host cloudweb1003.wikimedia.org with OS trixie co... [13:31:07] 10Horizon, 10Striker, 06tools-infrastructure-team: striker and openstack-dashboard crashloop on cloudweb1003 reboot - https://phabricator.wikimedia.org/T445201#12570898 (10taavi) 05Open→03Resolved `lang=shell-session taavi@cumin1004 ~ $ sudo confctl select name=cloudweb1003.wikimedia.org set/pooled=y... [13:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:53:06] 10Tool-inteGraality: Add type-checker to integraality - https://phabricator.wikimedia.org/T445213 (10JeanFred) 03NEW [13:54:58] (03update) 10raymond-ndibe: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) (owner: 10damian) [14:11:05] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations: Consider reducing memory from cloudbackup200X hosts - https://phabricator.wikimedia.org/T444903#12571085 (10Andrew) + agreed that we don't need all that RAM. It really comes down to whether the cost savings is worth de-standardizing things. [14:39:45] (03merge) 10tlepage: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:42:20] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.18-20261117143955-7726c142 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1524 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:42:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.18-20261117143955-7726c142 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1524 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:43:29] (03update) 10tlepage: alerts-api: Move prometheusUrls to values top level [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1523 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:44:30] (03update) 10tlepage: alerts-api: Move prometheusUrls to values top level [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1523 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:45:41] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [14:46:39] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [14:48:23] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [14:49:28] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [14:50:52] (03merge) 10tlepage: alerts-api: Move prometheusUrls to values top level [repos/cloud/toolforge/toolforge-deploy] (bump_alerts-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1523 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:50:53] (03update) 10tlepage: alerts-api: bump to 0.0.18-20261117143955-7726c142 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1524 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:52:36] (03merge) 10tlepage: alerts-api: bump to 0.0.18-20261117143955-7726c142 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1524 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:59:35] (03update) 10tlepage: http_session: move make_session out of the poller [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [14:59:51] (03update) 10tlepage: http_session: move make_session out of the poller [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:00:05] (03update) 10tlepage: http_session: move make_session out of the poller [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:01:57] (03merge) 10tlepage: http_session: move make_session out of the poller [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:05:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.19-20261117150214-f8417307 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1525 [15:05:43] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12571355 (10Redmin) [15:05:51] (03open) 10r4356th: Remove file options from individual files in galleries [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/61 (https://phabricator.wikimedia.org/T421530) [15:09:44] (03update) 10r4356th: Remove file options from individual files in galleries [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/61 (https://phabricator.wikimedia.org/T421530) [15:13:44] (03merge) 10r4356th: Remove file options from individual files in galleries [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/61 (https://phabricator.wikimedia.org/T421530) [15:14:08] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [15:14:37] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12571429 (10Redmin) [15:15:05] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [15:16:55] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [15:17:56] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [15:18:35] (03merge) 10tlepage: alerts-api: bump to 0.0.19-20261117150214-f8417307 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1525 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:19:28] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12571477 (10fnegri) > I think I already have this access @fnegri , not sure I can remember how that was configured, but yeaa I've deployed toolhub things end t... [15:22:58] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Give access to tlepage and rndibe to deploy Toolhub - https://phabricator.wikimedia.org/T445206#12571481 (10Raymond_Ndibe) @fnegri I'll check later, Thanks for creating! [15:28:06] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 24d 20h 24m 31s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:39:20] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:39:31] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] (dummy-api-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:40:20] 10Tool-k8s-status: k8s-status webservice often crashes with OOM when `/?purge` refresh job runs - https://phabricator.wikimedia.org/T445009#12571568 (10Andrew) Max pinged me about this as well. Is this something we can/should hack around with a quota bump until you have time to rework it fully? [15:46:49] 10Tool-delintbot: Edit across all namespaces in one run based on a sensible list - https://phabricator.wikimedia.org/T415890#12571595 (10Redmin) [16:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:17:18] (03update) 10fnegri: http_session: move make_session out of the poller [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:18:46] (03open) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [16:33:12] (03update) 10ladsgroup: Added initial examples to adoption gallery [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/9 (owner: 10dtorsani) [16:33:35] 10Striker: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232 (10fnegri) 03NEW [16:33:44] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12571941 (10fnegri) [16:33:50] (03merge) 10ladsgroup: Added initial examples to adoption gallery [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/9 (owner: 10dtorsani) [16:34:12] (03open) 10andrew: Draft: Testcheck [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/40 [16:34:56] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12571952 (10fnegri) [16:37:32] (03update) 10andrew: Draft: Testcheck [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/40 [16:38:45] (03update) 10fnegri: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:38:45] (03approved) 10fnegri: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:38:48] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Remove old Developer account association from Phabricator account - https://phabricator.wikimedia.org/T445179#12571966 (10bd808) 05Open→03In progress a:05MareikeHeuerWMDE→03bd808 [16:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:42:30] (03close) 10andrew: Draft: Testcheck [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/40 [16:46:14] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [16:47:26] (03open) 10ladsgroup: Add a bash script for generating the prod assets [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/12 [16:49:46] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [16:51:17] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:52:16] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [16:53:29] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12572033 (10Arendpieter) For moving Striker off LDAP auth: https://gerrit.wikimedia.org/r/c/labs/striker/+/1250537 (T359554) replaces the LDAP password login with CAS SSO via idp.wikimedia.org. Striker stil... [16:54:23] (03merge) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:54:23] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:57:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.20-20261117165433-0248f343 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1526 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:57:27] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.20-20261117165433-0248f343 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1526 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:58:02] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [16:58:56] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [17:01:35] (03PS9) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1250537 (https://phabricator.wikimedia.org/T359554) [17:04:05] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [17:05:03] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [17:06:00] (03approved) 10raymond-ndibe: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:06:20] (03unapproved) 10raymond-ndibe: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:07:27] (03merge) 10tlepage: alerts-api: bump to 0.0.20-20261117165433-0248f343 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1526 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:07:48] (03approved) 10raymond-ndibe: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:08:33] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [17:22:01] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Remove old Developer account association from Phabricator account - https://phabricator.wikimedia.org/T445179#12572263 (10bd808) 05In progress→03Resolved I don't know that I have ever documented this, so I will make some verbose notes here. Ob... [17:27:17] (03update) 10andrew: Update deps for K8s 1.33 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) [17:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:40:59] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [17:42:43] 10Openstack-Magnum, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Doing 😎): Rebuild the cloud-runner-staging magnum cluster in the zuul3 project with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437498#12572429 (10d... [17:43:54] (03merge) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [17:44:53] 10Striker, 06tools-platform-team: Striker should know which tool a specific tool maintains - https://phabricator.wikimedia.org/T443086#12572447 (10bd808) This would be similar to the "Your tools" list at https://toolsadmin.wikimedia.org/tools/ that shows an authenticated user the list of tools they have mainta... [17:46:24] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 [17:46:44] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.21-20261117174404-df623e1a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1527 [17:46:47] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.21-20261117174404-df623e1a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1527 [17:47:32] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [17:48:26] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [17:48:31] (03approved) 10fnegri: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 (owner: 10andrew) [17:50:53] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [17:51:03] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12572486 (10Yaron_Koren) Hm, yes, you might be right - showing the description is indis... [17:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:51:50] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api [17:54:10] (03update) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T414595) [17:54:17] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12572499 (10bd808) Just a note that Striker is currently deployed as https://toolsadmin-toolsbeta.wikimedia.org/ as well in a parallel Docker container. This second deployment would also ideally be moved. T... [17:54:23] (03merge) 10andrew: Update dependencies for k8s 1.34.12 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T414595) [17:54:40] (03merge) 10tlepage: alerts-api: bump to 0.0.21-20261117174404-df623e1a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1527 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:55:44] 10Tool-wiki-gender-stats: [TF] Setup temporary push-to-deploy - https://phabricator.wikimedia.org/T445247 (10Danya) 03NEW [17:55:58] 10Tool-wiki-gender-stats: [TF] Setup temporary push-to-deploy - https://phabricator.wikimedia.org/T445247#12572533 (10Danya) p:05Triage→03High [17:56:04] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12572537 (10bd808) >>! In T445232#12572033, @Arendpieter wrote: > For moving Striker off LDAP auth: https://gerrit.wikimedia.org/r/c/labs/striker/+/1250537 (T359554) replaces the LDAP password login with CA... [17:56:34] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12572544 (10Danya) A temporary solution will be implemented in T445247. [17:57:57] (03close) 10andrew: Update deps for K8s 1.33 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) [17:59:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-admission: bump to 0.0.45-20261117175433-df5a31d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1528 [18:00:56] (03approved) 10fnegri: Update Calico to 3.31.7 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/23 (https://phabricator.wikimedia.org/T441022) (owner: 10andrew) [18:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:05:21] 10Striker, 06tools-platform-team: Deploy Striker on WikiKube - https://phabricator.wikimedia.org/T445232#12572606 (10bd808) > I still dream of running striker inside toolforge proper, although that would require the significant LDAP access refactorings I've been dreaming for quite a while Just for the sake of... [18:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:09:45] 10Cloud-VPS, 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 07Puppet: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12572634 (10dancy) 05Open→03Resolved a:03dancy After letting https://gerrit.wikimedia.or... [18:27:15] (03update) 10fnegri: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [18:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:44:28] (03merge) 10andrew: Update Calico to 3.31.7 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/23 (https://phabricator.wikimedia.org/T441022) [18:46:12] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20261117184447-af8f7bc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1529 (https://phabricator.wikimedia.org/T441022) [18:46:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20261117184447-af8f7bc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1529 (https://phabricator.wikimedia.org/T441022) [18:48:08] (03merge) 10andrew: calico: bump to 0.0.20-20261117184447-af8f7bc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1529 (https://phabricator.wikimedia.org/T441022) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [19:05:14] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Remove old Developer account association from Phabricator account - https://phabricator.wikimedia.org/T445179#12572840 (10bd808) https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Striker/FAQ#Remove_an_unwanted_Phabricator_account_connection [19:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:53:49] (03open) 10andrew: calico: update kubeVersion for calico 3.31.7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1530 [19:54:23] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44163075399 (https://github.com/cluebotng/component-configs/commits/5d19f4ad2380fba1f484650fc3e56a76c9a4fbc0) [19:56:43] (03merge) 10andrew: calico: update kubeVersion for calico 3.31.7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1530 [20:00:41] 06tools-infrastructure-team, 06DC-Ops, 06SRE, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12573235 (10BTullis) >>! In T441202#12540066, @xcollazo wrote: > This is my pitch to go with RAID6: OK,... [21:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:48:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:57:34] (03open) 10andrew: Chart template: remove the 'beta' from beta.kubernetes.io/os [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/24 [22:16:52] (03CR) 10Lokal Profil: [C:03+2] api: Fetch the fields the map formats read [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1364771 (https://phabricator.wikimedia.org/T443832) (owner: 10Lokal Profil) [22:19:16] (03Merged) 10jenkins-bot: api: Fetch the fields the map formats read [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1364771 (https://phabricator.wikimedia.org/T443832) (owner: 10Lokal Profil) [22:29:40] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/calico/ctl:3.31.7 [22:29:51] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/calico/ctl:3.31.7 [22:34:37] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/calico/ctl:v3.31.7 [22:34:40] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/calico/ctl:v3.31.7 [22:38:06] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 24d 13h 16m 31s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [22:52:29] (03merge) 10andrew: Chart template: remove the 'beta' from beta.kubernetes.io/os [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/24 [22:53:59] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20261117225244-194bb79d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1531 [22:53:59] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20261117225244-194bb79d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1531 [23:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:45:12] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12573925 (10Southparkfan) 05In progress→03Resolved [23:46:44] (03update) 10atitkov: Embed the WikiRun web game in the native apps [toolforge-repos/wikirun-game] (wikirun-native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/4 (https://phabricator.wikimedia.org/T442998) [23:47:19] (03merge) 10atitkov: Embed the WikiRun web game in the native apps [toolforge-repos/wikirun-game] (wikirun-native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/4 (https://phabricator.wikimedia.org/T442998) [23:49:36] (03open) 10ibrahemqasim: Add metadata and pageview update workers [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/42 [23:51:18] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12573958 (10Samwilson) Yeah, `wikibase` as a name sounds good to me. I'm not sure about... [23:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown