[00:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:11:27] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477#12574071 (10Southparkfan) This VM hosts both RESTBase and a Cassandra 4.1 instance. Upgrading to... [00:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:03:44] 10Tool-delintbot: Handle all sorts of invalid font size values properly - https://phabricator.wikimedia.org/T443890#12574179 (10KomalPandey4) Hi @Redmin, yes! I have completed the WHATWG font-size keyword mapping, percentage parsing logic, and unit tests—all 103 tests and Ruff checks pass locally. My Wikimedia G... [01:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:23:22] (03update) 10ibrahemqasim: Add metadata and pageview update workers [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/42 [01:30:38] (03merge) 10ibrahemqasim: Add metadata and pageview update workers [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/42 [01:32:59] (03open) 10ibrahemqasim: Initialize typed results in worker base [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/43 [01:40:08] 10Tool-k8s-status: k8s-status webservice often crashes with OOM when `/?purge` refresh job runs - https://phabricator.wikimedia.org/T445009#12574217 (10bd808) >>! In T445009#12571568, @Andrew wrote: > Max pinged me about this as well. Is this something we can/should hack around with a quota bump until you have t... [01:42:59] (03merge) 10ibrahemqasim: Initialize typed results in worker base [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/43 [01:59:56] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12574234 (10DB111) This is not communism, so no expropriation/stealing by a passive retr... [02:42:41] (03open) 10emj912: Add QID lookup helper, extend CI, update test plan [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/39 [03:11:38] 10Tool-Pageviews: Massviews analysis of PetScan query by ID "fails to fetch" the PetScan query - https://phabricator.wikimedia.org/T443151#12574309 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Thanks for reporting! It looks like Petscan added CORS to their APIs recently, and that broke the previo... [03:15:19] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44228578877 (https://github.com/cluebotng/component-configs/commits/f2e36fecb128b70e75744cff2a5e1294d14f2b7a) [03:15:24] !log tools.cluebotng-editsets Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44228578722 (https://github.com/cluebotng/component-configs/commits/f2e36fecb128b70e75744cff2a5e1294d14f2b7a) [03:25:43] 10Tool-Pageviews: Massviews is giving an error where it can't see usage of template, maybe a larger issue? - https://phabricator.wikimedia.org/T444295#12574316 (10MusikAnimal) At quick glance, I noticed there was an underscore where a space was that separates the `hastemplate:` and `insource:` filters. I changed... [03:45:40] (03open) 10kitkat1100: T436544: Article linking API over the links data layer [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/40 [04:17:07] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44236149474 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:17:07] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44236149455 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:17:07] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44236149633 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:17:13] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/44236149453 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:19:20] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44236149449 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:19:30] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44236149466 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:19:50] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44236149590 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:19:52] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44236149450 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [04:20:03] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44236149549 (https://github.com/cluebotng/component-configs/commits/2d46f43bac437b295a98c98409f204dc15b6451a) [05:29:31] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12574398 (10Raymond_Ndibe) >>! In T422347#12572544, @Danya wrote: > A temporary solution will be implemented in T445247. Wanted to mention that `push-to-deploy` now supports deploying webservices, so this `Yes Danya,... [05:45:31] (03update) 10ibrahemqasim: Split admin metrics jobs and add jobs dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/44 [05:45:31] (03open) 10ibrahemqasim: Split admin metrics jobs and add jobs dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/44 [05:45:33] (03approved) 10ibrahemqasim: Split admin metrics jobs and add jobs dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/44 [05:47:35] (03merge) 10ibrahemqasim: Split admin metrics jobs and add jobs dashboard [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/44 [06:46:28] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12574458 (10Marostegui) [07:14:14] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44256756881 (https://github.com/cluebotng/component-configs/commits/7ec0a81aaad90bb9f967da5aa4fdd39eb28039f1) [07:14:53] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44256757048 (https://github.com/cluebotng/component-configs/commits/7ec0a81aaad90bb9f967da5aa4fdd39eb28039f1) [08:13:53] (03PS2) 10Thiemo Kreuz (WMDE): Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) [08:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:48:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:00:20] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Bump cpu overcommit to 3 or more - https://phabricator.wikimedia.org/T442997#12574825 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I thought we would have to poke the control plane manually after the configuration change, turns out th... [10:24:25] FIRING: [2x] SystemdUnitFailed: wmf-pt-kill@s3.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:54:25] RESOLVED: [2x] SystemdUnitFailed: wmf-pt-kill@s3.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:03:45] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44290544683 (https://github.com/cluebotng/component-configs/commits/c7dad16534226fd0a07c559c292de7b9e7a7b46d) [11:21:29] FIRING: SystemdUnitCrashLoop: wmf-pt-kill@s2.service crashloop on clouddb1027:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:21:55] FIRING: [2x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:26:29] RESOLVED: SystemdUnitCrashLoop: wmf-pt-kill@s2.service crashloop on clouddb1027:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:31:44] FIRING: [2x] SystemdUnitCrashLoop: wmf-pt-kill@s2.service crashloop on clouddb1027:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:31:55] FIRING: [4x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:32:10] FIRING: [6x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:35:59] RESOLVED: [2x] SystemdUnitCrashLoop: wmf-pt-kill@s2.service crashloop on clouddb1027:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:40:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1076 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [11:41:55] RESOLVED: [6x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:45:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1076 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [11:46:55] FIRING: [8x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:50:29] FIRING: SystemdUnitCrashLoop: wmf-pt-kill@s5.service crashloop on clouddb1033:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:51:55] FIRING: [10x] SystemdUnitFailed: wmf-pt-kill@s2.service on clouddb1027:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:55:29] RESOLVED: SystemdUnitCrashLoop: wmf-pt-kill@s5.service crashloop on clouddb1033:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [12:10:00] (03PS1) 10Lokal Profil: Remove the abandoned OAR location browser [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370942 [12:10:00] (03PS1) 10Lokal Profil: Remove the abandoned WLM 2011 rankings tool [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370943 [12:10:00] (03PS1) 10Lokal Profil: Remove the abandoned contest statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370944 [12:10:01] (03PS1) 10Lokal Profil: Remove the abandoned Layar endpoint and OSM map [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370945 [12:10:02] (03PS1) 10Lokal Profil: api: Escape harvested wikitext when turning it into HTML links [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370946 [12:10:46] (03PS1) 10Lokal Profil: Remove the unused prox_search table filler [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370947 [12:16:55] RESOLVED: [2x] SystemdUnitFailed: wmf-pt-kill@s5.service on clouddb1033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:28:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-test-nfs on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:46:29] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12575575 (10valerio.bozzolan) [12:48:28] (03open) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [12:50:51] (03merge) 10ladsgroup: Add a bash script for generating the prod assets [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/12 [12:53:01] (03CR) 10Lokal Profil: [C:03+2] Remove the abandoned OAR location browser [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370942 (owner: 10Lokal Profil) [12:54:04] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [12:55:10] (03Merged) 10jenkins-bot: Remove the abandoned OAR location browser [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370942 (owner: 10Lokal Profil) [12:55:41] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12575620 (10valerio.bozzolan) @DB111 I'm sorry for the situation. I don't know if this c... [12:56:26] (03CR) 10Lokal Profil: [C:03+2] Remove the abandoned WLM 2011 rankings tool [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370943 (owner: 10Lokal Profil) [12:58:36] (03Merged) 10jenkins-bot: Remove the abandoned WLM 2011 rankings tool [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370943 (owner: 10Lokal Profil) [13:03:19] (03Merged) 10jenkins-bot: Remove the abandoned contest statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370944 (owner: 10Lokal Profil) [13:05:57] (03CR) 10Lokal Profil: [C:03+2] Remove the abandoned Layar endpoint and OSM map [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370945 (owner: 10Lokal Profil) [13:15:18] (03CR) 10Lokal Profil: [C:03+2] api: Escape harvested wikitext when turning it into HTML links [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370946 (owner: 10Lokal Profil) [13:17:23] (03Merged) 10jenkins-bot: api: Escape harvested wikitext when turning it into HTML links [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370946 (owner: 10Lokal Profil) [13:26:50] (03CR) 10Lokal Profil: [C:03+2] Remove the unused prox_search table filler [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370947 (owner: 10Lokal Profil) [13:29:40] (03Merged) 10jenkins-bot: Remove the unused prox_search table filler [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370947 (owner: 10Lokal Profil) [13:31:10] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T440684) [13:31:17] T440684: Decom old cloudcephosd hardware - https://phabricator.wikimedia.org/T440684 [13:35:34] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [13:50:59] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12576042 (10Marostegui) [13:51:09] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12576045 (10Marostegui) [13:51:41] (03open) 10hashar: gerrit: ignore ScheduleDeploymentBot [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/81 [13:52:51] (03PS1) 10Lokal Profil: api: Only offer formats an action can produce [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370987 (https://phabricator.wikimedia.org/T443860) [13:53:59] (03PS2) 10Lokal Profil: api: Only offer formats an action can produce [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1370987 (https://phabricator.wikimedia.org/T443860) [14:06:28] (03open) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [14:11:22] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T440684) [14:11:28] T440684: Decom old cloudcephosd hardware - https://phabricator.wikimedia.org/T440684 [14:50:36] 10Tool-glam-tool-hospital, 10MediaWiki-extensions-Page_Forms: `values from wikidata` should store the underlying QID (while the user still sees and selects a human-readable label) - https://phabricator.wikimedia.org/T430098#12576402 (10Yaron_Koren) Alright, "wikibase" instead of "wikidata" it is - much as I do... [15:15:54] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [15:15:56] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [15:27:40] (03open) 10tlepage: Prometheus: read firing alerts and durations from the rules API [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/21 [15:29:06] (03update) 10tlepage: Prometheus: read firing alerts and durations from the rules API [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/21 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:29:40] (03update) 10tlepage: Prometheus: read firing alerts and durations from the rules API [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/21 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [15:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:39:13] (03open) 10hashar: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) [15:39:25] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [15:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:40:51] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44337459717 (https://github.com/cluebotng/component-configs/commits/9dc8d20e574dddcedb870a2d904d439d3c0795c1) [15:42:35] (03open) 10atitkov: Add the WikiRun content builder as a standalone Toolforge page [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/6 [15:42:39] (03CR) 10Krinkle: [C:03+2] Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) (owner: 10Thiemo Kreuz (WMDE)) [15:43:10] (03CR) 10CI reject: [V:04-1] Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) (owner: 10Thiemo Kreuz (WMDE)) [15:48:31] (03merge) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [15:48:38] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-state-metrics [15:48:41] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kube-state-metrics [15:51:09] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-state-metrics [15:51:13] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kube-state-metrics [15:52:19] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [15:54:32] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kube-state-metrics [15:54:36] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kube-state-metrics [15:55:31] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [15:55:35] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics [15:55:49] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component metrics [15:55:53] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component metrics [15:56:30] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [15:56:33] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics [15:58:00] (03update) 10andrew: envvars-admission: bump to 0.0.45-20261117175433-df5a31d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1528 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:03:51] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [16:04:17] (03CR) 10Krinkle: Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) (owner: 10Thiemo Kreuz (WMDE)) [16:04:19] (03CR) 10Krinkle: [C:03+2] Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) (owner: 10Thiemo Kreuz (WMDE)) [16:04:56] (03Merged) 10jenkins-bot: Fix incompatible setlocale() argument [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1369105 (https://phabricator.wikimedia.org/T410935) (owner: 10Thiemo Kreuz (WMDE)) [16:05:41] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:07:49] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [16:09:15] (03merge) 10andrew: envvars-admission: bump to 0.0.45-20261117175433-df5a31d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1528 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:09:21] !log andrew@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico [16:09:27] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:09:30] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:09:35] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars [16:09:36] 10Toolforge, 06tools-platform-team: [sample apps] update components config to expected standard - https://phabricator.wikimedia.org/T444784#12576953 (10komla) a:03komla [16:09:38] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars [16:11:13] (03open) 10tlepage: Prometheus: read firing alerts and durations from the rules API [repos/cloud/toolforge/alerts-api] (reduce-prometheus-calls) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/22 (https://phabricator.wikimedia.org/T441144) [16:14:14] (03update) 10andrew: calico: bump to 0.0.20-20261117225244-194bb79d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1531 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:15:38] (03merge) 10andrew: calico: bump to 0.0.20-20261117225244-194bb79d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1531 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:15:53] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:15:56] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:16:09] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:16:12] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:17:01] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:17:04] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:19:37] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:19:41] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:25:53] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:25:57] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:28:43] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [16:31:20] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:31:23] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:32:46] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars [16:32:49] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars [16:37:02] (03update) 10andrew: Update to k8s 1.34.13 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/350 (https://phabricator.wikimedia.org/T414595) [16:37:37] 10Toolforge, 06tools-platform-team: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224#12577125 (10Raymond_Ndibe) [16:37:52] 10Toolforge, 06tools-platform-team: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224#12577128 (10Raymond_Ndibe) [16:38:33] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:38:34] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12577132 (10Raymond_Ndibe) [16:38:36] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [16:45:50] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12577177 (10Danya) Ah thank you @Raymond_Ndibe, I didn’t know it was out already! Is there any documentation available? [16:46:07] (03update) 10hashar: Set bot mode (+B) when connecting [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/82 (https://phabricator.wikimedia.org/T445359) [16:56:38] 10Tool-k8s-status: k8s-status webservice often crashes with OOM when `/?purge` refresh job runs - https://phabricator.wikimedia.org/T445009#12577204 (10bd808) @Max is interested in helping fix this. I have just added him as a co-maintainer for the tool so he can poke around more and help define the extent of the... [16:59:03] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12577214 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:59:18] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12577218 (10Raymond_Ndibe) 05Open→03In progress [17:12:51] (03open) 10damian: kind - re-create the cluster on a config change [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/351 (https://phabricator.wikimedia.org/T380060) [17:14:07] 10Toolforge, 06tools-infrastructure-team, 07IPv6, 07Kubernetes, 13Patch-For-Review: Support IPv6 in Toolforge Kubernetes - https://phabricator.wikimedia.org/T380060#12577320 (10taavi) [17:14:22] (03open) 10damian: kind - disable the default CNI [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/352 (https://phabricator.wikimedia.org/T380060) [17:16:56] (03open) 10damian: basic_system - explicitly configure dns resolvers [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/353 (https://phabricator.wikimedia.org/T380060) [17:18:44] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12577339 (10Raymond_Ndibe) >>! In T422347#12577177, @Danya wrote: > Ah thank you @Raymond_Ndibe, I didn’t know it was out already! Is there any documentation available? Yes! we have https://wikitech.wikimedia.org/wik... [17:18:50] (03open) 10damian: kind - allow configuring dual stack networking [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/354 (https://phabricator.wikimedia.org/T380060) [17:21:48] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529#12577358 (10Chippyy) Adding @saper as I believe he has recently been working on maps-wma2. Hopefully there's some progress upgrading php on... [17:23:03] (03open) 10damian: Draft: Add IPv6 cluster subnet support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/355 (https://phabricator.wikimedia.org/T380060) [17:24:33] (03open) 10damian: Draft: Add IPv6 tunnel support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/356 (https://phabricator.wikimedia.org/T380060) [17:25:22] (03open) 10damian: Draft: namespace - add IPv6 offset annotation [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/96 (https://phabricator.wikimedia.org/T380060) [17:25:46] (03open) 10damian: Draft: Support enabling IPv6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/25 (https://phabricator.wikimedia.org/T380060) [17:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:42:19] (03update) 10damian: Draft: Add IPv6 tunnel support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/356 (https://phabricator.wikimedia.org/T380060) [18:12:19] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44366469744 (https://github.com/cluebotng/component-configs/commits/06c98e664cb03f4ac47947288254f7b5ea75d2a3) [18:13:13] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44366469562 (https://github.com/cluebotng/component-configs/commits/06c98e664cb03f4ac47947288254f7b5ea75d2a3) [18:25:40] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529#12577722 (10taavi) 05Open→03Resolved a:03taavi Hi! `maps-wma2` is still showing as running a vulnerable PHP 7.4 version. However,... [18:27:08] 10Cloud-VPS, 06tools-infrastructure-team: Re-audit math CloudVPS project - https://phabricator.wikimedia.org/T442858#12577730 (10taavi) Hello! Any updates here? [18:32:59] (03update) 10atitkov: Add the WikiRun content builder as a standalone Toolforge page [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/6 [18:44:35] (03CR) 10Lokal Profil: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270241 (owner: 10Lokal Profil) [19:06:39] (03update) 10atitkov: Add the WikiRun content builder as a standalone Toolforge page [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/6 [19:10:23] (03merge) 10atitkov: Add the WikiRun content builder as a standalone Toolforge page [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/6 [20:03:57] (03update) 10atitkov: Point OPS deployment commands at the games-native-wikirun tool [toolforge-repos/wikirun-game] (native-integration) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/5 [20:06:30] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12578171 (10bd808) >>! In T440145#12574234, @DB111 wrote: > This is not communism, so no... [20:12:17] FIRING: JobUnavailable: Reduced availability for job pdns_rec in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:17:17] RESOLVED: JobUnavailable: Reduced availability for job pdns_rec in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:38:53] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12578385 (10Danya) Thank you very much! I’ll take a look :) [20:40:39] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/44392912293 (https://github.com/cluebotng/component-configs/commits/4c74cf4dd2a5ad23777606afea2b0a62a5e3517a) [21:01:14] 10Cloud-VPS, 06tools-infrastructure-team: cloud-vps: run unattended upgrades more often - https://phabricator.wikimedia.org/T445398 (10Andrew) 03NEW [21:19:52] (03open) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:21:29] (03update) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:39:23] (03open) 10atitkov: Rework the draft list into an ordered, reorderable table [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/7 [21:39:32] (03merge) 10atitkov: Rework the draft list into an ordered, reorderable table [toolforge-repos/wikirun-game] (native-integration-content) - 10https://gitlab.wikimedia.org/toolforge-repos/wikirun-game/-/merge_requests/7 [21:40:59] (03update) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:41:05] (03approved) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:41:14] (03update) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:42:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:42:57] (03merge) 10ibrahemqasim: Add long-term error log statistics [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/45 [21:49:08] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12578758 (10aputhin) @DB111 if I may ask, would you be able to elaborate on your concern... [22:16:32] 10Cloud-VPS, 06tools-infrastructure-team: Re-audit math CloudVPS project - https://phabricator.wikimedia.org/T442858#12578880 (10Ihlec) yes we disabled the old unused containers and patched the others. [22:38:06] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1003.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 23d 16h 44m 19s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:51:19] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12579233 (10bd808) >>! In T440145#12557470, @bd808 wrote: >>>! In T440145#12556517, @DB1... [23:52:05] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12579234 (10bd808) >>! In T440145#12420330, @Bugreporter wrote: >>>! In T440145#1242001...