[00:27:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [02:17:14] (03PS1) 10Jsn.sherman: I18nHelper: guard dates against bad ICU locales [labs/xtools] - 10https://gerrit.wikimedia.org/r/1314218 (https://phabricator.wikimedia.org/T384711) [05:31:56] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:41:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:23:40] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12148600 (10Marostegui) [07:24:09] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12148601 (10Marostegui) clouddb1031 pooled in s2 and s7 [07:56:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#12148642 (10dcaro) 05Openā†’03Resolved a:03dcaro I have not seen failing in a few months, I'll close for now. [07:57:04] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [toolsbeta] probe flapping on ipv6 only - https://phabricator.wikimedia.org/T426584#12148647 (10dcaro) Fyi. Today tools probe also flapped once, on ip4 though (https://lists.wikimedia.org/hyperkitty/list/cloud-admin-fee... [08:13:13] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12148700 (10jnuche) Pinging @Andrew since the Catalyst team originally discussed this quota increase with him [08:14:55] 10Tool-wikinewsie: Live pageviews for Wikinewsie - https://phabricator.wikimedia.org/T432576#12148702 (10Pharos) This looks like the most relevant documentation https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Pageview_hourly [08:18:32] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12148711 (10KBach) [08:23:05] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Live pageviews for Wikinewsie - https://phabricator.wikimedia.org/T432576#12148717 (10Pharos) [08:31:15] (03approved) 10dcaro: debian-builder: update to latest images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/92 (https://phabricator.wikimedia.org/T426827) (owner: 10filippo) [08:34:23] (03update) 10filippo: debian-builder: update to latest images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/92 (https://phabricator.wikimedia.org/T426827) [08:34:42] (03merge) 10filippo: debian-builder: update to latest images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/92 (https://phabricator.wikimedia.org/T426827) [08:35:02] 10Tool-wikinewsie: Wikinewsie cards sometimes halve top cut off on mobile - https://phabricator.wikimedia.org/T432935 (10Pharos) 03NEW [08:35:25] 10Tool-wikinewsie: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12148782 (10Pharos) [08:47:46] 10Toolforge, 06tools-platform-team: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12148810 (10dcaro) Some workaround, for: ` WARN[0000] failed to find executable from os.Args[0] error="os.Args[0] is invalid: exec: \"limactl\": executable file not found in $PATH" WARN[00... [08:50:28] 10Toolforge, 06tools-platform-team: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12148816 (10dcaro) > That points to `limactl shell` actually forcing to run that command, instead of bashrc/other config trying to do it. > Potentially related https://github.com/lima-vm/lima/... [08:56:19] 10Toolforge, 06tools-platform-team: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12148840 (10dcaro) >>! In T429343#12148816, @dcaro wrote: >> That points to `limactl shell` actually forcing to run that command, instead of bashrc/other config trying to do it. >> Potentially... [09:00:39] (03open) 10filippo: dummy to test ci [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/115 [09:03:34] (03close) 10filippo: dummy to test ci [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/115 [09:03:35] (03update) 10filippo: dummy to test ci [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/115 [09:06:11] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938 (10Pharos) 03NEW [09:08:13] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12148878 (10Pharos) [09:11:34] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962#12148887 (10Pharos) This might be a good tool to use for this task, and other small semi-automated acts of annotation https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Large_Lan... [09:28:08] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29995256262 (https://github.com/cluebotng/component-configs/commits/c08c3e19100546a5f8ed26a2bfa7095cb4633147) [09:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:33:03] 10Tool-wikinewsie: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881#12149018 (10Pharos) Of course, another partial solution would be for the top 3 images to be square rather than horizontal, but that would only work part of the time and might be aesthetically bad, [09:34:50] 10Toolforge, 06tools-platform-team: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12149022 (10dcaro) > It's not, it works only if you are starting from your home, if you start from a different dir it fails to resolve the mounted home: > ` > 10:56 AM ~/Work/wikimedia/cloud_w... [09:47:21] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881#12149052 (10Pharos) [09:48:56] 10Tool-wikinewsie: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12149056 (10VIUK) Hi, Iā€™m interested in investigating this issue. Could you provide the URL of a Wikinewsie card where this problem occurs? I can test it on mobile browsers and look into the resp... [09:52:01] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12149072 (10Pharos) [09:52:04] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881#12149076 (10Csisc) At style.css in the src folder, there is the style settings for the "card-media" class from line 183 to line 190. Specially, in the line 187, "background-position: c... [09:54:20] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12149083 (10Pharos) >>! In T432935#12149056, @VIUK wrote: > Hi, Iā€™m interested in investigating this issue. > > Could you provide the URL of a Wikinewsie card where... [09:55:48] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12149084 (10dcaro) [09:57:16] (03open) 10dcaro: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 [09:59:39] (03update) 10dcaro: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 [09:59:43] (03update) 10dcaro: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 [10:01:08] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [10:02:52] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [10:07:03] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [10:07:26] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [10:09:36] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881#12149143 (10Csisc) Two choices. Either we set it on cdx-dialog in ArticleModal.vue or we create a JavaScript script and apply it as a function on the main page of the Project. [10:10:53] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [10:11:28] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29998195703 (https://github.com/cluebotng/component-configs/commits/e8762796572d00e69633a1b076413d5aa4ee3508) [10:11:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [10:12:40] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12149148 (10dcaro) [10:13:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12149149 (10dcaro) [10:18:41] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [10:24:37] 10Tool-wikinewsie: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12149203 (10Pharos) I think a starter regionalization could use these categories on English Wikipedia, then each continent can link to subpages for the countries as well: 2026 in Africa 2026 in Asia 2026... [10:26:46] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12149216 (10Pharos) [10:27:13] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12149218 (10Csisc) This thing can be a dropdown list on app-toolbar div. [10:28:49] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12149225 (10Csisc) This exists on App.vue. [10:44:41] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962#12149297 (10Csisc) This can be done using MediaWiki API: https://en.wikipedia.org/wiki/Special:ApiSandbox#action=compare&format=json&fromrev=1347027236&torev=1361473712&formatversion=2. [10:45:58] 10Tool-wikinewsie: Wikinewsie link to main updated section of Wikipedia article - https://phabricator.wikimedia.org/T431962#12149301 (10Csisc) There are two tags: for inserted parts and for deleted parts. The main point is to locate the first appearing ins or del tag and find the title directly comin... [10:55:11] (03update) 10raymond-ndibe: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 (owner: 10dcaro) [10:55:14] (03approved) 10raymond-ndibe: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 (owner: 10dcaro) [10:56:50] (03update) 10raymond-ndibe: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 (owner: 10dcaro) [10:56:53] (03approved) 10raymond-ndibe: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 (owner: 10dcaro) [10:57:27] (03update) 10raymond-ndibe: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 (owner: 10dcaro) [11:00:43] (03update) 10raymond-ndibe: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 (owner: 10dcaro) [11:02:21] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12149379 (10TheDJ) [11:03:23] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12149381 (10taavi) [11:03:24] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog šŸ“„): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12149383 (10taavi) [11:03:37] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12149384 (10taavi) [11:03:57] (03update) 10raymond-ndibe: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 (owner: 10dcaro) [11:03:59] (03approved) 10raymond-ndibe: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 (owner: 10dcaro) [11:04:18] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12149386 (10taavi) Removing subtasks not related to the coordination of the deprecation. Individual project upgrades are tracked in #cloud-vps-bullseye-deprecation. [11:05:20] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12149388 (10TheDJ) [11:06:07] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12149391 (10TheDJ) @saper has offered to help out with this [11:07:08] (03update) 10raymond-ndibe: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] (enable_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 (owner: 10dcaro) [11:07:11] (03approved) 10raymond-ndibe: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] (enable_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 (owner: 10dcaro) [11:13:15] (03update) 10raymond-ndibe: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 (owner: 10dcaro) [11:13:16] (03approved) 10raymond-ndibe: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 (owner: 10dcaro) [11:16:09] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12149470 (10taavi) As I understand it, Patch Demo is a tool for testing and demoing existing patches that have been developed within a developer's existing loc... [11:20:48] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-controller-2 [11:21:34] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-controller-2 [11:22:10] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-2 [11:22:56] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-2 [11:28:01] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/30003154188 (https://github.com/cluebotng/component-configs/commits/97520d86ff57924e8237a0446010f837e79b9062) [11:28:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [11:28:13] (03merge) 10dcaro: docs: add section with example env config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/330 [11:28:26] (03merge) 10dcaro: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 [11:28:29] (03update) 10dcaro: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 [11:28:42] (03merge) 10dcaro: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 [11:31:48] (03merge) 10dcaro: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 [11:32:41] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Live pageviews for Wikinewsie - https://phabricator.wikimedia.org/T432576#12149546 (10JAllemandou) We currently don't provide close-to-realtime pageviews, nor hourly pageviews in the pageview API. We however provide hourly pageviews per page every hour as dump fil... [11:33:47] (03open) 10dcaro: d/changelog: bump to 16.1.31 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/164 (https://phabricator.wikimedia.org/T432365 https://phabricator.wikimedia.org/T432592) [11:33:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.552-20260723112902-f1c6958f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1347 (https://phabricator.wikimedia.org/T432586) [11:33:50] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.552-20260723112902-f1c6958f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1347 (https://phabricator.wikimedia.org/T432586) [11:35:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:39:57] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [11:43:08] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [11:45:09] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [11:46:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:47:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:04] (03Abandoned) 10Nikerabbit: WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) (owner: 10Amire80) [12:00:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:02:57] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud [12:03:11] !log taavi@cloudcumin1001 metricsinfra END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud [12:04:05] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud [12:05:36] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud [12:05:58] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12149696 (10Xqt) [12:06:27] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055#12149699 (10Xqt) [12:08:41] FIRING: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:09:16] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-3 [12:09:36] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [12:09:53] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [12:10:03] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-3 [12:12:12] (03approved) 10dcaro: jobs-api: bump to 0.0.552-20260723112902-f1c6958f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1347 (https://phabricator.wikimedia.org/T432586) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:12:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [12:12:20] (03merge) 10dcaro: jobs-api: bump to 0.0.552-20260723112902-f1c6958f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1347 (https://phabricator.wikimedia.org/T432586) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:12:46] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] crashes if k8s object does not match expected format (un-guarded list access) - https://phabricator.wikimedia.org/T432586#12149711 (10dcaro) 05In progressā†’03Resolved Done :) [12:13:03] FIRING: [2x] TargetDown: Job main-nginx-https is unreachable in project project-proxy instance proxy-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:18:03] RESOLVED: [3x] TargetDown: Job main-nginx-https is unreachable in project project-proxy instance proxy-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:18:20] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12149722 (10dcaro) [12:20:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [12:22:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [12:26:41] (03open) 10dcaro: dotfiles: remove old dotfiles [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/331 [12:26:48] (03update) 10dcaro: dotfiles: remove old dotfiles [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/331 [12:28:49] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie cards sometimes have top cut off on mobile - https://phabricator.wikimedia.org/T432935#12149746 (10VIUK) I can now see the issue on the Wikinewsie card itself: the top of the card appears clipped on mobile. This looks like a mobile layout / safe-area is... [12:30:35] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [12:37:38] (03open) 10countcount: Repair missing mirrored block expiry values [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/60 [12:38:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:40:24] 10Tool-lexeme-forms, 06translatewiki.net: l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12149780 (10LucasWerkmeister) I think there might still be a problem ā€“ the `twn` branch was just updated (Thu Jul 23 14:30:58 2026 +020... [12:46:31] 10Tool-lexeme-forms, 06translatewiki.net: l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12149792 (10Nikerabbit) Yes I was just looking at the same thing: > Unable to create a pull request for wikidata-lexeme-forms: cURL err... [12:50:15] 10Tool-lexeme-forms, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12149796 (10Nikerabbit) p:05Triageā†’03High One idea is to use the read-repo... [12:53:30] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969 (10dcaro) 03NEW [12:58:03] 10Tool-lexeme-forms, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12149821 (10LucasWerkmeister) [12:58:26] 10Tool-lexeme-forms, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12149827 (10LucasWerkmeister) [12:59:20] (03approved) 10dcaro: d/changelog: bump to 16.1.31 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/164 (https://phabricator.wikimedia.org/T432365 https://phabricator.wikimedia.org/T432592) [12:59:28] (03merge) 10dcaro: d/changelog: bump to 16.1.31 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/164 (https://phabricator.wikimedia.org/T432365 https://phabricator.wikimedia.org/T432592) [13:03:11] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api,jobs-cli] `toolforge jobs load` uses PATCH for one-off jobs too - https://phabricator.wikimedia.org/T432592#12149850 (10dcaro) 05In progressā†’03Resolved Done! [13:11:12] (03CR) 10Majavah: [C:03+2] labsauth: Write SUL account details to LDAP on registration [labs/striker] - 10https://gerrit.wikimedia.org/r/1076815 (https://phabricator.wikimedia.org/T148048) (owner: 10Majavah) [13:13:51] (03Merged) 10jenkins-bot: labsauth: Write SUL account details to LDAP on registration [labs/striker] - 10https://gerrit.wikimedia.org/r/1076815 (https://phabricator.wikimedia.org/T148048) (owner: 10Majavah) [13:28:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:29:13] (03PS1) 10Majavah: profile: Reject PEM format SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/1314827 [13:32:59] 10Tool-wikinewsie: Portal:Current events mode for Wikinewsie - https://phabricator.wikimedia.org/T432971 (10Pharos) 03NEW [13:33:43] 10Tool-wikinewsie: Portal:Current events mode for Wikinewsie - https://phabricator.wikimedia.org/T432971#12149980 (10Pharos) [13:34:29] 10Tool-wikinewsie: Portal:Current events mode for Wikinewsie - https://phabricator.wikimedia.org/T432971#12149984 (10Pharos) [13:41:56] 10PAWS, 06tools-platform-team: PAWS giving 504 Gateway Time-out - https://phabricator.wikimedia.org/T432972 (10TBurmeister) 03NEW [13:42:35] 10PAWS, 06tools-platform-team: Provide a better index/site for browsing PAWS notebooks - https://phabricator.wikimedia.org/T268494#12150018 (10TBurmeister) [13:42:51] 10PAWS, 06tools-platform-team, 07Documentation: Provide a better index/site for browsing PAWS notebooks - https://phabricator.wikimedia.org/T268494#12150019 (10TBurmeister) [13:43:02] 10PAWS, 06tools-platform-team: Make forking notebooks in PAWS easier - https://phabricator.wikimedia.org/T264603#12150021 (10TBurmeister) [13:44:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate [13:44:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate [13:47:16] (03CR) 10Majavah: [C:03+2] profile: Reject PEM format SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/1314827 (owner: 10Majavah) [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:48:45] (03Merged) 10jenkins-bot: profile: Reject PEM format SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/1314827 (owner: 10Majavah) [14:07:47] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12150101 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7f1c1ea0-... [14:10:20] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12150115 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e840132a-... [14:25:35] 10Tool-wikinewsie: Portal:Current events mode for Wikinewsie - https://phabricator.wikimedia.org/T432971#12150173 (10Pharos) [14:35:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:36:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:36:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance paws-puppetserver-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:36:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:37:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance gitlab-runners-puppetserver-01 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:38:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:40:03] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [14:40:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:41:03] FIRING: [2x] PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:41:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:42:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance gitlab-runners-puppetserver-01 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:45:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:46:03] RESOLVED: [2x] PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:48:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:48:46] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:50:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:51:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance paws-puppetserver-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:53:46] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:57:44] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12150331 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [14:58:46] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:59:42] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12150338 (10dcaro) [15:03:46] RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:05:46] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:07:33] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [15:10:46] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:12:15] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12150409 (10fnegri) Went down then up again :/ {F95419195} [15:15:46] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:20:46] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:25:46] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:29:14] FIRING: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [15:30:46] RESOLVED: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:32:27] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12150496 (10bd808) >>! In T432617#12149470, @taavi wrote: > As I understand it, Patch Demo is a tool for testing and demoing existing patches that have been de... [15:32:46] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:34:14] RESOLVED: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [15:36:31] RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:37:46] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:41:31] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:42:46] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:43:44] FIRING: [9x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [15:46:31] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:47:46] RESOLVED: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:48:33] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:48:44] RESOLVED: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [15:51:31] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:51:44] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12150627 (10bd808) >>! In T432617#12148699, @jnuche wrote: > Pinging @Andrew since the Catalyst team originally discussed this quota increase with him There i... [15:53:33] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:55:50] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: Create the backend for leximap - https://phabricator.wikimedia.org/T432248#12150651 (10Kengkong1) 05Openā†’03Resolved [15:56:31] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:57:18] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12150657 (10Andrew) I've drained this again. dc-ops folks, this server is back in your court. It should still be under warranty. [15:58:20] 10VPS-project-Codesearch: searching for nothing should display an error to the user - https://phabricator.wikimedia.org/T432987 (10Novem_Linguae) 03NEW [15:58:33] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:59:44] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:01:31] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:03:33] RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:04:16] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:04:44] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:05:39] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12150723 (10taavi) >>! In T432617#12150495, @bd808 wrote: > I was a bit confused when @CCiufo-WMF, Levi, and Alexandros started talking about cloud dev environ... [16:06:31] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:09:16] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:11:31] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:14:16] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:16:31] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:18:38] 10PAWS, 06tools-platform-team: PAWS giving 504 Gateway Time-out - https://phabricator.wikimedia.org/T432972#12150771 (10fnegri) I can reproduce 504s on public-paws URLs like: * https://public-paws.wmcloud.org/ * https://public-paws.wmcloud.org/10005589/ [16:19:16] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:21:31] RESOLVED: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:22:36] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:24:16] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:27:36] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:29:16] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:32:36] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:34:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:34:16] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:37:36] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:39:14] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:39:16] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:42:36] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:44:16] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:44:43] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12150857 (10CCiufo-WMF) >>! In T432617#12150495, @bd808 wrote: > I think it is fair to have questions. I was a bit confused when @CCiufo-WMF, Levi, and Alexand... [16:47:36] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:49:14] FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:49:16] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:50:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram [16:50:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram [16:52:36] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:54:14] RESOLVED: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:54:16] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:56:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:57:36] RESOLVED: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:00:33] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:02:36] FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:05:33] RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:08:42] 10Toolforge, 06tools-infrastructure-team: [haproxy,infra] Move haproxy logs to their own volume/partition - https://phabricator.wikimedia.org/T432994 (10dcaro) 03NEW [17:14:03] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [17:14:31] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [17:15:39] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [17:15:39] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [17:16:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [17:16:44] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [17:26:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [17:33:02] 10PAWS, 06tools-platform-team: PAWS giving 504 Gateway Time-out - https://phabricator.wikimedia.org/T432972#12151016 (10fnegri) 05Openā†’03Resolved a:03dcaro @dcaro restarted the `nbserve` and `renderer` pods and things seem to be working again. [17:38:54] 10Toolforge, 06tools-infrastructure-team: [haproxy,infra] Move haproxy logs to their own volume/partition - https://phabricator.wikimedia.org/T432994#12151023 (10dcaro) We could also try to undersample the logs (see https://www.haproxy.com/blog/haproxy-log-sampling), though that might require tweaking toolview... [17:41:54] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12151035 (10fnegri) Merged the bugfix, before resolving this task I will do a final test rebooting all clouddbs tomorrow. [19:59:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12151429 (10Aklapper) [20:42:08] 10Tool-eventfeedback: Complete user documention - https://phabricator.wikimedia.org/T433015 (10Strainu) 03NEW [20:46:37] 10Tool-eventfeedback: Customizeable feedback - https://phabricator.wikimedia.org/T433016 (10Strainu) 03NEW [23:52:14] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12152055 (10bd808) >>! In T432426#12143509, @Andrew wrote: > This may be recurring on logging-logstash-04. Taavi (and now I) suspect that the issue is with systemd-networ... [23:53:35] 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12152060 (10ZhaoFJx) Would it be possible to use translatewiki.net? Reference: [[ https://translatewiki.net/wiki/Translating:New_project | Translating:New project ]]