[00:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:31:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:43:09] (03open) 10ibrahemqasim: Refactor job execution behind backend interface [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/38 [02:16:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:48:16] (03update) 10ibrahemqasim: Refactor job execution behind backend interface [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/38 [02:49:20] (03update) 10ibrahemqasim: Refactor job execution behind backend interface [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/38 [02:50:02] (03merge) 10ibrahemqasim: Refactor job execution behind backend interface [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/38 [02:57:17] (03update) 10ibrahemqasim: Refactor job execution behind backend interface [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/38 [03:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:10:19] (03open) 10ibrahemqasim: M [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/39 [03:16:04] (03update) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [03:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:31:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:15:42] (03update) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [05:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:38:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:01:27] 10Cloud-VPS, 06tools-infrastructure-team: puppet failing on metricsinfra-prometheus-2.metricsinfra due to expired crl - https://phabricator.wikimedia.org/T429298#12553621 (10taavi) 05Open→03Resolved a:03taavi [08:20:16] (03open) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:20:26] (03update) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:20:33] (03update) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:20:41] (03update) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:20:41] (03approved) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:27:24] (03update) 10essa237: Add Simulated Login for Offline or Local Development [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/2 (https://phabricator.wikimedia.org/T421158) (owner: 10janeeva1) [08:27:24] (03close) 10essa237: Add Simulated Login for Offline or Local Development [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/2 (https://phabricator.wikimedia.org/T421158) (owner: 10janeeva1) [08:27:38] (03update) 10essa237: Update Readme for with proper infor [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/1 (https://phabricator.wikimedia.org/T421577) (owner: 10collins) [08:27:39] (03close) 10essa237: Update Readme for with proper infor [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/1 (https://phabricator.wikimedia.org/T421577) (owner: 10collins) [08:28:19] (03merge) 10essa237: Add a merge request template and multi-contributor guidance [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/18 (https://phabricator.wikimedia.org/T444734) [08:31:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:32:06] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 26d 2h 38m 7s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [09:32:50] (03open) 10jinoytommanjaly: fix: upload original full-resolution files to Commons (#1) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/1 [09:37:06] RESOLVED: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1001.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 26d 2h 38m 7s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [09:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:14:00] 06Toolforge-standards-committee: Usurp h2bot tool account - https://phabricator.wikimedia.org/T442880#12554020 (10MBH) 05Stalled→03Open @taavi @JJMC89 After 15 days, no answer from tool owner on https://ru.wikipedia.org/wiki/User_talk:Haffman , https://wikitech.wikimedia.org/wiki/User_talk:Haffman and via em... [10:15:07] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554029 (10Danya) p:05Low→03Medium [10:15:08] (03open) 10fnegri: Remove outdated names and links [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/15 [10:17:32] 10Tool-wiki-gender-stats: Use local storage instead of cookies for FilterWidget - https://phabricator.wikimedia.org/T432533#12554044 (10Danya) [10:17:33] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12554045 (10Danya) [10:18:46] 10Tool-wiki-gender-stats: Make chart labels filtering more intuitive - https://phabricator.wikimedia.org/T434318#12554054 (10Danya) [10:18:47] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12554055 (10Danya) [10:19:44] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12554071 (10Danya) [10:19:45] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554070 (10Danya) [10:20:14] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554072 (10Danya) p:05Medium→03High [10:20:43] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554075 (10Danya) [10:31:04] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554283 (10Danya) [10:33:34] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12554303 (10Danya) [10:35:30] 10Tool-wiki-gender-stats: Add button to download charts as PNGs - https://phabricator.wikimedia.org/T432545#12554313 (10Danya) [10:35:36] 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12554314 (10Danya) [10:36:05] 10Tool-wiki-gender-stats: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314#12554315 (10Danya) [10:43:27] 10Cloud-VPS, 06tools-infrastructure-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [ceph] export number of bad sectors per-disk - https://phabricator.wikimedia.org/T348716#12554342 (10fnegri) [10:43:33] 10Toolforge, 06tools-platform-team, 03WMDE-TechWish-Kanban: Unable to request Toolforge membership - https://phabricator.wikimedia.org/T444782 (10MareikeHeuerWMDE) 03NEW [10:43:46] 10Cloud-VPS, 06tools-infrastructure-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [ceph] Investigate if there's a way to degrade instead of failing when jumbo frames are being dropped in the network - https://phabricator.wikimedia.org/T329778#12554354 (10fnegri) [10:44:02] 10Cloud-VPS, 06tools-infrastructure-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Unplanned: [ceph] Enable disk failure prediciton - https://phabricator.wikimedia.org/T349694#12554355 (10fnegri) [10:47:39] 10Toolforge, 06tools-platform-team: [sample-complex-app-backend] failing to deploy since last builder update - https://phabricator.wikimedia.org/T444783 (10DamianZaremba) 03NEW [10:47:41] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12554368 (10Aklapper) [10:48:10] 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-Projects: [trove] Database quota values are not updating correctly - https://phabricator.wikimedia.org/T373348#12554382 (10fnegri) [10:50:46] 10Data-Services, 06tools-platform-team, 06Data-Engineering, 06Data-Engineering-Icebox, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#12554390 (10fnegri) [10:51:02] 10Cloud-VPS, 06tools-infrastructure-team: Migrate cloudceph servers to nftables - https://phabricator.wikimedia.org/T361913#12554392 (10fnegri) [10:51:11] 10Toolforge, 06tools-platform-team: [sample apps] update components config to expected standard - https://phabricator.wikimedia.org/T444784 (10DamianZaremba) 03NEW [10:51:50] 10Cloud-VPS, 06tools-infrastructure-team: Add some monitoring/non-paging alerts to codfw1dev - https://phabricator.wikimedia.org/T344440#12554407 (10fnegri) [10:52:21] 10Cloud-VPS, 06tools-infrastructure-team: ceph.operationalization: Refactor the current puppet code to allow per-cluster configurations (as opposed to per-DC as it does currently) - https://phabricator.wikimedia.org/T281250#12554418 (10fnegri) [10:52:30] 10Cloud-VPS, 06tools-infrastructure-team: ceph.automation: Add non-host bound icinga downtimes to the upgrade scripts - https://phabricator.wikimedia.org/T281336#12554430 (10fnegri) [10:52:51] 10Cloud-VPS, 06tools-infrastructure-team: We almost never actually free up space from deleted VMs - https://phabricator.wikimedia.org/T289623#12554432 (10fnegri) [10:53:49] 10Cloud-VPS, 06tools-infrastructure-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned: [nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes - https://phabricator.wikimedia.org/T356621#12554464 (10fnegri) [10:54:49] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP - https://phabricator.wikimedia.org/T379550#12554500 (10fnegri) [10:58:19] 10Toolforge, 06tools-platform-team: [components-api] add order to the components deployment - https://phabricator.wikimedia.org/T362075#12554581 (10fnegri) [10:58:23] 10Data-Services, 06tools-platform-team, 06Data-Persistence: [wikireplicas] Route alerts to WMCS team - https://phabricator.wikimedia.org/T381589#12554583 (10fnegri) [10:58:29] 10Striker, 06tools-platform-team: Rebuild Striker demo server - https://phabricator.wikimedia.org/T329687#12554582 (10fnegri) [10:58:33] 10Tool-cdnjs, 10Toolforge, 06tools-platform-team, 13Patch-For-Review, 07Privacy: tools-static.wmflabs.org/cdnjs may return redirects to speedcf.cloudflareaccess.com, violating user privacy - https://phabricator.wikimedia.org/T399483#12554580 (10fnegri) [10:58:51] 10Toolforge, 06tools-platform-team: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#12554590 (10fnegri) [10:58:55] 10Toolforge, 06tools-platform-team: Port fourohfour tool to new Gateway system - https://phabricator.wikimedia.org/T418479#12554589 (10fnegri) [10:58:59] 10Quarry, 06tools-platform-team: [quarry] ensure that the DB and the models are in sync - https://phabricator.wikimedia.org/T288523#12554593 (10fnegri) [10:59:03] 10Toolforge, 06tools-platform-team: Track global and per-tool concurrent requests and 503 rate limiting responses from Toolforge CDN edge - https://phabricator.wikimedia.org/T409794#12554592 (10fnegri) [10:59:07] 10Toolforge, 06tools-platform-team: Identify candidate tools to test migrating to the Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407502#12554595 (10fnegri) [10:59:11] 10Toolforge, 06tools-platform-team: Decouple Toolforge API gateway authentication from Kubernetes certificates - https://phabricator.wikimedia.org/T332478#12554594 (10fnegri) [10:59:15] 10Toolforge, 06tools-platform-team: What are the parts of toolforge? - https://phabricator.wikimedia.org/T407299#12554599 (10fnegri) [10:59:19] 10Toolforge, 06tools-platform-team: [infra,k8s] Move to kubernetes VAPs and drop kyverno - https://phabricator.wikimedia.org/T364293#12554601 (10fnegri) [10:59:23] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#12554600 (10fnegri) [10:59:27] 10Toolforge, 06tools-platform-team, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#12554602 (10fnegri) [10:59:38] 10Toolforge, 06tools-platform-team: [toolforge,storage,swift,s3] Object store? - https://phabricator.wikimedia.org/T225190#12554603 (10fnegri) [10:59:42] 10Striker, 06tools-platform-team: Make it possible to maintain Toolforge tools via an easy-to-use web interface instead of a command-line one - https://phabricator.wikimedia.org/T332480#12554604 (10fnegri) [10:59:46] 10Toolforge, 06tools-platform-team: [toolforge-cli-gen] review the https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli client as potential consolidation - https://phabricator.wikimedia.org/T398651#12554606 (10fnegri) [10:59:50] 10Striker, 06tools-platform-team, 07Documentation: Document how to run Striker migrations - https://phabricator.wikimedia.org/T398062#12554608 (10fnegri) [11:00:00] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Duplicate operationId `cancel_tool_deployment` in components API spec - https://phabricator.wikimedia.org/T399362#12554607 (10fnegri) [11:00:04] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#12554609 (10fnegri) [11:00:08] 10Toolforge, 06tools-platform-team, 10MediaWiki-Email, 10MediaWiki-extensions-LoginNotify, and 3 others: BotPasswords + Toolforge combination causing daily "login from new device" warning emails - https://phabricator.wikimedia.org/T182867#12554610 (10fnegri) [11:00:14] 10Toolforge, 06tools-platform-team: [harbor] some artifacts and projects seems to have gone missing - https://phabricator.wikimedia.org/T380833#12554615 (10fnegri) [11:00:18] 10Toolforge, 06tools-platform-team, 05Goal: [harbor] Create backups and/or replication - https://phabricator.wikimedia.org/T336668#12554613 (10fnegri) [11:00:22] 10Toolforge, 06tools-platform-team: Ensure all ToolsDB databases comply with current naming conventions - https://phabricator.wikimedia.org/T269609#12554614 (10fnegri) [11:00:26] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] Add support for replacing a running scheduled job when an overlapping schedule fires (`concurrencyPolicy: Replace`) - https://phabricator.wikimedia.org/T377781#12554616 (10fnegri) [11:00:34] 06tools-infrastructure-team, 06DC-Ops, 06SRE, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12554621 (10fnegri) [11:03:35] 10Cloud-VPS, 06tools-infrastructure-team: Restrict service VM images (e.g. 'ubuntu-22.04-v1.36.5-for-magnum') to their associated services - https://phabricator.wikimedia.org/T444789 (10Andrew) 03NEW [11:04:13] 10Cloud-VPS, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 10Wikidata, and 2 others: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs - https://phabricator.wikimedia.org/T281122#12554652 (10fnegri) [11:15:10] 10Cloud-VPS, 06tools-infrastructure-team: Use mariabackup instead of xtrabackup for galera backups? (Or possibly for all maria backups?) - https://phabricator.wikimedia.org/T284157#12554730 (10fnegri) [11:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:28:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:51:01] RESOLVED: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:07:12] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [12:08:26] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [12:22:05] (03open) 10essa237: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T444733) [12:22:53] (03approved) 10essa237: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T444733) [12:23:08] (03update) 10essa237: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T444733) [12:30:19] (03update) 10essa237: Draft: T444733: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [13:34:06] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:39:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:44:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:45:37] (03update) 10jinoytommanjaly: fix: upload original full-resolution files to Commons (#1) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/1 [13:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:48:07] (03open) 10jinoytommanjaly: fix: upload original full-resolution files to Commons (#1) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/2 [13:48:41] (03update) 10jinoytommanjaly: fix: upload original full-resolution files to Commons (#1) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/1 [13:49:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:49:21] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:50:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:51:37] (03update) 10jinoytommanjaly: feat: title numbering, 1000-file bulk upload, license list and new upload form (#2) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/2 [13:51:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:54:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:59:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:04:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:05:53] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:07:54] (03update) 10jinoytommanjaly: feat: title numbering, 1000-file bulk upload, license list and new upload form (#2) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/2 [14:15:42] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.100 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/70 [14:15:45] (03open) 10renovatebot: Update cimg/rust Docker tag to v1.100 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/70 [14:19:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:22:24] (03update) 10jinoytommanjaly: feat: title numbering, 1000-file bulk upload, license list and new upload form (#2) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/2 [14:24:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:24:21] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:29:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:34:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:39:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:44:06] FIRING: [8x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:49:06] RESOLVED: [7x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:06:53] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:08:03] 10Toolforge, 06tools-platform-team: document the deployment process of embargoed patches on toolforge - https://phabricator.wikimedia.org/T444635#12555665 (10Raymond_Ndibe) First draft here https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Deploying_embargoed_security_fixes I believe I already teste... [15:28:02] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12555787 (10Doc_James) Amazing Amdrel, thank you for all your work on this. So the next step is updating the gadget layout. @1Veertje can help with this? [15:35:53] (03open) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [15:53:06] (03approved) 10raymond-ndibe: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:09:30] (03open) 10raymond-ndibe: build_test: add/improve test for previously untested but important behaviours [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/173 (https://phabricator.wikimedia.org/T444818) [16:16:20] (03update) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [16:26:51] (03update) 10tlepage: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:31:23] (03open) 10raymond-ndibe: build: get all TaskRuns in one call when listing builds [repos/cloud/toolforge/builds-api] (test_improvements) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T442194) [16:31:34] (03update) 10raymond-ndibe: build: get all TaskRuns in one call when listing builds [repos/cloud/toolforge/builds-api] (test_improvements) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T442194) [16:37:39] (03update) 10raymond-ndibe: build: get all TaskRuns in one call when listing builds [repos/cloud/toolforge/builds-api] (test_improvements) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T442194) [16:45:10] (03open) 10tlepage: Draft: alerts-api: Move prometheusUrls to values top level [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1523 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [16:50:14] (03update) 10fnegri: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:54:29] (03update) 10tlepage: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [17:01:50] (03update) 10tlepage: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [17:13:48] (03update) 10jinoytommanjaly: feat: title numbering, 1000-file bulk upload, license list and new upload form (#2) [toolforge-repos/g2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/g2commons/-/merge_requests/2 [17:21:21] (03update) 10fnegri: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:21:58] (03update) 10fnegri: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:22:10] (03update) 10fnegri: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:22:15] (03approved) 10fnegri: chart: move prometheusUrls to the top level of the values [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [17:29:48] (03CR) 10Lokal Profil: [C:03+2] tests: Mock Intuition rather than hand-writing a stub [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362224 (owner: 10Lokal Profil) [17:32:40] (03CR) 10CI reject: [V:04-1] tests: Mock Intuition rather than hand-writing a stub [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362224 (owner: 10Lokal Profil) [17:37:28] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] (dummy-api-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [18:15:37] (03update) 10renovatebot: Update dependency jsdom to v31 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/68 [18:29:58] (03CR) 10Jforrester: [C:03+1] "I've just fixed the CI PHP 8.5 images, so you can re-try this." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362224 (owner: 10Lokal Profil) [18:35:25] (03update) 10essa237: Draft: T444733: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [18:40:54] 10Wikibugs: wikibugs phab feed is failing - https://phabricator.wikimedia.org/T444793#12556507 (10bd808) This `Cursor "${poll_last_seen_chrono_key}" does not identify a valid object` thing has obviously been seen before as there is a fixture in the tests/test_phorge_polling.py that returns that error for the `te... [18:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:54:04] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12556517 (10DB111) I don't think the right to fork policy does apply here ("...after the... [19:10:10] (03update) 10essa237: Draft: T444733: Print the build job's memory limits in CI [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [19:33:56] (03update) 10essa237: Draft: T444733: Give the CI build a larger Node heap [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [19:35:26] (03update) 10essa237: Draft: T444733: Give the CI build a larger Node heap [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [19:37:45] (03update) 10essa237: T444733: Give the CI build a larger Node heap [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [19:39:18] (03merge) 10essa237: T444733: Give the CI build a larger Node heap [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/19 (https://phabricator.wikimedia.org/T444733) [19:42:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:43:31] 06Toolforge-standards-committee: Adoption request for h2bot - https://phabricator.wikimedia.org/T442880#12556599 (10JJMC89) p:05Triage→03Medium [19:44:04] 06Toolforge-standards-committee: Adoption request for h2bot - https://phabricator.wikimedia.org/T442880#12556605 (10JJMC89) [19:45:00] 06Toolforge-standards-committee: Adoption request for h2bot - https://phabricator.wikimedia.org/T442880#12556609 (10JJMC89) The tool is eligible for adoption, but not usurpation. [19:46:43] 06Toolforge-standards-committee: Adoption request for h2bot - https://phabricator.wikimedia.org/T442880#12556617 (10MBH) Yes, I mean adoption. [20:00:12] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Test Platform: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808#12556647 (10pwangai) a:05Peter→03pwangai [20:05:40] 10Tool-delintbot: Add ability to make only a certain number of edits - https://phabricator.wikimedia.org/T444833 (10Redmin) 03NEW [20:06:01] 10Tool-delintbot: Add ability to make only a certain number of edits - https://phabricator.wikimedia.org/T444833#12556673 (10Redmin) p:05Triage→03Medium [20:30:48] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12556734 (10valerio.bozzolan) My friend, sorry if I joke as friends together smiling: "t... [20:35:21] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12556747 (10valerio.bozzolan) [21:37:41] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12556914 (10bd808) >>! In T440145#12556517, @DB111 wrote: > I don't think the right to f... [21:38:27] 10Cloud-VPS, 06tools-infrastructure-team: cloudvirt hosts APT misconfiguration - https://phabricator.wikimedia.org/T444840 (10Volans) 03NEW [21:55:06] FIRING: [2x] HostBGPDown: BGP session for cloudlb2002-dev (172.20.5.3) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [22:00:06] RESOLVED: [4x] HostBGPDown: BGP session for cloudlb2002-dev (172.20.5.3) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [22:06:17] FIRING: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:10:21] FIRING: [8x] HostBGPDown: BGP session for cloudlb1002 (172.20.2.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [22:11:17] RESOLVED: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:11:36] RESOLVED: [8x] HostBGPDown: BGP session for cloudlb1002 (172.20.2.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [22:13:59] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12557444 (10bd808) >>! In T440145#12556517, @DB111 wrote: > It's rather demotivating to... [22:26:13] 06Toolforge-standards-committee, 10Tools, 03Wikimedia-Hackathon-2026 (Wikimedia Hackathon 2026 - Satellite Event Messina): Right to fork request: https://wikimap.toolforge.org/ - https://phabricator.wikimedia.org/T440145#12557470 (10bd808) >>! In T440145#12556517, @DB111 wrote: > I tried to repair a quasi-ab... [22:27:13] (03open) 10bd808: phorge: Discard chrono key when mentioned in Conduit error message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/79 (https://phabricator.wikimedia.org/T444793) [22:45:45] (03approved) 10bd808: Use SVG logo [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/20 (owner: 10taavi) [22:47:52] (03update) 10bd808: Use SVG logo [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/20 (https://phabricator.wikimedia.org/T129433) (owner: 10taavi) [22:48:20] RESOLVED: [4x] HostBGPDown: BGP session for cloudservices1005 (172.20.2.4) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [22:50:20] (03merge) 10bd808: phorge: Discard chrono key when mentioned in Conduit error message [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/79 (https://phabricator.wikimedia.org/T444793) [22:50:26] (03merge) 10taavi: Use SVG logo [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/20 (https://phabricator.wikimedia.org/T129433) [22:53:17] RESOLVED: [4x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:54:41] 10Tool-refill: Detect and reconcile duplicates - https://phabricator.wikimedia.org/T352397#12557573 (10ppelberg) [22:56:35] 10Tool-refill: Detect and reconcile duplicates - https://phabricator.wikimedia.org/T352397#12557683 (10ppelberg) [22:57:35] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#12557686 (10bd808) test [23:02:47] 10Wikibugs, 13Patch-For-Review: wikibugs phab feed is failing - https://phabricator.wikimedia.org/T444793#12557690 (10bd808) 05Open→03Resolved I still don't know what made Phabricator mad about the "7707614293742491866" and "7707679685703180148" chronologicalKey values, but I think I have added reasona... [23:03:41] 10Wikibugs, 13Patch-For-Review: wikibugs phab feed is failing with 'Cursor "$NUMBER" does not identify a valid object' Conduit error - https://phabricator.wikimedia.org/T444793#12557705 (10bd808) [23:30:57] 10Horizon, 06tools-infrastructure-team: "Sign Out" button on horizon.wikimedia.org doesn't log you out - https://phabricator.wikimedia.org/T443120#12557786 (10sbassett) [23:55:50] (03open) 10bd808: web: return 200 status for /api/config/refresh [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/80 (https://phabricator.wikimedia.org/T425229)