[00:29:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:43:56] (03Abandoned) 10Jsn.sherman: ProjectRepository: fail over to Sitematrix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [01:09:29] (03CR) 10Jsn.sherman: "@andre.goujot@netc.fr & @musikanimal@gmail.com:" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309732 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [01:32:10] (03CR) 10Jsn.sherman: "Thanks for pointing this out to me! I can see where you're going with this and it makes sense. conf for db/meta discovery, isWMF guard cla" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [01:59:16] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:07:42] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:24:56] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:25:14] 10PAWS, 06tools-platform-team: [feature] Cannot use PAWS since yesterday - https://phabricator.wikimedia.org/T431914 (10tototo_jpn) 03NEW [03:19:06] 10PAWS, 06tools-platform-team: Create a proper way to ban a user from PAWS - https://phabricator.wikimedia.org/T431915 (10Bugreporter) 03NEW [03:32:20] (03CR) 10MusikAnimal: [C:03+2] "Thank you!!! Some of this is already hotfixed in prod and I think that's why we've seen a lot less session DB errors." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [03:33:01] (03Merged) 10jenkins-bot: RateLimitSubscriber: cut anon session db traffic [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [03:36:13] (03CR) 10MusikAnimal: [C:03+2] "> messages were served from the library and the individual tool" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [03:39:06] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: User servers failing to spawn (timeout 30s) across multiple accounts - https://phabricator.wikimedia.org/T410837#12111488 (10Pppery) 05Resolved→03Declined [03:39:36] 10PAWS, 06tools-platform-team: Spawn failed: server did not respond in 30 seconds - https://phabricator.wikimedia.org/T398087#12111490 (10Pppery) 05Open→03Invalid [04:29:10] (03open) 10legoktm: Updates (July 2026) [toolforge-repos/masto-collab] - 10https://gitlab.wikimedia.org/toolforge-repos/masto-collab/-/merge_requests/4 [04:31:23] (03open) 10legoktm: Updates (July 2026) [toolforge-repos/newusers] - 10https://gitlab.wikimedia.org/toolforge-repos/newusers/-/merge_requests/5 [04:31:50] (03close) 10legoktm: Updates (July 2026) [toolforge-repos/newusers] - 10https://gitlab.wikimedia.org/toolforge-repos/newusers/-/merge_requests/5 [04:32:58] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12111498 (10ZhaoFJx) 05In progress→03Resolved [04:47:27] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12111499 (10ZhaoFJx) >>! In T431877#12110665, @Pharos wrote: > Probably not the same issue, but zh and ko both return no results, and jp returns "Query failed". For zh and ko, if we unsele... [04:48:00] (03merge) 10legoktm: Updates (July 2026) [toolforge-repos/masto-collab] - 10https://gitlab.wikimedia.org/toolforge-repos/masto-collab/-/merge_requests/4 [05:59:16] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:07:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:25:11] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:35:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:40:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:19:56] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:19:56] RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:47:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:52:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:08:19] 10Tool-sentinel: Allowlist wiki_url when creating bookmarks - https://phabricator.wikimedia.org/T431886#12111595 (10Arian_Ar) 05In progress→03Resolved [09:08:49] 10Tool-sentinel: Allowlist wiki_url on GET /api/feed/diff - https://phabricator.wikimedia.org/T431887#12111596 (10Arian_Ar) 05Open→03Resolved [09:09:03] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [09:09:36] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:14:03] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [09:14:36] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:59:01] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:59:30] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:07:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:09:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:16:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [10:16:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [10:45:46] (03open) 10arian-ar: Bypass MediaWiki Redis cache in the feed poller (T431888) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/11 (https://phabricator.wikimedia.org/T431888) [10:46:24] (03merge) 10arian-ar: Bypass MediaWiki Redis cache in the feed poller (T431888) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/11 (https://phabricator.wikimedia.org/T431888) [10:47:40] 10Tool-sentinel, 13Patch-For-Review: Feed poller must not use long-lived MediaWiki Redis cache - https://phabricator.wikimedia.org/T431888#12111654 (10Arian_Ar) 05Open→03In progress [11:03:27] 10Tool-wiki-gender-stats, 13Patch-For-Review: About page - https://phabricator.wikimedia.org/T425022#12111663 (10Danya) a:05Wyslijp16→03Danya [11:11:00] (03close) 10danyya: Draft: Resolve T425022 "About page [toolforge-repos/wiki-gender-stats] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/46 (owner: 10wyslijp16-toolforge) [11:12:43] (03open) 10danyya: Resolve T425022 "About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/69 [11:14:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [11:15:31] (03update) 10danyya: Resolve T425022 "About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/69 [12:27:07] (03CR) 10Jsn.sherman: ProjectRepository: fail over to Sitematrix (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [12:58:55] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12111715 (10dcaro) @Andrew the service to delete the files was failing due to permissions, just removed them manually: ` root@paws-nfs-2:~# systemctl status daily_file_cleanup.serv... [13:01:15] 10PAWS, 06tools-platform-team: [feature] Cannot use PAWS since yesterday - https://phabricator.wikimedia.org/T431914#12111716 (10dcaro) 05Open→03Resolved a:03dcaro Hi @tototo_jpn we ran out of space on the servers and that was stopping your notebooks from starting, should be solved now, feel free to... [13:09:17] (03CR) 10Jsn.sherman: "This approach makes a lot more sense than I29285b64b71ec75cd6e7986983f3b8c14ba9139c, and I think slots in perfectly in its place. One thin" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [13:12:57] (03open) 10countcount: Return the current live holder of a queried name from /nameusage [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/43 [13:13:52] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12111731 (10dcaro) Found also errors on previous run of the unit that move the big files around (running not to try to clean a bit more space): ` root@paws-nfs-2:~# journalctl -u s... [13:15:28] (03merge) 10countcount: Return the current live holder of a queried name from /nameusage [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/43 [13:20:46] (03PS11) 10Alien4444: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) [13:21:00] (03CR) 10Alien4444: Readd third-party suppport (033 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [13:31:47] 10PAWS, 06tools-platform-team: Create a proper way to ban a user from PAWS - https://phabricator.wikimedia.org/T431915#12111754 (10dcaro) Agree. As an fyi. there was a peak of misuse and we agreed (in the team meeting) to do the quick block now and try to be responsive for the tasks that might arise until we c... [13:33:23] 10PAWS, 06tools-platform-team: Create a proper way to ban a user from PAWS - https://phabricator.wikimedia.org/T431915#12111755 (10dcaro) >>! In T431915#12111754, @dcaro wrote: > Agree. As an fyi. there was a peak of misuse and we agreed (in the team meeting) to do the quick block now and try to be responsive... [13:34:12] (03CR) 10Alien4444: "I've cleaned up that twig template, but sure, feel free to rebase etc." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [13:48:44] (03open) 10countcount: Return localrenames by default from /info [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/44 [13:49:15] (03update) 10danyya: Resolve T425022 "About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/69 [13:51:25] (03merge) 10countcount: Return localrenames by default from /info [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/44 [13:52:19] (03update) 10danyya: Resolve T425022 "About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/69 [13:53:40] (03merge) 10danyya: Resolve T425022 "About page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/69 [13:55:07] 10Tool-wiki-gender-stats, 13Patch-For-Review: About page - https://phabricator.wikimedia.org/T425022#12111758 (10Danya) 05In progress→03Resolved [14:00:08] (03update) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 (owner: 10renovatebot) [14:03:24] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 (owner: 10renovatebot) [14:07:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:11:37] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 98.36%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:38:33] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:28:53] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12111809 (10Habst) @ZhaoFJx I added a Wikidata-based approach for Category:2026, thanks! [15:59:01] (03PS3) 10Alien4444: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) [15:59:45] (03CR) 10Alien4444: "I've switched it to an option. Re" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [16:20:23] 10Tool-wikinewsie: Semi-protection for images on Wikinewsie - https://phabricator.wikimedia.org/T431878#12111841 (10Habst) Done in https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/commit/d8555aec25427cd9cb30ba026d29fbe419a5f082; users are required to be auto-confirmed on either enwiki or whatever target... [16:31:46] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:32:14] FIRING: [10x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:35:49] (03open) 10agamyasamuel: chore: initialize uv project with pyproject.toml and lockfile [toolforge-repos/indicwiki-transliterate-api] - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/1 [16:36:46] FIRING: [14x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:37:14] FIRING: [22x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:37:32] FIRING: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:38:17] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927 (10Danya) 03NEW [16:38:58] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester. - https://phabricator.wikimedia.org/T431928 (10Danya) 03NEW [16:39:28] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927#12111884 (10Danya) a:03Danya [16:39:44] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester. - https://phabricator.wikimedia.org/T431928#12111888 (10Danya) [16:39:58] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927#12111890 (10Danya) p:05Triage→03High [16:40:00] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester. - https://phabricator.wikimedia.org/T431928#12111892 (10Danya) p:05Triage→03High [16:40:08] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester. - https://phabricator.wikimedia.org/T431928#12111904 (10Danya) 05Open→03In progress [16:40:13] 10Tool-wiki-gender-stats: Scheduler does not take full snapshots - https://phabricator.wikimedia.org/T423025#12111907 (10Danya) 05In progress→03Resolved [16:40:37] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester. - https://phabricator.wikimedia.org/T431928#12111908 (10Danya) [16:40:38] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12111909 (10Danya) [16:41:46] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:42:14] RESOLVED: [22x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:46:46] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:49:14] FIRING: ToolforgeKubernetesHAproxyUnknown: Toolforge HAProxy has unknown state. HAproxy might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyUnknown [16:51:46] FIRING: [15x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:52:32] RESOLVED: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:53:09] 10tool-wscontest, 06Wiki-Mentor-Africa, 07good first task, 13Patch-For-Review: WSContest Tool admin cannot edit contest page if their name is multi-word separated by underscore (they can edit if it's a space, instead) - https://phabricator.wikimedia.org/T336157#12111913 (10Ishta_P_Jain) [16:53:56] 10tool-wscontest, 06RoadToWiki, 06Wiki-Mentor-Africa, 07good first task, 13Patch-For-Review: WSContest Tool admin cannot edit contest page if their name is multi-word separated by underscore (they can edit if it's a space, instead) - https://phabricator.wikimedia.org/T336157#12111914 (10Ishta_P_Jain) [16:54:14] RESOLVED: ToolforgeKubernetesHAproxyUnknown: Toolforge HAProxy has unknown state. HAproxy might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyUnknown [16:55:44] FIRING: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:56:46] FIRING: [15x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:00:44] RESOLVED: [14x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:01:36] FIRING: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:01:46] FIRING: [17x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:02:14] FIRING: ToolforgeKubernetesHAproxyUnknown: Toolforge HAProxy has unknown state. HAproxy might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyUnknown [17:05:59] FIRING: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:06:36] RESOLVED: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:06:46] FIRING: [20x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:07:14] RESOLVED: ToolforgeKubernetesHAproxyUnknown: Toolforge HAProxy has unknown state. HAproxy might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyUnknown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyUnknown [17:07:44] RESOLVED: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:10:26] (03update) 10agamyasamuel: chore: initialize uv project with pyproject.toml and lockfile [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/1 [17:11:46] FIRING: [20x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:13:44] FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:16:46] FIRING: [19x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:17:01] FIRING: [19x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:18:44] RESOLVED: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:21:46] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:22:01] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:23:44] FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:25:59] RESOLVED: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:26:46] FIRING: [11x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:30:36] FIRING: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:31:46] FIRING: [13x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:32:44] FIRING: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:34:59] (03open) 10legoktm: Updates (July 2026) [toolforge-repos/slf-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/slf-bot/-/merge_requests/3 [17:35:28] (03open) 10legoktm: Updates (July 2026) [toolforge-repos/poty-stuff] - 10https://gitlab.wikimedia.org/toolforge-repos/poty-stuff/-/merge_requests/4 [17:35:36] RESOLVED: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:36:46] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:37:44] RESOLVED: [24x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:41:46] RESOLVED: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:45:37] (03merge) 10legoktm: Updates (July 2026) [toolforge-repos/parsoid-sandbox] - 10https://gitlab.wikimedia.org/toolforge-repos/parsoid-sandbox/-/merge_requests/4 [17:46:39] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb) [17:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [17:47:01] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:48:17] (03update) 10danyya: Draft: Resolve T431928 "Revert to python dumps ingester" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/70 [17:49:28] (03update) 10danyya: Draft: Resolve T431928 "Revert to python dumps ingester" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/70 [17:50:04] (03update) 10danyya: Draft: Resolve T431928 "Revert back to the Python-based dumps ingester" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/70 [17:50:25] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester - https://phabricator.wikimedia.org/T431928#12111937 (10Danya) [17:50:48] (03update) 10danyya: Resolve T431928 "Revert back to the Python-based dumps ingester" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/70 [17:50:54] (03merge) 10danyya: Resolve T431928 "Revert back to the Python-based dumps ingester" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/70 [17:51:35] 10Tool-wiki-gender-stats: Revert back to the Python-based dumps ingester - https://phabricator.wikimedia.org/T431928#12111940 (10Danya) 05In progress→03Resolved [17:51:37] 10Tool-wiki-gender-stats: Use Flask’s appcontext feature for database connections - https://phabricator.wikimedia.org/T428250#12111943 (10Danya) 05Open→03In progress [17:52:01] FIRING: [16x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:55:17] (03merge) 10legoktm: Updates (July 2026) [toolforge-repos/poty-stuff] - 10https://gitlab.wikimedia.org/toolforge-repos/poty-stuff/-/merge_requests/4 [17:57:01] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:59:14] FIRING: [10x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:02:01] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:02:03] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:04:14] RESOLVED: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:06:45] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977) [18:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [18:07:01] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:07:03] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:08:14] FIRING: [5x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:09:16] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977) [18:09:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [18:12:01] FIRING: [11x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:13:14] RESOLVED: [5x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:15:53] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 07Community-consensus-needed: Disable or restrict upload in Wikitech - https://phabricator.wikimedia.org/T431648#12111962 (10Bugreporter) > that are not welcome on Commons Note files are allowed on Commons as long as they are used in any Wikimedia project,... [18:16:26] 10Tool-wiki-gender-stats: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#12111968 (10Danya) [18:16:27] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12111969 (10Danya) [18:17:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:19:03] 10Tool-wiki-gender-stats: Button to download the whole database? - https://phabricator.wikimedia.org/T424282#12111973 (10Danya) 05Stalled→03Declined The database will be tens of gigabytes big, so this is not a viable idea. Cancelling this ticket. [18:20:12] 10Tool-wiki-gender-stats: Favicon - https://phabricator.wikimedia.org/T423004#12111978 (10Danya) 05Open→03Stalled Marking as stalled until we have an official icon. [18:21:07] 10Tool-wiki-gender-stats: Link to suggest a new feature - https://phabricator.wikimedia.org/T425484#12111980 (10Danya) [18:21:10] 10Tool-wiki-gender-stats: Link to suggest a new feature - https://phabricator.wikimedia.org/T425484#12111981 (10Danya) [18:32:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:38:14] FIRING: [5x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:42:01] FIRING: [9x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:43:14] RESOLVED: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:47:01] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:55:44] FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:57:01] FIRING: [9x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:57:03] FIRING: [10x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:00:44] RESOLVED: [13x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:02:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:04:47] 10Tool-wiki-gender-stats: Integrate May 2026 data from local copy - https://phabricator.wikimedia.org/T431933 (10Danya) 03NEW [19:04:53] 10Tool-wiki-gender-stats: Integrate May 2026 data from local copy - https://phabricator.wikimedia.org/T431933#12112037 (10Danya) p:05Triage→03Medium [19:05:07] 10Tool-wiki-gender-stats: Integrate May 2026 data from local copy - https://phabricator.wikimedia.org/T431933#12112039 (10Danya) p:05Medium→03High [19:06:35] 10Tool-wiki-gender-stats: Show current page name in tab title - https://phabricator.wikimedia.org/T431490#12112041 (10Danya) [19:12:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:15:03] (03update) 10danyya: Draft: Resolve T428250 "Use Flask’s appcontext feature for database connections" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/71 [19:17:01] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_api_svc_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:21:06] (03update) 10danyya: Resolve T428250 "Use Flask’s appcontext feature for database connections" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/71 [19:21:15] (03merge) 10danyya: Resolve T428250 "Use Flask’s appcontext feature for database connections" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/71 [19:21:55] 10Tool-wiki-gender-stats: Use Flask’s appcontext feature for database connections - https://phabricator.wikimedia.org/T428250#12112059 (10Danya) 05In progress→03Resolved [19:22:01] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_api_svc_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:22:50] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12112062 (10Danya) [19:22:51] 10Tool-wiki-gender-stats: Show current page name in tab title - https://phabricator.wikimedia.org/T431490#12112061 (10Danya) [19:22:52] 10Tool-wiki-gender-stats: Update Codex icons to latest version - https://phabricator.wikimedia.org/T429708#12112063 (10Danya) [19:22:53] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12112064 (10Danya) [19:22:56] 10Tool-wiki-gender-stats: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12112065 (10Danya) [19:22:57] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12112066 (10Danya) [19:27:01] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_api_svc_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:29:34] 10Tool-wiki-gender-stats: Show current page name in tab title - https://phabricator.wikimedia.org/T431490#12112102 (10Danya) 05Open→03In progress [19:30:11] (03open) 10danyya: Draft: Resolve T431490 "Show current page name in tab title" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/72 [19:30:29] FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:32:01] FIRING: [9x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:32:03] FIRING: [9x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:34:44] FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:35:29] RESOLVED: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:37:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:42:03] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:42:34] FIRING: EnvvarsAdmissionDown: EnvvarsAdmission is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsAdmissionDown [19:43:06] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977) [19:43:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [19:47:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:47:34] RESOLVED: EnvvarsAdmissionDown: EnvvarsAdmission is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsAdmissionDown [19:49:09] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977) [19:49:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [19:52:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:57:01] FIRING: [11x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:59:17] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17) [19:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [20:02:42] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17) [20:02:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [20:04:48] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17) [20:04:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [20:06:05] (03update) 10danyya: Draft: Resolve T431490 "Show current page name in tab title" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/72 [20:06:41] 10Tool-wiki-gender-stats: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#12112124 (10Danya) [20:06:43] (03update) 10danyya: Resolve T431490 "Show current page name in tab title" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/72 [20:06:48] (03merge) 10danyya: Resolve T431490 "Show current page name in tab title" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/72 [20:07:09] 10Tool-wiki-gender-stats, 13Patch-For-Review: Show current page name in tab title - https://phabricator.wikimedia.org/T431490#12112127 (10Danya) 05In progress→03Resolved [20:07:10] 10Tool-wiki-gender-stats: Update Codex icons to latest version - https://phabricator.wikimedia.org/T429708#12112130 (10Danya) 05Open→03In progress [20:08:25] (03open) 10danyya: Draft: Resolve T429708 "Update Codex icons to latest version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/73 [20:12:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:19:54] (03update) 10agamyasamuel: chore: initialize uv project with pyproject.toml and lockfile [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/1 [20:20:16] (03merge) 10agamyasamuel: chore: initialize uv project with pyproject.toml and lockfile [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/1 [20:22:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:25:52] (03update) 10danyya: Draft: Resolve T429708 "Update Codex icons to latest version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/73 [20:26:27] (03update) 10danyya: Draft: Resolve T429708 "Update Codex to latest version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/73 [20:26:34] 10Tool-wiki-gender-stats, 13Patch-For-Review: Update Codex to latest version - https://phabricator.wikimedia.org/T429708#12112138 (10Danya) [20:26:43] (03update) 10danyya: Resolve T429708 "Update Codex to latest version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/73 [20:27:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:27:45] (03merge) 10danyya: Resolve T429708 "Update Codex to latest version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/73 [20:28:05] 10Tool-wiki-gender-stats, 13Patch-For-Review: Update Codex to latest version - https://phabricator.wikimedia.org/T429708#12112142 (10Danya) 05In progress→03Resolved [20:32:05] 10Tool-wiki-gender-stats: Notify tool’s devs when snapshot capture fails - https://phabricator.wikimedia.org/T424279#12112148 (10Danya) [20:32:06] 10Tool-wiki-gender-stats: Unable to run Maven/Java within the Python-based image on Toolforge - https://phabricator.wikimedia.org/T431927#12112149 (10Danya) [20:33:25] 10Tool-wiki-gender-stats: Setup push to deploy - https://phabricator.wikimedia.org/T422347#12112154 (10Danya) [20:41:16] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055#12112157 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.5.0. * https://wikitech.wikimedia.org/wiki/PAWS/Admin#Update_the_pywikibot_version * https://gerrit.wikimedia.org/g/pywik... [20:41:17] 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12112158 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.5.0. * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image * https:/... [20:42:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:45:33] (03open) 10agamyasamuel: build: add structlog dependency for structured logging [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/2 [20:45:34] (03approved) 10agamyasamuel: build: add structlog dependency for structured logging [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/2 [20:45:50] (03merge) 10agamyasamuel: build: add structlog dependency for structured logging [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/2 [20:47:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:02:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:17:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:22:03] (03open) 10agamyasamuel: feat(cache): add SQLite-backed transliteration cache module [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/3 [21:22:13] (03approved) 10agamyasamuel: feat(cache): add SQLite-backed transliteration cache module [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/3 [21:22:19] (03merge) 10agamyasamuel: feat(cache): add SQLite-backed transliteration cache module [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/3 [21:26:18] (03PS12) 10Jsn.sherman: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [21:27:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:32:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:36:32] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:50:29] (03open) 10agamyasamuel: feat: standardize transliteration schemas to list-only and add retry config [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/4 [21:50:33] (03approved) 10agamyasamuel: feat: standardize transliteration schemas to list-only and add retry config [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/4 [21:50:38] (03merge) 10agamyasamuel: feat: standardize transliteration schemas to list-only and add retry config [toolforge-repos/indicwiki-transliterate-api] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/indicwiki-transliterate-api/-/merge_requests/4 [21:52:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:02:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:07:57] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:14:28] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [22:14:30] (03open) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [22:15:22] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [22:15:26] (03open) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [22:16:18] (03update) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [22:16:22] (03open) 10renovatebot: Update pnpm to v11.12.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [22:16:32] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:17:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:19:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-gateway-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:22:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:27:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:32:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:34:32] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-gateway-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:37:01] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:42:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:48:26] 10Tool-wiki-gender-stats: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937 (10Danya) 03NEW [22:48:48] 10Tool-wiki-gender-stats: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937#12112218 (10Danya) a:03Danya [22:48:50] 10Tool-wiki-gender-stats: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937#12112221 (10Danya) p:05Triage→03High [22:49:15] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12112226 (10Danya) [22:49:16] 10Tool-wiki-gender-stats: Translate card descriptions in About page - https://phabricator.wikimedia.org/T431937#12112225 (10Danya) [22:50:17] (03PS1) 10Jsn.sherman: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) [22:52:30] 10Tool-wikinewsie: Semi-protection for images on Wikinewsie - https://phabricator.wikimedia.org/T431878#12112229 (10ZhaoFJx) a:03Habst [22:52:36] 10Tool-wikinewsie: Semi-protection for images on Wikinewsie - https://phabricator.wikimedia.org/T431878#12112231 (10ZhaoFJx) 05Open→03Resolved [22:57:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:02:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:07:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:17:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:27:01] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:32:01] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:37:01] FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:38:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [23:42:01] FIRING: [11x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:43:14] FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [23:47:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:48:14] RESOLVED: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [23:48:44] FIRING: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [23:48:59] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [23:50:23] FIRING: ToolforgeKubernetesNodeNotReady: (no data) Multiple Kubernetes nodes are not ready #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [23:50:34] FIRING: EnvvarsAdmissionDown: EnvvarsAdmission is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsAdmissionDown [23:52:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:57:01] FIRING: [12x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown