[00:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:45:03] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:47:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:50:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:52:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:53:29] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:55:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:58:14] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [03:00:03] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [03:37:40] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:26:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:27:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:31:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:32:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:02:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:04:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:05:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:09:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:10:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:12:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:53:28] 10Tool-subtitler: Subtitler buttons to be shuffled - https://phabricator.wikimedia.org/T433193 (10psubhashish1) 03NEW [05:53:44] 10Tool-subtitler: Subtitler buttons to be shuffled/adjusted for better UX - https://phabricator.wikimedia.org/T433193#12156610 (10psubhashish1) [07:58:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:59:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:03:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:04:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:05:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:06:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:10:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:11:03] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:14:03] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:18:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:19:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:03:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:04:03] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:12:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:35:28] 10Tool-wiki-gender-stats: Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196 (10Danya) 03NEW [09:35:36] 10Tool-wiki-gender-stats: Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196#12156688 (10Danya) [09:35:41] 10Tool-wiki-gender-stats: Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196#12156690 (10Danya) p:05Triage→03High [09:38:07] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12156691 (10Danya) [09:42:51] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12156693 (10Danya) [09:58:09] 10Tool-wiki-gender-stats: Create an FAQ page - https://phabricator.wikimedia.org/T425482#12156697 (10Danya) [10:15:16] 10Tool-wiki-gender-stats: Use dumps instead of live QLever/MediaWiki queries? - https://phabricator.wikimedia.org/T424147#12156734 (10Danya) 05Declined→03Resolved [10:15:17] 10Tool-wiki-gender-stats: Merge all existing tables into a single one - https://phabricator.wikimedia.org/T425891#12156736 (10Danya) 05Declined→03Resolved [10:15:19] 10Tool-wiki-gender-stats: Link to the Cloud Services End User Terms of Use - https://phabricator.wikimedia.org/T426264#12156738 (10Danya) 05Declined→03Resolved [10:20:09] 10Tool-wiki-gender-stats: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314#12156748 (10Danya) p:05Medium→03High [10:20:10] 10Tool-wiki-gender-stats: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314#12156750 (10Danya) p:05High→03Medium [10:28:46] 10Tool-wiki-gender-stats: [TF] Check every week for the latest dump - https://phabricator.wikimedia.org/T427549#12156766 (10Danya) [10:29:03] 10Tool-wiki-gender-stats: [TF] Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12156767 (10Danya) [10:29:19] 10Tool-wiki-gender-stats: [TF] Integrate May 2026 data from Danÿa’s local copy - https://phabricator.wikimedia.org/T431933#12156768 (10Danya) [10:29:37] 10Tool-wiki-gender-stats: [TF] Setup push to deploy - https://phabricator.wikimedia.org/T422347#12156770 (10Danya) [10:29:43] 10Tool-wiki-gender-stats: [TF] Try to integrate data from Humaniki - https://phabricator.wikimedia.org/T432516#12156771 (10Danya) [10:32:22] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197 (10Danya) 03NEW [10:32:33] 10Tool-wiki-gender-stats: Improve i18n system - https://phabricator.wikimedia.org/T433197#12156784 (10Danya) p:05Triage→03Low [10:34:23] 10Tool-wiki-gender-stats: Content-Security-Policy error when loading OSM map tiles - https://phabricator.wikimedia.org/T432531#12156790 (10Danya) [11:41:50] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12156843 (10Leaderboard) [11:57:36] 10Tool-wiki-gender-stats: Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199 (10Danya) 03NEW [11:58:06] 10Tool-wiki-gender-stats: Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199#12156867 (10Danya) p:05Triage→03High [11:58:27] 10Tool-wiki-gender-stats: [TF] Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199#12156873 (10Danya) [12:10:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:23:21] 10Tool-wiki-gender-stats: [TF] Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199#12156905 (10Danya) [12:25:30] 10Tool-wiki-gender-stats: Add a search bar and sorting options in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12156906 (10Danya) [12:37:20] 10Tool-wiki-gender-stats: [TF] Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199#12156927 (10Danya) [12:55:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:12:40] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:37:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:38:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:42:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:43:03] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:48:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:49:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:53:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [13:54:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:54:46] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12157151 (10Chippyy) updates: Installed trixie vm locally Installed software in the vm Issue with compiling the mapserver ruby library - we have a n... [16:15:03] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:20:03] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:07:59] 10Tool-wiki-gender-stats: [VERSION] 0.4.0 - https://phabricator.wikimedia.org/T432521#12157205 (10Danya) 05Open→03In progress [17:10:13] 10Tool-wiki-gender-stats: Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196#12157208 (10Danya) [17:10:14] 10Tool-wiki-gender-stats: [VERSION] 0.4.0 - https://phabricator.wikimedia.org/T432521#12157209 (10Danya) [17:11:52] 10Tool-wiki-gender-stats: Website unable to access database while dumps ingester is running - https://phabricator.wikimedia.org/T432052#12157217 (10Danya) 05Open→03Stalled [17:12:40] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:53:31] (03open) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [19:08:56] 10Tool-wiki-gender-stats, 13Patch-For-Review: [TF] Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12157265 (10Danya) [19:17:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:22:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:22:03] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:23:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:27:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:32:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:37:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:42:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:43:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:47:03] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:22:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed