[00:27:07] <wmcs-alerts>	 FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing
[01:46:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:51:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:10:43] <wikibugs>	 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12152226 (10Yaron_Koren) Well, the first step is to add the right structure to the code, ideally by having an i18n/ directory containing at least two files, en.json and qqq.json. After that, it's certainl...
[07:58:43] <wikibugs>	 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12152478 (10AGhirelli-WMF) 05In progress→03Resolved
[08:01:39] <wikibugs>	 10Tool-wmf-openapi-linter, 06Test Platform, 03[MWI] FY2025-26 Q4, 10Continuous-Integration-Config, and 2 others: [5.2.5b Epic] Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#12152482 (10KBach) 05Open→03Resolved a:03KBach
[08:39:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[08:44:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[08:51:16] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] build: Updating mediawiki/mediawiki-codesniffer to 51.0.0 [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1283136 (owner: 10Libraryupgrader)
[08:53:00] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: Updating mediawiki/mediawiki-codesniffer to 51.0.0 [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1283136 (owner: 10Libraryupgrader)
[09:09:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on clouddb1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:20:07] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152603 (10ops-monitoring-bot) Rebooting clouddb1013.eqiad.wmnet
[09:25:48] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152610 (10ops-monitoring-bot) Reboot of clouddb1013.eqiad.wmnet completed
[09:26:03] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152614 (10ops-monitoring-bot) Rebooting clouddb1014.eqiad.wmnet
[09:27:18] <wikibugs>	 (03CR) 10Jforrester: [V:03+2 C:03+2] "Let's land this now and fix the guzzle issue next." [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1283136 (owner: 10Libraryupgrader)
[09:32:17] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152626 (10ops-monitoring-bot) Reboot of clouddb1014.eqiad.wmnet completed
[09:32:34] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152627 (10ops-monitoring-bot) Rebooting clouddb1015.eqiad.wmnet
[09:33:21] <wikibugs>	 (03PS1) 10Jforrester: Upgrade guzzlehttp/guzzle from ^6.3 to ^7.15.1 for security issue [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1315774
[09:40:40] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152633 (10ops-monitoring-bot) Reboot of clouddb1015.eqiad.wmnet completed
[09:41:30] <jinxer-wm>	 FIRING: [2x] SystemdUnitCrashLoop: wmf-pt-kill@s4.service crashloop on clouddb1015:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[09:42:00] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152635 (10ops-monitoring-bot) Rebooting clouddb1016.eqiad.wmnet
[09:44:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on clouddb1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:46:30] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitCrashLoop: wmf-pt-kill@s4.service crashloop on clouddb1015:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[09:47:13] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12152640 (10Marostegui) clouddb1032 replicating on s4 and s6
[09:48:35] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152643 (10ops-monitoring-bot) Reboot of clouddb1016.eqiad.wmnet completed
[09:48:52] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152644 (10ops-monitoring-bot) Rebooting clouddb1017.eqiad.wmnet
[09:53:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12152647 (10fnegri) 05Resolved→03In progress History length is still growing:  {F95587182}  Current long transactions, most of them are from s51434...
[09:54:48] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152653 (10ops-monitoring-bot) Reboot of clouddb1017.eqiad.wmnet completed
[09:54:59] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152654 (10ops-monitoring-bot) Rebooting clouddb1018.eqiad.wmnet
[10:00:37] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152710 (10ops-monitoring-bot) Reboot of clouddb1018.eqiad.wmnet completed
[10:00:51] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152711 (10ops-monitoring-bot) Rebooting clouddb1020.eqiad.wmnet
[10:02:31] <jinxer-wm>	 FIRING: SystemdUnitCrashLoop: wmf-pt-kill@s7.service crashloop on clouddb1018:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[10:06:45] <jinxer-wm>	 RESOLVED: SystemdUnitCrashLoop: wmf-pt-kill@s7.service crashloop on clouddb1018:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[10:07:15] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152717 (10ops-monitoring-bot) Reboot of clouddb1020.eqiad.wmnet completed
[10:07:31] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152718 (10ops-monitoring-bot) Rebooting clouddb1022.eqiad.wmnet
[10:07:58] <wikibugs>	 10Tool-techcontribs: add total gerrit patch count - https://phabricator.wikimedia.org/T428624#12152719 (10Chlod) 05Open→03In progress p:05Triage→03Medium a:03Chlod
[10:13:03] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152751 (10ops-monitoring-bot) Reboot of clouddb1022.eqiad.wmnet completed
[10:13:22] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152754 (10ops-monitoring-bot) Rebooting clouddb1023.eqiad.wmnet
[10:18:30] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152764 (10ops-monitoring-bot) Reboot of clouddb1023.eqiad.wmnet completed
[10:18:45] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152767 (10ops-monitoring-bot) Rebooting clouddb1024.eqiad.wmnet
[10:22:22] <jinxer-wm>	 FIRING: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section x4 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable
[10:22:57] <wikibugs>	 10Tool-techcontribs: Detect Wikimedia developer account from SUL username - https://phabricator.wikimedia.org/T432584#12152773 (10Chlod) Ta-da! {F95592331}
[10:23:07] <wikibugs>	 10Tool-techcontribs: Detect Wikimedia developer account from SUL username - https://phabricator.wikimedia.org/T432584#12152775 (10Chlod) 05Open→03Resolved
[10:23:10] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152776 (10fnegri) The cookbook worked fine until clouddb1024, which has only 1 section, but should still be considered a...
[10:23:11] <wikibugs>	 10Tool-techcontribs: add total gerrit patch count - https://phabricator.wikimedia.org/T428624#12152778 (10Chlod) 05In progress→03Resolved
[10:26:40] <wikibugs>	 10Tool-wikinewsie: Create new Wikidata items from Wikinewsie - https://phabricator.wikimedia.org/T432411#12152792 (10Pharos) This would presumably make use of [[ https://www.wikidata.org/wiki/Help:QuickStatements | QuickStatements ]] or some variant.  It would be good if, like QuickStatements, the Wikidata edit...
[10:29:06] <wikibugs>	 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12152795 (10AGhirelli-WMF)
[10:30:55] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152797 (10ops-monitoring-bot) Rebooting clouddb1024.eqiad.wmnet
[10:35:48] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152804 (10ops-monitoring-bot) Reboot of clouddb1024.eqiad.wmnet completed
[10:35:50] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152805 (10ops-monitoring-bot) Rebooting clouddb1025.eqiad.wmnet
[10:37:22] <jinxer-wm>	 RESOLVED: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section x4 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable
[10:39:56] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12152813 (10mikez-WMF) @Doc_James Those would be great to add as they're still open - we haven't planned anything for them.   Below is a summary of all the rela...
[10:42:09] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12152831 (10fnegri) > All the ones like prometheus-mysqld-exporter.service simply need a systemctl disable prometheus-mysqld-exporter.service...
[10:46:58] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12152842 (10Doc_James) We do have blurring here already on Image Annotation  https://image-annotation-tool.wmcloud.org/?file=File:Gout%20Signs%20and%20Symptoms.jpg
[10:48:28] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152845 (10ops-monitoring-bot) Rebooting clouddb1025.eqiad.wmnet
[10:49:31] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12152848 (10Doc_James) Go to "filters" and select "blur"  But I imagine folks just want the ability to blur certain parts of the image instead of the entire ima...
[11:02:02] <wikibugs>	 10Tool-wikinewsie: RTL Codex framework for Wikinewsie - https://phabricator.wikimedia.org/T433052 (10Pharos) 03NEW
[11:04:40] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS wlmitvisual01: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429723#12152926 (10Ysogo) A quick update: we have completed the logical moving of application and data from old server to the new servers.  I have no...
[11:10:01] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152936 (10fnegri) The new patch https://gerrit.wikimedia.org/r/1315819 fixed the issue in clouddb1...
[11:12:01] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152940 (10ops-monitoring-bot) Rebooting clouddb1025.eqiad.wmnet
[11:17:39] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152944 (10ops-monitoring-bot) Reboot of clouddb1025.eqiad.wmnet completed
[11:17:55] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152945 (10ops-monitoring-bot) Rebooting clouddb1026.eqiad.wmnet
[11:21:11] <wikibugs>	 10Tool-techcontribs: Detect Wikimedia developer account from SUL username - https://phabricator.wikimedia.org/T432584#12152947 (10valerio.bozzolan) Works perfectly ❤️ thanks Chlod. Tried all variances with/without underscores, and with/without first letter uppercase.
[11:22:21] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152950 (10ops-monitoring-bot) Reboot of clouddb1026.eqiad.wmnet completed
[11:22:37] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152951 (10ops-monitoring-bot) Rebooting clouddb1027.eqiad.wmnet
[11:27:46] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152960 (10ops-monitoring-bot) Reboot of clouddb1027.eqiad.wmnet completed
[11:28:04] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152961 (10ops-monitoring-bot) Rebooting clouddb1028.eqiad.wmnet
[11:33:10] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152986 (10ops-monitoring-bot) Reboot of clouddb1028.eqiad.wmnet completed
[11:33:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12152987 (10ops-monitoring-bot) Rebooting clouddb1029.eqiad.wmnet
[11:50:20] <wikibugs>	 10Tool-wikinewsie, 03Wikimania-Hackathon-2026: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12153012 (10Yaron_Koren) I added an en.json file here, with all the text messages I could find in App.vue and ArticleModal.vue:  https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/blob/main/i18n/en...
[12:07:11] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055 (10Seb35) 03NEW
[13:11:16] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153142 (10ops-monitoring-bot) Rebooting clouddb1029.eqiad.wmnet
[13:13:22] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153155 (10fnegri) clouddb1029 failed because `x3` was removed from that host, but `/etc/mysql/mysq...
[13:14:50] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153170 (10ops-monitoring-bot) Reboot of clouddb1029.eqiad.wmnet completed
[13:15:03] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153171 (10ops-monitoring-bot) Rebooting clouddb1030.eqiad.wmnet
[13:19:21] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153182 (10ops-monitoring-bot) Reboot of clouddb1030.eqiad.wmnet completed
[13:19:35] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153184 (10ops-monitoring-bot) Rebooting clouddb1031.eqiad.wmnet
[13:21:29] <jinxer-wm>	 FIRING: SystemdUnitCrashLoop: wmf-pt-kill@s1.service crashloop on clouddb1030:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[13:24:38] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12153190 (10ops-monitoring-bot) Reboot of clouddb1031.eqiad.wmnet completed
[13:26:29] <jinxer-wm>	 RESOLVED: SystemdUnitCrashLoop: wmf-pt-kill@s1.service crashloop on clouddb1030:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop
[13:35:57] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12153233 (10fnegri) > The wmf-pt-kill@x3.service and all the x3 ones are old and can be removed, that host used to have x3 but we cleaned up...
[13:41:32] <wikibugs>	 06tools-platform-team, 10Toolhub: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12153255 (10aputhin) Just a note, we ended up upgrading to Bookworm as Trixie would be a more complicated endeavor and we needed to move on from Bullseye as...
[13:43:43] <wikibugs>	 06tools-platform-team, 10Toolhub: Upgrade ToolHub base image to Trixie - https://phabricator.wikimedia.org/T433065 (10fnegri) 03NEW
[13:44:30] <wikibugs>	 06tools-platform-team, 10Toolhub: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12153272 (10fnegri) The followup is {T433065}.
[13:50:35] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12153289 (10fnegri) Everything else cleared on its own with the rolling restarts I did in {T420203}.
[13:52:40] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12153298 (10fnegri) 05Open→03Resolved a:03fnegri
[14:43:34] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10LibUp: Replace libup-web02 (bullseye) - https://phabricator.wikimedia.org/T432728#12153426 (10taavi) 05Open→03Resolved
[14:46:37] <wikibugs>	 (03PS2) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295)
[14:47:34] <wikibugs>	 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12153437 (10fnegri) 05Open→03In progress
[14:47:40] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12153439 (10fnegri) 05Open→03In progress
[14:47:48] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12153441 (10fnegri) a:03fnegri
[14:47:58] <wikibugs>	 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12153442 (10fnegri) a:03fnegri
[14:48:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds
[15:10:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0)
[15:18:45] <wikibugs>	 (03CR) 10Andrew Bogott: "tested latest in codfw1dev although of course that doesn't actually test the alerting infra fully" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott)
[15:30:37] <wikibugs>	 06cloud-services-team, 10Striker, 06tools-platform-team, 10Diffusion, and 2 others: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12153874 (10Aklapper) Striker's `repo.py::create()` first calls `phab.py::crea...
[15:37:30] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12153905 (10Bawolff) I think there is a risk of feature creep here
[15:41:55] <wikibugs>	 (03PS1) 10Aklapper: Remove unneeded IO setting for default readwrite Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341)
[15:44:26] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove unneeded IO setting for default readwrite Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper)
[15:51:11] <wikibugs>	 (03CR) 10Aklapper: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper)
[15:54:05] <wikibugs>	 (03CR) 10Majavah: "the failure is:" [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper)
[15:54:33] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12153977 (10fnegri) I have started looking into this.  A query like this one seems to identify the tools where the number of 5xx responses were more than 50% of a...
[16:11:38] <wikibugs>	 10Tool-sentinel: Restrict admin approve/reject to valid status transitions - https://phabricator.wikimedia.org/T431893#12154063 (10EbenezerRao) a:05Arian_Ar→03EbenezerRao Hiii I would like to work on this task..  Claiming it
[16:32:14] <wikibugs>	 (03open) 10ebenezerrao: Restrict admin approval and rejection to valid status transitions [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/14
[16:34:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:39:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:40:39] <wikibugs>	 10Tool-sentinel: Restrict admin approve/reject to valid status transitions - https://phabricator.wikimedia.org/T431893#12154190 (10EbenezerRao) Hi! @Arian_Ar   I've submitted a Merge Request implementing the requested validation for admin approve/reject status transitions.  MR: https://gitlab.wikimedia.org/toolf...
[16:46:07] <wikibugs>	 10Tool-sentinel: Remove redundant local import json in mediawiki.py - https://phabricator.wikimedia.org/T431907#12154207 (10EbenezerRao) Hi, @Arian_Ar   I checked backend/app/services/mediawiki.py and it appears the file already uses a top-level import json; I couldn't find any remaining local import json statem...
[16:53:50] <wikibugs>	 (03CR) 10Aklapper: "Thanks. I got lost clicking in the web interface. :-/" [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper)
[16:54:07] <wikibugs>	 (03PS2) 10Aklapper: Remove unneeded IO setting for default readwrite Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341)
[17:09:32] <wikibugs>	 10Tool-sentinel: Remove or wire unused SECRET_KEY setting - https://phabricator.wikimedia.org/T431905#12154272 (10EbenezerRao) a:05Arian_Ar→03EbenezerRao Hi!  I'd like to work on this task.  Claiming it.
[17:30:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Epic: Enable self-service Prometheus configuration management for project administrators - https://phabricator.wikimedia.org/T284993#12154323 (10Gerges)
[17:52:19] <wikibugs>	 (03open) 10ebenezerrao: Remove unused SECRET_KEY setting [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/15
[17:56:40] <wikibugs>	 10Tool-sentinel: Remove or wire unused SECRET_KEY setting - https://phabricator.wikimedia.org/T431905#12154416 (10EbenezerRao) Hi @Arian_Ar ,  I've submitted a Merge Request implementing the removal of the unused SECRET_KEY setting.  MR: https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/15...
[18:31:55] <wikibugs>	 (03CR) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott)
[18:41:49] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12154524 (10Andrew) I am setting   ` root@cloudcephmon1005:~# ceph config set osd.XX bdev_enable_discard true root@cloudce...
[19:49:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Request creation of wiki-community-metrics-service VPS project - https://phabricator.wikimedia.org/T429983#12154671 (10Aklapper) 05Stalled→03Declined Unfortunately closing this Phabricator task as no further information has been provide...
[20:12:30] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 (10Andrew) 03NEW
[20:13:02] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Upgrade toolforge to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 (10Andrew) 03NEW
[20:13:18] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12154837 (10Andrew)
[20:23:39] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12154870 (10Andrew)
[20:24:09] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12154871 (10Andrew)
[20:24:11] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12154873 (10Andrew)
[20:24:13] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12154872 (10Andrew)
[20:30:46] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786#12154877 (10Andrew)
[21:25:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:28:32] <wikibugs>	 10Tool-wiki-gender-stats: Add a search bar in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12155046 (10Danya) a:03Danya
[21:30:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:25:54] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12155174 (10TheDJ) Maybe we should start with having a good consistent, recognizable and usable place to have these tools, instead of spreading them around the...