[00:01:34] <wikibugs>	 (03PS3) 10MusikAnimal: RateLimitSubscriber: improve performance and use client IP [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888)
[00:03:13] <wikibugs>	 (03CR) 10MusikAnimal: "@jsherman@wikimedia.org (No rush to review this): I was not able conclude that your solution at T430888#12082192 isolated the rate limitin" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal)
[00:29:04] <wikibugs>	 06cloud-services-team, 10Striker: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12083896 (10bd808) I'm not seeing a duplicate: ` striker_admin@m5-master.eqiad.wmnet(striker)> select id from labsauth_labsuser where sulname = 'Kavaljeet Singh'\...
[02:09:04] <wikibugs>	 10Tool-tome: Optimise Tome for images and documents - https://phabricator.wikimedia.org/T431013#12083943 (10psubhashish1) Fixed at https://gitlab.wikimedia.org/toolforge-repos/tome/-/commit/1d2f631c3b50f337a17348d898271d2b6e97881d
[02:09:14] <wikibugs>	 10Tool-tome: Optimise Tome for images and documents - https://phabricator.wikimedia.org/T431013#12083944 (10psubhashish1) 05Open→03Resolved a:03psubhashish1
[02:19:48] <wikibugs>	 10Tool-tome: Tome 1.3.0 input field issues - https://phabricator.wikimedia.org/T431073 (10psubhashish1) 03NEW
[02:24:09] <wikibugs>	 10Tool-tome: Tome 1.3.0 input field issues - https://phabricator.wikimedia.org/T431073#12083961 (10psubhashish1)
[02:24:48] <wikibugs>	 10Tool-tome: Tome 1.3.0 input field issues - https://phabricator.wikimedia.org/T431073#12083962 (10psubhashish1)
[02:24:59] <wikibugs>	 10Tool-tome: Tome 1.3.0 input field issues - https://phabricator.wikimedia.org/T431073#12083963 (10psubhashish1)
[02:42:43] <wikibugs>	 10Cloud-VPS (Project-requests), 10Maps: Check current usage of project maps on CloudVPS - https://phabricator.wikimedia.org/T429446#12083977 (10dschwen) Thank you!
[02:51:13] <wikibugs>	 (03open) 10countcount: Add /currentname endpoint resolving historical names to the current SUL name and global user id [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/36
[03:03:21] <wikibugs>	 (03merge) 10countcount: Add /currentname endpoint resolving historical names to the current SUL name and global user id [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/36
[03:15:24] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28
[04:07:56] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:12:56] <wmcs-alerts>	 RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:55:04] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12084049 (10KBach)
[05:55:05] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: OAD linter gives warnings for missing response examples in RESTBase compatibility properties - https://phabricator.wikimedia.org/T431018#12084047 (10KBach)
[05:55:07] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), and 2 others: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12084048 (10KBach)
[05:55:47] <wikibugs>	 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support path-level summary and description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427362#12084050 (10KBach)
[05:55:48] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12084052 (10KBach)
[06:24:41] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12084057 (10KCVelaga_WMF) Hi @BTullis @Ottomata: can we move the schema out of development namespace...
[06:40:50] <wikibugs>	 06cloud-services-team, 10Striker: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12084069 (10fgiunchedi) I looked at, and reported, logs as part of clinic duty though did not touch the db FWIW
[06:50:16] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12084087 (10fgiunchedi) >>! In T411248#12080727, @ayounsi wrote: >>>! In T411248#12076020, @fg...
[08:00:56] <wikibugs>	 10Tool-tome: Tome 1.3.0 input field issues - https://phabricator.wikimedia.org/T431073#12084211 (10psubhashish1) 05Open→03Resolved a:03psubhashish1 [[ https://gitlab.wikimedia.org/toolforge-repos/tome/-/commit/bbe24ae880a64c6139fb355dd4c0b807b9917181 | Resolved ]]
[08:12:51] <wikibugs>	 10Tool-tome: Image-specific metadata vs. existing video-specific metadata - https://phabricator.wikimedia.org/T431082 (10psubhashish1) 03NEW
[08:28:54] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting, 13Patch-For-Review: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#12084341 (10fgiunchedi) On balance I like option 2, which ensures truly end-to-end monitoring of container build + deploy. In...
[08:33:35] <wikibugs>	 10Tool-tome: Tome: Auto-populate language, make names appear in category, adding subtitles details in other languages - https://phabricator.wikimedia.org/T431087 (10psubhashish1) 03NEW
[08:45:58] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#12084437 (10taavi) 05In progress→03Resolved
[09:04:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: puppet failing on metricsinfra-prometheus-2.metricsinfra due to expired crl - https://phabricator.wikimedia.org/T429298#12084513 (10fgiunchedi) Could be yeah, I don't know if puppet agent is actually supposed to be doing its own crl management/...
[09:05:25] <wm-bot2>	 !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650009604 (https://github.com/cluebotng/component-configs/commits/a66ec11b207598051416135000b2a5815ba30080)
[09:05:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[09:09:38] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: web proxy service: limit to one backend per web proxy - https://phabricator.wikimedia.org/T429960#12084522 (10taavi) a:03taavi
[09:14:41] <wm-bot2>	 !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650658730 (https://github.com/cluebotng/component-configs/commits/0d3a15b63afd3afe5965be4754401712df0b4ae1)
[09:14:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[09:17:17] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12084605 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet
[09:25:03] <wm-bot2>	 !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28651139847 (https://github.com/cluebotng/component-configs/commits/11ad538e5876626229416e2ba16bc9f7a7e5cace)
[09:25:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[09:45:33] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12084712 (10Gehel)
[09:46:03] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12084730 (10Gehel)
[09:46:18] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12084732 (10Gehel)
[09:46:36] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12084736 (10Gehel)
[09:52:16] <wikibugs>	 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12084859 (10Gehel)
[09:52:28] <wikibugs>	 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12084857 (10Gehel)
[09:54:38] <wikibugs>	 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12084890 (10Gehel)
[09:55:28] <wikibugs>	 10Toolforge, 06tools-platform-team: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12084925 (10fgiunchedi) Indeed now with the work by @taavi on {T392356} we do have per-tool status codes as seen by istio, i.e. the metric used in http...
[09:59:14] <wikibugs>	 10Toolforge, 06tools-platform-team: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12085036 (10fgiunchedi) The tool availability metric will need to be a recording rule because `istio_requests_total` is huge. And the same per-tool ava...
[10:03:00] <wikibugs>	 10Toolforge, 06tools-platform-team: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12085068 (10taavi) Moving the alerting to be based on Istio-level metrics means we lose visibility into any potential connectivity issues between HAPro...
[11:44:46] <wmcs-alerts>	 FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:45:14] <wmcs-alerts>	 FIRING: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:49:46] <wmcs-alerts>	 FIRING: [10x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:50:14] <wmcs-alerts>	 RESOLVED: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:50:34] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:54:46] <wmcs-alerts>	 RESOLVED: [9x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:55:34] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-haproxy-8 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:33:04] <wm-bot2>	 !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28663718696 (https://github.com/cluebotng/component-configs/commits/8b35b36bf642f83b59af6cf2903f9fcdd6d67007)
[13:33:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[13:48:20] <wikibugs>	 (03open) 10fnegri: ToolforgeWebHighErrorRate: increase threshold [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65 (https://phabricator.wikimedia.org/T429738)
[13:49:43] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12085766 (10fnegri) Maybe we could keep the haproxy alert but with a higher threshold? I opened https://gitlab.wikimedia.org/repo...
[13:50:41] <wikibugs>	 (03approved) 10taavi: ToolforgeWebHighErrorRate: increase threshold [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65 (https://phabricator.wikimedia.org/T429738) (owner: 10fnegri)
[13:51:29] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12085771 (10fnegri) To clarify my previous comment: maybe we could keep the haproxy alert at a 50% threshold (or higher), but als...
[13:55:43] <wikibugs>	 (03merge) 10fnegri: ToolforgeWebHighErrorRate: increase threshold [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/65 (https://phabricator.wikimedia.org/T429738)
[14:03:47] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12085830 (10fnegri) Restarted the `scholia` tool, and posted an update in https://github.com/WDscholia/scholia/issues/2794.
[14:07:31] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085846 (10EMill-WMF) @Soda Thanks for pushing here - your analysis is right, and we are fine with exposing this value without...
[14:11:33] <wmcs-alerts>	 FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 93.35%) at host paws-nfs-2 in project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull
[14:19:00] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, and 3 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085858 (10taavi) Similar to T402145, this table is currently marked as `visibility: private` in the catalog, and so isn't replicated to the Wi...
[14:26:54] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#12085885 (10fnegri) We discussed this during the last Toolforge Monthly Meeting ([meeting notes](https://wikitech.wikimedia.org/wiki/Portal:Toolforge/A...
[14:41:29] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Pull upstream heroku images weekly - https://phabricator.wikimedia.org/T431125 (10fnegri) 03NEW
[14:41:51] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Pull upstream heroku images weekly - https://phabricator.wikimedia.org/T431125#12085931 (10fnegri)
[14:41:53] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#12085932 (10fnegri)
[14:48:55] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Update the "stable" image once a month - https://phabricator.wikimedia.org/T431126 (10fnegri) 03NEW
[14:48:58] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Update the "stable" image once a month - https://phabricator.wikimedia.org/T431126#12085963 (10fnegri)
[14:48:59] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Pull upstream heroku images weekly - https://phabricator.wikimedia.org/T431125#12085964 (10fnegri)
[14:53:11] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, and 3 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085972 (10Marostegui) >>! In T344108#12085858, @taavi wrote: > Similar to T402145, this table is currently marked as `visibility: private` in...
[14:53:35] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, and 3 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085974 (10Marostegui) >>! In T344108#12085972, @Marostegui wrote: >>>! In T344108#12085858, @taavi wrote: >> Similar to T402145, this table is...
[14:55:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, and 3 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085978 (10taavi) >>! In T344108#12085973, @Marostegui wrote: > This would need to be signed off by Security.  That's T344108#12085846, unless...
[14:57:14] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Upgrade to "heroku-26" stack - https://phabricator.wikimedia.org/T431127 (10fnegri) 03NEW
[14:57:31] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-builder] Upgrade to "heroku-26" stack - https://phabricator.wikimedia.org/T431127#12085996 (10fnegri)
[14:57:34] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#12085997 (10fnegri)
[14:58:20] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, and 3 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12085999 (10Marostegui) Ah right! Missed that! THanks
[14:58:50] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#12086003 (10fnegri) 05In progress→03Resolved Follow-up tasks created, and linked as "parents" to this task: * {T431125} * {T431126} * {T431127}
[15:08:38] <wikibugs>	 06cloud-services-team, 10Toolforge: updatetools should alert instead of emailing - https://phabricator.wikimedia.org/T413099#12086027 (10fnegri) 05Open→03Declined `updatetools` was deleted in {T413431} so this is no longer an issue.
[16:14:00] <wikibugs>	 (03PS1) 10Harsh_Kushwaha07: Fix typo in tools/lsa js files to improve comments [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1307479 (https://phabricator.wikimedia.org/T201491)
[16:29:57] <wikibugs>	 06cloud-services-team, 10Quarry, 13Patch-For-Review, 07patch-welcome, 07Plural-Support: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12086212 (10Nihar_Chakravarti) Thanks for pointing that out, @Aklapper ! I've updated the PR—reverted the compiled JS file changes and cleared out...
[16:48:27] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[16:48:40] <wikibugs>	 (03close) 10arpithhhaaa: Improve UI layout and subtitle editor [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/8
[16:50:16] <wikibugs>	 (03open) 10arpithhhaaa: Finalmodification [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/9
[16:52:59] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[16:55:54] <wikibugs>	 (03open) 10arian-ar: Fix ToolsDB stale connection 500s after idle periods [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/7
[16:56:35] <wikibugs>	 (03merge) 10arian-ar: Fix ToolsDB stale connection 500s after idle periods [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/7
[16:58:07] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[17:02:49] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[17:07:00] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12086284 (10fnegri) @CWilliams-WMF thanks for testing this! I fixed the small issues you flagged on...
[17:20:42] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06tools-platform-team, 13Patch-For-Review: [wikireplicas] add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#12086295 (10fnegri) 05In progress→03Resolved Marking as Resolved, as there is nothing left...
[17:24:05] <wikibugs>	 06cloud-services-team, 10Data-Services, 13Patch-For-Review: [wikireplicas] Add an option to cookbooks to specify which hosts should be targeted - https://phabricator.wikimedia.org/T393387#12086314 (10fnegri) We could also add the new "--diff-mode" option (added in {T351637}) to `sre.wikireplicas.update-views...
[17:29:25] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[17:37:00] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[17:39:31] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[17:41:33] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[18:02:11] <wikibugs>	 10Tools: Toolforge CDNjs mirror can no longer obtain all versions of libraries - https://phabricator.wikimedia.org/T431039#12086329 (10LucasWerkmeister) I suppose an alternative solution would be to fetch the URLs for alternative solutions on demand as the user clicks that library version. In principle we could...
[18:10:15] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[18:27:16] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[18:27:41] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[18:47:25] <wikibugs>	 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12086392 (10TLepage-WMF) I believe this is caused by calling `datetime.now` on two separate occasions ([[ https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/mai...
[18:52:21] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[19:10:14] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12086426 (10Ottomata) Yes let's do it!
[19:16:13] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12086454 (10Ottomata) Oh I missed Ben's questions!  > Right, but do we do it before renaming the stre...
[20:22:52] <wikibugs>	 (03merge) 10sanskardubeydev: fix(a11y): add missing ARIA labels to interactive elements [toolforge-repos/wikisemanticimgsearch] - 10https://gitlab.wikimedia.org/toolforge-repos/wikisemanticimgsearch/-/merge_requests/1 (owner: 10zaidusyy)
[20:57:14] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: Easing pain points caused by divergence between cloudservices and production puppet usecases - https://phabricator.wikimedia.org/T285539#12086717 (10Tvassilian) 05Open→03In progress
[21:28:52] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[21:36:50] <wikibugs>	 (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45
[21:39:56] <wikibugs>	 (03CR) 10Alien4444: AppExtension: cache git hash shown in footer of every page (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal)
[21:53:37] <wikibugs>	 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146 (10LucasWerkmeister) 03NEW
[21:59:28] <wikibugs>	 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146#12086795 (10LucasWerkmeister) Tagging this with #OAuth in the hope that someone knows what might be...
[22:00:06] <wikibugs>	 10Tool-lexeme-forms, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Error “The refresh token is invalid.” after migrating Wikidata Lexeme Forms to OAuth 2 - https://phabricator.wikimedia.org/T431146#12086797 (10LucasWerkmeister) (The only other Phabricator task with this error message is T268256, w...