[04:55:15] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[07:16:16] <wikibugs>	 (03PS1) 10Joal: Sqoop CentralAuth's globaluser and localuser tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata)
[07:16:16] <wikibugs>	 (03CR) 10Joal: "A bunch of comments :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata)
[07:44:15] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11151003 (10Gehel)
[07:47:36] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, and 3 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11151022 (10Gehel)
[07:50:19] <wikibugs>	 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.08.15 - 2025.09.05), 07Essential-Work, 13Patch-For-Review: SUP: Use flink 1.20.1 - https://phabricator.wikimedia.org/T398159#11151035 (10Gehel)
[07:51:30] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, and 3 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11151043 (10Gehel) 05Open→03Resolved
[07:51:33] <wikibugs>	 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.08.15 - 2025.09.05), 07Essential-Work, 13Patch-For-Review: SUP: Use flink 1.20.1 - https://phabricator.wikimedia.org/T398159#11151047 (10Gehel) 05Open→03Resolved
[07:52:01] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11151058 (10Gehel) 05In progress→03Resolved
[08:23:17] <wikibugs>	 06Data-Engineering, 10CirrusSearch, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.09.05-2025.09.26), 13Patch-For-Review: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11151153 (10Gehel)
[08:23:19] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.09.05-2025.09.26), 13Patch-For-Review: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all depende... - https://phabricator.wikimedia.org/T367405#11151155
[08:23:51] <wikibugs>	 06Data-Engineering, 06Java-Scala-Standardization, 10Discovery-Search (2025.09.05-2025.09.26): Create Gitlab CI templates for JVM packages - https://phabricator.wikimedia.org/T386406#11151179 (10Gehel)
[08:48:56] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11151321 (10dcausse) 05Resolved→03In progress there are still some issues
[08:55:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[10:49:10] <wmf-insecte>	 Starting build #10 for job wikimedia-event-utilities-maven-release
[10:55:34] <wmf-insecte>	 Project wikimedia-event-utilities-maven-release build #10: 09SUCCESS in 6 min 23 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release/10/
[11:19:20] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11151594 (10dcausse) The SUP producer is failing with: ` Caused by: java.time.format.D...
[12:55:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[14:23:25] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 2 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11152179 (10Gehel)
[14:23:52] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Request for dedicated Airflow instance for WME - https://phabricator.wikimedia.org/T396672#11152191 (10Gehel)
[14:24:02] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Fix the hard dependency between the Airflow scheduler and the DataHub GMS service - https://phabricator.wikimedia.org/T395106#11152195 (10Gehel)
[14:24:08] <wikibugs>	 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11152199 (10Gehel)
[14:24:42] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 2 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11152217 (10Gehel)
[14:25:37] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11152239 (10Gehel)
[14:26:12] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11152255 (10Gehel)
[14:27:53] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11152290 (10Gehel)
[14:29:59] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11152332 (10Gehel)
[14:51:09] <wikibugs>	 06Data-Engineering, 10CirrusSearch, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.09.05 - 2025.09.26), 13Patch-For-Review: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11152410 (10BTullis) >>! In T366248#11145000, @xcol...
[14:52:54] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11152416 (10Krinkle)
[15:05:04] <wikibugs>	 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Essential-Work: Flink: Update k8s operator to 1.12.0 - https://phabricator.wikimedia.org/T398162#11152487 (10Gehel)
[15:06:01] <wikibugs>	 06Data-Engineering, 07Essential-Work: Ensure that blunderbuss uses the minimum HDFS file system permissions required - https://phabricator.wikimedia.org/T401103#11152501 (10Gehel)
[15:08:21] <wikibugs>	 06Data-Engineering, 07Essential-Work: Errors from refinery-sqoop-whole-mediawiki.service - September 2025 - re: categorylinks - https://phabricator.wikimedia.org/T403702#11152514 (10Gehel)
[15:09:44] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11152524 (10BTullis) I have an idea for helping with the publication stage of these dumps, although I understand that this part is out-of-scope fo...
[15:11:14] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 07Essential-Work: Request for Kerberos identity for querying SSAC table via statmachines - https://phabricator.wikimedia.org/T401827#11152540 (10Gehel)
[15:32:41] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11152703 (10Gehel)
[15:33:36] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11152705 (10dcausse)
[15:35:52] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11152714 (10FCeratto-WMF)
[15:44:05] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 5 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11152737 (10dcausse) The flink job finally got restarted after numerous issues: - fix...
[16:16:59] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11152860 (10Krinkle)
[16:17:47] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11152865 (10Krinkle)
[16:50:15] <jinxer-wm>	 RESOLVED: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[17:30:24] <wikibugs>	 (03PS1) 10Aleksandar Mastilovic: Fix wmf_raw.categorylinks column types [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1185169 (https://phabricator.wikimedia.org/T397923)
[18:12:38] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11153283 (10Krinkle)
[18:16:51] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11153301 (10Krinkle)
[18:23:15] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[18:53:15] <jinxer-wm>	 RESOLVED: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[19:52:18] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: EventBus - Add central user id to MediaWiki events - https://phabricator.wikimedia.org/T403664#11153642 (10Ottomata) Hm, I started with that, but [[ https://www.mediawiki.org/wiki/Manual:Central_ID | MediaWiki so...
[20:02:18] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): OpsWeek: spark_process_reconciliation_events for monthly reconcile fails with FetchFailedException - https://phabricator.wikimedia.org/T403635#11153680 (10xcollazo) This was finally successful sometime today.
[20:51:50] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11153781 (10Jdlrobson-WMF) Hey @krinkle!  In general, I'm fairly confident you have thought of everything but my time here has shown me there is always so...