[02:12:36] 10Data-Engineering-Roadmap, 07Epic: [Idea] Collect pageview data using client-side instrumentation - https://phabricator.wikimedia.org/T371321#10961733 (10nshahquinn-wmf) [02:18:19] 10Data-Engineering-Roadmap, 07Epic: Upgrade analytics-hadoop to Spark 3 + scala 2.12 - https://phabricator.wikimedia.org/T291464#10961734 (10nshahquinn-wmf) 05Open→03Resolved [03:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [03:54:24] 10Data-Engineering-Roadmap, 07Epic: [Idea] Collect pageview data using client-side instrumentation - https://phabricator.wikimedia.org/T371321#10961828 (10nshahquinn-wmf) Thank you, @dr0ptp4kt, @mpopov, and @Milimetric, for your super-thoughful responses! I've tried to summarize things in the task description;... [04:08:50] 10Data-Engineering-Roadmap, 07Epic: [Idea] Collect pageview data using client-side instrumentation - https://phabricator.wikimedia.org/T371321#10961830 (10nshahquinn-wmf) Also, a couple of notes that don't belong in the description: - @Milimetric and @Ottomata both said they're excited (rather than neutral or... [04:34:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [04:34:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:19:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:22:06] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:22:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [07:14:37] 06Data-Engineering, 10Datasets-General-or-Unknown, 10Pageviews-API: Pageviews / Mostread Data not available since June 28, 2025 - https://phabricator.wikimedia.org/T398150#10961936 (10JAllemandou) We have experienced a data ingestion issue last Saturday (28th of June) that we solved yesterday (Monday June 30... [07:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:38:05] 06Data-Engineering: mediawiki_api_request has no data for June 25 onwards - https://phabricator.wikimedia.org/T398173#10962000 (10Urbanecm_WMF) Thanks for the update @gmodena. I can confirm I see new data. I think this task can be declined then, as it is not possible to restore data from June 25-30. [07:38:17] 06Data-Engineering: mediawiki_api_request has no data for June 25 onwards - https://phabricator.wikimedia.org/T398173#10962001 (10Urbanecm_WMF) 05Open→03Declined [08:09:51] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [08:09:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [08:22:36] 10Data-Engineering (Q4 2025 April 1st - June 30th): Manage druid `webrequest_sampled_live` data size - https://phabricator.wikimedia.org/T398236#10962085 (10Joe) >>! In T398236#10960969, @JAllemandou wrote: >> Are there stats available for which columns are the heaviest on storage? > > I don't have those unfort... [08:39:37] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10962148 (10Anton.Kokh) {F62759310} Hello, Here's the key. Thank you! Anton [09:31:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:41:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:41:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:49:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:49:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:59:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:59:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:03:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:03:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:23:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:28:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:28:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:36:54] 10Data-Engineering-Roadmap, 07Epic: [Idea] Collect pageview data using client-side instrumentation - https://phabricator.wikimedia.org/T371321#10962525 (10Milimetric) > - @dr0ptp4kt said that, if we do this, it should be as a complement to log-based pageviews rather than a replacement. I very much agree; altho... [10:58:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:58:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:58:44] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to airflow-an and statboxes for htriedman - https://phabricator.wikimedia.org/T398075#10962577 (10Clement_Goubert) >>! In T398075#10961146, @Htriedman wrote: > Hi @Clement_Goubert! When I navigate to the L3 document... [11:02:16] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to airflow-an and statboxes for htriedman - https://phabricator.wikimedia.org/T398075#10962584 (10Clement_Goubert) [11:04:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:14:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:14:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:17:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:23:02] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10962642 (10Clement_Goubert) >>! In T395917#10962148, @Anton.Kokh wrote: >... [11:23:06] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10962643 (10Clement_Goubert) 05Stalled→03In progress [11:27:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:32:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:42:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:46:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [11:46:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [11:49:31] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to airflow-an and statboxes for htriedman - https://phabricator.wikimedia.org/T398075#10962776 (10brouberol) @Clement_Goubert: @Htriedman should have access to all airflow instances as part of... [11:51:53] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to airflow-an and statboxes for htriedman - https://phabricator.wikimedia.org/T398075#10962811 (10Clement_Goubert) >>! In T398075#10962776, @brouberol wrote: > @Clement_Goubert: @Htriedman shou... [11:58:03] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10962850 (10BTullis) I have confirmed that membership of `analytics-private... [12:43:45] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user (and LDAP nda, wmde) for Anton Kokh (WMDE) - https://phabricator.wikimedia.org/T395917#10963212 (10WMDE-leszek) Hi @BTullis, thanks, I appreciate thorough checkin... [13:09:38] (03CR) 10Joal: [V:03+2 C:03+2] "I'm going to merge this as I know it's important for incidents. I'll monitor data growth when I get back for holidays and we'll continue t" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1165054 (https://phabricator.wikimedia.org/T398213) (owner: 10Giuseppe Lavagetto) [13:16:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:16:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:19:17] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: hrwiki-20250620-pagelinks.sql.gz dump not correct? - https://phabricator.wikimedia.org/T398265#10963437 (10xcollazo) 05Open→03In progress [13:20:05] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: hrwiki-20250620-pagelinks.sql.gz dump not correct? - https://phabricator.wikimedia.org/T398265#10963440 (10xcollazo) a:03xcollazo [13:26:18] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#10963470 (10mforns) [14:09:05] q [14:09:09] oops [14:15:05] 06Data-Engineering, 10EventStreams: Figure out how Eventstreams connected client metrics went negative - https://phabricator.wikimedia.org/T398325 (10tchin) 03NEW [14:15:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:15:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:20:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:20:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:35:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:35:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:40:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:40:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:55:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:55:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:00:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:00:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:17:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:22:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:22:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:28:07] 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, and 2 others: Prepare EventLogging for temp accounts - https://phabricator.wikimedia.org/T374812#10964287 (10Milimetric) p:05Triage→03High [15:28:09] 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, and 2 others: Prepare EventLogging for temp accounts - https://phabricator.wikimedia.org/T374812#10964288 (10Milimetric) [15:31:40] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to airflow-an and statboxes for htriedman - https://phabricator.wikimedia.org/T398075#10964316 (10Htriedman) 05In progress→03Resolved @Clement_Goubert just verified that I can get into stat10XX and an-airflo... [15:33:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:33:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:39:38] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#10964348 (10Milimetric) [15:39:55] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#10964353 (10Milimetric) p:05Triage→03Medium [15:40:24] 06Data-Engineering, 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream - https://phabricator.wikimedia.org/T396359#10964354 (10Milimetric) [15:43:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:18:56] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10EventStreams: Figure out how Eventstreams connected client metrics went negative - https://phabricator.wikimedia.org/T398325#10964511 (10Ahoelzl) [16:47:12] 06Data-Engineering, 10Datasets-General-or-Unknown, 10Pageviews-API: Pageviews / Mostread Data not available since June 28, 2025 - https://phabricator.wikimedia.org/T398150#10964672 (10Dusan_Krehel) Pageviews are correct now. [16:47:44] 06Data-Engineering, 10Datasets-General-or-Unknown, 10Pageviews-API: Pageviews / Mostread Data not available since June 28, 2025 - https://phabricator.wikimedia.org/T398150#10964675 (10Dusan_Krehel) 05Open→03Resolved [19:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [20:01:37] 06Data-Engineering, 07Epic: [Idea] Collect pageview data using client-side instrumentation - https://phabricator.wikimedia.org/T371321#10965227 (10Ahoelzl) [23:18:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem