[02:23:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:23:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:23:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [02:28:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:28:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:28:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [02:31:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:31:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:31:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [02:36:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:36:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:36:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [08:33:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type Enrichment: Update flink code - https://phabricator.wikimedia.org/T424231#11859498 (10JMonton-WMF) > Wondering if we should handle these events, or letting them go to error sink is fine? I also think we should pass them along w... [08:40:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Update stream to use new deployment config - https://phabricator.wikimedia.org/T424223#11859578 (10brouberol) The application is now running ` brouberol@deploy1003:~$ kube-env mw-page-html-feature-counts-change-enri... [09:14:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11859842 (10daniel) p:05Triage→03High Bumping to high. With the ratelimit rollout happening this week, we really need this to investigate is... [09:57:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860157 (10brouberol) ` brouberol@deploy1003:~$ kenv mw-page-edit-type-enrich-next brouberol@deploy1003:~$ helm list WARNING: Kubernetes configuration file is group-... [09:57:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860158 (10brouberol) [09:58:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860179 (10brouberol) ` root@deploy1003:~# kubectl delete namespace mw-page-edit-type-enrich-next namespace "mw-page-edit-type-enrich-next" deleted ` (I had to delet... [09:58:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860180 (10brouberol) [10:00:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860192 (10brouberol) [10:17:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860252 (10brouberol) [10:20:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860284 (10brouberol) ` brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --regi... [10:29:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860337 (10brouberol) ` brouberol@cephosd1001:~$ sudo radosgw-admin user rm --uid=mw-page-edit-type-enrich-next brouberol@cephosd1001:~$ sudo r... [10:29:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860338 (10brouberol) [10:30:06] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11860339 (10brouberol) back to you @AKhatun_WMF [10:31:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Update stream to use new deployment config - https://phabricator.wikimedia.org/T424223#11860347 (10brouberol) [13:21:30] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11860894 (10brouberol) [13:30:07] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11860935 (10brouberol) ` brouberol@krb1002:~$ sudo manage_principals.py create suecarmol --email=s... [13:30:16] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11860938 (10brouberol) 05Open→03Resolved [13:30:33] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11860940 (10brouberol) @Scardenasmolinar you should have received an email with your kerberos cred... [13:35:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11860984 (10Snwachukwu) @Ladsgroup, your assumption is correct indeed. We started running it from 1st April that so I don't expect any data older than... [13:54:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11861119 (10Ladsgroup) Thank you! [14:02:48] 06Data-Engineering, 06Data-Platform-SRE, 10observability, 06serviceops-radar, and 4 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#11861150 (10brouberol) a:03brouberol [14:03:07] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524 (10saper) 03NEW [14:07:43] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Requesting Kerberos access for SCardenas (WMF) - https://phabricator.wikimedia.org/T418664#11861221 (10Scardenasmolinar) >>! In T418664#11860940, @brouberol wrote: > @Scardenasmolinar you should have received an... [14:17:16] (03CR) 10Milimetric: "I guess when we join to this we could define new namespaces as "other" by default. But I would expect this instrumentation to get turned " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1276803 (https://phabricator.wikimedia.org/T417050) (owner: 10Milimetric) [14:30:13] !log Test Kitchen edge-unique experiments (poll 166190) - adds: none; removes: logged-out-retention-round6; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:33:44] (03CR) 10Ottomata: "Will https://phabricator.wikimedia.org/T409462 help at all?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1276803 (https://phabricator.wikimedia.org/T417050) (owner: 10Milimetric) [14:36:06] (03CR) 10Milimetric: "yeah, it's not there, this is a manual arbitrary grouping for this analysis. The page_type is cool, and I'd encourage you to reach out to" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1276803 (https://phabricator.wikimedia.org/T417050) (owner: 10Milimetric) [14:44:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Incremental MediaWiki History - https://phabricator.wikimedia.org/T424350#11861514 (10Ottomata) [14:44:49] 14Analytics, 06Product-Analytics, 07Epic, 13Patch-For-Review: Data Lake incremental Data Updates - https://phabricator.wikimedia.org/T258511#11861515 (10Ottomata) [14:48:01] 06Data-Engineering, 06Data-Engineering-Radar, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Ad-hoc EventBus event submissions should be batched - https://phabricator.wikimedia.org/T382970#11861553 (10Ottomata) There is one unmerged JobQueue related patch. It is a JobQueue optimization that allows... [15:12:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Reader Experience Team, 10Test Kitchen, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31): Logged in reader retention logging - https://phabricator.wikimedia.org/T420621#11861682 (10tchin) Forgot to add an update from the Dublin offsite, there's now the `c... [15:31:18] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#11861789 (10bd808) [15:36:41] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#11861819 (10bd808) It isn't exactly the same, but this task does seem to rhyme with {T173511} where we were trying to figure out a sust... [15:41:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Essential-Work: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11861859 (... [16:02:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Add timeout - https://phabricator.wikimedia.org/T424547 (10AKhatun_WMF) 03NEW [16:11:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Alerting - https://phabricator.wikimedia.org/T423996#11862008 (10JMonton-WMF) [16:12:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11862022 (10AKhatun_WMF) [16:12:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type enrichment: Add timeout - https://phabricator.wikimedia.org/T424547#11862021 (10AKhatun_WMF) [16:13:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Debug edit type pipeline for production readiness - https://phabricator.wikimedia.org/T421026#11862030 (10AKhatun_WMF) [16:17:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Edit type Enrichment: Update documentation - https://phabricator.wikimedia.org/T424225#11862069 (10AKhatun_WMF) a:05JMonton-WMF→03AKhatun_WMF [16:27:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11862115 (10JMonton-WMF) I think we have some real metrics now. The application has been ru... [16:30:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11862142 (10AKhatun_WMF) [16:35:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11862188 (10AKhatun_WMF) [16:36:10] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Delete old edit-type stream - https://phabricator.wikimedia.org/T424364#11862193 (10AKhatun_WMF) [16:39:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11862205 (10AKhatun_WMF) [16:50:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11862255 (10Snwachukwu) >Also, it's weird that it says 60,000 plays also for the last link starting at April 1st. So this can only be right when the da... [17:04:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11862307 (10New_York-air) Hi @Snwachukwu, thank you for looking into it. If its not too much effort this would be great. When presenting the correct n... [18:07:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11862569 (10Ottomata) Met today with MW folks to learn more. [[ https://docs.google.com/document/d/1QZNCZhsBCxEKwogI8S1GFtELTPa0t9DYUBFoc3jI-oo/edit?pl... [20:36:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Sqoop file and filetypes tables from Commonswiki to the Hdfs - https://phabricator.wikimedia.org/T424365#11863116 (10Snwachukwu) 05Open→03Invalid [21:59:53] (03PS1) 10Conniecc1: Add event.wiki in EditAttemptStep to allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1277760 [23:51:39] 06Data-Engineering, 06DBA: Move SpecialWantedCategories computation to Hadoop - https://phabricator.wikimedia.org/T422450#11863741 (10Zabe) [23:54:14] (03PS1) 10Zabe: querypage: Add WantedCategories.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1277789 (https://phabricator.wikimedia.org/T422450)