[00:21:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11924024 (10Ottomata) [01:03:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:03:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:03:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [01:13:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:13:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:13:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [01:41:42] (03PS1) 10Snwachukwu: Add Sanitizer to clean up wprov value of x-analytics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) [02:02:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:02:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:02:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:12:01] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:12:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:12:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:27:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424350#11924142 (10xcollazo) [02:37:01] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:37:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:37:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:42:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:42:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:42:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [08:58:28] 06Data-Engineering, 10Pageviews-API, 10Tool-Pageviews: Mediaviews Analysis returns API not found error - https://phabricator.wikimedia.org/T426373#11924484 (10Aklapper) I don't see how this is related to RestBase? Why were all those folks added as task subscribers? [09:08:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#11924533 (10JMonton-WMF) Is the plan to backfill the data in batch, or to fo... [11:19:58] 06Data-Engineering: Make backfills resistant to Airflow restarts. - https://phabricator.wikimedia.org/T426398 (10GGoncalves-WMF) 03NEW [11:21:25] 06Data-Engineering: Make bot detection backfills resistant to Airflow restarts. - https://phabricator.wikimedia.org/T426398#11925018 (10GGoncalves-WMF) [14:10:47] (03CR) 10JavierMonton: [C:03+1] "The code looks good to me, but I don't know much about `wprov`. Just wondering if `wprov` could be empty, like `wprov=`. If that's a valid" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) (owner: 10Snwachukwu) [14:22:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): table_maintenance_iceberg_monthly permission issue fails task due to permission on Ivy cache artifact - https://phabricator.wikimedia.org/T418804#11925663 (10xcollazo) 05Open→03Resolved This only fixed table maintenance. The other jobs that also used... [14:24:18] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: wmf.mediawiki_history contains spurious event_type = 'create-page' for page entity rows - https://phabricator.wikimedia.org/T426242#11925672 (10xcollazo) 05Open→03Resolved [14:25:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Perform a one-time clean up of retained data sets in event_sanitize - https://phabricator.wikimedia.org/T417694#11925684 (10xcollazo) 05In progress→03Resolved [14:26:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add dbt base model for Wikipedia moderator actions metrics - https://phabricator.wikimedia.org/T423565#11925686 (10xcollazo) 05In progress→03Resolved [14:26:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Architectural design agreement: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424359#11925688 (10xcollazo) 05In progress→03Resolved [14:28:13] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11925694 (10xcollazo) @AKhatun_WMF doe the WARNINGs continue? [14:31:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE1.5 Consult on monthly active moderators data lake pipeline - https://phabricator.wikimedia.org/T419584#11925710 (10xcollazo) 05Open→03Resolved The consulting porting of this work is done. Work artifacts at T423565. [14:33:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Research: MediaWiki content history dataset issues - https://phabricator.wikimedia.org/T415311#11925716 (10xcollazo) ( Just checking in here to mention that, of the tasks on description, only T400632 remains, and @APizzata-WMF is working on it. ) [14:40:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 13Patch-For-Review: Archiva (archiva.wikimedia.org) returning HTTP 504 Gateway Timeout or no responses, breaking production Maven builds - https://phabricator.wikimedia.org/T426114#11925735 (10xcollazo) [14:44:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Relative Trending - Flink app for page_view - https://phabricator.wikimedia.org/T425624#11925747 (10JMonton-WMF) [14:48:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Flink page_view: Create docker build - https://phabricator.wikimedia.org/T426419 (10JMonton-WMF) 03NEW [14:58:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11925783 (10AKhatun_WMF) @xcollazo Yes, we still have very frequent warnings. https://airflow.wikimedia.org/dags/refine_webrequest_hourly_text... [15:18:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Relative Trending - Flink app for page_view - https://phabricator.wikimedia.org/T425624#11925841 (10JMonton-WMF) [15:29:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06SRE, 10Event-Platform: Flink Page View: Create K8s resources - https://phabricator.wikimedia.org/T426425 (10JMonton-WMF) 03NEW [15:29:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11925926 (10xcollazo) >Run mediawiky history. A typical run takes ~10.6h at 90 executors. We are using 128 below (~25... [15:30:08] 06Data-Engineering, 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426 (10Ottomata) 03NEW [15:32:57] 06Data-Engineering, 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11925954 (10Ottomata) [15:38:29] 06Data-Engineering, 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11925991 (10Ottomata) I gave up my root access a few months ago, so I log in and query cassandra directly anymore to see if the data is there. I can't find any page-analytics AQS s... [15:41:22] 06Data-Engineering, 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11926001 (10Ottomata) Oh, Eric commented [[ https://wikimedia.slack.com/archives/CSV483812/p1778858560416059?thread_ts=1778856943.557099&cid=CSV483812 | in Slack ]]. Investigating... [15:43:57] 06Data-Engineering, 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11926006 (10Ottomata) So, the data is not in Cassandra. That means the [[ https://airflow.wikimedia.org/dags/cassandra_load_pageview_top_pages_per_editor_monthly/grid?base_date=202... [15:45:33] (03PS1) 10Snwachukwu: Use SanitizeXAnalyticsWprovUDF to normalize x_analytics[wprov] values [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287909 (https://phabricator.wikimedia.org/T425787) [15:46:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11926026 (10Ahoelzl) p:05Triage→03High [15:46:46] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11926044 (10Ahoelzl) a:03amastilovic [15:51:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11926064 (10xcollazo) >>! In T425986#11925926, @xcollazo wrote: >>Run mediawiky history. A typical run takes ~10.6h a... [15:56:14] (03CR) 10Snwachukwu: "yes it could be a valid scenario. I'll add that test case thanks." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) (owner: 10Snwachukwu) [16:00:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11926111 (10diego) Adding context from the time-to-revert metric T424713, since the discussion t... [16:02:52] (03PS2) 10Snwachukwu: Add Sanitizer to clean up wprov value of x-analytics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) [16:08:22] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287909 (https://phabricator.wikimedia.org/T425787) (owner: 10Snwachukwu) [16:26:06] (03CR) 10Snwachukwu: [C:03+2] Add Sanitizer to clean up wprov value of x-analytics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) (owner: 10Snwachukwu) [16:30:23] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add Sanitizer to clean up wprov value of x-analytics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287508 (https://phabricator.wikimedia.org/T425787) (owner: 10Snwachukwu) [17:27:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11926498 (10mforns) Looking into this! [17:32:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11926512 (10xcollazo) > Thus will rerun with same parameters as in T425986#11923202. Unfortunately, we will have to w... [17:46:38] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06Wikimedia Enterprise: Include more namespaces in Wiktionary HTML dumps - https://phabricator.wikimedia.org/T303652#11926580 (10JArguello-WMF) 05Open→03Invalid [17:48:50] 06Data-Engineering, 10Pageviews-API, 10Tool-Pageviews: Mediaviews Analysis returns API not found error - https://phabricator.wikimedia.org/T426373#11926630 (10MusikAnimal) I'm told this may be related to T426426. Either way, Data Engineering is looking into it! Thanks for filing the task. [17:52:12] 06Data-Engineering, 10Dumps-Generation, 06Wikimedia Enterprise: Request: changelog for Enterprise API HTML dumps - https://phabricator.wikimedia.org/T348100#11926692 (10JArguello-WMF) 05Open→03Invalid [17:52:16] 06Data-Engineering, 10Dumps-Generation: {Investigation} Different file sizes for dumps - https://phabricator.wikimedia.org/T345176#11926696 (10JArguello-WMF) [17:52:58] 06Data-Engineering, 10Pageviews-API, 10Tool-Pageviews: Mediaviews Analysis returns API not found error - https://phabricator.wikimedia.org/T426373#11926709 (10MusikAnimal) [17:54:36] (03PS1) 10TChin: Update changelog.md for v0.3.14 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287926 [17:57:44] (03CR) 10Snwachukwu: [C:03+2] Update changelog.md for v0.3.14 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287926 (owner: 10TChin) [18:00:06] Starting build #59 for job analytics-refinery-maven-release [18:06:13] (03CR) 10Xcollazo: [C:03+1] change create table for mediawiki_content to become private (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1285337 (https://phabricator.wikimedia.org/T424355) (owner: 10A-pizzata) [18:09:19] (03Merged) 10jenkins-bot: Update changelog.md for v0.3.14 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1287926 (owner: 10TChin) [18:12:36] Project analytics-refinery-maven-release build #59: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/59/ [18:40:29] 06Data-Engineering, 10Pageviews-API, 10Tool-Pageviews: Mediaviews Analysis returns API not found error - https://phabricator.wikimedia.org/T426373#11926823 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Fixed! For the curious, this was due to upstream changes to the imageinfo API (T414338). [19:32:13] !log Test Kitchen edge-unique experiments (poll 1) - adds: logged-out-retention-round11, we-1-8-account-creation-form-v1, we-1-8-mobile-account-menu, logged-out-retention-round9, logged-out-retention-round8, growthexperiments-editattempt-anonwarning, mobile-page-previews, logged-out-retention-round10; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [19:32:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:32:23] !log Test Kitchen mw-user experiment (poll 1) - adds: fy25-26-we-1-7-8-suggestion-mode-beta, growthexperiments-revise-tone, incident_reporting_system_interaction, ab-test-email-confirmation-banner, we-1-10-articleguidance-v1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [19:32:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:36:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11927041 (10xcollazo) Maybe we should double all thresholds. [20:08:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.2; 2026-05-12): Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11927110 (10xcollazo) >>! In T423952#11915434, @Ottomata wrote: > W... [20:38:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11927168 (10Isaac) @diego I looked through the report/ticket but didn't see anywhere that someone... [20:50:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Spike: full-history revert detection for `mediawiki_history_incremental_v1` - https://phabricator.wikimedia.org/T426469 (10xcollazo) 03NEW [20:54:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: AQS pageviews/v3/top_pages_per_editor not returning data - https://phabricator.wikimedia.org/T426426#11927206 (10Ahoelzl) Data is available now in Cassandra / API ` % curl -X GET "https://wikimedia.org/api/rest_v1/metrics/pageviews/v3/top_pages_p... [23:27:34] (03PS7) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729)