[04:57:34] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10257611 (10ABran-WMF) [09:02:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Bump eventutilities to support flink 1.20 - https://phabricator.wikimedia.org/T377130#10257974 (10dcausse) >>! In T377130#10241834, @gmodena wrote: >> I've updated (n... [10:14:20] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Update eventutilities_python wrappers to support Flink 1.20 - https://phabricator.wikimedia.org/T374359#10258155 (10gmodena) >>! In T374359#10254572, @Ottomata wrote: > Are we still hoping to merge inclu... [11:15:41] FIRING: MediawikiPageContentChangeEnrichAvailability: ... [11:15:47] Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [11:25:41] RESOLVED: MediawikiPageContentChangeEnrichAvailability: ... [11:25:41] Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [12:16:34] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10258422 (10ABran-WMF) @Ladsgroup for those 2 edge cases: P70585 I guess I could pre depool the nodes to let them drain... [12:17:25] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10258424 (10ABran-WMF) 05In progress→03Stalled [14:04:58] (03CR) 10Gmodena: [V:03+2 C:03+2] static_data: add annwiki to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082330 (https://phabricator.wikimedia.org/T376332) (owner: 10Gmodena) [14:05:24] !log starting a refinery deployment to hotfix https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1082330 [14:05:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:31:57] !log refinery deployment completed [14:32:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:48:26] (03CR) 10Milimetric: [C:03+2] "just a style nit: looks like you have some trailing spaces:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082515 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [14:50:31] 14Analytics, 06Data-Engineering-Icebox, 06cloud-services-team, 10Data-Services: Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users - https://phabricator.wikimedia.org/T204950#10259085 (10taavi) 05Open→03Declined That seems unnecessarily complicated... [15:17:21] 06Data-Engineering, 06Growth-Team, 10MediaWiki-extensions-EventLogging, 10Notifications, and 3 others: [EPIC] Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#10259248 (10phuedx) [15:17:36] 06Data-Engineering, 06Data Products, 10MediaWiki-extensions-EventLogging, 10Metrics Platform: Remove deprecated $logger parameter from EventLogging::submit() - https://phabricator.wikimedia.org/T376410#10259256 (10phuedx) [15:18:00] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Data Products (Epics Timeline), 07Epic: [EPIC] Deprecate mw.eventLog.logEvent() - https://phabricator.wikimedia.org/T317874#10259250 (10phuedx) [15:18:38] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Data Products (Epics Timeline), 07Epic: [EPIC] Deprecate EventLogging::schemaValidate() - https://phabricator.wikimedia.org/T317793#10259255 (10phuedx) [15:31:36] 06Data-Engineering-Icebox: Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users - https://phabricator.wikimedia.org/T204950#10259320 (10Ottomata) 05Declined→03Open I'd like to keep this open, but I will remove the Data-Services tag. It is something we would re... [15:36:50] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Bump eventutilities to support flink 1.20 - https://phabricator.wikimedia.org/T377130#10259411 (10gmodena) >>! In T377130#10241164, @gmodena wrote: > > Good catch! I... [15:45:48] !log Deploy latest DAGs to analytics Airflow instance. T377999. [15:46:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:46:41] T377999: Run Dumps 2.0 main DAG at a daily cadence rather than hourly. - https://phabricator.wikimedia.org/T377999 [16:11:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.10.19 - 2024.11.08): Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10259599 (10Ottomata) [16:11:29] 10Data-Engineering (Q2 2024 October 1st - December 31th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#10259600 (10Ottomata) [16:25:23] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10259648 (10tchin) [16:38:57] 06Data-Engineering, 10Structured-Data-Backlog (Current Work): [L] Track commons deletion requests - https://phabricator.wikimedia.org/T370898#10259689 (10xcollazo) >>! In T370898#10254614, @Ottomata wrote: > I'll let @xcollazo confirm, but I'm suggesting to use the new Dumps 2 (not yet production ready, but ve... [17:41:48] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE, 06SRE, 13Patch-For-Review: Streamline Data Platform access approvals for WMF staff - https://phabricator.wikimedia.org/T370424#10259980 (10Ottomata) Updated docs: https://wikitech.wikimedia.org/w/index.php?title=SRE%2FProduction... [17:43:34] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 07Spike: [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors - https://phabricator.wikimedia.org/T360922#10259987 (10xcollazo) 05Open→03Resolved [18:02:53] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, 10Event-Platform, and 3 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10260100 (10mpopov) @Ottomata asked me to note that for Product Analytics it would be really he... [19:48:02] 06Data-Engineering, 10CampaignEvents, 06Data Products, 10EntitySchema, and 16 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#10260552 (10cjming) [23:50:16] hi team! I'm a student at the University of Waterloo working on a data science project on wikipedia metadata. Specifically, we'd like to query the analyrics API for view counts. we wrote a python script for this to do this in batches of 1000, but it seems we get rate limited quickly(429 response). Can you help us find a better way to get this data?