[00:58:34] 10Data-Engineering, 10Platform Engineering Roadmap: Audit/review pageviews test cases - https://phabricator.wikimedia.org/T305502 (10BPirkle) Comparing existing production "per-project" tests to the new ones [[ https://gitlab.wikimedia.org/repos/generated-data-platform/aqs/pageviews/-/blob/main/per_project_tes... [06:53:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [07:05:28] Hi folks - Back I am :) [07:06:39] 10Data-Engineering: Check home/HDFS leftovers of jdl - https://phabricator.wikimedia.org/T306412 (10MoritzMuehlenhoff) [07:39:25] 10Data-Engineering: Check home/HDFS leftovers of keepit-ssh - https://phabricator.wikimedia.org/T306415 (10MoritzMuehlenhoff) [08:23:15] 10Data-Engineering, 10Airflow: Add a new member to the airlfow-dags repository - https://phabricator.wikimedia.org/T305719 (10EChetty) a:03mforns [08:58:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:58:45] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:04:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Research, discuss and decide on DAG/task dependencies VS. success/failure files (Oozie style) - https://phabricator.wikimedia.org/T301568 (10JAllemandou) Adding something to check while looking at DAG/Task dependencies: how those feed into d... [09:18:30] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:18:45] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:23:30] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:36:56] (03CR) 10Btullis: [C: 03+2] Update the location of the datahub war file [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/780924 (https://phabricator.wikimedia.org/T306019) (owner: 10Btullis) [09:40:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:45:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [09:52:32] (03Merged) 10jenkins-bot: Update the location of the datahub war file [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/780924 (https://phabricator.wikimedia.org/T306019) (owner: 10Btullis) [09:58:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [10:03:30] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [10:36:46] (03PS2) 10NOkafor: Added browser_general and interlanguage hql files to /hql Bug: T300025, T302875 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/778550 (https://phabricator.wikimedia.org/T300025) [10:41:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [10:46:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [11:38:36] 10Analytics-Radar, 10Editing-team, 10MediaWiki-Page-editing, 10Platform Engineering, and 2 others: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588 (10Daimona) >>! In T251588#6860385, @daniel wrote: > Codesearch shows 22 extensions that use these hooks: https... [12:19:29] I'm about to try the upgrade of datahub to 0.8.32 again. [12:26:15] :-( Looks like I broke something. Authentication not working. [12:50:19] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing network firmware bnx2x/bnx2x-e2-7.13.21.0.fw in Debian 11 Bullseye - https://phabricator.wikimedia.org/T306148 (10MoritzMuehlenhoff) https://bugzilla.kernel.org/show_bug.cgi?id=215627 is specifically for a bug in 5.15 and doesn't... [13:09:25] Resolved. Datahub 0.8.32 is up and running. [13:14:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Epic, 10Patch-For-Review: Update DataHub to latest version for MVP - https://phabricator.wikimedia.org/T306019 (10BTullis) This is now done. Version 0.8.32 is now rolled out to datahub.wikimedia.org There was an issue with an incompatible... [13:15:31] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations, 10Patch-For-Review: clouddb1021 missing network firmware bnx2x/bnx2x-e2-7.13.21.0.fw in Debian 11 Bullseye - https://phabricator.wikimedia.org/T306148 (10MoritzMuehlenhoff) >>! In T306148#7864028, @MoritzMuehlenhoff wrote: > The case of the tg3 fi... [13:23:59] hello teammm! [13:24:23] * btullis waves frantically :-) [13:24:33] * joal too o//// [13:28:13] * ottomata ( * ^ *) ノシ [13:28:40] 10Data-Engineering, 10Airflow: Add a new member to the airlfow-dags repository - https://phabricator.wikimedia.org/T305719 (10mforns) Hey @bmansurov ! I added you as a repository developer. If that permit level is not enough, please let me know. Cheers! [13:50:03] :D [13:58:22] mforns: o/ i'm following https://phabricator.wikimedia.org/T305591 at about 60% right now, whatcha think? [13:58:30] also ping milimetric (but i think he's out?) [14:09:40] 10Data-Engineering, 10Generated Data Platform, 10SRE, 10SRE-Access-Requests: Request to grant cparle and mfossati login to an-airflow1003.eqiad.wmne - https://phabricator.wikimedia.org/T306057 (10Ottomata) > I could just add them directly to the analytics-platform-eng-admins I'm going to choose this optio... [14:13:17] (03CR) 10Ottomata: Image Suggestions feature schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/779052 (owner: 10Luke Bowmaker) [14:14:01] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:15:33] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Deploy an-test-coord1002 to facilitate failover testing of analytics coordinator role - https://phabricator.wikimedia.org/T287864 (10JArguello-WMF) a:05razzi→03BTullis Ben asked for the old se... [14:24:00] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:29:50] (03CR) 10Ottomata: "I am so happy that these are being migrated!!! And as new instrumentations too. <3" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (https://phabricator.wikimedia.org/T306385) (owner: 10Sharvaniharan) [14:31:36] ottomata: looking into https://phabricator.wikimedia.org/T305591 [14:38:28] ty let me know how to help [14:40:09] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Product-Analytics, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17): Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [14:58:52] (03PS3) 10NOkafor: Changed usage from hive to spark2-sql Bug: T300025, T302875 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/778550 (https://phabricator.wikimedia.org/T300025) [15:16:21] (03PS4) 10Mforns: Changed usage from hive to spark2-sql Bug: T300025, T302875 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/778550 (https://phabricator.wikimedia.org/T300025) (owner: 10NOkafor) [15:17:37] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/778550 (https://phabricator.wikimedia.org/T300025) (owner: 10NOkafor) [15:20:43] hey a-team, nothing in train etherpad, no train today? [15:20:48] https://etherpad.wikimedia.org/p/analytics-weekly-train [15:21:04] ottomata: we just merged some stuff in refienry [15:21:06] (03Abandoned) 10NOkafor: update usage from hive to spark2-sql Bug: T300025, T302875 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/779565 (https://phabricator.wikimedia.org/T300025) (owner: 10NOkafor) [15:21:40] but it's a no-op, so it would be just a refinery deploy, no-source, no bump ups, no restarts. [15:21:51] ottomata: will add it to the etherpad! [15:21:59] (03Abandoned) 10NOkafor: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/refinery Change-Id: If55278116320fb91c2475cb6d0fecd93048af221 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/779564 (owner: 10NOkafor) [15:22:11] (03Abandoned) 10NOkafor: updated usage from hive to spark2-sql Bug T300025, T302875 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/779523 (owner: 10NOkafor) [15:22:12] ok ty [15:22:25] (03Abandoned) 10NOkafor: updated usage comment [analytics/refinery] - 10https://gerrit.wikimedia.org/r/779521 (owner: 10NOkafor) [15:23:03] (03Abandoned) 10NOkafor: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/refinery [analytics/refinery] - 10https://gerrit.wikimedia.org/r/779522 (owner: 10NOkafor) [15:37:00] ok mforns just a regular refinery deploy, no refinery source? [15:37:13] yes [15:37:30] no refinery-source needed [15:39:17] mforns: btw am looking for a review of https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/17 :) [15:39:50] would be happy to give a walkthrogh first if that would be helpful [15:40:12] ottomata: ok! yes, I think walkthrough would help! let me know when's good for you! [15:40:17] now? [15:41:59] mforns: ^ [15:42:15] ottomata: ok! [15:42:19] bc! [15:42:21] ya [16:15:54] 10Data-Engineering-Kanban, 10Data-Engineering-Radar, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade clouddb* hosts to Bullseye - https://phabricator.wikimedia.org/T299480 (10Ottomata) a:03razzi [16:22:30] 10Data-Engineering, 10Anti-Harassment, 10Product-Analytics: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627 (10Ottomata) [16:38:20] wow an-launcher1002 is very busy with airflow dag stufff? [16:44:01] 10Data-Engineering, 10Anti-Harassment, 10Product-Analytics: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627 (10Ottomata) Hello! In case you aren't aware, there is a `event.mediawiki_user_blocks_change` Hive table that comes from the mediawiki.user-blocks-change stream... [17:19:28] (03PS1) 10Tsevener: Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 [17:19:29] 10Analytics-Clusters, 10Data-Engineering, 10Product-Analytics, 10Superset, 10Patch-For-Review: Add superset-next.wikimedia.org domain for superset staging - https://phabricator.wikimedia.org/T275575 (10mpopov) [17:19:57] 10Data-Engineering, 10Data-Engineering-Kanban, 10Superset, 10Patch-For-Review, 10Product-Analytics (Kanban): Upgrade Superset to 1.4.2 - https://phabricator.wikimedia.org/T304972 (10kzimmerman) Per Mikhail's comment, we're blocked on testing pending the new task of T275575. [17:20:06] (03CR) 10jerkins-bot: [V: 04-1] Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 (owner: 10Tsevener) [17:59:45] 10Data-Engineering, 10Data-Engineering-Kanban: Add the commons-entity dataset to the refinery-drop-mediawiki-snapshots script - https://phabricator.wikimedia.org/T303993 (10JAllemandou) [17:59:47] 10Data-Engineering: Error with refinery-drop-mediawiki-snapshots: table specs not matching partitions for wmf/wikidata/entity and wmf/wikidata/item_page_link - https://phabricator.wikimedia.org/T305591 (10JAllemandou) [18:00:04] mforns: do you have a minute now? [18:00:34] actually mforns - I'll ping you later - I only have 15mins [18:00:53] oh joal how you prefer, I'll be here later too [18:05:52] 10Analytics, 10API Platform: AQS 2.0 local tests fail when mwcli is running - https://phabricator.wikimedia.org/T304735 (10BPirkle) FWIW, here are the commands I executed to run tests, in order: make -C docker startup (leave this running) make -C docker bootstrap (this runs and quits) ./pageviews (leave this... [18:07:08] mforns: ok let's do it now :) [18:07:25] joal: ok! bc! [18:09:59] 10Data-Engineering: Error with refinery-drop-mediawiki-snapshots: table specs not matching partitions for wmf/wikidata/entity and wmf/wikidata/item_page_link - https://phabricator.wikimedia.org/T305591 (10JAllemandou) [18:10:01] 10Data-Engineering, 10Data-Engineering-Kanban: Refactor refinery-drop-mediawiki-snapshots so that it no longer uses a _SUCCESS file - https://phabricator.wikimedia.org/T303988 (10JAllemandou) [18:10:03] 10Data-Engineering, 10Data-Engineering-Kanban: Add the commons-entity dataset to the refinery-drop-mediawiki-snapshots script - https://phabricator.wikimedia.org/T303993 (10JAllemandou) [18:16:03] 10Data-Engineering: Error with refinery-drop-mediawiki-snapshots: table specs not matching partitions for wmf/wikidata/entity and wmf/wikidata/item_page_link - https://phabricator.wikimedia.org/T305591 (10mforns) I think the problem here is that the migrated Airflow job does not generate _SUCCESS files after the... [18:54:09] (03PS12) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [18:55:20] (03CR) 10Sharvaniharan: "Thank you for the review @Ottomata!" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [18:58:19] (03CR) 10Ottomata: [C: 03+1] "Okay great, +1 otherwise, the rest is up to you. Merge at will. Thank you!" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [19:05:59] 10Data-Engineering, 10Platform Engineering Roadmap: Audit/review pageviews test cases - https://phabricator.wikimedia.org/T305502 (10Eevans) >>! In T305502#7862830, @BPirkle wrote: > Comparing existing production "per-project" tests to the new ones [[ https://gitlab.wikimedia.org/repos/generated-data-platform/... [19:16:56] mforns: i'm trying to wrap up a bunch of little loose ends in various projects around the automated release stuff. would you be okay if i merged now and we handle changes in post merge review? can always make changes! [19:27:06] ottomata: sure, I will review the code nevertheless [19:27:13] okay, thank you. [20:12:48] (03PS2) 10Tsevener: Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 (https://phabricator.wikimedia.org/T290920) [20:24:32] (03PS3) 10Tsevener: Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 (https://phabricator.wikimedia.org/T290920) [20:41:58] (03PS4) 10Tsevener: Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 (https://phabricator.wikimedia.org/T290920) [20:58:50] (03PS5) 10Tsevener: Modify ios_notification_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/784293 (https://phabricator.wikimedia.org/T290920) [21:14:24] 10Analytics-Wikistats, 10Data-Engineering: Make it clear that WiViVi data is outdated - https://phabricator.wikimedia.org/T306478 (10Pcoombe) [22:33:15] 10Data-Engineering, 10Product-Analytics: Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10mpopov) p:05Triage→03High [23:40:31] 10Analytics-Radar, 10Fundraising-Backlog, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10sgrabarczuk)