[09:38:14] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9793627 (10gmodena) > F... [10:06:49] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9793797 (10gmodena) >>! In T351117#9781136, @Ottomata wrote: >> adopt topic names that follow EP conventions: . 10Quarry: quarry.wmcloud.org POST request to /api/query/stop does not work - https://phabricator.wikimedia.org/T364835 (10Oudedutchman) 03NEW [10:21:22] (03PS9) 10Gmodena: refinery-job: add webrequest instrumentation. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) [10:21:41] 10Quarry: quarry.wmcloud.org POST request to /api/query/stop does not work with queued queries - https://phabricator.wikimedia.org/T364835#9793876 (10Oudedutchman) [10:24:39] 10Quarry: quarry.wmcloud.org POST request to /api/query/stop does not work with queued queries - https://phabricator.wikimedia.org/T364835#9793884 (10Oudedutchman) →14Duplicate dup:03T362213 [10:25:05] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9793882 (10Oudedutchman) [10:46:37] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 10 others: Upgrade mobileapps to node 18 - https://phabricator.wikimedia.org/T363168#9793941 (10Dibohwendy377) 8141803742 opay Wendy chinasa Diboh [11:26:31] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9794050 (10Ladsgroup) >>! In T120242#9792008, @Ottomata wrote: >> So there is no real "source of truth". > >> So it is quite possible that we might even... [11:52:53] !log re-running refine_eventlogging_legacy for `event`.`centralnoticeimpression` /wmf/data/event/centralnoticeimpression/year=2024/month=5/day=13/hour=18 [11:52:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:03:40] 06Data-Engineering, 10Data-Platform (Data Platform Ops Week Working Group), 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Migrate data-engineering-alerts email list from Mailman to a Google Group - https://phabricator.wikimedia.org/T364632#9794182 (10CodeReviewBot) btullis merged https... [12:43:49] (03PS1) 10Gmodena: medaiwkihistory: typesafe access to compliance value. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031433 [12:46:30] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9794313 (10Fabfur) >>! In T351117#9793797, @gmodena wrote: >>>! In T351117#9781136, @Ottomata wrote: >>> adopt topic names th... [12:54:51] (03CR) 10Gmodena: Refine DeequColumnAnalysis code (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 (owner: 10Snwachukwu) [12:58:15] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9794349 (10gmodena) >>! In T351117#9794313, @Fabfur wrote: >>>! In T351117#9793797, @gmodena wrote: >>>>! In T351117#9781136,... [13:13:18] hi folks. we had a page yesterday that we believe was driven by an increase in internal traffic and it seems like the host was an-worker1165.eqiad.wmnet, 10.64.157.4 [13:13:28] https://w.wiki/A4ym [13:14:21] any thoughts on what could have caused this to happen? cdanis pointed out it was an analytics host and we were waiting for network_flows_internal to catch up and the graph shows the host in question [14:52:24] re: the same topic above: we noticed that there were kafka-jumbo restarts around the same time. we also noticed the traffic was to port 50010 which shows up in puppet as hadoop datanode-data. [14:52:58] on wikitech kafka and hadoop data imports "via gobblin" are mentioned and we were wondering if there could be any relation between these things. [14:53:11] thanks for the additional context mutante [14:53:46] briefly talked to Ryan about it and he pointed out we can check if the gobblin run matches (https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&forceLogin=&var-gobblin_job_name=event_default&var-kafka_topic=All&refresh=15m) [14:53:56] but it was 20 minutes off [15:07:21] 07Analytics-Data-Problem, 06Data Products, 06Data-Platform: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872 (10MNeisler) 03NEW [15:08:21] 07Analytics-Data-Problem, 06Data Products, 06Data-Platform: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872#9795149 (10MNeisler) [16:22:01] (03PS1) 10Btullis: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031506 [16:22:52] I am about to deploy refinery-source: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1031506 [16:22:59] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9795826 (10daniel) >>! In T120242#9792008, @Ottomata wrote: > IMO this is also the level of data inconsistency we should aim for in externalized state as... [16:28:22] 06Data-Engineering, 06Product-Analytics, 06Trust and Safety Product Team: Distinguish between types of block events in the Mediawiki user history table - https://phabricator.wikimedia.org/T213583#9795886 (10TAdeleye_WMF) [16:28:38] 06Data-Engineering, 06Product-Analytics, 06Trust and Safety Product Team: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627#9795888 (10TAdeleye_WMF) [16:31:24] (03PS2) 10Btullis: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031506 [16:33:08] (03CR) 10Joal: "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031506 (owner: 10Btullis) [16:52:31] (03CR) 10Btullis: [C:03+2] Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031506 (owner: 10Btullis) [16:53:40] 06Data-Engineering, 06Data-Platform: Add MW table 'cu_log' to data lake - https://phabricator.wikimedia.org/T364398#9796203 (10lbowmaker) [16:59:18] btullis: any insight on sukhe's question? [16:59:52] I realize that I should have probably filed a task. sorry, I can do that shortly [17:00:55] Ah, sorry. Got distracted and forgot to reply. Checking now. First guess is that gobblin is likely, since it loads from Kafaka jumbo into those datanode ports on HDFS. But let me check the timing. [17:01:11] btullis: thanks and np, let me know if I should put this in a task, happy to do so after a meeting [17:05:20] sukhe: Yes, I think it's worth a task please, if that's OK. an-worker1165 seems pretty inaccessible over SSH right now, although the graphs seem to be OK. [17:05:59] I see thanks, will do soh [17:06:00] *so [17:06:46] That gobblin dashboard isn't the easiest to work with either :-) It's pretty impenetrable unless you're trying to focus in on a certain import. [17:08:18] > an-worker1165 seems pretty inaccessible over SSH right now <--- scratch that. That was a layer 8 problem. [17:08:30] 07Analytics-Data-Problem, 06Data Products, 06Data-Platform, 06Movement-Insights: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872#9796296 (10Mayakp.wiki) [17:11:30] (03Merged) 10jenkins-bot: Updating changelog to prepare next deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031506 (owner: 10Btullis) [17:13:09] Starting build #6 for job analytics-refinery-maven-release [17:24:51] sukhe: I'm struggling to find a smoking gun for this spike at the moment, but I will carry on investigating. [17:25:03] btullis: filing the task and will add you to it, thanks [17:34:16] sukhe: Hi! Would you subscribe me too please? I'm interested in following up on this - thank you :) [17:34:51] sure thanks :) [17:38:36] Project analytics-refinery-maven-release build #6: 09SUCCESS in 25 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/6/ [17:39:15] * sukhe finaly getting to it after all delays [19:06:11] (03CR) 10Snwachukwu: medaiwkihistory: typesafe access to compliance value. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031433 (owner: 10Gmodena) [19:06:34] (03CR) 10Snwachukwu: [V:03+2 C:03+2] medaiwkihistory: typesafe access to compliance value. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031433 (owner: 10Gmodena) [19:10:13] (03CR) 10Snwachukwu: Refine DeequColumnAnalysis code (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 (owner: 10Snwachukwu) [19:17:14] (03CR) 10Snwachukwu: [C:03+2] Refine DeequColumnAnalysis code [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 (owner: 10Snwachukwu) [19:17:46] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Refine DeequColumnAnalysis code [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 (owner: 10Snwachukwu) [19:28:20] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9796972 (10Lydia_Pintscher) Just chiming in that it would be really really great to unblock @diego's work as the current model on Wikidata is not good e... [19:34:36] (03Merged) 10jenkins-bot: Refine DeequColumnAnalysis code [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 (owner: 10Snwachukwu) [19:34:36] (03CR) 10CI reject: [V:04-1] medaiwkihistory: typesafe access to compliance value. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031433 (owner: 10Gmodena) [20:07:46] (03CR) 10Snwachukwu: [V:03+1] medaiwkihistory: typesafe access to compliance value. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031433 (owner: 10Gmodena) [20:38:16] Starting build #6 for job analytics-refinery-update-jars [20:39:44] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.40 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1030564 [20:39:44] Project analytics-refinery-update-jars build #6: 09SUCCESS in 1 min 28 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/6/ [20:43:06] (03CR) 10Btullis: [V:03+2 C:03+2] Add refinery-source jars for v0.2.40 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1030564 (owner: 10Maven-release-user) [22:55:54] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 14), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9797895 (10nshahquinn-wmf)