[01:05:08] (03PS1) 10Lex Nasser: Add time interval limits to pageviews per-articles and mediarequests per-file AQS endpoints [analytics/aqs] - 10https://gerrit.wikimedia.org/r/661827 (https://phabricator.wikimedia.org/T261681) [01:50:01] 10Analytics, 10Patch-For-Review: Add time interval limits to pageview API - https://phabricator.wikimedia.org/T261681 (10lexnasser) Just submitted a patch set to address the first part of this task, enforcing a maximum time interval of 1 year for the pageviews/per-article and mediarequests/per-file endpoints.... [02:09:42] (03PS7) 10Awight: Added editor type preferences [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal) [02:10:39] (03CR) 10Awight: "PS 7: last-minute change, removing a redundant wikEd query." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal) [03:32:27] (03PS8) 10Awight: Added editor type preferences [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal) [03:36:25] (03CR) 10Awight: "PS 8: Optimizations: skip an intermediate query, summarize immediate; no need to case-cast boolean to int; avoid filesort." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/657362 (https://phabricator.wikimedia.org/T271902) (owner: 10Svantje Lilienthal) [03:41:02] (03PS4) 10Awight: Collect metrics of all wikis [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/655886 (https://phabricator.wikimedia.org/T271894) (owner: 10WMDE-Fisch) [03:42:36] (03CR) 10Awight: [C: 04-1] Collect metrics of all wikis (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/655886 (https://phabricator.wikimedia.org/T271894) (owner: 10WMDE-Fisch) [04:42:10] 10Analytics, 10Patch-For-Review: Add time interval limits to pageview API - https://phabricator.wikimedia.org/T261681 (10MusikAnimal) Hello! I'd like to point out that this will surely cause widespread breakage. #tool-pageviews, #XTools, and #event_metrics will all break (when requesting a range >1 year), and... [05:55:49] 10Analytics, 10Patch-For-Review: Add time interval limits to pageview API - https://phabricator.wikimedia.org/T261681 (10lexnasser) @MusikAnimal Thanks so much for pointing this out -- I personally overlooked this. To add, the pageviews/per-article endpoint has stability 'stable', meaning that it is "guarant... [07:03:11] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10EventStreams, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) 05Open→03Resolved [07:36:12] 10Analytics-Radar: Presto error in Superest - only when grouping - https://phabricator.wikimedia.org/T270503 (10elukey) After a bit of digging, the problem was found in Superset, namely @EYener's username was `EYener` instead of `eyener`. This seems to be a little change but since the username is the one that Ha... [08:29:46] joal: o/ missed your ping yesterday, was too late indeed :) [08:29:53] 10Analytics, 10observability, 10User-fgiunchedi: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10fgiunchedi) >>! In T273064#6805181, @CDanis wrote: > I think the change pushed today in the puppet private repo (hash b1b32d4ab) broke the [[ https://wikitech.wikimedia.org... [08:42:30] 10Analytics, 10Analytics-Wikistats: Wikistats New Feature: Bulgarian... - https://phabricator.wikimedia.org/T273678 (10Nikerabbit) 05Open→03Invalid It's not translated into Bulgarian: https://translatewiki.net/wiki/Special:MessageGroupStats?group=wikistats&x=D Feel free to sign up in translatewiki.net and... [09:21:24] 10Analytics: Add a presto query logger - https://phabricator.wikimedia.org/T269832 (10elukey) I tested in Hadoop test the AWS query logger (provided via Apache 2.0 license, very good) and it works nicely with little effort, but it doesn't seem log the query string: ` Feb 05, 2021 9:10:23 AM com.amazonaws.QueryE... [09:24:50] 10Analytics: Add a presto query logger - https://phabricator.wikimedia.org/T269832 (10elukey) Another thing that I have discovered is the Presto UI on the coordinator: {F34092823} This is way more useful, but it doesn't keep history of the queries after a restart. The info about the query execute is there, so... [10:15:18] 10Analytics, 10Patch-For-Review: Decide to move or not to PrestoSQL - https://phabricator.wikimedia.org/T266640 (10elukey) Opened https://github.com/prestodb/presto/issues/15685 [10:17:25] 10Analytics, 10Patch-For-Review: Decide to move or not to PrestoSQL/Trino - https://phabricator.wikimedia.org/T266640 (10elukey) [11:28:07] mforns: hola hola, are you around? [11:31:24] !log restart turnilo to pick up changes to the config (two new attributes to webrequest_128) [11:31:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:36:03] (already done np) [11:39:06] * elukey lunch! [13:49:20] elukey: o/ are kafka-main*.eqiad.wmnet brokers available from hadoop workers? [13:51:57] dcausse: o/ in theory yes, the firewall should allow them [13:52:08] thanks! [13:52:12] but in practice we need to be very careful dcausse [13:52:23] if we hammer kafka main with hadoop we'll be in trouble with SRE :D [13:52:37] sure [13:53:10] I want to get rid of kafka-jumbo for some application data, (so that wdqs1009 does not have to fetch from jumbo) [14:06:39] hm I having troubles with flink swift fs [14:06:50] I suspect some class loading issue [14:07:26] I'm hitting "Could not find a public truncate method on the Hadoop File System" [14:11:56] zpapierski: does this ring a bell? ^ [14:11:59] dcausse: I am wondering if it is a hadoop version issue, IIRC we had something like that in the past [14:12:36] dcausse: https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L181 [14:13:04] elukey: yes but we run 2.6 if I'm not mistaken [14:13:06] does flink need hadoop 2.7? [14:13:14] for some stuff yes [14:13:25] yes yes, I was about to say that on tuesday we'll bump it to 2.10.1 :) [14:13:53] but then I hope the truncate method will be there :) [14:14:16] but here I'm not sure to understand because it's working fine if I don't load the swift plugin [14:14:32] dcausse: yep https://issues.apache.org/jira/browse/FLINK-10203 [14:14:42] it seems that it wants hadoop 2.7 [14:15:29] yes only if you don't use OnCheckpointRollingPolicy [14:15:31] no idea about what's conflicting but it really sounds something that zpapierski asked a while ago [14:16:30] then no idea, but it feels as if the swift plugin makes flink hit a path that requires hadoop 2.7 [14:16:54] no clue :/ [14:17:04] dcausse: btw, on Tuesday morning I'd need airflow stopped for the hadoop upgrade, would it be possible to stop it the evening before? [14:17:20] I'm sure we can [14:17:23] ebernhardson: ^ [14:17:30] it would be great thanks :) [14:30:03] the flink swift plugin declares a lot more than just itself in META-INF/services/ [14:36:41] hmm [14:37:07] I'm trying to remember what I asked about it a while ago :) [14:37:31] I thought it required dependencies, but not hadoop itself [14:39:32] could fix one issue with https://gerrit.wikimedia.org/r/c/wikidata/query/flink-swift-plugin/+/661923 [14:57:27] +2ed [15:08:15] * elukey coffee [15:20:53] dcausse: if you like i can merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/661947/ for you [15:21:37] ottomata: if you don't mind then yes :) [15:24:58] thanks! [15:25:58] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): remove all Growth schemas from the schema whitelist - https://phabricator.wikimedia.org/T273826 (10mpopov) Thanks, @Rileych! The PA team will prioritize this in our next board refinement meeting (Feb 8). [15:26:09] 10Analytics, 10Growth-Scaling, 10Growth-Team, 10Product-Analytics: Growth: delete data older than 90 days - https://phabricator.wikimedia.org/T273821 (10mpopov) Thanks, @Rileych! The PA team will prioritize this in our next board refinement meeting (Feb 8). [15:26:59] 10Analytics, 10Growth-Scaling, 10Growth-Team, 10Product-Analytics: Growth: End wider data purge window - https://phabricator.wikimedia.org/T273815 (10mpopov) Thanks, @Rileych! The PA team will prioritize this in our next board refinement meeting (Feb 8). [15:32:41] 10Analytics, 10Product-Analytics: Provide a list of 100 most popular articles of Russian and English Wikipedias in terms of page views from Ukraine - https://phabricator.wikimedia.org/T273924 (10mpopov) The PA team will discuss and prioritize this at our next board refinement meeting (Feb 8). [15:37:48] hmm, am I doing something wrong? [15:37:56] I'm trying to do something simple,like: [15:38:03] https://www.irccloud.com/pastebin/JkFEOein/ [15:38:18] in Jupyter - and it basically hangs [15:38:29] it can read the schema, I checked [15:49:05] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) [15:53:14] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) [15:53:17] ok, it just takes a long time to produce a result (and show was incorrect), but I can't really understand why [15:57:37] zpapierski: on what stat node are you? [15:57:45] stat1007 [15:58:33] and it seems to change when I limit the dataframe via filtering (but even with 1h dataframe it takes about 10-15s) [15:59:32] maybe that's normal [15:59:49] it's the first time I'm doing that, I don't really know what to expect [16:03:25] well it depends on how much data you are pulling in from hdfs :D [16:03:46] should that be basically next to nothing? I'm going only for the first row [16:04:47] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) @wiki_willy, @hnowlan Move tickets have been created for db1111 (T273982), logstash1020 (T273984) and maps1001 (T273983). Francium has been de... [16:04:52] zpapierski: I'm not familiar with HivePartition, is that some fancy magic or are you just trying to use spark to read into a dataframe and you don't care much about the details? [16:05:15] zpapierski: if you don't filter with a partition you'll read almost all the data [16:06:15] milimetric: HivePartition is a tool Erik wrote to unify the partitioning scheme [16:07:07] dcausse: I'm doing this like that - [16:07:07] spec_fe = HivePartition.from_spec('event.searchsatisfaction/') [16:07:30] I guess I need to add some k=v there [16:07:41] oh, so to prevent you from doing stuff like readPath = s"/wmf/data/event/PrefUpdate/year=*/month=*/day=*/hour=*/*.parquet" etc. [16:07:51] yes [16:07:53] so I'm not sure how to use the magic, but usually to limit you'd be like [16:07:55] how can I know how's this partitioned [16:08:01] ? [16:08:06] readPath = s"/wmf/data/event/PrefUpdate/year=2021/month=2/day=2/hour=10/*.parquet" [16:08:24] e.g. something like HivePartition.from_spec('event.searchsatisfaction/year=2021,month=02,day=01,hour=10') [16:08:29] (sorry in this case) readPath = s"/wmf/data/event/SearchSatisfaction/year=2021/month=2/day=2/hour=10/*.parquet" [16:08:31] hmm, so I'm filtering in the wrong place [16:08:35] not sure about the "," [16:09:50] yeah, it's / [16:10:52] ok, it works, thx - I'm not sure it's any faster, but I'll assume this is basically what I can get [16:11:08] it will make joining on hour boundries difficult, though [16:11:16] use .cache() and keep the dataframe sometimes it helps [16:11:30] will do\ [16:12:47] if you run daily you can perfectly skip the hour criteria but for testing it's generally faster to work with a single partition [16:14:11] true, but honestly, it will be still an issue, I guess - I need to join two tables/dataframes on meta.request_id, I'm afraif that will be a quite taxing operation [16:14:25] but maybe not, we'll see [16:15:22] should be fine, I mean it'll take time but it's somewhat expected [16:22:29] 10Analytics-Clusters, 10Patch-For-Review: Improve logging for HDFS Namenodes - https://phabricator.wikimedia.org/T265126 (10elukey) >>! In T265126#6800421, @razzi wrote: > @Ottomata and I discussed next steps for this ticket, and came up with the following: > > - Create a puppet patch that allows a hiera sett... [16:25:18] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[18-41] - https://phabricator.wikimedia.org/T260445 (10elukey) @Cmjohnson wonderful news! I'll follow up in the task to help the owners of the hosts! [16:28:45] (03PS1) 10Ottomata: Enforce numeric bounds for all schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/661959 (https://phabricator.wikimedia.org/T273069) [16:29:16] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Rematerialize all event schemas with enforceeNumericBounds - https://phabricator.wikimedia.org/T273069 (10Ottomata) [16:30:43] 10Analytics-Clusters, 10Patch-For-Review: Improve logging for HDFS Namenodes - https://phabricator.wikimedia.org/T265126 (10Ottomata) > I'd prefer if we tested properly the whole procedure in Hadoop test. I quickly checked and the /srv partition is probably not really used, and it doesn't contain anything. You... [16:46:29] (03PS1) 10Mholloway: Update session_tick `test` defn to take string values [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/661963 [16:59:42] (03PS1) 10GoranSMilovanovic: Qurator [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/661964 [16:59:58] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Qurator [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/661964 (owner: 10GoranSMilovanovic) [17:03:25] (03PS1) 10GoranSMilovanovic: init [analytics/wmde/WD/WikidataQurator] - 10https://gerrit.wikimedia.org/r/661966 [17:03:42] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] init [analytics/wmde/WD/WikidataQurator] - 10https://gerrit.wikimedia.org/r/661966 (owner: 10GoranSMilovanovic) [17:25:55] (03PS1) 10Elukey: Release 2.8.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 [17:26:46] (03CR) 10jerkins-bot: [V: 04-1] Release 2.8.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey) [17:27:12] ufff [17:28:44] (03CR) 10Elukey: "recheck" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey) [17:28:58] (03CR) 10Elukey: "18:26:38 npm ERR! Error while executing:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey) [17:29:46] (03CR) 10jerkins-bot: [V: 04-1] Release 2.8.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey) [17:30:39] very weird, it seems that ci is broken [17:34:50] (03CR) 10Elukey: "I tried to come up with 2.8.5 but it seems that ci is broken, have you seen the issue before? If not I'll follow up with Antoine on Monday" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/661968 (owner: 10Elukey) [17:37:17] ottomata: https://issues.apache.org/jira/browse/BIGTOP-3488 :O :O :O [17:53:15] !log rebalance kafka partitions for eqiad.resource_change [17:53:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:53:37] !log rebalance kafka partitions for codfw.resource_change [17:53:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:38:50] !log rebalance kafka partitions for codfw.mediawiki.client.session_tick [18:38:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:42:30] * elukey afk! [18:42:35] have a nice weekend folks [18:49:23] (03CR) 10Nettrom: [C: 03+1] "This looks good to me! Adding Andrew as reviewer to see if there's something I've missed." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/661273 (https://phabricator.wikimedia.org/T263663) (owner: 10Eric Gardner) [19:00:57] (03CR) 10Ottomata: [C: 03+1] Update schema to handle quickview copy events [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/661273 (https://phabricator.wikimedia.org/T263663) (owner: 10Eric Gardner) [19:11:28] !log rebalance kafka partitions for eqiad.mediawiki.client.session_tick [19:11:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:48:09] (03CR) 10Milimetric: "looks good, left some comments. @Joal, what do you think of the pyspark idea? I kinda love it and it makes me want to rewrite webrequest" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/655804 (https://phabricator.wikimedia.org/T270140) (owner: 10Bmansurov) [20:00:51] 10Analytics, 10Patch-For-Review: Add time interval limits to pageview API - https://phabricator.wikimedia.org/T261681 (10Milimetric) The oversight is my fault, my apologies, I was too focused on our team's usage of the API. The initial motivation was high volume or wide timespan queries from a single user age... [20:15:44] 10Analytics, 10Analytics-Kanban: WMF-Last-Access cookie breaks Java client - https://phabricator.wikimedia.org/T98396 (10Milimetric) [20:27:31] 10Analytics, 10Product-Data-Infrastructure, 10Wikimedia-Logstash, 10observability: Create a separate logstash ElasticSearch index for schemaed events - https://phabricator.wikimedia.org/T265938 (10colewhite) 05Open→03Resolved a:03colewhite w3creportingapi logs are now in their custom schema. [20:31:06] !log rebalance kafka partitions for eventlogging_SearchSatisfaction [20:31:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:34:34] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) [21:53:18] 10Analytics, 10Analytics-Kanban, 10observability, 10Patch-For-Review: Modify Kafka max replica lag alert to only alert if increasing - https://phabricator.wikimedia.org/T273702 (10Ottomata) [23:12:48] 10Analytics, 10SRE, 10SRE-Access-Requests: Add kzeta to analytics-privatedata-users - https://phabricator.wikimedia.org/T272982 (10CDanis) I contacted Carol on Slack; this request is approved. [23:12:51] 10Analytics, 10SRE, 10SRE-Access-Requests: Add kzeta to analytics-privatedata-users - https://phabricator.wikimedia.org/T272982 (10CDanis) [23:17:33] 10Analytics, 10SRE, 10SRE-Access-Requests: Add kzeta to analytics-privatedata-users - https://phabricator.wikimedia.org/T272982 (10kzimmerman) Thank you @CDanis!