[03:26:01] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Hive is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Nuria) @JAllemandou we have code that prevents them from being so for self-reported bots: https://phabricator.wikimedia.org/T67508 [06:45:19] morning! [10:25:36] 10Analytics, 10Operations, 10SRE-Access-Requests: Allow access to Data Lake/Hive for Niharika - https://phabricator.wikimedia.org/T210022 (10jcrespo) a:03jcrespo [10:26:41] 10Analytics, 10Operations, 10SRE-Access-Requests: Allow access to Data Lake/Hive for Niharika - https://phabricator.wikimedia.org/T210022 (10jcrespo) [11:36:24] * elukey lunhc! [11:36:26] *lunch [11:56:07] joal: helloooo I've noticed that in the per project family uniques hive tables we have values for a few families that sound strange [11:57:08] namely: mediawiki, wikimediafoundation, wikidata and wikimedia [11:57:52] this conflicts with the editing metrics [11:58:14] see https://wikimedia.org/api/rest_v1/metrics/edited-pages/top-by-net-bytes-difference/all-mediawiki-projects/all-editor-types/all-page-types/2018/10/01 [11:58:39] versus https://wikimedia.org/api/rest_v1/metrics/edited-pages/top-by-net-bytes-difference/mediawiki/all-editor-types/all-page-types/2018/10/01 [12:20:15] (03PS1) 10GoranSMilovanovic: Sitelinks and Titles [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/475728 [12:20:33] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Sitelinks and Titles [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/475728 (owner: 10GoranSMilovanovic) [12:22:35] Hi fdans - the per-project-family uniques are defined through top-level cookies [12:24:10] hmm, this we have to keep in mind in the UI [12:24:47] fdans: This probably means we'll convert only certain all-project values? [12:25:17] joal: yes but in the patch i have in the UI we already define project families explicitly, so it shouldn't be problematic [12:25:31] k [12:25:56] fdans: We also want to be good in API :) [12:26:03] joal: through this thingy: https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/464583/2/src/apis/wikigroups.js [12:29:34] joal: btw got tangled in testing the family loading query and forgot about the jobs, is it ok for me to restart uniques jobs from the point the backfill ended the other day? [12:30:06] fdans: Please :) [12:30:14] thank youuu [12:30:25] fdans: Up to toda I imagine? [12:31:01] joal: but leave them running until we restart the bundle right? [12:31:27] fdans: if so, the coord from the bundle should be killed as well [12:31:52] fdans: second thing: there are 2 coords - One for daily, one for monthly - Are we talking about both? [12:33:26] joal: so the monthly data is up to date because we have uniques up to october, so no need to restart it now [12:33:46] daily is the one we need up to date to expose the data in aqs [12:34:08] fdans: I think we also show monthl data in AQS, no? [12:34:17] Ah, but it's up to date, right [12:34:24] yes that's what I mean :) [12:35:35] joal: so I think it's enough with just restarting the daily one [12:35:48] fdans: I don't agree :) [12:36:13] okeeei then I'm missing something joal :) [12:36:47] fdans: let's batcave, so that we disambiguise :) [12:36:59] omw [12:56:02] 10Analytics, 10Product-Analytics: Metrics request on portal namespace usage - https://phabricator.wikimedia.org/T205681 (10AfroThundr3007730) @Tbayer Just curious if the analytics team had time to pull any data for this yet. [13:21:38] 10Analytics, 10Analytics-Kanban, 10Core Platform Team, 10Core Platform Team Backlog, and 2 others: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) [13:46:39] joal: https://issues.apache.org/jira/browse/BIGTOP-3074 [13:46:44] "Drop oozie as it doesn't work with Hive-2.0+. See BIGTOP-2986" [13:46:48] /o\ [13:46:58] WAAAAAAT? [13:47:43] Holy moly - Upgrading will be hard ! /o\ [13:48:26] One comment on the bug says oozie-community is working on it :S [13:48:30] Maybe one of those days ... [13:48:50] but https://www.cloudera.com/documentation/enterprise/latest/topics/oozie.html [13:49:06] and I am pretty sure that in cdh6 there is hive 2.x [13:49:23] so they are most likely patching oozie to work with hive [13:50:12] yeah hive 2.1.1 [13:50:13] https://archive.cloudera.com/cdh6/6.0.0/ubuntu1604/apt/pool/contrib/h/hive/ [13:50:32] and oozie 5 https://archive.cloudera.com/cdh6/6.0.0/ubuntu1604/apt/pool/contrib/o/oozie/ [13:50:39] beta :D [13:52:04] woooow - interesting! [13:52:35] those packages will need to be rebuilt (or maybe used directly) for stretch [14:17:08] joal: do you have a minute to review some spark settings in bc ? [14:20:22] sure elukey - OMW (sorry for delay) [14:43:39] elukey: I was wondering if, by any chance, we could have had a go at trying ClickHouse installation on the new Presto-cloud nodes ;) [14:45:41] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) [14:47:16] joal: we could yes! They have to be configured to run VMs before that, but then we'll be able to experiment as much as we want [14:47:22] even in the analytics project [14:47:48] elukey: Great- I have looked, and indeed there are deb packages now [14:56:28] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) [14:56:52] joal: could it mean that we use clickhouse rather than presto if things are going well? [14:57:04] (just trying to figure out how to allocate time) [14:57:46] elukey: I don't think so - but I'd be interested in checking performance diffs more in detail [15:09:22] hey team :] [15:09:41] hola marceloooo [15:17:14] hey! [15:17:40] Hi milimetric - Do we take 5 minutes to warm-up for the upcoming meeting? [15:18:00] joal: yes, let's definitely do that, omw cave [15:18:52] (03CR) 10Mforns: "Looks pretty good! Left a couple comments." (033 comments) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/475618 (https://phabricator.wikimedia.org/T189475) (owner: 10Amire80) [15:21:37] FYI a-team! Pchelolo and I are going to deploy long awaited revision-score changes today. [15:21:41] plan is here [15:21:41] https://phabricator.wikimedia.org/T197000#4658652 [15:23:22] ottomata, we found a bug in profig, the lib that ConfigHelper uses. They merged our fix! And they are asking whether we are in a rush to get the change into a release build, do we need the release build? or can we import the fix anyway? [15:24:19] oh?! [15:24:22] link mforns wanna see [15:24:27] k [15:24:40] i'm pretty sure we use profig from upstream; would be nice if they released [15:24:43] https://github.com/outr/profig/issues/24 [15:24:47] we could fork/release but that' be annpying [15:25:02] https://github.com/outr/profig/pull/25 [15:25:19] ok, will ask them to do so [15:26:09] k [15:26:11] nice [15:26:19] ottomata, I was trying to activate RefineMonitor for ELSanitization, but passing regexes through property files wasn't working [15:26:36] it turns out property values that have = or : break [15:27:04] !log monthly and daily jobs for uniques killed, replaced with backfilling jobs until Dec 1st [15:27:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:28:19] aye [15:29:18] actually it's 40 past milimetric - sorry for having left too early :(0 [15:29:41] joal: join anyway, luca and I are here [15:29:44] k [15:41:46] !log stopped producing revision-score events with old schema; merged and deployed new schema; petr to deploy change to produce events with new schema soon. https://phabricator.wikimedia.org/T210013 [15:41:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:47:30] !log moved old raw revision-score data to hdfs in /user/otto/revision_score_old_schema_raw - T210013 [15:47:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:47:33] T210013: EventBus extension started emitting rev_count as a string - https://phabricator.wikimedia.org/T210013 [15:55:31] joal: we have revision-score events! [15:55:36] i'll set up a refine job.... :D [15:55:59] awesome ottomata :) In a meeting now, but I'll definitely have look later on [16:01:23] oh standup is in 30 mins... interestgiiing! [16:24:13] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10Ottomata) The Kafka security stuff shouldn't matter (yet) in this case. We'll likely have to set those one day soon. [16:31:47] ping ottomata standuppp [16:32:38] ping joal standup [16:33:52] UH OH [16:33:53] coming [16:39:23] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews top endpoint in descending order as of 2018-11-20 - https://phabricator.wikimedia.org/T210091 (10Milimetric) Just to follow up to put this issue to rest, the caches are all cleared and the responses are all consistent, going fr... [16:49:01] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) I'm setting a deadline for myself to name this thing and to create a WMF repo this week. At the moment I'm leaning... [17:06:09] ping fdans [17:06:32] fdans: groooosskinnnn [17:21:01] PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is CRITICAL: CRITICAL [17:22:19] just remounted --^ [17:45:26] 10Analytics: Link to User Contribution page in wikistats UI - https://phabricator.wikimedia.org/T210422 (10Nuria) [17:45:40] 10Analytics: Link to User Contribution page in wikistats UI rather than user page - https://phabricator.wikimedia.org/T210422 (10Nuria) [17:46:07] 10Analytics: Link to User Contribution page in wikistats UI rather than user page - https://phabricator.wikimedia.org/T210422 (10fdans) [17:46:09] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10fdans) [17:46:48] 10Analytics: Wikistats2 metric: top article creators - https://phabricator.wikimedia.org/T210423 (10Nuria) [17:47:39] (03PS3) 10Milimetric: Sqoop from logging_compat [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473256 [17:48:25] 10Analytics: Wikistats2 metric: top article creators - https://phabricator.wikimedia.org/T210423 (10fdans) [17:48:27] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10fdans) [17:48:45] (03CR) 10Milimetric: "going with this solution for now, as a stop-gap measure to make sure sqooping works for the 2018-11 snapshot." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473256 (owner: 10Milimetric) [17:50:25] 10Analytics: Wikistats2: UX bug table option should not be available in table option (example table view of top edits) - https://phabricator.wikimedia.org/T210424 (10Nuria) [17:51:03] 10Analytics: Wikistats2 UX bug: table option should not be available in table option (example table view of top edits) - https://phabricator.wikimedia.org/T210424 (10Nuria) [17:51:05] 10Analytics: Wikistats2 UX bug: table option should not be available in table option (example table view of top edits) - https://phabricator.wikimedia.org/T210424 (10fdans) [17:51:09] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10fdans) [17:51:10] RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is OK: OK [17:51:55] (03PS4) 10Milimetric: Sqoop from logging_compat [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473256 [17:52:31] (03PS1) 10Milimetric: [WIP] brainstorming the sanitizing pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/475799 [17:58:46] 10Analytics: Wikistats2 metric: top edited pages by number of editors - https://phabricator.wikimedia.org/T210425 (10Nuria) [17:59:11] 10Analytics: Wikistats2 metric: top edited pages by number of editors - https://phabricator.wikimedia.org/T210425 (10fdans) [17:59:13] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10fdans) [18:01:35] 10Analytics: Wikistats2 metric: top edited pages by number of editors - https://phabricator.wikimedia.org/T210425 (10Nuria) [18:08:11] a-team are we planning on doing some team thing right after all hands or something? sorry my brain forgets things [18:08:23] All hands looks like it is only the first 3 days of the week [18:08:30] OH [18:08:32] OHHH [18:08:35] no it is all week [18:08:36] sorry [18:08:41] meeting days and all hands conference days [18:08:41] ok [18:08:44] its all week great! [18:10:19] ottomata: UTAH [18:10:47] fdans: not gonna maaake iiiiit sorrryyyy got plans already [18:11:03] :____( [18:14:44] * elukey off! [18:16:08] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create reports in wikistats UI for "most prolific editors" (a.k.a "top contributors") - https://phabricator.wikimedia.org/T189882 (10Nuria) [18:16:12] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) 05Resolved>03Open [18:16:57] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) Reopening this, as @ezachte pointed out our metric is articles with more "contributions" in terms of 'edits" bu... [18:18:38] 10Analytics: Wikistats2 metric: top edited pages by number of editors - https://phabricator.wikimedia.org/T210425 (10Nuria) [18:18:42] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) [18:23:24] (03PS1) 10Michael Große: WIP [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 [18:23:41] (03CR) 10jerkins-bot: [V: 04-1] WIP [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (owner: 10Michael Große) [18:26:08] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Hive is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10chelsyx) Thanks for looking into this @Nuria ! [18:32:04] joal: we have mediawiki_revision_score! [18:33:51] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10jcrespo) Yes, RAID 10 with 256K stripe is our default setup, sorry for not specifying it. Only on very specific setup we will not want that (parsercaches or o... [18:35:16] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10jcrespo) [18:36:20] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10jcrespo) [18:36:25] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10Cmjohnson) Thanks @jcrespo! I didn't want to assume anything. [18:37:16] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10jcrespo) We are cool! Better to ask than make you work twice :-D [18:38:54] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10awight) @bmansurov This might be interesting to you. Please let us know if the design will be compatible with your articl... [18:42:11] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Ottomata) Woot, we have a Hive event.mediawiki_revision_score table! :) [18:42:27] 10Analytics, 10Datasets-Archiving, 10Research: Make HTML dumps available - https://phabricator.wikimedia.org/T182351 (10Sj) See also: https://phabricator.wikimedia.org/T210413 for maintaining an IPFS-friendly dump (which could be generated from an unzipped HTML dump) [18:44:11] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10Ottomata) Hey heyyy! We deployed changes for T197000 today. I also re-enabled Hive refinement of this data, so we now ha... [18:47:31] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 3 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Pchelolo) 05Open>03Resolved Ok. The events with the new schema are being emitted, there are no rejections on the #eve... [18:49:31] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) It doesn't seem like the codec choice should affect this either. [18:55:39] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) I don't have SSH access to `logstash1007` in order to check which processes are running/erroring. Should I submit an access request or can... [19:03:24] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10bmansurov) @awight thanks for the ping. I'll keep an eye on the task. [19:06:10] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10Ottomata) @phuedx come find me in #wikimedia-analytics in about 30 mins and we'll look together [19:08:15] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10Ottomata) Actually, now is fine come find me! :) [19:10:21] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10Nuria) >1a What is group bot vs name bot. I can't find explanation and I'm not familiar with this breakdown. Having in mind that we still have pending our tasks about... [19:10:28] joal: yt? [19:13:20] Nettrom: I'm going to generate some events on testwiki now, will send you a user id in a sec [19:13:37] kostajh: sounds good, I'm ready to check for those [19:15:25] Nettrom: user id 41496 [19:15:31] kostajh: saw them! [19:15:38] Hi nuria [19:15:40] Here I am [19:16:15] Nettrom: cool, can you send me the json, in a phab paste or via gchat? [19:16:16] joal: erik had made our point about the "articles with most contributors" here: https://phabricator.wikimedia.org/T204965 [19:16:58] kostajh: will do in a minute [19:17:14] joal: i think we closed that ticket too eagerly and implemented "articles with most edits" instead cc fdans [19:17:29] Nettrom: thanks, I just logged out so the last event should be a logout event [19:18:17] kostajh: got it, chat incoming [19:18:18] nuria: wooooow - I think I get it - Very probable that I misread and implemented the wrong one :( Oh my :( [19:20:27] Nettrom: hmm, is it possible that we might have missed the first event for that user? [19:20:31] joal: I did too, i reopened ticket and given that thsi was part of our goals I think we should spend some cycles thinking about it and if possible, implement it this quarter [19:20:58] Nettrom: first several events, rather [19:21:04] kostajh: no, that's my copy & paste error, sorry! [19:21:08] kostajh: let me fix that [19:21:17] 😅 [19:22:09] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10Cmjohnson) [19:22:18] I hear that nuria - Will spend cycles indeed - The "Top pages by number of editors" is actually in the digests parts of our metrics, so trickier to get right - Will think [19:22:34] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10Cmjohnson) a:05Cmjohnson>03RobH @robh these are ready to for installs. [19:23:17] Nettrom: ok, all looks good, thanks! [19:23:28] kostajh: great to hear, yay! :) [19:26:19] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4757923, @JAllemandou wrote: >>>! In T209731#4754979, @Nuria wrote: >> It is worth looking at already existing event data, if we want to reuse... [19:26:23] (03CR) 10Joal: "LGTM! Let's merge :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473256 (owner: 10Milimetric) [19:26:50] (03CR) 10Joal: [V: 032 C: 032] "Let's ACTUALLY merge ..." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473256 (owner: 10Milimetric) [19:27:01] :) thanks jo [19:29:44] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10JAllemandou) I support the idea of using model name and version as partitions. Wiki_db would possibly be another good fit if requests will most often be on singular... [19:31:32] milimetric: thinking of something you mentioned in the sqooping-report: We should REALLY make a checker for sql-schema changes [19:31:36] Not hard I htink [19:32:37] not hard, no, just a few queries to information_schema and a diff [19:36:36] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) Rather than backfilling which implies you are "filling a hole of data" this is a model recalculation completely. As in: you are recalculating scores due to e... [19:41:19] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4775194, @JAllemandou wrote: > I support the idea of using model name and version as partitions. Wiki_db would possibly be another good fit if... [19:50:58] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) @awight: Can you explain a bit the consumer use cases? Data in hive of this nature is mostly consumed by automated processes that create derived datasets. Do... [19:56:42] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4775228, @Nuria wrote: > Rather than backfilling which implies you are "filling a hole of data" this is a model recalculation completely. As... [19:56:46] Insight landed !! [19:57:12] joal: ahem... insight about ? [19:57:37] nuria: https://en.wikipedia.org/wiki/InSight [19:59:00] joal: wowowowow [19:59:40] indeed ! those folks just succeeded in sending and grounding some stuff a few millions kilometers away :D [20:00:55] also amazing: they updated the wikipedia article in like 2 seconds [20:02:04] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4775303, @Nuria wrote: > @awight: Can you explain a bit the consumer use cases? Data in hive of this nature is mostly consumed by automated p... [20:07:46] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10JAllemandou) > Will the order of partitions make a difference? For example, if consumers are more likely to get multiple models of scores for a single wiki, vs. mul... [20:10:09] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) >Some processes such as the recommendation API are already running on Hive, and for example might benefit from the new scores table, by finding the top 1% qua... [20:21:48] 10Analytics, 10Anti-Harassment (AHT Sprint 34): 👩‍👧 Track how often blocked user attempt to edit - https://phabricator.wikimedia.org/T189724 (10TBolliger) [20:26:49] milimetric: meeting notes in https://office.wikimedia.org/wiki/User:Milimetric_(WMF)/Analytics_context_on_sanitizing cc joseph, please add /edit as needed [20:30:37] Thanks nuria - Looks good --^ [20:31:38] Gone for tonight team - Will see y'all tomorrow [20:55:48] I am trying to see messages in eqiad.mediawiki.page-delete topic and do not see anything [20:56:05] anybody knows if I'm doing something wrong or there's nothing there? [20:58:19] this seems to be empty too: https://grafana.wikimedia.org/dashboard/db/kafka-by-topic?refresh=5m&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-kafka_cluster=main-eqiad&var-kafka_broker=All&var-topic=eqiad.mediawiki.page_delete&from=now-7d&to=now [21:01:10] SMalyshev: might be caused by https://phabricator.wikimedia.org/T210013 [21:01:55] ottomata: this talks about revision score, but I see no messages in deletion scheme... [21:02:58] OH oops that is a bad log message SMalyshev [21:03:04] i logged with the wrong Phab ticket id earlier [21:03:12] its about the article delete hook [21:03:27] its causing validation errors for page delete [21:03:55] so, how long back it goes? Because I don't see any deletes in the topic at all [21:04:04] which means it's at least 90 days back? [21:04:22] or 14 days? [21:04:30] I am not sure what's retention in Kafka [21:04:38] SMalyshev: you are looking at kafka main? [21:04:50] ottomata: kafka1001.eqiad.wmnet:9092,kafka1002.eqiad.wmnet:9092,kafka1003.eqiad.wmnet:9092 [21:04:56] not sure which one is that [21:05:03] that's main [21:05:08] i think its only 7 days back there [21:05:11] ok, so I see nothing in the topic [21:05:38] which means probably WDQS also did not get any deletes from Kafka [21:06:06] ya looks like nobody got any deletes [21:06:24] huh that's bad [21:06:41] its especially bad that we didn't notice it [21:06:44] and it's still not fixed? [21:06:47] its waiting on deploy [21:06:49] thi week train [21:06:50] i think [21:06:56] i just found out about it today too [21:07:24] its bad that we didnt' notice; i think the volume was low enough (page delete), and the error mundane enough (validation error) [21:07:50] but, its kind of MW's fault. Tim changed the hook which caused the type of the param to change [21:07:58] haha, this is even more so a problem of loosely typed PHP :p [21:08:00] yeah I've got people complaining WDQS is missing deletes... I thought it's some WDQS problem but then I see there's nothing in delete topic at all [21:08:50] we probably need to SWAT that change ASAP then... I see it's from Nov 21 and still not deployed? [21:09:18] ya maybe, we should ping Petr if that is so i think [21:11:57] 10Analytics, 10EventBus, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) [21:12:19] 10Analytics, 10EventBus, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) p:05Triage>03Unbreak! [21:12:26] ottomata: ok, I've created an UBN task here: https://phabricator.wikimedia.org/T210451 to get page-deleted messages running again [21:12:59] and will probably need to figure out a way to backpatch all missing updates since then... [21:13:11] ottomata: do you know when exactly the breakage started? [21:14:18] the patch is from Oct 23 - does it mean we've been running for a month without working kafka topic and just noticed it now? This is really bad if so... [21:15:49] 10Analytics, 10EventBus, 10Operations, 10Wikidata, and 2 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) [21:18:28] SMalyshev: i think so yes. [21:18:44] :( [21:18:46] i think i saw another ticket about getting alerts for that type of thing [21:19:17] the only other way I know to figure this out...would either be to query the mediawiki database, or to use the mediawiki history table in hive [21:19:44] hm, i suppose...are page-deletes in recentchange? if so, there is a recentchange table in hive that would have this. since recentchagne doesn't use this schema directly, i doubt it was affected [21:20:17] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) >>! In T209731#4775407, @Nuria wrote: >>Once we've piped a mediawiki_revision_score event into our new table, we'll never need to read that event again. Does... [21:20:19] ottomata: which table it would be? List of all deletes for the duration of incident would help a lot [21:23:03] milimetric: can you help SMalyshev here? [21:23:12] he needs page delete events [21:23:18] in mw history [21:23:21] is that possible? [21:23:30] more specifically, in wikidata history [21:23:55] SMalyshev: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Mediawiki_history [21:24:12] I probably could extract it via some scripting and RC API but if I can save the effort it'd be nicer [21:24:13] although [21:24:19] that table is only updated once a month [21:24:20] its a snapshot [21:24:29] so it should have Oct in there [21:24:31] but not Nov yet [21:24:35] ottomata, the profig fix has been released as 2.3.3 :] [21:24:39] mforns: nice! [21:25:03] ottomata: I need November... [21:26:33] yeah [21:26:40] it'll happen, but prob not until dec 10 [21:26:46] ish [21:27:03] SMalyshev: let's try the event.mediawiki_recentchange table [21:27:16] OH nope [21:27:18] that one won't work either [21:27:25] as a table. [21:27:35] we can't use that one because it is basically schemaless [21:27:42] so we can't use it in SQL contexts lik ehive [21:27:54] we do import the raw json though [21:27:57] i think... [21:28:13] yes we do [21:28:14] ok [21:28:21] 10Analytics, 10EventBus, 10Operations, 10Wikidata, and 2 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Pchelolo) Yes, [[ https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/475011/ | the fix ]] has not been deployed yet. Asked to put it on Euro Mid... [21:28:27] SMalyshev: this is going to be annoying, because it is just random JSON lines in hadoop [21:28:53] SMalyshev: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging#Hadoop_Raw_Data [21:29:07] this isn't eventlogging analytics data [21:29:10] so the path is different [21:29:23] it will be [21:29:51] /wmf/data/raw/event/eqiad_mediawiki_recentchange [21:29:51] and [21:29:51] /wmf/data/raw/event/codfs_mediawiki_recentchange [21:29:53] oops [21:30:02]  /wmf/data/raw/event/eqiad_mediawiki_recentchange [21:30:02] and [21:30:02]  /wmf/data/raw/event/codfw_mediawiki_recentchange [21:30:23] the wikitech link shows how you can use either hive or spark to get to that data [21:31:10] wait, so how I use it? is it from command line on stat machines? [21:31:14] hm the spark docs there are old...for spark 1 instead of 2 [21:31:22] spark 2 should be mostly the same, but the API might be slightly different [21:31:23] not sure [21:31:34] you might be able to do spark.read.json(path/here) [21:31:39] SMalyshev: sure [21:31:41] you can try [21:31:48] spark2-shell (for scala) or pyspark2 (python) [21:32:22] SMalyshev: https://spark.apache.org/docs/2.3.1/sql-programming-guide.html#json-datasets [21:32:29] hmm I have no idea what that all means... [21:32:48] SMalyshev: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark [21:35:09] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10mobrovac) p:05Unbreak!>03High Will be done in the EU SWAT window on 2018-11-27. Lowering the priority as it is a known issue and does not... [21:36:28] SMalyshev: i'm not 100% how sure spark.read.json() will work on the recentchange data; it may have conflicting enough schema that it wont' be able to load it [21:37:08] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) It does cause pretty severe breakage - all delete updates are missing from WDQS and people are complaining (in fact, have been comp... [21:38:47] hmm it's kinda hard to make sense of these pages without prior knowledge... do you have any recommendation which example I should use to extract delete events on wikidata for the last month? [21:39:11] those all seem to do some different things and not sure even which ones... [21:41:29] SMalyshev: i just tried real quick on recentchange; doesn't look good [21:41:36] the data is too nasty [21:41:46] maybe I'll just use the original recentchanges table in the db [21:41:48] at least, not with spark [21:41:49] may be simpler [21:42:23] SMalyshev: might be simpler; the data in Hadoop there will be simliar to what's in the db, but in JSON, and kinda random. its pretty much what you'd get if you subscribed to the IRC recentchange feed [21:42:52] so, i could help you download the json as files, and you could parse them yourself...but at that point querying the db might be easier [21:43:09] ottomata: we can detect that type of no event scenario looking for last dates of events [21:43:51] nuria: you mean lack of events? [21:44:01] ottomata: yes [21:44:04] we'd have to configure alerts for specific streams; i think we have some for e.g. revision-create [21:44:11] but every stream has its own volume [21:44:20] so either alerts are configured manually for every one [21:44:39] or we have some auto anomoly detection on all of them [21:45:33] ottomata: i think a total drop does not need anomaly detection if you look at all streams and you sort by time you will see one falling two days behind all the rest at some point [21:45:41] nuria: really what we need is alerting on validation errors [21:45:41] https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.11.26/root?id=AWdR-4Ll8MXbBsCPIbSw&_g=h@7bf0c26 [21:46:19] ottomata: was it validation errors? I thought it was errors not being sent .. if the earlier that is sure easier [21:46:47] yes validation errors this time [21:46:59] Tim made a change to the hook that was used to emit this event [21:47:04] and the type cahnged from int to string [21:47:19] ottomata: wait, why did the errors alarm not jumped then? cause volume was that low? [21:47:40] this was the fix: the volume is pretty low [21:47:42] oops [21:47:45] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/475011/3/includes/page/WikiPage.php [21:48:16] nuria: i don't think there are validation alerts for eventbus [21:48:35] was in meeting, have to take some notes and will read up [21:48:52] ottomata: i see, this one is EL alone no? https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=13&fullscreen&orgId=1&from=now-30d&to=now-5m [21:49:09] ya [21:49:38] eventbus doesn't emit to a topic like that. new stream intake service will tho [21:51:04] ottomata: ok, which means that once we have the event intake service ingesting events we shoudl be able to send validation errors to a topic [21:51:17] ottomata: what happens now to validation errors in eventbus? [21:51:28] ottomata: they just get rejected at the http level, right? [21:53:05] 10Analytics: Alert on validation errors on new stream intake service - https://phabricator.wikimedia.org/T210457 (10Nuria) [21:54:02] yes, they also go to a local file [21:54:27] OH ho hoooo [21:54:45] SMalyshev: i do have all failed page delete events after all [21:54:52] ottomata: a local file where? [21:55:03] on the box where the service runs [21:55:06] each box [21:55:14] HMMMMM [21:55:19] i *could* replay them all [21:55:24] ottomata: ok, then we can have alarms [21:55:25] if I parse and fix the int [21:55:40] oh nuria the errors also go to logstash... [21:55:42] ottomata: you can backfill YES [21:55:44] not sure if we can make a logstash alert [21:55:55] oof that backfill is going to be a pain [21:55:57] but possible [21:57:25] ottomata: probably the alarm is not that hard to do if logstash is at all connected to our icinga infrastructure [21:58:07] ottomata:as for backfilling, who relies on that data for tier-1 use cases besides SMalyshev ? [21:58:14] ottomata: do we know? [21:58:50] change prop maybe? [21:58:52] ottomata: I think I already extracted the page IDs from the database [21:58:53] not sure [21:58:56] ya? [21:59:00] so I'll just update them manually [21:59:07] it's only 8k items so not too bad [21:59:14] ya the volume is pretty low [21:59:24] the priority is to get the channel running going forward [22:01:34] SMalyshev: you can query the logging table, unless I'm misunderstanding what you need [22:01:49] like select * from logging where log_type = 'delete' and log_action = 'delete'? [22:02:00] and you can consider restore if you want [22:02:03] yep, I've figured it out [22:02:12] ok, sorry I was late to the party, was in meeting [22:02:15] I just used DB rc table [22:02:25] that works, yeah, but only goes back 90 days [22:02:26] milimetric: no problem :) [22:02:39] yeah fortunately I only need it back a month [22:02:45] k, great. let me know if you need anything [22:03:55] ottomata: ok, let's talk to peter and if change prop needs those deletes probably peter can backfill, i think we need to look into alarms on validation for this eventbus though [22:04:10] mainly now we need to get the channel back to running and add some monitoring to it [22:11:51] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) Process above seems a bit error prone (as you do not want to hit your live pipeline to recalculate scores for events 15 years behind), those two calculations... [22:13:09] 10Analytics: Explore adding validation alarms to eventbus using logstash - https://phabricator.wikimedia.org/T210459 (10Nuria) [22:13:26] SMalyshev: talks created , please add yourself as subscriber if you wish: https://phabricator.wikimedia.org/T210459 [22:26:12] 10Analytics, 10EventBus, 10Services (watching): Explore adding validation alarms to eventbus using logstash - https://phabricator.wikimedia.org/T210459 (10Smalyshev) Example of topic needing alarms here: {T210451} [22:26:15] 10Analytics, 10EventBus, 10Services (watching): Explore adding validation alarms to eventbus using logstash - https://phabricator.wikimedia.org/T210459 (10Pchelolo) If I understand the proposal correctly, it's easier to do it using grafana alerts which I'm proposing in T210031 I will start experimenting wit... [22:27:39] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Pchelolo) [22:27:51] 10Analytics, 10Analytics-Dashiki, 10Google-Code-in-2018, 10goodfirstbug: Add external link to tabs layout - https://phabricator.wikimedia.org/T146774 (10Milimetric) I tried to register at the https://codein.withgoogle.com/ site, but that site has some major bugs... For example, when signing in, it says I... [22:29:06] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10RobH) [22:30:26] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Pchelolo) Currently, if the alerts will be enabled, the emails will go to only to [[ https://github.com/wikimedia/puppet/blob/b347052863d4d2e87b37d6c2d9f44f833cfd9dc2/modules/ic... [22:32:57] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10RobH) a:05RobH>03Cmjohnson @Cmjohnson dbstore1004 shows production network cable issue? Switch shows it is admin enabled but no link: ge-1/0/13 u... [22:33:20] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install dbstore100[3-5].eqiad.wmnet - https://phabricator.wikimedia.org/T209620 (10RobH) dbstore1003 & dbstore1005 are fully installed and now online, standing by with role:spare applied. [22:33:38] 10Analytics: Merge metadata from filtered-tables.txt and maintain-views.yaml - https://phabricator.wikimedia.org/T210462 (10Milimetric) [22:34:05] nuria: thanks, did so [22:35:10] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Nuria) Pchelo : are those alarms taking advantage of icinga/nagios? Cause for critical alarms we want to shy away from e-mail as it is hard to tell what alarms are open and bein... [22:37:21] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Pchelolo) @Nuria yes. Sample alert we're getting for #restbase ` Notification Type: PROBLEM Service: https://grafana.wikimedia.org/dashboard/db/restbase grafana alert Host:... [22:39:24] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Nuria) I see, then it should go to analytics-alerts and i take that also appears on IRC? [22:40:10] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Public Data for the WMDE Banner Campaigns Dashboard - https://phabricator.wikimedia.org/T210463 (10GoranSMilovanovic) p:05Triage>03High [22:41:05] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Pchelolo) The ones we have for RESTBase doesn't go to IRC, but I think for analytics it should appear on IRC judging by https://github.com/wikimedia/puppet/blob/ce74f598509d2b44... [22:43:16] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Nuria) @Pchelolo excellent, sounds like that is what we need until we have a more robust error setup with the new intake service and we can report distinctively on validation er... [22:44:12] AH SMalyshev i think is not relevant anymore; but I made a mistake with spark before [22:44:20] it was my fault that I couldn't read the json; i was doing it wrong [22:45:07] 10Analytics, 10EventBus, 10Services (later): Create alert on EventBus 400 error rate - https://phabricator.wikimedia.org/T210031 (10Pchelolo) So here's the plan: tomorrow I will test some alerts manually, if all goes well - we will add `analytics` into the contactgroup for `eventbus` dashboard. However, cu... [22:52:23] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) Thanks for all the help! @JAllemandou Can you confirm that we should partition on model_version? Will that make it possible to efficiently purge all data f... [22:55:43] 10Analytics: Refinery Spark HiveExtensions schema merge should support merging of arrays with struct elements - https://phabricator.wikimedia.org/T210465 (10Ottomata) [22:56:55] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) a:05fdans>03JAllemandou [22:57:22] 10Analytics: Refinery Spark HiveExtensions schema merge should support merging of arrays with struct elements - https://phabricator.wikimedia.org/T210465 (10Ottomata) Input schema has: ` |-- scores: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- error: struct (nullabl... [23:12:35] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) >Will that make it possible to efficiently purge all data from an old model version? Yes, it would. Purging means you are going to drop the "whole" "model ver... [23:16:08] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) Wonderful, that answers all the questions I have from our perspective. I'll update the task description… [23:17:31] 10Analytics, 10ORES, 10Scoring-platform-team: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) 05Open>03Resolved [23:17:34] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10awight) [23:24:53] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Allow access to Data Lake/Hive for Niharika - https://phabricator.wikimedia.org/T210022 (10Niharika) 05Open>03Resolved Thanks all! [23:42:33] 10Analytics, 10EventBus, 10Operations, 10WMF-JobQueue, and 5 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451 (10Smalyshev) To make the problem worse, looks like breakage started more than 30 days ago - which means we don't have a record of the old delete... [23:43:12] ottomata: looks like some of the missing deletes are older than 30 days, which means recentchanges tables do not have them anymore. Do we have older data? [23:43:26] SMalyshev: i think i found older data [23:43:30] let me try to gather it [23:43:55] ottomata: thanks, that would help a lot! Basically I need all deletes on Wikidata from about Oct 23 to October 27. [23:44:19] rc goes from somewhere in Oct 27 so I got those ones [23:47:19] SMalyshev: the " select * from logging where log_type = 'delete' and log_action = 'delete'?" would also do it, i think ottomata is going to get validation failures from hosts which might not look as pretty [23:47:46] nuria: does logging extend more back? [23:50:57] SMalyshev: i thought it was older, but maybe i got it wrong ...checking [23:51:32] hmm yes looks like it has stuff older [23:51:38] maybe this would work then [23:52:23] yeah looks like old stuff shows up, thanks! [23:52:35] ottomata: this can be a solution for me then [23:53:03] ottomata: ya, i think evens from tables will be more sensical than from validation errors cc SMalyshev [23:55:47] indeed [23:59:37] ok, just in case, i gathered all of the failed events log files on stat1004 [23:59:47] great, thanks [23:59:47] i'll stop here, but if it doesn't work SMalyshev we should be able to get them from these