[01:13:26] milimetric: For the geoeditor dumps, should they continue to include both lower and upper bounds or just upper bounds? [01:15:17] 10Analytics, 10Event-Platform, 10Product-Analytics, 10CPT Initiatives (Modern Event Platform (TEC2)): Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10Ottomata) I haven't looked at the data, but are meta.id and meta.dt different? If s... [01:22:09] lexnasser: lower and upper is probably good for dumps; we'll probably turn those off after we incorporate your API into the wikistats UI [01:22:23] milimetric: sounds good, thanks [01:30:15] 10Analytics, 10Product-Analytics: Give clear recommendations for Spark settings - https://phabricator.wikimedia.org/T245897 (10nshahquinn-wmf) >>! In T245897#5913626, @JAllemandou wrote: > There is some misunderstanding here between recommendations and examples IMO. the links pasted in the task definition show... [01:35:47] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Analysts cannot reliably use Spark to run SQL queries against Hive databases - https://phabricator.wikimedia.org/T245891 (10nshahquinn-wmf) [02:25:26] 10Analytics, 10Product-Analytics (Kanban): Update wmfdata to support multiple SQL engines for Hive databases - https://phabricator.wikimedia.org/T246060 (10nshahquinn-wmf) [02:25:41] lexnasser: the dumps should not change, they can include the same data they have now [02:26:02] ah sorry I see milimetric responded [02:26:42] hi nuria :) I see you're working late after reading all the coronavirus news you could find, same as me :) [02:26:42] 10Analytics, 10Product-Analytics (Kanban): Update wmfdata to support multiple SQL engines for Hive databases - https://phabricator.wikimedia.org/T246060 (10nshahquinn-wmf) Leaving this unprioritized for now, pending more thinking and discussion about the best way to deal with these issues. [02:27:02] 10Analytics, 10Product-Analytics: wmfdata cannot recover from a crashed Spark session - https://phabricator.wikimedia.org/T245713 (10nshahquinn-wmf) a:05kzimmerman→03nshahquinn-wmf [08:09:19] 10Analytics: Check home/HDFS leftovers of flemmerich - https://phabricator.wikimedia.org/T246070 (10MoritzMuehlenhoff) [09:00:13] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: Add Fsalutari to nda LDAP group - https://phabricator.wikimedia.org/T245997 (10MoritzMuehlenhoff) >>! In T245997#5912965, @Ottomata wrote: > @Muehlenhoff just double checking: Fsalutari has an NDA, can I just add to `nda` LDAP group? Y... [10:12:38] (03PS1) 10Joal: Update spark kernels settings for consistency [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/574710 (https://phabricator.wikimedia.org/T245897) [10:14:03] 10Analytics, 10Product-Analytics, 10Patch-For-Review: Give clear recommendations for Spark settings - https://phabricator.wikimedia.org/T245897 (10JAllemandou) > If these are the best starting points, maybe the examples should use them instead. I have updated the [[ https://wikitech.wikimedia.org/wiki/SWA... [10:15:35] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Give clear recommendations for Spark settings - https://phabricator.wikimedia.org/T245897 (10JAllemandou) a:03JAllemandou [12:06:09] (03CR) 10Fdans: "@MarcoAurelio that might be because this change has the wrong parent by accident. Let me try to change it to the correct one." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/573953 (https://phabricator.wikimedia.org/T240621) (owner: 10Fdans) [12:26:10] (03PS2) 10Fdans: Remove aa.json, replace untranslatable string [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/573953 (https://phabricator.wikimedia.org/T240621) [12:26:46] (03CR) 10Fdans: [C: 03+2] Remove aa.json, replace untranslatable string [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/573953 (https://phabricator.wikimedia.org/T240621) (owner: 10Fdans) [12:28:12] (03Merged) 10jenkins-bot: Remove aa.json, replace untranslatable string [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/573953 (https://phabricator.wikimedia.org/T240621) (owner: 10Fdans) [13:15:44] 10Analytics, 10Analytics-Wikistats, 10translatewiki.net, 10Patch-For-Review: Add stats.wikimedia.org to translatewiki.net - https://phabricator.wikimedia.org/T240621 (10fdans) @abi_ thank you for the list! Here's the original ring file: https://www.mediawiki.org/wiki/File:WMF_Analytics_-_Ring_Logo_(thick)... [14:15:07] (03CR) 10Ottomata: "I'm not so sure about 12g memory for local. notebook hosts only have 64g of memory and are often mostly used. Can we make this 6g? If mo" [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/574710 (https://phabricator.wikimedia.org/T245897) (owner: 10Joal) [14:25:41] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: Add Fsalutari to nda LDAP group - https://phabricator.wikimedia.org/T245997 (10Ottomata) @Fsalutari try now. @Muehlenhoff this user already has a shell entry in data.yaml...is that what you mean? [14:34:21] (03PS2) 10Joal: Update spark kernels settings for consistency [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/574710 (https://phabricator.wikimedia.org/T245897) [14:35:58] (03CR) 10Joal: "Updated to 4 executors and 6g for spark local. I however think we should have a script killing long-running processes on the notebook. It " [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/574710 (https://phabricator.wikimedia.org/T245897) (owner: 10Joal) [14:36:47] milimetric, ottomata - are we meeting? [14:45:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10akosiaris) @Ottomata, tokens created and being propagaged across the clus... [14:46:06] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) Thank you! [14:46:19] ping bis ottomata milimetric ? [14:46:31] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: Add Fsalutari to nda LDAP group - https://phabricator.wikimedia.org/T245997 (10Fsalutari) I can now log in! Thanks! [14:46:50] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [14:46:51] OHNO [14:50:05] mforns: o/ [14:50:16] when you are onlinez let's check RU [14:50:30] I also prepped a change to move the mysql jobs out of stat1006 [14:51:04] (03CR) 10Joal: "Documentation pages created." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/572834 (https://phabricator.wikimedia.org/T244707) (owner: 10Joal) [14:55:49] ottomata: Yooohooo [14:56:18] ohk! [15:06:12] 10Analytics, 10Analytics-Wikistats, 10translatewiki.net, 10Patch-For-Review: Add stats.wikimedia.org to translatewiki.net - https://phabricator.wikimedia.org/T240621 (10abi_) >>! In T240621#5915497, @fdans wrote: > @abi_ thank you for the list! > > Here's the original ring file: https://www.mediawiki.org/... [15:10:32] do you guys know where RU reports are published? [15:13:22] is it in /srv/analytics.wikimedia.org/published/datasets/periodic/reports on thorium? [15:14:56] Luca, why don't you read /usr/local/bin/published-sync on an-launcher? [15:15:02] come on it has been 4y in here [15:15:06] still lost in this stuff? [15:16:14] ah /srv/published-rsynced on thorium, that of course doesn't contain stuff from an-launcher [15:18:12] ah! hosts_allow => $::statistics::servers, [15:19:51] heya team :] [15:20:02] elukey, ok, checking RU [15:25:34] elukey, reports failed at query time... [15:25:40] kerberos? [15:26:27] mforns: which one failed? [15:26:38] reference-previews [15:27:12] ah lovely doesn't return non-zero in this case [15:29:17] elukey, no RU tries to execute all other reports [15:29:47] even if all fail at query time, RU is chillin' [15:29:54] :/ [15:30:39] no bueno [15:31:20] i don't recall how RU uses kerberos though [15:31:26] in puppet we don't define anything [15:32:09] also the error seems to be ERROR - Report "popups" could not be executed because of error: object of type 'NoneType' has no len() [15:33:25] elukey, yes, this means the query returned no data [15:33:36] elukey, if I try to execute the query manually, I get: FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException [15:34:10] yes but in all it fails for [15:34:11] empty_row = [report.start] + [None] * (len(normalized_header) - 1) [15:34:30] so one thing to notice is that RU now runs with python 3.7, not 3.5 [15:35:15] anyway, it is something to fix later [15:35:27] mforns: how did you find/run the query? [15:35:28] oh aha [15:35:47] I opened the corresponding file in the reportupdater-queries repo [15:35:49] i.e. [15:36:11] reference-previews/baseline [15:36:41] copied it and replaced all references to $1 with a date: 2020-02-24 [15:37:07] mforns: do you recall how we contact hive with Kerberos though? There is no trace of kerberos-run-command in puppet [15:37:11] I don't recall how we did it [15:39:29] elukey, if I execute the following query from stat1007 and from an-launcher1001 it works for the first and fails for the second: [15:39:29] select * from event.referencepreviewsbaseline where year=2020 and month=2 and day=24 limit 10; [15:39:54] under my kinited user [15:40:41] seems like a hive problem no? [15:40:52] mforns: fails how? :) [15:40:55] more details please [15:41:01] also, using the hive cli? [15:41:04] the error that I pasted before [15:41:06] yes [15:41:12] FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException [15:42:00] okok can repro [15:42:02] wanna bc? [15:42:52] elukey, actually any comand that I use in hive fails with the same error [15:42:56] even selec 1; [15:43:01] select 1; [15:43:01] nono I got what's wrong [15:43:05] or use mforns; [15:43:06] puppet, luca is stupid [15:45:40] fix incoming [15:46:33] checking pingback in the meanting [15:46:36] meantime [15:47:10] mforns: the reports are currenlty not rsynced to thorium, I discovered that one setting is missing too [15:47:16] for the rsync to work [15:47:28] aha [15:47:34] how do you check if those get published? [15:47:43] good that you saw that, no idea where that goes [15:47:58] I go to.. [15:48:06] I trusted alarms yesterday, my bad [15:48:15] (and one run of a RU job, that was good) [15:48:21] https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/ [15:51:37] ok now hive cli works [15:51:56] yay! [15:52:02] testing [15:52:07] just restarted reportupdater-reference-previews.service [15:52:41] ok [15:52:48] that seems working [15:53:18] aha, my test query seems too [15:53:25] yea :] [15:53:47] mforns: ok so now my question is, how does RU authenticate with hive? [15:54:19] RU runs a script via Popen I think [15:54:27] and the script uses hive -e "..." [15:54:46] ok but without kerberos run command? [15:54:50] yes [15:54:53] without [15:55:26] and how does that work? [15:55:40] mforns: bc? [15:55:53] ¯\_(ツ)_/¯ [15:55:55] yes [15:56:35] mforns: it is busy [15:56:38] let's use something else [15:56:44] http://bit.ly/a-TARDIS [15:56:53] use the TARDIS :) [15:56:56] (it's lonely) [15:56:56] ok [15:56:58] ack! [16:00:25] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: Add Fsalutari to nda LDAP group - https://phabricator.wikimedia.org/T245997 (10MoritzMuehlenhoff) If the user already has shell access, no additional entry is needed. [16:18:14] 10Analytics: create kerberos identity for jmorgan - https://phabricator.wikimedia.org/T246118 (10Capt_Swing) [16:18:37] 10Analytics: create kerberos identity for jmorgan - https://phabricator.wikimedia.org/T246118 (10Capt_Swing) [16:23:35] drum roll [16:23:42] we've never kerberized RU [16:24:02] it was working on stat1007 since other timers were populating the analytics cred cache [16:24:14] * elukey plays sad_trombone.wav [16:24:18] OHHh haha cool [16:24:28] not so bad tho! [16:25:34] 10Analytics, 10Fundraising-Backlog, 10fundraising-tech-ops: Install superset on front end server for analytics - https://phabricator.wikimedia.org/T245755 (10EYener) It would be good to touch base, @Milimetric - I'll find time on your calendar for later next week. Please feel free to add others to the meeti... [16:29:47] elukey: That story is worth a blog post :) [16:30:23] elukey: on a less funny aspect - Shouldn't we sret a rule about using analytics creds only on specific machines to prevent easy-reusing? [16:30:45] joal: ? [16:31:36] elukey: using for instance an-coord1001 when doing opsy stuff for analytics creds-cache not to be reusable on other machines [16:31:36] in theory after I move the jobs that we own to an-launcher I'll remove the analytics keytab from all stats [16:31:42] \o/ [16:31:48] this answers my question :) [16:31:51] ah okok :D [16:31:53] Thanks elukey :) [16:32:24] mforns: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/574786/ [16:32:29] * joal should put a not on always thinking elukey has thought about about my thoughts before me, and is actually acting on it [16:32:42] ah no the patch is not correct [16:32:59] sorry mforns 1 min :D [16:34:11] np [16:34:28] Hello! I am trying to move files from notebook to stat with rsync. it doesn't seem to work. [16:35:07] djellel: --verbose :) [16:35:36] https://www.irccloud.com/pastebin/RMAElZvj/ [16:36:18] mforns: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/574786/ [16:36:33] elookin' [16:37:17] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [16:38:40] mforns: just send another patch, simplified a bit, pcc was complaining [16:39:33] djellel: our fault, hosts allow = notebook*.*.wmnet localhost [16:40:00] I'll chat with ottomata to add stat/notebook rsync [16:40:07] is it urgent? [16:40:08] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [16:40:47] elukey, you removed the absent? [16:41:08] mforns: yes exactly [16:41:09] won't that leave lots of jobs there to clean manually? [16:41:24] elukey: not urgent. I will try to move things through hdfs [16:41:29] mforns: in theory no, they will be updated with kerberos-run-command things [16:41:32] ok [16:41:58] mforns: see https://puppet-compiler.wmflabs.org/compiler1002/21054/an-launcher1001.eqiad.wmnet/ [16:42:38] looks good [16:43:12] aha [16:45:38] all right done [17:00:16] is https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os supposed to be empty? [17:01:28] no... [17:14:22] elukey: the browser folder is missing from https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/ which means something is rsyncing from where we deleted it? [17:14:47] (that's why the dashboard above is empty) [17:15:45] milimetric: yes I am still fixing the rsync, but I know why it disappeared [17:15:48] lemme try to fix [17:19:44] elukey: is it on an-launcher /srv/published? [17:19:51] is it in stat1007 /srv/published? [17:20:00] there shouldn't be any deleteing from the published rsync stuff [17:20:13] ottomata: it was my bad, I moved /srv/reportupdater to a backup dir on stat1007 [17:20:24] an-launcher1001 is still not in the stat servers list [17:20:42] hm, but, why would it disappear? we don't rsync wiwth --delete for published stuff [17:20:55] I have no idea [17:38:11] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Nuria) [17:39:30] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Nuria) a:03Ottomata [17:41:22] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Nuria) Let's also document clearly how to re-start the kernel [17:42:21] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Analysts cannot reliably use wmfdata to run SQL queries against Hive databases - https://phabricator.wikimedia.org/T245891 (10nshahquinn-wmf) [17:52:33] elukey: I'm trying to ops-week an issue with a Turnilo config, but https://wikitech.wikimedia.org/wiki/Analytics/Systems/Turnilo seems out of date. I'll be digging around puppet to update it, FYI just in case you did something similar or have thoughts [18:19:39] 10Analytics: Dumps NFS mounts not available on stat1006 - https://phabricator.wikimedia.org/T243775 (10elukey) 05Open→03Resolved a:03elukey [18:26:22] 10Analytics, 10Product-Analytics: wmfdata cannot recover from a crashed Spark session - https://phabricator.wikimedia.org/T245713 (10Nuria) @nshahquinn-wmf Kerberos tickets expire after 1 day, once they do spark context will no longer work, so you need to restart your kernel entirely. We will be extending expi... [18:27:45] 10Analytics, 10Epic, 10Product-Analytics (Kanban): Spark applications crash when running large queries - https://phabricator.wikimedia.org/T245896 (10Nuria) Per our conversation, this can be somewhat alleviated with better settings but hive is a better alternative for large amounts of data. I think @Ottomata... [18:37:09] 10Analytics: create kerberos identity for jmorgan - https://phabricator.wikimedia.org/T246118 (10elukey) Please check your inbox :) ` elukey@krb1001:~$ sudo manage_principals.py create jmorgan --email_address=jmorgan@wikimedia.org Principal successfully created. Make sure to update data.yaml in Puppet. Successf... [18:50:27] milimetric: i think turnilo docs are fine , do you see any issues? [18:51:01] yes sorry forgot to follow up, what is the issue? [18:51:15] nuria: server and repos are the old ones I think [18:51:34] it’s ok, updating per ops week [18:51:46] (but I need a lunch break now) [19:07:55] * elukey off! [19:53:54] (03CR) 10Joal: "2 comments, nothing major" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/571365 (https://phabricator.wikimedia.org/T244771) (owner: 10Ottomata) [20:01:09] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 6 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) [20:03:57] 10Analytics: kinit "Failed to store credentials" error - https://phabricator.wikimedia.org/T246151 (10dr0ptp4kt) [20:17:37] Is there a utility to process the current version wikipedia of a given language on the cluster? I am going through the painful process of downloading the whole dump and I have a script based on mwparserfromhell to extract simple things such as all Wikilinks. This takes forever on a single machine especially for enwiki. [20:23:22] 10Analytics: Should reportupdater Pingback reports be refactored? - https://phabricator.wikimedia.org/T246154 (10mforns) [20:30:44] joal: yt? [20:30:48] yup [20:31:02] can 't figure out how to make an e.g setSchema more readable [20:31:12] ottomata: batcave? [20:31:14] i had a var dfReader for a second [20:31:17] sure [20:35:20] (03CR) 10Ottomata: Refine - Warn when merging incompatible types; FAILFAST when reading JSON data with a schema (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/571365 (https://phabricator.wikimedia.org/T244771) (owner: 10Ottomata) [20:52:35] 10Analytics, 10Product-Analytics: Presto: missing partitions causes queries to fail - https://phabricator.wikimedia.org/T246034 (10mforns) I recreated tables plus repaired partitions of event_sanitized.helppanel, event_sanitized.homepagemodule and event_sanitized.homepagevisit. This eliminated loose Hive parti... [20:53:08] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Presto: missing partitions causes queries to fail - https://phabricator.wikimedia.org/T246034 (10mforns) a:03mforns [20:55:21] (03CR) 10Nuria: [C: 04-1] Add wikidata item_page_link spark job (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572746 (owner: 10Joal) [21:03:17] (03CR) 10Nuria: [C: 04-1] "There are no chnages on patch 2?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572726 (https://phabricator.wikimedia.org/T245453) (owner: 10Joal) [21:04:06] (03PS5) 10Ottomata: Refine - Warn when merging incompatible types; FAILFAST when reading JSON data with a schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/571365 (https://phabricator.wikimedia.org/T244771) [21:04:14] 10Analytics, 10Analytics-Kanban: Make history and current wikitext available in hadoop - https://phabricator.wikimedia.org/T238858 (10Nuria) 05Open→03Resolved [21:04:16] 10Analytics: Provide data dumps in the Analytics Data Lake - https://phabricator.wikimedia.org/T186559 (10Nuria) [21:06:15] (03CR) 10Nuria: Pass spark_job_jar as an argument in ArticlePlaceholder oozie job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/572713 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [21:07:07] (03CR) 10Joal: [C: 03+2] "Thanks for the change :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/571365 (https://phabricator.wikimedia.org/T244771) (owner: 10Ottomata) [21:08:35] (03CR) 10Joal: "The 2nd patch was a rebase after deploy. Let's discuss on whether to force qualifiers to be in that list or not." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572726 (https://phabricator.wikimedia.org/T245453) (owner: 10Joal) [21:10:24] 10Analytics: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10Varnent) To repo the outcome, go to the list of pages, click on the page overlay option, and see if the overylays actually appear. The page itself loads, but the overlays do not appear. Having others check as well - and... [21:11:29] (03Merged) 10jenkins-bot: Refine - Warn when merging incompatible types; FAILFAST when reading JSON data with a schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/571365 (https://phabricator.wikimedia.org/T244771) (owner: 10Ottomata) [21:11:43] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Refining is failing to refine centranoticeimpression events - https://phabricator.wikimedia.org/T244771 (10Nuria) @ottomata: can we alter table to latest schema and re-refine the last 90 days of data? [21:13:11] 10Analytics, 10Fundraising-Backlog, 10WMDE-Analytics-Engineering, 10WMDE-FUN-Team, 10WMDE-Fundraising-Tech: Find a better way for WMDE to get impression counts for their banners - https://phabricator.wikimedia.org/T243092 (10Nuria) >Do I understand correctly, that event.centralnoticeimpression does not c... [21:15:28] 10Analytics, 10Fundraising-Backlog, 10WMDE-Analytics-Engineering, 10WMDE-FUN-Team, 10WMDE-Fundraising-Tech: Find a better way for WMDE to get impression counts for their banners - https://phabricator.wikimedia.org/T243092 (10Nuria) Now, I think if you rely on that data source strongly you need to communi... [21:15:31] (03CR) 10Joal: "@nuria: It looks like you'd prefer the job to directly update the hive table instead of generating files as we do for other similar jobs. " (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572746 (owner: 10Joal) [21:16:24] (03PS7) 10Joal: Add wikidata item_page_link spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572746 (https://phabricator.wikimedia.org/T244707) [21:17:44] (03CR) 10Nuria: "I think generating files is fine, now, let's make clear on the job this is one step of many" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572746 (https://phabricator.wikimedia.org/T244707) (owner: 10Joal) [21:18:13] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review: Refining is failing to refine centranoticeimpression events - https://phabricator.wikimedia.org/T244771 (10Nuria) Selecting from table now I get: ` hive (event)> select * from CentralNoticeImpression where year=2... [21:18:28] @ottomata can you take a look at https://phabricator.wikimedia.org/T244771 [21:19:56] (03CR) 10Joal: Add wikidata item_page_link spark job (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572746 (https://phabricator.wikimedia.org/T244707) (owner: 10Joal) [21:21:12] joal: yt? [21:21:16] Yes ! [21:21:19] jaja [21:21:25] joal: in this one https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/553726/3/refinery-hive/src/main/java/org/wikimedia/analytics/refinery/hive/GetGeoDataUDF.java [21:21:36] yes [21:23:17] joal: why suynchronize? [21:25:38] joal: also from commit change i cannot tell why the initialization of the db needs fixing [21:25:40] nuria: you suggested that in a previous comment, and I understood it as a need to prevent calls from `configure` and `initialize` to be done async [21:26:19] joal: let me understand, what was broken here that led to the udf not work all of a suddenm in a plain hive script? [21:26:41] nuria: UDF was not working in local-task [21:26:52] when hive launches a single local-task as an optim [21:26:55] joal: because? [21:27:09] because DB was not initialized properly [21:28:12] joal: and why did it stopped being initialized properly, do we know? cause that used to work before [21:28:14] nuria: In map-reduce context, initialization is done through the 'configure' method [21:28:27] joal: aham [21:29:51] nuria: and when not in map-reduce, initilization needs to be done in a different place - See comments line 118+ [21:30:29] joal: but what constitutes "local execution" [21:30:34] About the reason it didn't bit us before: we changed config in hive to prevent local-tasks at some point, since it was making jobs fail due to memory issues [21:30:41] nuria: let's batcave :) [21:30:54] joal: sorry i cannot, i am in a super noisy place [21:31:07] nuria: ok - writing :) [21:31:13] joal: but that is ok, you can explain tomorrow! [21:31:14] really [21:31:44] hive triggers local-execution as an optimization - For instance when you do : select from bah limit 10 [21:31:56] Only 10 rows are needed, read from files [21:32:33] Instead of starting a headvy map-reduce, hive starts a local-task that reads the file and print - a lot faster [21:33:02] At some point we disabled that optim, cause some jobs where failing in local-mode optimisation due memory settings [21:33:33] Memory changes have been made, local mode reenabled, and geo-udf started to fail in local-mode scripts [21:33:36] nuria: --^ [21:35:54] joal:ok get it [21:36:20] \o/ :) [21:37:00] joal: but I am sorry i suggedted the synchronization, * i think* that is wrong [21:37:11] possible :) [21:37:51] nuria: I don't know how configure and initialize are called - If in same thread we're good - if not, sync is usefull [21:41:36] (03CR) 10Joal: "I think that patch should be abandonned in favor of the approach using normalized_project (https://gerrit.wikimedia.org/r/#/c/analytics/re" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/572713 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [21:41:42] (03CR) 10Joal: "I think that patch should be abandonned in favor of the approach using normalized_project (https://gerrit.wikimedia.org/r/#/c/analytics/re" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572711 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [22:11:38] 10Analytics: Should reportupdater Pingback reports be refactored? - https://phabricator.wikimedia.org/T246154 (10CCicalese_WMF) That is disappointing. What's the next step in getting them working again? [22:34:48] 10Analytics: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10EdErhart-WMF) The overlays are not loading for me either. Windows machine on Firefox. (To be clear, at the link provided you need to hover over "index" and click "open page overlay") [22:40:39] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: Add Fsalutari to nda LDAP group - https://phabricator.wikimedia.org/T245997 (10Dzahn) 05Open→03Resolved This seems done. Confirmed user is already in data.yaml and he @Fsalutari confirmed he can login. Resolving. [22:46:06] 10Analytics-Kanban, 10Patch-For-Review: Productionize streaming jobs - https://phabricator.wikimedia.org/T176983 (10Nuria) [22:47:38] 10Analytics: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10Nuria) a:03Milimetric [22:47:51] 10Analytics: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10Nuria) Assigning to @Milimetric per ops week [22:48:03] 10Analytics, 10Analytics-Kanban: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10Nuria) [22:52:43] 10Analytics: Should reportupdater Pingback reports be refactored? - https://phabricator.wikimedia.org/T246154 (10Nuria) @CCicalese_WMF The queries need to be entirely rewritten so they do not scan the whole table all records every time. We would like to suggest that this would be a good item to work on for someo... [22:53:52] djellel: if you are there ping us again, the dumps a re available on the cluster in a batter format to parse them than xml [22:54:02] djellel: a ticket also would work [22:54:26] ottomata: when you can can you take a look at https://phabricator.wikimedia.org/T244771 [23:20:07] hey a-team, looks like notebook1004 has run out of disk space? [23:33:10] PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1004 is CRITICAL: connect to address 10.64.36.107 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration%23Fixing_HDFS_mount_at_/mnt/hdfs [23:58:43] (03PS3) 10Nuria: Fix webrequest host normalization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/572726 (https://phabricator.wikimedia.org/T245453) (owner: 10Joal)