[07:28:05] Good morning [08:44:51] (03PS1) 10Joal: Update oozie jobs parameters for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533861 (https://phabricator.wikimedia.org/T231787) [09:24:09] (03CR) 10Ladsgroup: [C: 03+2] Track *.jar files as git lfs [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/533507 (https://phabricator.wikimedia.org/T230015) (owner: 10Ladsgroup) [09:24:18] (03Merged) 10jenkins-bot: Track *.jar files as git lfs [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/533507 (https://phabricator.wikimedia.org/T230015) (owner: 10Ladsgroup) [09:48:25] hello joal (starting a bit late today), whenever you want we can deal with those jobs that need restarting :) [09:49:04] fdans: I'm actually providing more and moar code-reviews that would take benefit of all-jobs-restart [09:49:24] fdans: So I'd like to wait for reviews and possible merge before moving if ok for you :) [09:49:36] actually I'm gonna add you as a reviewer to that last one [09:49:44] sounds good! [09:49:48] (you're gonna hate me) [09:50:14] Done [09:50:34] And fdans, please excuse my rudeness - Good morning :) I hope you're good [09:50:49] yes joal how dare you [09:50:51] :D [09:52:33] :) [10:13:16] holaaa [10:14:20] fdans: did you see the geoeditors job that failed? [10:14:28] holaaa a-team [10:15:01] jelo [10:15:07] Hi nuria [10:15:29] nuria: was just looking at it, I started a lil late today [10:15:44] fdans: ok, let us know what you find [10:17:07] fdans: do ping if you need help [10:18:24] joal: a 98-file code review is what everyone needs to start the week right! ;) [10:18:46] I have looked at it and provided a major CR (francisco noticed) about the issue [10:18:51] nuria: --^ [10:19:01] sorry fdans :S [10:19:26] nono :) thank you for doing this work [10:19:35] fdans: I tried to find 2 more files to get a round number but didn't [10:22:13] joal: so the jdbc url is what made the geoeditors job fail? this one https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/533861/? [10:22:33] (03CR) 10Fdans: [C: 03+1] "Looked at all files, couldn't find anything weird. Thanks for doing this!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533861 (https://phabricator.wikimedia.org/T231787) (owner: 10Joal) [10:23:00] Correct nuria - the 2 load jobs (geoeditor and mediawiki-history) failed becasue of missing jdbc-url param [10:23:25] I have updated all workflow/coords to make the parameter mandatory, for fail-fast [10:24:02] And I have also updated more edits-hourly and data-quality adding missing parameters, for the same reason [10:24:54] joal: and the change to remove hive-site.xml? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/533861/1/oozie/banner_activity/druid/monthly/coordinator.properties ? [10:25:30] nuria: parameter not needed, this job doesn't use hive [10:26:09] nuria: I used search-tricks to find files to update, and found other discrepencies in config, and corrected them [10:31:42] nuria: joal "javax.servlet.jsp.el.ELException: variable [hive2_jdbc_url] cannot be resolved" [10:31:57] yes nuria [10:31:59] joal: some good audit here, and this one? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/533861/1/oozie/data_quality/hourly/bundle.xml [10:32:45] I updated the same way as for the jdbc-url param, and added the other ones [10:33:47] nuria: parameters in oozie are passed from bundle to coord to workflow even if not defined in the param section - defining them in the parameters section makes them mandatory even if not used [10:34:00] joal: ahahahaha [10:34:14] Therefore Iadded all the needed params here, preventing to have to wait for a workflow execution to see an error because of a missing param [10:34:25] nuria: in meeting, will be bac kt otalk in 1/2h [10:34:38] joal: nice, that is something for all of us to look in CRs [10:36:18] (03CR) 10Nuria: [C: 03+2] "Thanks for doing this massive audit" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533861 (https://phabricator.wikimedia.org/T231787) (owner: 10Joal) [11:07:13] nuria: about geoeditors, I re-added wikidata portion after the comment from leila - Your comment looks like it's ok - I'm gonna add doc to this page: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors [11:07:40] nuria: Are you ok with me merging the patch? [11:08:21] joal: it's fine, yes. Going forward let's not add work such us this that nobody has request it , it is another piece to maintain that, at this time we do not need [11:08:47] nuria: I heard leila's comment as a request, but I hear your point [11:08:54] Thanks :) [11:09:32] joal: the GII are the users of this data and at this time (and probably next year as well) thay will not be using wikidata's data [11:09:34] *they [11:10:23] nuria: in the task nuria mentionned being after wikidata to try to push itto them as innovation index - even if not needed per say, having the data is mandatory to do what leila wants [11:10:34] s/nuria/leila sorry [11:11:03] (03PS5) 10Joal: Update geoditors-yearly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533169 (https://phabricator.wikimedia.org/T215655) [11:11:14] joal: i rather approach it the other way around, if when GII needs that data it can be added [11:12:23] (03PS6) 10Joal: Update geoditors-yearly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533169 (https://phabricator.wikimedia.org/T215655) [11:12:38] sorry for the spam, trying to get my commit-message fixed [11:13:01] ok looks good - merging [11:13:12] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533169 (https://phabricator.wikimedia.org/T215655) (owner: 10Joal) [11:15:48] (03PS2) 10Joal: Update oozie jobs parameters for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533861 (https://phabricator.wikimedia.org/T231787) [11:16:25] joal: making a list of jobs to restart/reschedule to be certain [11:16:38] ok fdans :) Thanks for that [11:16:48] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533861 (https://phabricator.wikimedia.org/T231787) (owner: 10Joal) [11:18:52] (03PS3) 10Joal: Add SLA-email alerts to all oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533173 (https://phabricator.wikimedia.org/T228747) [11:19:33] ok - that one --^ is the last one I'd like to see merged before we deploy and restart everything - Waiting for Marcel to discuss comments [11:23:47] joal: hmmm so we are restarting every single oozie job right? [11:23:59] fdans: almost :D [11:24:15] fdans: maybe not even almost, possibly all [11:24:43] The Analytics Restartapocalypse of September 2019 [11:25:23] :) [11:39:20] (03CR) 10Joal: "Answers to all comments, let's discuss them :)" (0312 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533173 (https://phabricator.wikimedia.org/T228747) (owner: 10Joal) [11:39:48] (03PS4) 10Joal: Add SLA-email alerts to all oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533173 (https://phabricator.wikimedia.org/T228747) [13:53:06] Gone for kids [14:13:02] heya teammm [14:39:39] * fdans is very suspicious that today everything's working on the first try [14:47:37] (03PS1) 10Fdans: (wip) Add cassandra loading job for requests per file metric [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533921 (https://phabricator.wikimedia.org/T228149) [15:01:34] ping fdans standdup [15:38:25] a-team: I'm having a problem installing the nbdime package (used to diff Jupyter notebook) with pip on notebook1003. The main issue seems to be `ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/srv/home/neilpquinn-wmf/venv/lib/python3.5/site-packages/tornado-6.0.3.dist-info/METADATA'` [15:38:41] However, I also get `WARNING: No metadata found in ./venv/lib/python3.5/site-packages` [15:39:51] Full output of pip: https://phabricator.wikimedia.org/P9025. Let me know if I should give the verbose version as well [15:40:24] It worked correctly on notebook1004 so it may be an issue with another package I have installed, but I don't know how to troubleshoot [15:40:30] Hi neilpquinn - Our op is away today - I don't think you'll get an answer before tomorrow :( [15:40:56] joal: okay, thanks. I guess I'll file a Phab task then :) [15:41:06] best idea neilpquinn :) [16:07:04] neilpquinn: nbdime requires tornado [16:08:52] nuria: but tornado is already installed. when I run `pip install --upgrade tornado` I get the following: [16:08:57] https://www.irccloud.com/pastebin/PAQWc7jl/ [16:11:06] neilpquinn: I would try deleting tornado and reinstalling it [16:11:27] nuria: I just tried...both operations fail with that same error message [16:11:43] neilpquinn: is there a metadatafile on /srv/home/neilpquinn-wmf/venv/lib/python3.5/site-packages/tornado-6.0.3.dist-info/? [16:11:58] neilpquinn: metadata something (lower case)? [16:12:33] nuria: there is a metadata.json file [16:12:53] should I symlink the METADATA file it wants to that? [16:13:05] neilpquinn: do copy it to a file called METADATA and see whether that fixes matters, be aware of permits [16:14:16] neilpquinn: copy, rather than symlink [16:20:18] nuria: it turns out either copying or symlinking solved it. Thank you so much 😁 [16:20:28] neilpquinn: np [16:21:32] joal, I keep reading you changed the docs at /pageview/hourly but I can not find such directory in the review. I think I'm missing something... [16:23:25] mforns: I must have messed the name :( Checking [16:24:43] mforns: It was actually the projecview/hourly coordinator - Sorry about that :( [16:24:57] oh! ok, no problem at all! [16:26:01] ok, I see, thanks! [16:32:09] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533173 (https://phabricator.wikimedia.org/T228747) (owner: 10Joal) [16:32:27] joal, merged it, thanks for waiting :] [16:32:44] mforns: Thank you for the review !! [16:33:37] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533173 (https://phabricator.wikimedia.org/T228747) (owner: 10Joal) [16:34:24] Gone for diner [16:49:21] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/531148 (https://phabricator.wikimedia.org/T230514) (owner: 10Fdans) [16:49:31] thank you mforns [16:49:43] np fdans :] [16:52:01] (03PS2) 10Fdans: (wip) Add cassandra loading job for requests per file metric [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533921 (https://phabricator.wikimedia.org/T228149) [16:53:46] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/533629 (https://phabricator.wikimedia.org/T228557) (owner: 10Nuria) [16:54:04] (03PS1) 10Fdans: Release 2.6.7 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/533951 [16:57:53] (03PS2) 10Fdans: Release 2.6.8 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/533951 [16:58:18] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.6.8 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/533951 (owner: 10Fdans) [17:00:56] (03Merged) 10jenkins-bot: Correcting column name as spark is case sensitive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/533629 (https://phabricator.wikimedia.org/T228557) (owner: 10Nuria) [17:03:39] fdans: FYI that my refinery-source change does not need to be deployed this week , can be so next week cause i think we have enough with restart-apocalipse [18:40:36] a-team: I'm been having all sorts of package-related problems ever since I tried to install nbdime (compounded by my troubleshooting attempts). Can someone reset my virtual environment? https://wikitech.wikimedia.org/wiki/SWAP#Resetting_user_virtualenvs [18:42:34] oh, important detail: this is on notebook1003 [19:04:00] neilpquinn, trying now! [19:05:01] neilpquinn, is your username neilpquinn-wmf ? [19:05:25] mforns: yes, that's me. thank you! [19:05:32] cool thx [19:13:02] neilpquinn, I don't think the analytics user (the one I have sudo) is allowed to restart your service in notebooks1003... [19:13:11] I'm getting permission issues... [19:15:08] neilpquinn, (a-team correct me if I'm wrong) but I think you need an ops sudoer, and exceptionally today we Analytics don't have one... [19:21:41] mforns: okay, I will ask again tomorrow [19:21:48] thanks for trying :)