[02:19:41] 10Analytics: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10MMiller_WMF) [07:03:45] 10Analytics, 10Analytics-Kanban, 10ArticlePlaceholder, 10Wikidata, and 4 others: ArticlePlaceholder dashboard stopped tracking page views - https://phabricator.wikimedia.org/T236895 (10Ladsgroup) a:05Ladsgroup→03None We are at middle of something rather big, I don't think we can work on it for the next... [07:30:54] !log restart turnilo after updating the webrequest_sampled_128's config [07:30:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:33:50] !log kill test_elukey_webrequest_sampled_128 from druid [07:33:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:38:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add request_bytes as measure in Druid's webrequest_sampled_128 - https://phabricator.wikimedia.org/T240681 (10elukey) Just checked, all done! [07:45:23] !log re-run failed data-quality-stats-event.navigationtiming-useragent_entropy-hourly-coord 2020/01/09T00 [07:45:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:57:59] 10Analytics, 10Datasets-Archiving, 10Research-Backlog: Make HTML dumps available - https://phabricator.wikimedia.org/T182351 (10ArielGlenn) @leila I still really want these to happen. As RESTbasse moves towards being phased out I'm trying to have the discussion about access to its replacement and how we migh... [08:05:23] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Erik_Zachte) @Nuria are you saying a fix that might take an hour, if not less, is not done, because another issue might popup in the future? It's not that you're committing for eternity to uph... [08:12:38] good morning elukey [08:14:21] 10Analytics: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) Sigh :( Let's re-analyze logs then! The error starts from the Hive2 server: ` 2020-01-09 02:17:20,476 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1927)) - PriviledgedActionException as:mmiller (aut... [08:16:13] this problem is weird --^ :( [08:35:13] joal: bonjour! It is yes, but from the logs it seems that the patch already applied should work [08:35:38] my next step is to try the zookeeper setting with Hadoop test, to see if znodes are created [08:35:48] maybe the db option is not available for hive 1.2? [08:36:01] I tried to look for when it was introduced but didn't find much [08:36:08] (I am currently checking another thing) [08:36:13] elukey: I kinda wonder about that - the patch you merged should work for restarts, this I understand - The error case here seems weirder - Or maybe it;s a leftover from yesterday restart? [08:39:49] 10Analytics, 10Pywikibot, 10Wikimedia-Site-requests, 10User-Urbanecm: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Urbanecm) Bot usernames are (mostly) connected with exactly one operator. As said in previous comment, I'm happy to contac... [08:43:46] joal: in theory no, we haven't restarted anything related to hive (server/metastore) and hue right? [08:44:10] elukey: haven't we restarted for the patch? Ah it was the day before [08:44:50] ahhh okok yes I did it [08:44:54] and also restarted hue [08:44:57] and everything worked [08:45:01] but then it stopped again [08:45:24] now the timing matches, more or less, to when I applied the limits for the hive queries to hue [08:45:49] I am wondering if those limits are causing this [08:46:05] we haven't seen the issue before [08:46:27] but there were holidays in the middle.. [08:46:37] we could try to revert the limits, and see [08:50:46] 10Analytics: Check home leftovers of tieu - https://phabricator.wikimedia.org/T241713 (10gabriel-wmde) I was a colleague of him and had a look - you can safely delete the home directories. [09:20:55] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10elukey) >>! In T238243#5776581, @Erik_Zachte wrote: >> We are leaning towards turning these jobs off because they've been broken for almost a year and nobody has complained so far. > > I als... [09:31:26] 10Analytics: Check home leftovers of tieu - https://phabricator.wikimedia.org/T241713 (10elukey) 05Open→03Resolved a:03elukey Thanks @gabriel-wmde, all cleaned up! [10:04:27] 10Analytics, 10GLOW, 10User-Elukey: Access to DataGrip refused - https://phabricator.wikimedia.org/T241170 (10elukey) @Iflorez I had a chat with @joal and we discovered that Datagrip doesn't work with ssh tunnels and kerberos, see: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360005112460... [10:27:23] 10Analytics: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) To remove variables I am reverting https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563144/ This was added last week to limit huge Hue/Hive queries and prevent memory exhaustion of the Hue's host, but the timeline is... [10:29:47] 10Analytics, 10Cite, 10Reference Previews, 10Research, and 3 others: Instrument Cite to record the nubmer of footnote marks and references list entries rendered in each article - https://phabricator.wikimedia.org/T241833 (10awight) [10:30:01] !log revert hue's hive query limit and restart hue - T242306 [10:30:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:30:03] T242306: No queries run in Hue - https://phabricator.wikimedia.org/T242306 [10:31:23] 10Analytics: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) @MMiller_WMF Hue now works for me, but as we discovered before the errors affect only some users so let me know if it works now or not. I reverted a change that might be related, let's see if it works. [10:31:42] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) p:05Triage→03High a:03elukey [10:32:58] 10Analytics: Kerberos credentials for jfishback - https://phabricator.wikimedia.org/T242245 (10elukey) ` elukey@krb1001:~$ sudo manage_principals.py create jfishback --email_address=jfishback@wikimedia.org Principal successfully created. Make sure to update data.yaml in Puppet. Successfully sent email to jfishba... [11:10:22] !log remove old accounts (user: absent) from Superset [11:10:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:10:38] this was not a big deal since ldap auth was preventing old users to connect, but still [11:12:38] 10Analytics: Kerberos credentials for jfishback - https://phabricator.wikimedia.org/T242245 (10elukey) 05Open→03Resolved [11:13:05] 10Analytics: Kerberos credentials for dsharpe - https://phabricator.wikimedia.org/T242244 (10elukey) 05Open→03Resolved [11:14:16] 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T241755 (10elukey) 05Open→03Invalid Since this task was created without any explanation I am marking it as invalid :) [11:16:16] 10Analytics: Kerberos password for user mepps - https://phabricator.wikimedia.org/T242222 (10elukey) ` elukey@krb1001:~$ sudo manage_principals.py create mepps --email_address=mepps@wikimedia.org Principal successfully created. Make sure to update data.yaml in Puppet. Successfully sent email to mepps@wikimedia.o... [11:17:18] !log installing cyrus-sasl security updates [11:17:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:17:29] moritzm: wrong chan I think :D [11:17:54] 10Analytics: Requesting kerberos access for snowick - https://phabricator.wikimedia.org/T242046 (10elukey) ` elukey@krb1001:~$ sudo manage_principals.py create snowick --email_address=snowick@wikimedia.org Principal successfully created. Make sure to update data.yaml in Puppet. Successfully sent email to snowick... [11:23:50] good point :) [11:27:23] * elukey lunch! [12:07:57] ottomata: good morning! Let's talk when you have a minute on hdfs-rsync please :) [12:35:57] (03PS12) 10Mforns: Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) [14:10:57] hello joal! [14:14:19] Hi ottomata :) [14:14:46] Are you coffeinated enough for some scala ottomata ? [14:15:40] sure! [14:15:57] batcave ottomata ? [14:16:02] k [14:23:52] 10Analytics: Kerberos password for user mepps - https://phabricator.wikimedia.org/T242222 (10elukey) 05Open→03Resolved [14:24:19] 10Analytics: Requesting kerberos access for snowick - https://phabricator.wikimedia.org/T242046 (10elukey) 05Open→03Resolved [14:25:19] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) [14:38:10] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) @EBernhardson hi! I am looping you in since you are our top spark user :D We are testing encryption for Spark RPCs in Hadoop test, and excluding some Heisenbugs we are in a good st... [14:40:22] ottomata: PR updated - shall I merge or left you do it? [14:41:46] joal: when you have time/patience, can you re-do a pass in Hadoop test to see if Spark now works with encryption? [14:41:55] elukey: I can do that! [14:42:06] <3 [14:42:10] elukey: I'll do it after kids, I won't have time to finish before standup [14:42:20] oh yes even tomorrow [14:57:23] elukey: in https://phabricator.wikimedia.org/T241860 [14:57:29] do you think we should make a new naming convention? [14:57:35] for analytics vs public druids? [14:57:40] as is now we are going to add these to analytics, right? [14:57:44] if we kept current naming [14:57:48] ottomata: hello :) [14:57:48] analytics druid would have [14:58:00] IIRC we thought to add 1 to each cluster [14:58:01] druid100[1-3] + druid100[7-9] [14:58:03] OH [14:58:05] really? [14:58:17] OH [14:58:21] makes sense!~ [14:58:23] ok [14:58:25] still naming will be weird tho [14:58:26] that was my understanding but we can change it! [14:58:35] yes I agree, you are right [14:58:43] an-druid? [14:58:52] maybe we can keep the other druid as just druid? [14:59:16] could be an option yes, even if the other one falls in the analytics realm too.. [14:59:26] analyitcs realm, but not analytics vlan ! :) [14:59:43] yes yes [14:59:44] and we call it the druid_analytics cluster [14:59:48] I am not opposing :) [14:59:51] Gone for kids - ottomata may I let you merge hdfs-rsync please? I'll release later today if you want [14:59:54] k will note in ticket [15:00:00] joal: ok! did you psh? [15:00:01] push* [15:00:05] I did :) [15:00:10] I actually push FORCE :) [15:01:36] elukey: [15:01:36] https://wikitech.wikimedia.org/w/index.php?title=Infrastructure_naming_conventions&type=revision&diff=1850173&oldid=1849724 [15:02:16] ack +1 [15:02:32] joal: were you going to make the skip function not return the result? [15:02:41] i guess it doesn't matteer that much, just thought that is what we dicussed [15:02:49] just make it a logging wrapper [15:02:51] and explicitly return [15:03:00] sincee now skip is only ever used to return None [15:03:21] will comment!~ [15:09:06] elukey: do we use 10G for druids? [15:09:11] could be useful for ingestion? [15:10:27] I'd say no judging from the grafana metrics, we should be good with 1g [15:10:32] yeah i think we dno't [15:10:32] ok [15:12:34] speaking of 10g, I'd need to ping John to install the new cards on jumbo! [15:19:42] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Switch all eventgate clients to use new TLS port - https://phabricator.wikimedia.org/T242224 (10Ottomata) [15:20:36] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Switch all eventgate clients to use new TLS port - https://phabricator.wikimedia.org/T242224 (10Ottomata) @Joe Q for you. If php-fpm will not do HTTPS requests under load...what should we do for TLS everywhere with... [15:20:50] oh yeah! did we get new nodes there too? i can't remember [15:20:59] (am clearly not following this hw stuff as much as I used to...) [15:30:20] not yet, only new cards as far as I know [15:32:53] k [15:46:48] joal: https://www.ericlin.me/2018/06/oozie-spark-action-not-loading-spark-configurations/ :O [15:46:56] also ottomata --^ [15:47:05] this seems really nice [15:47:44] I noticed that the setting that is required to load Spark configuration is missing: oozie.service.SparkConfigurationService.spark.configurations ???? [15:47:50] WHY WOULD OOZIE HAVE SUCH A SETTING?! [15:47:51] hahah [15:48:21] ??? [15:49:42] https://oozie.apache.org/docs/4.2.0/oozie-default.xml [15:49:44] seems supported [15:49:48] elukey that was a quote from the article [15:49:49] i mean [15:49:55] why would oozie ever NOT want to load spark confs? [15:50:01] i would think that would be up to spark [15:50:37] but oozie runs spark actions, and we list settings in every workflow [15:50:56] something that we'd need to do for spark.authentica=true as well [15:51:11] yes souunds good! but i mean, it is weird that /etc/spark/conf isn't used by default [15:51:12] I consider it something like spark-submit no? [15:51:17] ahhhh [15:51:18] yes yes [15:51:55] it uses spark-conf in the oozie's main dir [15:52:12] so if we put something like *=/etc/spark2/conf it should pick them up [15:52:22] (03PS1) 10Mforns: Add anomaly detection to data quality stats workflow [analytics/refinery] - 10https://gerrit.wikimedia.org/r/563200 (https://phabricator.wikimedia.org/T235486) [15:52:40] huh ok [15:52:42] aye cool [15:52:52] if this work it would be a big relief so we'd avoid to change workflows in oozie [15:53:14] *works [15:54:15] (03CR) 10Mforns: [C: 04-2] "Still WIP." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/563200 (https://phabricator.wikimedia.org/T235486) (owner: 10Mforns) [15:54:36] yeah [15:58:37] noooooo it is only for oozie 4.2 [15:58:45] * elukey flips the table [15:58:58] haha [15:59:07] (╯°□°）╯︵ ┻━┻ [16:00:08] but! [16:00:09] https://archive.cloudera.com/cdh5/cdh/5/oozie/oozie-default.xml [16:00:19] lists it! [16:01:03] so possibly our version is a frankenstein [16:01:10] like the rest of the cdh packages [16:02:23] worth a try! [16:02:33] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) @joal I found https://www.ericlin.me/2018/06/oozie-spark-action-not-loading-spark-configurations/ today, there is an option listed that seems good to test: ` o... [16:03:01] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) a:03elukey [16:04:55] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Operations, and 5 others: Public EventGate endpoint for analytics event intake - https://phabricator.wikimedia.org/T233629 (10Ottomata) [16:09:37] 2020-01-09 16:08:58,351 INFO SparkConfigurationService:520 - SERVER[analytics1030.eqiad.wmnet] Loaded Spark Configuration: *=/etc/spark2/conf/spark-defaults.conf [16:09:43] yesssss [16:10:09] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) Applied manually on analytics1030 in Hadoop test: ` 2020-01-09 16:08:58,351 INFO SparkConfigurationService:520 - SERVER[analytics1030.eqiad.wmnet] Loaded Spark Configuration: *=/e... [16:20:50] 10Analytics, 10Pywikibot, 10Wikimedia-Site-requests, 10User-Urbanecm: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Lokal_Profil) >>! In T242157#5785171, @Urbanecm wrote: > Maybe we can add the deprecation to the tech news, or even massme... [16:32:37] are we doing standup or staff meeting today? :) [16:35:09] last time we did standup and goals [16:39:22] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Operations, and 5 others: Public EventGate instance and endpoint for analytics event intake: eventgate-analytics-external - https://phabricator.wikimedia.org/T233629 (10Ottomata) p:05Normal→03High [16:48:39] ottomata, nuria is not here I think, so you decide! [16:48:46] as overlord [16:50:26] i've never really been to a staff meeting [16:50:28] let's go to staff! [16:51:05] joal: https://grafana.wikimedia.org/d/kUVKEvaWz/cassandra-storage :) [16:51:08] really nice! [16:55:11] ok! [16:59:47] a-team, will be 2 min late, sorry [17:00:02] fdans, we're going to staff today [17:00:32] ok staff meeting it is then [17:01:07] ottomata: excuse me I did half the work at push - release excitement [17:02:03] ok! :) [17:02:46] ah ok thanks mforns!!! [17:08:19] could i get someone with hdfs "root" (whatever it is) to chown /wmf/data/discovery/mjolnir/* directories to analytics-search user? [17:08:52] ottomata: In order not to repeat the 'None', is that ok if I keep the 'skip' function without the res param send None? [17:10:09] ebernhardson: sure - give me a minute [17:10:34] joal: thanks! [17:11:25] ebernhardson: shall I change /wmf/data/discovery/mjolnir itself as well ? [17:11:31] joal: if you prefer i think that's ok [17:11:44] ack ottomata - thanks [17:11:58] i think it is more readable the other way, and makes the skip function a simple logging wrapper, but ya you can just make the ret type of skip always None [17:12:18] ottomata: I like less None lines :) [17:12:27] but if you prefer the readability of the other, I'll do it :) [17:13:16] ebernhardson: launched, including parent folder (mjolnir) [17:13:47] joal: excellent! cleaning up my mess from being lazy and running this myself... [17:14:04] no bother ebernhardson - done! [17:17:32] ottomata: branch force-pushed :) [17:21:19] merged [17:21:25] \o/ [17:21:29] Thanks ottomata [17:47:08] fdans: great email (slack) :) [17:48:47] ottomata: aw thanks! it was sent with a lot of hesitation [17:49:17] alternate option: heres the slack alternative we will use, and we now invest in it [17:49:26] matrix or riot whatever [17:49:41] I'll reply with that actually... :() [17:51:09] there are alternatives, but no willingness to explore them, this is probably what caused friction in the past [17:51:26] one above all, https://mattermost.com/ [17:51:43] (we could self host it in theory( [17:51:55] * elukey brb [17:57:25] ottomata: elukey at this point I'll agree with any non-irc solution, but it seems like a bit of a stretch to make everyone in the org switch to a 4th platform [18:01:47] fdans: if there was top down priority for another option [18:01:55] it would possibly help make everyone happy [18:02:07] it'd be: easier and better than IRC (hopefully), but also floss [18:05:37] ottomata: I can't release (no write to wikimedia repo I think) [18:05:49] hmmmm [18:05:52] lemme see [18:05:54] also ottomata, I create a PR to add scm [18:06:00] I'll update it [18:07:25] ottomata: PR ready, but with those settings I can't prepare (failure to login to push) [18:07:36] ottomata: maybe due to 2FA in github for me? [18:07:41] fdans: fully agree, we should find a good compromise, especially in light of the fact that we already use tools like the gsuite etc.. [18:07:59] not a valid reason to admit any kind of proprietary tool [18:08:11] but we also need to strive for a good compromise [18:08:16] we'll see [18:08:16] joal can you log into archiva? [18:08:17] :) [18:08:28] github? [18:08:35] ottomata: problem is not pushing to archive (I actually uploaded 0.0.2-SNAPSHOT) [18:08:42] ottomata: github, for scm tag [18:09:06] ah hm [18:09:33] hmm it worked for me before.... hmmm [18:09:42] ottomata: without scm settings? [18:09:46] yes [18:09:49] WEIRD! [18:10:21] butoh [18:10:22] oh [18:10:23] mvn deploy [18:10:30] i think i manually tagged maybe [18:10:36] Ah [18:10:49] https://wikitech.wikimedia.org/wiki/Archiva#Deploy_to_Archiva [18:10:50] And manually updated to non-snapshot etc? [18:10:53] yes [18:11:02] Ahhh! Makes sense [18:11:13] ok, will do so [18:11:27] I'll close the PR bout SCM and open a new one with pom update [18:11:30] ottomata: -^ [18:11:30] ok [18:11:31] i mean [18:11:34] we could get mvn release to work [18:11:36] that woudl be cooler [18:11:44] I'd love to :) [18:11:46] but maybe 2fa is a problem? [18:11:57] I'm gonna dig a bit and see ot [18:12:01] k [18:19:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Create and use new schema repositories - https://phabricator.wikimedia.org/T240985 (10Ottomata) Ok, I've deployed usage of the new schema repos totally to eventgate-logging-external, but only to staging for eventgate... [18:22:03] ottomata: I think the current setting for scm should work (with SSH key and push write to wikimedia repo) [18:22:20] ottomata: https://github.com/wikimedia/hdfs-tools/pull/5 [18:22:23] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Erik_Zachte) @elukey thanks for continuing a constructive dialogue. Fixing the cron job, the rsync rights, and monitoring updates is all in the domain of ops, right? Or should I say bash rel... [18:24:24] oh joal you haven't tried it? [18:24:37] or you need it merged to try? [18:24:41] ottomata: I didn't want to polute - I can try though :) [18:24:48] or so I think [18:24:53] trying now [18:25:54] ottomata: I think it needs to be merged first [18:26:30] question for elukey or joal (or anyone else who might know this): is there a way to access files in dumps.wikimedia.org. Need to read & extract the legacy & modern pagecounts for the months they overlap but don't want to scrape index pages and download over HTTP [18:26:40] ok lets merge and try [18:26:57] bearloga: by dumps you mean pageview files? [18:26:59] joal: merged [18:27:05] thanks ottomata - will try [18:27:14] joal: yep [18:28:13] bearloga: hdfs://wmf/data/archive/... [18:28:35] bearloga: I think you're interested in pagecounts-all-sites or pagecounts-raw [18:29:42] joal: perfect! thank you! [18:29:47] np bearloga :) [18:32:23] ottomata: still uploading but I think it worked :) [18:33:23] ottomata: Ah actually no, not sure :( [18:33:24] meh [18:35:08] ottomata: git push rejected :( [18:35:34] ottomata: it says non-fast-forward - I don't get it why :( [18:36:23] hmmm [18:36:35] joal: you are on tip of master? [18:37:05] maybe just to be sure: [18:37:05] git checkout -b m; git branch -D master; git checkout -b master origin/master; git branch -D m; [18:37:14] ottomata: not master - I use mw_master for the mw-repo (origin is my own) [18:37:53] hm [18:37:58] oh [18:37:59] hm [18:38:05] ok joal can I try? [18:38:34] ottomata: weird - tag got created [18:38:39] hmm [18:38:43] ottomata: I'm gonna try again [18:38:46] k [18:38:48] with all clean stuff [18:39:06] joal: maybe that's why its failing? [18:39:09] delete the tag and try again? [18:39:17] indeed ottomata [18:39:35] joal it looked like you pushed mw_master too [18:39:37] to github [18:39:47] hmm [18:39:54] ok - I'm gonna do it differently [18:39:56] k [18:40:06] I'm gonna make origin to the wikimedia repo [18:40:16] (03PS1) 10Milimetric: Fix file mode for hive script [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/563242 [18:40:38] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fix file mode for hive script [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/563242 (owner: 10Milimetric) [18:43:50] joal probably a good idea :) [18:44:15] ottomata: seems working - Thanks a lot [18:44:24] great! [18:46:08] ottomata: Done with success! I did: mvn release:clean release:prepare release:perform - All good :) [18:46:18] artifact is in archiva :) [18:46:21] \o/ [18:50:24] joal: do you have a min? [18:50:31] sure elukey [18:51:00] I am trying to launch the mobile_apps session metrics job in hadoop test [18:51:14] I have created my own /user/elukey/oozie with up to date stuff [18:51:28] and followed the changes that you have in your refinery on an-tool1006 [18:51:33] elukey: the one I use is a modified one [18:51:35] when I launch the coord I get Error: E0803 : E0803: IO error, undefined dataset: webrequest_text [18:51:38] You can use it! [18:51:53] awesoome! [18:52:42] for some reason it was erroring before, now it is not, lovely [18:53:00] hm [18:53:19] probably my bad [18:53:45] joal: did you see https://phabricator.wikimedia.org/T240934#5789947 ? [18:53:59] (03PS1) 10Ottomata: Deploy hdfs-tools 0.0.2 [analytics/hdfs-tools/deploy] - 10https://gerrit.wikimedia.org/r/563248 [18:54:07] ottomata: Ahhh - I was doing it :) [18:54:16] ottomata: you;re too fast :) [18:54:31] I have seen that elukey yes - I need to test! [18:54:35] (03PS2) 10Ottomata: Deploy hdfs-tools 0.0.2 [analytics/hdfs-tools/deploy] - 10https://gerrit.wikimedia.org/r/563248 [18:54:37] hehe [18:55:21] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Deploy hdfs-tools 0.0.2 [analytics/hdfs-tools/deploy] - 10https://gerrit.wikimedia.org/r/563248 (owner: 10Ottomata) [18:55:23] ottomata: shall I merge? [18:55:25] ok [18:55:28] :) [18:55:32] haha just did [18:55:54] ottomata: I kinda get it, you're TOO FAST [18:56:00] :) [18:57:15] deployed joal! [18:57:32] Yay! [18:57:36] we can test :) [18:57:58] elukey: I guess tomorrow being friday we'll wish to wait for Monday? [18:58:06] elukey: about testing hdfs-rsnc [18:58:38] joal: I'd prefer so yes, but we can test some manual runs on the labstore nodes [18:58:44] joal: I think you can test some rsyncs! but maybe we'll wait to merge the big one until monday [18:58:47] the big one as a cron [18:58:54] / systemd timer [18:59:00] yup, works for me :) [18:59:06] luca and I think alike! [18:59:07] tomorrow test [19:00:59] logging off for today, talk with you tomorrow! [19:01:01] o/ [19:01:07] elukey: I'm runnig a tst now [19:01:13] ah! [19:01:16] I can wait then [19:01:24] I am too tired to fight with oozie :) [19:01:25] elukey: SUCESS [19:01:47] joal: it works without spark.authenticate=true etc.. ? [19:02:24] I can't recall the conf - let me check [19:02:47] I think that you added it among the parameters [19:03:01] (it is still listed in your refinery's git diff) [19:03:15] indeed [19:03:19] will remove it [19:03:23] super [19:08:21] elukey: there is an error on every worker when they start, but it's not fatal [19:08:28] And the job succeeds [19:08:57] what error? [19:09:32] elukey: java.io.FileNotFoundException: /var/lib/hadoop/data/e/yarn/local/usercache/analytics/appcache/application_1576771377404_17683/container_e05_1576771377404_17683_01_000002/spark-oozie-job_1576771377404_17683.log (No such file or directory) [19:09:53] whattt [19:10:01] But except from that it all works great - no need for extra param, result written, all good [19:10:28] \o/ [19:10:57] I hope that spark defaults will not interfere with settings in workflows [19:11:21] like max number of workers [19:11:58] but it looks good for the moment, it would be great if we could avoid a oozie roll restart [19:12:09] for sure! [19:12:40] Interesting elukey: sprk-submit with python works, but pyspark2 --master yarn still fails for me [19:12:48] Let's see that tomorrow :) [19:12:59] I did a spark.sql query today and it worked [19:13:01] ufff [19:15:04] ah yes now it doesn't [19:15:42] elukey: WEIRDDDD! [19:15:56] elukey: let's table that to tomorrow? [19:16:29] (03Abandoned) 10Ottomata: [WIP] Eventlogging schema loader quick and dirty refactor on top of andrew's changes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493157 (owner: 10Nuria) [19:16:33] joal: sure, sigh [19:16:59] 10Analytics, 10Analytics-Kanban: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10JAllemandou) Latest development on my end: * Oozie worked with Luca's patch above * spark-submit with python worked as well * pyspark2 still fails More investigations tomorrow [19:17:02] elukey: --^ [19:17:34] thanks :) [19:17:38] milimetric: not sure if you saw but you have a nit comment at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/524575#message-67b2925df37b112a25a5e9e1e53c9cbe8dd0b99d [19:17:42] have a good evening! [19:17:44] o/ [19:17:47] byeee! [19:17:51] bye elukey [19:19:15] oh thx ottomata hadn’t seen [19:20:15] milimetric: perhaps we should rebase https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/554893 on top of your queue change to start the work on that [19:20:16] ? [19:20:23] or do you think your queue change will still have a lot more patches to go? [19:20:32] ok gone as well :) [19:20:49] laters! [19:21:37] ottomata: as far as I know my change is fairly solid. Just needs review [19:21:50] so I say rebase [19:22:07] Thanks again ottomata for the review and merge and all :) [19:23:53] ty too! [19:37:01] 10Analytics, 10Pywikibot, 10Wikimedia-Site-requests, 10User-Urbanecm: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Xqt) BTW I created T242120 for migration support (and did a few) [19:43:37] milimetric: yeehaw also got timo to respond :D [19:48:41] 10Analytics, 10Event-Platform, 10WMF-JobQueue, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: EventBus extension must not send batches that are too large - https://phabricator.wikimedia.org/T232392 (10WDoranWMF) a:03Clarakosi [19:50:22] 10Analytics, 10Event-Platform, 10MediaWiki-Maintenance-scripts, 10WMF-JobQueue, and 2 others: showJobs.php maintenance script useless and misleading in production - https://phabricator.wikimedia.org/T221224 (10WDoranWMF) a:03Clarakosi [19:51:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Create common http subobject for re-use in event schemas - https://phabricator.wikimedia.org/T242363 (10Ottomata) [20:05:09] 10Analytics, 10Pywikibot, 10Wikimedia-Site-requests, 10User-Urbanecm: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Urbanecm) >>! In T242157#5790000, @Lokal_Profil wrote: >>>! In T242157#5785171, @Urbanecm wrote: >> Maybe we can add the d... [20:19:28] wowow, awesome, got work to do :) [20:31:48] 10Analytics, 10GLOW, 10User-Elukey: Access to DataGrip refused - https://phabricator.wikimedia.org/T241170 (10Iflorez) Thank you @elukey I appreciate your troubleshooting and assessment [20:32:10] 10Analytics, 10GLOW, 10User-Elukey: Access to DataGrip refused - https://phabricator.wikimedia.org/T241170 (10Iflorez) 05Open→03Resolved [20:34:26] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.30x-O-Ontbijtkoek): EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196 (10ABorbaWMF) Find in page seems to be working for me on 2.7.50308-beta-2020-01-06 [20:40:15] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.30x-O-Ontbijtkoek): EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196 (10Dbrant) Note: Analytics should see a reduction in these errors after we release our update to... [20:56:43] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Create common http subobject for re-use in event schemas - https://phabricator.wikimedia.org/T242363 (10Ottomata) @Pchelolo and @Milimetric, what do you think about https://gerrit.wikimedia.org/r/563270? @jlinehan a... [20:58:44] 10Analytics, 10Android-app-Bugs, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.30x-O-Ontbijtkoek): EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196 (10Krinkle) I'll note that if this event has indeed been unconditionally rejected since 2015 by E... [20:59:41] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10MMiller_WMF) @elukey -- thanks for working on this. I am still receiving the same error. [21:03:53] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Create common http subobject for re-use in event schemas - https://phabricator.wikimedia.org/T242363 (10Pchelolo) I like the idea of separating partial and full schemas very much. For whatever reason `_fragment` doe... [21:06:32] milimetric: thoughts ^ [21:06:33] ? [21:10:38] ottomata: util? [21:10:43] (I dislike the _ too) [21:10:58] snippet... fragment without the _ [21:11:12] (this bikeshed will self-destruct in 5 minutes) [21:12:06] haha [21:12:53] well, the _ makes it sort nicely [21:12:59] +1 to moving them to a folder though, agree that makes sense to separate fragments [21:13:04] otherwise it is in the middle of the list [21:13:13] right, but hopefully whatever name is obvious enough [21:13:16] might have been nice if we had originally made everything else under /event [21:13:24] ¯\_(ツ)_/¯ [21:13:26] like there's a part of my brain that looks for "util" [21:13:31] don't love util [21:13:35] nor lib [21:13:36] yea bad here [21:13:48] fragement or component is descriptive [21:13:54] fragment a little more specific [21:14:49] __fragment__ [21:14:53] ... hm [21:15:47] BOOOM [21:15:53] oh man, pieces of that shed went EVERYWHERE [21:16:55] haha [21:16:59] OH hey man i'm multitasking! [21:17:02] sheesh [21:17:25] milimetric: maybe you are right, no _ [21:26:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 3 others: Create common http subobject for re-use in event schemas - https://phabricator.wikimedia.org/T242363 (10Ottomata) Changed to just /fragment [21:33:53] has anyone here thought about using their yubikey to authenticate with kerberos? when pressing the button it generates 44 alpha characters (the first 12 are always the same & unique to device, the rest are random), so theoretically those first 12 characters could be a password to make authenticating with kinit as easy as pressing a button [21:39:40] bearloga: no, but we have some other ideas [21:39:56] i think we could use the ssh key that already exists [21:40:01] for your user [21:40:21] and kinit at login transparently [21:40:31] but tbd... [21:43:25] ah