[00:05:53] Im [00:06:03] having issues with hue again :( [00:06:23] Error while compiling statement: FAILED: SemanticException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient [01:26:02] 10Analytics, 10Security-Team: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10Reedy) [01:28:33] 10Analytics, 10Security-Team: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10Reedy) Do we have any "approved" VPN apps/protocols for production? What auth is the VPN going to use? Is authing via VPN going to be the only auth? Then there will be no auth to use t... [06:42:58] Seddon: o/ [06:44:15] Seddon: there is a workaround in https://phabricator.wikimedia.org/T242306 [06:44:23] other people are experiencing the same issue :( [06:44:32] can you try it and see if it works? [06:45:06] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) Adding @Seddon to this task since he reported the issue in #wikimedia-analytics :) [06:45:21] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) @MMiller_WMF any re-occurrence of the issue? [06:58:24] (afk for a bit more) [07:38:45] 10Analytics, 10Security-Team: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10elukey) >>! In T242998#5811743, @Reedy wrote: > Do we have any "approved" VPN apps/protocols for production? > > What auth is the VPN going to use? > > Is authing via VPN going to be t... [07:38:58] 10Analytics, 10Security-Team, 10User-Elukey: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10elukey) [07:42:59] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10elukey) [08:07:58] good morning team :) [08:08:01] bonjour! [08:13:51] elukey: mediacounts have yesterday's file - Looks like the stuff works :) [08:13:57] \o/ [08:14:01] joal: \o/ [08:14:08] we can convert the others if you want [08:14:16] and clean up the puppet stuff on monday [08:14:18] elukey: let's do that on monday :) [08:14:37] ok as you prefer :) [08:14:48] elukey: there is some work to clean and all, so let's make in not before weekend :) [08:14:49] I am off from Wed onward, this is why I was saying that [08:15:01] Ah - Ok, I can do it now then ) [08:15:20] I meant only adding hdfs rsync to all the timers, not the clean up:) [08:15:33] ah also, unrelated - https://gerrit.wikimedia.org/r/#/c/565509/ [08:15:38] ok for you? [08:16:16] wow [08:16:22] yes, ok for me :) [08:16:34] so the FsImages are around ~100G now :D [08:16:49] so far we have never used FsImage (no hdfs issue) [08:17:01] don't jinx it joal :D [08:17:17] that is the nuclear option [08:17:40] :) [08:18:01] also elukey, I have found an interesting problem [08:18:11] Namely bz2 decompress issue in java [08:18:33] in what context? [08:18:57] parsing dumps xml dumps :) [08:19:25] the issue we experienced 2 month back is back, with the same exact problem - Exception at decompress [08:20:24] Only a single file is causing trouble, and interestingly linux-bz2 command works for it, but java uncompress using apache commons-compress package fails, meaning its not hadoop/spark related [08:22:07] ah snap [08:22:14] indeed :S [08:57:04] I am trying to re-run the refine failed mediawiki_job occurrence, the emails are mine [08:57:25] k [09:00:08] 10Analytics: Requesting kerberos access for jwang - https://phabricator.wikimedia.org/T242813 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff I created your Kerberos account. You should have a mail for your Kerberos account (required to access Hadoop) with further instructions. [09:05:50] random thought - I just noticed that we use "--deploy-mode cluster" for spark refine, and now I kinda understand why we needed to add the --keytab --principal options [09:06:23] with the driver in Yarn it might make sense that we need to pass the keytab to the Yarn's cache [09:06:26] elukey: RIGHT!!!!! [09:06:56] elukey: but I thought spark was still launching a local something (not driver, but connector) that was providing tickets [09:07:01] from cache [09:07:20] I think it is the driver that takes care of it.. [09:07:40] I don't think so, it's an interesting try :) [09:08:50] what runs on the host when we launch a spark job in cluster mode? Basically only a yarn app checker? [09:09:07] elukey: shall I go for a all-in-one CR for hdfs-rsync-dumps (meaning cleaning at the sxame time), or just change others so that they work the same way the first ones do? [09:09:37] nono let's just move all the timers to hdfs-rsync, clean up as separate step [09:09:45] since we'll need to remove the /mnt/hdfs stuff as well [09:09:50] elukey: when launching a spark-job in cluster-mode, there is some local connector to the driver - The one that shows you that stuff is running [09:10:03] ack elukey - CR coming :) [09:10:39] makes sense, but I am not sure if the connector has anything to say in kerberos credentials [09:10:58] it seems more a lightweight reference to the driver that runs in yarn [09:11:02] without much logic [09:11:06] elukey: I think it has, I have launched jobs in cluster mode from my user not having to pass keytabs [09:11:08] anyhow, just a though :) [09:11:43] ok if you tested it then it might be something else [09:12:01] I am trying to understand why that refine job keeps failing for kerb auth, strange [09:12:47] :( [09:16:07] joal: diff in https://puppet-compiler.wmflabs.org/compiler1003/20408/labstore1006.wikimedia.org/ [09:16:52] elukey: there is a new resource (geoeditors was not yet converted, can't check that specific one [09:17:35] joal: it is in the change catalog, but it seems very easy so I'd just go for it [09:19:06] elukey: all changes in the file look good :) [09:22:11] ok done! [09:22:26] no more pulls from stat1007 [09:35:19] elukey: aren't there the datasets ones? [09:35:58] ah - they are actual files, not from hdfs :) [09:39:23] yes sorry [09:44:37] Gone for errand - back before lunch [09:50:30] ok something is definitely weird [09:50:45] I don't see the --principal --keytab options for refine jobs [09:50:50] and profile::analytics::refinery::job::refine::use_kerberos is not in puppet [09:52:16] whattt [10:00:13] this is super crazy [10:00:30] I think I have never enabled it [10:01:25] Luca, why on earth then it has been working so far? [10:01:35] there is a plausible explanation [10:02:16] when we enable kerberos, we just execute the kerberos-run-command, that populates the credentials cache where spark picks credentials [10:02:40] since on the coordinator other jobs do it periodically, the refine spark jobs were picking up creds just fine [10:02:59] now there is a mediawiki_job that wants to alter a table in hive [10:03:06] or just create one I think [10:03:18] (since it seems the first set of events of its kind) [10:03:47] so it fails, since we established that this use case neeed the keytab on the yarn's cache [10:04:02] and we currently don't pass it [10:04:14] what a pebcak [10:04:18] Bad Luca [10:06:21] ah and in test it was working since kerberos is always enabled [10:17:44] all fixed [10:49:18] 10Analytics, 10Research, 10Security-Team: Release data from a public health related research conducted by WMF and formal collaborators - https://phabricator.wikimedia.org/T242844 (10Miriam) >>! In T235309#5805098, @Nuria wrote: > We will met with @JFishback_WMF and let you know of next steps thanks @Nuria! [11:19:52] great analysis elukey! [11:20:43] sigh that was a bit miss in the upgrade [11:21:17] elukey: there were so few that I can still say it was one the best complicated upgrade I've ever done :) [11:23:15] :) [11:23:43] going afk for lunch + gym, ttl! [11:23:49] later! [12:20:40] * joal has warm thoughts for halAFK and isaacj https://twitter.com/CaliaDomenico/status/1218068815336103937 :) [12:36:40] (03PS1) 10Joal: Enforce distinct revision in xml-dumps converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565554 [12:47:10] elukey: Thanks for that work around! Can confirm this worked for myself [12:47:33] hi Seddon - Glad it worked :) [12:51:08] 10Analytics, 10Analytics-Kanban: Fix sqoop after changes - https://phabricator.wikimedia.org/T242015 (10JAllemandou) p:05Triage→03High [12:51:47] Hue is my friend :) Never let it disappear. For now I am lost without it [12:52:49] Seddon: we have heard that feedback from others than you as well - If we drop hue, we won't let you down (sorry for the pun, couldn't resist) - We'll be after soemthing that provide the same value in term of facilitating querying [12:55:04] 10Analytics, 10Analytics-Kanban: Update wikitext-processing on hadoop various aspects - https://phabricator.wikimedia.org/T238858 (10JAllemandou) [12:55:27] (03PS1) 10Joal: Update wikitext oozie job adding current [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565558 (https://phabricator.wikimedia.org/T238858) [12:55:50] (03PS2) 10Joal: Enforce distinct revision in xml-dumps converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565554 (https://phabricator.wikimedia.org/T238858) [12:56:34] joal: I can confirm unfortunately that that's pretty on point. On the bright side, it's supposed to warm up (and we'll get a foot of snow). Totally unrelated but I am looking forward to all-hands... [12:57:25] isaacj: so am I (looking forward to all-hands) - It'll be so great to meet y'all again :) [13:19:25] (03PS3) 10Joal: Enforce distinct revision in xml-dumps converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565554 (https://phabricator.wikimedia.org/T238858) [14:15:26] joal: feeling bold today, just merged the clean up [14:15:32] no op on both labstores, good job :) [14:15:34] WAAAAAAW :) [14:15:43] Thanks mate :) [14:16:04] elukey: I guess you can move that task of yours about labstore to done :D [14:16:08] joal: we can move the task to done now :) [14:16:11] ahhahaha [14:16:16] huhuhu [14:16:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts - https://phabricator.wikimedia.org/T234229 (10elukey) [14:17:13] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts - https://phabricator.wikimedia.org/T234229 (10elukey) [14:17:18] done! [14:17:25] \o/ [14:42:43] o/ joal [14:42:49] hi halfak :) [14:42:53] Here's one of my photos from last week: v [14:42:55] https://imgur.com/a/dsuTlM9 [14:43:12] Was out riding my bike for a couple of hours. [14:43:24] Maaaaaan! [14:43:28] ~ -14C [14:43:51] You have ice-teeth at the chin! [14:43:59] Mwahahaha! [14:44:21] I guess you need to do something like biking when getting out with that temperature! [14:44:38] Honestly, I was quite comfortable! Believe it or not, it's all about dressing for the weather. [14:44:57] Oh yeah. Doesn't hurt to be constantly generating a ton of body heat either :) [14:45:08] :) [14:45:09] halfak: what about the face though? isn't it like getting little knives under your skin when the wind blows? [14:45:49] anyway, great resistance to the weather :D [14:45:54] Na. You get used to the cold on your bare skin. Though I find that there is a limit to that at about -20C, I need to have my skin mostly covered. [14:46:03] ahahahah [14:46:07] -20C [14:46:14] "reasonable" :D [14:46:24] Maintaining a beard or wearing an artificial beard is important at this temp. ;) [14:46:45] Artificial beard: https://en.wikipedia.org/wiki/Balaclava_(clothing) [14:47:15] Right - I was trying not to imaginate you putting artificial hair on your face [14:50:34] :DE [14:55:50] this room is so offtopic today, love it :D [14:56:04] joal: not sure if it is my impression or not, but after moving the hive delegation tokens to the db it seems that we are not getting those random failures for kerberos from oozie [14:56:28] elukey: I don't recall having seen the error either :) [14:56:33] Definitely Friday(TM) [14:56:52] elukey: It's weird a bit, elukey, since those should only have been related to restarts, but ehh [14:57:38] joal: no idea what happened in there [14:58:47] Need to go for kids - See you at standup team [14:59:12] halfak: See you at all-hands! And enjoy your ice-teeth :) [14:59:35] \o/ I'm very exited to see y'all there ^_^ [14:59:45] Hopefully I won't bring any frigid weather with me [15:01:00] Hi a-team, please can you tell me which mysql server should I use to access the enwiki database? I used to use analytics-store, but that is not working anymore [15:01:59] dsaez: there's a script on all the boxes that finds the connection (they're sharded now over multiple hosts) [15:02:04] dsaez: you do... one sec [15:02:20] ooh, ok, great... thx [15:02:26] dsaez: analytics-mysql enwiki [15:02:31] yep :) [15:02:49] excellent! thanks...you are my favorite team :D [15:03:23] and if you need the connection info itself, you can look at the code that the script runs here: https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/util.py#L238 [15:03:36] cool! [15:03:39] I bet dsaez says that to all the teams... [15:04:42] haha..no, no, just to you :D ... I also just complain about HD space to elukey :D ... but love is the predominant feeling [15:04:53] :) [15:15:46] (03PS14) 10Fdans: Add vue-i18n integration, English strings [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) [15:17:37] milimetric: helloooo so I'm happy with how the code looks right now if you want to take a final look [15:17:50] i applied the last changes you suggested [15:18:46] k, lookin [15:34:01] fdans: it renders 2019-12-01 pageview data as "Nov 2019", that should be December but was this on purpose somehow? [15:34:09] (when comparing old to new site) [15:34:53] the widgets are fine, those all show it as "D" for december [15:35:01] just the detail page [15:35:36] milimetric: can you pass a snapshot? might be utc shenanigans from datefns [15:36:13] oh fdans I see what's going on, it's that bug where the right side of the time interval decrements by one month [15:36:15] it's back [15:36:41] that might've been why we had that other buggy code that you removed, remember? [15:37:00] see http://localhost:5000/dist-dev/#/all-projects/content/pages-to-date/normal|line|2-year|page_type~non-content|monthly [15:37:04] and toggle splits [15:50:28] joal: when we added the TLS column to webrequest, did you run an alter table manually? [15:50:55] (I am getting failures in test related to that, trying to bring it up to speed with the last code) [15:55:27] (03CR) 10Milimetric: [C: 04-1] "-1 mainly for the re-appearance of the weird bug that decrements the end of the time range when toggling filters on the detail page. Besi" (037 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [16:06:35] going away for ~30 mins, be back in a bit! [16:23:26] fdans: will be here for our 1 on 1 [16:25:02] nuria: cool!! [16:37:22] milimetric: "this doesn't seem to be filled in, but I can't figure out why..." => can't reproduce, the status overlay looks fine to me? [16:38:10] it says The metric is not ... [16:38:17] instead of The "Edits" metric is not ... [16:39:03] fdans: look at the editors metric for all wikis [16:39:24] and marvel at how anal I am that I found that :) [16:41:25] Hey folks. Where could I find stats on browser usage for contributors or readers? [16:59:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10cchen) @Milimetric Thanks for working on this! i noticed the updates in hive table,... [17:03:38] halfak: I don't think we have stats for browser of contributors [17:03:58] halfak: for readers: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os [17:05:33] halfak: i mean it's all together, but if you want you can query webrequest for urls that look like editing activity and categorize by parsed user agent properties [17:06:11] Thanks! This works for us :) [17:21:08] 10Analytics, 10Research, 10Privacy, 10Security: Release data from a public health related research conducted by WMF and formal collaborators - https://phabricator.wikimedia.org/T242844 (10chasemp) My understanding is a #privacy_engineering was already engaged on this in {T235309} and #security-team has no... [17:22:53] 10Analytics, 10Research, 10Privacy, 10Security: Release data from a public health related research conducted by WMF and formal collaborators - https://phabricator.wikimedia.org/T242844 (10chasemp) p:05Triage→03Normal [17:26:13] 10Analytics: Spike. Try to ML models distributted in jupyter notebooks with dask - https://phabricator.wikimedia.org/T243089 (10Nuria) [17:30:53] elukey: indeed, I updated the table manually :) [17:31:07] elukey: the webrequest ones [17:54:34] Gone for tonight team - see you on monday [18:12:41] joal: yt? [18:12:48] or dsaez [18:15:21] * elukey off! [18:15:22] o/ [18:17:19] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10MarcoAurelio) Hi @elukey, and thanks for the ping. Our SUL watcher bots do certainly use this channel to report account creations to `#cvn-unif... [18:23:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Nuria) on edits_hourly in turnilo I see: text_bytes_diff -> as a metric redirect_c... [18:24:34] 10Analytics, 10Analytics-Kanban, 10Product-Analytics (Kanban): Add new dimensions to virtual_pageview_hourly - https://phabricator.wikimedia.org/T243090 (10Nuria) [18:26:27] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Nuria) As far i can see month, day_of_week were not added but those can be easily o... [18:31:14] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10cchen) Yes, i am seeing those fields now in Turnilo, but not in Superset. [18:56:02] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10MMiller_WMF) @elukey -- no re-occurrence. I am still doing fine in Hue. Thanks! [18:56:03] folks, heads up that I announced the office hours next week on LinkedIn now: https://www.linkedin.com/feed/update/urn:li:activity:6624010848925036544/ [18:56:24] feel free to like, share, ignore, ... [19:20:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Nuria) cchen superset needs to have the "columns" on data source updated. see: http... [19:32:08] 10Analytics, 10Analytics-Kanban, 10Product-Analytics (Kanban): Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10Nuria) [19:39:31] nuria: here for a minute [19:39:43] joal: nvm let's talk next week [19:39:50] joal: not urgent [19:40:10] nuria: ok :) I can spend a minute explaining the bot date trick if you want :) [19:40:22] joal: ok, let's batcave [19:40:33] joining [23:07:37] 10Analytics, 10Analytics-SWAP, 10GLOW: Viewing Santali and Javanese characters on SWAP via Chrome only displays Tofu signs - https://phabricator.wikimedia.org/T242490 (10Iflorez) Hi @Aklapper, on my screen, I see tofu characters for the characters on the Javanese script page, essentially just boxes. {F315157...