[05:58:55] goood morning! [05:59:02] so list of fixes for today: [05:59:57] 1) https://gerrit.wikimedia.org/r/#/c/437056/ - this one replaces in puppet s/webrequest/webrequest_sampled_128, should be straightforward but I'll wait for a +1. Should resolve the "Couldn't get datasource metadata for datasource webrequest. Aborting." [06:00:31] 2) I've installed python3-dateutil on an1003 to solve "ImportError: No module named 'dateutil' [06:01:07] I also added the pkg in https://gerrit.wikimedia.org/r/#/c/436012/ (still to be merged, basically a refactor of the packages needed in our cluster) [06:05:09] #) last but not the least, we'd need to send an email to analytics@ informing people about the s/-/_/g in the druid's datasource names (got a ping in one task about it) [06:21:20] Hi elukey [06:21:25] Nice list! [06:21:52] elukey: Thanks for all the patches [06:23:01] hello joal! Bon jour :) [06:23:10] Bonjour à toi ! [06:27:15] elukey: I think we'll have to manually restart the sqoop job having failed this weekend [06:27:51] elukey: druid backfill of _ datasources is done, I have deleted the old ones (except mediawiki-history-beta) [06:28:02] nice! [06:28:16] can I start a tmux session and manually run the cron? [06:28:19] elukey: Can you please restart turnilo so that the list gets updated? [06:28:29] elukey: yessir [06:28:48] turnilo restarted [06:29:06] Looks good :) [06:29:10] Thanks elukey [06:33:06] weird, if I try to run the sqoop-mediawiki-private cron I get grep: Invalid back reference [06:34:37] could it be [06:34:38] /srv/deployment/analytics/refinery/python/refinery/util.py:47: 'grep -q "\({0}\).*\(RUNNING\|ACCEPTED\)"'.format(job_name) ? [06:36:26] mmm no [06:37:05] doesn't sound failiar o me elukey :( [06:39:11] so it is running now, the logs show correctly sqooping as far as I can see [06:39:41] elukey: what was the issue? [06:39:48] great elukey :) [06:41:47] well it still emits the grep error [06:41:52] but then it proceeds [06:42:08] and the only grep reference seems to be the one that I added above [06:42:21] (used to check if a yarn app is running) [06:42:41] I have done this [06:42:45] sudo -u hdfs /bin/bash [06:42:47] export PYTHONPATH=${PYTHONPATH}:/srv/deployment/analytics/refinery/python [06:43:00] and then /usr/bin/python3 etc.. [06:46:05] the grep error seems to be related to the actual grep command, which doesn't make sense! [06:48:15] I am trying to reproduce but I can't [06:48:42] same for me [06:48:44] weird [06:48:50] 10Analytics: Error when accessing webrequest on hue.wikimedia.org - https://phabricator.wikimedia.org/T196318#4252377 (10JAllemandou) For Hive to support JSON files with 1 record per line, explicit import of the hcatalog jar in session is needed (see https://github.com/wikimedia/analytics-refinery/blob/master/h... [06:48:51] Could be related to jobname? [06:49:16] in my shell I can see this [06:49:29] (pvt) [06:54:39] elukey: writing email about datasources in druid, will send you for review [06:54:58] <# [06:55:00] <3 [07:05:49] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#4252716 (10Vgutierrez) >>! In T182993#4248709, @Ottomata wrote: > Hm, ya, sounds like a way off before we get that in Debian then, ya? Is that... [07:13:45] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837#4252731 (10MoritzMuehlenhoff) 05Open>03Resolved a:03MoritzMuehlenhoff @Gilles You can now log int... [07:33:51] Wow - Good catch on stat1006 elukey ! [07:34:19] the script has references to Dan's home in it :D [07:34:42] ah wait! [07:34:48] this is why it is failing! [07:45:12] but probably this thing will be superseeded by Dan's new work on geowiki [07:45:17] I'll wait for his input [08:12:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deploy Turnilo (possible pivot replacement) - https://phabricator.wikimedia.org/T194427#4252840 (10elukey) [08:12:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Pageviews-daily broken after move from Pivot to Turnilo - https://phabricator.wikimedia.org/T195819#4252838 (10elukey) 05Open>03Resolved Email to analytics@ sent, closing task! [08:50:53] thought of the morning - should be deprecate the usage of python 2 across worker/stat boxes? [08:52:13] elukey: some of our users will probably be unhappy [08:52:52] elukey: But that would be no problem for me [08:57:17] :) [09:02:26] joal: I added the pkgs requested in https://gerrit.wikimedia.org/r/#/c/436012/ [09:03:36] That's great elukey :) [09:24:12] * elukey realizes that it is the 4th and only 4 days are left for the self+peers reviews [09:40:00] * joal continues to look away from the list of reviews to write [09:46:08] 2018-06-04T09:36:57 INFO Wrote Success file /wmf/data/raw/mediawiki_private/tables/cu_changes/month=2018-05/_SUCCESS [09:46:11] joal: --^ [09:46:20] \o/ ! [09:47:07] elukey: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0055992-180330093100664-oozie-oozi-C/ [09:47:10] :) [09:48:04] nice! [10:06:10] 10Analytics, 10Services (watching): Enable TLS and authorization for cross DC MirrorMaker - https://phabricator.wikimedia.org/T196081#4253084 (10Pchelolo) [10:25:04] 10Analytics, 10Operations, 10hardware-requests: Site: eqiad | hardware request for a dedicated stat analytics host for the Research team - https://phabricator.wikimedia.org/T196080#4253157 (10elukey) 05Open>03declined Sure we can decline and start another one. For the specs we don't have specific require... [10:27:29] 10Analytics, 10Operations, 10hardware-requests: Site: eqiad | hardware request a new stat analytics host - https://phabricator.wikimedia.org/T196345#4253175 (10elukey) [10:32:30] * elukey errand + lunch! [11:52:50] 10Analytics-Tech-community-metrics, 10Developer-Relations: Consider enabling GitHub backend in wikimedia.biterg.io to cover canonical Wikimedia repositories not in Gerrit - https://phabricator.wikimedia.org/T186736#4253401 (10Aklapper) I think I mentioned this somewhere before and than forgot, but something [?... [12:00:07] 10Analytics, 10Analytics-Cluster, 10Services (doing): Move EventStreams to new jumbo cluster. - https://phabricator.wikimedia.org/T185225#4253425 (10Pchelolo) [12:30:26] 10Analytics, 10DC-Ops, 10Operations, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072#4245913 (10faidon) We have a number of spreadsheets tracking inventory, refreshes, CapEx budgets etc. Which one are you referring to specifically (doc & sheet)? [12:47:50] 10Analytics, 10DC-Ops, 10Operations, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072#4253576 (10elukey) 05Open>03Resolved a:03elukey Wrong tab in the spreadsheet! :) [13:10:41] 10Analytics, 10Analytics-Cluster, 10Services (doing): Move EventStreams to new jumbo cluster. - https://phabricator.wikimedia.org/T185225#4253691 (10Pchelolo) I've looked into reusing our existing rate limiter for concurrent connections limiting instead of the request rate limiting, but it requires significa... [13:13:09] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4253706 (10Ottomata) Done, try now! Use your shell username (mmiller) and your wikitech/ldap password. [13:52:12] o/ [13:54:48] o/ [14:40:45] (03PS5) 10Mforns: Fixed accessibility/markup issues of Wikistats 2.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/436002 (https://phabricator.wikimedia.org/T185533) (owner: 10Sahil505) [14:41:25] (03CR) 10Mforns: [V: 032 C: 032] Fixed accessibility/markup issues of Wikistats 2.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/436002 (https://phabricator.wikimedia.org/T185533) (owner: 10Sahil505) [14:51:40] (03CR) 10VolkerE: [C: 031] Fixed accessibility/markup issues of Wikistats 2.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/436002 (https://phabricator.wikimedia.org/T185533) (owner: 10Sahil505) [14:53:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deploy Turnilo (possible pivot replacement) - https://phabricator.wikimedia.org/T194427#4253980 (10Nuria) 05Open>03Resolved [14:56:21] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837#4253993 (10Nuria) Some docs for @Gilles regarding eventlogging access in hive: https://wikitech.wikimed... [15:01:46] ping ottomata [15:06:23] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventBus service can drop a few messages during kafka leadership change - https://phabricator.wikimedia.org/T196077#4254028 (10Ottomata) [15:09:02] (03PS1) 10Mforns: [WIP] Allow partial whitelisting of map fields [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/437269 (https://phabricator.wikimedia.org/T193176) [15:10:15] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4254036 (10Joe) [15:10:19] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#4254037 (10Joe) [15:10:23] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Create an LVS endpoint for jobrunners on videoscalers - https://phabricator.wikimedia.org/T188947#4254035 (10Joe) 05Open>03Resolved [15:11:02] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Create an LVS endpoint for jobrunners on videoscalers - https://phabricator.wikimedia.org/T188947#4024921 (10Joe) The LVS endpoint is now available at `videoscaler.discovery.wmnet` [15:47:39] 10Analytics, 10Analytics-EventLogging: Add A/B testing support in EventLogging - https://phabricator.wikimedia.org/T87459#4254163 (10fdans) 05Open>03Resolved a:03fdans A/B testing support is available in Wikimedia events https://www.mediawiki.org/wiki/Extension:WikimediaEvents [15:52:24] 10Analytics: Error when accessing webrequest on hue.wikimedia.org - https://phabricator.wikimedia.org/T196318#4254193 (10fdans) 05Open>03declined Closing since wmf.webrequest does work in Hue. The raw table is not for user consumption except for very specific use cases that should be discussed with Analytics. [15:53:43] 10Analytics, 10Analytics-Kanban: Review Burrow alarms in order to avoid false positives when restarting it - https://phabricator.wikimedia.org/T196158#4254195 (10fdans) p:05Triage>03Normal [16:17:12] ottomata: fyi https://phabricator.wikimedia.org/T189741 [16:21:31] thanks [17:02:19] joaL: the webrequest dataset still does not have the ip entity right? https://turnilo.wikimedia.org/#webrequest_sampled_128/3/N4IgbglgzgrghgGwgLzgFwgewHYgFwhqZqJQgA0hEAtgKbI634gCiaAxgPQCqAKgMIUQAMwgI0tAE5k8AbVBoAngAcmBDHSGTaw5gH09GppSMAFKVgAmM+SEsxJ6LLgKmAjABEhS1cwTpaKDQhINplfABaNwBfAF04yihlJDQbWMo6OFhtGVBoAFkYcQh8YVJaRIhsAHMENRAACwhUoWoijHxZRuayeMplKuxaSw8aWmwoZzT+weGAZUxJYIImlso66vHLfGwihEom6oakI+XdhARooA [17:02:23] cc elukey [17:11:09] nuria_: yep, IIRC it would have been a high cardinality dimension that we didn't want to manage [17:12:20] ottomata: Hello! The first option you mentioned in https://phabricator.wikimedia.org/T192819#4151345, the schema-less object, is it still not future proof? I want to add a schema-less object in my table cuz otherwise I'll have to list all languages of Wikipedia as keys. But I want to check with you how bad that would be... [17:12:55] * elukey off! [17:36:52] a-team I'm going to overwrite the druid geowiki_archive_country datasource so that it contains ISO country codes instead of names 🌩 [17:58:37] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4254662 (10MMiller_WMF) Thanks @ottomata -- it's working. [17:59:23] hm, chelsyx, not sure what you want to do [17:59:29] but it seems bad to make a dimension into a key [18:07:48] ottomata: so in the next release, users will allowed to turn off certain feed card for certain languages, so we want to log that like: {feature_article:{en:1, es:0, zh:1}} [18:08:36] This means user turn on the the feed from English and Chinese wikipedia, turn off feed from Spanish wikipedia [18:09:29] chelsyx: [18:09:48] feature_article: { enabled: [en, zh], disabled: [es] } [18:09:48] or [18:09:51] even better maybe [18:10:24] feature_article: [en, zh] [18:10:24] prior_state { [18:10:24] feature_article: [es] [18:10:24] } [18:10:25] or whatever [18:10:40] maybe ^? [18:11:13] (brb) [18:17:19] ottomata: Thx! I think `feature_article: { enabled: [en, zh], disabled: [es] }` will work. :) [18:22:08] it seemed struct is easier to query than array to me, but array should work as well :) [18:26:12] its true, but having variable column names never ends well [18:26:18] imagine creating a sql table [18:26:24] every single key you make will be a sql column [18:26:37] chelsyx: ^ [18:30:08] ottomata: yeah [18:58:21] chelsyx: ya, +1 to ottomata 's suggestion [19:07:39] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4254879 (10mpopov) [19:10:59] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventBus service can drop a few messages during kafka leadership change - https://phabricator.wikimedia.org/T196077#4254897 (10Ottomata) Ok! I think this helped. Deploying my fix somehow exposed the underlying probl... [19:13:19] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4254905 (10Ottomata) > ts field for client-side timestamps in case the device goes offline and the event is q... [19:40:54] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4255009 (10mpopov) [19:42:06] qq for JS folks [19:42:11] mayyybe fdans still there? [19:42:20] if you were parsing a date string (maybe just ISO-8601) [19:42:24] (or maybe anything?) [19:42:32] would you use just Date.parse() [19:42:34] or something like moment.js? [19:44:47] ottomata: have you tried just putting the string in the constructor? for a lot of formats it just works! [19:45:12] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4255030 (10mpopov) >>! In T191859#4254905, @Ottomata wrote: >> ts field for client-side timestamps in case th... [19:53:13] yes tried it, but just wondering what is best? [19:53:16] you'd just use built in stuff then ya? [19:53:23] fdans: ^ [20:05:03] sorry ottomata makin dinner! it depends on whatcha want to do with the dates [20:05:50] if it involves a lot of timezone wrangling momentjs is way more convenient [20:06:21] hm [20:06:26] no timezone i think [20:06:31] i can tell people to always use utc...i think [20:06:39] but, i think i want to accept either: millisecond unix ts [20:06:40] or [20:06:42] ISO-8601 [20:06:46] or if also easy, other string date format [20:06:47] s [20:38:21] (03PS2) 10Milimetric: Adjust date formatting in the hover box [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/434557 (https://phabricator.wikimedia.org/T194430) [20:39:30] (03CR) 10Milimetric: "This is rebased and ready to review." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/434557 (https://phabricator.wikimedia.org/T194430) (owner: 10Milimetric) [21:05:49] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Stream: EventBus should produce messages to Kafka with event time set to meta.dt - https://phabricator.wikimedia.org/T196407#4255251 (10Ottomata) p:05Triage>03Normal [21:06:44] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Stream, 10Services (watching): EventBus should produce messages to Kafka with event time set to meta.dt - https://phabricator.wikimedia.org/T196407#4255265 (10Pchelolo) [21:10:39] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4255283 (10Ottomata) `client_dt`? We are trying to use the convention that fields named after `dt` are ISO-8... [21:11:10] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Stream, 10Services (watching): Consider increasing retention for mediawiki event topics - https://phabricator.wikimedia.org/T196409#4255284 (10Pchelolo) [21:12:55] 10Analytics, 10Product-Analytics, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, and 3 others: [EPIC] Reading List Sync service analytics - https://phabricator.wikimedia.org/T191859#4255297 (10mpopov) Sounds good, @Ottomata! Updated and I'll keep this in mind going forward. [21:20:25] 10Analytics, 10EventBus, 10ORES, 10Patch-For-Review, and 3 others: Numeric keys in ORES models causing downstream Hive ingestion to fail - https://phabricator.wikimedia.org/T195979#4255301 (10Ladsgroup) https://github.com/wiki-ai/articlequality/pull/68 Also announced it in wikisource-l, ai-l and Village p... [21:29:29] (03PS1) 10Milimetric: Use unrestricted characters [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/437363 [21:29:45] (03CR) 10Milimetric: [V: 032 C: 032] Use unrestricted characters [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/437363 (owner: 10Milimetric) [21:33:01] (03CR) 10Milimetric: "Replying to everything:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/434500 (https://phabricator.wikimedia.org/T179444) (owner: 10Milimetric)