[00:17:21] 10Analytics, 10Operations, 10Research, 10Patch-For-Review, 10User-Banyek: Import recommendations into production database - https://phabricator.wikimedia.org/T208622 (10Nuria) Both 2 and 3 above can start being worked as soon as @bmansurov has any bandwidth. [00:22:23] (03CR) 10Nuria: [V: 032 C: 032] Clickstream failing due to not being able to access hive tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/478952 (https://phabricator.wikimedia.org/T211717) (owner: 10Joal) [00:23:56] 10Analytics, 10Operations, 10Research, 10Patch-For-Review, 10User-Banyek: Import recommendations into production database - https://phabricator.wikimedia.org/T208622 (10leila) [00:24:22] 10Analytics-Kanban, 10Patch-For-Review: Clickstream job failing due to change of types of namespace column - https://phabricator.wikimedia.org/T211717 (10Nuria) a:05Nuria>03JAllemandou [00:24:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Failure while refining webrequest upload 2018-12-01-14. Upgrade alarms - https://phabricator.wikimedia.org/T211000 (10Nuria) [00:28:28] (03CR) 10Nuria: "Confirming that the problem I saw is no longer there, I leave up to milimetric to CR changes." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T205665) (owner: 10Fdans) [00:29:46] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) a:05fdans>03elukey [00:30:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Wikistats2 UX bug: table option should not be available in table graph selected - https://phabricator.wikimedia.org/T210424 (10Nuria) [00:31:28] 10Analytics, 10Analytics-Kanban: Link to User Contribution page in wikistats UI rather than user page - https://phabricator.wikimedia.org/T210422 (10Nuria) @fdans let's do this as part of our next wikistats deploy that will include families [00:45:23] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) We can port change to 0.27 https://github.com/apache/incubator-superset/pull/5931 if we need to, the bigger question of us moving to 3.6 on superset remains cause we will need to do that sooner o... [07:39:17] (03PS1) 10Elukey: Update README.md with some guidelines [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479164 [07:40:40] (03CR) 10Elukey: [V: 032 C: 032] Update README.md with some guidelines [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479164 (owner: 10Elukey) [08:27:47] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) The materialized view generation completed. The total size of the materialized views are ~150 G all tog... [09:12:43] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10MoritzMuehlenhoff) Sure, this sounds like a sensible candidate for next Q's initial evaluation/migration of buster (we'll investigate/fix up the base layer and also migrate a few systems) Looking at http... [09:28:10] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10elukey) The first breaking change that I can see (use of f-strings) happened in commit https://github.com/apache/incubator-superset/commit/cc3a625a4bb6b0e581b30f3112315ff5a8ab6807 that should be in the up... [10:02:40] (03PS8) 10Fdans: Adds logic and configuration for project families [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T205665) [10:09:48] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10MoritzMuehlenhoff) >>! In T211605#4816351, @elukey wrote: > We'll need to migrate to Buster anyway though to get Python 3.6, my only doubt though is if Superset will require say 3.7/3.8 in the future. At... [10:11:38] elukey@builder:~/incubator-superset$ cat pypi_push.sh [10:11:38] # first bump up package.json manually, commit and tag [10:11:38] rm superset/assets/dist/* [10:11:38] cd superset/assets/ [10:11:39] npm run build [10:11:43] /o\ [11:38:25] * elukey lunch! [11:41:10] 10Analytics-Tech-community-metrics, 10Developer-Advocacy (Oct-Dec 2018), 10Documentation: Update documentation on [[mw:Community_metrics]] to reflect new UI on wikimedia.biterg.io - https://phabricator.wikimedia.org/T211759 (10Aklapper) [13:10:12] 10Analytics, 10Operations, 10Security-Team, 10WMF-Legal, 10Software-Licensing: Can exfat be used in WMF production? - https://phabricator.wikimedia.org/T210667 (10JBennett) >>! In T210667#4812704, @Legoktm wrote: >>>! In T210667#4795435, @JBennett wrote: >> Thanks everyone of for their thoughtful conside... [13:39:16] 10Analytics, 10Analytics-Kanban: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10MoritzMuehlenhoff) >>! In T211605#4816351, @elukey wrote: > The first breaking change that I can see (use of f-strings) happened in commit https://github.com/apache/incubator-superset/commit/cc3a625a4bb6b... [14:05:23] 10Analytics, 10Analytics-Kanban: Presto on Cloud Platform Design Document - https://phabricator.wikimedia.org/T208614 (10JAllemandou) Asking for comments @Nuria , please :) [14:13:51] 10Analytics, 10Operations, 10Research, 10Patch-For-Review, 10User-Banyek: Import recommendations into production database - https://phabricator.wikimedia.org/T208622 (10bmansurov) @Nuria, what you mention makes sense. I created this task in order to get the current recommendations into MySQL. I think we... [14:38:24] (03CR) 10Milimetric: [C: 032] Adds logic and configuration for project families [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464583 (https://phabricator.wikimedia.org/T205665) (owner: 10Fdans) [14:38:59] ottomata: o/ [14:39:13] morninggg [14:39:45] o/ [14:39:45] morning [14:40:16] today I tried to mess a bit with Superset to build a custom wheel without the code that requires python 3.6, and I didn't manage to make it happen (but I think I am close). On analytics-tools1003 though I just noticed that we do [14:40:21] /srv/deployment/analytics/superset/venv/bin/python3 /srv/deployment/analytics/superse [14:40:24] t/venv/bin/gunicorn --config /etc/superset/gunicorn_config.py superset:app [14:40:33] so we don't use, afaics, the python installed [14:40:50] I am wondering if we could use 3.6 with these settings [14:43:52] we don't use the python installed? [14:43:54] oh... [14:44:00] hm [14:44:13] yeah but what is that python3... does it actually install a binary there? maybe it does... [14:44:24] good point [14:44:26] it links to that python because it will have access to the venv [14:44:55] /srv/deployment/analytics/superset/venv/bin/python3: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=dbfc2e1a3c58b6d241b3f9af7b2fb3a24b81b90e, stripped [14:44:56] it is a binary... [14:44:58] yeah [14:45:04] elukey: worth a try! [14:45:28] ottomata: whenever you have time I'd need your brain for a python-build question, not urgent :) [14:45:37] I am really ignorant in the subject [14:45:39] elukey: now is good [14:45:48] bc for a second? [14:46:47] k [14:46:57] let's make it an early ops sync? :) [14:48:35] (03PS1) 10Fdans: Release 2.5.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479216 [14:51:33] (03CR) 10Fdans: [V: 032 C: 032] Release 2.5.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479216 (owner: 10Fdans) [15:12:18] ottomata: do you think this would work to build a prototype of daniel's dependency management system? https://github.com/rayokota/kafka-graphs [15:12:38] and would something like that be worthwhile [15:12:57] (03PS1) 10Fdans: Add version number to footer [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479223 [15:13:35] milimetric mforns I added this lil thing, what yall think? [15:21:50] hey a-team, i'll miss standup today (and maybe on wednesdays in general now?) for another better use of data meeting [15:24:02] hey team :] [15:24:59] milimetric: interesting! [15:25:13] its hard for me to say really; i don't know a lot about graph processing... [15:25:47] well, it's not going to be as fast as a good graph db, but should be useful as a very quick way to get started, because it's just a jar and we can write it right now [15:26:03] milimetric: you can do that with flink to :p [15:26:04] its not just a jar [15:26:06] kafka streams is a cluster [15:26:15] wekk [15:26:15] well [15:26:24] you'd deploy it as a cluster [15:26:31] it'd just be your app [15:26:38] but you'd still deploy it as a cluster [15:26:44] in k8s or something [15:27:24] this thin looks like it is a kafka streams app, right? so you launch the kakfa-graphs-rest-app [15:27:58] also have another kafka streams or ksql running elsewhere that is runnign the Ktables [15:28:00] (03CR) 10Milimetric: Add version number to footer (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479223 (owner: 10Fdans) [15:28:10] then you can POST graph operations to the kafka-graphs-rest-app? [15:29:18] ottomata: as far as I understand, no, you can just run a java app on stat1007 in the background. Based on what they say, as long as you use these jars, that's all you need. [15:29:38] so you'd need the Kafka Streams jars and the Kafka Graphs jar [15:29:43] ottomata: $build_venv/bin/pip wheel -w $wheels_dir/ superset-0.28.1-py3-none-any.whl [15:29:46] \o/ [15:29:47] it works! [15:29:49] otherwise they're lying :) [15:33:01] milimetric: right [15:33:04] you can run it locally [15:33:06] but [15:33:11] you can also run flink locally! :p [15:33:23] all you need is the flink jar [15:33:36] milimetric: but you do need to create the ktables [15:33:39] so you need a running kafka cluster [15:33:53] don't remember exactly how to create ktables...i've done it with KSQL before [15:36:32] I thought the jars did that for you in the background, but maybe not [15:36:56] flink has Gelly, right? [15:37:24] I think we should do a day hacking where we try both of these and figure out which one we use to play with, try out new ideas [15:38:26] it'd be great to have a prototyping method that we can use to understand the scale of a new data flow [15:39:23] so like, if your prototype in Kafka Streams can handle, say 30 events / second, then you should do X in production, but if it can handle 3000 events / second, you should do Y [15:41:58] milimetric: aye, that'd be fun, we'd prob want to collab with marko and petr and/or daniel [15:42:09] i don't know much about graph processing or even what they really are trying to do [15:42:12] i understand the problem [15:42:19] but not really how they want to solve it :) [15:45:17] I get daniel's proposal, it's fairly straightforward, I think it'd be fun to hack it up in something simple and get a better feel for how it scales [15:49:49] woudl def be fun [15:55:51] (03PS3) 10Milimetric: Update hive and oozie for labs/prod sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476855 (https://phabricator.wikimedia.org/T210542) (owner: 10Joal) [16:41:14] (03PS5) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) [16:42:39] (03CR) 10jerkins-bot: [V: 04-1] Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) (owner: 10Mforns) [16:45:59] milimetric: Heya - I'm interested to try to understand what you are trying to achive with the kafka-graph lib :) [16:46:16] joal: we can catch up before standup if you want [16:46:24] let's do that :) [16:46:28] OMW ! [16:49:07] joal: looked at doc and i think is almost there, i just think we need to change afew concepts like "WMF data" by "wikimedia projects analytics data" [16:49:20] joal: and similar. Will work on it a bit. [17:01:01] ping ottomata elukey [17:01:48] nuria: missing standup today, better use of data meeting conflict [17:02:02] ping elukey [17:02:03] status: hive is running in cloud-analytics, had to fight with mariadb puppet stuff [17:02:10] looked into monitoring [17:02:14] its gonna be nasty, not sure what to do [17:02:18] but for now am just working on presto [17:02:57] (03PS1) 10Elukey: [WIP] Release 0.28.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/479249 [17:10:11] !log restarted turnilo to clear deleted test datasource [17:10:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:28:42] (03PS1) 10Fdans: Don't add links if all-projects is selected [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479251 [17:30:51] ottomata: When ou have a working prototype of prest, let me know :) I can help testing (maybe ;) [17:38:53] k! [17:39:05] elukey: another downside of cloud vps: no PCC testing on nodes :( [17:39:37] ooooofff [17:39:43] bye elukey [17:39:59] joal: ? [17:40:15] Ah - This was not "off" with a long o :) [17:40:18] :D [17:40:20] haahahahah [17:40:23] yesyes sorry! [17:40:44] no problemo at all :) I'll be carefull as well next time ;) [17:41:29] I'm a bit sad that Dan didn't get my data-spanish joke in the chan :) [17:44:37] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create .deb package for Presto - https://phabricator.wikimedia.org/T203115 (10Ottomata) https://apt.wikimedia.org/wikimedia/pool/main/p/presto/ [17:53:27] this time off for real! :) [17:53:28] o/ [17:53:36] bye then :) [18:20:14] (03CR) 10Nuria: Don't add links if all-projects is selected (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479251 (owner: 10Fdans) [18:36:08] 10Analytics, 10EventBus: Call to a member function format() on a non-object (boolean) - https://phabricator.wikimedia.org/T211805 (10zeljkofilipin) p:05Triage>03Unbreak! [18:37:41] 10Analytics, 10Analytics-Kanban: Presto on Cloud Platform Design Document - https://phabricator.wikimedia.org/T208614 (10Nuria) Added comments to: https://docs.google.com/document/d/1jYu7j3wev935McqlWDHClqLQpjYfaauGtuO0RaPuF5c/edit#heading=h.okd4m7zhyc5z [18:43:21] 10Analytics, 10EventBus: Call to a member function format() on a non-object (boolean) - https://phabricator.wikimedia.org/T211805 (10zeljkofilipin) Looks like the only thing that changed in EventBus from wmf.6 to wmf.8 is this, and it does not look related. 😐 https://www.mediawiki.org/wiki/MediaWiki_1.33/wmf.... [18:50:05] (03CR) 10Nuria: [C: 031] "I imagine we have tested this last patch that includes the flatening" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) (owner: 10Mforns) [18:50:46] (03PS4) 10Milimetric: Update hive and oozie for labs/prod sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476855 (https://phabricator.wikimedia.org/T210542) (owner: 10Joal) [18:50:58] 10Analytics, 10EventBus: Call to a member function format() on a non-object (boolean) - https://phabricator.wikimedia.org/T211805 (10Pchelolo) I think this is the reason and in particular @mobrovac last comment that for some reason went unnoticed. https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventB... [18:57:05] (03PS5) 10Milimetric: Update hive and oozie for labs/prod sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476855 (https://phabricator.wikimedia.org/T210542) (owner: 10Joal) [19:06:29] (03PS6) 10Milimetric: Update hive and oozie for labs/prod sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476855 (https://phabricator.wikimedia.org/T210542) (owner: 10Joal) [19:15:27] joal: https://hue.wikimedia.org/oozie/list_oozie_workflow/0041340-181112144035577-oozie-oozi-W/?coordinator_job_id=0041339-181112144035577-oozie-oozi-C [19:15:35] I always get unreasonably nervous about running oozie jobs [19:15:40] like I'm auditioning to be Pope or something [19:18:31] milimetric: If the Pope had anything to do with oozie I'm sure we'd have to write conf in latin [19:18:59] * milimetric would have a hard time picking between latin and XML [19:19:11] XMlatin [19:23:32] milimetric: * i think* (cc joal) that we should rerun scoop to make sure things are actually working and that issues you run into where indeed a fluke [19:24:04] A smaller sqoop is enough for that --^ On ly a few projects [19:24:25] nuria / joal: that's the weird thing, I ran the same exact sqoop on fewer wikis and it was completely fine [19:24:41] so I don't think we changed anything that would affect a full sqoop in some weird different way [19:25:07] I'm more inclined to think the sudden bastion hiccup is to blame, along with me potentially spamming Ctrl+C into the darkness that is the internet [19:27:19] milimetric: mmmm... [19:27:30] milimetric: did we run it just for enwiki? [19:27:39] milimetric: no smaller wikis but rather 1 larger one? [19:28:10] sure, maybe one large wiki and a couple of small ones, to simulate most characteristics of a full run [19:28:18] milimetric: i think i am too old , this change for python strings seems total bs: https://github.com/apache/incubator-superset/commit/cc3a625a4bb6b0e581b30f3112315ff5a8ab6807 [19:28:26] ok, I'll do that once we confirm that everything else is working (if it's not, we may have to run another full sqoop anyway) [19:28:31] milimetric (side note, nothing to do with anything) [19:28:47] milimetric: k, that sounds good. going to look at dashiki now [19:30:15] nuria: I have no idea about the bug they reference, but f strings look really cool! [19:30:26] milimetric: see IAM TOO OLD [19:30:30] milimetric: i was like weren [19:30:42] strings were ok before [19:30:57] :) sure, they're ok, but now they're AMAZING :) [19:31:38] I feel you, it's like, people would rather build something new than improve something old. Like, if there's really a problem with .format, they should FIX THAT!!! Not like require everyone to change how they do strings... [19:32:18] milimetric: I'm they wanted python to feel more scala-ish :-P [19:32:23] +sure [19:32:43] hahaha, ok, now I'm too old for that [19:32:46] :D [19:33:29] nuria: after reading again the page on why we picked presto, I feel the section I had written in the doc doesn't bring much - Shall I just drop it? [19:33:53] nuria: instead of trying to merge/modify the one already not-too-bad on wiki [19:33:54] joal: i think so yes, let's keep that info in 1 place [19:34:01] ok cool [19:35:35] nuria: thanks also for the rewording of the intro [19:36:04] I however don't like the title that much :( [19:36:36] joal: do change it [19:36:39] joal: please [19:36:48] Updated, removed the Better, feel better :) [19:39:39] Service announcement: Research Showcase on the “Why the World Read Wikipedia” study just started. You can follow the live stream here: https://www.youtube.com/watch?v=RKMFvi_CCB0 [19:43:14] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10Nuria) 05Open>03Resolved [19:43:16] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) [19:43:27] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10Nuria) [19:43:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Wikistats2 UX bug: table option should not be available in table graph selected - https://phabricator.wikimedia.org/T210424 (10Nuria) 05Open>03Resolved [19:44:55] 10Analytics-Kanban: Per Family Unique Devices Counts - https://phabricator.wikimedia.org/T143927 (10Jdforrester-WMF) This looks done: * https://stats.wikimedia.org/v2/#/all-wikipedia-projects * https://stats.wikimedia.org/v2/#/all-wikiquote-projects * https://stats.wikimedia.org/v2/#/all-wikivoyage-projects [19:45:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539 (10Nuria) @fdans please update docs with new per family and the addition of offset and underestimate [19:47:34] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10Nuria) [19:47:36] 10Analytics, 10Analytics-Kanban: Link to User Contribution page in wikistats UI rather than user page - https://phabricator.wikimedia.org/T210422 (10Nuria) 05Open>03Resolved [19:48:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: AQS unique devices api should report offset/underestimate separately - https://phabricator.wikimedia.org/T164201 (10Nuria) [19:49:18] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539 (10Nuria) [19:49:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: AQS unique devices api should report offset/underestimate separately - https://phabricator.wikimedia.org/T164201 (10Nuria) 05Open>03Resolved [19:50:52] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) These changes have been deployed for a few metrics, see for example new registedred users for all wikipedias: h... [19:51:00] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) [19:51:02] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256 (10Nuria) [19:51:07] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) 05Open>03Resolved [19:51:32] 10Analytics: Update html language for per-domain uniques - https://phabricator.wikimedia.org/T168477 (10Nuria) a:05Milimetric>03fdans [19:51:48] 10Analytics: Update html language for per-domain uniques - https://phabricator.wikimedia.org/T168477 (10Nuria) Let's please update all docs pertaining to unique devices documentation [19:52:04] 10Analytics, 10Analytics-Kanban: Update html language for per-domain uniques - https://phabricator.wikimedia.org/T168477 (10Nuria) [19:52:33] 10Analytics-Kanban, 10Patch-For-Review: Count project-wide unique devices (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928 (10Nuria) [19:52:35] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550 (10Nuria) 05Open>03Resolved [19:53:33] 10Analytics-Kanban: Per Family Unique Devices Counts - https://phabricator.wikimedia.org/T143927 (10Nuria) Closing, we have a remaing task of updating unique devices API wikitech docs [19:53:47] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Nuria) [19:53:49] 10Analytics-Kanban: Per Family Unique Devices Counts - https://phabricator.wikimedia.org/T143927 (10Nuria) 05Open>03Resolved [19:54:28] 10Analytics: Serve global unique device counts externally - https://phabricator.wikimedia.org/T157981 (10Nuria) [19:54:30] 10Analytics-Kanban: Per Family Unique Devices Counts - https://phabricator.wikimedia.org/T143927 (10Nuria) [19:58:11] 10Analytics, 10Analytics-Wikistats: Active Editors metric per project family - https://phabricator.wikimedia.org/T188265 (10Nuria) Update: we have had to postpone this work but we have been able to workin other metrics for project families that are computed more easily than this one. Please see, for example,... [20:03:31] 10Analytics, 10Analytics-Wikistats: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626 (10Nuria) p:05Normal>03Unbreak! [20:03:48] 10Analytics, 10Product-Analytics: As a user of Superset I would like it to be up-to-date so I'm not blocked by bugs that have already been fixed - https://phabricator.wikimedia.org/T211606 (10Nuria) p:05Triage>03High [20:04:38] fdans: closed a bunch of tickets regarding families and unqiue devices, last remainig item is to update dopcs of AQS on wikitech [20:07:33] !log restarted turnilo to clear deleted test datasource [20:07:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:07:51] * unique devices [20:12:21] 10Analytics, 10Product-Analytics: MobileWebSectionUsage schema is whitelisting both session ids and page ids - https://phabricator.wikimedia.org/T209049 (10Nuria) Ping on thsi as it is been a month cc @Tbayer [20:12:56] 10Analytics: ReadingDepth schema is whitelisting both session ids and page ids - https://phabricator.wikimedia.org/T209051 (10Nuria) Ping on this @Tbayer [20:13:15] 10Analytics, 10Readers-Web-Backlog: Print schema is whitelisting both session ids and page ids - https://phabricator.wikimedia.org/T209050 (10Nuria) ping on this @Tbayer [20:14:23] 10Analytics: Update pageview_hourly to include timestamp for better druid indexation - https://phabricator.wikimedia.org/T208230 (10Nuria) [20:14:26] (03CR) 10Mforns: [C: 04-2] ">" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) (owner: 10Mforns) [20:14:59] 10Analytics: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10Nuria) a:03mforns [20:15:21] 10Analytics: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10Nuria) I think this can be resolved with your latest changes, let me know otherwise. [20:17:20] 10Analytics: [EL2Druid] Make RefineTarget compatible with Druid and use it from EventLoggingToDruid - https://phabricator.wikimedia.org/T207207 (10Nuria) ping @mforns I think with recent changes this ticket might need updating. [20:18:35] joal: sparksql is working right? see https://phabricator.wikimedia.org/T196661 which i think can be closed [20:18:53] cc milimetric this can be closed right? https://phabricator.wikimedia.org/T196661 [20:19:14] +1 nuria [20:19:56] 10Analytics: Set up sparksql to try out sql access to hdfs data - https://phabricator.wikimedia.org/T196661 (10Nuria) 05Open>03Resolved [20:20:13] also +1, except I haven't checked whether or not it still spams lots of logs [20:20:35] milimetric: I found a trick for that: spark2-sql --master yarn -S [20:20:40] A lot less logs [20:20:48] oh, cool, will try [20:20:49] thx [20:20:51] np [20:20:56] 10Analytics: [EL2Druid] Make RefineTarget compatible with Druid and use it from EventLoggingToDruid - https://phabricator.wikimedia.org/T207207 (10mforns) @Nuria Are you referencing the "double ingestion" (first hourly, then after a couple days daily), that is supposed to reduce the backfilling ingestion proble... [20:24:02] 10Analytics, 10Analytics-Wikistats: Wikistats. How about historic data? - https://phabricator.wikimedia.org/T186124 (10Nuria) p:05Lowest>03Triage [20:25:12] mforns: i think this ticket can be resolved with your latest changes: https://phabricator.wikimedia.org/T208589 [20:25:23] mforns: if so please resolve [20:25:27] nuria, responding to that right now [20:25:39] no, I think it needs some (small) work [20:27:48] mforns: k [20:30:55] 10Analytics: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10mforns) Hm! good point... I think part of it has been solved by the recent changes in T210099. Namely there was a bug in accessing capsule fields that had underscores in them, like... [20:34:55] nuria, I can even include the needed changes in the current patch if necessary [20:35:08] it might be faster overall [20:35:24] if we need both things for the bannerimpression loading [20:35:31] mforns: not need, i was going through tickets that might be closed and found this one and a few more [20:35:38] k [20:35:41] mforns: banner impression sends geo info [20:35:46] mforns: so not for now [20:35:49] yes, ok [20:36:09] nuria, not yet [20:37:01] 10Analytics, 10Research: Provide data dumps in the Analytics Data Lake - https://phabricator.wikimedia.org/T186559 (10Nuria) [20:37:03] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Copy monthly XML files from public-dumps to HDFS - https://phabricator.wikimedia.org/T202489 (10Nuria) 05Open>03Resolved [20:40:39] 10Analytics, 10EventBus, 10Patch-For-Review: Call to a member function format() on a non-object (boolean) - https://phabricator.wikimedia.org/T211805 (10Reedy) [21:48:44] 10Analytics, 10EventBus, 10MW-1.33-notes (1.33.0-wmf.9; 2018-12-18): Call to a member function format() on a non-object (boolean) - https://phabricator.wikimedia.org/T211805 (10Reedy) 05Open>03Resolved a:03Pchelolo [21:56:04] 10Analytics, 10Wikimedia-Stream, 10Patch-For-Review, 10Services (watching): EventStreams process occasionally OOMs - https://phabricator.wikimedia.org/T210741 (10Pchelolo) [21:56:49] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10RobH) [21:59:03] 10Analytics, 10DC-Ops, 10decommission: decommission oxygen.eqiad.wmnet - https://phabricator.wikimedia.org/T211826 (10RobH) [21:59:20] 10Analytics, 10DC-Ops, 10decommission: decommission oxygen.eqiad.wmnet - https://phabricator.wikimedia.org/T211826 (10RobH) [21:59:23] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10RobH) [22:00:10] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10RobH) a:05RobH>03elukey So this should either go to @elukey or @Ottomata, as this is ready to go into serivice and replace oxygen, then we can decommission oxygen on T211826. Th... [22:19:31] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10Ottomata) @elukey can do this if he wants to, but I don't think Analytics considers oxygen to be part of its domain :) It's used only by SRE. [22:33:19] 10Analytics, 10Operations: setup/install weblog1001/WMF4750 as oxygen replacement - https://phabricator.wikimedia.org/T207760 (10RobH) Ahh, due to past discussions via linked tasks, I assumed he was part of the refresh-replace, so I made assumptions! if this needs to go to someone else @elukey let me know! [22:49:39] !log restarted turnilo to clear deleted test datasource [22:49:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:29:44] (03PS6) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) [23:39:00] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check analytics1037 power supply status - https://phabricator.wikimedia.org/T179192 (10RobH) 05stalled>03Resolved This hasn't reoccured in a very long time, none since this task creation, resolving.