[01:48:34] 10Analytics: Use native timestamp types in Data Lake edit data (needs Hive 1.2) - https://phabricator.wikimedia.org/T161150#4141650 (10Neil_P._Quinn_WMF) [06:05:36] 10Analytics-Kanban, 10Patch-For-Review, 10Puppet: Puppetize job that saves old versions of Maxmind geoIP database - https://phabricator.wikimedia.org/T136732#2345955 (10Joe) I would strongly suggest that any system that wants to archive geoip data from maxmind should create its own repository of data and NOT... [06:08:01] 10Analytics-Kanban, 10Patch-For-Review, 10Puppet: Puppetize job that saves old versions of Maxmind geoIP database - https://phabricator.wikimedia.org/T136732#4141827 (10Joe) >>! In T136732#4139610, @Ottomata wrote: > We could do that, but we wanted something centralized and reproducable (e.g. include a puppe... [06:22:41] joal: morningggg [06:22:57] when you are ready/caffeinated/etc.. lemme know what you think about https://phabricator.wikimedia.org/T164008#4139249 [06:59:28] Hi elukey :) [07:01:45] 2 things to discuss this morning: druid version for upgrade, and AQS->Druid staying alive connection [07:02:40] https://www.youtube.com/watch?v=-lXMx5f9XiU [07:04:50] wooowww [07:05:25] * elukey notes down the band name [07:05:33] +1 for that elukey :) [07:06:28] a lot of things changed since we attempted the 0.10 upgrade, do you think that we'd need more labs testing? [07:07:55] elukey: depends on what we decide to do :) [07:08:22] joal: do you mean what version to pick? I'd go for 0.10 now [07:08:28] elukey: for 0.10.0, rollback is `easy` - regular labs testing sounds good [07:08:45] If we go for more, indeed way more testing [07:09:17] for 0.10.0 - Indexation + sample querying in labs looks goods [07:09:20] nono I'd say 0.10, my question is if it was already tested extensively in labs before the first attempt [07:09:37] just to figure out how much time to allocate [07:09:39] elukey: I wonder about realtime [07:09:55] ah yes that might be an issue [07:15:47] elukey: how do you want us to proceed for labs testing? [07:17:13] now that we have a hadoop cluster in labs we could spin up 2/3 druid nodes in there and configure them [07:17:26] elukey: works for me ! [07:17:34] do you think that we could test 0.10 (and then 0.11) within labs? [07:17:44] elukey: I didn't realize: The hadoop cluser in labs doesn't die fropm too many logs anymore? [07:18:01] I havent' checked in a while, not sure what's the status :D [07:18:06] Ah [07:18:14] Could be a good surprise :) [07:18:34] elukey: Let's go and test 0.10.0 first, then 0.11.0 ? [07:22:12] yeah I think it is a good and conservative way to go [07:22:22] 10 -> 11 might be tricky [07:22:30] yup [07:24:03] so things to do: 1) three new druid instances in labs, creation of the 0.10 deb (basically done thanks to Andrew) [07:24:23] 2) test 0.9.2 in labs (to verify that works), then rolling upgrade [07:25:04] sounds good [07:25:30] I need to think of ways to test indexation [07:25:36] and possibly realtime [07:25:44] hm [07:26:38] elukey: I also have 2 things today that'll delay me - Doctor for Naé at 10:30, and presentation-time with researcher who wrote https://arxiv.org/abs/1803.09461 [07:30:11] joal: don't worry it will take me a bit to set up the whole thing, we can do tomorrow :) [07:30:30] elukey: I need some prep as well, to be ready for testing :) [07:30:54] ah joal, fyi later on I'll migrate zookeeper in eqiad to 3.4.9 [07:30:58] when Marko is online [07:31:07] elukey: ack ! [07:59:44] elukey: reenabling some data generation on hadoop, or shall I wait later? [08:00:32] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10MediaWiki-extensions-Translate, and 2 others: Unable to mark pages for translation in Meta - https://phabricator.wikimedia.org/T192107#4141900 (10Nikerabbit) a:03Nikerabbit [08:01:35] joal: you can proceed [08:02:00] elukey: when do we plan to test? This afternoon? [08:03:45] joal: possibly yes, I'll try to have everything ready in labs this morning [08:09:15] awesome [08:10:03] elukey: Given the presentation is beginning of afternoon and then I catch the kids, I'll be able to test after standup - probably to late for you? [08:12:37] joal: super fine, even tomorrow! Maybe let me know in IRC what are the things to kick/restart/etc.. :D [08:13:00] elukey: I think nothing will needed really [08:13:16] I found the script Andrew used to generate fake webrequests, so wer'll have data [08:13:37] For realtime, I'm gonna write a small streaming job ingesting thos webrequest in order to test [08:14:10] (03CR) 10Fdans: [C: 031] "LGTM! Plusoneing in case any other reviewer wants to take a look, will merge later otherwise." (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427042 (https://phabricator.wikimedia.org/T182718) (owner: 10Nuria) [08:14:36] joal: if you are ok, I'd prep the 0.9.2 cluster to verify first that our baseline works [08:14:42] then I'll upgrade to 0.10 with you [08:14:44] and test again [08:14:47] how does it sound? [08:15:09] Sounds great :) [08:15:45] elukey: I'll reeanble data gen when I get back from the presentation, and ask you where we stand :) [08:15:55] Ok, time for doctor with Naé - Back in a whiule [08:19:09] ah nice it seems we already have d-1.analytics.eqiad.wmflabs [08:19:16] a single node druid worker cluster [08:53:55] (03PS1) 10Fdans: Point uap-core submodule to latest commit [analytics/ua-parser] - 10https://gerrit.wikimedia.org/r/427620 (https://phabricator.wikimedia.org/T192464) [09:00:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update ua-parser-core submodule in wmf ua-parser - https://phabricator.wikimedia.org/T192464#4142069 (10fdans) The ua-parser repo we use (https://github.com/wikimedia/analytics-ua-parser) has two submodules: uap-core and uap-java. UAP-core contains the reg... [09:01:15] 10Analytics, 10Analytics-Kanban: Update version of ua-parser in eventlogging - https://phabricator.wikimedia.org/T192529#4142071 (10fdans) p:05Triage>03High [10:00:26] as FYI I just migrated zookeeper main-eqiad to 3.4.9 [10:00:28] so far all good [11:02:26] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Adjust to Grimoirelab / Bitergia moving to GitLab - https://phabricator.wikimedia.org/T171290#4142387 (10Aklapper) 05Resolved>03Open More to do here, hence reopening [11:08:11] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Adjust to Grimoirelab / Bitergia moving to GitLab - https://phabricator.wikimedia.org/T171290#4142407 (10Aklapper) As discussed in https://gitlab.com/Bitergia/c/Wikimedia/support/issues/30, we will decommission the data dumps on Github.... [11:26:13] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Adjust to Grimoirelab / Bitergia moving to GitLab - https://phabricator.wikimedia.org/T171290#4142454 (10Aklapper) 05Open>03Resolved Updated `User:AKlapper/continuity#Appendix` on officewiki accordingly. Done. [11:31:11] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to stats.wikimedia.org - https://phabricator.wikimedia.org/T165634#4142460 (10Ooswesthoesbes) Yes, everything looks to be working okay. This can be closed. Thanks! :) [11:51:58] * elukey lunch! [13:16:27] ottomata: o/ [13:16:35] yoohoo [13:16:37] zk in main eqiad migrated to 3.4.9 [13:16:41] nothing exploding so far [13:17:16] awseooome [13:17:20] 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-GoranSMilovanovic: SparkR on Spark 2.3.0 - Testing on Large Data Sets - https://phabricator.wikimedia.org/T192348#4142633 (10Ottomata) Gonna ping @jallemandou on this one ^ :) [13:36:29] (03PS1) 10Fdans: Adds wikis to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427658 [13:39:11] (03CR) 10Elukey: [C: 031] Adds wikis to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427658 (owner: 10Fdans) [13:48:54] 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-GoranSMilovanovic: SparkR on Spark 2.3.0 - Testing on Large Data Sets - https://phabricator.wikimedia.org/T192348#4142685 (10JAllemandou) Hi @GoranSMilovanovic. I am not fluent in sparkR but here are a few thoughts: - The example... [13:57:58] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4142715 (10elukey) After a chat with the team we decided to proceed with Druid 0.10 for the moment, since we have basically everything that we need ready to go. I added a couple... [13:59:33] (03CR) 10Ottomata: [C: 031] Add PartitionedDataFrame to spark Refine job (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427187 (owner: 10Joal) [14:00:32] ottomata: need to drop to catch kids, but we can discuss about the spark patch after standup? [14:01:11] elukey: also https://gerrit.wikimedia.org/r/#/c/355471/ [14:01:14] also ottomata, elukey, I'm having issues on labs hadoop-coordinator-1 because of disk-space - Onedeploy of refinery is eating ~10Gb - I'm a bit surprised :( [14:01:55] joal sure [14:01:58] ottomata: I was trying to update the hdfs crontab for camus to run in labs, but diskspace issue :( [14:02:31] du tells me /srv/deployment/analytics/refinery-cache/revs/f99e7dd041c4cee3c796690c9b8a27d31970d93d takes 6.7Gb, but nothing in it takes more than 3Gb :( [14:02:39] * joal don't understand [14:03:17] hm, dunno? use the hammer? [14:03:22] ahahahah [14:03:24] :D [14:03:31] yeah joal lemme clean it up [14:03:53] ok elukey - We'll still need the refinery code though, for camus, ooie and all [14:04:08] !log disabled druid1002's worker as prep step for restart - jvms with a old version running realtime indexation [14:04:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:04:52] I think that the labs hadoop cluster would need to be rebuilt (eventually) with bigger disk space [14:04:57] I didn't think about it at the time [14:04:57] ufff [14:05:03] :( [14:05:03] * elukey blames himself [14:05:20] * joal blames elukey for blaming himself [14:05:26] :) [14:05:28] in theory now that all hiera etc.. is set up it might be a matter of creating new instances [14:05:48] elukey: I really to drop now, will be back for standup, we can go for that at that mone [14:07:09] ack! [14:08:36] so there is only one reve on hadoop-coordinator for refinery [14:08:45] /srv/deployment/analytics/refinery-cache/revs/f99e7dd041c4cee3c796690c9b8a27d31970d93d is 6.7GB, but .git is 3.4GB :) [14:09:50] ah! [14:09:51] elukey@hadoop-coordinator-1:~$ sudo pvs PV VG Fmt Attr PSize PFree /dev/vda4 vd lvm2 a-- 60.50g 60.50g [14:10:00] so there are 60g free [14:10:08] * elukey creates a partition [14:19:01] joal: /dev/mapper/vd-srv 59G 6.7G 50G 12% /srv [14:19:03] better now :) [14:19:23] !log add 60G /srv partition to hadoop-coordinator-1 in analytics labs [14:19:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:22:50] 10Analytics-Kanban, 10Patch-For-Review, 10Puppet: Puppetize job that saves old versions of Maxmind geoIP database - https://phabricator.wikimedia.org/T136732#4142770 (10faidon) >>! In T136732#4139610, @Ottomata wrote: > We could do that, but we wanted something centralized and reproducable (e.g. include a pu... [14:27:24] (03PS1) 10Lucas Werkmeister (WMDE): Don’t count deprecated constraint statements [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/427673 [14:30:00] 10Analytics-Kanban, 10Patch-For-Review, 10Puppet: Puppetize job that saves old versions of Maxmind geoIP database - https://phabricator.wikimedia.org/T136732#4142808 (10Ottomata) I don't have much context of how geowiki runs, but storing this in HDFS would be fine. We (I?) just thought it would be better to... [14:30:03] !log disabled druid1001's middlemanager, restarted 1002's [14:30:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:35:10] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code: When cursor is out of graph overlay should not display - https://phabricator.wikimedia.org/T192416#4142816 (10Nuria) [14:36:18] (03CR) 10Nuria: [C: 032] Add job and query for page previews aggregation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [14:38:11] 10Analytics-Kanban, 10Patch-For-Review, 10Puppet: Puppetize job that saves old versions of Maxmind geoIP database - https://phabricator.wikimedia.org/T136732#4142823 (10fdans) Got it, yeah uploading to HDFS seems pretty sensible. The only documented application for this archive is history reconstruction, so... [14:40:46] (03CR) 10Nuria: [V: 032 C: 032] Adds wikis to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427658 (owner: 10Fdans) [14:45:30] (03CR) 10Nuria: "* Why do we need a fork of this package rather than the original package? In what way is our code different?" [analytics/ua-parser/uap-core] - 10https://gerrit.wikimedia.org/r/427415 (https://phabricator.wikimedia.org/T192465) (owner: 10Fdans) [15:03:28] nuria_: I may be wrong, but i just ran a diff between the original repo and our fork ok uap-core and the only difference is the release tags (cc ottomata, about Nuria's comment above) [15:03:46] fdans: on meeting , can talk later [15:03:51] i don't rembemer or know :/ [15:04:07] 10Analytics: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4142906 (10elukey) p:05Triage>03Normal [15:04:11] https://www.irccloud.com/pastebin/WtfotAU5/ [15:09:22] in the meantime, reimaging analytics1069 to stretch [15:09:35] :O :) [15:09:42] you da best [15:10:30] ottomata: funny, I don't think we've ever touched this fork other than to pull stuff from upstream [15:10:53] fdans: maybe madhuvishy remembers? [15:11:04] if there's no reason to use the fork, we might as well just put the dep in our pom directly, ya? [15:12:51] Definitely predates even me joining the team, so I don't know or remember why either [15:12:56] hm ok! [15:13:00] fdans: let's try without then? [15:13:10] thanks madhu [15:13:30] oh ottomata, the original repo is not published in archiva? [15:13:43] and there's no releases in the repo from 2015 [15:13:49] fdans it should be proxied if you try to build [15:14:00] ok, lemme try [15:14:05] 10Analytics, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4142906 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1069.eqiad.wmnet'] ``` The log can be found... [15:16:17] ottomata: I don't understand what you mean by proxied [15:16:21] right now the dep is [15:16:24] https://www.irccloud.com/pastebin/hksDdDTu/ [15:16:40] this points to our fork [15:17:11] oh wait i'm dumb [15:17:19] i'm mixing things in my head [15:17:54] do you have a minute in batcave ottomata ? [15:18:06] fdans: [15:18:06] https://search.maven.org/#artifactdetails%7Ccom.github.coveo%7Cuap-java%7C1.3.1-coveo1%7Cjar [15:18:20] if you put that Dependency Information in our pom [15:18:26] archiva will dl it from maven central and cache it [15:18:32] during build [15:18:43] coveo? [15:18:47] not sure if that is the right one [15:18:56] i don't think it is [15:18:58] maybe that's why we fork/build? so that we can upload something somewhere (archiva) [15:19:09] yeah i was thinking that [15:19:50] ok here's the thing: we have this repo: https://github.com/wikimedia/analytics-ua-parser [15:19:51] ha fdans https://github.com/ua-parser/uap-java/issues/1#issuecomment-108633895 [15:20:50] ha [15:21:59] oh [15:22:05] fdans: our repo is not a fork? [15:22:09] no [15:22:12] it is just a container with submodules [15:22:13] ok [15:22:14] it's two submodules [15:22:15] this makes sense [15:22:22] and i'm real confuzzled by it [15:22:26] i think? [15:22:34] because where are we importing it from? [15:22:40] https://github.com/wikimedia/analytics-ua-parser/commit/16a688ca793c6472479f59da6071bd452e26c2cc [15:23:02] hm [15:23:06] yea yea that's what i was doing [15:23:07] (03CR) 10Milimetric: [C: 04-1] "looks good overall, but I want to standardize how we do computed properties and triggering, so that the code is more homogeneous. The com" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427042 (https://phabricator.wikimedia.org/T182718) (owner: 10Nuria) [15:24:18] ottomata: this tree of tasks makes sense: https://phabricator.wikimedia.org/T189230 [15:24:19] BUT [15:24:25] fdans: i think...the wmf* builds just combine the uap-java with uap-core regexes [15:24:26] ? [15:24:31] that's why we build our own? [15:25:03] but uap-java and uap-core aren't directly linked right? [15:25:15] right [15:25:24] we probably don't need to updated uap-java, i doubt it has changed (has it?) [15:26:49] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#4143041 (10mobrovac) [15:27:00] yeah, the only thing to update is the core [15:27:45] OOOOOOO [15:28:23] ottomata: this feels evil, but here's the last piece of the mystery https://github.com/wikimedia/analytics-ua-parser-uap-java/blob/master/pom.xml#L25-L29 [15:28:35] mforns: for https://gerrit.wikimedia.org/r/#/c/425281/4..5/oozie/virtualpageview/hourly/virtualpageview_hourly.hql on line 75, what's wrong with just passing useragent as user_agent_map? [15:28:43] it has too many fields? [15:28:58] or the bit above that fdans, [15:29:00] not testResources [15:29:01] ottomata: it's getting the regexes yaml by assuming it's a submodule [15:29:02] but resources, ya [15:29:06] yeayea [15:29:11] oh right [15:29:11] milimetric, EL useragent is a struct, not a map [15:29:12] ../ [15:29:26] mforns: ooh, ok, thanks, yeah, that's the only way I can think of too then [15:29:31] hive does not know how to convert... [15:29:39] that's a bit irregular no ottomata ? [15:29:40] mforns: if you tested that we can merge and deploy, no? [15:29:44] milimetric, also, useragent has at least an extra field is_bot [15:29:54] yea, I tested with real data [15:30:04] fdans: yes, but what else? the regexes are maintained as a separate repo, which is not a maven module of any kind [15:30:07] ok, cool, let's deploy then? Better to get them the data sooner, right? [15:30:24] (03CR) 10Milimetric: [V: 032] Add job and query for page previews aggregation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/425281 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [15:30:40] milimetric, it will also help me test the druid indexation [15:30:41] woudl be better if we uap-java took a regexes.yaml file in its constructor or something [15:30:51] so we wouldn't only have to update our regexes file in refinery source or something [15:30:57] thanks milimetric :] [15:31:06] ottomata: it feels like uap-core should be a dependency of uap-java [15:31:08] but alas [15:31:08] https://github.com/wikimedia/analytics-ua-parser-uap-java/blob/master/src/main/java/ua_parser/Parser.java#L34 [15:31:09] fdans: it's your ops week, wanna deploy together? [15:31:10] but in our case it's a sibling [15:31:30] fdans: if you want to make this real and right... you'd have to fix uap-java :p [15:32:07] uap-java expects "/ua_parser/regexes.yaml", i think it will find it in the .jar [15:32:07] milimetric: yeah! [15:32:22] ok, to the batcave! [15:33:13] ottomata: I think for now I'm happy to document the whole thing and totally kick the can [15:33:21] unless you feel strongly that it should be refactored [15:35:25] fdans: i think that if we do not need a fork we should not mainatin one [15:35:43] we do need it tho nuria_ it seems [15:35:55] fdans: it makes updates easier, so if we do not (ottomata might know best) then let's just do away with it [15:35:59] nuria_: we do [15:36:09] we just figure dit out [15:36:18] mforns: saw your comments on state, will get to those today [15:36:22] because we have to repackage the thing with an updated version of the regexes ourselves [15:36:29] no one is maintaining uap-java [15:36:55] ottomata: isn't that a different package [15:36:56] although, fdans, we don't technically need a fork of uap-java? [15:37:03] we just need the submodule checked out [15:37:08] it could check out the ua-parser repo [15:37:17] ua-parser/uap-java [15:37:28] nuria_: uap-java builds with the content of uap-core in it [15:37:37] so we need to build uap-java to update ua regexes [15:37:46] see: https://github.com/ua-parser/uap-java/blob/master/pom.xml#L17 [15:37:58] and this nastiness: https://github.com/ua-parser/uap-java/blob/master/src/main/java/ua_parser/Parser.java#L34 [15:38:21] ottomata: so we have forks of 1 package (uap-java?) or 2 (uap-java and uap-python)? [15:38:41] ottomata: wow Parser.java, deluxe [15:40:33] !log deploying refinery [15:40:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:43:31] nuria_: i think the only reason we have the 'fork' is to change the versions [15:43:33] and build [15:43:58] it looks like we do fork https://github.com/wikimedia/analytics-ua-parser-uap-core [15:44:02] but i'm not sure we need to do that [15:44:21] we could just make the submodule in https://github.com/wikimedia/analytics-ua-parser point to the upstream uap-core [15:48:56] nuria_, I don't think I left comments on state...? [15:55:46] ottomata nuria_: maybe we do it not to do a submodule from gerrit to github? [15:55:58] you know, keepin it all in the family? [15:59:44] * joal has read the backlog and knows why elukey is ops :) [15:59:52] Thanks elukey for hthe partition [16:01:27] mforns: yohooo [16:06:02] 10Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#4143192 (10faidon) I'm a Pivot newbie -- how could this be inferred? I've tried adding an `Ip ~ ":"` but that can only appear as a filter, not under split; in split I can only add "Ip" as a field, but... [16:06:53] (03PS1) 10Mforns: Add jobs for druid indexing of virtualpageviews [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427696 (https://phabricator.wikimedia.org/T192305) [16:07:09] (03PS2) 10Mforns: [WIP] Add jobs for druid indexing of virtualpageviews [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427696 (https://phabricator.wikimedia.org/T192305) [16:24:01] 10Analytics, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4143250 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1069.eqiad.wmnet'] ``` and were **ALL** successful. [16:27:53] !log analytics1069 reimaged to Debian stretch [16:27:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:37:43] 10Analytics, 10Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#4143303 (10Milimetric) This is so cool, @EBernhardson, thank you. Formatting for my future reference: ``` select count(1) as n_ip, percentile_approx( n_... [16:38:32] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Reimage the Debian Jessie Analytics worker nodes to Stretch. - https://phabricator.wikimedia.org/T192557#4143306 (10fdans) [16:39:20] elukey: I have reenabled camus on labs, but the job doesn't succeed :( [16:39:29] elukey: it starts, but doesn't even run [16:40:24] joal: any error/trace that indicates what is happening? [16:43:23] 10Analytics, 10Research: Refactor pagecounts-ez generation - https://phabricator.wikimedia.org/T192474#4143320 (10fdans) p:05Triage>03Low [16:45:05] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code: When cursor is out of graph overlay should not display - https://phabricator.wikimedia.org/T192416#4137990 (10fdans) p:05Triage>03Normal [16:47:50] 10Analytics, 10MediaWiki-Vagrant, 10Services (watching): Vagrant's /var/log/daemon.log filling up with kafka errors - https://phabricator.wikimedia.org/T187102#4143336 (10fdans) p:05Triage>03Normal [16:50:19] 10Analytics, 10Analytics-Wikistats: Feedback on hive table mediawiki_history by Erik Z - https://phabricator.wikimedia.org/T178591#4143363 (10mforns) p:05Triage>03Normal [16:50:20] elukey: I'm after some [16:51:35] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4143367 (10mforns) [16:51:41] 10Analytics, 10Analytics-Kanban: Update user_history and page_history column naming convention - https://phabricator.wikimedia.org/T188669#4143368 (10mforns) p:05Triage>03Unbreak! [16:53:32] 10Analytics: Mediawiki History: moves counted twice in Revision - https://phabricator.wikimedia.org/T189044#4143373 (10mforns) p:05Triage>03Normal [16:54:11] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: [EL sanitization] Ensure presence of EL YAML whitelist in analytics1003 - https://phabricator.wikimedia.org/T189691#4143379 (10mforns) p:05Triage>03Normal [16:55:49] 10Analytics: Review parent task for any potential pageview definition improvements - https://phabricator.wikimedia.org/T156656#4143386 (10mforns) p:05Normal>03Low [16:56:15] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149#4143389 (10mforns) p:05Normal>03High [17:01:35] 10Analytics: Some fields in webrequest druid dataset should eb ingested as numbers - https://phabricator.wikimedia.org/T167494#3334762 (10mforns) We are upgrading Druid in short. However, we don't know if the frozen version of Pivot that we have will support the new Druid version. And we don't know if this will... [17:02:00] 10Analytics: Some fields in webrequest druid dataset should eb ingested as numbers - https://phabricator.wikimedia.org/T167494#4143449 (10mforns) p:05Normal>03Low [17:03:24] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Product-Analytics: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009#4143456 (10mforns) p:05Normal>03Triage [17:03:26] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Product-Analytics: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009#3482778 (10mforns) p:05Triage>03Normal [17:03:51] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Product-Analytics: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009#4143458 (10mforns) a:03Nuria [17:05:35] * awight crawls out from under a rock [17:07:24] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [17:07:27] ottomata: --^ [17:07:30] am there! [17:12:02] elukey: batcave 2? [17:12:13] see link above elukey --^^ [17:15:53] joal: sure [17:20:43] fdans: did the sync to hdfs finish? [17:20:56] if so, just log here that it's done [17:26:03] 10Analytics, 10Analytics-Wikistats: Routing code allows invalid routes - https://phabricator.wikimedia.org/T188792#4143539 (10Milimetric) p:05Unbreak!>03High [17:26:29] nuria_: if you want to join we're triaging wikistats stuff [17:29:01] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180#4143542 (10Milimetric) p:05Triage>03Normal [17:30:06] 10Analytics, 10Analytics-Wikistats: Present Wikistats 2 charts for the period selected by the user. - https://phabricator.wikimedia.org/T183183#4143543 (10Milimetric) p:05Triage>03High [17:30:40] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Intervals/buckets for data arround pageviews per country in wikistats maps - https://phabricator.wikimedia.org/T188928#4143546 (10Milimetric) p:05Triage>03Normal [17:31:58] 10Analytics, 10Analytics-Wikistats: Add wikistats metric "top-by-edits" - https://phabricator.wikimedia.org/T189620#4143553 (10Milimetric) p:05Triage>03Normal [17:32:32] 10Analytics, 10Analytics-Wikistats: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626#4143554 (10Milimetric) [17:33:48] 10Analytics, 10Analytics-Wikistats: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626#4143567 (10Milimetric) p:05Triage>03Normal [17:35:08] 10Analytics: Add a --dry-run option to the sqoop script - https://phabricator.wikimedia.org/T188556#4143583 (10Milimetric) p:05Triage>03Normal [17:35:10] !log refinery deployment - sync to hdfs finished [17:35:14] done milimetric [17:35:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:35:27] thanks fdans [17:41:18] * elukey off! [17:45:29] (03PS1) 10Mforns: Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) [18:01:20] (03PS2) 10Joal: Add PartitionedDataFrame to spark Refine job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427187 [18:01:35] milimetric: just got back sorry [18:02:44] ottomata: --^ As discussed [18:03:22] ottomata: I think I could do with some help on hadoop-labs [18:03:56] (03PS2) 10Mforns: Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) [18:04:11] (03PS3) 10Mforns: [WIP] Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) [18:09:30] ottomata: From the yarn-UI in labs, I see that yarn thinks it has no workers [18:12:07] (03CR) 10Nuria: [C: 031] [WIP] Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [18:15:12] ok ottomata - soft-rebooting the workers did the trick [18:18:36] (03CR) 10Nuria: [WIP] Add jobs for druid indexing of virtualpageviews (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427696 (https://phabricator.wikimedia.org/T192305) (owner: 10Mforns) [18:19:53] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code: When cursor is out of graph overlay should not display - https://phabricator.wikimedia.org/T192416#4143723 (10Nuria) p:05Normal>03High [18:21:41] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Intervals/buckets for data arround pageviews per country in wikistats maps - https://phabricator.wikimedia.org/T188928#4143725 (10Nuria) p:05Normal>03High [18:21:54] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Intervals/buckets for data arround pageviews per country in wikistats maps - https://phabricator.wikimedia.org/T188928#4024359 (10Nuria) Moving to high priority (cc @milimetric) as we agreed to do this this quarter [18:22:04] milimetric: do you want to brain bounce geo wiki vetting? [18:23:05] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4143728 (10Nuria) Let's make sure to test whether pivot works with this release [19:17:24] nuria_: late lunch, back now, no, gotta still step through, I was helping with deploy this morning [19:27:51] (03PS1) 10Joal: [TMP] Add fake webrequest-stream job for labs test [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427766 [19:41:38] (03PS1) 10Sahil505: Hide "Load more rows..." once max data is visible in Table Chart [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427774 (https://phabricator.wikimedia.org/T192407) [19:44:26] (03CR) 10Sahil505: "> Uploaded patch set 1." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427774 (https://phabricator.wikimedia.org/T192407) (owner: 10Sahil505) [19:49:10] (03PS2) 10Sahil505: Hide "Load more rows..." once max data is visible in Table Chart [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427774 (https://phabricator.wikimedia.org/T192407) [19:49:21] (03PS4) 10Mforns: [WIP] Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) [19:49:59] (03PS5) 10Mforns: Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) [19:50:44] (03CR) 10Mforns: [V: 032] "OK, it's tested now. I had some problems with a couple lines I forgot :/ Should be fine now!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [19:53:51] (03CR) 10Nuria: [C: 031] "As long as it is tested should be ok to merge." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [20:20:28] (03CR) 10Milimetric: [C: 032] Correct hardcoded ADD JAR in virtualpageview_hourly query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/427722 (https://phabricator.wikimedia.org/T186728) (owner: 10Mforns) [20:20:44] thx! [20:21:25] mforns: are you still working? [20:21:34] yep [20:21:36] so late! [20:21:42] ok, I'll deploy and if you want we can start it [20:21:43] started late [20:21:58] ok, but is IRC OK? [20:22:04] ofc [20:22:09] cool [20:22:27] I'm in the living room, people asleep [20:22:58] :) no prob, I've started scap, will sync, and then start the job [20:23:06] so I'll only ping you if something goes wrong [20:23:42] oh, and I created the table a while back, so that's ready [20:32:28] (03CR) 10Nuria: [C: 04-1] "Please do take a second look, the loadMorerRrows button does not display on 1st load of the page." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/427774 (https://phabricator.wikimedia.org/T192407) (owner: 10Sahil505) [20:36:10] !log Synced latest refinery version to HDFS [20:36:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:39:18] !log launched virtual pageviews job, it has id 0026169-180330093100664-oozie-oozi-C [20:39:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:41:28] nice job, mforns, it's running and I checked the data it generates and it looks good [20:41:36] woohooo! [20:42:02] tomorrow will test the druid loading then [20:42:09] thanks for deploying! [20:42:48] np at all, have a good night [21:11:50] (03PS2) 10Joal: [TMP] Add fake webrequest-stream job for labs test [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/427766 [21:13:03] git fetch https://gerrit.wikimedia.org/r/analytics/refinery/source refs/changes/66/427766/2 && git checkout FETCH_HEAD [21:13:07] soory [21:35:52] milimetric: one thing i do not understand about this https://github.com/wikimedia/analytics-wikistats2/blob/master/src/components/detail/Detail.vue#L84 [21:36:00] milimetric: is why s=do we need Object.assign? [21:36:58] milimetric: ahhhh i get it , sorry! [21:37:33] milimetric: i wonder how this is going to play out with mixings, trying now [21:46:28] 10Analytics, 10Patch-For-Review, 10User-Elukey: Update druid to latest release (0.11) - https://phabricator.wikimedia.org/T164008#4144424 (10JAllemandou) First step of testing confirmed on labs with druid 0.9.2: - Indexation from hadoop - Realtime indexation with tranquility Only issue we should fix befor... [21:52:52] Gone for tonight a-team [21:52:56] See you tomorrow ! [22:18:28] sorry nuria_ I’m around now if you have trouble [22:18:57] should be no different for a mixin, as that just executes like the component itself [23:36:32] milimetric: indeed, i think i got it working