[00:02:00] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#3912816 (10Platonides) > However, it's operating without any redundancy, in terms of both individual hardware failure, an... [00:24:56] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#3919263 (10Platonides) [00:33:23] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#3919279 (10faidon) >>! In T185319#3919216, @Platonides wrote: >> However, it's operating without any redundancy, in terms... [00:34:14] 10Analytics-Data-Quality, 10Operations, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3919281 (10Tbayer) Same for November (ulsfo): ``` total_bytes requests 2124819017324015 65380125071 1 row selected (3809.329 second... [00:38:05] 10Analytics-Data-Quality, 10Operations, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3919287 (10faidon) Seems fairly consistent: LibreNMS has recorded November as 2.45PB. October is incomplete, unfortunately, so we can't compare... [01:13:27] 10Analytics-Data-Quality, 10Operations, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3919331 (10Tbayer) >>! In T185350#3918346, @Tbayer wrote: > @ottomata notes that the response_size field should correspond to the "Size of respo... [01:43:06] 10Analytics-Data-Quality, 10Operations, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3919416 (10Ottomata) Correct! :) [01:45:52] ottomata: thanks for confirming ;) check my documentation edits too https://wikitech.wikimedia.org/w/index.php?title=Analytics/Data_Lake/Traffic/Webrequest&diff=prev&oldid=1781167 https://wikitech.wikimedia.org/w/index.php?title=Cache_log_format&diff=prev&oldid=1781166 (i guess the latter page could use further updates) [01:48:12] 10Analytics-Data-Quality, 10Operations, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3919423 (10Tbayer) Regarding 1.: We haven't yet heard back from @Milimetric about what the issue was back then, but after looking at [[https://g... [01:51:04] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#3919427 (10Platonides) I was just trying to address the stated concern, not make it a perfect solution (we will probably... [02:01:33] 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#3912816 (10Krenair) Pretty sure MW has supported having multiple destinations for these streams for years now. So you cou... [02:03:06] 10Analytics, 10Analytics-Wikistats: Questionable metrics from Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3919443 (10Pine) Thanks for explaining that @JAllemandou, I was thinking that "g" == "grand". [02:04:19] 10Analytics, 10Analytics-Wikistats: Questionable metrics from Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3919444 (10Pine) May I suggest replacing "g" with "b" for this usage? That would me more consistent with the use of "m" for "million", I think. [02:05:50] 10Analytics, 10Analytics-Wikistats: Confusing abbreviation on Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3919445 (10Pine) 05Invalid>03Open [02:06:22] 10Analytics, 10Analytics-Wikistats: Confusing abbreviation on Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3869685 (10Pine) I've renamed this task to reflect the nature of the problem and re-opened it. [09:15:23] 10Analytics, 10Analytics-Wikistats: Confusing abbreviation on Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3919602 (10JAllemandou) @Pine: We have discusse this naming convention within the team a while ago, and decided to go for "g" instead of "b" for internationalization reasons. "b" stands... [09:19:59] I am trying to run the eventlogging test suite locally [09:20:09] but it doesn't seem to be easy on macos [09:21:29] elukey: I can't even recall having ever done that (I'm actually sure I did, but can't recall ... How baaaaaad) [09:22:27] joal: nothing urgent, I was trying to add tests for https://gerrit.wikimedia.org/r/#/c/405687/9/eventlogging/handlers.py [09:22:37] this is what me and Andrew came up with to patch EL [09:22:41] for the duplicate events [09:23:25] elukey: looks good (from a 2 minutes read ;) [09:24:11] joal: the difficult part is now testing it :D [09:25:06] :) [09:25:20] elukey: as always with async code and race conditionsd [09:26:15] !log Dataloss warning for upload and text 2018-01-23:06 is confirmed to be false positive [09:26:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:36:12] (03PS4) 10Joal: Refactor json-refine and json-to-Druid for spark2 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405770 [11:34:08] * elukey lunch + errand! [11:41:52] 10Analytics, 10Analytics-Cluster: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3919735 (10JAllemandou) Found the precise line: https://github.com/wikimedia/analytics-camus/blob/master/camus-etl-kafka/src/main/java/com/linkedin/camus/e... [11:45:34] (03PS5) 10Joal: Refactor json-refine and json-to-Druid for spark2 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405770 [11:49:57] (03PS1) 10Joal: Update camus part checker topic name normalization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405867 (https://phabricator.wikimedia.org/T171099) [13:45:51] ottomata: o/ [13:52:06] 10Analytics-Kanban, 10ChangeProp, 10EventBus, 10Services (watching), 10User-Elukey: Export burrow metrics to prometheus - https://phabricator.wikimedia.org/T180442#3919816 (10elukey) I tried today to build the https://github.com/jirwin/burrow_exporter package with a simple Debian configuration on boron a... [13:53:00] so I checked our burrow repo on gerrit and I found two [13:53:01] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3919817 (10JAllemandou) a:03JAllemandou [13:53:22] I think that https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/burrow is the right one but let me know otherwise [13:53:29] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats v2 has a weird localization issues on search - https://phabricator.wikimedia.org/T182987#3919832 (10fdans) I've tested this by changing my browser language and it seems like this problem has been solved by @mforns 's big wikiselector change. [13:53:46] what I'd do is add a new origin (github), pull all the new commits in master and push them to our origin [13:53:47] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats v2 has a weird localization issues on search - https://phabricator.wikimedia.org/T182987#3919837 (10fdans) 05Open>03Resolved a:03fdans [13:54:20] then update the debian branch accordingly (we may not need the golang pkgs anymore since Debian supports a lot of them now) [14:04:49] https://packages.debian.org/search?searchon=names&keywords=golang-github [14:25:53] (03CR) 10Joal: "Tested on eventbus data - Seems to work correctly :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405867 (https://phabricator.wikimedia.org/T171099) (owner: 10Joal) [14:35:34] (03PS5) 10Joal: Refactor geo-coding function and add ISP [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/403916 (https://phabricator.wikimedia.org/T167907) [14:36:29] (03CR) 10Joal: "Amost working - Hitting that limitation: https://issues.apache.org/jira/browse/SPARK-19261" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405770 (owner: 10Joal) [14:39:41] 10Analytics-Kanban, 10ChangeProp, 10EventBus, 10Services (watching), 10User-Elukey: Export burrow metrics to prometheus - https://phabricator.wikimedia.org/T180442#3919966 (10elukey) For burrow it turns out that the following list can replace the godeps in the debian dir: ``` Build-Depends: debhelper (>... [14:45:39] (03CR) 10Ottomata: [C: 031] Update camus part checker topic name normalization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405867 (https://phabricator.wikimedia.org/T171099) (owner: 10Joal) [15:22:34] elukey: i think the execpt Exception: raise is equivalent to finally: [15:22:43] but, maybe not? [15:22:54] i don't know which is the better way, but it seems finally was made for this purpose [15:22:58] which is why I suggested it [15:23:29] what we want to do is always flush() when the loop is done, whether or not it is from a break or an exception [15:23:52] heckay it might even be nice to abstract this out somehow into streams.py instead of handlers [15:24:00] and make creating a stream take a done_callback [15:24:16] then any handler could automatically have a done_callback called when the stream finally closes [15:24:22] :) [15:24:26] but, not today ... :) [15:24:44] either way, i'm not picky for htis patch, and I haven't tested, so if you like the except raise better, i'm fine with it [15:25:01] joal: how goes archiva stuff? [15:25:31] also, can I test your spark 2 refine? i probably want to rebase mine on top of yours, because its not a minimal amount of work changing scopt to properties [15:26:04] 10Analytics-Kanban, 10ChangeProp, 10EventBus, 10Services (watching), 10User-Elukey: Export burrow metrics to prometheus - https://phabricator.wikimedia.org/T180442#3920086 (10elukey) >>! In T180442#3769470, @fgiunchedi wrote: > I took a quick look at burrow_exporter (without building/running it) and it l... [15:26:46] 10Analytics-Kanban, 10ChangeProp, 10EventBus, 10Services (watching), 10User-Elukey: Export burrow metrics to prometheus - https://phabricator.wikimedia.org/T180442#3920092 (10Ottomata) > If the number of consumer groups and partitions grows It almost certainly will. :) [15:27:35] ottomata: sorry just seen the pings! I still need to test all of this so we can definitely chat about it in SF. Don't want to push for any solution, I'll check finally better and we'll see [15:27:48] k! :) [15:28:00] haha, we'd better be careful too many things to talk about in SF! :p [15:28:45] be prepared because I'd really like to get an introduction to our use of archiva if you'll have time/patience :P [15:29:38] (03CR) 10Ottomata: "Oh, nasty. Spark 2.2. Ya we can upgrade no problem! Probably won't be difficult now at all. I think you should just change this to refi" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/405770 (owner: 10Joal) [15:29:48] elukey: sure! [15:29:53] ottomata: about burrow - shall we put some effort in upgrading to 1.0.0? Seems only one golang dep away [15:30:06] elukey: guess you've read https://wikitech.wikimedia.org/wiki/Archiva ? [15:30:12] elukey: def [15:30:14] i think we shoudl do so now [15:30:21] I haven't! [15:30:22] eventlogging is the only use of it afaik [15:30:27] and giving it some love would be good [15:30:31] +2 to 1.0 [15:30:35] i think we should just do it [15:31:31] my idea was to add a new origin for github, pull from upstream, push all the new commits to our master (175!) [15:31:51] and then merge them to debian, update the debian dir, review/merge/blabla [15:34:18] les do iiit [15:34:40] elukey: usually i just fetch the tag and then merge it to debian branch and push that to gerrit [15:34:53] then make a gerrit review patch for the debian/ change in the debian branch [15:34:58] so, fetch tag, merge to debian [15:35:20] push tag and debian directly to git/gerrit (so you don't have to review the upstream changes in gerrrit), then make whatever changes to debian/ you need and git review [15:37:00] ottomata: checking how to fetch the specific tag [15:37:09] the main issue is that one dep is not in debian yet [15:37:18] so we should build it and then we'd be ok [15:37:19] aye saw that [15:37:22] ya [15:41:55] 10Analytics-Cluster, 10Analytics-Kanban: Upgrade spark2 .deb to spark 2.2.1 - https://phabricator.wikimedia.org/T185581#3920149 (10Ottomata) [15:42:00] 10Analytics-Cluster, 10Analytics-Kanban: Upgrade spark2 .deb to spark 2.2.1 - https://phabricator.wikimedia.org/T185581#3920160 (10Ottomata) [16:30:59] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248#3920273 (10Ottomata) [16:31:01] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Upgrade spark2 .deb to spark 2.2.1 - https://phabricator.wikimedia.org/T185581#3920272 (10Ottomata) [16:31:17] elukey: have you sent email announcement about cluster downtime feb 6? [16:31:35] ottomata: not yet, I was about to ask if we need to do it before SF [16:32:24] probalby should send email out, feb 6 is not long after SF [16:32:30] for full cluster downtime we should give people lots of notice [16:32:44] also, just thought of something: will feb 6 be enough time for beginning of month jobs to finish? [16:32:44] hmmm [16:32:48] joal: ^^? [16:33:30] ottomata: feb 6th everything should be done (except in case of failures) [16:35:47] ottomata, elukey - Which data from isp-maxmind should I push to Druid (if any)? [16:38:03] joal: no context about this sorry :( [16:38:26] np elukey :) It can wait anyhow (I should probably as Faidon about that) [16:38:55] (03PS1) 10Joal: Add ISP data to webrequest table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405899 (https://phabricator.wikimedia.org/T167907) [16:39:33] (03CR) 10Joal: "To be merged and deploy AFTER deploy of refinery-source with https://gerrit.wikimedia.org/r/#/c/403916/" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405899 (https://phabricator.wikimedia.org/T167907) (owner: 10Joal) [16:44:39] ottomata,joal - ok to send the email then? analytics@ and engineering@ ? [16:44:45] works for me elukey [16:45:00] elukey: can you add research? [16:45:52] yaaa let's do it [16:47:50] joal: wiki-research-l ? [16:48:01] yessir [16:48:07] ack! [16:49:00] ottomata,joal - you have a draft in your inbox [16:49:10] is there anything else that needs to be mentioned? [16:50:16] elukey: sounds good, maybe format it it so the date and times are very visible on their own with some bold or something :) [16:50:21] but yaaa loooks good, send away [16:50:36] Luca: I'd put a TL;DR in bold at the beginning, just in case (3 times is good -> Header, TlDR, core message :) [16:50:56] ahaha okok [16:51:01] ottomata: you read my mind :) [16:58:29] sent! [16:58:45] Thanks a lot elukey :) [16:59:16] going afk, will check later! [16:59:20] Bye elukey [17:08:47] thanks! [17:34:27] Ok a-team, gone for diner [18:45:10] phuedx: https://www.slideshare.net/MarkusWinand/modern-sql [18:45:17] thanks pnorman! [18:45:34] HaeB: you might enjoy reading over that link too! :] [19:14:42] hey analytics, how do I select in wikistats 2 a family of projects (e.g. all WIkipedias)? [19:14:59] is it a UI glitch or are project family aggregations not available? [19:15:11] DarTar: Not available yet [19:15:22] joal: oh I see [19:15:29] DarTar: Backend supports it, but front not yet [19:15:40] acutally, middleware not yet [19:15:44] I am looking for a canonical and citable reference on the total # of articles in WIkipedia across languages [19:16:03] wikistats 1.0 doesn’t update any more and this is not on wikistats 2 yet [19:16:08] any suggestions? [19:16:20] nope [19:16:36] I can provide you with a value, but it doesn't help [19:16:56] gotcha [19:17:01] there’s this of course https://meta.wikimedia.org/wiki/List_of_Wikipedias [19:17:52] which is the source of statements such as: [19:17:56] DarTar: Making requests per wiki to the API would work, but not scalable nor nice [19:18:04] “Distribution of the 47,348,405 articles in different language editions (as of 23 January 2018)” on the enwiki article about Wikipedia [19:18:22] I think that’s fine, I’ll use a generic “more than 40M” [19:18:30] which can be backed by the legacy reports [19:18:38] thanks joal [19:18:51] np, sorry for not having it yet [19:22:26] joal: understandable, and I am glad I can finally stop trying every possible combination of keywords in the UI ;) [19:25:25] (03CR) 10Joal: [C: 031] "Tested over an hour of data." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/403916 (https://phabricator.wikimedia.org/T167907) (owner: 10Joal) [20:01:29] Hi ottomata - still here? [20:02:26] yaa [20:02:56] ottomata: I wonder about https://phabricator.wikimedia.org/T185419 [20:04:37] hm [20:04:47] ottomata: Looking at the files permission, there is some weird things [20:05:22] I'm asuming problem comes from parent folder not having write right for group - but I'd rather confirm [20:06:28] OOO interesting [20:06:43] that sounds likely, but write? i see 2018 files in that directory [20:06:44] in dhfs [20:06:45] hdfs [20:07:45] oh top1000...? [20:07:53] ottomata: my understanding is that top files are computed by Erik [20:07:57] ah hm [20:08:00] i betcha that is the problem [20:08:07] with correct group [20:08:08] joal i'm going to chgrp 2018 and the parent [20:08:10] yeah [20:08:15] ottomata: Many thanks :) [20:08:43] oh, its just the mode? [20:08:43] hmm [20:08:58] ohh yeah [20:08:58] hm [20:09:10] so the group ownership is correct [20:09:11] hm [20:09:36] i thought hadoop did this correctly... [20:10:04] !log hdfs dfs -chmod 775 /wmf/data/archive/mediacounts/daily/2018 for T185419 [20:10:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:10:06] T185419: Mediacounts missing top1000 files after 2018-01-01 - https://phabricator.wikimedia.org/T185419 [20:12:15] huh joal! [20:12:20] group ownership is preserved [20:12:24] when making a directory inside a dir [20:12:28] of the parent dir [20:12:32] but file mode is not. [20:14:39] 10Analytics-Cluster, 10Analytics-Kanban, 10Datasets-Archiving, 10Datasets-Webstatscollector: Mediacounts missing top1000 files after 2018-01-01 - https://phabricator.wikimedia.org/T185419#3915720 (10Ottomata) Huh! Ok, so when the new 2018 directory was created by the Hadoop jobs that compute the daily med... [20:22:20] ottomata: mwarf - not cool :( [20:22:33] ottomata: Thanks for having (presumably) fixed :) [20:22:40] ottomata: Will followup on tcket [20:23:02] ok - enough for today [20:23:10] Bye a-team - Have safe trips [20:25:40] laters! [20:30:39] (03PS1) 10Ottomata: Chmod yearly mediacounts directory so ezachte's scripts can write top1000 files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405938 (https://phabricator.wikimedia.org/T185419) [20:31:26] (03CR) 10Ottomata: "(untested)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405938 (https://phabricator.wikimedia.org/T185419) (owner: 10Ottomata) [21:18:58] 10Analytics-Kanban, 10Patch-For-Review: Launch top per country pageviews on UI - https://phabricator.wikimedia.org/T185510#3917834 (10Nuria) Units are a bit strange, see screen shot. [21:19:07] 10Analytics-Kanban, 10Patch-For-Review: Launch top per country pageviews on UI - https://phabricator.wikimedia.org/T185510#3920934 (10Nuria) {F12754154} [21:20:28] 10Analytics-Kanban, 10Patch-For-Review: Launch top per country pageviews on UI - https://phabricator.wikimedia.org/T185510#3920935 (10Nuria) 1g reads like 1"gram" , so we probably need to work on ux of that prior to merge. Also 1some -10 some should probably be written like 1-10 some [21:26:09] (03CR) 10Nuria: "Almost there. Please see comments on ticket about units: https://phabricator.wikimedia.org/T185510" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/405708 (https://phabricator.wikimedia.org/T185510) (owner: 10Fdans) [21:34:33] (03CR) 10Nuria: Add ISP data to webrequest table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/405899 (https://phabricator.wikimedia.org/T167907) (owner: 10Joal) [21:38:23] (03CR) 10Nuria: "Let's make sure to also test udf in case we might have missed an NPE." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/403916 (https://phabricator.wikimedia.org/T167907) (owner: 10Joal)