[00:37:22] Hi! anyone have any details on the values of "hostname" in Webrequest in Hive? https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest [00:38:20] As far as I understand, requests can bounce about among hosts, sometimes between data centers, depending on cache contents and other factors.... https://wikitech.wikimedia.org/wiki/Varnish#X-Cache [00:38:52] Maybe hostename is the first in the chain? Or the last? [00:59:55] milimetric: nuria: ^ thx in advance, apologies for the bother... ;) [01:35:42] AndyRussG: the hostname is the host that was originally requested by the user agent [01:36:08] bouncing inside our clusters doesn't change the hostname, it just might delegate the response to different machines [01:36:36] oh wait... I might be talking about uri_host [01:36:40] lemme get some samples [01:37:58] ok, AndyRussG, so it's the machine that served the response to the requesting user agent [01:38:14] why do you need to know this? [01:39:51] milimetric: trying to find what casued T152122 [01:39:51] T152122: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122 [01:40:50] Still a mystery... [01:41:31] AndyRussG: I see, ok, did you graph the traffic broken down by hostname around that time and see anything weird? [01:41:48] I mean if some of the hosts go down and others don't, it could be machine trouble I suppose [01:42:14] but generally the traffic would just be bounced elsewhere and we'd have heard about it, I don't remember anything like that from Luca who keeps in touch with the traffic team [01:43:46] milimetric: yeah so we haven't identified any definitive factor, but maybe one of the ones we've seen (such as language) in combination w/ responding DC might be it [09:26:37] 06Analytics-Kanban, 06Operations, 10Traffic, 13Patch-For-Review: Ganglia varnishkafka python module crashing repeatedly - https://phabricator.wikimedia.org/T152093#2849995 (10elukey) 05Open>03Resolved [10:34:45] (03PS4) 10Addshore: Add ExactValueQuantityProcessor [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/324906 [10:35:04] (03PS4) 10Addshore: Add ExactValueQuantityProcessor [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/325308 [11:03:52] * elukey battles with clickhouse [11:05:14] Arf elukey [11:05:19] elukey: need halp? [11:06:46] ahahah no no [11:06:56] I am trying to build the pkg [11:07:16] Ah, ok, indeed I'd be more of a burden than a help :) [11:08:31] I am really a newbie in debian packaging :) [11:08:45] one interesting thing is that it requires gcc 5 or 6 [11:08:54] Ah, this makes a lot more experience than I have ! [11:08:55] so I'll need to build against sdi [11:08:58] *sid [11:15:49] (03CR) 10Addshore: [C: 032 V: 032] Add ExactValueQuantityProcessor [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/324906 (owner: 10Addshore) [11:15:52] (03CR) 10Addshore: [C: 032 V: 032] Add ExactValueQuantityProcessor [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/325308 (owner: 10Addshore) [11:37:37] 10Analytics-Tech-community-metrics: When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#2850157 (10Aklapper) Going through today's diff, just one example of five detached uidentities I'll have to manually merge, as the... [11:41:10] all right Cmake configure deps solved [11:41:15] now it is building [11:43:09] the idea is to build with SID (so gcc-6, boosts, etc..) and then to specify GLIB_COMPATIBILITY at build time [11:43:50] ldd on druid said that clickhouse-server depends on libc, libm and few other very common libs [11:44:02] so I guess that all the other ones are statically compiled [11:45:42] now another cmake horrible error [11:45:46] :/ [12:07:18] 06Analytics-Kanban: Create SLA alarms for pageview_hourly jobs - https://phabricator.wikimedia.org/T152109#2850198 (10JAllemandou) a:03JAllemandou [12:15:32] (03PS1) 10Joal: Add SLA alerts for pageview jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/325540 (https://phabricator.wikimedia.org/T152109) [12:20:10] (03CR) 10Elukey: [C: 031] Add SLA alerts for pageview jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/325540 (https://phabricator.wikimedia.org/T152109) (owner: 10Joal) [12:20:24] Thanks elukey :) [12:22:43] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2850224 (10Physikerwelt) [12:26:49] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2850230 (10Physikerwelt) I added the Analytics project. Maybe some else is using analytics vagrant role and has experienced similar problems. [12:59:33] 10Analytics, 07Documentation: Document a proposal for bundling other than load-refine jobs together (see refine/diagram) - https://phabricator.wikimedia.org/T130734#2850302 (10JAllemandou) Seems that the only group that could easily be done is grouping pageview and projectview generation together. The rest of... [13:04:06] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2850306 (10Physikerwelt) After destroying and recreating the instance, at least the hdfs was created. ``` vagrant@mediawiki-vagrant:/mnt/hdfs$ hadoop dfs -ls / DEPRECATED: Use of this s... [13:05:06] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2850308 (10Physikerwelt) p:05Triage>03Normal [13:06:09] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2830159 (10Physikerwelt) @mschwarzer as long as we do not reboot the instance you can now continue with the things you inteded to do. I suggest that we ignore the fuse error ``` ==> def... [13:08:02] 10Analytics, 10MediaWiki-Vagrant: Cannot enable 'analytics' role on Labs instance - https://phabricator.wikimedia.org/T151861#2850315 (10Physikerwelt) just to be sure I uncommented the following line in /etc/fstab ``` hadoop-fuse-dfs#dfs://mediawiki-vagrant.dev:8020 /mnt/hdfs fuse allow_other,u... [13:51:51] dbms/include/DB/Dictionaries/MongoDBBlockInputStream.h:13:35: fatal error: mongo/client/dbclient.h: No such file or directory [13:51:59] and I put DISABLE_MONGO=1 [14:14:39] c++: internal compiler error: Killed (program cc1plus) [14:14:40] ahhahhaa [14:14:44] today is a great day [14:14:51] :S [14:16:12] ah snap this one means that too much ram was used /o\ [14:17:16] but I figured out the mongo problem [14:59:39] finally I managed to build \o/ [15:01:00] no too soon :( [15:07:36] elukey rocks [15:07:48] elukey FTW ! [15:16:57] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2850655 (10elukey) Opened https://github.com/yandex/ClickHouse/issues/228 to upstream to ask for some clarification of the debian files. [15:17:07] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2850658 (10elukey) p:05Triage>03Normal [15:17:31] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching): RecentChanges in Kafka - https://phabricator.wikimedia.org/T152030#2850659 (10Ottomata) [15:17:45] sorry guys I said victory too soon, now I get to 100% but ld gives me some issue [15:17:58] it may be related to the choice of the architecture, but I need to rebuild [15:25:49] hehe, you rock anyway :] [15:26:03] * elukey hugs mforns [15:26:30] hehehe [15:56:39] joal: sorry was out this morning [15:57:19] are you going to name the new druid data cube mediawiki_history? [16:00:18] ottomata, joal: staddduppp [16:01:54] milimetric: I want to, but I found a tricky thing with my last successfull run, so I'm willing to correct before rerunning and uploading :( [16:02:26] AHHHH [16:07:05] 10Analytics, 06Operations, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2850856 (10Gilles) [16:32:56] elukey: retrooo? [16:44:20] (03PS2) 10MarkTraceur: Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) [16:44:34] (03PS3) 10MarkTraceur: Add illustration queries for enwiki [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/267722 (https://phabricator.wikimedia.org/T111793) [17:30:34] lunchin bbl [17:50:15] (03CR) 10Nuria: [C: 032 V: 032] Add SLA alerts for pageview jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/325540 (https://phabricator.wikimedia.org/T152109) (owner: 10Joal) [18:05:18] mforns: https://gerrit.wikimedia.org/r/#/c/324946/ shouldn't have been necessary, I don't think I have time to follow up on this now, but that seems like a bug [18:06:33] milimetric, looking [18:06:45] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/zero - https://phabricator.wikimedia.org/T92343#2851206 (10Nuria) 05Open>03Resolved [18:07:04] 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2851209 (10Nuria) 05Open>03Resolved [18:07:22] 06Analytics-Kanban, 10ChangeProp, 10EventBus, 10Wikimedia-Stream, and 2 others: Write node-rdkafka event.stats callback that reports stats to statsd - https://phabricator.wikimedia.org/T145099#2851211 (10Nuria) 05Open>03Resolved [18:07:25] milimetric, mmm, makes sense to me that the column names need to be different, or maybe was I forgetting some parameter in the wiki config?? [18:07:38] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2851212 (10Nuria) [18:07:41] 06Analytics-Kanban, 06Operations, 06Zero, 07Mobile, and 2 others: Purge > 90 days stat1002:/a/squid/archive/mobile - https://phabricator.wikimedia.org/T92341#2851213 (10Nuria) 05Open>03Resolved [18:07:57] 06Analytics-Kanban, 06Operations, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#2851215 (10Nuria) 05Open>03Resolved [18:08:28] milimetric: that cleanup makes me so HAPPY [18:08:40] the logs? [18:08:47] milimetric: yes [18:08:54] cool, me too [18:09:03] milimetric: one less thing to be worried about [18:09:10] I'm having some second thoughts about cleaning up the regular jobs [18:09:15] the glam_nara one, sure [18:09:24] and maybe the api one too [18:09:34] milimetric: and sampled logs from webrequest right? [18:09:43] but the sampled ones might be interesting [18:09:44] milimetric: sorry , "main" requests [18:11:04] milimetric: but what info (other than past data) can you get from there that you cannot get from webrequest logs? [18:13:45] i guess that's true, especially since you can tablesample [18:13:58] and the parquet storage is pretty good [18:14:11] yeah, these made more sense before the refined data was stable [18:15:23] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#1106670 (10Dzahn) @Milimetric is this resolved? just noticed the parent task was closed today. [18:22:58] 06Analytics-Kanban, 06Operations, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#1106670 (10Milimetric) 05Open>03Resolved (just forgot to move this over to the done, the data was deleted and this task is resolved.) [18:28:26] mforns, milimetric : where is that spreadsheet for sql compration of old way and new data lake fashion? [18:28:54] nuria, looking [18:30:20] nuria, https://docs.google.com/spreadsheets/d/12nHxfp5cerKwAc1Q7W_DudSJ-ZhmynDK6VINzb857zE/edit#gid=1232097690 [18:31:12] mforns / ashgrigas / nuria: we doing the data types meeting? [18:31:18] yes [18:31:23] milimetric, yes! batcave? [18:31:25] ashgrigas: yes, was loking for link [18:31:25] joining now [18:31:31] ashgrigas: ok, batcave [18:31:43] sorry just assumed its on the irc channel [18:31:46] the batcave link [18:31:52] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave [18:32:06] ashgrigas: ^ [18:32:37] ashgrigas: we are in batcave [18:37:27] wikimedia/mediawiki-extensions-EventLogging#624 (wmf/1.29.0-wmf.5 - 5658eb3 : Translation updater bot): The build has errored. [18:37:27] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.5 [18:37:27] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/181740644 [18:47:04] 10Analytics, 06Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2851396 (10HJiang-WMF) Synced with @mforns and finished one set of deployment staging, and some metrics(daily edits, and daily edits by registered users) are deployed to htt... [19:05:57] Hi AndyRussG, quick hello for the same (small) problem as the other day: it seems there are 2 pyspark launched when you use only one :) [19:28:48] joal: Hmmm I don't know why that is [19:30:40] joal: I just shut down the one I've been using, and both entries with my username disappeared from https://yarn.wikimedia.org/cluster/scheduler [19:30:58] Weird AndyRussG -- Are you using a notebook, or CLI? [19:31:03] Notebook [19:31:14] I guess it's because of the options I'm using to start: pyspark --master yarn --deploy-mode client --num-executors 2 --executor-memory 2g --executor-cores 2 [19:31:24] Just copied that from the doc [19:31:25] hm, that might be the thing: I notice that behavior before when ellery was using notebooks [19:31:44] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Spark#Spark_and_Ipython [19:31:58] This command looks ok to me [19:32:03] Hmmm [19:32:12] "--executor-cores 2" ? [19:32:17] But, maybe the notebook launches it's own thing behind the scene? [19:32:33] AndyRussG: Means every executor has 2 slots for tasks [19:32:36] Maybe..... really not very familiar with how this works under the hood [19:32:39] :) [19:32:48] AndyRussG: You can trust me on that one ;) [19:32:52] K [19:33:26] In any case, if there's a different way of running it, pls lmk... Since they both died when I did ctrl-c, they must have been linked somehow... [19:33:42] I'm also running commands on beeline here and there [19:37:41] AndyRussG: Given the resource you ask for and the duration of your spark shells, that's no big deal, but that's something ottomata or elukey should be aware of :) [19:37:54] AndyRussG: I'm creating a task to not forget [19:39:53] joal: K thx!! :) [19:40:00] 10Analytics, 10Analytics-Cluster: Prevent notebooks on spark to launch 2 pyspark instances instead of 1 - https://phabricator.wikimedia.org/T152522#2851673 (10JAllemandou) [19:40:12] AndyRussG: --^ [19:40:13] :) [19:40:18] Thank you AndyRussG ! [19:42:26] ;D [19:42:57] :) [20:10:43] joal: add me as reviewer to scala change [20:21:42] ottomata: are the 5xx legacy tsvs used by ops? https://github.com/wikimedia/analytics-refinery/tree/master/oozie/webrequest/legacy_tsvs [20:21:49] 5xx-misc, text, upload, etc. [20:22:00] or should I ask people if it's ok to delete those too? [20:42:44] ottomata: scala as in mediawiki history I guess? [20:46:02] milimetric, mforns: any of you still here? [20:46:08] joal, yea [20:46:52] mforns: The bug I found is in populateByteDiff process, and I'm surprised it has not impacted more [20:47:00] aha [20:47:12] what happened, do you want to cave [20:47:13] ? [20:47:19] i'm here too [20:47:34] but we can talk tomorrow, no need to keep jo up [20:47:38] mforns: For some pages on certain wikis (commonswiki is one of them), there are 2 revisions with the same revId [20:47:42] yea sure [20:47:56] mforns, milimetric: if you have time, I can debrief for a minute [20:47:59] we haven't vetted the byte diffs yet, so maybe that's why we didn't see it [20:48:01] sure, omw [20:48:16] https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave-2 [20:49:40] joal: ya [20:49:45] offered to help review [20:49:53] if you like [20:49:53] ottomata: Yay, thanks mate :) [20:53:25] 10Analytics: de-duplicate archive records matching revision records in mediawiki_history - https://phabricator.wikimedia.org/T152546#2852209 (10Milimetric) [20:58:55] milimetric: do we know Zhou Zhou? is he someone with access to stat1002/3? [20:58:56] heh, I thought I was going crazy 'cause my logs that I deleted re-appeared [20:59:12] but yeah, i'm not crazy /usr/bin/rsync -rt --perms --chmod=g-w stat1002.eqiad.wmnet::hdfs-archive/webrequest/* /a/log/webrequest/archive/ [20:59:18] ottomata: yes, legal team [20:59:25] he can hive / etc. [20:59:38] I'll tell him where to put his data on stat box so it ends up in the folder he wants [21:01:21] ok cool [21:05:09] ottomata: I think best for you to start reviewing would be to have a walk through the code with us if you wish [21:07:48] yeah that would be great joal [21:07:50] this is a biggy [21:08:01] but [21:08:02] only if that is helpful [21:08:08] if it is a burden don't worry about it, yall are great :) [21:09:35] ottomata: more readers, better code :) [21:09:49] ottomata: I bet you gonna love scala ;) [21:10:01] hahaha [21:10:02] MAYBE [21:10:11] i'm gonna be like: SO AWSEOME! BUT ALSO so confusing! [21:10:11] :) [21:10:33] probable :) [21:11:42] milimetric: i added MaxSem to https://gerrit.wikimedia.org/r/#/c/323178/ (and also yurik) in the hopes that they can CR the patch and we can merge it [21:11:46] haha 6000+ lines, holy crapcrackers! :o [21:12:22] 6000 lines of scala should be enough to describe the known universe [21:12:49] I sometimes regret I didn't push hyper-normalization :) [21:13:17] 6K lines of scala must be like 6 million lines of java [21:13:48] joal my tomorrow morning walkthrough? [21:15:06] ottomata: bring some food.... [21:16:17] ottomata: Sure, tomorrow morning :) [21:16:30] I'll have coffee as well :) [21:17:21] hehehe [21:18:22] i'll try to get good sleep [21:18:27] 10Analytics, 10Analytics-Wikistats: Design new UI for Wikistats 2.0 - https://phabricator.wikimedia.org/T140000#2852324 (10Milimetric) [21:18:32] maybe i'll wear some sweatbands [21:19:12] * milimetric is really looking forward to the most epic code review ever [21:19:57] ha, it looks like the bulk of those lines are tests though [21:20:02] so, that makes it slightly less daunting [21:20:51] is this why joal's beard is so long? [21:21:34] .map(e => (e._2._1, e._2._2._2)) // RDD[Edge(id-fromKey, id-toKey, 0L)] - edge value not used (0L) [21:21:36] * joal is discovered ! [21:21:45] all i can say, has got some comments [21:22:03] ottomata: not everywhere, and not enough [21:22:05] at it has* [21:22:10] haha [21:22:17] probably most of my review will be "let's add some comments here" [21:23:38] (03CR) 10Ottomata: Add mediawiki history spark jobs to refinery-job (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) (owner: 10Joal) [21:23:44] there. review done. :p [21:24:38] ottomata: Thanks mate :) [21:24:46] I knew you'd be the man [21:36:13] mforns: subtle thing: to help with working on the spreadsheet I updated the titles of the reports on https://www.mediawiki.org/wiki/Analytics/Wikistats/DumpReports/Future_per_report with the current vote count [21:36:28] so you can look at just the TOC and jump to the next most popular one [21:37:08] milimetric, thank you! I surely will loom at that or will take too much time [21:37:22] milimetric, I'm logging off for today now though [21:37:33] 'course - nite [21:37:48] good night, thanks! [21:50:09] (03PS3) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) [22:08:57] 10Analytics, 10EventBus, 06Services (watching): Allow multiple schemas in a single EventBus topic. - https://phabricator.wikimedia.org/T152557#2852495 (10Ottomata)