[04:27:31] 10Analytics, 10EventBus, 10Wikimedia-Stream: Hits from private AbuseFilters aren't in the stream - https://phabricator.wikimedia.org/T175438#3645302 (10Nirmos) When someone trips a public filter, I get a JSON blob like this: ``` { "bot":false, "comment":"", "id":null, "log_action":"hit",... [08:36:22] Mornin' a-team :) [08:36:44] hello :) [08:36:58] o/ [08:54:49] elukey: Would you mind braindumping with me a minute on that notebook networking stuff? [08:56:35] joal: any issue if we do it in 10m? [08:56:42] Absolutely not :) [08:56:46] Ping me when ready :) [08:56:51] elukey: --^ [08:56:52] ok :) [09:19:23] joal: here I am :) [09:19:24] 10Analytics, 10PageViewInfo, 10Pageviews-API: Query PageViews pages selector using English equivalent of a special page is not working - https://phabricator.wikimedia.org/T177049#3645632 (10Trizek-WMF) [09:20:08] Hi elukey [09:20:09] Batcave? [09:20:17] sure [09:41:03] 10Analytics-EventLogging, 10Analytics-Kanban, 10Page-Previews, 10Readers-Web-Backlog, and 5 others: EventLogging subscriber module in ready state but not sending tracked events - https://phabricator.wikimedia.org/T175918#3645697 (10phuedx) 05Open>03Resolved 🎉🎉🎉 Since the Page Previews A/B test has bee... [09:42:17] ^ that phuedx person is all up in eventlogging again ;D [11:28:45] elukey: have a minute before lunch? [11:30:23] joal: on irc it would be better since there is an issue with the job queeus [11:30:27] :( [11:30:32] mwarf :() [11:30:46] if it's not the right moment, it'll wait - let me know [11:30:52] elukey: --^ [11:32:32] thanks :) [11:32:44] (03PS1) 10GoranSMilovanovic: Change dates in Search and Sqoop [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381425 [11:41:04] (03CR) 10GoranSMilovanovic: [C: 032] Change dates in Search and Sqoop [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381425 (owner: 10GoranSMilovanovic) [11:41:10] (03Merged) 10jenkins-bot: Change dates in Search and Sqoop [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381425 (owner: 10GoranSMilovanovic) [11:50:42] (03CR) 10Addshore: "So, in future you can add" [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381425 (owner: 10GoranSMilovanovic) [11:52:35] joal: I am almost free now :) [11:53:16] ok elukey - I have weird behaviors of python package :( [11:53:33] on the notebooks? [11:53:51] Yes - it's actually more of a python question ... [11:54:54] Queryin [11:54:59] again sorry [11:55:06] 10Analytics-Kanban: Write document on wikitech on why do we want to migrate back to gerrit from differential - https://phabricator.wikimedia.org/T176145#3645976 (10fdans) @Nuria @mforns @Milimetric let me know what you think about this: https://wikitech.wikimedia.org/wiki/Analytics/Differential [11:55:17] Querying using the pydruid lib doesn't work - I get a 503 [11:55:41] But when I copy paste the querying code in my notebook, it works just fine :( [11:56:14] * joal no understand [11:57:32] a 503? [11:57:39] Yes [11:57:43] it smells like squid proxy [11:57:49] hm [11:57:54] hmmmmmmm hm [11:58:05] will check more in depth [12:03:15] yeah it looks really weird that druid returns a 503, meanwhile we were getting something like that before while testing [12:03:36] joal: we can check with tcpdump very quicky [12:03:39] *quickly [12:03:47] elukey: You were right [12:04:32] elukey: I had to force my way through a weird thing, but it worked - I think I'm gonna do a PR on the pydruid package [12:04:41] \o/ [12:04:53] Thanks again mate :) [12:05:02] no problem! [12:05:05] your nose on network errors helped a lot :) [12:05:20] ok, let's take a break now :0 [12:05:25] later [12:05:29] laterzzz [12:05:30] * elukey lunch! [12:12:46] helooo [12:14:38] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats unique devices metrics needs some copy that says "monthly" - https://phabricator.wikimedia.org/T176240#3646021 (10fdans) [12:33:53] (03PS1) 10GoranSMilovanovic: change timestamp in job report filenames [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381432 [12:34:19] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] change timestamp in job report filenames [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381432 (owner: 10GoranSMilovanovic) [12:34:25] (03Merged) 10jenkins-bot: change timestamp in job report filenames [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/381432 (owner: 10GoranSMilovanovic) [13:17:43] elukey: o/ [13:17:46] good morniiing [13:20:50] ottomata: morningzzzz [13:20:58] how is our strata guy?? [13:21:03] all good at the presentation? [13:21:46] ya it was good! [13:22:08] elukey: apologies if I gave you a chill factor for those commits! I probably just commented really hastily while I was at the conference! i was not upset at all :) [13:22:55] you should make commits and merge them, just because I comment later doesn't mean you shouldn't do it, i'm fine if we debate them and we can always make changes after the fact [13:23:22] ottomata: sure sure but I mistakenly thought that I pissed you off for some reason and I was super sad :( [13:23:45] yeahhhh i'm sorry i was not pissed off at all! [13:28:14] super, sorry for the misunderstanding :) [13:29:03] milimetric, fdans, heyyy! any of you has 10 mins to help me with wikistats breakdowns? [13:29:30] mforns: I can jump in! my environment might be a bit noisy though :) [13:29:39] batcave? [13:29:56] mforns: I'll be right there but my computer froze [13:30:10] sure! omw [13:36:12] ottomata: https://grafana-admin.wikimedia.org/dashboard/db/prometheus-kafka - first line comes out of prometheus metrics :) [13:37:02] OOOOOOOO fancy! [13:37:04] v nice! [13:37:48] just fixed it, the last two were broken [13:37:53] buuuuut we have metrics! [13:39:03] ottomata: if you want to play with metrics, you can use the prometheus web ui via ssh -NL 9900:localhost:9900 prometheus1004.eqiad.wmnet (or similar) [13:39:18] it is handy to find metrics, test them, etc.. [13:40:13] vv cool [13:42:24] elukey: q [13:42:27] cluster=kafka_jumbo [13:42:36] what would that be if there was a kafka jumbo cluster in codfw? [13:45:47] joal: yt? [13:49:37] ottomata: it would be in another prometheus datasource, ops/codfw [13:50:42] another datasource? [13:55:02] yes top left corner of the dashboard, there is a "datasource" template variable [13:55:20] it selects what prometheus datasource you get the data from [13:55:29] we have one for each DC/type [13:55:45] so ops/codfw, ops/eqiad, ops/esams, etc.. and then services/esams, etc.. [13:56:28] oh in grafana [13:56:36] but waht about in prometheus itself? [13:56:40] oh it'd be a different instance ? [13:56:54] yes exactly [13:57:18] the metrics are polled by different instances in differenc dcs, that becomes the datasources in grafana [13:59:08] joal / halfak: not sure if you noticed I moved live systems meeting, it's not now it's in 1.5 hours [13:59:26] it was better for shilad then, but we can have it now if later doesn't work for you [14:00:30] just remembered [14:00:42] hmm, ok interesting [14:05:15] filling up the blank graphs but I think we have everything [14:05:17] \o/ [14:06:46] nice! vm_classes_loaded_total{cluster="kafka_jumbo",instance="kafka-jumbo1001:7800",job="jmx_kafka"} -> ~4k [14:07:04] err *jvm_classes_loaded [14:07:53] errrr elukey what about camel case vs snake case?! :D [14:18:19] I thought we decided to go with the all lowercase, cassandra does already the same and it seemed that having a single standard was good [14:18:34] maybe I talked only with Filippo about this :D [14:18:40] ayyyeye didn't talk to me [14:18:41] i hate it! [14:18:42] hahah [14:18:46] haahhaha sorrryyyyy [14:18:51] i think cassandra only does it because they didn't care [14:18:59] which is not a good reason! :[p [14:19:00] :p [14:19:16] I personally hate camel case :P [14:19:22] in general, i agree [14:19:29] if we were making a style guide for programming [14:19:38] but, these metrics come directly out of java, and are named camel case already [14:19:50] and, we don't have the ability to convert them to snake case directly [14:20:04] e.g. it doesn't insert underscores between lowerCapital places [14:20:08] so it isn't really snake case at all [14:20:12] it is just lowercase [14:20:19] which loses any meaning of word separators that were there to begin with [14:20:33] mmmm not really from what I can see, it is clear for me [14:21:01] so, i'd really prefer to keep them 1. the same as they were originally (since we can't do a proper conversion anyway), and 2. with the word separators [14:21:26] this could get really bad though, java can use long symbols in camel case [14:21:55] tcpinterfacebytebufferoverflow [14:22:04] (just made that up :p) [14:22:32] elukey: if we could actually convert these to snake case, I would agree with you, but this isn't snake case [14:22:48] it is just lower case, with . replaced by _ for metric separators [14:23:09] yeah, snake is only for attributes sadly, not sure why [14:25:31] plus I know you a bit and I am pretty sure you are not going to change idea hahah [14:25:34] elukey: for attributes? [14:25:40] haha [14:26:03] hahaa, YEahhHhHHH, i mean, i think I have to convince filippo too, right? [14:26:05] yes there is a tunable for the prometheus_jmx_exporter [14:26:23] this case i feel pretty strongly about, but if i can't convince either filippo OR you, i'll have to just deal with it [14:26:36] elukey: not sure i know what you mean just attributes? [14:26:44] where are the attributes being converted to snake case properly? [14:27:40] mbeans attribute I mean (IIUC the values of all the keys) [14:28:16] what I meant was that the keys of a mbean are not snake cased, the only option available is for attributes but we don't really care [14:28:28] at least, I didn't find a good way to make them work [14:29:01] oh, i see, so the conversion we are doing isn't even called to snake case? its just to lower case on the mbean names? [14:29:12] yes exactly [14:29:23] upstream could probably add a feature for this [14:29:25] I guess [14:29:42] attrNameSnakeCase [14:29:46] Converts the attribute name to snake case. This is seen in the names matched by the pattern and the default format. For example, anAttrName to an_attr_name. Defaults to false. [14:30:27] ahhh [14:30:57] yeah, if we could make the mbean names snake cased, i'd be ok with that. i'd slightly prefer still to keep them the same as their original jmx names (for consistency), but i wouldn't be so strongly opposed to converting to snake case :) [14:31:22] ottomata: in any case, would you agree that the kafka goal is done? [14:31:30] metrics naming plus alarming can be done later on [14:31:32] yeah i thikn so [14:31:35] super [14:31:39] nice job! :) [14:31:52] you too, we both worked on it a ton :) [14:31:55] i can probably work on alerts a little today, now that there are metrics [14:32:24] Filippo told me that we could create a task to decide the convention to use [14:32:26] elukey: how do you think we can resolve the lower case issue, should I comment on a phab task and ask for feedback from filippo? [14:32:29] ok great [14:32:31] i will do so! [14:32:36] nice [14:32:38] this is specifically for jmx metrics, right? [14:32:41] yes yes [14:32:42] the convention? [14:32:42] ok [14:50:30] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Decide on casing convention for JMX metrics in Prometheus - https://phabricator.wikimedia.org/T177078#3646534 (10Ottomata) [14:50:55] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Decide on casing convention for JMX metrics in Prometheus - https://phabricator.wikimedia.org/T177078#3646534 (10Ottomata) [14:56:13] Hey ottomata - Was gone, how may I help? [14:56:30] heya, was gonna ask you some qs about some hadoop block hashing stuff [14:56:36] hmmm, we can talk real quick before standup, wanna? [14:56:46] sure [14:56:50] k in bc [15:03:48] 10Analytics-Kanban: Rename datasources and fields in Druid to use hyphens instead of underscores - https://phabricator.wikimedia.org/T175162#3584677 (10Ottomata) Why hyphens? I think underscores would be better, so that we can be consistent with places where we can't use hyphens for dataset names (like Hive tab... [15:36:03] milimetric, joal: I am waiting to join the call. I suspect you all are coming from sup, but I wanted to make sure I wasn't missing the call because of some technical issue. [15:36:22] yes Shilad we're in standup still [15:36:23] Shilad: indeed ! [15:36:35] great. I will not worry then :) [15:41:28] Shilad: we're there [15:41:34] just got out of s-up [15:42:35] 10Analytics-Kanban: Rename datasources and fields in Druid to use hyphens instead of underscores - https://phabricator.wikimedia.org/T175162#3584677 (10Nuria) Because we do not want to break bookmarks for datasets that have them and hyphen-named datasets are the majority. Sounds OK? [15:46:47] 10Analytics-Kanban, 10Analytics-Wikistats: Implement Topic Selector Widget - https://phabricator.wikimedia.org/T167676#3646774 (10fdans) [15:47:10] milimetric: want to talk differential? [15:47:35] yes, in this meeting for another little bit [15:51:42] coooool [15:52:03] also, I share your frustration about those unintended diffs in your topic selector revision :) [15:52:26] right [15:52:27] :) [15:54:29] joal: did you figure out the deal with talking to druid from notebooks? [15:54:44] Yes Madhu !! [15:54:50] oh awesome! [15:55:07] It's due to webproxy being set by default [15:55:10] madhuvishy: --^ [15:55:15] aaah [15:55:52] madhuvishy: We are thinking of removing the by-defaut proxy so that it works by default internally, and you'ld have to set it up to pip install [15:55:57] madhuvishy: any thoughts? [15:57:38] fdans: wanna play with the topic selector until we hang out in a few minutes, that way I can get your thoughts on that too? [15:57:59] milimetric: I'm on that right now :) [15:58:04] sweet [15:58:43] joal: right, need to think about it. It's weird that it's trying to go through it for an internal host [16:00:37] a-team: fyi the ops team is now alarming on long running tmux/screen sessions, so I just whitelisted druid* and stat*. I don't think that we'll need more, but let me know otherwise. [16:01:00] sounds good elukey [16:01:04] elukey: Why druid? [16:01:51] you have tmux sessions in there for tranquillity etc.. :) [16:02:02] elukey: shouldn't, that's why I ask [16:02:12] elukey: tranquility is not running on druid anymore [16:02:34] joal 23088 0.0 0.0 27968 2484 ? Ss Jun20 0:18 SCREEN -S tranq-banners [16:02:44] joal 22568 0.0 0.0 28028 2460 ? Ss Jun20 0:21 SCREEN -S tranq-netflow [16:02:53] elukey: Let me check, I should be able to kill those [16:02:59] druid1003 [16:02:59] elukey: druid1001? [16:03:02] thanks [16:05:16] 10Analytics: Combine Hive Year / Month / Day / Hour partitions into ISO date string - https://phabricator.wikimedia.org/T177097#3646934 (10Milimetric) [16:05:21] no more screens elukey :D [16:05:28] ahhh nice [16:05:36] fdans: omw to the batcave [16:06:16] joal: I think the right thing is to have the squid proxy know that this service is internal [16:06:46] madhuvishy: that'd be the best for sure ! [16:06:52] 10Analytics, 10Analytics-Cluster: Combine Hive Year / Month / Day / Hour partitions into ISO date string - https://phabricator.wikimedia.org/T177097#3646957 (10Ottomata) [16:07:04] like, sutff in .eqiad.wmnet shouldn't be proxied [16:07:13] madhuvishy: --^ [16:07:22] madhuvishy: I however don't know how to do that :( [16:08:04] I think it's here - https://github.com/wikimedia/puppet/blob/production/modules/role/templates/caching-proxy/squid.conf.erb [16:08:40] the eqiad stuff is fine, this is already a 10. ip [16:08:47] and the ACL does allow it [16:09:00] http_access deny !Safe_ports [16:09:09] may be what's causing the problem [16:10:16] it does say `acl Safe_ports port 1025-65535 # unregistered ports` [16:10:19] hmm not sure [16:58:46] * elukey off! [17:08:04] * mforns off! [17:44:12] 10Analytics: PageContenSaveComplete. Stop collecting - https://phabricator.wikimedia.org/T177101#3647234 (10Nuria) [18:01:40] elukey: we can abandon this, right? https://gerrit.wikimedia.org/r/#/c/379290/1/modules/role/manifests/prometheus/services.pp [18:07:08] hey madhuvishy joal ottomata So yesterday I was able to get pydruid to query Druid from notebook1001, from the console, within the same virtualenv as the Jupyter notebook uses [18:07:26] https://tools.wmflabs.org/paste/view/b114d636 [18:07:37] However the same thing run from within the notebook didn't work [18:07:43] AndyRussG: yeah notebooks export HTTP_PROXY to make sure you can pip install [18:07:58] Ahhh hmmmm [18:08:03] but it seems like the proxy is for whatever reason preventing druid access [18:08:11] which shouldn't be the case for 10. things [18:08:15] but for now [18:08:17] Yeah I was about to say, I couldn't do a normal python-based http request either [18:08:30] if you just do unset http_proxy [18:08:39] 10Analytics: PageContenSaveComplete. Stop collecting - https://phabricator.wikimedia.org/T177101#3647294 (10Nuria) [18:08:52] 10Analytics: PageContenSaveComplete. Stop collecting - https://phabricator.wikimedia.org/T177101#3647234 (10Nuria) a:05DarTar>03Nuria [18:09:22] in the notebook !unset http_proxy [18:09:35] AndyRussG: that should help I think [18:10:33] if you need to export it again for pip installs you can do `export http_proxy=http://webproxy.eqiad.wmnet:8080 [18:10:42] `export http_proxy=http://webproxy.eqiad.wmnet:8080` [18:11:04] The squid proxy behavior should probably be fixed though [18:13:19] Nice! Trying now :) [18:17:35] madhuvishy: hmmm doesn't let me unset [18:17:50] definitely makes sense that this is the issue, tho [18:18:17] !unset http_proxy [18:18:19] !echo $http_proxy [18:18:21] !unset http_proxy [18:18:23] !echo $http_proxy [18:18:25] http://webproxy.eqiad.wmnet:8080 [18:19:12] Or [18:19:17] !export http_proxy='foo' [18:19:19] !echo $http_proxy [18:19:20] http://webproxy.eqiad.wmnet:8080 [18:19:53] (sorry repeated messages above) [18:20:12] maybe some config in pydruid can get around it..... [18:20:22] AndyRussG: I see [18:23:27] fun, eh? [18:24:02] From a different perspective, actually seems like a pretty solid behaviour on the part of Jupyter [18:25:34] right [18:25:53] yeah, the squid proxy should let you talk to internal services [18:26:06] so that's the behavior that needs to get fixed [18:27:51] I can do [18:27:54] https://www.irccloud.com/pastebin/bMvHye5h/ [18:36:02] madhuvishy: fantastic, thanks so much! [18:47:12] joal: AndyRussG if you wanna patch pydruid :) https://gist.github.com/madhuvishy/1d24b556fcf0ed619476200b2018977c [18:51:48] madhuvishy: wow!!!! fantasmic!! :) [18:52:27] I stole joal's druid stuff, I only added two lines to the _post function :) [18:52:27] I was trying something much hackier, manually adding some bits in ~/venv/lib/python3.4/site-packages/pydruid/client.py [18:52:52] your approach is way better [18:53:18] :) Inheritance is cool! [18:53:19] K, this totally works for now [18:53:51] awesome, I'll let you know if I figure out what's wrong with the webproxy [18:54:19] madhuvishy: thanks so much once again :D [18:54:29] yw :D [19:48:29] 10Analytics-EventLogging, 10Analytics-Kanban: PageContenSaveComplete. Stop collecting - https://phabricator.wikimedia.org/T177101#3647519 (10Nuria) [19:48:47] madhuvishy: i went a a pretty cool presentation about JupyterLab at strata [19:50:06] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Port Kafka alerts from check_graphite to check_prometheus - https://phabricator.wikimedia.org/T175923#3608073 (10Ottomata) a:03Ottomata [19:50:08] madhuvishy: https://github.com/jobar/pydruid/commit/611f6b31faded42fefe8f7dc2fce3488714ef1ab [19:51:29] AndyRussG: Heya, sorry was away for diner [19:52:06] AndyRussG: I pushed a pull-request for PyDruid - If you want you can add the change to your lib versio [19:53:50] AndyRussG: Open a terminal in jupyter, then cd venv/lib/python3.4/site-packages/pydruid, then replace client.py with https://raw.githubusercontent.com/jobar/pydruid/611f6b31faded42fefe8f7dc2fce3488714ef1ab/pydruid/client.py [19:54:49] AndyRussG: Restart the kernel of your pydruid notebook and update PyDruid instanciation: PyDruid('http://druid1001.eqiad.wmnet:8082', 'druid/v2', proxies = {}) [19:54:55] You're ready to go :) [19:56:23] Gone back to sleep :) [19:56:25] Later a-team [20:01:57] latesr~ [20:10:01] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3647570 (10Aklapper) [20:10:05] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3647571 (10Aklapper) [20:13:09] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Fix duplicated enrollments in database - https://phabricator.wikimedia.org/T176786#3647577 (10Aklapper) [[ https://phabricator.wikimedia.org/T157898#3639876 | owlbot overwrote my updates ]], so while this is 'ready to push' it has to wai... [20:13:20] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Fix duplicated enrollments in database - https://phabricator.wikimedia.org/T176786#3647581 (10Aklapper) [20:13:23] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3020071 (10Aklapper) [20:13:30] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Fix duplicated enrollments in database - https://phabricator.wikimedia.org/T176786#3636616 (10Aklapper) 05Open>03stalled [20:14:24] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Fix duplicated enrollments in database - https://phabricator.wikimedia.org/T176786#3636616 (10Aklapper) p:05Normal>03Low List is shorter than expected btw: ``` sortinghat withdraw 025124bdf0ea09267b3a6fddca2ae4552dd44a4d "Wikimedia F... [20:24:14] Anybody know about the dangers involved in using webrequest.page_id vs reconstructing the page_id from pageview_info? [20:40:22] 10Analytics, 10PageViewInfo, 10Pageviews-API: Query PageViews pages selector using English equivalent of a special page is not working - https://phabricator.wikimedia.org/T177049#3647616 (10Milimetric) 05Open>03declined Without the page id, it would be quite hard to do this. We're looking at allowing pe... [20:47:44] 10Analytics: Making geowiki data public - https://phabricator.wikimedia.org/T131280#3647645 (10Milimetric) It looks like this is now getting priority, so I'd like to get involved and set up a hopefully useful approach. Here's what I'm thinking: * Finish the subtask that Nuria just created, which is necessary f... [21:09:38] Shilad: FYI (when you change your code to desktop only) that some projects are 80% mobile like indonesian wikipedia, most others are about < 50% [21:09:45] Shilad: let me know if this makes sense [21:10:01] Shilad: pageview_info? [21:10:17] Shilad: I am not surte i understand [21:14:19] * sure