[09:21:54] Analytics-Tech-community-metrics, Engineering-Community, WMF-Product-Strategy: Review ECM analytics needs - https://phabricator.wikimedia.org/T92807#1120786 (Aklapper) [10:35:45] Analytics-Tech-community-metrics, Engineering-Community, WMF-Product-Strategy: Review Engineering Community analytics needs - https://phabricator.wikimedia.org/T92807#1120989 (Qgil) [10:59:33] Analytics-Cluster: Stop Cluster from $SOMETIMES not running oozie jobs - https://phabricator.wikimedia.org/T92819#1121036 (QChris) NEW [11:31:14] Analytics, Analytics-Kanban: Dashboard Directory Design Feedback - https://phabricator.wikimedia.org/T92502#1121131 (Pginer-WMF) [14:54:38] ottomata: yt? [14:57:51] yup [14:57:52] hiya [14:58:52] ottomata: so ... [14:59:00] ottomata: trying to do git-deploy [14:59:03] i get [14:59:06] https://www.irccloud.com/pastebin/kKWlift9 [14:59:36] ottomata: actually git pull (after git deploy-start) [14:59:49] ottomata: this is teh full sequence of commands: [14:59:53] https://www.irccloud.com/pastebin/ASUMnpL8 [15:02:35] i usually do git pull before git deploys tart [15:02:40] pretty sure git deploys start makes a tag [15:02:46] from current checkoiut [15:02:48] and that is what it deployed [15:02:59] oh but [15:03:01] hm [15:03:06] checking [15:03:12] cause that looks like a perm problem [15:03:49] hmm, perms for that look fine [15:03:55] maybe just try it the other way nuria? dunno [15:04:01] try that and let me know if it still does the same thing [15:05:05] ottomata: so I did > git deploy abort [15:05:09] >git pull [15:05:13] and same thing . [15:06:10] hm ok [15:06:53] ottomata: lack of permits in tin? [15:07:35] i'm looking, the permits seem fine [15:07:40] drwxrwsr-x 248 sartoris wikidev 4096 Dec 18 13:57 .git/objects [15:07:45] it happens for me too though [15:08:14] ottomata: seems that we should be in a group in which we are not right? [15:08:29] nuria@tin:/srv/deployment/eventlogging/EventLogging$ groups [15:08:29] wikidev deployment [15:08:45] ottomata: although ya, i seem to be in 'deployment' group [15:08:45] you are in wikidev, and that dir is group writable by wikidev, as it should be [15:09:18] nuria: going to try some stuff... [15:09:40] Some files did not have group writability. [15:09:51] I added them now. [15:09:56] That might have done the trick. [15:09:59] which ones? [15:10:03] nuria: Can you try again? [15:10:20] ottomata: Some files underneath the .git directory. [15:10:29] qchris: ah yes, working now. Is that something we should do everytime? [15:10:30] Let me see if scrollback goes that far back. [15:10:50] nuria: you shouldn't have to, no [15:10:55] ottomata: k [15:11:25] I also agree that one should not have too. [15:11:45] But some files in the .git directory are owned by qchris, nu-ria, or or-i. [15:12:10] And due to default umasks (I guess), they loose group writability. [15:12:24] Not sure how git-deploy is supposed to work, but [15:12:33] this gets in the way herep [15:12:37] s/herep/here/ [15:12:46] qchris: ok, will write that gotcha [15:13:16] I ran into the issue before with or-i once. He just adjusted permissions on all files again and things seemed to work again. ... until now it seems. [15:14:48] ottomata: One directory that was lacking group writability was /srv/deployment/eventlogging/EventLogging/.git/objects/a2 [15:15:04] I see that nuria's pull wanted to create a file in there: [15:15:14] qchris, ottomata: many thanks, will document gotchas here: https://wikitech.wikimedia.org/wiki/EventLogging#Deploying_EventLogging [15:15:19] /srv/deployment/eventlogging/EventLogging/.git/objects/a2/bf69e169ab30c4f839108c2d06b236c9c1c65b [15:16:27] hm, ok [15:18:03] me no comprendooo [15:18:20] qchris: git deploy sync is supposed to send code to vanadium/hafnium? [15:18:58] nuria: otto-mata asked about an example where the permissions have been in the way. And the above directory/file is such an example [15:19:21] qchris: ah ya, sorry, i was asking about once that is fixed [15:19:26] nuria: Yes, last time I deployed "git deploy sync" brought the code to vanadium/hafnium. [15:19:30] qchris: we do > git pull [15:19:36] we do> git deploy sync [15:20:10] but qchris , ottomata after i get: [15:20:17] https://www.irccloud.com/pastebin/lYwYFHZC [15:20:47] 0/3 is not good :-) [15:20:51] and no new code is on: nuria@vanadium:/srv/deployment/eventlogging/EventLogging$ [15:21:02] qchris: ya... that is what i thought... [15:21:06] 2/3 is the usual thing (as documented on wiki) [15:21:14] Mhmm. [15:21:14] qchris: right, right [15:21:44] I'd try (on tin) "git deploy abort" [15:21:48] then "git deploy start" [15:21:57] then "git checkout master" [15:22:01] then "git pull" [15:22:07] then "git deploy sync" [15:23:13] qchris: aham... so then git deploy start must come BEFORE git pull [15:23:23] qchris: thank you , will document that too [15:23:34] well ... no, it's not that it needs to. [15:23:49] It's just that the docs that are linked on wiki recommend it. [15:24:05] And it's nice, since (as otto-mata said) "git deploy start" creates a tag. [15:24:20] So you know where the repo was before you pulled. [15:24:36] This helps a bit if you have to do archeology on the repo. [15:25:08] But there is no strict requirement to do "git deploy start" before "git pull". (Or rather: I do not know of such a requirement) [15:25:36] qchris: seems that otherwise things do not work so pretty but fixed now, code is on vanadium [15:25:45] k [15:25:57] nuria: sometimes salt is slower than git deploy wants it to be [15:26:01] if it says 0/3 [15:26:04] tell it to retry [15:26:15] ottomata: ahaham [16:05:17] mforns, milimetric: can any of you CR this one: https://gerrit.wikimedia.org/r/#/c/197070/2 [16:05:24] * milimetric looks [16:11:53] milimetric: thank youuu [16:12:01] np [16:13:52] milimetric: argh, yes, mean to type 400 [16:15:05] milimetric: but "configurable" in EL means: it's a constant on python. I wish we had a configuration package but we do not since all code is deployed together [16:16:22] nuria: fair enough [16:16:48] milimetric: correctd now [16:16:52] *corrected [16:25:10] (PS1) Milimetric: Add VIM mode [analytics/quarry/web] - https://gerrit.wikimedia.org/r/197076 [16:39:33] milimetric: corrected & deployed to vanadium [16:39:48] milimetric: so vanadium is running latest code [17:01:34] ottomata: looked at kafka code and looked at github of kafka-python package, just have 1 question regarding re-starts [17:04:52] yes [17:05:14] ? [17:05:50] ottomata: what about re-starts? [17:05:55] ? [17:06:01] whatcha mena? [17:06:03] mean? [17:06:06] restarts of? [17:06:20] ottomata: since the kafka buffer has durability, how do we keep track of events consumed so we do not re-consume them? [17:06:34] ottomata: restarts of the kafka consumer [17:06:38] ah, that is a good question that I have not looked into much yet [17:06:46] the client should be able to store offsets in zookeeper [17:06:50] but i haven't tried to make that work yet [17:07:09] right now i thikn it just continues from the latest offset after a resart [17:07:27] same behavior as udp2log one now [17:07:57] ottomata: ok, let me know when you get there cause i think we would need to address offsets before deploying for real, right? [17:08:44] not sure, is that something we want to support right away? or maybe it is good to get that working, so that if we turn it on we know it already works that way, and we can avoid a behavior changing deployment later? [17:08:54] but, i'd worry about eventlogging if it had to consume tons of data for some reason [17:09:06] right now, it will never do that, because it only starts at the latest [17:09:22] if there were a big backlog, say, a day, would eventlogging get overloaded because it would consume as much as it could really fast? [17:09:41] now? [17:10:09] with the zeromq-> db consumer if there is a burst the limit is how much the db can swallow [17:10:09] no, now it can't do that. but say we allowed this kafka processor to consume from where it last left off [17:10:18] aham [17:10:25] if it hadn't run for a long time [17:10:29] it would start from where it left off [17:10:36] and consume events from kafka as fast as possible [17:10:47] which would shove a bunch of events into 0mq really fast [17:10:58] ottomata: which is less than ideal [17:11:15] right, which is why i'm not sure if we want to implement offset storage yet [17:11:24] i mean, we could, and disable it, but i would,n't want to turn it on for eventlogging yet, because of that reason [17:11:39] ottomata: but then what is the behaviour now? [17:11:48] now/ withoutkafka? [17:11:51] ottomata: wouldn't it re-consume what has alredy consumed? [17:11:53] there is no buffering for udp2log [17:11:56] oh with kafka [17:12:00] ? [17:12:07] ottomata: with kafka consumer as is [17:12:15] ah, no [17:12:19] we should double check this [17:12:21] but i'm pretty sure now [17:12:29] it just consumes from the latest offset when it starts [17:12:37] so, the most recent message and starts from there [17:12:43] ottomata: ok, let's triple check that [17:12:44] it doesn't try to consume the backlog [17:12:56] well, i mean, i know it isn't storing offsets [17:12:58] i checked that [17:13:03] but, i don't know if it isn't trying to :) [17:13:10] we should make sure it doesn't try to [17:14:25] ottomata: right, i saw that from the consumer instantiation (no offsets) which to me mean: read from the beginning of buffer not the most recent message [17:14:30] let me do some seraches [17:14:41] *research [17:18:36] ah, nuria [17:18:37] https://github.com/mumrah/kafka-python/blob/v0.9.3/kafka/consumer/kafka.py#L124 [17:19:18] and [17:19:22] auto_commit_enable = False [17:19:23] by default [17:19:35] i should specify those manually to be explicit in the eventlogging code [17:19:56] ottomata: ok, right, as we want to fall in this one, right? https://github.com/mumrah/kafka-python/blob/v0.9.3/kafka/consumer/kafka.py#L651 [17:20:14] yes, largest is latest [17:20:18] that is , make sure we always start at latest [17:20:31] ottomata: ok, so changing that should take care of restarts [17:20:38] ottomata: very goooooddd [18:10:15] Analytics, VisualEditor: Tons of Eventlogging Edit events For VE not validating - https://phabricator.wikimedia.org/T92869#1122239 (Nuria) [18:18:26] milimetric: plis take a look at : https://phabricator.wikimedia.org/T92869 as abort rates might be off since this is happening. [18:27:58] Analytics-Engineering, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122344 (Ottomata) NEW [18:28:45] Analytics-Engineering, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122351 (Ottomata) [18:32:05] Analytics-Engineering, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122371 (dr0ptp4kt) I prefer header enrichment be done at the MediaWiki API tier if possible. The parameter name-val... [18:32:56] Analytics-Engineering, MediaWiki-API, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122374 (Krenair) [18:34:46] Analytics-Engineering, MediaWiki-API, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122384 (yuvipanda) +1 for doing it at the mediawiki API level. I think that would necessitate a... [18:39:36] milimetric: Do you need review/+1 on https://gerrit.wikimedia.org/r/#/c/195895/ ? [18:51:20] ottomata: are you testing in EL beta labs? [18:51:39] i'm setting up the varnishkafka part now, ops meeting was just over, about to do that [18:53:34] gotta run out for just a few mins htough, back shortly [18:53:42] ottomata: ok, i was going to re-start as i saw a bunch of errors but i will not touch nothing [18:53:51] nuria, i haven't done anything with EL there at all [18:53:57] all i [18:54:03] all i've done so far is set up a kafka instance [18:54:07] haven't modified EL at all [18:54:12] ottomata|afk: ok, then let me re-start and deploy latest [18:54:14] so, feel free to fix whatever in betalabs [18:54:17] aye, cool, back in a bit [18:55:23] Analytics-EventLogging, Analytics-Kanban, Easy: Eventlogging should log timestamps for all error messages {oryx} - https://phabricator.wikimedia.org/T89162#1122499 (Nuria) [18:56:17] James_F: no, https://gerrit.wikimedia.org/r/#/c/195895/ is stalled on trying to figure out if the event_page.revid is always the revision.rev_parent_id, even in the case of action = saveSuccess [18:56:43] James_F: but then that's blocked by me finishing up the dashboard layout and side-by-side comparison with Wikitext [18:56:49] milimetric: event_page.revid for event.action = saveSuccess should be the newly-saved revision's ID. [18:57:11] James_F: yes, but some quick queries last week showed me that this wasn't so [18:57:12] So… revision.ID rather than revision.rev_parent_id. [18:57:13] I think. [18:57:15] Hmm. [18:57:24] It's certainly /sometimes/ true. [18:57:25] and I didn't have time to dig too much, since I have to finish the other work [18:57:31] yeah, it's definitely sometimes true [18:57:34] 'Cos my manual digging for diffs is based on it. [19:25:24] ottomata|afk: beta labs El restarted & deployed & all nice & ready for-ya [19:27:50] ya thanks! [19:30:24] Analytics-EventLogging, Analytics-Kanban, Wikipedia-App-iOS-App, Mobile App Sprint 52 - iOS: Some events not validating for MobileWikiAppSavedPages schema - https://phabricator.wikimedia.org/T91290#1122650 (Nuria) Talked to Adam and changes here are done, scheduled to be released on march 30th. [19:35:21] Analytics-Engineering, MediaWiki-API, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1122692 (Anomie) Hmm, no #MediaWiki-extensions-XAnalytics ? As far as I understand this, that's w... [19:45:58] nuria: agh. [19:46:14] i need to make another instance (or find a trusty one) on which to run zookeeper in the labs deployment project [19:46:23] i was going to run it on my kafka node instance [19:46:28] buuut, i have to run kafka in precise for now [19:46:35] and the zookeeper version is mismatched in precise [19:46:44] but, i can't seem to create another instance, i guess the project is full [19:46:47] hm. [19:49:44] ottomata: mmm [19:49:55] ottomata: project meaning beta labs, right? [19:51:05] s'ok, i asked in labs, andrewb is upping the project quota [19:51:11] sorry to ping :) [19:51:37] ottomata: good, it will be awesome to have kafka there permanently [19:51:53] welll, i would set this up more robustly if this was a really permanent setup, perhaps I will [19:51:56] right now it is only one node [19:52:06] can't do much kafka failover testing with that [20:37:43] milimetric: I need to get a dump of the existing WikiGrok data from the eventlogging table (log.MobileWebWikiGrokResponse_10352279), but I seem to have permission trouble whenever I try to dump it. Do you have time to help? [20:38:13] kaldari: sure, I can take a look [20:38:58] kaldari: ok, I can select from that table, whatcha need? [20:39:46] oh, kaldari, the password changed, that might be what's up with your permissions, do you use /etc/mysql/conf.d/research-client.cnf for your credentials? [20:40:30] milimetric: I can select too, but wasn’t able to either use mysqldump or pipe a select to a file for some reason. [20:40:44] weird, so what data do you need [20:41:08] milimetric: Does the following work for you: [20:41:13] mysql -e --defaults-file=/etc/mysql/conf.d/research-client.cnf -h analytics-store.eqiad.wmnet "select * from log.MobileWebWikiGrokResponse_10352279" > file_name.tsv [20:41:57] heh, no... hm, one sec [20:42:23] oh, that command's backwards [20:42:50] oops, my mysql foo is weak :) [20:42:54] mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf -h analytics-store.eqiad.wmnet -e "select * from log.MobileWebWikiGrokResponse_10352279" > file_name.tsv [20:43:00] try that [20:43:05] it's not you, the tool's weak :) [20:43:26] but it just needed the command to follow -e [20:43:38] milimetric: yay, that works. Sorry for the dumb question :P [20:44:05] np, everyone that has trouble with mysql makes me feel better for the dozens of times I've embarassed myself :) [20:47:10] nuria: did you see this one too? [20:47:10] https://gerrit.wikimedia.org/r/#/c/196637/2/server/bin/eventlogging-processor [20:47:50] also, i am ready to test this change in beta now, how would you suggest I deploy? [20:48:26] ottomata: what's the question, re socket identities? [20:48:49] milimetric, can you help me with vagrant? :] [20:48:59] i guess, previously it was passing identity directly to sub_socket [20:49:07] and now is setting it via the uri [20:49:16] that should be fine, yes? [20:49:16] mforns: I can try [20:49:30] milimetric, batcave? [20:49:33] sure [20:50:11] ottomata: where is uri_append_query_items set, btw? [20:51:16] ori, https://gerrit.wikimedia.org/r/#/c/196073/6/server/eventlogging/utils.py [20:52:16] this is a bit awful, though the awfulness is not introduced by your change, you're just surfacing existing awfulness for which I am the culprit [20:52:23] haha [20:52:27] i am adding to the awfulness [20:52:32] howso, by overloading the uris for this? [20:52:33] nuria has been nagging about proper configurability for a while and I think she's right [20:52:40] well, the URLs are already overloaded [20:52:50] I think we can live with this for now, though [20:53:36] even though it doesn't look so elegant, i am finding that encoding the config in the uri is pretty slick, especially the way you have it working with handlers so transparently [20:53:38] ottomata: lemme seee [20:53:46] so easy to add a new handler that takes different args [20:53:53] and it automatically works with a uri that has those as query params [20:54:23] i guess you could do that if there was some config thing too, just grab all the config keys of some kind and try to pass them [20:54:33] nuria: i can deploy from deployment-bastion, yes? [20:54:54] ottomata: no, it doesn't work but you can deploy on the box, see: [20:55:05] ottomata: https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs#How_to_deploy_code [20:55:22] orly, hum [20:55:27] ori: i second ottomata with urls, i think those are supper handy [20:56:04] ottomata: or maybe deployment bastion is something i do not know about, i think when i tried last it did not work [21:06:40] Analytics-Engineering, MediaWiki-API, Wikipedia-App-Android-App, Wikipedia-App-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1123130 (Deskana) Seems like it needs to be done in the extension, rather than in the app. [21:06:57] nuria: how do I restart a single service, like, evnetlogging-forwarder? [21:07:05] $ sudo service eventlogging/processor status [21:07:05] status: Unknown parameter: NAME [21:07:55] ottomata: like [21:07:57] start eventlogging/consumer NAME=graphite CONFIG=/etc/eventlogging.d/consumers/graphite [21:08:00] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1123141 (awight) @Jalexander: Was this the information you needed, and is it usable? Do you need the numbers for any other date ranges? [21:09:35] sudo service eventlogging/forwarder NAME=8421 status [21:09:35] eventlogging/forwarder: unrecognized service [21:09:45] oh CONFIG? [21:09:45] ok [21:11:06] hm no good either [21:11:07] sudo service eventlogging/forwarder NAME=8421 CONFIG=/etc/eventlogging.d/forwarders/8421 status [21:11:07] eventlogging/forwarder: unrecognized service [21:11:08] nuria: ^ [21:11:32] AH [21:11:32] got it [21:11:33] status first [21:11:37] ottomata: would do Kill -9 to kill svc [21:11:39] prams after [21:11:46] sudo service eventlogging/forwarder status NAME=8421 CONFIG=/etc/eventlogging.d/forwarders/8421 [21:11:47] is good [21:12:34] ottomata: ok, good, i think there are some variations that work [21:12:59] COoooL [21:12:59] 2015-03-16 21:12:55,361 (MainThread) Forwarding udp://0.0.0.0:8421?raw=True => tcp:8421... [21:13:20] ping if you need me, i'm a bit distracted but can answer qs in a pinch [21:21:42] Analytics-Kanban, Analytics-Visualization: Remove isZero field from data in Pentaho - https://phabricator.wikimedia.org/T91587#1123257 (kevinator) Open>declined a:kevinator WP Zero team just requested to keep this data. It is more important to them to have the the field so they can generate numb... [21:21:54] nuria: q [21:21:55] yt? [21:21:59] yes [21:22:14] ok, just making sure these forwarders work with the same setup as before [21:22:25] i'm tailing /var/log/eventlogging/client-side-events.log [21:22:26] ottomata: have 5 mins before i have to leave [21:22:40] and then hitting event.gif [21:22:42] and, isee the logs [21:22:45] but, i see them twice [21:22:47] was that happening before? [21:23:54] aha [21:24:18] ottomata: every hit on event.gif appear only once [21:24:20] HMMMm, maybe not. actually,i only see it twice if I am curling the url [21:24:34] if i load a page, i only see unique events [21:24:47] ottomata: that's even MORE odd [21:25:45] hm strange, yeah [21:26:11] hmmm, maybe i'm not. hmm [21:26:50] nawi think i'm just confused [21:27:29] hm, actually, i am seeing two diffent but similar requests when I curl [21:27:59] ottomata: will be back in 45 mins and i can take a look at whatever is needed. [21:28:08] i'm going to head out soon too, we can talk tomorrow [21:28:11] i think it is fine [21:28:23] maybe somehing weird happens with eventlogging when req doesn't come from JS [21:28:29] i see two similar requests, different UAs [21:28:32] body is the same, except for [21:28:47] one has scriptLoaded%22%3Atrue [21:28:47] the other has scriptLoaded%22%3Afalse% [21:28:50] strange! [21:28:54] anyway, looks like it is working! [21:30:34] ottomata: that ahem looks like needs to be looked at [21:30:54] ottomata: there are plenty request for EL that come to event.gif that do not use JS [21:37:29] i dunno, all i know is that a curl causes two different but very similar looking requests to be logged, at least that's what i'm seeing here [22:19:27] (PS1) Milimetric: [WIP] Add Sunburst Visualizer [analytics/dashiki] - https://gerrit.wikimedia.org/r/197234 [23:55:50] Analytics, Analytics-Kanban: Turn off Zero - Limn dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1123876 (kevinator) NEW