[06:07:14] (PS1) KartikMistry: Add 'gu' and 'vi' in dashboard [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/202002 [08:16:46] Analytics-EventLogging, operations, ops-eqiad: vanadium failed disk /dev/sda - https://phabricator.wikimedia.org/T94926#1181595 (yuvipanda) [11:06:52] Analytics, Labs, Tool-Labs: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1181937 (scfc) p:Triage>Normal [12:46:37] (CR) Mforns: [C: 2 V: 2] Add support for wiki explosion and others. [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/200239 (owner: Mforns) [13:51:38] (CR) Ottomata: "Huh?" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [13:53:02] Analytics-EventLogging, Ops-Access-Requests, operations: Grant user 'tomasz' access to dbstore1002 for Event Logging data - https://phabricator.wikimedia.org/T95036#1182107 (Ottomata) Thomas will need to be added to the 'researchers' group. Thomas, you can then read the file /etc/mysql/conf.d/researc... [13:54:34] Analytics-Tech-community-metrics: "Who contributes code" page metrics are not updating - https://phabricator.wikimedia.org/T95166#1182110 (Qgil) NEW a:Dicortazar [13:55:26] (CR) Mforns: "LGTM!" (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 (owner: Milimetric) [14:08:03] (PS14) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [14:16:01] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1182128 (kevinator) Moving to Done column as this was deployed last week (around April 1st) [14:22:05] Analytics-Tech-community-metrics, Phabricator, Wikimedia-Hackathon-2015, ECT-April-2015, ECT-March-2015: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#1182136 (Qgil) Status update: Bitergia has sent us a quote to develop a Maniphest backend for Metrics Grimoire, and to update htt... [14:22:58] hafnium graphite process is restarted [14:24:06] cc: mforns , milimetric , this is how to do it: https://wikitech.wikimedia.org/wiki/EventLogging#Fix_graphite_counts_not_working.3F [14:30:49] nuria: https://phabricator.wikimedia.org/T74747 has been deployed, correct? Can you mark the task as resolved? I'll send en email to wikimetrics-l later today. [14:31:21] kevinator: done [14:31:27] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Story: WikimetricsUser reads user names in a JSON report [8 pts] - https://phabricator.wikimedia.org/T74747#1182143 (Nuria) Open>Resolved [14:31:29] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1182145 (kevinator) [14:31:31] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Story: WikimetricsUser reads user names in a JSON report [8 pts] - https://phabricator.wikimedia.org/T74747#1182144 (kevinator) [14:34:24] (PS1) Mforns: Adjust config to run reportupdater [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/202033 (https://phabricator.wikimedia.org/T89251) [14:39:12] (CR) Mforns: "Thanks!" [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 (owner: Milimetric) [14:47:22] (PS1) Mforns: Fix bug in wiki.txt file path [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/202036 (https://phabricator.wikimedia.org/T89251) [14:49:02] kevinator, milimetric , mforns : i think with this upcoming quarter it will be worth having an on-call rotation for EL support [14:50:50] nuria, ok, lets talk about that on the meeting [14:51:00] how long would the rotations be? [14:51:07] kevinator: 1 week is plenty [14:51:32] nuria, kevinator, during weekends also? [14:52:15] mforns: well, depends on teh level of support we eant to provide [14:52:19] *we want [14:53:49] nuria, ok, if we feel the pain we'll make it more robust [14:54:11] mforns, milimetric , kevinator : this is a good topic for staff [14:58:26] Analytics-Tech-community-metrics: "Who contributes code" page metrics are not updating - https://phabricator.wikimedia.org/T95166#1182203 (Dicortazar) Indeed, the dashboard is not being updated. You'll probably see similar issues around. We're considering the option of downgrading the dashboard to the versi... [14:59:19] (CR) Mforns: [C: 2 V: 2] Adjust config to run reportupdater [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/202033 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [15:00:19] (CR) Mforns: [C: 2 V: 2] Fix bug in wiki.txt file path [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/202036 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [15:02:57] ottomata: yt? [15:04:46] ottomata: ping me when you get in and i explian what was happening with logs [15:05:40] yup hey [15:05:42] i'm here nuria [15:05:43] what's up? [15:06:14] ottomata: so, what was happening is that 1) disk got full 2) after that every record that we could not write to disk on /var/log/eventlogging [15:06:35] AHHH [15:06:37] 3) was being written to /var/log/upstart by the processors [15:06:44] 4) restart fixed it [15:06:49] ah ok, so / was full, which caused more logs to be written to / [15:06:51] ok cool [15:07:01] so those logs aren't being written to upstart/ now, ja? [15:07:04] but good fix is moving stuff arround so disk does not get full [15:07:10] correct [15:07:21] ALSO we should have seen a disk full alarm, did you see it? [15:07:46] no, don't htink I did [15:07:56] i just got alerts from eventlogging [15:08:01] i am going to make upstart be on /srv/ though [15:08:06] ottomata: i also restarted EL on hafnium and made sure there is an outgoing connection to the new machine [15:08:16] ok [15:08:17] ottomata: ok, and how do we setup a "disk full" alarm [15:08:18] ? [15:08:27] like the ones we used to have in vadaium [15:08:37] *vanadium [15:08:39] i'm surprised that there isn't one [15:08:44] i appllied all the same configs [15:08:46] ottomata: ya, me to [15:09:11] ottomata: cause that seems it should be part of puppet standard for the 'basic' machine provisioning [15:09:14] there is one [15:09:43] ottomata: ah ok, then maybe there was tons of space in the other partition [15:09:46] it triggered too [15:09:47] https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=eventlog1001&service=Disk+space [15:09:57] Analytics-Tech-community-metrics, ECT-April-2015, ECT-March-2015: Migrate Korma identitites database to SortingHat - https://phabricator.wikimedia.org/T92953#1182217 (Qgil) This task was planed for March. Do you have an estimated delivery time? [15:10:02] ottomata: ah, ok, so how do we do to receive those? [15:10:07] i don't know [15:10:17] did you used to get alerts for disk space from vanadium? [15:10:47] about disk space? [15:11:04] ottomata: mmmm....disk full i think so but others like 'disk failure" no... [15:13:28] nuria, logs in upstart are still growing fast [15:13:29] check [15:13:33] /var/log/upstart/eventlogging_consumer-all-events-log.log [15:14:28] ottomata: mmm.. what is up with that one? cause server one is smaller [15:14:50] ottomata: argh, and same for client one [15:14:50] eventlogging_consumer-client-side-events-log.log [15:15:00] growing at about 160kB/s [15:15:05] ottomata: is the other partition full again? [15:15:21] about 260 events per second [15:15:24] no [15:15:27] Analytics-Tech-community-metrics, ECT-April-2015, ECT-March-2015, Patch-For-Review: "Age of unreviewed changesets by affiliation" shows negative number of changesets - https://phabricator.wikimedia.org/T72600#1182225 (Qgil) >>! In T72600#1124857, @Qgil wrote: > In my browsers the graph ends in Janua... [15:15:30] /srv was never full [15:15:58] and "/var/log/eventlogging#" symlinks to "/srv" right? [15:16:03] yes [15:16:10] it was /var/log/upstart that caused this problem in the first place [15:16:56] ottomata: Now server side log is dumping everything AGAIN [15:17:03] eventlogging_consumer-server-side-events-log.log is also growing [15:17:03] ottomata: arghhhh [15:17:03] yup [15:17:14] ottomata: it was fine for a while [15:17:21] it is writing to /var/log/eventlogging just fine too [15:17:31] it isn't printing any errors though [15:17:34] in /var/log/upsart [15:17:43] why would it write its logs to upstart if it didn't have errors? [15:18:03] ottomata: it was printing errors a bit ago , but now it's dumping all again... man ... [15:18:26] Analytics-Tech-community-metrics: Consolidating time ranges across tech community metrics - https://phabricator.wikimedia.org/T86630#1182236 (Qgil) [15:18:28] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit, ECT-April-2015: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#1182234 (Qgil) [15:18:30] Analytics-Tech-community-metrics, ECT-April-2015: Migrate Korma identitites database to SortingHat - https://phabricator.wikimedia.org/T92953#1182235 (Qgil) [15:18:30] ottomata: where are logging levels for this thing [15:19:04] we have a logger ... but were are the logging levels? [15:19:22] hardcoded i think [15:19:44] nuria: what's up with eventlogging_consumer-mysql-m4-master.log [15:19:45] are those normal? [15:20:25] ottomata: that is fine i think, it's more verbose [15:20:30] ok [15:20:40] ottomata: and logs insertion times but it is doing fine [15:21:07] ottomata: also , schema counts are not coming in from hafnium -> we are doing great #not [15:21:59] nuria: immediately after you restarted eventlogging client side events consumer [15:22:03] it started printing logs to that file [15:22:09] grep -A 20 'Driving ' eventlogging_consumer-client-side-events-log.log [15:22:22] i just see logs, i don't see any errors there [15:23:07] oh what? [15:23:43] nuria: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/handlers.py#L148 [15:23:47] has this always been happening? [15:24:13] log.addHandler(handler) [15:24:45] ottomata: no, i think we have 1 extra consumer? [15:25:06] huh, no, this is the hanlder [15:25:10] it uses logging to write to the file [15:25:17] to write to the /var/log/eventlogging file [15:25:27] but, i would think, logging at level of info would make the log also be written to stdout [15:25:34] if we are configured to write info logs to stdout too [15:25:36] which i think we are [15:25:52] or stderr [15:25:53] https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/utils.py#L192 [15:26:20] https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/bin/eventlogging-consumer#L35 [15:27:45] ottomata: ya...but what i do not get is how this was not happening before, or maybe it was ...and we did not see it? [15:27:47] nooo [15:28:39] i don't get that part either [15:29:26] nuria i am looking at logs there [15:29:28] on vanadium [15:29:30] it was happening before [15:29:34] ya me too [15:29:35] / was just much bigger there [15:30:00] ok, ya i see it [15:30:13] now, this makes sense , i mean , it's bad but makes sense [15:30:44] ottomata: ok, let's lower the login level [15:30:59] Analytics-Kanban: Dashboard Directory- Hay collaboration - https://phabricator.wikimedia.org/T95172#1182277 (ggellerman) NEW [15:32:13] ottomata: let me look at login level for logger python module [15:32:48] Analytics-Kanban: Dashboard Directory- Hay collaboration - https://phabricator.wikimedia.org/T95172#1182285 (ggellerman) [15:33:42] well, nuria [15:33:46] hm [15:33:50] ottomata: yesss? [15:34:18] ottomata: seems that default level is warning [15:34:18] we should lower the logging level from debug to info [15:34:19] but [15:34:27] that wont' stop us from getting the lgos on stderr [15:34:43] how come? [15:35:29] because [15:35:29] https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/handlers.py#L142 [15:35:39] the logger for the /var/log/eventlogging file logs at level info [15:36:17] hm, nuria, maybe instead of calling addHandler [15:36:21] we could just call handler.emit() [15:36:22] manually [15:36:25] insteads of log.info() [15:36:26] ... [15:36:26] not sure [15:36:27] :) [15:36:45] ottomata: ok, have a meeting for goals now, give me 30 mins to look at it [15:41:50] naw, i found the right way :) [15:43:41] Analytics-EventLogging, Ops-Access-Requests, operations: Grant user 'tomasz' access to dbstore1002 for Event Logging data - https://phabricator.wikimedia.org/T95036#1182337 (Andrew) p:Triage>High [15:44:01] nuria: https://gerrit.wikimedia.org/r/#/c/202043/ [15:44:32] nuria: i'm going to manually edit python code on eventlog1001 and try that real quick [15:44:39] ottomata: k [15:46:41] gah, do you remember where python setup install actually installs this stuff? [15:47:52] yes, [15:48:55] (CR) KartikMistry: [C: 2] Add 'gu' and 'vi' in dashboard [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/202002 (owner: KartikMistry) [15:49:02] (Merged) jenkins-bot: Add 'gu' and 'vi' in dashboard [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/202002 (owner: KartikMistry) [15:49:25] ottomata: /usr/local/lib/python2.7/dist-packages/ [15:49:34] danke [15:51:41] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1182358 (Andrew) Once we have approval from both Toby and Howie, the clock will start ticking for this. A simple reply to this ticket from both is s... [15:52:33] ok, nuria, that looks good [15:53:06] ja, logs now going into /srv, no logs going into upstart [15:53:08] ottomata: ok, merging [15:53:12] danke, i will deploy [15:53:19] ottomata: k [15:53:24] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1182362 (Andrew) [15:53:57] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1176345 (Andrew) Sorry, not Howie obviously. Whoever Jon's direct manager is, which I can't currently tell from looking at the staff page :/ [15:56:56] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1182388 (Andrew) So, I guess just Toby. [16:00:15] ottomata: did you deployed & restarted? [16:00:25] i restarted a few mins ago [16:00:30] with the manually applied patch [16:00:33] haven't deploy with the actual change [16:00:42] was thinking about moving /va/rlog/upstart to /srv before I restarted again [16:01:05] ottomata: k, we are set then [16:01:09] going to deploy [16:01:54] ottomata: kj [16:03:21] Analytics-Cluster, Analytics-Kanban: Add map of tags to refined webrequest table - https://phabricator.wikimedia.org/T95178#1182414 (JAllemandou) NEW a:JAllemandou [16:05:05] ottomata: crisis averted for today -seems like- , only thing left would be "disk full" alarms [16:10:31] nuria: i don't think you ever got alarms about disk full. [16:10:38] hm [16:10:49] ottomata: I believe you [16:11:25] ottomata: i bet yuvi was the proxy [16:12:49] errgggghhh nuria and turning them for just analytics folks and just for eventlog1001 on would be difficult [16:13:06] ottomata: ok, i see [16:13:07] base -> base::monitoring::host -> nrpe::monitor_service { 'disk_space': [16:13:19] hmm, we could probably add a second check that did the same thing though [16:13:20] hmmmm [16:13:22] yeah...>... [16:13:37] ottomata: it's fine if you receive them [16:13:46] can someone tell me what the heck the password is for the MySQL db on dan-pentaho? [16:13:49] there aren't alerts sent for them [16:13:54] it isn't marked as critical [16:13:59] but, yes, we can make a new alert [16:14:03] that alerts us [16:14:04] i think [16:14:08] ottomata: ah, that would be something to change [16:14:16] ottomata: they should be critical [16:15:02] Ironholds: mysql? [16:16:27] nuria, the one the pentaho data lives in [16:16:43] milimetric, do you have root on stat1003? [16:19:54] Ironholds: maybe mforns knows it: it's not in : mysql -uroot -p -Dwarehouse [16:20:10] sorry: https://wikitech.wikimedia.org/wiki/Analytics/Pentaho [16:20:17] hi [16:20:29] I think I know it [16:20:32] hey mforns [16:20:33] Ironholds, ^ [16:20:36] great! Can you send it to me? [16:20:42] sure [16:23:04] ottomata: wait .... [16:23:51] Ironholds, sent it, lmk if it works [16:23:54] ta [16:25:07] ottomata: counts for schema rates are not coming in from hafnium, lemme look at what is being sent to statsd [16:27:02] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1182506 (JKatzWMF) @ ottomata my direct manager is toby, thanks! [16:30:50] mforns: no root, but I can sudo -u stats [16:31:15] milimetric, oh! ok [16:31:17] mforns: what do you need? [16:31:28] can you delete the following file on stat1003? [16:32:09] milimetric, /srv/limn-mobile-data/.reportupdater.history [16:32:29] mforns: done [16:33:15] milimetric, txh [16:33:19] thx! [16:33:26] nuria: https://gerrit.wikimedia.org/r/#/c/202048/1/manifests/role/eventlogging.pp [16:34:11] ottomata: nice. [16:34:30] Analytics-EventLogging, operations, ops-eqiad: vanadium failed disk /dev/sda - https://phabricator.wikimedia.org/T94926#1182542 (Andrew) p:Triage>High [16:35:23] Analytics-EventLogging, operations, ops-eqiad: vanadium failed disk /dev/sda - https://phabricator.wikimedia.org/T94926#1182543 (Ottomata) FYI, because of this, on Friday we replaced vanadium with eventlog1001. vanadium is no longer in production, and will be decommissioned this week. [16:36:49] ottomata: and last thing, iam looking at reports to statsd from hafnium (via tcpdump) and thins are reported ok, [16:36:59] ottomata: i have to see why rates are not showing up [16:37:45] ottomata: maybe rates were never there? [16:39:51] Analytics-EventLogging, operations, ops-eqiad: vanadium failed disk /dev/sda - https://phabricator.wikimedia.org/T94926#1182554 (yuvipanda) p:High>Low [16:39:59] ottomata: ok, they broke a while back: (3/25) http://graphite.wikimedia.org/render/?width=588&height=311&_salt=1428338381.823&from=00%3A00_20150301&until=23%3A59_20150406&target=eventlogging.schema.Edit.rate&target=eventlogging.schema.MediaViewer.rate [16:47:15] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1182571 (Tnegrin) approved [16:48:20] i just got a disk spasce alert! [16:49:58] ottomata: ok, good [16:50:05] well, not really, all of ops got paged too [16:50:25] ottomata: ay ay [16:50:39] ottomata: still looking at hafnium, i think there is sometthing odd there [16:53:31] k will be with you in a bit [17:14:17] Ironholds: you there ? [17:16:21] joal, yes, but in a meeting.w hat's up? [17:16:57] Ironholds: Wanted to talk a bit on utl tagging [17:17:11] Ironholds: Let me know when you have a minute :) [17:17:17] kk [17:18:43] thx :) [17:24:24] Analytics, Labs, Tool-Labs: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1182724 (DarTar) We released [[ http://figshare.com/articles/Wikipedia_Clickstream/1305770 | static datasets ]] and experimented with a lightweight API on Labs, designed by @... [17:25:22] Analytics, Labs, Tool-Labs: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1182729 (DarTar) [17:37:26] ottomata: k, let me know when you are back [17:38:28] nuria: back, was fixin something, eating lunch, and watching that 60 minutes thing :) [17:38:45] ottomata: is the 60 min thing worth it? [17:39:56] ja it sgood [17:40:15] lots of 'I know that person!' [17:40:15] :) [17:40:22] ottomata: jajaja [17:41:01] ottomata: ok, i think we broke graphite counts when we deployed adding the new kafka consumer [17:41:22] Hm, ok... [17:41:27] ottomata: http://graphite.wikimedia.org/render/?width=588&height=311&_salt=1428342071.451&from=00%3A00_20150301&until=23%3A59_20150406&target=eventlogging.schema.Edit.rate&target=eventlogging.schema.MediaViewer.rate [17:41:31] i thought we fixed that by commenting out the kafka part [17:41:32] of the reporter [17:41:53] ottomata: ya in vanadium [17:42:01] ottomata: but i think not in hafnium [17:42:20] ottomata: now in hafnium there are no errors in the log [17:42:40] hmmm, halfnium didn't deploy properly today [17:43:04] ottomata: i take it back : i did deploy my change there [17:43:05] 28a0bf667a3869e95af0997c90af28dd329f6485 [17:43:06] hm, but your rerporter change is there [17:43:16] " Statsd reporter checks format of processor files" [17:43:25] ya, but checked tcpdump [17:43:38] with: tcpdump host statsd.eqiad.wmnet -nnXSs 0 | grep -C4 Edit [17:43:57] and the stats sent are some stuff ori has there dealing with perf [17:43:58] hmmm, nuria [17:44:03] [@hafnium:/etc/eventlogging.d] $ cat consumers/graphite [17:44:06] tcp://hafnium.wikimedia.org:8600?socket_id=graphite [17:44:06] statsd://statsd.eqiad.wmnet:8125 [17:44:19] that should subscribe to eventlog1001, shouldn't it? [17:44:26] ya, but if you listen on tcp://hafnium.wikimedia.org:8600?socket_id=graphite nothing comes in right? [17:44:26] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#1182788 (Qgil) Currently multiple affiliations can be assigned to a user, but for different time periods (i.e. until January 2015 Mary was Independent, from Febr... [17:44:31] right [17:44:32] no, no [17:44:35] wait. [17:44:35] no? [17:45:03] there is an outgoing connection from hafnium -> eventlogging1001 [17:45:10] you can listen to it [17:45:21] ? [17:45:29] see: netstat -tulpen | grep eventlog1001.eqiad.wmnet [17:45:41] don't see anything [17:45:55] ah sorry, try IP [17:46:06] same [17:46:21] i see tcpdump traffic though [17:46:34] ah ok, see it [17:46:40] argh sorry, it's not tulpen [17:46:53] is: netstat -nputw [17:47:18] https://www.irccloud.com/pastebin/kzbtUXVB [17:47:33] (PS1) Mforns: Change output folder [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/202067 (https://phabricator.wikimedia.org/T89251) [17:48:12] milimetric, yt? [17:48:44] nuria, i htink this is my fault [17:48:58] well, hm [17:48:58] ottomata: it's nobody's fault [17:49:03] ottomata: this is software [17:49:51] haha' [17:49:52] i mean [17:49:53] this change [17:49:53] https://phabricator.wikimedia.org/rOPUP31cae723f550b7761d3a061278cc83954a8508a4#f8c267fa [17:49:55] did it [17:50:07] i thought that role::eventlogging::graphite was included on vanadium [17:51:05] i'm confused. [17:51:23] ahhhhh [17:51:28] got it [17:51:29] hm [17:51:34] ottomata: i looked ta those chnages but i still do not get it [17:51:40] *at [17:51:41] eventlogging::service::consumer { 'graphite': [17:51:41] input => "tcp://${::fqdn}:8600", [17:51:51] seems like all the right configs got changed. [17:51:55] i thought that consumer was being deployed on the main eventloging host [17:52:03] that was the most puzzling one [17:52:11] so, i just made it be a dynamic var so we wouldn't have to change it if we moved evenltogging again [17:52:16] but, that needs to point explicitly at eventlog1001 [17:52:34] ottomata; ok, was looking at that earlier [17:52:49] ottomata: should be easy to fix then [17:54:42] nuria: https://gerrit.wikimedia.org/r/#/c/202070/2/manifests/role/eventlogging.pp [17:55:44] ottomata: ok, from all i looked at before that seemed the most likely candidate, i know i am repeating myself but i like it when things make sense [17:55:47] mforns: yes, here [17:56:02] milimetric, hey the reportupdater worked [17:56:08] woo! [17:56:09] :) [17:56:18] it wrote the reports to the wrong directory though :[ [17:56:22] can you move them? [17:56:24] ottomata: so next run of puppet in hafnium should fix this [17:56:45] mforns: ok :) [17:56:48] milimetric, I already pushed the change to fix it: https://gerrit.wikimedia.org/r/#/c/202067/ [17:56:52] gotcha [17:56:59] waiting on jenkins... [17:57:07] looking now, moving [17:57:08] there it is [17:57:17] milimetric, It took 30 mins to compute 1 day for the 4 metrics [17:57:22] running puppet [17:57:28] mforns: that seems plenty fast [17:57:39] but we'll have to backfill slowly [17:58:02] milimetric, yes, if we have to backfill, however, it's good to take this into account [17:58:12] xD, yes [17:58:18] we'll have to backfill about 4 more days I guess [17:58:29] because good data started flowing in at the end of march basically [17:58:45] milimetric, oh! so maybe I can add the date change to the config changeset [17:59:18] milimetric, ok, let me change the dates before merging, then [17:59:40] ottomata: we need to re-start the consumer too, after puppet run [17:59:44] mforns: yeah, I think just the 4 days should be fine, it'll take a couple of hours to backfill but that sounds good [18:00:24] mforns: hm, wait a sec [18:00:31] yep [18:00:31] puppet hsould restart it, no? [18:00:37] Notice: /Stage[main]/Eventlogging/Service[eventlogging/init]: Triggered 'refresh' from 1 events [18:01:02] ottomata: no if it is alreday running [18:01:07] ok... [18:01:12] doing that [18:01:14] *already , right? [18:01:24] just done [18:02:06] ottomata: ok, we will keep an eye for graphite , let me see the tcpdump to statsd [18:02:42] ottomata: ok, now i see the counts [18:02:49] (CR) Ottomata: "Oh, got it, sorry." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [18:03:28] with this: tcpdump host statsd.eqiad.wmnet -nnXSs 0 | grep -C4 Edit | more [18:03:32] (CR) Ottomata: "It is filed:" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [18:06:08] ottomata: looks good, see: tcpdump host statsd.eqiad.wmnet -nnXSs 0 | grep -C4 Edit [18:06:09] Analytics-Engineering, MediaWiki-API, Wikipedia-Android-App, Wikipedia-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1182917 (Ottomata) Hm? The extension is meant to be used by MW code. We wouldn't code usages of the ext... [18:10:16] joal, now around [18:10:23] nuria: ja little perk up [18:10:23] http://graphite.wikimedia.org/render/?width=588&height=311&_salt=1428338381.823&from=00:00_20150401&until=23:59_20150406&target=eventlogging.schema.Edit.rate&target=eventlogging.schema.MediaViewer.rate [18:10:28] although I'm going to eat lunch soonish so... [18:10:29] ottomata: ok, i think today's excitement about El is over! [18:10:59] ottomata: something every day , man .... [18:11:11] back to mobile sessions [18:15:08] heh [18:15:10] yeh [18:20:13] hey Ironholds , sorry went away a few minutes [18:20:20] batcave ? [18:21:36] Analytics-Engineering, MediaWiki-API, Wikipedia-Android-App, Wikipedia-iOS-App: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#1182984 (Anomie) No. It would more likely do like rEXAN64e43ad61363ff69d2bad173a6693ff311876ec0#C1546702N... [18:25:14] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#1183004 (Krenair) >>! In T53050#1182788, @Qgil wrote: > Currently multiple affiliations can be assigned to a user, but for different time periods (i.e. until Jan... [18:28:03] Ironholds: My time to get to eat [18:28:12] Maybe later, or tomorrow :) [18:28:22] joal, sorry, didn't see your ping [18:28:25] I'm around from now onwards [18:28:41] ok, if you have some time now, let's go for that :) [18:29:01] Ironholds: should be short enough [18:29:40] Ironholds: batcave ? [18:31:12] Analytics, Analytics-Kanban: Publish aggregate geodumps of article pageviews - https://phabricator.wikimedia.org/T91331#1183029 (kevinator) p:Triage>Normal [18:31:26] Analytics-Tech-community-metrics: Ensure that most basic Community Metrics are in place - https://phabricator.wikimedia.org/T94578#1183032 (Qgil) Should this task be part of #ECT-April-2015 ? [18:31:31] joal, sure [18:31:41] in npow [18:39:59] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#1183053 (Qgil) a:RyanLane>None [18:42:42] Analytics-Engineering, Engineering-Community, ECT-April-2015: Tech Talk: May 2015: Kanban - https://phabricator.wikimedia.org/T95202#1183058 (Rfarrand) NEW a:Rfarrand [18:43:22] Lads, I'll see you tomorrow ! [18:43:37] Good end of day to y'all :) [18:44:23]