[03:34:42] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2781773 (10MoritzMuehlenhoff) I've just completed a backport of 6.9.1 for jessie-wikimedia. My build no longer links against the shared library copy of c-ar... [03:41:01] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2781774 (10Yurik) I have tried running Kartotherian (without rebuilding) under nodejs 6.9.1, and had some issues - Kartotherian wouldn't start due to mapnik... [08:51:43] (03CR) 10Elukey: "Added some comments for the comments :)" (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) (owner: 10Mforns) [09:02:52] I checked wf-text-2016-11-8-12 (last oozie warning) and the number of requests logged with dt - is ~50, that is almost irrelevant compared to the total traffic [09:03:31] With Marcel's new alarms we will probably not even get these anymore [09:06:03] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2781963 (10MoritzMuehlenhoff) >>! In T149331#2781773, @MoritzMuehlenhoff wrote: > I've just completed a backport of 6.9.1 for jessie-wikimedia. My build no... [09:30:57] 06Analytics-Kanban, 06Operations, 10Traffic: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2781980 (10elukey) We had a discussion on #wikimedia-traffic about this and the Analytics team completely agrees with what Brandon sai... [09:32:39] mforns: --^ :) [09:42:03] That's awesome elukey :) [09:43:01] \o/ [09:43:10] I am rebooting kafka hosts [09:43:14] k [09:43:15] for kernel + openjdk upgrades [09:48:23] elukey: Just saw for the superboos in US .... I'm gonna have a coffee in front of my garden ... [09:55:38] joal: :D [10:24:21] 10Analytics, 10Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2782033 (10elukey) Quick script to match UUID and mountpoint: ``` elukey@kafka1012:~$ cat /proc/mounts | grep spool | cut -d " " -f 1,2 | sort | while read pa... [11:08:31] going to take an early lunch [11:08:59] kafka101[83] and kafka1020 have been rebooted (kafka1013 only restarted since I rebooted it yesterday) [11:09:17] will finish the reboots after lunch (fixing fstab in the process) [11:45:49] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2782147 (10mobrovac) >>! In T149331#2781774, @Yurik wrote: > I have tried running Kartotherian (without rebuilding) under nodejs 6.9.1, and had some issues... [11:56:11] joal: where is the best place to see uniquie page views (on a graph if possible) for wikimedia sites? [11:56:30] Hi addshore [11:56:35] So far I have found https://reportcard.wmflabs.org/ and https://analytics.wikimedia.org/dashboards/vital-signs/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=Pageviews should I be pointing a person to one of those or somewhere else? [11:56:42] addshore: What do you mean by unique? [11:57:12] oh actually, I have re read the email and the request is for "unique visitors" [11:57:41] addshore: right :) [11:58:33] i guress vital-signs is the right thing then! [11:59:35] addshore: Looks correct [11:59:40] Great! :) [12:00:13] addshore: For more detailed pageviews you could go for: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-20&pages=Cat|Dog [12:00:17] For instance [12:28:35] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2782174 (10MoritzMuehlenhoff) > I'll refrain from uploading this to apt.wikimedia.org until the karthotherian problems are sorted out. In the mean time I'v... [12:59:46] (rebooting the other kafka nodes :) [13:14:42] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2782209 (10KartikMistry) I suggest we should test new Node in Beta cluster first. [13:25:13] Hey elukey, should we do something about node 6 migration? [13:25:29] ? [13:25:44] ahh nodejs [13:25:51] right, sorry :) [13:25:58] I think that we should test AQS in beta first [13:26:03] then upgrade [13:33:28] 10Analytics, 10Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2782250 (10elukey) ``` cat /proc/mounts | grep spool | cut -d " " -f 1,2 | sort | while read partition mountpoint; do uuid=$(sudo blkid | grep $partition | cut... [13:51:48] holy cow [13:52:03] I'm not sure how functional I can be today [13:53:03] UHHHHH [13:53:09] Hi milimetric ... I've been wandering a lot [13:53:16] 10Analytics, 10Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2782306 (10elukey) kafka1012 is up and running. To sanity check the kafka spools mount point I did the following: 1) `for el in `ls`; do sudo find $el; done >... [13:54:12] kafka1012 up and running again with UUIDs in the fstab [13:54:15] ottomata: --^ [13:54:19] You rock elukey [13:54:41] elukey: nice! [13:54:45] joal: I have a magic script to help :D https://phabricator.wikimedia.org/T147879#2782306 [13:54:51] it looks horrible now but it works [13:54:55] :) [13:55:28] :) [14:01:14] joal / halfak: we meeting? [14:01:49] milimetric, halfak: At your disposal [14:03:15] joal: we're in the hangout [14:03:23] Ah, joining [14:20:44] kafka hosts rebooted and fixed the remaining ones without UUIDs [14:20:57] (except eventbus but we'll do it for the next round of reboots) [14:23:00] yeehaw [14:23:24] elukey: i think main kafka already use uuids, since they aren't jbod [14:23:41] hello team, elukey thanks for the review! if I didn't mess up, the code should send an email also if the incomplete records surpass the threshold, but I'll have a closer look later today and also improve the other comments [14:26:19] joal, yt? [14:31:15] mforns: ah ok! it would be awesome if the message was different for the two use cases.. [14:31:57] ottomata: yep sorry didn't check, you are right! [14:32:23] Hi mfornsm I'm in meeting with milimetric and halfak [14:32:32] ottomata: qq about hive and the Job trackers.. would a simple job using hive issue a conn request to an1003 port 10000? [14:32:34] elukey, I see, mmmh OK will think about it [14:32:34] mforns: --^ sorry [14:32:44] joal, np [14:33:14] mforns: just if it is feasible [14:33:17] elukey: not totally sure, i think it depends on the way the job talks to hive. some i think only use the metastore [14:33:21] but others might use hive-server [14:33:31] okok :) [14:34:01] i could be wrong though, but i don't see why we'd restrict it anyway [14:34:28] sure sure I was just curious [14:34:36] about how it works :) [14:34:41] I agree with ANALYTICS [14:34:45] seems the best choice [14:37:02] aye :) [14:51:23] 10Analytics, 10Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2782435 (10elukey) Next and last step is to update the documentation for new hosts to use UUIDs. [14:51:32] 10Analytics, 10Analytics-Cluster: Audit fstabs on Kafka and Hadoop nodes to use UUIDs instead of /dev paths - https://phabricator.wikimedia.org/T147879#2782436 (10elukey) p:05High>03Normal [14:57:30] elukey: we still gotta do hadoop nodes too [14:59:30] ah yes [15:00:09] amended :) [15:01:20] mforns: not sure how you're dealing with output that has different dates, but I found out about unix join [15:01:48] for daily unique page creators I put the output in two files with date--tab-->count and did this: [15:01:57] join -a1 -a2 -eNULL -j1 -o 1.2,2.2,1.1,2.1 new.tsv old.tsv > join.tsv [15:02:20] lemme know if that'd be useful, I can explain more [15:02:59] hey mforns, I can talk now if you want [15:03:21] hi milimetric and joal [15:03:52] milimetric, not sure what you mean with different dates? [15:04:21] joal, I figured it out, I had a problem with specifying the partition in join clauses with wheres [15:04:28] joal, thanks anyway :] [15:04:37] Ah, ok mforns :) [15:05:06] joal, it seems you have to specify the partition in both the ON clause and the WHERE clause [15:05:11] for both tables [15:05:30] mforns: like if you select daily output and all dates are not represented in each result [15:05:37] milimetric, I see [15:05:54] then you have to basically full outer join the two outputs [15:05:55] k mforns, didn't know about that [15:05:59] milimetric, what I did was take from 20160101 onwards and it has data for all dates [15:06:01] and that's what that join does [15:06:07] :) [15:06:17] milimetric, ah! cool [15:06:35] k, that works too, but will ignore problems in early history (which is probably ok, I might have captured them in my queries) [15:06:57] milimetric, sure, I did an ugly workaround, the real solution is yours :] [15:07:14] full disclosure: I'm in a crazy emotional state right now and working if possible but probably taking time off for distress [15:07:35] when I get to vet all edit metric I'll change them to the mili-join :] [15:08:00] milimetric, I guess... [15:27:49] elukey: joal, yall mind if we push our meeting to :45 so i can watch this concession speech ? :( [15:27:56] no prob ottomata [15:27:58] nope [15:28:11] if it starts on time, that is [15:36:32] elukey: joal, not started, wanna just do meeting? [15:39:48] ottomata: sure, joining in a sec [15:40:25] k [15:50:22] ahha elukey 'starting' [15:51:38] ottomata: do you want to rejoin batcave? [15:52:20] yeah [15:52:24] I am reading the comments flowing in https://www.youtube.com/watch?v=b-B6AtuNlMY and I can't stop laughing [15:52:39] haha [15:52:42] gotta hide those [15:52:44] haha [16:11:35] joal: pivot restarted [16:55:46] 10Analytics: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2782769 (10Nuria) [17:40:09] milimetric, I hope you don't mind if I ping you in threads like this: https://en.wikipedia.org/wiki/User_talk:EpochFail#A_cup_of_coffee_for_you.21 [17:40:10] :D [17:40:46] hey mforns, want to say hello to Lino ? [17:41:07] joal, sure :] [17:41:34] mforns: he just arrived from the bath, and asked for colleagues [17:48:52] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 12 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2782913 (10Yurik) @KartikMistry I agree, but we don't have Kartotherian/Tilerator running in beta cluster (we should). @mobrovac, seems like my git repo got... [17:49:50] joal: I see that /user/joal/wmf/data/wmf/edit_history_prod/alized/_common_metadata is not updated, is the new data somewhere else? [17:49:59] or is the metadata file different... [17:51:00] halfak: I'm still not prioritizing mw pings properly, but I definitely want to be pinged anywhere I'm needed. I'll try to improve how I manage those, thanks for pinging me about the ping in the meantime [17:52:27] No problem :) [17:52:40] BTW, you can tell MW to email you when you get an on-wiki ping. [17:54:35] 10Analytics, 06Commons, 06Multimedia, 10Tabular-Data, and 4 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2782925 (10Yurik) Getting ready for deployment: here's tabular data example on beta cluster, that also supports localization, shared dat... [17:57:27] milimetric: it is ! [17:57:43] milimetric: /user/joal/wmf/data/wmf/edit_history/denormalized [17:57:48] aha, thx [18:07:12] cool, milimetric.mediawiki_history is pointing to the new data now, looks great, thanks. I'm going to go offline, drop off the car, and go meet yurik in manhattan [18:08:21] goin afk for a bit [19:04:36] milimetric, mforns : I managed to have calcite query Druid [19:04:44] Will play more with tomorrow [19:04:44] joal, oooh [19:04:56] But at least it's functional [19:05:02] this is awesome news [19:05:06] It is :) [19:05:17] I want to test: complex queries, response times etc [19:05:22] aha [19:05:27] But first baby step is done :) [19:05:46] And I'm done for tonight ;) [19:05:52] Tomorrow a-team ! [19:06:04] bye joal! cya tomorrow [19:37:30] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 11 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2748980 (10Yurik) [19:40:30] 10Analytics, 10Android-app-Bugs, 06Wikipedia-Android-App-Backlog: EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196#2783281 (10Niedzielski) p:05Triage>03Normal [19:49:07] 10Analytics, 10ChangeProp, 10Citoid, 10ContentTranslation-CXserver, and 11 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2783313 (10GWicke) @MoritzMuehlenhoff, thanks a lot for preparing the node debs! [20:18:39] 10Analytics-General-or-Unknown, 10Graphite, 06Operations, 06Performance-Team, 07Wikimedia-Incident: statsv outage on 2016-11-09 - https://phabricator.wikimedia.org/T150359#2783449 (10Krinkle) [20:19:08] 10Analytics-General-or-Unknown, 10Graphite, 06Operations, 06Performance-Team, 07Wikimedia-Incident: statsv outage on 2016-11-09 - https://phabricator.wikimedia.org/T150359#2783461 (10Krinkle) [20:22:03] 10Analytics-General-or-Unknown, 10Graphite, 06Operations, 06Performance-Team, 07Wikimedia-Incident: statsv outage on 2016-11-09 - https://phabricator.wikimedia.org/T150359#2783470 (10Krinkle) [20:29:19] 10Analytics-General-or-Unknown, 10Graphite, 06Operations, 06Performance-Team, 07Wikimedia-Incident: statsv outage on 2016-11-09 - https://phabricator.wikimedia.org/T150359#2783488 (10Krinkle) [20:29:33] joaL; nice! [20:34:47] OOooooboy [20:34:47] http://codepen.io/ottomata/pen/VKNyEw?editors=0011 [20:36:04] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review, 06Services (watching): Prepare eventstreams (with KafkaSSE) for deployment - https://phabricator.wikimedia.org/T148779#2783542 (10Ottomata) Yes! Deployed to beta via puppet and scap. This works! http://codepen.io/ottomata/pen/VKNyE... [20:49:07] 10Analytics, 10EventBus, 10Wikimedia-Stream: Productionize Public Event Stream Prototype - https://phabricator.wikimedia.org/T143925#2783646 (10Ottomata) [20:49:10] 10Analytics, 10Wikimedia-Stream, 10service-runner, 06Services (watching): Support node cluster sticky-session in service-runner - https://phabricator.wikimedia.org/T145805#2783644 (10Ottomata) 05Open>03declined We decided to use HTTP + SSE, instead of socket.io. Declining this ticket. [20:51:03] 10Analytics-EventLogging: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#2783668 (10kaldari) [20:55:37] 10Analytics, 10EventBus, 10Wikimedia-Stream: Productionize Public Event Stream Prototype - https://phabricator.wikimedia.org/T143925#2783705 (10Ottomata) [20:56:17] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review, 06Services (watching): Prepare eventstreams (with KafkaSSE) for deployment - https://phabricator.wikimedia.org/T148779#2783709 (10Ottomata) [20:56:37] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2783710 (10Ottomata) [20:56:39] 10Analytics, 10EventBus, 10Wikimedia-Stream: Productionize Public EventStreams - https://phabricator.wikimedia.org/T143925#2583449 (10Ottomata) [20:56:43] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Productionize Public EventStreams - https://phabricator.wikimedia.org/T143925#2583449 (10Ottomata) [20:56:45] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Productionize Public EventStreams - https://phabricator.wikimedia.org/T143925#2583449 (10Ottomata) [21:02:11] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Productionize Public EventStreams - https://phabricator.wikimedia.org/T143925#2783742 (10Ottomata) > How to get this into REST API, and at what path? `/api/rest_v1/stream/*` -> `eventstreams.svc.$site.wmnet/v1/stream/*` This needs to be done in some nginx o... [21:26:31] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Productionize and deploy Public EventStreams - https://phabricator.wikimedia.org/T143925#2783859 (10Ottomata) [21:47:04] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Productionize and deploy Public EventStreams - https://phabricator.wikimedia.org/T143925#2783942 (10Ottomata) Hm, possibly the domain / path at which this will be hosted is controversial. To keep things centralized, let's discuss on the [[ https://phabrica... [21:49:25] 10Analytics-EventLogging: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#2783953 (10Neil_P._Quinn_WMF) [22:10:22] (03CR) 10Yurik: [C: 032] "Lets get this running, and once analytics is ready, we can migrate it to their platform" [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: 10MaxSem) [22:12:27] byyyeyee [22:41:42] (03CR) 10MaxSem: [V: 032] Count pages with geo tags [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: 10MaxSem)