[00:33:22] nuria: https://github.com/wikimedia/node-rdkafka-stats [09:57:30] Analytics, ChangeProp, Citoid, Graphoid, and 6 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2759397 (mobrovac) [12:23:43] Analytics, ChangeProp, Citoid, Graphoid, and 6 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2759571 (faidon) >>! In T149331#2758280, @GWicke wrote: > So, here is a proposal: > > - Double-check Node 6 support for all production services. We have been testing major s... [12:27:39] Analytics, ChangeProp, Citoid, Graphoid, and 6 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2759582 (KartikMistry) @GWicke Manually testing using experimental packages is fine? ``` sudo apt -t experimental install nodejs ``` [12:28:15] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 9 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2759583 (KartikMistry) [12:29:56] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 9 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2759586 (MoritzMuehlenhoff) @KartikMistry : Yeah, should be fine. That's (module some jessie backport changes) what we'll be using in production as well. [13:08:51] Analytics, Analytics-Kanban, Pageviews-API: Non existing article is one of the most viewed according to the data returned by the /metrics/pageviews/top/ API - https://phabricator.wikimedia.org/T149178#2759658 (JAllemandou) a:JAllemandou [13:42:46] (CR) Ottomata: [C: 2 V: 2] Travis test with node 4, 6 & latest. [analytics/aqs] - https://gerrit.wikimedia.org/r/318995 (owner: GWicke) [13:56:48] ottomata: Hello ! [13:57:03] hiya! [13:57:08] ottomata: Howdy? [13:58:35] hiya, howdy things are good. in lancaster Pennsylvania this week [13:58:42] will be on the lookout for some amish [13:58:49] how you?\ [13:59:02] Back home ;) [13:59:13] After a great weekend [13:59:39] went to some music concerts where my brother played, always great :) [13:59:57] ottomata: If you have time and know how, I'd have some request [14:00:00] ottomata: I have reloaded data in druid edit-history-test data-source, and I think Pivot needs a little something to catch up with those [14:00:13] where your brother played? what does he play? [14:00:25] oh, hm, ok joal what does it need? [14:00:28] My brother was playing in Toulouse, he plays saxophone :) [14:00:31] a little soft kick in the butt? [14:00:36] awesome! what kind of sax? [14:00:38] ottomata: possibly yes [14:00:39] jazz or what? [14:00:58] ottomata: Mostly jazz and impro, but sometimes pop, sometimes even classical ;) [14:00:58] !log restarting pivot [14:00:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:01:04] cool [14:01:17] done, so easy thanks to luca yeehaw [14:01:34] This weekend two concerts : One classical (so AWESOME !), and one swing (so dancy !) [14:02:49] ottomata: If you like, a part of last year concert https://www.youtube.com/watch?v=Ve-B0Yt44EQ [14:03:18] ah cool, your bro is in the orchestra here? [14:03:29] Yup ! Tenor sax [14:04:04] nice [14:04:20] the piano solo is completely weird on that track :) [14:05:45] hm, weird in the meaning fantastic, great, awesome, and what else (weird was definitely the correct word here) [14:09:55] thanks ottomata for the restart, it worked - I'll ask you for another one later on today: still some missing fields [14:10:23] great [14:23:29] (PS1) Joal: Update jar version for webrequest load job [analytics/refinery] - https://gerrit.wikimedia.org/r/319084 (https://phabricator.wikimedia.org/T130249) [14:23:37] I messed up yesterday with nuria [14:23:46] I told her to restart [14:24:06] the pageview job, while it was the webrequest_load one that needed to be restarted [14:24:16] mwarf ... [14:27:35] doh! [14:27:44] ottomata: indeed [14:27:52] why did webrequest job need restarted? [14:28:21] ottomata: nuria patched the pageview definition, applied at load (now containing refine) stage [14:28:40] And I mecanically thought - Let's restart pageview job [14:29:09] ottomata: And actually the patch I wrote (see 4 lines above) is also needed before restart [14:29:33] applied at load/refine? [14:29:52] oh righ tright [14:29:53] right [14:29:55] is_pageview? [14:29:59] correct sir [14:30:13] right [14:30:14] aye [14:30:15] k [14:30:15] pageview job is actually extraction and format convertion [14:30:26] Man so easy to mistake [14:30:33] yeah [14:30:38] no harm done though [14:30:44] joal: shall I merge this since I'm looking at it? [14:30:46] jar bump? [14:30:50] no big deal - nothing broken, but still - Lucky I checked data this morning [14:30:58] ottomata: You can [14:31:00] (CR) Ottomata: [C: 2 V: 2] Update jar version for webrequest load job [analytics/refinery] - https://gerrit.wikimedia.org/r/319084 (https://phabricator.wikimedia.org/T130249) (owner: Joal) [14:31:01] Thanks mate [14:31:12] Will deploy it now [14:33:03] !log deploying refinery onto the cluster [14:33:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:36:21] ottomata: weird behavior from scap today :( [14:36:37] It has deployed successfully on stat1002, but doesn't seem to deploy on other nodes [14:41:52] hm [14:42:03] ottomata: I tried to force, but didn't change [14:42:05] and stat1002 is canary? [14:42:08] looking [14:42:08] yessir [14:42:42] ottomata: Just tried again and it worked [14:43:10] I think I have understood: explicit yes doesn't work as an answer to continue - needs to be y6 [14:43:15] ay [14:43:16] ah [14:43:17] aye [14:43:18] ok [14:43:18] cool [14:43:25] sorry for bothering ottomata [14:43:33] np [14:45:08] !log Restart webrequest load job to apply [14:45:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:35:43] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 9 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2760054 (GWicke) [15:38:12] headed to cafe, bbiab [15:45:39] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Merge unattached Phab and MW accounts in user data - https://phabricator.wikimedia.org/T149700#2760102 (Aklapper) [15:46:02] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Merge unattached Phab and MW accounts in user data - https://phabricator.wikimedia.org/T149700#2760102 (Aklapper) p:Triage>Low [16:01:28] (PS10) Milimetric: Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) [16:03:07] nuria (I know you're in a meeting, no hurry): on my system, d3 gets installed at the top level when I add vega to package.json, I have no idea why. I added d3 to package.json anyway, and on my system it has no effect, but it should fix the problem on your system [16:15:46] milimetric: what npm version do you have? [16:16:43] 3.10.6 [16:17:40] milimetric: does this display something? npm ls d3 -g [16:18:19] hm... /usr/local/lib [16:18:19] └─┬ vega@1.4.0 [16:18:19] └── d3@3.4.6 [16:18:31] I'll uninstall that and see [16:19:45] milimetric: right, I think that is where they are coming from [16:19:52] milimetric: let me know [16:20:23] no, still gets installed even after I remove vega [16:20:31] npm ls d3 -g now returns empty [16:20:45] it's just broken :) [16:23:25] milimetric: aham, let me see for a sec [16:23:48] milimetric: batcave? [16:24:11] nuria: did you get that repo created? [16:24:22] if not, want to just put it up somewhere so I can review ahead of time? [16:25:42] nuria: omw [16:27:07] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 11 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2760265 (KartikMistry) [16:34:08] (PS25) Joal: [WIP] Join and denormalize all histories into one [analytics/refinery/source] - https://gerrit.wikimedia.org/r/307903 (owner: Milimetric) [16:40:00] ottomata: o/ [16:40:44] I checked icinga briefly and it seems that kafka1002 has undefined datapoints (??) [16:40:51] jmxtrans shows a lot of [16:40:51] javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: kafka1002.eqiad.wmnet; nested exception is: [16:40:58] but restarting is not helping.. [16:41:03] have you seen the issue before? [16:41:04] HMMM [16:41:08] das weird [16:41:20] no [16:41:25] looking [16:41:27] if i can [16:41:32] i went to this cafe, expecting internet [16:41:43] i expected wrong! phone internet is being SLowowowww [16:42:08] elukey: its just 1002? [16:42:50] the alarm was also for 2001 but now it is gone.. [16:42:51] milimetric: all good now, build two layouts and things work as far as i can tell [16:42:51] mmmm [16:42:57] maybe only temporarily glitch? [16:43:04] milimetric: should we merge and deploy to analytics. wikimedia.org? [16:43:08] nuria: cool [16:43:17] ottomata: let me look to see if depot was created [16:43:18] nuria: we can merge, no urgent need to deploy [16:43:29] milimetric: i think we should deploy to avoid surprises [16:43:34] sure, no problem [16:43:42] milimetric: merging then [16:44:00] elukey: yeah hm [16:44:29] (CR) Nuria: [C: 2 V: 2] Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) (owner: Milimetric) [16:45:26] ottomata: so, no i cannot see module under operations/debs yet [16:46:01] ottomata: weird, but probably not a big issue :) [16:46:07] * elukey afk again :) [16:46:30] a-team: alert on pageview unexpected values is for those: https://gist.github.com/jobar/c758b34bd91479be02a9b2f5ef69e6db [16:46:34] operations debs??? [16:46:46] nuria: did you ask for diffusion or gerrit? [16:46:47] We should discuss this (maybe post-staff?) [16:47:31] joal: it's my ops week, I'm looking [16:48:18] joal: so those are showing up due to nuria's patch being deployed and running on load? [16:48:25] correct milimetric [16:48:30] so I should just add them to the whitelist, anything else? [16:48:52] ottomata: gerrit, wait isn't top level what you wanted let me see [16:48:59] milimetric: adding to the list will prevent alarm - We just need to make sure thos porjects should be accepted :) [16:49:05] milimetric: I guess yes :) [16:49:31] (PS28) Milimetric: Script sqooping mediawiki tables into hdfs [analytics/refinery] - https://gerrit.wikimedia.org/r/306292 (https://phabricator.wikimedia.org/T141476) [16:49:52] nuria: yes. nuria if gerrit, i can create it [16:49:58] joal: yeah, I see, we were originally only going to accept some of them [16:49:59] will do so now [16:50:02] I see now that we're doing all of them? [16:50:10] ottomata: operations/node-rdkafka-stats then? [16:50:21] i see [16:50:22] https://gerrit.wikimedia.org/r/#/admin/projects/node-rdkafka-stats [16:50:24] but wrong name! [16:50:28] just node-rdkafka-statsd [16:50:29] I'll add them to a patch and we'll merge after we talk in staff [16:50:30] top level [16:50:31] no operations/ [16:50:39] will delete that one and create new [16:50:43] ottomata: k [16:52:07] nuria: https://gerrit.wikimedia.org/r/#/admin/projects/node-rdkafka-statsd [16:54:40] ottomata: many thanks, did you deleted the other one? [16:54:40] nuria: for staff, can you call my phone? internet is SO slow [16:54:52] nuria: gerrit has changed the way it deletes, trying to figure out how [16:54:56] yes [16:54:57] also asking in -devtools [16:55:00] ottomata: will call, yes [16:58:09] Analytics, Mobile: Calculate monthly mobile Wikipedia edits for Major Gifts report - https://phabricator.wikimedia.org/T149710#2760424 (Reedy) [17:00:37] (PS1) Milimetric: Include pageviews for all wikis in whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/319105 (https://phabricator.wikimedia.org/T130249) [17:00:39] Analytics, Beta-Cluster-Infrastructure, WikimediaPageViewInfo: Deploy WikimediaPageViewInfo extension to beta cluster - https://phabricator.wikimedia.org/T129602#2110544 (Reedy) Don't want to do this until T148775 is fixed, but I'm happy to deploy this to beta and/or production afterwards as deemed a... [17:26:38] Analytics: Add help page in wikitech on what the analytics team can do for you similar to release engineering page - https://phabricator.wikimedia.org/T116188#1742698 (Nuria) Some notes on what we can do for you on staff etherpad. [17:28:16] wikimedia/mediawiki-extensions-EventLogging#616 (wmf/1.29.0-wmf.1 - 41f8b94 : Chad Horohoe): The build has errored. [17:28:16] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.1 [17:28:16] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/172356535 [17:32:06] Analytics-Wikimetrics, Quarry: Include Tulu Wikipedia in Metrics and Quarry - https://phabricator.wikimedia.org/T148950#2760605 (Pavanaja) [17:39:15] nuria: so I'll submit the stuff you +2 now and deploy [17:39:44] milimetric: k [17:40:24] (CR) Nuria: [C: 2 V: 2] Include pageviews for all wikis in whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/319105 (https://phabricator.wikimedia.org/T130249) (owner: Milimetric) [17:40:44] (Abandoned) Milimetric: Improve build a bit more [analytics/dashiki] - https://gerrit.wikimedia.org/r/314622 (owner: Milimetric) [17:40:57] (CR) Milimetric: [C: 2 V: 2] Clean up folder hierarchy [analytics/dashiki] - https://gerrit.wikimedia.org/r/316834 (https://phabricator.wikimedia.org/T147884) (owner: Milimetric) [17:41:07] joal: we need to just update table in hive right? no need to deploy a new jar and such [17:41:25] (PS2) Milimetric: Clean up folder hierarchy [analytics/dashiki] - https://gerrit.wikimedia.org/r/316834 (https://phabricator.wikimedia.org/T147884) [17:41:29] (CR) Milimetric: [V: 2] Clean up folder hierarchy [analytics/dashiki] - https://gerrit.wikimedia.org/r/316834 (https://phabricator.wikimedia.org/T147884) (owner: Milimetric) [17:48:04] (PS11) Milimetric: Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) [17:48:36] nuria: let me think about that twice [17:49:10] (CR) Milimetric: [C: 2 V: 2] Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) (owner: Milimetric) [17:49:35] nuria: correct - the pageview_whitelist table is one of the few cases where we have a static path [17:50:42] nuria: sorry, more precision: no need to release/deploy refinery-source, everything happen in refinery [17:51:09] nuria: also no need to restart the load job, hive should pick up the new values magically [17:52:04] (CR) Joal: [C: 2 V: 2] "Merging, discussed with team." [analytics/refinery] - https://gerrit.wikimedia.org/r/319105 (https://phabricator.wikimedia.org/T130249) (owner: Milimetric) [17:52:21] !log Deploying refinery [17:52:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:53:15] magic! [17:53:15] :) [17:53:23] milimetric: :) [17:54:09] joal: ok, will wait a bit, do some queries and update pageview definition page on meta [17:54:21] k nuria, thanks :) [17:55:22] (PS1) Milimetric: Build dashboards with new local fonts [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/319111 [17:55:41] (CR) Milimetric: [C: 2 V: 2] Build dashboards with new local fonts [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/319111 (owner: Milimetric) [17:59:50] nuria: ok, that's merged and will be deployed by puppet automatically. Do check out the fonts when you're done, because it turns out the old way we were building it was never including the google fonts at all, so we were just defaulting to Arial [18:07:57] nuria: patch applied, hoppefully not too many more to go with :) [18:08:09] milimetric: --^ [18:08:15] awesome [18:08:17] thx [18:34:57] ottomata: Heya, mind restarting pivot again? [18:35:29] np... [18:35:45] done [18:35:49] Thanks ! [18:48:42] a-team, going for today, see you tomorroe [18:49:00] laters joal [18:57:02] joal (for tomorrow) updated changelog: https://meta.wikimedia.org/wiki/Research:Page_view#Change_log [19:47:20] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T130651#2761071 (Ottomata) [19:47:23] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T130651#2761071 (Ottomata) [19:48:31] oops [19:50:03] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2761079 (Ottomata) [19:50:06] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2761079 (Ottomata) [19:50:35] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2761082 (Ottomata) [19:50:39] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2761082 (Ottomata) [19:52:21] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2761097 (Ottomata) Whoops, meant to create a subtask, not edit this one. Reverted. [19:52:24] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2761097 (Ottomata) Whoops, meant to create a subtask, not edit this one. Reverted. [19:52:38] Analytics-Kanban, Wikipedia-iOS-App-Backlog, iOS-app-feature-Analytics: Drop in iOS app pageviews since version 5.2.0 - https://phabricator.wikimedia.org/T148663#2761099 (JMinor) @dr0ptp4kt this seems out of the scope of iOS team engineering development (submitting CR's for analytics code). Is this s... [19:52:41] Analytics-Kanban, Wikipedia-iOS-App-Backlog, iOS-app-feature-Analytics: Drop in iOS app pageviews since version 5.2.0 - https://phabricator.wikimedia.org/T148663#2761099 (JMinor) @dr0ptp4kt this seems out of the scope of iOS team engineering development (submitting CR's for analytics code). Is this s... [21:28:25] Analytics, Dumps-Generation: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2761520 (Bawolff) btw, I'm untagging from security since there is no security impact on MediaWiki. [21:52:04] Quarry, Labs: 502 Bad Gateway on HTTP-requests to quarry.wmflabs.org - https://phabricator.wikimedia.org/T121502#1880330 (MuhammadShuaib) Again we face the same issue. [21:52:48] Quarry, Labs: 502 Bad Gateway on HTTP-requests to quarry.wmflabs.org - https://phabricator.wikimedia.org/T121502#2761620 (MuhammadShuaib) Resolved>Open [21:53:41] Quarry, Labs: 502 Bad Gateway on HTTP-requests to quarry.wmflabs.org - https://phabricator.wikimedia.org/T121502#1880330 (Matthewrbowker) >>! In T121502#2761617, @MuhammadShuaib wrote: > Again we face the same issue. Appears to be related to https://lists.wikimedia.org/pipermail/labs-announce/2016-Nove... [22:08:43] Quarry, Labs: 502 Bad Gateway on HTTP-requests to quarry.wmflabs.org - https://phabricator.wikimedia.org/T121502#2761641 (Matthewrbowker) Open>Resolved Works for me now. Is related to my previous comment. [22:15:52] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Merge unattached Phab and MW accounts in user data - https://phabricator.wikimedia.org/T149700#2761646 (Aklapper) Open>Resolved Alright, approx. 600-700 accounts merged in https://github.com/Bitergia/mediawiki-identities/commit... [22:35:21] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2761685 (Krinkle) @Ottomata Is `recentchanges` not yet included? [23:04:39] Analytics-Kanban, Wikipedia-iOS-App-Backlog, iOS-app-feature-Analytics: Drop in iOS app pageviews since version 5.2.0 - https://phabricator.wikimedia.org/T148663#2761742 (Fjalapeno) @Nuria Sorry I only provided the phone version of the user agent string. As you pointed out there is a "Tablet" variant...