[01:05:52] Analytics-Engineering, Community-Tech: [AOI] Add page view statistics to page information pages (action=info) - https://phabricator.wikimedia.org/T110147#1569847 (kaldari) NEW [01:06:38] Analytics-Engineering, Community-Tech: [AOI] Add page view statistics to page information pages (action=info) - https://phabricator.wikimedia.org/T110147#1569857 (kaldari) [01:11:59] Analytics-Engineering, Community-Tech: [AOI] Add page view statistics to page information pages (action=info) - https://phabricator.wikimedia.org/T110147#1569862 (kaldari) [01:44:03] Analytics-EventLogging, Analytics-Kanban: Update Schema Talk pages {tick} [8 pts] - https://phabricator.wikimedia.org/T103133#1569899 (madhuvishy) a:madhuvishy [03:13:52] @tgr Hey! Just saw your comment on my talk page. Yes, that shouldn't have happened. I was updating all the talk pages with EL schema purging information. I'm scanning through the list and making sure I revert any such documentation [03:14:08] Sorry about that [05:15:49] Analytics-Tech-community-metrics, Research consulting, Research-and-Data: Data for audit report - https://phabricator.wikimedia.org/T110067#1570065 (Tbayer) For the first question, feel free to reuse the method and numbers for the WMF quarterly report from T106502#1523010 : - [[https://meta.wikimed... [12:00:28] Analytics-Tech-community-metrics, ECT-August-2015: "Median time to review for Gerrit Changesets, per month": External vs. WMF/WMDE/etc patch authors - https://phabricator.wikimedia.org/T100189#1570762 (Qgil) p:Normal>High This task is blocking {T88531} and {T107562} and, as said above, even touches... [12:37:02] Analytics-Tech-community-metrics, ECT-August-2015: Exclude third-party / pulled upstream code repositories from metrics - https://phabricator.wikimedia.org/T103984#1570832 (Qgil) Varnish doesn't appear at https://github.com/Bitergia/mediawiki-repositories/blob/master/gerrit_projects.conf anymore, but is s... [12:41:40] Analytics-Tech-community-metrics, ECT-August-2015: Remove deprecated repositories from korma.wmflabs.org code review metrics - https://phabricator.wikimedia.org/T101777#1570844 (Qgil) https://github.com/Bitergia/mediawiki-repositories/pull/1 requested in order to remove ExternalArticles. The other repos l... [13:01:05] o/ milimetric & joal, I might need to reboot. [13:01:12] So I'm going to be a bit late. [13:01:21] Got a weird chrome bug that is acting up. [13:01:33] hey halfak no prob [13:03:47] morning! FYI, joal, last night I bumped up num replica threads on the 3 new brokers, and started moving upload partitions [13:04:10] ok [13:04:17] thx for letting me know ottomata :) [13:04:49] The max lag has indeed taken a good bump ! [13:05:18] hehe, its kinda pretty over time! [13:05:22] we're making rainbows! [13:05:26] ) [13:05:28] :) [13:06:16] heh, broker log size is kinda cool too, eventually the two sets of lines will converge [13:09:19] halfak: sorry! [13:09:30] I had a rough trip back yesterday, got home late and overslept [13:09:30] No worries. Just started :) [13:09:34] k, brt [13:11:39] Quarry: Show all published queries in profile - https://phabricator.wikimedia.org/T77948#1570952 (matej_suchanek) [13:12:47] milimetric: not now, but later today, let's make a report, eh? of data loss? [13:17:59] ottomata: yes, I'm game [14:10:00] Analytics-Dashiki: vital-signs doesn't display pageviews graph - https://phabricator.wikimedia.org/T109693#1571054 (Milimetric) What Browser and version is this? I'm trying to find out if it's a race condition or a browser bug. It works ok for me in Firefox 39 and Chrome 44.0.2403.130 (64-bit) and I haven'... [14:10:45] Analytics-Backlog, Analytics-Dashiki: vital-signs doesn't display pageviews graph - https://phabricator.wikimedia.org/T109693#1571055 (Milimetric) p:Triage>High [14:19:59] hi a-team! [14:20:18] hello! [14:20:27] hi mforns :) [14:23:21] :] [14:35:16] Analytics-Backlog: Stats for en.wikinews.org not working - https://phabricator.wikimedia.org/T109146#1571112 (Milimetric) Open>Invalid a:Milimetric @DragonFire1024, it sounds like there's no issue here other than normal workflow. The next version of wikistats aims to update more frequently and in a... [14:51:22] Analytics-Backlog, Analytics-EventLogging: Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts] - https://phabricator.wikimedia.org/T106254#1571164 (Ottomata) [14:54:41] Analytics-Backlog, Analytics-EventLogging: Make EventLogging monitoring and alerts based on Kafka metrics {stag} [8 pts] - https://phabricator.wikimedia.org/T106254#1571170 (Ottomata) ToDo: [] Undo `eventlogging::monitoring::ganglia` [] Undo `role::eventlogging::reporter` (on hafnium) [] Undo `role::even... [14:55:16] joal, I think I have concluded that nothing around Bzip2Codec references "blocksize" or "compression level" [14:55:59] halfak: hm, so no config for bz2 compression in hadoop [14:56:06] I wouldn't have guessed so :( [14:56:47] Yeah. Nothing that I can find. I'd appreciate if you could take a look with your own methods. No rush though. When you're done with the Pageview push is cool. [14:57:03] Just letting you know that I have given up and I'll just recompress bz2 with the default settings. [15:00:12] * halfak finds a snappy utility [15:00:18] I might not even convert to bz2 [15:00:40] :) [15:15:38] Analytics-Kanban: Create Hadoop Job to load data into cassandra [21?? pts] {slug} - https://phabricator.wikimedia.org/T108174#1571211 (JAllemandou) I have tried to test https://github.com/spotify/hdfs2cass. It is not directly usable for two reasons: - I doesn't handle username/password authentication for the... [15:20:46] halfak: altiscale meeting today ? [15:21:04] I figured we could ask 'em about compression. [15:21:44] fai [15:21:51] fair sorry [15:24:18] I filed a ticket. [15:24:23] ok [15:24:47] I'd also like to push on them a bit for the difficulty in transfering data to and from the cluster. It is a pain to use AWS buckets for all of this. [15:30:10] halfak: makes sense [15:30:20] standup! [15:30:21] ah! [15:33:31] milimetric: having some internet trouble, trying to join standup. Might be a bit late. Can you let team know [15:33:40] k [15:36:36] madhuvishy: want us to call your cell? [15:37:28] Analytics, Analytics-Kanban: Transform to XML-->JSON in sorted file format - https://phabricator.wikimedia.org/T108684#1571291 (JAllemandou) [15:37:50] Analytics, Analytics-Kanban: Transform to XML-->JSON in sorted file format - https://phabricator.wikimedia.org/T108684#1571292 (JAllemandou) p:Triage>Normal [15:39:19] Analytics-Cluster, Analytics-Kanban, operations, Monitoring: Replace uses of monitoring::ganglia with monitoring::graphite_* - https://phabricator.wikimedia.org/T90642#1571305 (Ottomata) We removed a bunch of monitoring::ganglia usages as part of the Kafka upgrade and expansion. The only one that i... [15:52:46] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Move Eventlogging Kafka writer to use pykafka's Producer instead of python-kafka {stag} [8 pts] - https://phabricator.wikimedia.org/T109244#1571384 (madhuvishy) Open>declined [15:52:47] Analytics-EventLogging, Analytics-Kanban: {stag} EventLogging on Kafka - https://phabricator.wikimedia.org/T102225#1571385 (madhuvishy) [16:03:54] Analytics-Backlog: Stats for en.wikinews.org not working - https://phabricator.wikimedia.org/T109146#1571428 (ezachte) Some background: Wikistats used to publish fully automated. But once, when a bug got numbers totally wrong, no-one told me, I only learned of it when an article had been posted on the issue i... [16:33:34] Analytics-Kanban, operations, Monitoring, Patch-For-Review: Overhaul reqstats - https://phabricator.wikimedia.org/T83580#1571705 (Ottomata) Natively share the dict? Hm. Just quickly tried this, and I get an immediate segfault: ``` Aug 25 16:32:25 cp1052 kernel: [8455259.595360] python[7589]: segfa... [17:21:09] Analytics-Backlog, Analytics-Dashiki, Browser-Support-Firefox: vital-signs doesn't display pageviews graph in Firefox 41, 42 - https://phabricator.wikimedia.org/T109693#1572276 (Jdforrester-WMF) [18:26:40] milimetric: can you send me a GPG-signed mail? -> dario@wikimedia.org [18:26:57] or I’ll text you [18:27:12] DarTar: no idea how to do that stuff [18:27:39] text is probably safe in this case [18:29:55] milimetric: kk [18:47:57] milimetric: I saw wikimetrics and went whaaa [18:48:21] if you're watching the video [18:50:04] madhuvishy: yeah, of course, people love that idea, but we just coded it for the wrong audience :) [18:50:19] we'll make it better, I think amanda's going in the right direction with it [18:50:53] milimetric: :) [19:06:28] Analytics-Backlog: Stats for en.wikinews.org not working - https://phabricator.wikimedia.org/T109146#1572782 (Milimetric) Right, the only way to go back to automated would be if we had monitoring in place that would let us know when processing is not going as expected. [19:46:37] halfak: You there ? [19:46:45] o/ [19:46:47] What's up? [19:47:01] Got some code for you :) [19:47:10] https://github.com/jobar/analytics-wikihadoop/tree/json_sorted_revs [19:47:26] wanna batcave for a minute ? [19:47:58] sure! [20:21:07] milimetric, do you have 10 minutes later on today? not for dashiki, but to show you some sql [20:27:18] mforns: do you think we can do this now? [20:27:19] https://phabricator.wikimedia.org/T108857 [20:27:29] madhuvishy, looking [20:27:52] Bye a-team, I'm off to bed :) [20:27:59] later joal! [20:28:00] bye joal! [20:28:13] Bye! [20:29:28] madhuvishy, I'm still not sure if we can do this or Sean will want to take part? [20:30:05] latesr joal! [20:30:07] mforns: alright. i guess he could do the dropping tables part. we'll do the page archival? [20:30:10] madhuvishy, I'm not even sure if we have the permits to delete tables in the db [20:30:25] madhuvishy, yes sure, that part we can do :] [20:30:59] mforns: alright - I'll make a tiny separate subtask for it and archive them [20:31:21] madhuvishy, cool :] [20:32:42] mforns: I've been in meetings and running numbers all day, maybe we can catch up tomorrow morning [20:32:55] milimetric, sure :] np [21:48:28] Analytics-Kanban: Archive obsolete schema pages {tick} - https://phabricator.wikimedia.org/T110247#1573390 (madhuvishy) NEW a:madhuvishy [22:02:48] Analytics-Kanban: back up data used to crunch numbers - https://phabricator.wikimedia.org/T110255#1573492 (Milimetric) NEW [22:25:24] milimetric: around? [22:26:00] hey madhuvishy what's up? [22:26:04] small question - how did you archive old pages during the documentation sprint we had a while back [22:26:52] milimetric: ^ [22:27:36] ooh, I forgot [22:27:57] sorry, head is full :( [22:30:03] milimetric: no problem! ping me if you remember - even if tomorrow [22:30:28] madhuvishy: i think we never archived them, at least i didn't, i just put them in the archive section of the etherpad [22:30:37] aah [23:29:29] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Upgrade Camus - https://phabricator.wikimedia.org/T109860#1573695 (kevinator) Open>Resolved [23:31:02] Analytics-Cluster, Analytics-Kanban: Add hourly aggregate sequence stats creation to webrequest load job - https://phabricator.wikimedia.org/T109136#1573703 (kevinator) Open>Resolved