[00:00:28] ottomata, do you have a sense for what it means to consider working with a project that is in the "Apache Incubator"? [00:01:26] i.e. is it likely to be a mess? [00:38:07] Analytics-EventLogging, Analytics-Engineering: Community Engineers have a working EventLogging dev server in vagrant - https://phabricator.wikimedia.org/T86067#961462 (kevinator) p:Triage>Normal [00:43:49] Analytics-Wikimetrics, Analytics-Engineering: Epic: Grantmaking User gets reports on Wikimetrics usage - https://phabricator.wikimedia.org/T76106#961493 (kevinator) When Wikimetrics is deployed on Thursday January 8 2015, I'll send out a notice. [00:58:07] (PS11) Nuria: Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 [01:06:20] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Performance histogram for MediaViewer and File page - https://phabricator.wikimedia.org/T77694#961599 (Tgr) p:Normal>Low [01:07:58] (CR) MarkTraceur: [C: 2] Do not count Commons opt-ins as opt-outs [analytics/multimedia] - https://gerrit.wikimedia.org/r/183392 (owner: Gergő Tisza) [01:08:03] (Merged) jenkins-bot: Do not count Commons opt-ins as opt-outs [analytics/multimedia] - https://gerrit.wikimedia.org/r/183392 (owner: Gergő Tisza) [01:27:35] Services, MediaWiki-General-or-Unknown, Analytics, operations, Wikidata, wikidata-query-service: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#961647 (GWicke) >>! In T84923#961622, @bd808 wrote: >> can support large delays (order of days) for individual consumers > > Do yo... [05:33:02] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Add Last-Modified to performance logging - https://phabricator.wikimedia.org/T78767#961790 (Gilles) Open>Resolved [05:33:15] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Add Last-Modified to performance logging - https://phabricator.wikimedia.org/T78767#852822 (Gilles) [05:33:17] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Update network performance schema queries - https://phabricator.wikimedia.org/T84852#961792 (Gilles) Open>Resolved [06:34:02] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#961885 (Qgil) [06:44:24] Analytics-Tech-community-metrics, Phabricator: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#961889 (Qgil) [06:47:35] Analytics-Tech-community-metrics, Phabricator: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#961908 (Qgil) I think the concept of [[ https://wikitech.wikimedia.org/wiki/Key_Wikimedia_software_projects | key Wikimedia software projects ]] is interesting but has its drawbacks as well. All projec... [06:58:35] Analytics-Tech-community-metrics, Phabricator: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#961911 (Qgil) @kevinator said in T1003#844983: > When I saw the wiki with number of active bugzilla users, I wondered if you knew about the graph extension that lets you embed graphs into wiki pages. >... [07:00:57] Analytics-Tech-community-metrics, Phabricator: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#961915 (Qgil) [07:04:37] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#961920 (Qgil) The report send on January 1 looks good! Thank you very much. I have [[ https://lists.wikimedia.org/pipermail/wikitech-l/2015-January... [07:15:22] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#961937 (KartikMistry) Is it possible to get teamwise statistics? (eg: Language Engineering) (Just wishlist) :) [07:51:02] Analytics-Engineering: Epic: Offboarding Transition - https://phabricator.wikimedia.org/T86135#962018 (kevinator) NEW [07:53:25] Analytics-Engineering, Analytics-EventLogging: Story: Analytics Eng can monitor database replication lag - https://phabricator.wikimedia.org/T86136#962024 (kevinator) NEW [07:53:34] Analytics-Engineering, Analytics-EventLogging: Story: Analytics Eng can monitor database replication lag - https://phabricator.wikimedia.org/T86136#962024 (kevinator) p:Normal>High [07:53:53] Analytics-Engineering: Epic: Offboarding Transition - https://phabricator.wikimedia.org/T86135#962032 (kevinator) [07:54:27] Analytics-Engineering: Epic: Offboarding Transition - https://phabricator.wikimedia.org/T86135#962018 (kevinator) [08:05:30] Analytics-Engineering, Analytics: Engineer adds data to X-Analytics header using mediawiki extension - https://phabricator.wikimedia.org/T78801#962061 (kevinator) [08:05:32] Analytics-Cluster, Analytics-Engineering: Allow Mediawiki to set X-Analytics fields which means that Mediawiki code can set fields - https://phabricator.wikimedia.org/T77389#962062 (kevinator) [08:07:07] Analytics-Cluster, Analytics-Engineering: Mediawiki Eng adds fields to X-Analytics header - https://phabricator.wikimedia.org/T77389#962064 (kevinator) [08:09:56] Analytics-Cluster, Analytics-Engineering: Add page_id in the X-Analytics field - https://phabricator.wikimedia.org/T77416#962076 (kevinator) [08:12:40] Analytics-Engineering, Analytics-EventLogging: EL office hours - https://phabricator.wikimedia.org/T76796#962081 (kevinator) a:kevinator [08:13:08] Analytics-Cluster, Analytics-Engineering: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406#962082 (kevinator) p:High>Normal [08:20:31] Analytics-Cluster, Analytics-Engineering: Researchers have page_id in X-Analytics field of webrequest logs - https://phabricator.wikimedia.org/T77416#962089 (kevinator) [08:27:39] Analytics-EventLogging, Analytics-Engineering: Client Engineer passes null to EventLogging::logEvent for optional boolean fields - https://phabricator.wikimedia.org/T78325#962093 (kevinator) [08:29:52] Analytics-Wikimetrics, Analytics-Engineering: Wikimetrics-l receives email about Lab’s Terms of Use - https://phabricator.wikimedia.org/T76108#962094 (kevinator) a:kevinator [08:34:47] Analytics-EventLogging, Analytics-Engineering: Create a new read-only permission group on vanadium for people to be able to access the original log and set the appropriate restrictions to make sure these users don’t perform computationally intensive operations - https://phabricator.wikimedia.org/T85027#962104 (... [08:37:01] Analytics-EventLogging, Analytics-Engineering: Create a new read-only permission group on vanadium for people to be able to access the original log and set the appropriate restrictions to make sure these users don’t perform computationally intensive operations - https://phabricator.wikimedia.org/T85027#962114 (... [08:43:35] Analytics-EventLogging, Analytics-Engineering: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#962119 (kevinator) [08:45:33] Analytics-EventLogging, Analytics-Engineering: Researchers access log of events failing validation - https://phabricator.wikimedia.org/T85028#962121 (kevinator) p:Triage>Normal [08:49:17] Analytics-Engineering, Analytics-Cluster, Analytics-Dashiki: Vital Signs user selects Pageviews using new definition - https://phabricator.wikimedia.org/T86141#962128 (kevinator) NEW [09:43:13] (CR) QChris: [C: 2 V: 2] "> I think it should be OK to self-merge changes of this type." [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182681 (owner: QChris) [09:44:01] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182682 (owner: QChris) [09:44:18] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182683 (https://phabricator.wikimedia.org/T76406) (owner: QChris) [09:44:39] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182684 (owner: QChris) [09:44:57] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182685 (owner: QChris) [09:45:14] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182686 (owner: QChris) [09:45:32] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182687 (owner: QChris) [09:45:51] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182688 (owner: QChris) [09:46:13] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/182689 (owner: QChris) [09:46:31] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/183150 (owner: QChris) [09:47:44] (CR) QChris: [C: 2] Remove debugging output from monitoring script [analytics/aggregator] - https://gerrit.wikimedia.org/r/183073 (owner: QChris) [09:47:54] (Merged) jenkins-bot: Remove debugging output from monitoring script [analytics/aggregator] - https://gerrit.wikimedia.org/r/183073 (owner: QChris) [09:51:43] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#962235 (Qgil) >>! In T1003#961937, @KartikMistry wrote: > Is it possible to get teamwise statistics? (eg: Language Engineering) (Just wishlist) :)... [11:00:04] Analytics-Tech-community-metrics: Graph waiting time for shell access requests - https://phabricator.wikimedia.org/T66702#962315 (Qgil) p:Triage>Low What is the motivation of this metric? If the problem is that requests are taking too long to be processed, maybe the first thing to do should be to look a... [11:10:06] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#962321 (Qgil) At this point, it would make more sense to take as reference Wikimedia Phabricator users instead of wikitech.wikimedia.org users... [11:12:01] Analytics-Tech-community-metrics: Tech contributors metrics show only bugs closed, not opened - https://phabricator.wikimedia.org/T56470#962322 (Qgil) Open>declined a:Qgil We are not going to touch Bugzilla metrics in korma at this point. Let's decline this task. Any request related to Maniphest shou... [11:21:48] Analytics-Tech-community-metrics: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#962335 (Qgil) p:Low>Normal What about this: * How many users have exercised +2 OR -2 in a given month. * Percentage of users that have exercised +2 OR -2 by affiliation... [11:25:06] Analytics-Tech-community-metrics: Contributor pages without data should include an explanation - https://phabricator.wikimedia.org/T58111#962341 (Qgil) p:Normal>Low [11:25:07] Analytics-Tech-community-metrics: "Age of unreviewed changesets by affiliation" shows negative number of changesets - https://phabricator.wikimedia.org/T72600#962342 (Qgil) p:High>Low [11:25:09] Analytics-Tech-community-metrics: Tech metrics should talk about "Affiliation" instead of organizations or companies - https://phabricator.wikimedia.org/T62091#962343 (Qgil) p:High>Low [11:25:10] Analytics-Tech-community-metrics: Graphs for median/average should report absolute numbers - https://phabricator.wikimedia.org/T68266#962344 (Qgil) p:High>Low [11:25:11] Analytics-Tech-community-metrics: "Volume of open changesets" graph should show reviews pending every month - https://phabricator.wikimedia.org/T72278#962345 (Qgil) p:High>Low [11:25:12] Analytics-Tech-community-metrics: Change "Date of submission of open changesets" to Date of upload - https://phabricator.wikimedia.org/T72650#962346 (Qgil) p:High>Low [11:25:15] Analytics-Tech-community-metrics: div.navbar-inner obscures content on Internet Explorer 10 - https://phabricator.wikimedia.org/T55635#962349 (Qgil) p:Normal>Low [11:25:16] Analytics-Tech-community-metrics: Guess the name from id (not from URL parameter) - https://phabricator.wikimedia.org/T65062#962347 (Qgil) p:Normal>Low [11:25:17] Analytics-Tech-community-metrics: Failed to load resource: the server responded with a status of 404 (Not Found) - https://phabricator.wikimedia.org/T65061#962348 (Qgil) p:Normal>Low [11:25:18] Analytics-Tech-community-metrics: Add i18n/l10n support - https://phabricator.wikimedia.org/T62073#962350 (Qgil) p:Normal>Low [11:25:20] Analytics-Tech-community-metrics: Unclear definition for "Aging charts" - https://phabricator.wikimedia.org/T62070#962351 (Qgil) p:Normal>Low [11:25:21] Analytics-Tech-community-metrics: MediaWiki.org stats should also consider discussion - https://phabricator.wikimedia.org/T62074#962353 (Qgil) p:Normal>Low [11:25:22] Analytics-Tech-community-metrics: Tech metrics missing IRC channels - https://phabricator.wikimedia.org/T56230#962352 (Qgil) p:Normal>Low [11:31:39] Analytics-Tech-community-metrics: List of Phabricator users - https://phabricator.wikimedia.org/T37508#962365 (Qgil) Open>Resolved a:Qgil I'm going to resolve this task, but let me explain. Initially this task was about offering a full list of Gerrit users. Gerrit didn't offer such feature, and it i... [11:31:56] Analytics-Tech-community-metrics, Phabricator.org: List of Phabricator users - https://phabricator.wikimedia.org/T37508#962369 (Qgil) [11:33:58] Analytics-Tech-community-metrics: Code review time must be on merged patches, not closed ones - https://phabricator.wikimedia.org/T68265#962372 (Qgil) p:Normal>Low [11:34:29] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#962373 (Qgil) p:Normal>Low [11:41:07] Analytics-Tech-community-metrics: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#962385 (Qgil) NEW [11:41:41] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#962385 (Qgil) [11:42:30] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#962385 (Qgil) [11:43:54] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#641085 (Qgil) [11:54:52] Analytics-Tech-community-metrics: Remove the filter for key Wikimedia software projects in korma.wmflabs.org - https://phabricator.wikimedia.org/T86154#962420 (Qgil) NEW [12:36:01] Analytics-Tech-community-metrics: Relating tech contributors with countries - https://phabricator.wikimedia.org/T57626#962532 (Qgil) Open>declined Declining myself. I don't think there is much point in polishing details about metrics by country before having solved {T60585}, and we don't have any plans... [12:36:53] Analytics-Tech-community-metrics: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#622887 (Qgil) [12:47:35] (PS2) QChris: Remove dia backup files upon 'make clean' [analytics/refinery] - https://gerrit.wikimedia.org/r/183166 [12:47:37] (PS3) QChris: Add Oozie setup to generate main webrequest tsvs [analytics/refinery] - https://gerrit.wikimedia.org/r/162589 [12:47:39] (PS1) QChris: Stop enforcing content file being 000000_0 for Oozie's archive workflow [analytics/refinery] - https://gerrit.wikimedia.org/r/183487 [12:54:34] (PS2) QChris: Stop enforcing content file being 000000_0 for Oozie's archive workflow [analytics/refinery] - https://gerrit.wikimedia.org/r/183487 [12:54:36] (PS4) QChris: Add Oozie setup to generate main webrequest tsvs [analytics/refinery] - https://gerrit.wikimedia.org/r/162589 [13:22:26] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#962609 (Krenair) This entire idea is flawed because it does not account for multiple affiliations (e.g. Independent and WMF/WMDE). [13:30:52] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#962629 (Qgil) The list of people is short, and we can look at the details when we have it. Potentially flawed data is still better than no data, imho. [13:31:26] (PS1) QChris: Restore refinery-tools' groupId [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183489 [13:32:40] (CR) QChris: "Do not take this version bumping as template for future" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/182949 (owner: Ottomata) [13:32:44] Analytics-Tech-community-metrics: Automate creating charts from Bugzilla Weekly Report - https://phabricator.wikimedia.org/T51744#962635 (Qgil) Open>declined a:Qgil [14:53:00] qchris_away: hi are you still away? :) [14:53:34] I am forever unable to find the log request format. I don't understand how people search wikis. I've searched everywhere and my head is going to explode [14:53:52] webrequest log i mean [14:56:56] https://wikitech.wikimedia.org/wiki/Cache_log_format [14:56:58] milimetric: ^ [14:57:19] :P [14:57:20] god [14:57:25] (btw. Google let me join the hangout without issues today :-D ) [14:57:28] yay! [14:57:33] i was just about to ask [15:08:57] waves folks in your hangout for me please :] [15:26:26] Analytics-Engineering, Analytics-EventLogging: Unit tests for Event Logging - https://phabricator.wikimedia.org/T86188#963055 (ggellerman) NEW [15:26:31] milimetric: whatever happened to moving wikimetrics to prod? [15:28:50] hashar: Argh. Did google not let you in? [15:29:50] qchris_meeting: I haven't attempted joining :D [15:30:27] Oh I see :-) [15:31:47] YuviPanda: we're going to do it. More important is that we get our data in a nice schema we can optimize [15:31:50] Ironholds: where is your pageview definition in wiki? ( in words for people to understand) [15:31:54] Ironholds: not code [15:32:01] because some of the metrics we're trying to compute basically never finish on labs [15:32:02] Analytics: measure how do autopromote values affect participation, vandalism, editor retention, etc. - https://phabricator.wikimedia.org/T86189#963072 (Amire80) NEW [15:32:20] nuria, https://meta.wikimedia.org/wiki/Research:Page_view although I need to tweak it [15:32:29] milimetric: :D I was asking because I noticed again how wikimetrics instance has puppet disabled forever [15:32:37] but YuviPanda: that's why I cleaned up the OAuth stuff before the break and we've got a couple more tasks to do to get it in prod [15:32:42] nice [15:32:55] yea [15:32:55] puppet messes it up, restarts things in non-friendly ways [15:33:15] (PS1) Ottomata: Fix for in refinery-core/pom.xml [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183509 [15:33:40] (CR) Ottomata: [C: 2 V: 2] Fix for in refinery-core/pom.xml [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183509 (owner: Ottomata) [15:33:52] milimetric: yeah, I would say that the puppet code is written wrong there (shouldn’t use puppet for deployment). Hopefully deployment will be handled separately in prod [15:34:31] we don't use puppet for deployment [15:34:39] except deployment of infrastructure changes, of course [15:35:48] halfak: in meetings for a while but if I don't ping you later, ping me after your lunch [15:35:55] i definitely want to talk about Samza :) [15:36:19] Will do :) [15:39:11] Analytics-Wikimetrics: Make wikimetrics role use hiera - https://phabricator.wikimedia.org/T86190#963086 (yuvipanda) NEW [15:51:50] milimetric: hello [15:51:57] hi kart_ [15:52:07] so do you have an idea of what SQL you want to run on each wiki? [15:52:14] you can test it out on labs, which has the same structure [15:52:19] Yep [15:52:37] milimetric: https://phabricator.wikimedia.org/T85106 [15:52:45] it is quite simple sql. [15:53:11] I've pythong script, but it will be great if any other alternative is available (simple shell script?) [15:55:10] kart_: yeah, cron can fire anything there. So I would suggest, write a script (python, shell, whatever you wish) that runs and outputs exactly what you need and emails you [15:55:26] ananthrk, ping [15:55:32] and of course the data is not available in labs, but just so you have that part done [15:57:09] kart_: just use python. [15:57:26] scripts of any size tend to grow bigger, and I’ve recently had to rewrite several bash scripts into python. [15:59:23] hello [15:59:36] @Ironholds [15:59:45] ananthrk_, are you the ananth of UDF renown? ;) [16:00:02] yup :) [16:00:10] YuviPanda: agree. [16:00:23] yay! [16:00:32] UDF renknown? [16:00:34] so, I'm Oliver. You may have heard about me, or, if you're really unlucky, seen my code [16:00:43] YuviPanda: milimetric do we have any similar example of quering DB scripts? [16:00:49] a toby-shaped person tells me you're working on the geocoding UDFs; is this the case? [16:00:56] and, if so: are you using MaxMind's V1 or V2 API? [16:01:19] am building off Andrew's code but am using V2 API [16:03:49] ananthrk_, awesome! Okay, that is the end of my questions; just wanted to check :D [16:04:23] ha..sure! will be around if you have more questions :) [16:06:25] oh, qchris_meeting, i didn't see you groupid fix [16:06:30] i just fixed it myself [16:12:00] (Abandoned) Ottomata: Restore refinery-tools' groupId [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183489 (owner: QChris) [16:45:33] Analytics-Engineering, Analytics-Cluster: Cluster monitoring: turn on duplicate monitoring of partitions - https://phabricator.wikimedia.org/T86197#963230 (ggellerman) NEW [16:46:36] Analytics: Geodata: replicate this in our home directory in the mean time - https://phabricator.wikimedia.org/T86199#963242 (ggellerman) NEW [16:47:31] Analytics-Engineering: Geodata: replicate this in our home directory in the mean time - https://phabricator.wikimedia.org/T86199#963242 (ggellerman) [16:49:04] Analytics-Engineering: Event Logging Monitoring - https://phabricator.wikimedia.org/T86200#963258 (ggellerman) NEW [16:49:39] milimetric: Thanks for the bug report, looks like wikitech isn't configured properly [16:49:55] np [16:50:52] kart_: sorry! sql examples running against event logging data are here: https://github.com/wikimedia/analytics-limn-mobile-data/tree/master/mobile [16:51:10] flow data here: https://github.com/wikimedia/analytics-limn-flow-data/tree/master/flow/ [16:51:32] kart_: and editing team: https://github.com/wikimedia/analytics-limn-edit-data/tree/master/edit [16:55:57] Analytics-Engineering: ImageView script: update and merge UDF - https://phabricator.wikimedia.org/T86202#963279 (ggellerman) NEW [16:56:40] Analytics-Engineering: ImageView script: Release the script but explain very clearly how the data is imprecise. (Let Erik or Dan write this explanation) - https://phabricator.wikimedia.org/T86203#963288 (ggellerman) NEW [16:57:20] Analytics-Engineering: ImageView script: dump the data - https://phabricator.wikimedia.org/T86204#963296 (ggellerman) NEW [16:57:30] (CR) Ottomata: [C: 2 V: 2] Remove dia backup files upon 'make clean' [analytics/refinery] - https://gerrit.wikimedia.org/r/183166 (owner: QChris) [16:58:15] Analytics-Engineering: udp2log: Announce new stream so people can compare streams - https://phabricator.wikimedia.org/T86205#963309 (ggellerman) NEW [16:59:32] ottomata1: I am not sure I understand the comment about a maven puppet module ... You mean creating something like "modules/cdh" for just the maven setup ... like "modules/maven"? [17:00:52] I mean ... it will be just two files. [17:00:59] The role, and the settings.xml. [17:01:23] milimetric: thanks! [17:02:09] kart_: ah, sorry, but those are just mostly raw SQL [17:02:32] so that doesn't help you much. But is this an accurate representation of what you need: [17:02:36] ottomata1: Ok, you convinced me by silence :-P [17:02:41] qchris: ha [17:02:46] yeah, modules/maven [17:02:59] bah, i dunno [17:03:02] ottomata: But I'll just put it in the main repo, not start a separate repo [17:03:05] milimetric: so, it is good if I go via 'long' way, creating similar for language team? [17:03:06] i mean, qchris, i am on the fence to [17:03:09] yesh, for sure [17:03:13] not a separate repo [17:03:17] use enwiki; run query1 run query2; use dewiki; run query1 run query2; etc. [17:03:20] milimetric: instead of writing cron? [17:03:31] kart_: what do you mean by long way? [17:03:33] qchris: just a role could be fine, but i'm not sure where to put it. [17:03:40] we could start with a role and make a module later if we need to [17:03:50] ottomata: k. i'll create the module after I got something to eat. [17:03:51] milimetric: same as flow and editing? [17:03:55] k [17:04:45] milimetric: if you can help, it won't be complex, it is just one query this time we want :) [17:04:45] kart_: you mean you want to use the same pipeline and have a dashboard like the teams I mentioned? [17:04:56] milimetric: yes. if doable. [17:05:05] like, kart_ here's an example dashboard: http://edit-reportcard.wmflabs.org/ [17:05:08] (within few days) [17:05:43] milimetric: looks fantastic. [17:06:34] kart_: but the difference for you is you need to run the query on multiple databases, right? [17:06:43] which databases exactly? [17:06:44] milimetric: yes. [17:06:45] all? some? [17:08:16] milimetric: 8 wikis, enwiki, cawiki, eswiki, ptwiki, dawiki, eowiki, idwiki, mswiki, sawiki [17:08:48] ok, cool. kart_ I'll set up that dashboard, it'll take me a few hours and I'm in meetings, but I'll ping you or the language team over email when it's done [17:09:13] once that's done, I'll have to think how to do the multi-database thing, but then I'll set it up [17:09:19] milimetric: do you need sql query to run? [17:09:35] so kart_ last thing: i know you lined me to your task but can you paste me the exact final query you need to run? It can change later, just so I know I have latest [17:09:40] milimetric: it is: at https://phabricator.wikimedia.org/T85106 [17:09:50] that exact sql there in the description? [17:10:02] select count(*) from user_properties where up_property = 'cx'; [17:10:03] select count(*) from user_properties where up_property = 'cx'; [17:10:14] ok, hm... wonder how that performs on enwiki :P [17:10:31] :) [17:10:52] * kart_ tries to find real name of milimetric ;) [17:11:00] Dan Andreescu at your service :) [17:11:14] hello Dan :) [17:11:47] milimetric: to note that, we've not yet deployed Content Translation on wikis yet. [17:12:03] We plan to do that on Monday the 12th. [17:14:11] milimetric: do you need anything else to start? [17:14:30] kart_: nope, all set - I just need to not be in meetings ;) [17:15:01] milimetric: I'll try to be awake for next couple of hours. Ping me/leave me msg after that. [17:15:14] (email if I'm dead :)) [17:15:23] no worries kart_ you can sleep, I will send out at least a status update later [17:15:44] milimetric: thanks a lot. [17:15:52] Analytics-EventLogging, Analytics-Engineering: Client Engineer passes null to EventLogging::logEvent for optional boolean fields - https://phabricator.wikimedia.org/T78325#963367 (Nuria) Is the validation you are referring to happening on the php client for EL? [17:20:42] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#963373 (Umherirrender) Under http://koti.kapsi.fi/~federico/crstats/core.txt is a list of actions, but having that under control of wmf is better for updates and tr... [17:22:47] hey qchris_away, lemme know when you are back [17:25:16] Analytics-EventLogging, Analytics-Engineering: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 points] - https://phabricator.wikimedia.org/T78325#963376 (ggellerman) [17:26:49] Analytics-EventLogging, Analytics-Engineering: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#842558 (ggellerman) [17:29:53] Analytics-EventLogging, Analytics-Engineering: Community Engineers have a working EventLogging dev server in vagrant - https://phabricator.wikimedia.org/T86067#963381 (Nuria) Traceback (most recent call last): File "./bin/eventlogging-devserver", line 72, in parser = eventlogging.LogParser('%q... [17:30:11] Analytics-EventLogging, Analytics-Engineering: Community Engineers have a working EventLogging dev server in vagrant - https://phabricator.wikimedia.org/T86067#963382 (Nuria) Trace above of error on devserver [17:31:43] Analytics-EventLogging, Analytics-Engineering: Community Engineers have a working EventLogging dev server in vagrant [8 pts] - https://phabricator.wikimedia.org/T86067#963389 (ggellerman) [17:47:26] (CR) Ottomata: [C: 2] "Cool." [analytics/refinery] - https://gerrit.wikimedia.org/r/183487 (owner: QChris) [17:48:50] Analytics-Engineering, Analytics-EventLogging: Investigate EventLogging Monitoring with Ops DBA - https://phabricator.wikimedia.org/T86200#963593 (kevinator) p:Triage>High [17:51:34] Analytics-Engineering, Analytics-Cluster: Analytics Eng has duplicate monitoring for partitions coming through Kafka - https://phabricator.wikimedia.org/T86197#963615 (kevinator) [17:51:41] stats.wikimedia seems to be stuck in october, no newer data available. Known issue? [17:52:14] Analytics-Engineering, Analytics-Cluster: Analytics Eng has duplicate monitoring for partitions coming through Kafka - https://phabricator.wikimedia.org/T86197#963230 (kevinator) p:Triage>High [17:52:19] Analytics-Engineering, Analytics-EventLogging: Story: Analytics Eng can monitor database replication lag - https://phabricator.wikimedia.org/T86136#963629 (kevinator) p:High>Normal [17:54:06] ottomata|lunch: lemme know when you are backkk [17:55:28] Analytics-Engineering: Analytics Eng has geodata scripts and cron job running elsewhere than a personal home directory - https://phabricator.wikimedia.org/T86199#963651 (kevinator) [17:55:44] Analytics-Engineering, Analytics-Cluster: Analytics Eng has geodata scripts and cron job running elsewhere than a personal home directory - https://phabricator.wikimedia.org/T86199#963242 (kevinator) p:Triage>High [17:56:13] Analytics-EventLogging, Analytics-Engineering: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#963659 (kevinator) p:Normal>High [18:02:15] mforns: where'd everyone go? [18:02:24] I thought we were just about to have a meeting [18:02:32] milimetric, hehe sorry I thought you heard Toby [18:02:36] nope [18:02:49] we are in the room that is in the calendar event [18:02:56] sigh [18:04:18] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#963724 (Dzahn) >>! In T1003#961920, @Qgil wrote: > The report send on January 1 looks good! Thank you very much. I have [[ https://lists.wikimedia.... [18:07:34] Analytics-Engineering: EPIC: Prepare and host Event Logging hackathon at MWDS - https://phabricator.wikimedia.org/T86212#963739 (ggellerman) NEW [18:14:52] sir ottomata|lunch, you called. [18:14:58] Oh. lunch :-D [18:18:38] (PS1) Ananthrk: [WIP] UDF to get country code from IP address UDF to determine client IP address given values from remote_addr and XFF headers Change-Id: I6dd39300173cd378b1836649f926b21ec27cc032 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 [18:33:50] ja, qchris, but i commented on change instead [18:34:05] the output format thing? [18:34:12] I am just replying to that. [18:34:39] ja [18:34:39] k [18:36:54] qchris: i added you as a reviewer to ananthrk's WIP patchset, feel free to ignore, or remove yourself as reviewer [18:37:05] k [18:43:54] hey nuria [18:43:58] just thinking [18:44:03] HMMMM [18:44:06] maybe i'm not thikning [18:44:21] ottomata: yess.... [18:44:37] ottomata: listening ...... [18:44:57] would it be better to inset the mobile app data into a hive table and then get the data out of? like we do for the webstats/pagecoutns-all-sites table? [18:44:58] not sure. [18:44:59] maybe not. [18:45:42] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#963841 (Dzahn) >>! In T1003#963724, @Dzahn wrote: > > Yes, let me try to make one for November manually. I will attach it here. I can give you th... [18:46:42] ottomata: but we do need it available to query right? [18:48:04] spose not [18:48:11] i guess you are right, it is very small, yes? [18:48:19] Analytics-Engineering, Analytics-Cluster: Mobile Product manager has daily App Uniques report generated using Hive [13 pts] - https://phabricator.wikimedia.org/T76534#963855 (ggellerman) [18:48:34] ottomata: then we might as well calculate it and drop the table, super samll [18:48:50] mmk [18:48:52] yeah nm then [18:49:57] ottomata: other thing [18:50:21] i was trying to use the workflow that moves things arround that christian did [18:50:27] nuria, why both temporary_destination_directory AND hive_query_destination_directory [18:50:28] ? [18:50:36] you set one to the other: [18:50:36] hive_query_destination_directory [18:50:36] javascript:; [18:50:36] ${temporary_destination_directory} [18:50:46] ah yes..? [18:50:54] ottomata: right taht is a mistake to correct [18:51:24] sooo...i was trying to use the workflow that moves things arround that christian did but that is centered arround having a SUCCESS file [18:52:04] ottomata: which is not the case here, we are storing teh query output [18:52:28] ottomata: in a temp dir and moving it later (no processing) to a different dir [18:54:05] hm [18:54:39] ottomata: makes sense? [18:55:00] * qchris sees that he got pinged, but qchris has no clue what you are talking about. If I can help, let me know. [18:56:13] qchris, here: https://gerrit.wikimedia.org/r/#/c/181017/11/oozie/mobile-apps/generate_daily_uniques/workflow.xml [18:56:23] in the move_apps_uniques_to_arrchive action [18:56:29] Ah. Some other change. I see. [18:56:33] Thanks for the context. [18:56:44] nuria is doing something similar to what your arvhive_job_output workflow does [18:56:48] i was suggesting that she use it instead [18:56:51] cool. [18:57:18] nuria, i don't see the depenence on the _SUCCESS flag in christian's stuff [18:57:22] milimetric: are we going to the meeting in few min? I noticed only I have responded yes. ;-) [18:57:36] oh done_file [18:57:36] hm [18:57:48] right right [18:57:49] hm [18:58:03] https://www.irccloud.com/pastebin/NR63126J [18:58:04] But a done flag can easily get generated. [18:58:11] ottomata: right, you just saw it [18:58:13] that's true. [18:58:33] oh I should've asked ottomata I guess. [18:58:45] ja nuria, just add a done-flag to your get_apps_uniques action, i thik [18:59:21] leila: sure [18:59:24] Analytics-Wikimetrics: Re-run Wikimetrics data once Labs issues are fixed [8 pts] - https://phabricator.wikimedia.org/T78305#963896 (kevinator) [18:59:29] i didn't reply? [18:59:30] ottomata: i can do that but note that then every hive query needs to output to a different dir [18:59:31] :( me [18:59:34] Analytics-Engineering, Analytics-Dashiki: Vital Signs user knows to click on metric title to open definition [3 pts] - https://phabricator.wikimedia.org/T76741#963897 (kevinator) [18:59:45] aren't you deleteing the dir every time anyway? [18:59:47] ottomata, nuria: /util/mark_directory_done/workflow.xml [19:00:05] ^ that is an abstraction for creating the done flag [19:01:13] ottomata: ok, can do that, the other things is this one: [19:02:14] source="${concat(concat(source_directory,'/000000_0'),expected_filename_ending eq 'EMPTY' ? '' : expected_filename_ending)}" [19:02:14] [19:03:00] so -in this case- file will be one and always called 000000_0 [19:03:15] ^ ottomata [19:04:09] nuria: https://gerrit.wikimedia.org/r/#/c/183487/ [19:04:54] (CR) QChris: "A separate output format would be cool." [analytics/refinery] - https://gerrit.wikimedia.org/r/183487 (owner: QChris) [19:05:01] ottomata: i see [19:05:36] very well, will try to use both workflows then, the 'mark as done' and 'move' [19:06:19] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#963915 (Aklapper) I doubt the November numbers because "Number of open and stalled tasks in total" is 18296 while it was 18125 in December. This re... [19:07:15] milimetric: should we deploy wikimetrics/dashiki ? [19:08:21] Analytics-Tech-community-metrics, Engineering-Community, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#963921 (Aklapper) >>! In T1003#961937, @KartikMistry wrote: > Is it possible to get teamwise statistics? (eg: Language Engineering) (Just wishlist)... [19:08:45] mforns: will try to get to the dashiki CR today [19:13:03] nuria, ok! but I think that given the bunch of things we (specially you) have, this can wait [19:42:58] Ironholds: Is there a quick way of getting "how many revisions have the tag 'visualeditor' across all wikis as of now"? [19:43:22] Ironholds: I can add up the displayed numbers on a dozen wikis by hand and guesstimate, but… [19:43:36] * James_F is assuming not. [19:45:40] James_F, I mean, a global query, but otherwise, no [19:45:46] if this is a dashboard you want, poke kevinator [19:45:56] Ironholds: No, I need it for a Lila talk. [19:47:06] Analytics-Tech-community-metrics: Graph waiting time for shell access requests - https://phabricator.wikimedia.org/T66702#964040 (scfc) The motivation of this metric is to see if there is a problem? [19:47:45] then...no [19:58:58] ottomata: is there a way that i can test sending jobs to the cluster with the hdfs user? does -Duser override user in cmd line? [20:01:24] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964075 (Jdforrester-WMF) That's a message from the #Analytics-EventLogging extension, because `ve.track` writes events to that. They don't have any im... [20:02:35] Ironholds: two gerrit things for you to look at regarding the whole en.m to enwiki mobile thing [20:02:37] https://gerrit.wikimedia.org/r/#/c/169974/7/python/refinery/projectcounts.py [20:02:59] milimetric, danke! [20:03:00] look at the "db name to wsc" conversion there towards the bottom [20:03:07] this is decidedly the simpler way to do the conversion [20:03:11] then, look at this: [20:03:14] https://gerrit.wikimedia.org/r/#/c/169974/2/python/refinery/projectcounts.py [20:03:17] Analytics-Engineering, Analytics-EventLogging: Analytics Eng team present at Wikimedia Developer Summit - https://phabricator.wikimedia.org/T86228#964078 (kevinator) NEW [20:03:30] that's the other way (called translate project and also lower on the page) [20:03:43] Ironholds: both of these are part of an abandoned gerrit change [20:03:45] thanks! [20:03:50] running to the shop but will review when I get back :) [20:04:04] Christian productionized it separately but used for the most part that first logic I mention [20:04:10] kk, no rush later [20:04:19] Analytics-Engineering, Analytics-EventLogging: Community Engineers have a working EventLogging dev server in vagrant [8 pts] - https://phabricator.wikimedia.org/T86067#964084 (mforns) a:mforns [20:05:42] YAY [20:05:44] R SWAG [20:16:06] Analytics-EventLogging, §analytics-sprint-2015-01-08: Analytics Eng team present at Wikimedia Developer Summit - https://phabricator.wikimedia.org/T86228#964111 (kevinator) [20:16:13] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964113 (Negative24) Can you post the messages here? I am getting some messages about unknown dependencies but I don't think it has anything to do with... [20:16:24] Analytics-EventLogging, §analytics-sprint-2015-01-08: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#964114 (kevinator) [20:16:58] Analytics-Engineering, Analytics-Wikimetrics, §analytics-sprint-2015-01-08: Eng has vetted data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T78019#964120 (kevinator) [20:17:17] Analytics-Engineering, Analytics-Cluster, §analytics-sprint-2015-01-08: Mobile Product manager has daily App Uniques report generated using Hive [13 pts] - https://phabricator.wikimedia.org/T76534#964121 (kevinator) [20:17:26] Analytics-Cluster, §analytics-sprint-2015-01-08: Mobile Product manager has daily App Uniques report generated using Hive [13 pts] - https://phabricator.wikimedia.org/T76534#802714 (kevinator) [20:17:40] Analytics-Wikimetrics, §analytics-sprint-2015-01-08: Eng has vetted data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T78019#834072 (kevinator) [20:18:07] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964124 (Krenair) > Uncaught Error: Unknown dependency: schema.Edit was the error caused by EventLogging. No others, VE loads fine for me. [20:18:55] Analytics-EventLogging, §analytics-sprint-2015-01-08: Analytics Eng team present at Wikimedia Developer Summit [34 pts] - https://phabricator.wikimedia.org/T86228#964135 (kevinator) [20:19:03] Analytics-EventLogging, §analytics-sprint-2015-01-08: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#964137 (kevinator) [20:19:13] Analytics-Wikimetrics, §analytics-sprint-2015-01-08: Eng has vetted data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T78019#964140 (kevinator) [20:19:35] Analytics-Cluster, §analytics-sprint-2015-01-08: Mobile Product manager has daily App Uniques report generated using Hive [13 pts] - https://phabricator.wikimedia.org/T76534#964142 (kevinator) [20:21:32] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964144 (Negative24) I'm getting that as well. I was told that it is safe to ignore those messages. Do you think this could be a problem with Parsoid? [20:27:10] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964160 (Jdforrester-WMF) >>! In T86112#964144, @Negative24 wrote: > I'm getting that as well. I was told that it is safe to ignore those messages. Do... [20:34:54] VisualEditor, Analytics-EventLogging: VisualEditor not loading unless edit page is refreshed with the cache cleared. - https://phabricator.wikimedia.org/T86112#964172 (Negative24) I'll try with a different browser when I get home. Jdforrester, you tested it on the Performance wiki? [21:02:04] i was about to release some aggregate data publicly [21:02:09] and figured I'd double check here [21:02:11] select count(*) from user_properties where up_property = 'cx'; [21:02:21] nuria, halfak, Ironholds ^ [21:02:23] what thinks you? [21:02:44] milimetric: on meeting give me 15 mins [21:05:51] what's cx? [21:06:06] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#964237 (Dzahn) As requested in comments above i have added wikitech-l to the recipients of this mail. More specifically wikitech-l is now one of th... [21:09:15] Analytics-Tech-community-metrics: Graph waiting time for shell access requests - https://phabricator.wikimedia.org/T66702#964241 (Qgil) Has anybody reported a problem? I mean, the shell request process is so linear and affects relatively few people. If there is a problem I guess someone will complain sooner o... [21:20:12] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#964263 (Dzahn) andre__ added the address as allowed sender to wikitech-l. That means the list should now receive stats starting on Feb 1st. [21:27:42] nuria, sorry, have been in meetings for a while [21:27:43] milimetric, I'm in a-batcave when you're ready [21:28:50] nruia, no, you can't send with hdfs user, you'd need some sudo properties. which maybe we can getcha... [21:29:00] especially with chri s tian leaving [21:30:02] (Abandoned) QChris: Drop doubled assignment in RunReports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/154801 (owner: QChris) [21:30:47] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#964292 (Qgil) Thank you very much! >>! In T1003#963841, @Dzahn wrote: > Number of accounts created in (2014-11): 299 > Number of active users (an... [21:45:15] ottomata1, can you lose like a codeword for the leaving? [21:45:24] like, "taking some time to smell the daisies", or something? [21:45:30] because every time I think about it it makes me sad ;p [21:53:10] milimetric: lemme know if you want to talk about anything [21:54:05] nuria: I just pasted that query above, if you or Ironholds could look at it and double check it wouldn't be bad to release the results for it publicly, that'd be great [21:56:21] (CR) Ottomata: [V: 2] "Psh, no way, less work! Create a table and specify location, insert into it, and then done." [analytics/refinery] - https://gerrit.wikimedia.org/r/183487 (owner: QChris) [21:58:45] ottomata1: about the output format ... how is an unneeded extra table (possibly a separate one for each tsv) less work than a CONCAT? [21:59:33] After all we want the data in a single file and control what the file looks like, [21:59:51] so we need to extract the data through Hive, needn't we? [22:00:34] if you could control the name of each partition file via an input format [22:00:50] then you wouldn' tneed your shell script, and you wouldn't need an extra 'archive' step [22:00:59] (well, maybe you would to have them all in the same directory) [22:01:27] The different files might have different compressions. [22:01:35] LIke the media files will be bz2 [22:01:41] while pagecounts are gz [22:01:50] and the projectcounts are uncompressed. [22:01:57] Is that possible within a single table? [22:02:10] And they also have different columns. [22:02:18] i think it is possible, but that's not what i'm suggesting [22:02:23] yes, you'd have difffernent tables for each one [22:02:36] Analytics-EventLogging, Analytics-Engineering: Teams needs to do perf testing of event logging to correctly estimate perf thresholds - https://phabricator.wikimedia.org/T86244#964389 (Nuria) NEW [22:02:40] but you'd know the name of the single file in each partitiong [22:02:42] partition [22:02:49] that would be predictable [22:03:07] Yes, sure. If we had a separate table for each tsv that we generate, it's possible. [22:03:16] But that's a real overhead, isn't it? [22:03:25] Analytics-EventLogging, Analytics-Engineering: Performance testing of event logging to correctly estimate perf thresholds - https://phabricator.wikimedia.org/T86244#964403 (Nuria) [22:03:39] more than the script and oozie action? dunno. i think writing this hypothetical input format would be more work for sure :) [22:03:45] *outputformat [22:03:58] :-D [22:04:09] especially since you already have a working shell script [22:07:02] ottomata: btw. Did you get a chance to look at the change that generates the tsvs? [22:07:19] Is it roughly ok, so I can prepare an alike change for the other tsvs? [22:07:46] https://gerrit.wikimedia.org/r/#/c/162589/ [22:08:03] milimetric: ok, got it [22:08:43] yes, qchris, in the middle of looking at them [22:08:52] ah. great. thanks. [22:08:53] sory was in 2.5 hours of meetings this after noon :) [22:09:02] no worries. [22:09:11] sorry for pressing on those changes. [22:11:15] Hey ottomata, how much trouble would it be to install "jq" on the hadoop nodes [22:11:16] ? [22:13:35] purty easy [22:13:44] gimme just a few [22:16:33] halfak: you want to use that for hadoop streaming, i assume? :) [22:21:03] qchris: i guess when we feel like the new refined webrequest table is more stable, we should use that for the tsv queries, right? [22:21:05] with tablesample? [22:21:12] Totally! [22:21:32] Should we ... Can we use it already? [22:21:45] Do you trust it enough, or should we wait a bit? [22:22:50] i think we should wait [22:22:53] or. [22:22:54] hm. [22:22:57] for these jobs... [22:23:04] haha, well, ihaven't checked on it today [22:23:12] :-) [22:23:23] But IIRC, it now has mobile and text. [22:23:30] sampled-1000 would also need upload. [22:23:45] ah [22:23:52] But that's just a matter of configuration. [22:24:08] (Except that upload has many faulty partitions these days) [22:24:59] yes. i will turn bits off tomorrow morning [22:25:05] yeah [22:25:06] well, hm [22:25:34] (CR) Ottomata: [C: 2] ":D" [analytics/refinery] - https://gerrit.wikimedia.org/r/162589 (owner: QChris) [22:25:41] (CR) Ottomata: [V: 2] ":D" [analytics/refinery] - https://gerrit.wikimedia.org/r/162589 (owner: QChris) [22:26:24] Yippie! [22:27:07] Would it be ok if I try deploying them? [22:27:23] do it! [22:27:31] mayb etry the mobile and zero ones on the refined table, eh? [22:27:58] I'll compare runtimes. [22:28:12] But by what you said, I am sure it'll be a win. [22:29:10] i think it will be faster, but since you are selecting pretty much every column, not a huge one. [22:29:16] not as huge anyway [22:29:25] tablesampel could help, but i haven't actually tested that... [22:29:59] I'll find out :-) [22:30:17] But looking at the clock ... maybe I'll deploy tomorrow. [22:32:37] :) [23:53:37] (PS1) Milimetric: Set up initial example query and config [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/183739 [23:53:51] (CR) Milimetric: [C: 2 V: 2] Set up initial example query and config [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/183739 (owner: Milimetric) [23:59:14] nuria, milimetric: are you guys still able to deploy wikimetrics today? [23:59:42] ah :( [23:59:46] i knew i was missing something [23:59:50] kevinator: i thought we were not doing it today for some reason, it was not on the calendar , but i can do it