[00:02:05] (CR) Milimetric: [C: -1] "this is exciting, couple of nits, I know you're still working on tests, and the new bower module. But I can't wait to refactor some of th" (8 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063) (owner: Nuria) [00:02:18] lol [00:02:38] goodness, tensions are so high, it's crazy [05:42:18] apergos: Could you rename me on OfficeWiki? (To my full name.) [05:42:39] "Olivneh" just looks and sounds horrible [05:44:12] Don't ask me why I asked here, I apparently can't operate an IRC client. [06:35:33] (PS1) Yuvipanda: Allow forking your own queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/273184 [06:41:02] (CR) Yuvipanda: [C: 2] Allow forking your own queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/273184 (owner: Yuvipanda) [06:41:29] (Merged) jenkins-bot: Allow forking your own queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/273184 (owner: Yuvipanda) [08:04:12] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2062643 (jcrespo) The purging has finished on the master, I... [09:07:56] Quarry: Allow users to choose free licenses other than CC0 - https://phabricator.wikimedia.org/T124528#2062669 (yuvipanda) Open>declined I don't really know the answer to the question of copy pasting stuff from fawiki to here. However, that problem will be present no matter what individual license is p... [09:43:39] elukey: Hi ! [09:49:43] joal: hellooooooo [09:49:49] Hey :) [09:50:08] I backloged yesterday's talk, and saw you might some more help with oozie :) [09:50:33] elukey: let me know if you want me to try to :) [09:51:13] ah you are always super helpful :) I will ping you later on during the day, I need to re-image hosts today to Debian :D [09:51:22] I thought that oozie was more.. intuitive [09:51:26] it is not :P [09:51:30] elukey: told you it was not ;) [09:51:42] let me know when you want :) [09:52:30] joal: oh yes I listened to your advices, but didn't fully get them until oozie told me "no!" several times in a row :P [09:52:56] I am not surprised, that's also why I'd prefer you to try ;) [09:53:13] Trying is usually a good way to understand :) [09:54:56] Good luck with debian images :) [09:56:44] looooong work, pooling/depooling them from memcached/redis pools is boooring [09:56:47] :) [09:56:55] Yeah, can understand that [09:57:08] Boring, but possibly less frustrating than oozie ;) [09:57:35] I have not seen many tools as frustrating as this one :) [10:17:20] Analytics-Tech-community-metrics, DevRel-February-2016: top-contributors.html empty due to 404s for several JSON files - https://phabricator.wikimedia.org/T126971#2062752 (Aklapper) Open>Resolved Confirming it is working now on http://korma.wmflabs.org/browser/top-contributors.html - thanks! Setting... [10:22:02] (PS1) Addshore: Add first build [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/273194 (https://phabricator.wikimedia.org/T127946) [10:23:04] (PS1) Addshore: Add README [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/273195 [10:23:36] (CR) Addshore: [C: 2 V: 2] Add first build [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/273194 (https://phabricator.wikimedia.org/T127946) (owner: Addshore) [10:23:53] (CR) Addshore: [C: 2 V: 2] Add README [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/273195 (owner: Addshore) [11:10:02] Analytics-Tech-community-metrics, DevRel-February-2016: Affiliations and country of resident should be visible in Korma's user profiles - https://phabricator.wikimedia.org/T112528#2062868 (Aklapper) This looks really good! Thank you a lot! [12:48:36] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2063000 (Dicortazar) @Aklapper, I just sent to you an email with... [13:02:31] (PS1) Addshore: Adjust for new version of phabricator [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/273207 [13:03:11] (PS2) Addshore: Adjust for new version of phabricator [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/273207 (https://phabricator.wikimedia.org/T127934) [13:04:02] (CR) Addshore: [C: 2 V: 2] Adjust for new version of phabricator [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/273207 (https://phabricator.wikimedia.org/T127934) (owner: Addshore) [13:06:38] (PS1) Addshore: Update for new repo name [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/273210 [13:06:54] (CR) Addshore: [C: 2 V: 2] Update for new repo name [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/273210 (owner: Addshore) [13:31:14] we can chat here if you like, hashar [13:36:12] was of no importance ;-] [13:36:58] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2063149 (Aklapper) a:Dicortazar>Aklapper Thanks! I'll take... [14:08:02] Analytics-Tech-community-metrics, DevRel-February-2016, developer-notice: Affiliations and country of resident should be visible in Korma's user profiles - https://phabricator.wikimedia.org/T112528#2063227 (Qgil) [14:29:30] ottomata: HELLOOOOOO :) [14:30:57] HlelLO [14:31:56] ottomata: if you want to look at some crazy stuff, I am investigating the pyspark issue [14:34:22] oh ja? [14:34:30] also, i see your patch about camus checker classpath [14:34:35] There are things that don't make sense ... [14:34:37] i had updated the spark-env.sh we use from puppet yesterday [14:34:43] and it now has SPARK_DIST_CLASSPATH [14:34:45] like you say [14:34:57] so it may be possible to use spark-env.sh instead of $(hadoop classpath | sed ...) [14:35:09] but yes, please! [14:35:14] tell me about pyspark and SPARK_HOME [14:35:21] i agree something is fishy [14:35:22] ottomata: it works with $(hadoop classpath | sed ...), then no change :) [14:35:32] hehe [14:35:44] ottomata: IRC or batcave? [14:35:49] batcave sure [15:44:10] did we ever solve this whole thing: [15:44:15] https://www.irccloud.com/pastebin/yfpUAdab/ [15:44:45] (the lots of INFO messages printing while we query) [15:53:30] hello! How are we going to do for the tasking mforns? [15:53:41] also milimetric :) [15:54:09] it sounds like a couple of us thought the research talk was important [15:54:18] so we'll work tasking around that [15:54:24] we'll do some before, and if needed some after [15:54:41] ah ok, all right :) [15:59:41] oh wait, a-team, the tasking today also conflicts with the flat-org reading meeting that both ottomata and I want to attend [16:00:47] I, absolving myself of any implicit authority I might have as a grumpy opinionated man, respectfully request that I be allowed to skip tasking today :) [16:01:17] * joal approves this message --^ [16:01:30] * milimetric supports joal for president [16:01:48] * elukey supports joal for president too [16:02:00] Analytics-Tech-community-metrics, DevRel-February-2016: "Tickets" (defunct Bugzilla) vs "Maniphest" sections on korma are confusing - https://phabricator.wikimedia.org/T106037#2063687 (Lcanasdiaz) Added to the side bar. {F3428283} I've also included this text on the [[ http://korma.wmflabs.org/browser/i... [16:02:06] * joal is not even candidate !!!! [16:02:40] :] [16:03:24] so are we rescheduling/skipping for today? [16:03:40] I am still fighting with Debian [16:03:42] :D [16:04:13] milimetric: postpotne meeting (to later time) or skipping today is fine, let's go over everybody's tasks after standup [16:04:31] milimetric: to make sure everyone has work. [16:04:50] :( but I have to skip standup too to make it to this meetingg [16:05:11] I feel bad because I don't even feel conflicted [16:05:31] milimetric: ok, I will go over with everybody else [16:05:36] I'm just 100% bought into the idea that the real pain is not here yet [16:05:47] and I want to do everything I can to protect myself and my team [16:06:38] * elukey supports milimetric as vice-president when joal is not available [16:06:47] :) [16:07:09] * joal supports milimetric for emperor !P [16:09:27] * milimetric checks to see if he has any clothes on [16:09:32] milimetric: one question if you have time [16:09:37] ofc [16:09:39] what's up nuria [16:09:55] milimetric: saw your comment here: https://gerrit.wikimedia.org/r/#/c/270867/5/src/app/apis/pageview-api.js [16:10:02] and i need to move some code arround [16:10:08] but one key diffrence now [16:10:38] is that the promise that fetches "agreggated" pageviews cannot be the same than the one that fetches "split" counts [16:10:49] as those are two different http requests [16:11:02] unlike before [16:11:17] not sure if this makes sense [16:11:27] cc milimetric [16:11:32] no that makes sense, nuria [16:11:38] I was trying to think how to help :) [16:11:49] it's true that now instead of 1 promise we have x promises [16:11:55] exactly [16:12:12] i can move things around to reduce duplication [16:12:20] but we can just have a promises = accessMethods.map(... transform each access method into a pageview api query) [16:12:31] then just do Promise.all regardless of promises.length [16:12:43] but you cannot reuse that promise for a different request [16:13:05] those internal promises aren't passed out or reused anyway [16:13:15] we create a deferred that we return from the outer function [16:13:26] these promises are inside the "done" of the config call [16:13:57] yes, but the deferred returned needs to be a different one depending on requests initiated [16:14:09] as otherwise it would have been "resolved" [16:14:15] for "all-access" [16:14:40] mmm, I don't think I see what you mean. so let's break it down for both cases [16:14:50] 1. access-method: all-access [16:15:15] make a generic deferred, start an async call to config, handled by a "done" callback, return the generic deferred [16:15:24] 2. access-method: [desktop, mobile-web] [16:15:41] make a generic deferred, start an async call to config, handled by a "done" callback, return the generic deferred [16:15:55] that's the outer structure that I'm saying should be shared [16:15:58] now the "done" callback [16:16:05] 1. access-method: all-access [16:16:44] promises = ['all-access'].map(... make a request with 'all-access' ...), Promises.all(promises, callback) [16:16:53] 2. access-method: [desktop, mobile-web] [16:17:14] promises = ['desktop', 'mobile-web'].map( ... same ... ), Promises.all(promises, callback) [16:17:33] so my point is, if the "showBreakdowns" parameter just gets renamed to accessMethods, we're good [16:18:28] right, ok, i think we are in agreement , there are still two [promises] on wikimetrics-visualizer but less code on api [16:20:42] oh nuria I get what you mean [16:20:46] but why not call var timeseriesData = _.flatten(arguments); in both cases? [16:20:54] that's the only difference and it does nothing in the no-breakdown case [16:21:22] or just flatten the results before resolving the generic deferred with them in the api [16:21:26] yes, that is fine, the api code can be refactor that way [16:21:47] but we will still have two different promises, otherwise you would be calling a promise that has alredy "resolved" [16:21:51] *already [16:22:07] two different promises on the caller code on visualizer [16:22:28] i will send code whenever i have a min today [16:22:45] sure, I think I'm missing the problem but I see kind of what you mean, code will help [16:23:02] Analytics-Wikistats: Wikisources Wikistats Inconsistencies - https://phabricator.wikimedia.org/T127657#2063756 (Zdzislaw) >>! In T127657#2061647, @ezachte wrote: > Here is today's list of countable namespaces, mostly from the API + presets mainly for wikisource, I wonder should we remove the presets? That wo... [16:31:44] a-team: standdup [16:42:24] ottomata: Tested on cluster using discovery code: job luanched (not yet finished, but at least started ! [16:42:54] Analytics-Tech-community-metrics: Add "Ticket Openers" to Korma's "Activity by contributors" - https://phabricator.wikimedia.org/T105634#2063831 (Lcanasdiaz) [16:42:57] Analytics-Tech-community-metrics, Phabricator: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#2063832 (Lcanasdiaz) [16:43:00] Analytics-Tech-community-metrics, DevRel-February-2016: "Tickets" (defunct Bugzilla) vs "Maniphest" sections on korma are confusing - https://phabricator.wikimedia.org/T106037#2063830 (Lcanasdiaz) Open>Resolved [16:44:55] mforns: do you mind to keep me in the loop for the puppet changes that you are making? Really interested :) [16:45:09] elukey, sure, will add you as a reviewer [16:45:22] if you want, we can also work together at some point [16:47:22] yep that would be great [16:47:36] I won't be as useful as ottomata though, I have to warn you :P [16:49:21] Analytics-Wikistats: Wikisources Wikistats Inconsistencies - https://phabricator.wikimedia.org/T127657#2049752 (Nemo_bis) > it should be ws,pl,0|104|100|102 That's what the content namespaces configuration thinks too: https://pl.wikisource.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces It seems... [16:54:06] Analytics-Wikistats: Wikistats doesn't yet know of all content namespaces on Wikisource - https://phabricator.wikimedia.org/T127657#2063850 (Nemo_bis) [16:54:46] elukey, let me know then when you'll have time today and I show you what I'm doing, sure you can help [16:55:34] elukey, although probably today you will be logging off soon [16:56:38] mforns: tomorrow would be better, is it ok for you? [16:56:51] elukey, sure [16:57:28] elukey, btw do you know how we test new puppet code? [16:58:09] analytics in particular no :) [16:58:38] I only know the self hosted puppet master instance with multiple roles [16:58:53] aha [16:59:42] ok elukey np, will ask the other yalls [17:00:36] mforns: I am curious to know how people test things, I am still puzzled [17:00:54] hehe [17:01:21] theoretically spinning up a puppet master containing your role in labs is not that overkill, but surely not easy too [17:02:49] elukey, what things? Puppet I don't know. For EventLogging, Wikimetrics, Dashiki, reportupdater, ... we have testing environments or analog things [17:03:05] ok, good idea [17:03:39] mforns: yes yes things == puppet changes sorry :) [17:03:54] aha, yes I also don't know [17:03:57] :] [17:16:21] ebernhardson: Good morning :) [17:16:35] Shall I kill my test job transfering to ES ? [17:16:39] ebernhardson: --^ [17:17:16] ebernhardson: I also have question about remaining oozie coordinators from before upgrade: shall we kill them ? [17:18:01] joal: yes i think i will need to kill all the coordinators and push a new version of our repository [17:18:17] ebernhardson: I do that now ? [17:18:19] i just started up a test run from your suggestions shipping to a plain http server on stat1002 and it looks to be working [17:18:22] sure [17:18:29] ok ! [17:19:01] although something else odd is happening, probably unrelated to all of this, but my http server on stat1002 is receiving requests at a *much* slower rate than i remember in prior tests [17:19:17] hm :( [17:21:47] ebernhardson: oozie job illed [17:23:24] joal: thanks [17:23:44] elukey: ya, i thought teh same thing like a 100 times [17:24:14] elukey: i think ottomata runs changes in his mind , cc madhuvishy ... how do you do in general to test puppet changes? [17:25:06] nuria: I use self hosted puppet masters in labs [17:25:33] madhuvishy: meaning that you get a labs machine with puppet git depot and run updates there [17:25:36] madhuvishy: correct? [17:25:41] elukey: yes [17:25:44] Uhh [17:25:48] nuria: [17:26:00] madhuvishy: as in change your code and run puppet there in teh labs machine [17:26:01] *the [17:27:01] nuria: yup. When you ask a labs instance to be self hosted, it'll clone a copy of ops/puppet in /var/lib/git . you can then pull your gerrit changes there and restart puppet to test [17:28:50] cc elukey so it looks like that is the only way [17:28:56] using labs [17:29:03] nuria: after creating a new instance and ensuring first puppet run, from wikitech, you should go to configure instance on wikitech and check the role::puppet:: self or sth like that which makes in self hosted. [17:29:08] madhuvishy: do you need to use heira and the wiki in some weird way? I remember that you told me something like this in SD [17:29:10] Yeah its not too hard. [17:29:11] *SF [17:29:34] elukey: labs lets you have hiera config in wiki pages [17:29:37] Yes [17:30:04] but only if I want to modify them, otherwise I can just merge my commit from gerrit and run puppet to see how it works [17:30:29] elukey: not merge but pull locally your commit to labs [17:31:03] elukey: yes [17:31:20] nuria: yes sorry, you're right :) [17:31:21] And yes no need to merge if you want to test first [17:31:31] DarTar: where's the meeting, how do I join? [17:31:39] Hiera:Analytics [17:31:46] On wikitech [17:31:48] elukey: k, cause if you self-merge puppet stuff ops ... ahem.. might not love you as much [17:32:11] Is the config page for analytics project. [17:32:31] ah, staff calendar [17:32:41] Analytics-Tech-community-metrics, DevRel-February-2016: "Tickets" (defunct Bugzilla) vs "Maniphest" sections on korma are confusing - https://phabricator.wikimedia.org/T106037#2063935 (Aklapper) I don't see that reflected yet in the side bar on korma.wmflabs.org but looking forward to it. Any related cod... [17:33:27] In general it can be, Hiera:LabsProject or Hiera:LabsProject/host/...eqiad.wmflabs if you want host specific [17:33:40] elukey: [17:34:00] milimetric, I can't find the staff calendar [17:34:02] madhuvishy: https://wikitech.wikimedia.org/wiki/Hiera:Analytics and here I can find all the beta hosts [17:34:20] Analytics, Analytics-EventLogging: Send raw server side events to Kafka using a PHP Kafka Client {oryx} - https://phabricator.wikimedia.org/T106257#2063936 (Nuria) This should not be needed if these set of chnages work: https://phabricator.wikimedia.org/T127209 [17:34:26] I know that your have already explained this to me, but it got flushed away [17:34:27] elukey: labs 😀 not beta [17:34:52] Beta cluster is the stuff in deploymentprep project [17:34:56] Analytics-Kanban: Add regexps that match the bots that follow the User-Agent policy {hawk} - https://phabricator.wikimedia.org/T125731#2063937 (Nuria) [17:35:20] Analytics-Kanban: Add regexps that match the bots that follow the User-Agent policy {hawk} - https://phabricator.wikimedia.org/T125731#1996003 (Nuria) Moving this task to kanban so it is up for grabs given that it is pretty small [17:35:27] madhuvishy: okok I was terrified for a second. My "beta" meant "testing-environment" [17:35:32] Aaah [17:35:37] He he [17:35:38] but yes I got your point, you're right :) [17:36:03] joal, did you manage to enter the research brownbag? [17:36:09] I did [17:36:13] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2063941 (Dicortazar) Work in progress. I've finally added the feature to the code at https://github.com/VizGrimoire/GrimoireLib/commit/9568489b5196fd6da4ae96d5b7d2... [17:36:19] a lot of people mforns [17:36:29] joal, I can't find the staff calendar [17:36:51] mforns: https://plus.google.com/hangouts/_/wikimedia.org/research-group?authuser=0&hceid=bGVpbGFAd2lraW1lZGlhLm9yZw.4b3fnafnendfk83t8ilan6mf98 [17:36:52] mforns: https://plus.google.com/hangouts/_/wikimedia.org/research-group?authuser=0&hceid=bGVpbGFAd2lraW1lZGlhLm9yZw.4b3fnafnendfk83t8ilan6mf98 [17:36:56] beat you! ha [17:37:01] hehehe [17:37:18] :) [17:37:26] Analytics-Kanban: Parse User-Agent strings with OS like "Windows 7" correctly into the user agent map {hawk} - https://phabricator.wikimedia.org/T127324#2063947 (Nuria) [17:37:46] mforns: I just go to someone who might be attending like Dario's calendar to figure it out. I cannot find staff calendar either :P [17:37:48] Analytics-Kanban: Parse User-Agent strings with OS like "Windows 7" correctly into the user agent map {hawk} - https://phabricator.wikimedia.org/T127324#2039833 (Nuria) Moving to kanban so this task is up for grabs as it is not so big. [17:39:07] a-team: added couple tasks to kanban just in case anyone finds themselves with some spare time: both are pretty small but they need through testing: https://phabricator.wikimedia.org/tag/analytics-kanban/ [17:39:21] ok nuria [17:39:36] thanks madhuvishy I'm in [17:39:40] are people trying to join the hangout for the privacy talk? we're close to the limit [17:51:47] Analytics-Wikistats: Wikistats doesn't yet know of all content namespaces on Wikisource - https://phabricator.wikimedia.org/T127657#2063994 (Zdzislaw) Yes, @Nemo_bis is right! "presets" values are wrong and should be removed for all ws; it is required to use the API for all wikisource. Author, Page and Inde... [18:04:48] have you guys seen http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines ? [18:04:54] --^ joal ottomata [18:05:15] hm, no haven't seen this one yet elukey [18:05:41] Schema management: The ability of the data pipeline to carry schema information where it is available. In the absence of this capability, you end up having to recreate it downstream. Furthermore, if there are multiple consumers for the same data, then each consumer has to recreate it. We will cover the various nuances of schema management for data pipelines in a future blog post. [18:05:55] it remembers me of something :) [18:07:07] also http://docs.confluent.io/2.0.0/connect/connect-hdfs/docs/index.html [18:13:25] Analytics-Wikistats: Wikistats doesn't yet know of all content namespaces on Wikisource - https://phabricator.wikimedia.org/T127657#2064138 (ezachte) Yes, that's why I asked: "I wonder should we remove the presets? That would require the API to be up date for all wikisource wikis. Is that the case?" So ap... [18:28:53] (PS1) Milimetric: Update for February Meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/273287 [18:29:02] (CR) Milimetric: [C: 2 V: 2] Update for February Meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/273287 (owner: Milimetric) [18:42:49] nuria: looks like pageview on vital-signs are not updated since feb 3rd :( [18:43:10] nuria: can you do something around that ? [18:43:16] cd [18:43:21] oops [18:43:25] joal: sure [18:43:45] joal: but last i looked they were updating. let me look again [18:44:15] nuria: data is updated in files (just double checked [18:44:24] But not on chart (maybe caching issue for me ?) [18:45:52] joal: no, taht would not be it [18:46:03] joal: maybe the cron i added is been removed? [18:46:07] nuria: yeah, tried on a new session, still not updated [18:46:13] ?:S [18:47:59] joal: mmm.. cron has updated data: [18:48:04] https://www.irccloud.com/pastebin/nBiQoYy0/ [18:48:32] hm ... weird ! [18:51:08] joal: and files on wikimetrics instance are fine: [18:51:11] https://www.irccloud.com/pastebin/2WisKJRd/ [18:52:32] joal: so these files are being served from elsewhere [18:52:48] nuria: on stat1002 we have up to 2016-02-24, but 02-22 is good emough :) [18:53:02] joal: ya that is fine [18:53:21] But not updated in charts ... WEIRD !!! [18:57:02] joal: no, it makes sense cause this [18:57:05] https://www.irccloud.com/pastebin/YkClO6IP/ [18:57:09] is wrong. [19:00:20] madhuvishy: how can i know whether the dns metrics.wmflabs.org has been moved to a different machine? [19:03:51] nuria: [19:03:56] joal: it is a caching problem elsewhere, see commit [19:03:57] wikimetrics1 is the wrong instance [19:04:10] wikimetrics-01.wikimetrics [19:04:18] are you using this? [19:04:40] madhuvishy: aham, did we changed that with puppet refactor? [19:04:51] nuria: yes! we have the new wikimetrics project [19:05:23] Analytics-Wikistats: Wikistats doesn't yet know of all content namespaces on Wikisource - https://phabricator.wikimedia.org/T127657#2064381 (Zdzislaw) >>! In T127657#2064138, @ezachte wrote: > Yes, that's why I asked: > > "I wonder should we remove the presets? That would require the API to be up date for... [19:06:04] so ssh is wikimetrics-01.wikimetrics.wmflabs? [19:06:34] nuria: wikimetrics-01.eqiad.wmflabs or wikimetrics-01.wikimetrics.eqiad.wmflabs [19:13:00] madhuvishy: k, will fix cron trick [19:13:15] nuria: you're close to merging your thing no [19:13:17] Analytics: Remove cron on wikimetrics instance that updates vital signs - https://phabricator.wikimedia.org/T125751#2064417 (Nuria) Wikimetrics was moved to wikimetrics-01.eqiad.wmflabs or wikimetrics-01.wikimetrics.eqiad.wmflabs [19:13:28] may be we dont need the cron? [19:13:34] if it was never there [19:13:36] madhuvishy: not today probably [19:13:44] broke piwik?! :) [19:13:44] okay [19:13:55] :D [19:14:01] ottomata: yes [19:14:05] nuria: [19:14:08] oops sorry [19:14:12] ottomata: we had to remove SEO traffic [19:14:19] ottomata: and it was down for a bit too [19:14:27] heheh [19:24:14] madhuvishy: i left it like the other instance (i.e. with cron) things should be updating now [19:24:33] joal: vital signs updated [19:24:41] Thanks a lot nuria ! [19:25:35] hmm nuria, not yet for me :( [19:25:42] probably caching issues still [19:25:57] joal: now is caching on your end, [19:26:06] ok [19:26:06] joal: on your browser [19:26:13] joal: just clear cache [19:26:31] joal: 1) open developer tools/ go to network tab (bottom) [19:26:37] 2) untick cache [19:26:59] Tried with a clear sessionYay [19:27:12] Thanks ! [19:28:22] madhuvishy: is there a plavce to look at dns domains versus physical instances? [19:28:31] nuria: [19:28:34] ummm [19:28:40] *place [19:28:51] i dont know a super cool way - but i'd look it wikitech - Manage Web proxies [19:29:12] k [19:29:23] just to make sure no new coolness had been added [19:29:58] nuria: :) well it's not apache on 80 anymore - uwsgi server is serving at 8080 [19:30:01] Analytics-Tech-community-metrics, DevRel-February-2016: Many profiles on profile.html do not display identity's name though data is available - https://phabricator.wikimedia.org/T117871#2064483 (Lcanasdiaz) The way this data is calculated is the following: # we get the people listed in the different to... [19:30:17] may be i'll update all the docs in the wiki/write up some new stuff sometime [19:45:22] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2064505 (jcrespo) Lag should be ok now. I just need a last... [19:45:49] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, WMF-deploy-2016-02-09_(1.27.0-wmf.13): Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#2064507 (jcrespo) a:jcrespo>None [19:53:11] logging off a-team! talk with you tomorrow :) [19:53:18] bye elukey! [19:53:28] byee [19:56:44] Bye ! [20:01:08] joal: you shoud probably delete those python zip files ja? [20:01:10] that you put in HDFS? [20:01:13] so it doesn't bite us later? [20:01:19] (maybe you already did...) [20:01:23] ottomata: for sure ! [20:01:37] ottomata: doing now [20:02:14] ottomata: done ! [20:03:17] danke [20:06:53] nuria: got some interesting results on uniques randomness [20:07:02] Ok to review that tomorrow ? [20:07:46] Analytics, Analytics-Kanban, HyperSwitch, RESTBase, and 2 others: Separate AQS off of RESTBase - https://phabricator.wikimedia.org/T126294#2064547 (Ottomata) [20:08:02] ottomata, o/ qq: how can I test new puppet code? [20:08:24] mforns: :) [20:08:31] many ways! mw-vagrant? labs? [20:08:38] labs self hosted puppetmaster? [20:08:47] aha [20:08:50] Live in prod! [20:09:02] puppet compiler compiler [20:09:09] :] [20:09:43] mforns: ja this can help [20:09:43] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build?delay=0sec [20:09:51] but its usually better for testing changes [20:09:54] rather than new things [20:10:05] like, if you want to be careful you aren't breaking something [20:10:06] ottomata, aha [20:10:15] it'll show you the change catalog [20:10:19] but its hard to read for lots of changes [20:10:20] just a big diff [20:10:36] aha [20:11:36] mw-vagrant would be a good option, but my installation is broken, the last 2 times this happened, it took me hours to fix... [20:13:55] thx ottomata [20:19:02] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2064595 (Lcanasdiaz) Guys, we deployed a simple visualization. To be honest I'm not pretty sure if it is worth to create other kind of tables as we are speeding up... [20:22:51] mforns: hh, yeah [20:23:00] i reinstalled a bunch of stuff and got mine working a week or two ago [20:41:52] joal: yt? [20:42:02] joal: can talk about results [20:42:18] joal:tomorrow works too [21:08:13] Analytics-Tech-community-metrics, DevRel-February-2016: "Tickets" (defunct Bugzilla) vs "Maniphest" sections on korma are confusing - https://phabricator.wikimedia.org/T106037#2064746 (Lcanasdiaz) >>! In T106037#2063935, @Aklapper wrote: > I don't see that reflected yet in the side bar on korma.wmflabs.or... [21:31:48] Analytics-Wikistats: Wikistats doesn't yet know of all content namespaces on Wikisource - https://phabricator.wikimedia.org/T127657#2064817 (Nemo_bis) p:Triage>Normal [21:35:16] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064831 (Danielsberger) [21:43:39] (PS6) Nuria: Fetch Pageview Data from Pageview API [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063) [21:45:48] (CR) Nuria: "Added bower package and removed a bunch of redundant code. Please take a look, now only tests are pending." (3 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063) (owner: Nuria) [21:51:08] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064831 (kevinator) @Danielsberger How big does the data set need to be? One hour? One day? It gets big very quickly. [22:05:24] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064831 (Nuria) For privacy this data needs to be sampled and also timestamps cannot be disclosed as they are but rather as increments from a point in the past. [22:07:38] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064967 (Danielsberger) The 2007 dataset covers a large time span: September 19th 2007 until January 2nd 2008. With on average 2GB of logs per day, that's about 250 GB overall. I understand that... [22:08:41] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064971 (Danielsberger) An increment counter was also used in the 2007 dataset. Seems like a good solution to me. [22:09:15] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2064973 (Nuria) @Danielsberger : your best bet is to restrict the dataset to a project maybe? We get >100.000 res per sec [22:19:00] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2065051 (Danielsberger) The 2007 dataset needs roughly 20 Bytes per request with gzip compression (112 Bytes per request without compression). If we add little more information or find a good rep... [22:22:26] Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#2065056 (Ottomata) Open>Resolved This works on terbium now: ``` python /srv/deployment/eventlogging/eventlogging/bin/eventlogging-consumer 'kaf... [22:28:14] Analytics, Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Set up Webrequest -> kafka flow in beta. - https://phabricator.wikimedia.org/T127369#2065072 (Ottomata) wmf_raw.webrequest and wmf.webrequest table partitions are created by load and refine jobs. [23:02:43] Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2065154 (Dicortazar) Just to mention that this contains the top 10 oldest changesets per organization for all of them. We can use another limit if you prefer to ha... [23:14:20] (CR) Milimetric: "structure is as clean as possible, I think. Awesome. Few more nits but up to you, and I'll wait for the tests then merge." (6 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063) (owner: Nuria)