[01:16:12] Analytics, Patch-For-Review: Remove LegacyPageviews from vital-signs - https://phabricator.wikimedia.org/T124244#1954392 (Nuria) >Disable the dashboard on Vital Signs Done >Stop the aggregator from generating data and archive/deprecate the source code repo. The code that generates the new and old pagevie... [01:16:30] Analytics-Kanban, Patch-For-Review: Remove LegacyPageviews from vital-signs - https://phabricator.wikimedia.org/T124244#1954393 (Nuria) [01:16:36] ottomata: https://github.com/yuvipanda/ldapauthenticator finished the ldap authenticator :) [01:16:50] I can hopefully move it to the jupyter organizaiton at some point [01:30:46] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1954432 (Legoktm) @Sadads: @Krenair brought up the fact that the current subdomain splitting p... [02:55:05] Analytics, Editing-Analysis: Data on the edit analysis dashboard stops in December 2015 - https://phabricator.wikimedia.org/T124383#1954533 (Neil_P._Quinn_WMF) NEW [02:55:42] Analytics, Editing-Analysis: Queries for the edit analysis dashboard failing since December 2015 - https://phabricator.wikimedia.org/T124383#1954542 (Neil_P._Quinn_WMF) [02:56:33] Analytics, Editing-Analysis: Queries for the edit analysis dashboard failing since December 2015 - https://phabricator.wikimedia.org/T124383#1954533 (Neil_P._Quinn_WMF) @Milimetric, I'd be happy to do some joint investigation with you if you're in the mood to transfer knowledge! [06:09:21] Analytics-Kanban, operations, HTTPS, Patch-For-Review: EventLogging sees too few distinct client IPs {oryx} [8 pts] - https://phabricator.wikimedia.org/T119144#1954687 (leila) @Ottomata I checked couple of tables that I knew and the diversity of hashed IPs looks healthy. I also looked at two of Tilma... [06:49:10] Analytics, Editing-Analysis: Queries for the edit analysis dashboard failing since December 2015 - https://phabricator.wikimedia.org/T124383#1954735 (Neil_P._Quinn_WMF) Interesting...it's now on to the same query for 5 December. So conceivably the queries aren't failing, they're just really far behind (an... [06:49:19] Analytics, Editing-Analysis: Queries for the edit analysis dashboard failing since December 2015 - https://phabricator.wikimedia.org/T124383#1954736 (Neil_P._Quinn_WMF) [07:29:00] Analytics-Tech-community-metrics, DevRel-January-2016: Legend for "review time for reviewers" and other strings on repository.html - https://phabricator.wikimedia.org/T103469#1954784 (Lcanasdiaz) We've just included a new widget in order to show the description name, it is called MetricDesc. See: * http:... [07:29:15] Analytics-Tech-community-metrics, DevRel-January-2016: Legend for "review time for reviewers" and other strings on repository.html - https://phabricator.wikimedia.org/T103469#1954785 (Lcanasdiaz) Open>Resolved [09:10:56] Analytics-Tech-community-metrics, DevRel-January-2016: Make GrimoireLib display *one* consistent name for one user, plus the *current* affiliation of a user - https://phabricator.wikimedia.org/T118169#1954889 (Aklapper) [09:10:58] Analytics-Tech-community-metrics: top-contributors should have real names for the main contributors - https://phabricator.wikimedia.org/T124346#1954888 (Aklapper) [09:11:23] Analytics-Tech-community-metrics: top-contributors should have real names for the main contributors - https://phabricator.wikimedia.org/T124346#1954891 (Aklapper) p:Triage>Lowest Requires fixing T118169 for good and then some manual editing of the `name` parameter of such accounts... [09:30:53] Analytics-Tech-community-metrics, DevRel-January-2016: Many profiles on profile.html do not display identity's name though data is available - https://phabricator.wikimedia.org/T117871#1954918 (Aklapper) >>! In T117871#1953215, @jayvdb wrote: > There are many cases where this is not happening. @jayvdb: C... [09:31:40] YuviPanda: ldapauthenticator means jupyter for stat100X ? [09:34:09] joal: o/ [09:34:26] Hi elukey [09:38:30] I am drawing a graph for Event Loggin [09:38:33] *Logging [09:38:41] to put in the oncall docs [09:39:05] Great elukey, it will help have a good visualisation of the streams / workers etc [09:39:09] :) [09:40:31] I am sure that I'll miss something :P [09:41:24] elukey: If you miss only one thing in EL after 3 weeks at the foundation, you're definitely good :D [09:50:22] joal: https://wikitech.wikimedia.org/wiki/File:EventLogging.png (first draft) [09:52:05] * joal reads [09:53:16] elukey: some comments [09:53:40] hm, maybe in batcave, to be faster ? [09:53:54] if you have time sure! [09:54:12] I always take time when somebody asks me to review good documentation :) [09:55:09] cave ? [09:55:15] yep joinging [09:55:18] *joining [09:56:17] Analytics-Tech-community-metrics, DevRel-February-2016: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#1954941 (Aklapper) [09:56:28] Analytics-Tech-community-metrics, DevRel-February-2016: Mailing lists recently added to korma do not have "Top senders" data created (JSON file is 404) - https://phabricator.wikimedia.org/T123929#1954942 (Aklapper) [10:03:30] Analytics-Tech-community-metrics, DevRel-January-2016: "Age of open changesets by Affiliation" has some "NaN" values - https://phabricator.wikimedia.org/T110875#1954943 (Dicortazar) Open>Resolved [10:04:13] Analytics-Tech-community-metrics, DevRel-January-2016: "Age of open changesets by Affiliation" has some "NaN" values - https://phabricator.wikimedia.org/T110875#1588704 (Dicortazar) Kudos to @Lcanasdiaz ! [10:07:00] ah snap I forgot about the main one drawn by ottomata :( [10:12:58] Analytics-Tech-community-metrics, pywikibot-core, DevRel-January-2016: Statistics for SCM project 'core' mix pywikibot/core, mediawiki/core and oojs/core - https://phabricator.wikimedia.org/T123808#1954971 (Aklapper) a:Dicortazar [10:26:46] Analytics-Tech-community-metrics: "Last 30 days" stats for specific mailing list display an account as one list item per username character - https://phabricator.wikimedia.org/T123927#1954992 (Dicortazar) [10:29:31] Analytics-Tech-community-metrics, DevRel-January-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#1954993 (Dicortazar) p:Low>High [10:43:10] Analytics-Tech-community-metrics, DevRel-February-2016: Mailing lists recently added to korma do not have "Top senders" data created (JSON file is 404) - https://phabricator.wikimedia.org/T123929#1955001 (Aklapper) a:Lcanasdiaz [10:44:48] Analytics-Tech-community-metrics, DevRel-February-2016: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#1955002 (Aklapper) a:Dicortazar [10:44:50] Analytics-Tech-community-metrics, DevRel-February-2016: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#1955003 (Dicortazar) [10:49:17] Analytics-Tech-community-metrics, DevRel-March-2016, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#1955010 (Aklapper) [10:52:26] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, DevRel-March-2016: Who are the top 50 independent contributors and what do they need from the WMF? - https://phabricator.wikimedia.org/T85600#1955013 (Aklapper) [10:53:10] Analytics-Tech-community-metrics, Developer-Relations, DevRel-March-2016, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#1955014 (Aklapper) [10:54:43] Analytics-Tech-community-metrics, DevRel-January-2016: Time axis on repository.html only displays two months, repeated several items - https://phabricator.wikimedia.org/T115872#1955016 (Aklapper) Problem with the library. It needs several points, if you have less, you have this faulty chart. "If there is... [10:55:17] Analytics-Tech-community-metrics, DevRel-January-2016: Contributor pages which show user name but not any other data should include an explanation - https://phabricator.wikimedia.org/T58111#1955018 (Aklapper) These might come from before using SortingHat. (Only potentially related: T112528) [10:56:56] Analytics-Tech-community-metrics, DevRel-January-2016: Affiliations and country of resident should be visible in Korma's user profiles - https://phabricator.wikimedia.org/T112528#1955020 (Aklapper) p:Low>Normal [10:58:04] Analytics-Tech-community-metrics, DevRel-February-2016: Data in korma project pages has confusing labels, is difficult to understand - https://phabricator.wikimedia.org/T110524#1955023 (Aklapper) p:Normal>Low a:Aklapper [11:01:59] joal: do you remember what triggered https://phabricator.wikimedia.org/T123634 ? Did we receive an alarm or was it somebody checking graphs? [11:02:31] elukey: I think it came from somebody saying that there were lag [11:02:51] elukey: IRRC there have been a task created to actually create an alert based on replication lag [11:03:07] ah it might have been also assigned to me, let me check [11:03:09] elukey: And I actualy think it's been assigned to you ;) [11:03:12] :) [11:03:32] exactly: https://phabricator.wikimedia.org/T124306 [11:03:33] hahaha [11:03:45] I am still not used to phab notifications [11:04:25] :) [11:08:47] all right updated https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/Oncall and consolidated all the howtos [11:11:19] Ahhh here's the explanation for the custom replication [11:11:21] https://phabricator.wikimedia.org/T124307 [11:11:28] let me add it to the docs [11:11:46] Arf elukey, another thing I just noticed: The errors event are imported into hadoop, and they probably shouldn't be on the same arrow as valid ones :) [11:14:26] you are completely right, fixing it.. I'll also add a note about the custom replication [12:11:03] Hi! Could anyone please review this patch for me? https://gerrit.wikimedia.org/r/#/c/263911/ Thanks! [12:27:16] anmol_wassan, hi! is there any Phabricator task related to this patch? [12:37:40] Yes there is! https://phabricator.wikimedia.org/T76914 [12:38:09] mforns: You might want to have a look. :) [12:38:43] anmol_wassan, thanks! [12:38:57] No problem. [13:02:59] Analytics-Tech-community-metrics, DevRel-January-2016, Patch-For-Review: gerrit_review_queue's "waiting for review" column name misleading (also includes unmerged CR+1 patches) - https://phabricator.wikimedia.org/T121495#1955206 (Aklapper) https://github.com/Bitergia/mediawiki-dashboard/pull/78 merged;... [13:26:04] Analytics-Tech-community-metrics, DevRel-January-2016: Many profiles on profile.html do not display identity's name though data is available - https://phabricator.wikimedia.org/T117871#1955255 (Aklapper) Ah. Going to http://korma.wmflabs.org/browser/mls.html , under "Last 30 days" I see "Phil" listed as #... [13:33:02] Analytics-Kanban: Polish script that checks eventlogging lag to use it for alarming - https://phabricator.wikimedia.org/T124306#1955259 (elukey) I took a look to the script and it would be really great to push metrics to statsd about the lag observed for each table. After that alarming with graphite/icinga s... [13:37:14] Analytics-Kanban: Polish script that checks eventlogging lag to use it for alarming - https://phabricator.wikimedia.org/T124306#1955260 (Reedy) [15:19:11] Analytics-Tech-community-metrics, pywikibot-core, DevRel-January-2016: Statistics for SCM project 'core' mix pywikibot/core, mediawiki/core and oojs/core - https://phabricator.wikimedia.org/T123808#1955474 (Lcanasdiaz) This is a bug of the UI and the way we should the repositories. I'm working on it n... [15:29:11] milimetric: I'm seeing new data, woo [15:29:20] I think uploads isn't being updated but that's OK [15:29:50] hm, weird, i'll check the logs [15:29:56] but yeah, it might take a while to backfill all the way [15:30:04] but then going forward shouldn't be a problem since it's monthly [15:30:09] Right [15:30:19] I'm thinking I might add daily ones, but that's OK [15:30:36] I'll probably not do them since the beginning of time [15:32:30] (PS2) Milimetric: Track cross-wiki notifications beta feature [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/264700 (owner: Catrope) [15:33:09] (CR) Milimetric: [C: 2 V: 2] Track cross-wiki notifications beta feature [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/264700 (owner: Catrope) [15:36:29] MarkTraceur: no errors, just taking a long time I think [15:36:35] OK then [15:36:41] I thought uploads was one of the faster ones actually [15:40:13] joal: I bumped the compression block size for the per-article table in aqs slightly from 128 to 256k [15:40:32] hey gwicke [15:40:53] this is supposed to compress more at the expense of more CPU work, correct ? [15:41:02] yeah [15:41:24] with rotating disks the main cost is seeks, and this might slightly reduce those as well [15:41:26] And if I assume correctly, reduce the number of SStables [15:41:38] makes sense [15:42:01] now gwicke, this change will be applied only to newly (re)compressed data, correct ? [15:42:22] currently ratio is 8% of input, so at 128k each read is for only ~10k of compressed data [15:42:58] yes, it's only applied to newly compressed sstables [15:43:03] so the effect will be very gradual [15:43:07] sure [15:46:09] gwicke: it's reaaly interesting how how number of SStables per read seems strongly explaining latency [15:46:33] And I forgot the principal gwicke : Thanks a lot :) [15:47:02] yw! [15:47:41] and yes, especially in a seek-limited environment touching many sstables is a bad idea [15:48:45] especially if they are large & not in page cache [15:49:20] makes complete sense [15:56:04] Analytics, Analytics-Cluster: Create and maintain an Analytics Cluster in Beta Cluster in labs. - https://phabricator.wikimedia.org/T109859#1955608 (Ottomata) @elukey woo woooo [15:57:14] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1955620 (jcrespo) It is now executing ALTER 132/261, although that may be a bit misleading because it has already converte... [16:04:51] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1955662 (jcrespo) I want to point out the reduction on disk usage that was converting the Edit table, which is a considera... [16:16:19] milimetric: James_F tells me you own the edit-analysis.wmflabs.org dashiki instance, and suggests that might be a good place to put our numberz [16:17:37] milimetric: I can either give you the config I have for dashiki or add it myself if you give me access (and that would let me add more numberz too) [16:20:50] MarkTraceur: I just need the config url, it's Config:MultimediaHealth right? [16:20:58] Believe so, yeah [16:21:04] But I think I had to do more to make it work [16:21:21] so you want edit-analysis.wmflabs.org/multimedia-health or a separate URL? The proxies can hit anything [16:21:41] hm... shouldn't, otherwise something's wrong [16:24:03] MarkTraceur: ^ [16:33:47] milimetric: Yeah, that looks like a fine URL [16:34:15] ok, MarkTraceur gimme a sec, in a QR [16:34:26] No problem [16:41:39] MarkTraceur: https://edit-analysis.wmflabs.org/multimedia-health/#projects=commonswiki,dewiki,enwiki/metrics=Uploads [16:42:26] \o/ [16:42:33] James_F: ^^ we gots the numbahz [16:42:44] It's still generating, but yeah, graphs 'n' stuff [16:50:19] (PS1) MarkTraceur: Add deletions script [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/265757 [16:51:04] MarkTraceur: Gosh. [16:51:44] Right? So exciting [16:52:10] I kinda wish I had different colours for the different wikis, still [16:52:17] 'New uploaders' is a bit sparse, but it's a start. [16:52:32] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, operations, and 2 others: schedule a daily run of ContentTranslation analytics scripts - https://phabricator.wikimedia.org/T122479#1955827 (Nuria) [16:52:48] 'Uploaders' has different colours but 'Uploads' doesn't… [16:53:11] Oh, weird. [16:53:16] Analytics-Kanban: Productionize last access jobs for daily and monthly calculations {bear} - https://phabricator.wikimedia.org/T122514#1955829 (Nuria) a:Nuria>JAllemandou [16:53:38] Oh, I know exactly why that is [16:53:43] The TSVs are still the old ones [16:54:13] So the column heads all say "uploads" instead of "commonswiki" or "dewiki" or "enwiki" [16:54:27] But they're getting replaced (slowly) by the new data [16:54:46] Right-o. [16:54:56] I guess the rsync job doesn't sync deletions. [16:55:02] MarkTraceur, milimetric: Nice work. :-) [16:55:53] I didn't do much this time, which I will take as a self-congratulatory moment for the way dashiki and reportupdater have grown up [16:56:45] :-) [16:59:03] Analytics, Analytics-Cluster: Create and maintain an Analytics Cluster in Beta Cluster in labs. - https://phabricator.wikimedia.org/T109859#1955856 (JAllemandou) @elukey I'll ensure you a beer supplied night a next offsite for that one :) [17:01:12] Analytics, EventBus: Continuous Integration for mediawiki/event-schemas - https://phabricator.wikimedia.org/T124438#1955858 (Ottomata) NEW [17:07:32] (PS13) Nuria: [WIP] Daily and monthly uniques oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/216341 (https://phabricator.wikimedia.org/T92977) (owner: Madhuvishy) [17:10:15] a-team standup now? [17:10:41] yes [17:10:47] k, joining [17:10:54] elukey: (in casehe is not in a-team) [17:10:58] *case [17:11:01] i'll join in a minute [17:11:01] standuppp [17:11:13] nope I am! joining in a sec [17:11:58] madhuvishy: standup now ? [17:13:49] sorry had to move between floors [17:13:49] joining [17:14:11] Analytics-Cluster, Analytics-Kanban, EventBus, Patch-For-Review: Update MirrorMaker in Kafka .deb and puppet module - https://phabricator.wikimedia.org/T124077#1955912 (Ottomata) [17:14:37] Analytics-Cluster, Analytics-Kanban, EventBus, Patch-For-Review: Update MirrorMaker in Kafka .deb and puppet module [13 pts] - https://phabricator.wikimedia.org/T124077#1955917 (Ottomata) [17:44:08] (PS8) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [17:47:24] Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#1956019 (GWicke) Open>Resolved a:GWicke A basic event bus is now available in production, and is being populated with edit events f... [17:47:35] (PS5) Mforns: Divide app session metrics job into global and split [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) [17:50:30] Analytics-Kanban, Patch-For-Review: Provide weekly app session metrics separately for Android and iOS, and move to 7 day counts [13 pts] - https://phabricator.wikimedia.org/T117615#1956029 (mforns) Rebased both patches after the mobile->text revert. So, it's ready for CR. Cheers! [19:00:31] joal: best graphana we have for pageview API is this one? https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-system [19:00:33] cc mili|constructio [19:00:51] depends what you want to get nuria [19:01:06] For traffic, this one is not bad : https://grafana.wikimedia.org/dashboard/db/pageviews [19:01:40] nuria: the aqs-cassandra-X ones are cassandra-technically-oriented [19:02:34] joal: so this is better: https://grafana.wikimedia.org/dashboard/db/restbase [19:02:58] restbase is not us [19:03:38] joal: ah, ok, that is what i thought. [19:03:42] :) [19:03:46] But pageviews is [19:03:50] joal: is there a better dashboard then? [19:04:03] joal: ahhh https://grafana.wikimedia.org/dashboard/db/pageviews [19:04:06] 20:01:06 < joal> For traffic, this one is not bad : https://grafana.wikimedia.org/dashboard/db/pageviews [19:04:13] nuria: --^ [19:04:26] nuria: not as complete as the restbase one, but still ok [19:04:43] nuria: signing off for the weekend ! [19:04:50] joal: same here [19:04:50] See you a-team on monday ! [19:04:56] laters! [19:04:59] joal, have a nice weekend! [19:05:01] bye! [19:05:06] Thx mforns, you too :) [19:19:25] joal, nuria: we can fairly easily export & adapt any of the RB dashboards for your use [19:19:49] another one that might be interesting to you is https://grafana.wikimedia.org/dashboard/db/restbase-cassandra-cf-performance [19:25:14] joal: yes :) [19:25:25] joal: that's what it meaaaans :) [19:25:44] (CR) Aklapper: "Milimetric: Could this contributed patch get a review please?" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/263911 (owner: Wassan.anmol) [19:27:51] gwicke: I think we are going to have to have two dashboards: one for outside world to see at a glance response times and request ratios [19:28:03] gwicke: and other for us with more detail to troubleshoot issues [19:31:50] makes sense [19:55:51] !log deleting /a/mw-log/archive/CirrusSearchRequests per request from ebernhardson (https://gerrit.wikimedia.org/r/#/c/265773/) [20:01:02] Analytics, Wikimedia-Logstash: Kafka logging to Logstash - https://phabricator.wikimedia.org/T84907#1956812 (Ottomata) Open>declined a:Ottomata [20:08:20] Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1956833 (Ottomata) [20:13:27] Analytics-Cluster, EventBus, Services, operations: 4 more Kafka brokers, 2 in eqiad and 2 codfw - https://phabricator.wikimedia.org/T124469#1956846 (Ottomata) NEW a:Ottomata [21:29:54] Analytics-Tech-community-metrics: Eliminate duplicated «"source": "wikimedia:its"» identities in korma identities DB - https://phabricator.wikimedia.org/T124475#1957196 (Aklapper) NEW [22:32:47] Analytics, Learning-and-Evaluation: Add instruction text next to the input fields in the Program Global Metrics Report {kudu} - https://phabricator.wikimedia.org/T121899#1957511 (Abit) [23:59:21] Analytics: Add wm:BE-Wikimedia-Belgium to Wikimetrics tags {dove} - https://phabricator.wikimedia.org/T124492#1958065 (kevinator) NEW