[01:51:27] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [30.0] [06:10:09] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [06:23:39] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [30.0] [06:56:05] Analytics-Tech-community-metrics, DevRel-November-2015: Explain / sort out / fix SCR repository number mismatch on korma - https://phabricator.wikimedia.org/T116484#1834721 (Lcanasdiaz) New version of the library deployed. Waiting results .. [06:56:55] Analytics-Tech-community-metrics, DevRel-November-2015: Explain / sort out / fix SCM repository number mismatch on korma - https://phabricator.wikimedia.org/T116483#1834722 (Lcanasdiaz) New version of the library deployed. Waiting results .. [07:18:22] Analytics-Tech-community-metrics, DevRel-November-2015: Automated generation of (Git) repositories for Korma - https://phabricator.wikimedia.org/T110678#1834730 (Lcanasdiaz) Our new micro tool is being executed right now to update automagically the list of Git repos based on the following lists: - https... [07:24:18] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [08:09:47] Analytics-Tech-community-metrics, DevRel-November-2015: Automated generation of (Git) repositories for Korma - https://phabricator.wikimedia.org/T110678#1834741 (Lcanasdiaz) It works like a charm. I executed two times, find below the log of both executions ``` owl@atari:~/dashboards/mediawiki$ cat log/u... [10:08:43] hi a-team [10:17:20] (CR) DCausse: Implement ArraySum UDF (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/254452 (owner: EBernhardson) [10:35:31] Hi mforns_ [10:35:39] hi joal [10:35:49] How are you today ? [10:35:54] how are you doing? [10:35:54] have you seen the EL thing? [10:36:08] Just saw that [10:36:11] I'm fine thx [10:36:11] Seems real [10:36:35] Good thing from an alert perspective, bad thing from an EL one [10:37:09] Have you looked into it or not yet ? [10:37:11] Ori wrote to analytics about that, it seems the same problem that we saw some weeks ago [10:37:21] hmmm [10:37:27] * joal reads [10:37:36] It has the same behavior than the other time [10:38:05] I'll do a histogram of the database contents, to look for holes [10:46:05] joal, there is a ~65% loss from 01:30 UTC to 07:00 UTC today [10:46:35] mforns_: man, that's uncool [10:47:57] but it is recoverable I think, from the logs [10:48:43] mforns_: I dont understand how, being based on Kafka, we lose data [10:48:58] mforns_: Great news that it is recoverable, but I still don't get it :) [10:50:16] joal, you're right, after the changes we did, EL should block reading from kafka when the memory is full [10:51:35] hrmhrm [10:51:51] mforns_: you are the lead on this backfilling stuff [10:51:57] mforns_: how may I help ? [10:52:09] we'll have to look at process kills... and at the mysql consumer log [10:52:40] joal, I can do that during today, it's not a very hard thing, just takes time [10:52:48] but if you want to have a short root cause analysis session on the batcave? [10:53:31] sure mforns_ [10:53:35] now, later? [10:54:14] whenever you prefer [10:54:53] same, I'm in da cave ! [10:57:59] (PS1) Addshore: Add initial README [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255682 [10:58:14] (CR) Addshore: [C: 2 V: 2] Add initial README [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255682 (owner: Addshore) [10:58:26] joal, ^ [10:59:44] (PS1) Addshore: Add rc.php to minutely.sh [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255683 [11:00:04] (CR) Addshore: [C: 2 V: 2] Add rc.php to minutely.sh [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255683 (owner: Addshore) [11:05:36] (PS1) Addshore: Shebangs shebangs..... everywhere... [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255684 [11:06:14] (CR) Addshore: [C: 2 V: 2] Shebangs shebangs..... everywhere... [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255684 (owner: Addshore) [11:35:21] (PS1) Addshore: Add statements_per_entity script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255686 [11:39:05] (PS2) Addshore: Add statements_per_entity script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255686 (https://phabricator.wikimedia.org/T119609) [11:42:25] (PS1) Addshore: Add sitelinks_per_site script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255688 (https://bugzilla.wikimedia.org/119609) [11:43:33] (PS1) Addshore: Add last 2 scripts to daily cron script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255689 [11:44:54] (PS2) Addshore: Add last 2 scripts to daily cron script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255689 [11:44:57] (CR) Addshore: [C: 2] Add statements_per_entity script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255686 (https://phabricator.wikimedia.org/T119609) (owner: Addshore) [11:45:04] (CR) Addshore: [C: 2] Add sitelinks_per_site script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255688 (https://bugzilla.wikimedia.org/119609) (owner: Addshore) [11:45:20] (CR) Addshore: [C: 2] Add last 2 scripts to daily cron script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255689 (owner: Addshore) [11:45:23] (Merged) jenkins-bot: Add statements_per_entity script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255686 (https://phabricator.wikimedia.org/T119609) (owner: Addshore) [11:45:33] (Merged) jenkins-bot: Add sitelinks_per_site script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255688 (https://bugzilla.wikimedia.org/119609) (owner: Addshore) [11:45:42] Analytics-Tech-community-metrics, DevRel-November-2015: OwlBot seems to merge random user accounts in korma user data - https://phabricator.wikimedia.org/T119755#1834991 (Aklapper) NEW [11:47:58] (CR) Addshore: [V: 2] Add last 2 scripts to daily cron script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255689 (owner: Addshore) [11:58:32] (PS1) Addshore: Add ranks sparql script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255690 (https://phabricator.wikimedia.org/T119606) [11:59:50] (PS2) Addshore: Add ranks sparql script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255690 (https://phabricator.wikimedia.org/T119606) [12:01:19] (PS3) Addshore: Add ranks sparql script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255690 (https://phabricator.wikimedia.org/T119606) [12:01:38] (CR) Addshore: [C: 2 V: 2] Add ranks sparql script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255690 (https://phabricator.wikimedia.org/T119606) (owner: Addshore) [12:08:49] (PS1) Addshore: Add sitelinks_per_item script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255691 (https://phabricator.wikimedia.org/T119611) [12:09:41] (PS2) Addshore: Add sitelinks_per_item script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255691 (https://phabricator.wikimedia.org/T119611) [12:10:54] (CR) Addshore: [C: 2 V: 2] Add sitelinks_per_item script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255691 (https://phabricator.wikimedia.org/T119611) (owner: Addshore) [12:27:49] joal, I managed to log in to IRC [12:27:53] :D [12:27:59] Ho great mforns :) [12:28:06] So, omw :) [12:39:17] Analytics-Tech-community-metrics, DevRel-November-2015: OwlBot seems to merge random user accounts in korma user data - https://phabricator.wikimedia.org/T119755#1835029 (Aklapper) [12:48:58] Hi! [12:49:11] If I had some questions about the pagecounts, cfr. https://wikitech.wikimedia.org/wiki/Analytics/Data/Pagecounts-all-sites, stream with whom should I talk? [12:51:22] Hi CristianCantoro [12:51:34] Hi joal :) [12:51:38] The easiest way would be to send an email to the analytics leisT :) [12:51:57] where sorry? [12:52:36] analytics list: https://lists.wikimedia.org/mailman/listinfo/analytics [12:52:40] CristianCantoro: --^ [12:52:45] oh right [12:52:46] :) [12:53:12] Also just for you to know CristianCantoro, it's Thanksgiving week, we are only european workers :) [12:53:27] right! [12:53:29] You can expect more reactivity next week ;) [13:01:03] Analytics-Tech-community-metrics, DevRel-December-2015: Kill out-of-date scr-organizations-summary in korma - https://phabricator.wikimedia.org/T119756#1835036 (Aklapper) NEW a:Aklapper [13:01:54] Analytics-Tech-community-metrics: What is contributors.html for in korma? - https://phabricator.wikimedia.org/T118522#1835043 (Aklapper) What I see on custom http://korma.wmflabs.org/browser/contributors.html is covered by http://korma.wmflabs.org/browser/scr-contributors.html and http://korma.wmflabs.org/br... [14:21:22] Analytics-Tech-community-metrics: What is contributors.html for in contrast to who_contributes_code.html and sc[m,r]-contributors.html? - https://phabricator.wikimedia.org/T118522#1835119 (Aklapper) [14:23:40] Analytics-Tech-community-metrics, DevRel-November-2015: Improve Key performance indicator: code contributors new / gone - https://phabricator.wikimedia.org/T63563#1835122 (Aklapper) The page does not really show any trends for a specific contributor. I'm not even sure what its scope should be. 6 months o... [14:23:49] Analytics-Tech-community-metrics, DevRel-December-2015, DevRel-November-2015: Improve Key performance indicator: code contributors new / gone - https://phabricator.wikimedia.org/T63563#1835123 (Aklapper) [14:24:34] Analytics-Tech-community-metrics: What is contributors.html for, in contrast to who_contributes_code.html and sc[m,r]-contributors.html and top-contributors.html? - https://phabricator.wikimedia.org/T118522#1835124 (Aklapper) [14:24:55] Analytics-Tech-community-metrics, DevRel-December-2015: What is contributors.html for, in contrast to who_contributes_code.html and sc[m,r]-contributors.html and top-contributors.html? - https://phabricator.wikimedia.org/T118522#1802427 (Aklapper) [14:28:19] Analytics-Tech-community-metrics, DevRel-December-2015: Key performance indicator: Top contributors: Should have sane Ranking algorithm which takes (un)reliability of user data into account - https://phabricator.wikimedia.org/T64221#1835126 (Aklapper) [15:09:47] mforns: I HAVE FOUND A WAY :) [15:09:51] Ahahaha :) [15:47:40] joal, I'm back [15:47:45] Heya [15:47:48] cave ? [16:11:18] hello a-team [16:11:24] Hi nuria [16:11:29] did you figure out waht was going on with EL? [16:11:30] hi nuria :] [16:11:43] nuria, it seems the same thing than the other time [16:11:48] tha we backfilled [16:12:17] mforns: sleep did not worked I guess, let's look at logs [16:12:29] nuria, I think sleep worked! [16:12:49] it is actually backfilling by itself [16:12:55] but it't taking a while [16:13:13] by itself? [16:13:21] so wait mysql process got restarted [16:13:29] and now is proceeding from its offset? [16:13:38] ^ mforns [16:14:31] nuria, it has been backfilling during the morning [16:14:41] but how? [16:14:52] by means of stopping? [16:14:55] meaning [16:15:12] when it got restarted, it started consuming from kafka at the point it stopped [16:15:22] right, at the offset as i said [16:15:27] the other time, this was too much for the process and it got killed for oom [16:15:27] but .. ahem ... [16:15:42] but what kill the process? [16:15:58] now, when the queue got full, it slowed down kafka consumption [16:16:32] the other time, the consumer read so many events from kafka, that the memory got full and the process was killed by the system for oom [16:17:11] this time, the process seems to have controlled the input flow from kafka [16:17:32] and it is backfilling slowly [16:17:50] catching up, ya [16:18:08] I think i missunderstood then, cause i thought you mentioned that mysql process was killed [16:18:23] I am saying that, because of what we observed in the morning with joal, but the last query I did to the database is a bit confusing... :( [16:18:34] ya, it is because [16:18:43] the events get there by UDP & kafka [16:18:58] so UDP would have been persisted (those are server side sent by mw directly) [16:19:04] but kafka ones not [16:19:07] makes sense? [16:19:23] ^ mforns [16:19:31] nuria, UDP is what happend even before validation [16:19:38] mmm [16:19:39] joal: no [16:19:50] we have different streams of events [16:19:59] and mediawiki (server side) [16:20:06] doesn't send events to kafka [16:20:08] directly [16:20:28] ah but wait, it is possible udp does validation after [16:20:34] You are talking about the first step (using a forwarder from UDP or send directly from kafka) [16:20:35] (although mw has validated those earlier) [16:21:03] Then the forwarder send the event using kafka to the processor that validates [16:21:29] yes, i think you are right [16:22:09] nuria: https://www.mediawiki.org/w/index.php?title=File%3AEventLogging_on_Kafka_-_Lightning_Talk.pdf&page=6 [16:23:30] joal: you are right, i though mw stream was directly sent w/o kafka (cause it is already validated) [16:23:40] but no , it is been validated twice, which is fine [16:25:08] nuria, do you know exactly what the metrics "Sum of schema events" and "eventlogging-valid-mixed" mean? [16:25:39] mforns: eventlogging-valid-mixed is the addition of valid events for all topics [16:26:34] nuria: So the two metrics should be exactly the same --> They differ by 2 or 3 all the time (WEIRDOOOOO) [16:26:53] jaja [16:27:11] nuria, but the metrics get increased everytime an event enters the topic? [16:27:14] but where is teh sum-of-schema-events? I do not see it here: https://grafana.wikimedia.org/dashboard/db/eventlogging [16:27:15] nuria: want to talk in da cave with us ? [16:27:35] mforns: no, the metrics get increased everytime they get reported to statsd [16:27:43] sure [16:29:08] Analytics-Backlog, Analytics-Wikistats, DevRel-November-2015: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1673629 (Qgil) [17:19:18] Analytics-Kanban, Analytics-Wikistats, Patch-For-Review: Feed Wikistats traffic reports with aggregated hive data {lama} [21 pts] - https://phabricator.wikimedia.org/T114379#1835286 (ezachte) Added 3 more charts for per project totals, e.g. http://stats.wikimedia.org/EN/draft/SummaryZZ.htm (preview loc... [18:13:38] Analytics-Kanban: Make Eventlloging more resilient to kafka outages - https://phabricator.wikimedia.org/T119770#1835359 (Nuria) NEW [18:21:17] Analytics-Kanban: Create eventloggging alarm that triggers when sql insertion goes to zero - https://phabricator.wikimedia.org/T119771#1835372 (Nuria) NEW [19:07:04] AndyRussG: did some preliminary testing for 1hr of traffic and it looks like at least 1% of requests do not have cookies enabled [19:07:30] nuria: ah fantastic! thanks so much! [19:07:55] I wonder what percentage of those are JS-unsupported [19:08:42] (the even more specific and less tractable question that'd be interesting to answer is, what percentage of users who see CentralNotice banners don't have cookies enabled...) [19:09:26] AndyRussG: better said (preliminary research) "at least 1% of our pageviews for the hour i studied were coming from browsers with cookies off" [19:10:08] AndyRussG: you can answer that with data in cluster (it will be an estimation) [19:10:23] AndyRussG: you just have to do the work to correlate a bunch of things [19:10:36] AndyRussG: for teh estimate i gave you see: https://wikitech.wikimedia.org/wiki/Analytics/Unique_clients/Last_access_solution/BotResearch#Requests_with_no_cookies [19:11:39] nuria: wohoo K thanks much! [19:12:08] If you're curious, here's the (now less urgent than before) task that this relates to: https://phabricator.wikimedia.org/T119538 [19:12:23] (I can add that link to the task...) [19:16:18] AndyRussG: To be clear the banner insights you are interested on you can gather yourself from the data in the cluster, the "at least 1% of pageviews" is a low ball estimate [19:16:33] nuria: K interesting [19:17:04] Yeah the medium-term issue is fallback mechanisms in general if cookies and/or LocalStorage is not available [19:17:17] I wonder what % of overlap there is on no LocalStorage and no cookies [19:19:23] BTW I pulled a bit of data from our banner history logs, and found in a 1-hr sample that 0.06% of pageviews that were sampled to send BannerHistory logs (1:100 sample of pageviews in FR campaigns) had the event logging event with the a tag saying no LocalStorage available [19:19:25] nuria: ^ [19:20:15] AndyRussG: If you make a wikitech doc outlining your selects we can look at your logs [19:20:21] AndyRussG: but number looks wrong [19:20:47] AndyRussG: as we know a much higher percentage than that does not have javascript enabled [19:20:58] nuria: well, it's only a % of the users getting banners. So yes, that excludes those with no JS [19:21:19] As well as platforms where banners don't work, such as (as of recently) IE8, as well as Opera Mini [19:21:27] AndyRussG:0.06% of banners requests have no local storage then? [19:22:33] AndyRussG: without counting ie8 and other platforms were we either do not serve js or js is not enabled correct? [19:22:50] yeah [19:23:37] On a 1-hr sample of banner history logs (which were sent from 1% of pageviews in certain regions on platforms on which CentralNotice is functioning completely) 0.06% of the logs were marked as empty because LocalStorage wasn't available [19:23:54] You can also re-sample any period yourself [19:24:18] Check out what hive has for the schema: https://meta.wikimedia.org/wiki/Schema:CentralNoticeBannerHistory [19:24:49] All you need is to get the percentage of events that include an "e" property [19:26:48] When trying to save a banner history event to LocalStorage, we check if LocalStorage was available. Then, when we send the logs back (whih we do for 1% of users in campaigns with that feature enabled, which is now all FR campaigns I think) if LocalStorage wasn't available, we set the "e" property. On normal cases, the "e" property won't be set. [19:27:12] So you can just check the % of banner history logs with the e property as a percentage of the total for any time period [19:27:58] nuria: ^ [19:32:07] AndyRussG: ok, then , it seems a plausible number [19:36:07] nuria: cool! [19:36:47] AndyRussG: but i would do a page like teh one i sent you taht outlines your methodology/results and code so if someone needs to double check your numbers they can do it [19:37:25] AndyRussG: otherwise it is hard to gauge how correct that number is [19:42:05] nuria: OK you bet! It was just a preliminary test but I'd be happy if even so it could be of use to someone or a starting point for further data [19:42:46] I think once we start processing the banner history data in earnest, we'll have a bigger sample on this (though the same basic methodology) [19:43:54] nuria: gotta do some family driving in a few minutes! I'll be back online in a bit tho [20:00:49] nuria: hello! [20:00:58] ottomata: holaaaaa [20:01:07] how goes?! [20:01:11] things a little funky this morning eh? [20:01:29] ottomata: well, learning the inners of EL consumers from kafka... [20:01:41] ottomata: but not to bad i'd say [20:01:48] ottomata: we can talk whenever [20:02:11] nuria: https://phabricator.wikimedia.org/T118315 [20:02:13] i'm working a bit now [20:02:28] ottomata: ah, duplicate of [20:03:22] ottomata: https://phabricator.wikimedia.org/T119770 [20:03:38] Analytics-Backlog, Analytics-EventLogging: EventLogging (MySQL?) Kafka consumer stops consuming after Kafka metadata change - https://phabricator.wikimedia.org/T118315#1835447 (Nuria) [20:03:39] Analytics-Kanban: Make Eventlloging more resilient to kafka outages - https://phabricator.wikimedia.org/T119770#1835446 (Nuria) [20:03:56] Analytics-EventLogging, Analytics-Kanban: EventLogging (MySQL?) Kafka consumer stops consuming after Kafka metadata change - https://phabricator.wikimedia.org/T118315#1797263 (Nuria) [20:07:49] ok nuria i should do a kafka leader election [20:07:59] but, i suspect this will cause consumers to stop [20:08:03] as that is the behavior we have seen [20:08:14] shall we do so and just watch logs to see, and then restart eventlogging? [20:08:31] ottomata: if it is a test we should do it on labs [20:08:51] ottomata: if we need to do a leader in prod election let's stop EL [20:08:56] ottomata: do election [20:09:00] ottomata: and re-start [20:09:03] it is not a test [20:09:14] ok, i mean, its effectilvey the same eithe rway [20:09:16] and i've done it before [20:09:30] its nice to see if we see the same error still (i expect we will) [20:10:59] ottomata: I rather have production not fail if possible [20:11:04] ok [20:11:08] let's stop el then [20:11:26] !log stopping eventlogging and doing a kafka leader election, then starting eventlogging [20:13:13] hmm, nuria looking at mysql consumer logs, don't see any data inserted messages... [20:13:14] hm [20:13:32] oh, there they are [20:13:33] its ok [20:13:53] guess it takes a bit on start up for insert queues to fill before it starts inserting [20:13:55] looking good [20:14:00] k [20:14:39] ottomata: i am going to do an alarm on sql insertion cause it seems a good proxy to detect these type of issues, the raw-validated seems to fail too often [20:15:08] yeah, that sounds good [20:15:14] cause sql insertion going to zero for a while it's indicative of many issues, teh most likely one a kafka one [20:15:20] maybe a check_graphite_anomoly one! [20:15:24] or 0, eyah [20:15:26] that would be good [20:15:27] good idea [20:15:37] what's the deal with server side metric being wrong? [20:15:39] k, let me send you a cR [20:16:13] ottomata: i do not know, but in graphana at times it is, we can look it together whenever you can [20:17:35] hm [20:17:41] i'm looking at just graphite now, and it looks fine there [20:19:54] nuria: this looks good, right? [20:19:55] https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=6&fullscreen [20:20:23] client side and server side are the pinkish stacked ones [20:20:25] ottomata: when you mouseover [20:20:26] no [20:20:32] ottomata: when you look at the panel yes [20:20:55] when you mouseover, its just showing the summed stacked value, that matches up with the y axis [20:21:10] ottomata: ahem, whattt??? [20:21:40] ottomata: when you mouseover "raw server side" is a sum? [20:22:08] yeah, not sure how to make it show you the raw value [20:22:18] but in this graph, server and client side raw are stacked [20:22:32] that's why the server side is a darker pink color on top of the lighter pink color [20:22:59] ahem .. ok, Ui sadness but i guess values (which is what matter) are ok [20:30:05] Analytics-Kanban: Create eventloggging alarm that triggers when sql insertion goes to zero - https://phabricator.wikimedia.org/T119771#1835479 (Nuria) a:Nuria [20:47:06] Analytics-Kanban, Datasets-Webstatscollector: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article [5 pts] {slug} - https://phabricator.wikimedia.org/T118933#1835494 (Nuria) Open>Resolved [20:47:39] Analytics-Kanban: Prepare Pageview API lightning talk {melc} [5 pts] - https://phabricator.wikimedia.org/T119091#1835495 (Nuria) Open>Resolved [20:48:01] Analytics-Kanban, Patch-For-Review: Create eventloggging alarm that triggers when sql insertion goes to zero [3] - https://phabricator.wikimedia.org/T119771#1835496 (Nuria) [20:49:14] Analytics-Kanban, WMDE-Analytics-Engineering, Wikidata, Patch-For-Review: Fix '.*http.*' not being tagged as spiders in webrequest [5 pts] {hawk} - https://phabricator.wikimedia.org/T119054#1835498 (Nuria) Open>Resolved [20:49:52] Analytics-Kanban: JsonRevisionsSortedPerPage failed on enwiki-20150901-pages-meta-history [13 pts] {paon} - https://phabricator.wikimedia.org/T114359#1835499 (Nuria) Open>Resolved [20:51:09] Analytics-Cluster, Analytics-Kanban: Estimate number of users (or requests) that have cookies off (due to fresh session or incognito mode)[3] - https://phabricator.wikimedia.org/T119653#1835512 (Nuria) [20:51:48] Analytics-Cluster, Analytics-Kanban: Estimate number of users (or requests) that have cookies off (due to fresh session or incognito mode)[3] - https://phabricator.wikimedia.org/T119653#1832269 (Nuria) See; https://wikitech.wikimedia.org/wiki/Analytics/Unique_clients/Last_access_solution/BotResearch