[04:15:35] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2179692 (Nuria) @Tbayer: We do not have the ability to backfill pageviews only for iOs and reruning pageview... [06:52:26] (PS1) Amire80: Add a script for checking number of pages published despite failures [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282312 [07:00:18] (CR) KartikMistry: [C: 2] Add ady to language list [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282190 (owner: Amire80) [07:01:27] (CR) KartikMistry: Add sorted errors (1 comment) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 (owner: Amire80) [07:03:21] (CR) KartikMistry: [C: 2] Show the top user name along with the top article count [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/280827 (owner: Amire80) [07:07:43] (Merged) jenkins-bot: Add ady to language list [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282190 (owner: Amire80) [07:07:45] (Merged) jenkins-bot: Show the top user name along with the top article count [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/280827 (owner: Amire80) [08:10:40] Analytics-Kanban, Operations: Upgrade aqs* to nodejs 4.3 - https://phabricator.wikimedia.org/T123629#2189365 (elukey) Open>Resolved [08:52:34] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#1934774 (Aklapper) [08:55:46] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Contributor pages which show user name but not any other data should include an explanation - https://phabricator.wikimedia.org/T58111#2189432 (Aklapper) [08:56:06] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Make GrimoireLib display *one* consistent name for one user - https://phabricator.wikimedia.org/T118169#2189433 (Aklapper) [08:57:14] hi elukey, here ? [08:57:58] joal: o/ [08:58:08] you want to deploy AQS [08:58:21] elukey: You READ MY MIND ! [08:58:28] I knewwwww ittttt [08:58:31] * joal hides away from elukey !!!! [08:58:43] :D just read Dan's email :D [08:58:54] ;) [08:58:59] So what d'you think? [08:59:07] Do you back me up [08:59:09] ? [08:59:38] and by the way a-team: https://wikimedia.org/api/rest_v1/metrics/unique-devices/en.wikipedia/all-sites/daily/20160101/20160201 [09:00:50] * elukey dances [09:01:00] * joal dances with elukey :) [09:01:40] while I don't like deployments on a Friday, if it is low risk and you double checked I have no objections [09:01:57] elukey: I do agree on friday deploys [09:02:14] very low risk, and triple check everything [09:02:26] +1 then :) [09:03:45] elukey: currently double checking what the deploy involves [09:42:19] joal: working with Ariel atm to swtich dumps.w.o to https, ping me if you need anything :) [09:42:28] elukey: okey ! [09:52:49] (PS1) Joal: Update aqs to 7445dec [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/282329 [09:53:46] Analytics, Datasets-General-or-Unknown, Operations, Traffic, Patch-For-Review: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2189490 (ArielGlenn) Open>Resolved with collaboration with elukey this is done. [09:58:39] (CR) Joal: [C: 2 V: 2] "Merging for deploy." [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/282329 (owner: Joal) [10:01:23] (PS1) Joal: Add execution rights to aqs manual test script [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/282332 [10:01:56] (CR) Joal: [C: 2 V: 2] "Self merging minor update for deploy." [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/282332 (owner: Joal) [10:03:11] !log deploying aqs from tin on aqs1001 [10:03:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:08:22] !log deploying aqs from tin on aqs1002 [10:08:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:09:39] !log deploying aqs from tin on aqs1003 [10:09:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:14:31] :) [10:16:42] elukey: SUCCESS ! [10:17:03] elukey: job tested the various stuff, everything works fine :) [10:17:43] * joal is back to multithreading debugging [10:18:44] goooood! [10:19:41] * milimetric dances [10:20:22] hey milimetric ! far too early for you to work, but ok for a dance :) [10:23:08] I think dancing is mandatory when there are so many things in the done column!! Nice job everyone [10:23:20] milimetric: +100 ! [10:23:46] Our quarterly review will be like: [10:23:49] a d [10:24:08] And then, we did this, and then this, and then alsoooo this [10:24:15] :D [10:24:21] :) [10:33:12] milimetric: now that uniques are served though API, how long would it take to include them in vital signs? [10:40:16] elukey: from what I read on phab, dumps->https done ? [10:41:35] yep! [10:41:41] elukey: awesome :) [10:48:02] * joal just thinks he has fixed loading stuff [10:48:11] * joal is jumping in joy in his office [10:49:12] wooooo [10:49:20] http://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams --> Wikipedia Demo on Kafka Streams - step by step [10:49:23] ahahahah [11:18:34] (PS1) Joal: Correct CqlRecordWriter in cassandra module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282338 [11:19:12] elukey: If you have minute, I'll gladly have your opinion on --^ [11:20:23] joal: would it be fine in ~1hr? [11:20:31] elukey: no rush :) [11:20:41] super (lunch and review!) [11:20:48] elukey: enjoy food :) [11:38:45] Analytics-Kanban: Document the new AQS endpoint and launch it - https://phabricator.wikimedia.org/T129520#2189670 (JAllemandou) [12:03:13] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): top-contributors should have real names for the main contributors - https://phabricator.wikimedia.org/T124346#2189697 (Aklapper) This should pretty much be fixed by https://github.com/Bitergia/mediawiki-identities/commit/6e10a3e8d33265... [12:09:43] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Contributor pages which show user name but not any other data should include an explanation - https://phabricator.wikimedia.org/T58111#2189700 (Aklapper) Verified. Thanks! [12:16:02] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Make GrimoireLib display *one* consistent name for one user - https://phabricator.wikimedia.org/T118169#2189703 (Aklapper) @Lcanasdiaz: Thanks! One issue: Going to [[ http://korma.wmflabs.org/browser/scr-contributors.html | scr-contrib... [12:19:54] joal: looks good! Even if I am super sure about all the consequences in the overall code flow (partially forgot from yesterday). nuria will surely be able to do a better review! :( [12:20:15] elukey: Thanks mate :) [12:25:41] * joal takes a greak, [12:26:01] * joal takes a *break - Will be back later ! [13:30:48] morning joal! checking email... [13:39:28] (CR) Ottomata: [C: 1] "IIInteresting" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282338 (owner: Joal) [13:40:40] Analytics, WMDE-Analytics-Engineering, Wikidata: [Task] dashboard showing browser usage distribution for Wikidata - https://phabricator.wikimedia.org/T130102#2190011 (Lydia_Pintscher) Ok can one of you get this to me in an accessible way please? I'd just like a chart of this and I'm happy. [13:52:21] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016), developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2190033 (Aklapper) >>! In T103292#1933663, @Qgil wrote: > whether the curr... [14:04:24] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016), developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2190046 (Aklapper) As Lcanasdiaz said, the script was not running. Now it'... [14:05:35] hmmmm, i forget, who knows things about aggregator? [14:07:07] (PS1) Addshore: Add note to statements_per_entity.php [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/282369 [14:07:30] (CR) Addshore: [C: 2 V: 2] Add note to statements_per_entity.php [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/282369 (owner: Addshore) [14:09:24] Analytics, WMDE-Analytics-Engineering, Wikidata: [Task] dashboard showing browser usage distribution for Wikidata - https://phabricator.wikimedia.org/T130102#2190056 (Addshore) >>! In T130102#2190011, @Lydia_Pintscher wrote: > Ok can one of you get this to me in an accessible way please? I'd just like... [14:10:37] Analytics, WMDE-Analytics-Engineering, Wikidata: [Task] dashboard showing browser usage distribution for Wikidata - https://phabricator.wikimedia.org/T130102#2190057 (Lydia_Pintscher) Wikidata [14:22:31] (PS1) Addshore: Also count items,props and statements from dumps [analytics/wmde/toolkit-analyzer] - https://gerrit.wikimedia.org/r/282372 [14:22:34] (PS1) Addshore: Also count avg statements per item/prop from dumps [analytics/wmde/toolkit-analyzer] - https://gerrit.wikimedia.org/r/282373 [14:24:58] ottomata: updated code review, building the new package now :) [14:25:17] coool [14:26:44] hm elukey it hink you can remove varnishkafka.postinst altogether [14:26:47] it doesn't do anyting anymore [14:27:04] (PS1) Addshore: New build with: [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/282374 [14:27:45] (CR) Addshore: [C: 2 V: 2] Also count items,props and statements from dumps [analytics/wmde/toolkit-analyzer] - https://gerrit.wikimedia.org/r/282372 (owner: Addshore) [14:27:48] (CR) Addshore: [C: 2 V: 2] Also count avg statements per item/prop from dumps [analytics/wmde/toolkit-analyzer] - https://gerrit.wikimedia.org/r/282373 (owner: Addshore) [14:27:51] (CR) Addshore: [C: 2 V: 2] New build with: [analytics/wmde/toolkit-analyzer-build] - https://gerrit.wikimedia.org/r/282374 (owner: Addshore) [14:28:25] ottomata: wasn't sure, ok removing [14:28:48] k [14:33:24] Analytics-Cluster, Operations: setup stat1004/WMF4721 for hadoop client usage - https://phabricator.wikimedia.org/T131877#2190122 (Ottomata) Thanks! [14:34:10] heyy elukey we got 2 new nodes coming in for setup: stat1004 (analytics clluster client node) and analytics1003 (hive/oozie server) [14:34:15] stat1004 is ready [14:34:20] I do one, you do the other? [14:34:22] your choice! [14:34:23] :) [14:35:09] stat1004 is prob pretty easy, just applying puppet, etc. analytisc1003 will require a migration and alittle bit of hive/oozie downtime [14:43:39] I can do both if you want! Or just check what you do.. need some experience :) [14:46:27] ok! well, either way, maybe we can do analytics1003 together whne it is time [14:46:31] stat1004 will be easy [14:46:40] its already online and puppetized, just needs analytics stuffnow [14:47:11] it needs analytics_cluster::client, analytics_cluster::refinery I think that's it [14:47:25] i'll assign task to you for that one [14:49:03] Analytics-Cluster, Analytics-Kanban, Operations: setup stat1004/WMF4721 for hadoop client usage - https://phabricator.wikimedia.org/T131877#2190216 (Ottomata) a:Ottomata>elukey [14:49:27] Analytics-Cluster, Analytics-Kanban, Operations: setup stat1004/WMF4721 for hadoop client usage - https://phabricator.wikimedia.org/T131877#2181451 (Ottomata) Ok! @elukey this is online as stat1004.eqiad.wmnet. It has base puppet. It needs the following two roles: - analytics_cluster::client - an... [14:49:30] Analytics-Cluster, Analytics-Kanban, Operations: setup stat1004/WMF4721 for hadoop client usage - https://phabricator.wikimedia.org/T131877#2190229 (Ottomata) [14:58:09] Analytics, Analytics-EventLogging, Performance-Team, Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#2190538 (Ottomata) I talked with @Krinkle on IRC yesterday and we came up with a plan of action. There are two parts to... [14:58:18] Analytics, Analytics-EventLogging, Performance-Team, Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#2190542 (Ottomata) a:Krinkle [14:59:58] Hey ottomata P! [15:00:11] ottomata: you've seen, weirdo stuff in cassandra :) [15:00:28] ottomata: as for aggregator, either milimetric or me I think [15:00:39] joal: its ok! i figured aggregator out [15:00:43] so being here, please shoot :) [15:00:46] just stuff i had forgotten [15:00:48] ah, ok :) [15:01:03] but ja, so, in summary, your patch just makes it try to connect a second time? [15:01:08] is that right?" [15:01:20] in case the first failure/timeout/disconnect was a threading fluke? [15:01:36] ottomata: not really, there is a connection initialisation (with method extraction) [15:02:10] There was NullPointerException expected because of no initialisation, this bit is just cleaner code [15:02:40] The actual fix is in using poll instead of take for the first data to get, and therefore not use interupt at close time [15:02:44] ottomata: --^ [15:03:29] Analytics-EventLogging, Analytics-Kanban, Scap3 (Scap3-Adoption-Phase1): Stop using global eventlogging install on hafnium (and any other eventlogging lib user) - https://phabricator.wikimedia.org/T131977#2190579 (Ottomata) a:Ottomata>Krinkle T110903 is about changing EL consumers to use Kafka.... [15:03:48] hm [15:04:28] interesting [15:04:52] joal it hink you'd have to outline the whole issue to me if you want me to understand it...buuut if you have a working solution you mihgt not have to [15:04:57] ottomata: the downside is that termination is less event driven (before, interuption was fired, releasing thread if blocked) [15:05:19] ottomata: If you're interested, I'll share, if you don't care, I'll keep ;) [15:05:44] haha, uhhh,i care fi you think I can help, but idunno...? [15:06:24] ottomata: I don't think it's for help really, more for info :) [15:06:37] so, I understand you might other stuff to do ;) [15:15:10] joal: jaaa if you think it will be fun to tell me, i'm interested! :) [15:15:21] also don't want to take your time :) [15:15:39] ottomata: Let's go for it, another set of eyes is always good :) [15:15:43] batcave? [15:15:54] k [15:21:55] Analytics-Kanban, Patch-For-Review: Fix limn-mobile-data mobile-options-last-3-months report after RU changes {lama} - https://phabricator.wikimedia.org/T131849#2190642 (mforns) I found a bug in RU, in the method get_date_threshold, used by the mobile-options-last-3-months report, that is breaking the exe... [15:25:53] (PS8) Nuria: Allow filtering of data breakdowns [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [15:34:56] a-team: if you are ok I am going to skip stand-up (and send e-scrum) to join an ops presentation happening at the same time [15:39:00] elukey, ok by me :] [15:42:22] Analytics: Add JJB support for Jenkins Maven Release Plugin {hawk} - https://phabricator.wikimedia.org/T132175#2190696 (madhuvishy) [15:43:10] Analytics: Add a maven-release user to Gerrit {hawk} - https://phabricator.wikimedia.org/T132176#2190712 (madhuvishy) [15:43:51] Analytics: Create separate archiva credentials to be loaded to the Jenkins cred store {hawk} - https://phabricator.wikimedia.org/T132177#2190727 (madhuvishy) [15:44:24] Analytics: Have archiva server credentials available via the Config File Builder in global maven settings.xml - https://phabricator.wikimedia.org/T132178#2190741 (madhuvishy) [15:45:06] Analytics: Figure out a rollback strategy if the release job fails {hawk} - https://phabricator.wikimedia.org/T132179#2190755 (madhuvishy) [15:45:39] Analytics: Figure out the exact strategy for release {hawk} - https://phabricator.wikimedia.org/T132180#2190769 (madhuvishy) [15:46:21] Analytics: Figure out if the Changelog file can be updated in the release process by Jenkins {hawk} - https://phabricator.wikimedia.org/T132181#2190783 (madhuvishy) [15:47:23] Analytics: Translate the analytics-release-test job to YAML config in integration/config {hawk} - https://phabricator.wikimedia.org/T132182#2190797 (madhuvishy) [15:51:51] joal, there is a rise in 5xx responses from pageviews recently: https://grafana-admin.wikimedia.org/dashboard/db/pageviews?panelId=1&fullscreen [15:52:51] looks correlated with disk read throughput: https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-system?panelId=7&fullscreen [15:53:02] gwicke: yeah, due to timeout :S [15:53:53] we do have logs of timed-out requests if you want to have a look at parameters [15:54:24] do you have any limits on the requested data set size? [15:55:28] re logs, you can seach for 'analytics.wikimedia.org' in https://logstash.wikimedia.org/#/dashboard/elasticsearch/restbase [15:55:40] gwicke: thanks for the extra metric! I am going to add another panel for all the status codes [15:57:18] * elukey adds documentation about logs [15:57:34] most timed out requests are for 1-2 months worth of data for a given article [15:59:22] (brb meeting) [16:00:25] mforns: standdduppp [16:00:28] yesss [16:01:30] gwicke: we don't restrict datasets [16:01:43] gwicke: in standup, will get back in minutes [16:34:05] nuria_, I think the percentage display changes have not been deployed to the browser reports yet, I tested with changing the report source and it works but there is no % sign [16:35:19] nuria_, oh! no, I think there is some configuration missing on my side... [16:35:20] mforns: The code that enables % is there but might need to be configured on the metric what is the default format, let me see [16:35:30] yesss [16:41:50] mforns: the config for teh dashboard does not have percentages: https://meta.wikimedia.org/wiki/Config:SimpleRequestBreakdowns [16:51:09] joal: just realized that i have been using 1.6 all this time in intelij [16:53:20] (PS1) Joal: Add pageview definition special case for iOs App [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282388 [16:53:34] nuria_: old loves never stop ;) [16:54:16] joal: man ... this is sadness, fixed that and now looking at your code and learning about https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html [17:01:24] nuria_: I think I have a even better approach for the thing [17:01:29] Testing while you read [17:01:32] k [17:04:21] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191115 (Tbayer) >>! In T131824#2189130, @Nuria wrote: > @Tbayer: We do not have the ability to backfill pag... [17:05:35] ottomata: time to check https://gerrit.wikimedia.org/r/#/c/282161 ? [17:13:23] nuria_: while fun, the idea I had doesn't work :) [17:13:33] nuria_: the code you have is best I can do :) [17:13:47] nuria_: let me know if you want to discuss it [17:15:57] nuria_, some of the browser reports had not been converted to rates by the ad-hoc script successfully, so I deleted them, and will wait for RU to backfill them [17:16:31] after that, I'll edit the new config in meta and [17:16:41] elukey: cool, +2 merge at will! [17:20:17] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191154 (Nuria) >Even if it is not completed in time for the Reading team's quarterly review (April 12) for... [17:20:27] joal: ay, i do not understand the fix, will comment [17:21:39] nuria_: wanna batcave for explanations? [17:21:45] joal:sure [17:21:54] ottomata: thanksss! I am going to package another version with ema's latest CR (s/LIBS/LDFLAGs/g) on monday, after that we should be done done done [17:22:13] oook awesome! [17:24:12] also I hope to be close to add full submodule support to the puppet compiler [17:24:27] so pcc will work directly with a code review filed against a submodule repo [17:25:00] yaaa saw that [17:25:02] looks awesome [17:25:05] very useful [17:25:43] the idea is to chdir to modules/module_name and run fetch/pull/etc.. and then finally git submodule --init [17:25:59] but I am not sure if it is enough [17:29:42] ottomata --^ [17:34:34] (CR) Nuria: [C: 2 V: 2] "Did not run code but fix concurrency wise makes more sense, removing take which is blocking and replacing it with poll. Also removing NPE " [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282338 (owner: Joal) [17:38:42] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191179 (Nuria) >Even if it is not completed in time for the Reading team's quarterly review (April 12) for... [17:41:15] just fetch pull every time? [17:41:27] hm, i think there probably needs to be a way to specify a submodule and sha to check [17:41:35] elukey: ^ [17:41:51] (CR) Nuria: [C: 2 V: 2] Add pageview definition special case for iOs App [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282388 (owner: Joal) [17:46:13] ottomata: the code automagically finds the correct dir for the module, changes dir, fetch/pull, back to main dir, git submodule --init [17:46:27] I'll see if it works, might not be enough [17:47:31] a-team: added metrics to https://grafana.wikimedia.org/dashboard/db/pageviews - there are some 50x registered recently (also viewable in https://logstash.wikimedia.org/#/dashboard/elasticsearch/restbase -> query: analytics.wikimedia.org) [17:48:43] oh if you just pass it the gerrit change for thesubmodule commit [17:48:45] mmm [17:48:46] cool [17:48:48] ja that should work [17:50:36] yesss exactly! [17:55:39] all right, going offline, let me know if the 50x issue is known (timeouts? SSDs?) or if we need to open a phab task! [17:55:43] byyyyeeeeeee o/ [17:55:50] have a good weekend folks [17:57:12] Bye elukey ! [17:57:22] byeee [18:51:16] elukey: looked at dashboard, boy do we need larms of pageview api [18:51:20] *alarms [18:52:08] elukey: super thnaks for updating dashboard [19:05:17] nuria_: here ? [19:12:13] ottomata: could be fun : https://engineering.linkedin.com/blog/2016/04/dr-elephant-open-source-self-serve-performance-tuning-hadoop-spark [19:17:05] ha awesome [19:32:42] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191536 (Mhurd) @Nuria > code is in gerrit not github Oh sorry, I am aware. I just used that link for conv... [19:49:34] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191578 (Mhurd) @Nuria Could we just change it from this... ``` private static final Pattern appAgent... [19:50:35] milimetric: hiii, yt? [19:53:34] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191582 (JMinor) > I understand, what we are saying is that In our end we have to balance the amount of work... [19:54:32] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191584 (Mhurd) The 5.0.0, 5.0.1 and 5.0.2 user agent strings also contain `iOS` and `Scale` if that helps:... [20:09:59] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191619 (Tbayer) >>! In T131824#2191154, @Nuria wrote: >>Even if it is not completed in time for the Reading... [20:10:25] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191622 (Tbayer) [20:13:56] nuria_: i'm honestly a bit confused by your statements in https://phabricator.wikimedia.org/T131824 , see my followup question. in any case, i would agree that this bug is not the Analytics team's fault, but would appreciate a solution-focused discussion [20:42:59] (Abandoned) Yuvipanda: Update compiled javascript template for previous patch [analytics/quarry/web] - https://gerrit.wikimedia.org/r/218131 (owner: Yuvipanda) [21:04:54] HaeB: at this time we have made changes to update UA so pageview computations as of when we deploy next (monday) will be including the new UA string. [21:07:43] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191762 (Nuria) >I'm not sure I understand why this is so costly, and I have no way to engage in a discussio... [21:08:36] nuria_: that's great; i was talking about solutions for the current, invalid data though [21:13:04] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191763 (Nuria) >Is the issue that re-running the pageviews can only be done across all requests and so all... [21:14:07] HaeB: Updated ticket. [21:19:26] nuria_: thanks - read it, but it's still not clear to me what the answer to my question at https://phabricator.wikimedia.org/T131824#2191619 is. [21:19:44] are you saying it is impossible to run a query against the webrequest table that covers one month's worth of data (March 11-April 11, the time between the rollout of the faulty user agent and the update of the definition)? [21:50:16] bye a-team, have a nice week-end! [21:53:47] byeee [22:28:02] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191961 (Mhurd) @nuria @JAllemandou Does this need to be updated as well? https://phabricator.wikimedia.o... [22:32:55] Hi! Are real prod cookie names logged, or sampled and logged, anywhere? [22:33:02] HaeB: not sure what you mean? query for you to count the pageviews or recompute? [22:35:13] HaeB: I will be here for a bit more, ping me if you are around. [22:36:12] Analytics, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, iOS-app-Bugs, iOS-app-v5.0.3-Disco: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2191965 (Nuria) @Mhurd : indeed! Thanks for catching that. [22:36:33] nuria_: see the link - the question there is "Are you stating that it is impossible for capacity reasons to run a webrequest query as described in my previous comment?" [22:37:09] HaeB: webrequest query for what purpose sorry? recomputing includes a lot more things that running a query [22:37:49] nuria_: again, please see the link [22:37:59] the previous comment (linked there) describes it [22:38:41] HaeB: Maybe you can be so kind as to repeat what you need so I can make sure I understand? [22:39:45] nuria_: well, could you read the phabricator comment first? I already wrote it down there [22:40:26] ...if it's unclear there, we can try to take it from there [23:10:42] (PS1) Nuria: Add pageview definition special case for iOs App [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282439 (https://phabricator.wikimedia.org/T131824) [23:13:36] HaeB: I have re-read the ticket, i think you are perhaps mixing my comments about recomputation with queries that you want to run to calculate that data manually? [23:17:08] HaeB: If you just want the pageviews for IOS you can get them easily from webrequest table. Recomputing them involves doing updates in several other places besides webrequest and pageview hourly which are only used internally. Let me cut and paste a query you can use to calculate pageviews from webrequest data. [23:19:41] Hi nuria_! .... any ideas about my question above, how to find a list of cookie names floating in the wild? [23:21:23] AndyRussG: a list of cookie names? those are not in the cluster if they do not get set in x-analytics header so I would look at VCL code on varnish templates [23:21:41] Hmmm [23:22:53] nuria_: yeah I didn't see them in x-analytics... K thx much! [23:49:59] nuria_: (re) i was aware of the distinction between backfilling the tables and merely calculating (computing) the numbers from the webrequest table, that's why i asked for confirmation ("Are you stating ..."). perhaps we are just using different definitions of https://en.wiktionary.org/wiki/compute#Verb ;) [23:50:14] ...a query snippet would be great