[00:12:38] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2683339 (Nuria) Resolved>Open [00:18:06] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2683342 (Nuria) We are still getting tons of events not valid from the C... [01:01:13] (CR) Nuria: "Run query in secs w/o issues." [analytics/refinery] - https://gerrit.wikimedia.org/r/311964 (https://phabricator.wikimedia.org/T146064) (owner: Nschaaf) [02:25:33] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (bd808) The the long time maintainer of [[https://pecl.php.net/package/yaml|YAML bindings for PHP]], I oppose a mo... [07:30:57] morning! [07:47:34] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (TheDJ) >>! In T147158#2683314, @Legoktm wrote: > I think the premise here is flawed. I don't think we should ever... [07:47:40] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (jayvdb) If we switched to YAML, we should define a subset that fixes only these problems. Instead I recommend TO... [07:48:25] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (RobLa-WMF) I get where Yuri is coming from. YAML's comments and more flexible handling of formatting make it a m... [07:57:16] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683640 (Harej) >>! In T147158#2683314, @Legoktm wrote: > I don't think we should ever have users editing raw JSON themsel... [10:00:27] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [10:01:13] buuuuu [10:03:18] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [10:05:04] elukey: makes a few days that those alarms fire up once in a while [10:05:16] elukey: dixit nuria_ they are user-scehma related [10:05:28] yep I saw the emails :) [10:05:46] :) [11:04:29] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [30.0] [11:06:27] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [11:14:04] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [30.0] [11:21:44] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [11:26:22] so I can see a RecentChanges recurrence in the logs with (None is not of type 'integer') [11:26:29] is there a schema like this one ? [11:29:38] ah from logstash seems clear, ChangesListFilters and MobileWikiAppFindInPage [11:29:51] mforns: whenever you have time can we check --^ ? [11:30:13] elukey, yes! [11:31:31] ah just seen https://phabricator.wikimedia.org/T146674#2683342 [11:32:33] ok nevermind, probably we'll just need to ping people [11:32:40] not sure if the change has been deployed though [11:32:43] (it is merged) [11:33:14] don't think so [11:34:55] elukey, from the error messages, the error still looks the same, the namespace field is not populated. [11:36:11] yeah, probably they need to deploy it :/ [11:36:15] thanks for the check mforns !! [11:36:52] elukey, I could look at the other error, it has some frequency too, and ping their owners [11:36:59] will do that [11:51:19] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668022 (Milimetric) quick reminder: User Agents and browsing activity to... [11:52:05] (just made a note on the phab task but remember not to copy paste raw data from event logging. Even though we don't have IPs any more, the user agent / page name combination or even the user agent by itself is sensitive) [11:54:39] cool milimetric [12:03:00] * elukey lunch! [12:15:22] (still debugging a OCG issue) [13:25:59] Analytics: EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196#2684286 (mforns) [13:26:39] thanks mforns --^ !! [13:26:48] elukey, np :] [13:30:03] OCG? Original code gangsters? [13:39:47] ahahah [13:40:02] milimetric: https://wikitech.wikimedia.org/wiki/OCG [13:45:38] Analytics, Wikipedia-Android-App-Backlog: EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196#2684329 (Dbrant) [15:01:53] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (matmarex) YAML is nice for writing it (you have to worry less about quotes and strings and escape characters), bu... [15:03:47] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Deploy hive-site.xml to HDFS separately from refinery - https://phabricator.wikimedia.org/T133208#2684530 (Nuria) Open>Resolved [15:04:16] Analytics-Kanban: Scale MySQL edit history reconstruction data extraction - https://phabricator.wikimedia.org/T134791#2684532 (Nuria) [15:04:18] Analytics-Kanban: Productionize scala code for edit reconstruction - https://phabricator.wikimedia.org/T142552#2684531 (Nuria) Open>Resolved [15:04:29] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2684537 (Nuria) Open>Resolved [15:04:46] Analytics-Kanban: Missing pageviews dumps files for October 1 - 26 - https://phabricator.wikimedia.org/T146029#2684539 (Nuria) Open>Resolved [15:05:06] Analytics-Kanban, Patch-For-Review: Productionize Pivot UI - https://phabricator.wikimedia.org/T138262#2684542 (Nuria) [15:05:08] Analytics-Kanban, Patch-For-Review: Upload the final version of the pivot repo and load test - https://phabricator.wikimedia.org/T146389#2684541 (Nuria) Open>Resolved [15:10:47] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [15:13:27] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [15:32:08] Analytics, Wikipedia-Android-App-Backlog: EventLogging sees MobileWikiAppFindInPage parsing errors - https://phabricator.wikimedia.org/T147196#2684627 (Milimetric) a:Dbrant [15:33:12] Analytics, Analytics-Dashiki: Create a Universal Layout for Dashiki for staging / testing config purposes - https://phabricator.wikimedia.org/T147009#2677693 (Milimetric) p:Triage>Low [15:33:27] Analytics, Analytics-Dashiki: Dashiki (layout fixes) mostly mobile - https://phabricator.wikimedia.org/T147007#2677640 (Milimetric) p:Triage>Normal [15:35:36] Analytics-Kanban: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2684654 (Milimetric) [15:35:39] Analytics: Quantify false positives when filtering for number of distinct user agents per page in top pages computation - https://phabricator.wikimedia.org/T146911#2674479 (Milimetric) [15:35:44] Analytics: Quantify false positives when filtering for number of distinct user agents per page in top pages computation - https://phabricator.wikimedia.org/T146911#2674479 (Milimetric) [15:35:46] Analytics-Kanban: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2618003 (Milimetric) [15:36:38] Analytics, Analytics-Dashiki: Add external link to tabs layout - https://phabricator.wikimedia.org/T146774#2670812 (Milimetric) p:Triage>Normal [15:36:50] Analytics, Analytics-Dashiki, Easy: Add external link to tabs layout - https://phabricator.wikimedia.org/T146774#2684660 (Milimetric) [15:42:48] Analytics, Analytics-EventLogging, Spike: {Spike} Design where to specify purge schedule for schema {oryx} - https://phabricator.wikimedia.org/T88646#2684669 (Milimetric) Open>Resolved a:Milimetric We have a comprehensive talk-page driven purge policy now. [15:42:50] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#2684672 (Milimetric) [15:43:11] Analytics, Analytics-Dashiki, Analytics-Visualization: Build reactive filters on wiki and date in Dashiki - https://phabricator.wikimedia.org/T88375#2684673 (Milimetric) Open>Invalid [15:43:22] Analytics, Analytics-Dashiki, Analytics-Visualization: Build low level visualization of the paths through the application (starburst) in Dashiki - https://phabricator.wikimedia.org/T88374#2684674 (Milimetric) Open>Resolved a:Milimetric [15:43:41] Analytics, Analytics-Dashiki, Analytics-Visualization: Build high level timeseries view of key metrics (save / ready rate) in Dashiki - https://phabricator.wikimedia.org/T88373#2684676 (Milimetric) Open>Resolved a:Milimetric [15:46:25] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2684678 (bd808) >>! In T147158#2684521, @matmarex wrote: > I honestly don't think introducing YAML content would further y... [15:48:12] Analytics, Analytics-EventLogging: Add ops-reportcard dashboard with analysis that shows the http to https slowdown on russian wikipedia - https://phabricator.wikimedia.org/T87604#2684683 (Milimetric) Wondering whether this dashboard solves this problem: https://grafana.wikimedia.org/dashboard/db/navigat... [15:50:17] Analytics: Productionize Pageview_sanitization hive code with Oozie job and refinery inclusion {hawk} - https://phabricator.wikimedia.org/T118839#2684692 (Milimetric) [15:50:19] Analytics, Analytics-Cluster: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066753 (Milimetric) [15:50:46] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#2684694 (Milimetric) [15:50:48] Analytics, Analytics-EventLogging: Make DB logging configurable in the schema {oryx} - https://phabricator.wikimedia.org/T87177#2684693 (Milimetric) Open>Invalid [15:53:53] Analytics, Analytics-Cluster, Epic: Epic: qchris transition - https://phabricator.wikimedia.org/T86135#2684703 (Milimetric) Open>Resolved a:Milimetric We decided to join the geowiki project into the edit history / wikistats 2.0 project. So the remaining MaxMind task will stay open but th... [15:54:54] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#2684715 (Milimetric) [15:54:56] Analytics, Analytics-EventLogging: Engineer finds events that fail validation as early as possible {oryx} - https://phabricator.wikimedia.org/T78355#2684712 (Milimetric) Open>Resolved a:Milimetric Debugging steps: locally in vagrant on the beta cluster with the logstash EventLogging errors d... [15:55:30] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#819854 (Milimetric) [15:55:46] Analytics, Analytics-EventLogging: Analytics Eng has capacity & monitoring for EventLogging {oryx} - https://phabricator.wikimedia.org/T76803#2684716 (Milimetric) Open>Resolved a:Milimetric [15:56:08] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#819854 (Milimetric) Open>Resolved a:Milimetric Marcel did a review and we've been updating docs, we feel they're complete [15:57:45] Analytics: Process new PV hourly dumps for Vital Signs {hawk} - https://phabricator.wikimedia.org/T94592#2684726 (Milimetric) [15:57:47] Analytics, Analytics-Cluster, Story: Story: AnalyticsEng has hourly PageView counts {hawk} - https://phabricator.wikimedia.org/T69804#2684724 (Milimetric) Open>Resolved a:Milimetric [15:58:59] Analytics: Oozie sends emails when any job fails {hawk} - https://phabricator.wikimedia.org/T114901#2684729 (Milimetric) Open>Resolved a:Milimetric [15:59:28] Analytics, Analytics-Dashiki: Upgrade Dashiki to semantic-2 for all layouts - https://phabricator.wikimedia.org/T125409#2684731 (Milimetric) Open>Resolved a:Milimetric [16:01:13] Analytics: Visualization of Zika access by geography - https://phabricator.wikimedia.org/T125856#2684747 (Milimetric) Open>Resolved Working with external researches to get this data. [16:02:20] Analytics, Documentation: Clean up and possibly refine UDP sampled logs (which go back to 2014) - https://phabricator.wikimedia.org/T114381#2684752 (Milimetric) [16:05:22] Analytics, Documentation: Clean up and possibly refine UDP sampled logs (which go back to 2014) - https://phabricator.wikimedia.org/T114381#2684758 (Milimetric) [16:05:43] mforns: let's talk about correction of tasks a bit later today cc milimetric [16:05:47] Analytics, Documentation: Clean up and possibly refine UDP sampled logs (which go back to 2014) - https://phabricator.wikimedia.org/T114381#1693705 (Milimetric) [16:41:08] hey milimetric [16:41:15] yes sir [16:41:24] what do you prefer for me to take right now, resumes or review your code? [16:42:12] no rush on either one, mforns, so whichever you like. I guess resumes we have plenty, and reviewing the code could bring the improvements to people faster (the dashboards) [16:42:24] so I'd do the code if I was me, but I'm not you :) [16:43:15] cool [16:43:26] thx :] [17:03:30] mforns, milimetric : want to sync up about tasks correction? [17:07:24] nuria_ / mforns: I haven't had lunch and I'm getting hangry, I think I'm going to skip our recruiting meeting and do that [17:07:33] milimetric: sounds good [17:07:44] ok! [17:07:59] no big rush on the tasks, we'll have plenty, but I have some thoughts from when I last evaluated these queue ones [17:08:11] we can meet later today maybe? Or tomorrow morning [17:12:16] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2685104 (Mooeypoo) The fix that was submitted (and that will make sure th... [17:15:01] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2685118 (Doc_James) Hey Milimetric can we get data for Sept for WPMED? Not sure if it is avaliable yet. Also by NDA you mean "non disclosure agreement"? [17:15:37] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2685119 (Doc_James) I update this table as I get it https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Popular_pages [17:16:10] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2685122 (Mooeypoo) >>! In T146674#2684070, @Milimetric wrote: > quick rem... [17:22:12] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668022 (Catrope) @Nuria The patch has been merged, but could not be depl... [17:34:14] nuria_, aren't we having the recruiting meeting? [17:51:29] mforns: milimetric couldn't make it, let's try tomorrow [17:55:57] Analytics-EventLogging, ArchCom-RfC, Discovery, Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#2683091 (Esanders) JSON with comment stripping seems reasonable. We can still provide pure JSON via a parameter. I don't t... [17:56:19] nuria_, if you want to reschedule to today before 11:30, I'm OK with that, I'll be here [17:56:27] mforns: sorry, you mean our sync up with recruiting. [17:57:03] nuria_, yes, I meant that one [18:05:47] mforns: let's reschedule rather [18:05:53] nuria_, ok [18:14:35] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2685397 (Nuria) @Doc_James : i think @milimetric was going to do this as a 1 off query but the intent was not to repeat it every month until we have a process by which we can run these queries automatically. I m... [18:18:30] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016, Patch-For-Review, WMF-deploy-2016-10-04_(1.28.0-wmf.21): EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2685441 (Nuria) @Catrope: nah, it doesn't. My mistake, I was under the im... [18:39:08] Analytics-Kanban: Special characters showing up as question marks in /pageviews/top endpoint - https://phabricator.wikimedia.org/T145043#2685512 (MusikAnimal) At least for the examples I provided, this should be resolved by T146496, I believe [18:40:07] Analytics-Kanban, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2685531 (Nuria) [18:40:29] Analytics-Kanban, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2649398 (Nuria) a:schana>Nuria [18:42:33] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2685540 (Milimetric) Hi @Doc_James, yes, I only intended to run this manually once. We have plans to prioritize the long-term version of this in Q3 (starting January 2017). Until then, we're busy with editing d... [18:46:47] Analytics, MediaWiki-API, User-bd808: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#2685565 (Milimetric) Same offer to help with oozie applies to @Tgr, of course, and bonus: I'm now a lot better at setting up oozie jobs. [18:47:28] nuria_ / mforns: I'm back if yall wanna chat about any recruiting stuff [18:47:33] milimetric, sure [18:48:15] I don't have anything myself, just if you wanted to make-up for our missed meeting [18:48:51] milimetric: I spoked to recruiting ( and apologized again). I had alredy looked yesterday at all screenings that were done and provided feedback [18:48:59] me neither, except nuria_ wanted to sync-up on how to evaluate tasks [18:49:11] ok, you wanna do that now? [18:49:12] milimetric: so i think the only thing is for us to sync up on tasks [18:49:17] k [18:49:27] sure [18:49:35] give me 15 minutes [18:49:41] np [18:49:43] ok [18:50:16] Analytics-Kanban: Make top pages for WP:MED articles - https://phabricator.wikimedia.org/T139324#2685584 (Doc_James) Okay thanks. @Ladsgroup do you think you could run September for us? [19:08:03] mforns, milimetric : can sync up about tasks now, do you have time? [19:08:09] nuria_, yes [19:08:13] yes [19:08:15] to the cave [19:08:19] ok [19:41:56] (CR) Mforns: [C: 2 V: 2] "Amazing!" [analytics/dashiki] - https://gerrit.wikimedia.org/r/312430 (https://phabricator.wikimedia.org/T118846) (owner: Milimetric) [19:46:12] really better, mobile layout looks awesome [19:47:57] mforns: it's not as good on actual mobile as on the simulator, but it might just be a small browser difference. Either way flexbox is so easy and fun to work with [19:48:23] mforns: lemme know if you run out of S and we can split the rest of the letters [20:35:45] Analytics-EventLogging, MobileFrontend, Reading-Web-Backlog, Reading-Web-Tech-Debt, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#2685940 (MBinder_WMF) [21:00:56] Analytics, Analytics-EventLogging: Add ops-reportcard dashboard with analysis that shows the http to https slowdown on russian wikipedia - https://phabricator.wikimedia.org/T87604#2686016 (Milimetric) Notes from Ori: * this would be interesting to see, but even though it's pretty easy to get the data br...