[00:35:39] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) @lexnasser let's do a bit more testing, from my tries with the \x{n] notation it does not work on a string context (fails at runtime) . Adding... [00:36:42] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) @awight: this is indeed a small projects without any aim to change mediawiki code [00:47:52] 10Analytics, 10Privacy Engineering, 10Product-Analytics, 10Security-Team, and 2 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Nuria) [00:48:33] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10lexnasser) @Nuria This is the code I'm using: `Pattern.compile("^[ %!\"$&'()*,\\-.\\/0-9:;=?@A-Z\\\\^_`a-z~\\x{80}-\\x{10FFFF}\\+]+$");` It has a f... [00:50:32] (03CR) 10Nuria: [C: 03+1] "+2 on my end but waiting for sam to confirm. This would not delete all data on events_sanitized and will require some manual deletion. Thi" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588105 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [01:06:37] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) @lexnasser ok, changing those fixed it, just tested that it works on scala 2.11 and java1.8 (scala uses java's regexes but just to be extra sur... [01:37:16] 10Analytics, 10Privacy Engineering, 10Product-Analytics, 10Security-Team, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Peachey88) [03:57:02] (03CR) 10Phuedx: [C: 03+1] "I think having the flexibility of being able to query a dataset is preferable." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588105 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [07:36:28] he elukey, just wrote you an email. Problem solved :) [07:38:38] dsaez: nice! [08:03:21] helloooo team [08:15:01] hey fran :) [08:19:09] (03PS1) 10Amire80: Use "pages" instead of "articles" consistently [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/588369 [08:25:18] 10Analytics, 10Pageviews-API, 10Pageviews-Anomaly: "Venuše (planeta)" on cs.wp has surprisingly high numbers in Pageviews Analysis (and also Topviews Analysis) - https://phabricator.wikimedia.org/T239532 (10matej_suchanek) [08:38:52] fdans: just fixed the refine alert left from last week [08:40:05] going afk, if needed ping me on the phone :) [08:40:08] elukey: awesome, thank you Luca, now get out of here and enjoy your day off [10:05:39] Hi team - Happy Easter :) day off for me with kids and chocolate :) [10:54:00] joal: I'm suffering from MAJOR chocolate envy right now [10:54:06] no chocolate in this household [12:42:33] (03PS1) 10Fdans: Handle punctuation chars in paths for mediarequests per file [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) [13:35:17] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Ottomata) Not sure if this is related, but just in case: {T219279} [13:37:17] (03PS1) 10Fdans: Correct typo in mobile metric areas i18n [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/588408 (https://phabricator.wikimedia.org/T247725) [13:47:50] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto dashboards on superset - https://phabricator.wikimedia.org/T249923 (10dr0ptp4kt) @elukey that worked, yes - thanks. [14:02:09] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10awight) >>! In T245468#6051656, @Ottomata wrote: > Not sure if this is related, but just in case: {T219279} Thanks for the tip! Luckily, I don't thi... [14:50:20] 10Analytics, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10Nuria) hold on, cause i think @dr0ptp4kt will be able to use superset to view this data, @dr0ptp4kt let us know otherwise [14:53:35] (03CR) 10Nuria: [C: 03+2] eventlogging: Remove unused props from PrefUpdate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588105 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [14:53:37] (03CR) 10Nuria: [V: 03+2 C: 03+2] eventlogging: Remove unused props from PrefUpdate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588105 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [14:55:02] (03CR) 10Nuria: "The amount of glue code in this api to make up for shortcomings of loading or processing is getting to be a bit much, let's discuss." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [15:01:10] ping ottomata , standup [15:03:29] IUH OH [15:13:48] 10Analytics, 10Privacy Engineering, 10Product-Analytics, 10Privacy, and 2 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Dsharpe) reviewed in Clinic [15:18:38] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Milimetric) p:05Triage→03High a:03fdans [15:20:51] 10Analytics, 10Analytics-Wikistats: Wikistats New Feature-Country pageview breakdown by language - https://phabricator.wikimedia.org/T250001 (10Milimetric) p:05Triage→03Medium Thanks for the request, we have to triage it medium as we have other important infrastructure to get to first. But good idea, than... [15:23:25] 10Analytics, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10Milimetric) [15:23:30] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10Milimetric) [15:26:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10Milimetric) [15:27:26] 10Analytics, 10Analytics-Data-Quality, 10Analytics-EventLogging, 10WikiEditor, 10Mobile: WikiEditor records all edits as desktop edits in EventLogging - https://phabricator.wikimedia.org/T249944 (10Milimetric) needs to be fixed in instrumentation [15:28:30] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10Milimetric) p:05Triage→03High [15:29:14] 10Analytics: Explore in jupyter notebook whether the raw pageview timeseries can help on outage/censhorsip automatic detection - https://phabricator.wikimedia.org/T249849 (10Milimetric) p:05Triage→03Medium [15:29:41] 10Analytics: Explore in jupyter notebook whether the raw pageview timeseries can help on outage/censhorsip automatic detection - https://phabricator.wikimedia.org/T249849 (10Milimetric) ping @mforns to explore later with Nuria [15:32:38] 10Analytics: Superset: "Error while fetching database list" - https://phabricator.wikimedia.org/T249825 (10Milimetric) We just tried it in FF and couldn't reproduce. Can you confirm with someone from your team that should have the same access, like @srishakatux? [15:34:01] 10Analytics, 10Analytics-Wikistats: Wikistats New Feature-Country pageview breakdown by language - https://phabricator.wikimedia.org/T250001 (10Milimetric) [15:34:03] 10Analytics, 10Analytics-Kanban: Statement of work for new designer in wikistats - https://phabricator.wikimedia.org/T223478 (10Milimetric) [15:35:16] 10Analytics: Superset: Repeatedly asking to re-log in - https://phabricator.wikimedia.org/T249824 (10Milimetric) SQL Lab had some problems with Presto that we fixed, maybe the timing was unfortunate and you were using it during the fixes (for reference: T249923). Please try again and let us know. [15:35:41] 10Analytics: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Ottomata) [15:38:46] 10Analytics, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Milimetric) p:05Triage→03Medium Not sure when we can get to this, but do let us know if it's more urgent than we think. [15:41:17] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10Milimetric) p:05Triage→03Medium [15:42:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add hourly resolution to data quality outage/censhorship alarms - https://phabricator.wikimedia.org/T249759 (10Milimetric) p:05Triage→03High [15:42:26] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10Ottomata) Hm interesting! You are the first person aside from Joseph that I heard uses Scala in Jupyter! Can you try this on stat1005? IIUC There is a newer version of Toree (the Scala Spark... [15:43:43] 10Analytics: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Milimetric) p:05Triage→03High a:03Milimetric [15:46:28] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Combine filters and splits on wikistats UI - https://phabricator.wikimedia.org/T249758 (10Milimetric) p:05Triage→03High a:03fdans [15:47:09] 10Analytics, 10Cassandra: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10Milimetric) p:05Triage→03High [15:47:48] 10Analytics, 10Cassandra: Cassandra3 migration for Analytics AQS - https://phabricator.wikimedia.org/T249755 (10Milimetric) p:05Triage→03High a:03elukey [15:48:05] 10Analytics, 10Analytics-Kanban: Unify stat1007 puppet role with the rest of the stats cluster - https://phabricator.wikimedia.org/T249754 (10Milimetric) p:05Triage→03High [15:48:29] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Milimetric) p:05Triage→03Medium [15:49:22] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Define reduce calculations needed to compute active editors per project family - https://phabricator.wikimedia.org/T249751 (10Milimetric) p:05Triage→03High [15:51:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10dr0ptp4kt) We'd like to be able to use both Turnilo and Superset. As an aside, for Superset, what are the steps to ensure that a Presto-backed result set allows... [15:51:12] 10Analytics: Rewrite cassandra loading In spark - https://phabricator.wikimedia.org/T249735 (10Milimetric) p:05Triage→03Medium [15:52:55] 10Analytics, 10Analytics-EventLogging, 10Timeless: EventLogging revision popup gets hidden behind content in Timeless - https://phabricator.wikimedia.org/T249557 (10Milimetric) 05Open→03Declined We're migrating to putting schemas on https://schema.wikimedia.org/ and we won't be maintaining this. (I also... [15:55:22] 10Analytics, 10Growth-Team, 10Product-Analytics: Growth: validate that data is purged after 270 days - https://phabricator.wikimedia.org/T249666 (10Milimetric) p:05Medium→03High [15:56:18] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog: Develop test environment solution for MEP analytics events - https://phabricator.wikimedia.org/T238837 (10Milimetric) p:05Low→03High [15:57:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10Nuria) >We'd like to be able to use both Turnilo and Superset. then, let's just go ahead with this task and ingest the data on druid, there is no need to have... [15:57:42] 10Analytics, 10Analytics-Kanban: Delete raw events for mediawiki_ , the refined data is kept indefinately - https://phabricator.wikimedia.org/T245126 (10Milimetric) p:05Triage→03High a:03Ottomata [15:57:47] 10Analytics, 10Analytics-Kanban: Analytics Ops Technical Debt - https://phabricator.wikimedia.org/T240437 (10Milimetric) p:05Triage→03High [15:57:53] 10Analytics, 10Analytics-Kanban: Fix wikidata_item_page_link job - https://phabricator.wikimedia.org/T248228 (10Milimetric) p:05Triage→03High [15:58:03] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto dashboards on superset - https://phabricator.wikimedia.org/T249923 (10Milimetric) p:05Triage→03High [15:58:11] 10Analytics, 10Event-Platform, 10User-Elukey: Create EventStream's equivalent to irc.wikimedia.org's #central channel - https://phabricator.wikimedia.org/T240182 (10Milimetric) p:05Triage→03Medium [15:59:57] 10Analytics, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10Milimetric) [16:00:11] 10Analytics, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10Milimetric) [16:00:14] 10Analytics: Outdated project codes in pagecounts-ez - https://phabricator.wikimedia.org/T219914 (10Milimetric) [16:00:51] 10Analytics, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10Milimetric) p:05Triage→03High a:03fdans [16:29:27] (03CR) 10Nuria: "Per conversation we are going to document how encoding happens on current pipeline so we can decide where it makes sense to do encoding fi" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [17:09:57] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10nettrom_WMF) In the Product Analytics team we'd like to understand more about this task. Is background inform... [17:17:05] 10Analytics, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Isaac) > Not sure when we can get to this, but do let us know if it's more urgent than we think. @Milimetric totally understandable. We currently have the work... [17:55:02] (03Restored) 10Nuria: eventlogging: Purge prefupdate after 90 days [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588106 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [17:55:27] (03CR) 10Nuria: [C: 03+2] eventlogging: Purge prefupdate after 90 days [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588106 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [17:56:12] (03Abandoned) 10Nuria: eventlogging: Purge prefupdate after 90 days [analytics/refinery] - 10https://gerrit.wikimedia.org/r/588106 (https://phabricator.wikimedia.org/T249894) (owner: 10Krinkle) [19:35:28] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10awight) >>! In T249761#6052153, @Ottomata wrote: > Hm interesting! You are the first person aside from Joseph that I heard uses Scala in Jupyter! Honestly, it's only because I'm lazy and there... [19:38:22] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10Ottomata) > Minor side note: it seems like using jupyter directly on stat* machines is a normal practice? This is new; Luca is working on unifying all stat and notebook box configurations. We'l... [19:52:40] (03PS5) 10Ottomata: Unify Refine transform functions and add user agent parser transform [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) [20:01:30] (03PS6) 10Ottomata: Unify Refine transform functions and add user agent parser transform [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) [20:02:52] (03CR) 10Ottomata: "Ok @Joal @Nuria I think this is ready and finally works." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [20:10:21] !log remove deprecated mediawiki schema repository from schema.wikimedia.org [20:22:49] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Repository-Admins, 10Services (watching): Archive mediawiki/event-schemas repository from gerrit - https://phabricator.wikimedia.org/T250113 (10Ottomata) [20:25:36] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Create and use new schema repositories - https://phabricator.wikimedia.org/T240985 (10Ottomata) [20:25:48] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Repository-Admins, 10Services (watching): Archive mediawiki/event-schemas repository from gerrit - https://phabricator.wikimedia.org/T250113 (10Ottomata) a:05Ottomata→03None [20:26:02] 10Analytics, 10Event-Platform, 10Repository-Admins, 10Services (watching): Archive mediawiki/event-schemas repository from gerrit - https://phabricator.wikimedia.org/T250113 (10Ottomata) [20:30:44] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Create and use new schema repositories - https://phabricator.wikimedia.org/T240985 (10Ottomata) Phew we are done here yeehaw! [20:37:41] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10Cmjohnson) These are on 1G racks. If you need 10G they will have to be moved. [20:39:21] 10Analytics, 10Event-Platform, 10Projects-Cleanup, 10Repository-Admins, 10Services (watching): Archive mediawiki/event-schemas repository from gerrit - https://phabricator.wikimedia.org/T250113 (10Peachey88) [20:40:46] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10elukey) >>! In T244506#6053265, @Cmjohnson wrote: > These are on 1G racks. If you need 10G they will have to be moved. Yep we'd need 10G, but regardless... [20:48:54] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10elukey) > It's been impossible to run Spark Scala notebooks in SWAP for some time now. This would be wonderful to restore if possible. Can you please add more info about what does this mean? Wh... [21:20:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10leila) [21:27:22] 10Analytics: Investigate tools.wmflabs.org to toolforge.org migration - https://phabricator.wikimedia.org/T250116 (10Milimetric) [21:28:39] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review, 10User-Elukey: Add SWAP profile to stat1005 - https://phabricator.wikimedia.org/T245179 (10leila) I'm removing the Research tag. Please ping us if we can support in any way. [21:28:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add SWAP profile to stat1005 - https://phabricator.wikimedia.org/T245179 (10leila) [21:45:23] 10Analytics: Superset: "Error while fetching database list" - https://phabricator.wikimedia.org/T249825 (10srishakatux) I am not able to log into https://superset.wikimedia.org/ anymore :( I believe I'm using the same credentials I use to log into Wikitech with username `srishakatux`. [22:10:07] 10Analytics, 10Cite, 10Reference Previews, 10Research, and 2 others: Instrument Cite to record the nubmer of footnote marks and references list entries rendered in each article - https://phabricator.wikimedia.org/T241833 (10leila) I'm removing the Research tag as that's the one we use to track our team's t... [22:10:17] 10Analytics, 10Cite, 10Reference Previews, 10Patch-For-Review, 10User-awight: Instrument Cite to record the nubmer of footnote marks and references list entries rendered in each article - https://phabricator.wikimedia.org/T241833 (10leila) [22:11:05] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10awight) >>! In T249761#6053290, @elukey wrote: > Can you please add more info about what does this mean? What error do you get? How can we reproduce? Thanks for the nudge, I'll put more detail... [22:14:59] 10Analytics, 10Operations, 10Research, 10Traffic: Wikipedia Accessibility, check false positives and false negatives of traffic alarms - https://phabricator.wikimedia.org/T245166 (10leila) @Nuria do you need our team's support in any way for this task? (I'm reviewing our tasks in Staged.) [22:29:38] 10Analytics, 10Analytics-SWAP: Spark Scala kernel dying under Jupyter - https://phabricator.wikimedia.org/T249761 (10awight) [23:39:57] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10lexnasser) On @Milimetric 's suggestion, I tested all 3 methods against each other to verify their consistency, and found they all behaved the same ov...