[00:15:45] 10Analytics: Superset throwing up performance errors - https://phabricator.wikimedia.org/T231614 (10JKatzWMF) @Nuria it's not working on even 1 year. [01:26:34] 10Analytics, 10Research: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 (10leila) @Milimetric @mforns here is your update: * We have 6 responses to the survey. I know of one more which may come in in the next 14 hours. I'll close the survey after th... [07:35:59] good morning, I'm looking at the failed mediacounts [07:38:06] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10raja_wmde) Hi @Nuria, for me it's the same format as for Verena, my wikitech account log in user is: Raja Gumienny (WMDE) I just created it. Thanks, Raja [07:46:25] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10raja_wmde) @fdans I'm not sure if I have signed the NDA yet, I wanted to a while ago, but I don't know where or how to get the NDA. Thanks for giving me more information on this. Best,... [07:56:04] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Nuria) [07:59:10] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Nuria) Both of you need to sign NDAS, here is the info on how to do that: https://wikitech.wikimedia.org/wiki/Volunteer_NDA#Create_a_request [08:01:51] 10Analytics: Superset throwing up performance errors - https://phabricator.wikimedia.org/T231614 (10Nuria) Link provided does not work , can you please run it through a shorterner and repost? [08:28:00] a-team: apparently ottomata didn't get a chance to reset my virtual environments on the notebook servers...how can I get this done ASAP? Pretty much all my work has been blocked for two days now... [08:46:35] !log Fix mediacounts-archive SLA and kill-restart job [08:46:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:51:00] (03PS1) 10Joal: Correct mediacounts-archive SLA [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 [09:14:01] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10ema) >>! In T230772#5461321, @Nuria wrote: > I can see the cache-control: max-age=604800 , I think @ema needs to change something on his end so varnish /A... [12:19:48] heya teammm [12:22:44] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) I think things look good now: [nuriaruiz@nurieta][~/tips]$ curl -v https://piwik.wikimedia.org/piwik.js > piwik * Trying 91.198.174.192... % T... [12:24:30] 10Analytics, 10Research: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 (10mforns) Thanks @leila! I reviewed the survey responses yesterday, both on survey results and comments in the Phabricator task(s). There were good insights! I wanted to write... [12:34:56] neilpquinn: sorry about that, can you do the "sudo mv /home/$user/venv /home/$user/venv-old.$(date +%s)"? [12:36:43] neilpquinn: fastest way to get your problem fixed is piging ottomata on irc [12:36:54] neilpquinn: he is on east coast time though [12:38:48] (03CR) 10Nuria: Correct mediacounts-archive SLA (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 (owner: 10Joal) [12:57:45] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10mforns) I thought the 'hourly' in pageview_hourly meant aggregated hourly, not updated hourly. In general I would name a data set after what does it contain, rather than how it is processed or when it is updated. Now, edit_h... [13:03:36] (03CR) 10Mforns: [C: 03+1] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 (owner: 10Joal) [13:16:11] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10Ottomata) > I thought the 'hourly' in pageview_hourly meant aggregated hourly Me too! [13:17:34] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Jclark-ctr) @Cmjohnson removed 2nd dac cable yes it plugged into d7 xe-7/0/2 [13:26:26] joal: yt? [13:29:52] (03CR) 10Fdans: [C: 03+1] "What Nuria said, but looks good to me!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 (owner: 10Joal) [13:44:52] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) While the caching header is correctly served, when the request is in the context of the foundation website, Varnish is doing a pass: {F30222372,... [13:50:38] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) Inddeed! I see server-timing: cache;desc="pass" on reg chrome window as well, thanks @Gilles for catching that [13:53:37] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10ema) >>! In T230772#5464917, @Gilles wrote: > I'm guessing it might be coming from the cookies? Which the Chrome developer tools weren't showing. We've ha... [13:55:40] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) I think the issue is that misc_recv_pass is applied to every site. You want it to apply to wikis (where people can log in), but not on non-wiki we... [13:56:27] a-team fyi! neil and i just figured out that users can reset their own swap venvs [13:56:29] https://wikitech.wikimedia.org/wiki/SWAP#Resetting_user_virtualenvs [13:56:31] updated ^ [13:56:39] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) Or I guess, to be conservative, add Wikimedia wiki login cookies to the filter only if you're in the context of a non-wiki site. [13:57:11] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Gilles) This should probably be its own task, though, it's not specific to piwik.js [13:58:56] ottomata: sounds good , thank you [14:04:31] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) Right this is an issue for all resources served for https://stats.wikimedia.org/v2 and, I am gusessing, other non wiki domains. Can we consider b... [14:04:32] yes, thank you ottomata! thrilled that we can do it ourselves and it now fully documented [14:04:36] :) [14:18:47] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Ottomata) Too bad this is .NET. http://schemastore.org/json/ https://github.com/schemastore/sc... [14:25:09] Hello analytics, I have a request to add KaiOS to the list of OS query options for pageviews in Turnilo. I assume there is some rule somewhere that assigns an OS value based on the user-agent. Is this the basic set-up? [14:25:53] If so then I'll advise her to open a ticket and tag analytics, but if it's more technically complicated then maybe I'll have to direct her over to y'all to chat some more about it? [14:26:54] hip: iiuc, use use ua-parser regexes for that stuff: https://github.com/ua-parser/uap-core [14:27:29] we haven't updated our version of it since april 2018, so perhaps it has been added there already? [14:31:03] hmm, looks like they don't have a rule for it in their repo, is there any kind of extension system where we could add a regex without having to fork or submit a PR? [14:31:26] (or upgrade) [14:31:54] we probably could code that in by applying a regex a second time, but the best thing to do would be to get it upstreamed. I wouldnt' be surprised if they would accept patches [14:32:23] I agree, but in that case we'd have to be up for upgrading to that patch version, is that feasable do we think? [14:35:18] I'll make a ticket and ppl can follow up there async [14:45:48] yeah i think so. [14:46:07] a ticket is a good idea, and we have upgraded ua-parser to get new changes their before, i think we could do it again [14:46:38] awesome, thanks for the quick response [14:56:22] 10Analytics, 10New-Readers: Add KaiOS to the list of OS query options for pageviews in Turnilo - https://phabricator.wikimedia.org/T231998 (10jlinehan) [15:05:03] 10Analytics, 10Research: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 (10leila) @mforns great. (I just checked and all answers that I expected are in.) I'll assign this task to you for now, let me know if I can help somewhere. [15:05:12] 10Analytics, 10Research: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 (10leila) a:05leila→03mforns [15:37:52] 10Analytics, 10New-Readers: Add KaiOS to the list of OS query options for pageviews in Turnilo - https://phabricator.wikimedia.org/T231998 (10SBisson) It makes sense to me. My Alcatel Go Flip 2 phone reports `Mozilla/5.0 (Mobile; ALCATEL40440; rv:37.0) Gecko/37.0 Firefox/37.0 KaiOS/1.0` By scanning the rege... [15:51:24] (03PS2) 10Joal: Correct mediacounts-archive SLA [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 (https://phabricator.wikimedia.org/T228747) [15:53:36] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534388 (https://phabricator.wikimedia.org/T228747) (owner: 10Joal) [15:54:32] Awesome trick on reseting virtual-envs ottomata! Thanks for the docs :) [15:55:14] !log Deploy refinery using scap (fix for SLAs) [15:55:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:28:17] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) The eventgate-main config now includes the new brokers i... [17:33:34] for anyone interested in privacy: https://brave.com/google-gdpr-workaround/ [19:11:11] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) [19:11:32] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Ottomata) [19:23:16] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (watching): CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) Petr and I met today to discuss. We decided that the idea of using a meta schema to validate... [19:28:53] 10Analytics, 10Product-Analytics: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529 (10Ottomata) 05Open→03Resolved a:03Ottomata The subtasks here are closed, closing this ticket as well. Feel free to reopen for more tasks or discussion. [19:29:01] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) [20:03:05] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) We are finally starting to work on {T206814}, so we can start to think about some other conventions we mig...