[09:16:26] good morning a-team! [09:17:03] holaaaa [09:19:32] (03PS6) 10Jonas Kress (WMDE): Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) [09:32:44] a-team: as FYI grafana is going under maintenance for T170150 [09:32:45] T170150: Evaluate Grafana's LDAP group options and deprecate grafana-admin if possible - https://phabricator.wikimedia.org/T170150 [09:36:00] (03PS1) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [09:38:00] (03CR) 10Sahil505: [C: 04-1] "This requires improvement." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [10:23:22] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4318615 (10fdans) >>! In T190856#4303045, @mforns wrote: > `geowiki_archive_monthly_edits_country` Looks good to me overall as well. > There's anothe... [10:34:15] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4318703 (10fdans) >>! In T190856#4303158, @mforns wrote: > In Superset the 'Geowiki legacy archive' dashboard works well and shows correct data. > I ha... [10:35:10] * elukey afk for ~2h! [10:56:08] Hi fdans :) [10:56:24] hellooooo joseph! [11:33:23] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4272223 (10Jdrewniak) a:03Jdrewniak [12:29:37] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4319113 (10mforns) @fdans > Yes, it's better for the map component in Superset to have standard country names, and the names in the original dataset... [12:30:24] hey everyoneeeee [12:30:46] (03CR) 10Joal: "Comments inline - I think some changes are needed :)" (033 comments) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/440136 (https://phabricator.wikimedia.org/T188928) (owner: 10Fdans) [12:57:32] !log set Piwik in maintenance mode as prep step for backup + upgrade [12:57:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:08:01] !log piwik upgraded to 3.2.1 on bohrium + started the db migration procedure (will last 2/3h probably) [13:08:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:11:13] 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4319248 (10elukey) @jcrespo, @Marostegui: I'd really like to get your opinion on this if you have time. What I am currently trying to do is get rid of the single point of failures in the Analytics infrastructure, and a... [13:19:26] 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4319265 (10jcrespo) > In this case, we'd like to set up an automatic failover if possible The dbproxy roles should be generic enough, for writes and reads respectively (one is supposed to point to a master, and fails... [13:23:37] 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4319273 (10jcrespo) If you need more advanced proxying, we also have proxysql, but only the module and debian package, without a lot of actual deploys (needs more work, but it is supposed to be on production soon): ``... [13:29:39] 10Analytics: Add a safe failover for analytics1003 - https://phabricator.wikimedia.org/T198093#4319297 (10jcrespo) Unlike other services, actually I wouldn't have any issue with just moving this to db misc cluster- basically owning everything at db layer, from high availability to backups. I would see, however,... [13:57:58] (03PS2) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [14:42:32] (03PS3) 10Mforns: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [14:51:53] (03CR) 10jerkins-bot: [V: 04-1] Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [14:54:53] so piwik is still altering tables [14:55:01] we are half way through [14:55:41] 10Analytics: [Wikistats Design] integrate browser dashboard data into wikistats - https://phabricator.wikimedia.org/T198333#4319515 (10Nuria) [15:10:04] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4319559 (10Jdrewniak) Looking over the task description Inside the Popups code,... [15:13:13] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4319571 (10Ottomata) @Jdrewniak, I think it'd be fine to modify the eventlogging co... [15:18:49] (03PS4) 10Fdans: Add glue code to turn "ceiled" pageview values into intervals [analytics/aqs] - 10https://gerrit.wikimedia.org/r/440136 (https://phabricator.wikimedia.org/T188928) [15:26:43] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4319599 (10phuedx) >>! In T196904#4319559, @Jdrewniak wrote: > The alternative to th... [15:27:59] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4319601 (10Ottomata) Ya fine with me too. A lot of these bits will likely get refac... [15:32:42] AH [15:45:58] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4319665 (10Jdlrobson) What you say sounds right. That said the 1000 char limit does... [15:53:23] 10Analytics: Reimage thorium to Debian Stretch - https://phabricator.wikimedia.org/T192641#4319680 (10Ottomata) Luca and I just discussed, and decided that we should upgrade thorium to stretch anyway, and then later think about moving sites elsewhere. [15:53:37] 10Analytics, 10Analytics-Kanban: Reimage thorium to Debian Stretch - https://phabricator.wikimedia.org/T192641#4319681 (10Ottomata) [15:58:42] a-team - I have the "Ocercoming unconscious bias" session tonight [15:58:51] k joal [15:58:56] Had we planned on having goals sessions, or tomorrow? [16:02:31] joal, it will be tomorrow in groskin [16:02:40] Great - hanks mforns [16:02:44] :] [16:06:59] (03PS1) 10Fdans: Release 2.3.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442334 [16:08:53] (03PS4) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [16:16:19] (03PS5) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [16:19:15] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4319731 (10fdans) > We could document this detail on-wiki no? The way of estimating monthly values is documented in the wikitech page (https://wikitec... [16:24:24] (03CR) 10Fdans: [C: 032] Release 2.3.1 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442334 (owner: 10Fdans) [16:47:49] 10Analytics, 10Analytics-Kanban: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4319792 (10mforns) Oh yea, I meant the tiny inconsistency created by the normalization rounding. I added a one-liner to the docs: https://wikitech.wiki... [16:53:53] (03PS6) 10Sahil505: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) [16:54:51] (03CR) 10Sahil505: [C: 031] "Consider this as the final draft!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [16:55:49] very confusing [16:55:51] Executing ALTER TABLE `piwik_user` CHANGE `password` `password` VARCHAR(255) NOT NULL;... Done. [14 / 18] [16:55:54] then after a while [16:56:02] Executing ALTER TABLE `piwik_goal` ADD COLUMN `description` VARCHAR(255) NOT NULL DEFAULT '' AFTER `name`;... Done. [21 / 18] [16:56:19] what do the "18" mean then? [17:12:05] anyhow, there is a huge alter table in progress, the one related to the table that logs the piwik visits [17:12:12] so I suppose that it will take a bit [17:14:50] (03CR) 10Mforns: "I added Dan, Nuria and Fran as reviewers." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [17:24:20] (03PS1) 10Fdans: Release 2.3.1 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/442351 [17:49:21] (03CR) 10Fdans: [C: 032] Release 2.3.1 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/442351 (owner: 10Fdans) [17:49:42] (03CR) 10Fdans: [V: 032 C: 032] Release 2.3.1 [analytics/wikistats2] (release) - 10https://gerrit.wikimedia.org/r/442351 (owner: 10Fdans) [17:57:35] * elukey off! [17:58:05] fdans: Notice: /Stage[main]/Statistics::Sites::Stats/Git::Clone[wikistats-v2]/Exec[git_pull_wikistats-v2]/returns: executed successfully [17:58:24] should be live now [17:58:31] elukey: aw, thank you for letting me know luca [17:58:35] have a nice evening! [18:08:06] Hey fdans - currently looking at your last patch - Looks like there is a new unneded file, but except from that it's great :) [18:08:42] (03CR) 10Joal: [C: 031] "I think he testUtils.js file addition is not needed, but except from that all good :)" (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/440136 (https://phabricator.wikimedia.org/T188928) (owner: 10Fdans) [18:09:04] joal: omg there's always something I'm messing up [18:09:24] never do git add --all kids [18:09:34] fdans: I try in [18:09:59] again - fdans: I try indeed not to do it, because anytime I do I mess up (and still do it once in while) [18:10:06] (03PS5) 10Fdans: Add glue code to turn "ceiled" pageview values into intervals [analytics/aqs] - 10https://gerrit.wikimedia.org/r/440136 (https://phabricator.wikimedia.org/T188928) [18:10:09] nuria_: would you by any chance close-by? [18:10:24] that should be correct now joal, thank you so much for the CR [18:10:33] fdans: Thanks a mil for the patch :) [18:12:34] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4319996 (10aaron) I can't reproduce this in eval.php (after https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/442245/). [18:12:37] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4319997 (10aaron) I can't reproduce this in eval.php (after https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/442245/). [18:13:35] fdans: I +1ed it, works for me when you want :) [18:15:27] fdans: went over the patch for hive - looks good as well :) [18:15:33] fdans: Looks like yo're ready to go !! [18:23:40] 10Analytics, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog: As an end-user I shouldn't see non-articles in the list of top articles - https://phabricator.wikimedia.org/T124082#4320022 (10Mholloway) 05Open>03Resolved a:03Mholloway Services consuming the r... [18:31:53] joal: ok to merge both then? :) [18:55:44] (piwik is still altering tables, will check once in a while..) [18:58:02] fdans: ok for me - anything else to change? [19:05:31] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4320085 (10Jdrewniak) URIError can be avoided by searching for the last % charac... [19:09:51] (03PS1) 10EBernhardson: Partition keys containing ints can be strings [analytics/refinery] - 10https://gerrit.wikimedia.org/r/442369 [19:30:53] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4320172 (10Jdrewniak) ``` 'http://url...ウィキペディア'.split('').reduce( (url, x) => {... [19:59:38] joal: here! [19:59:43] Hi nuria_ :) [19:59:58] nuria_: Do you have aminute about history check (again) [20:00:00] ? [20:00:20] joal: yes, batcave? [20:00:23] yes! [20:05:58] (one big alter still ongoing from 3.7h ago.. [20:15:02] 10Analytics, 10Patch-For-Review, 10Wikipedia-Android-App-Backlog (Android-app-release-v2.7.24x-I-Ice-lolly): Include HTTP Referer header when navigating through internal links - https://phabricator.wikimedia.org/T192779#4320394 (10Dbrant) 05Open>03Resolved [20:33:07] FYI: https://phabricator.wikimedia.org/T198190 [20:33:22] Is a data access request -- looking to geocode editors who are contributing to German Wikipedia [20:33:33] In order to target outreach efforts. [20:33:40] (From a chapter organizer) [20:34:19] I'm not sure if y'all have reporting tools for supporting this kind of request or if accessing the DB directly and geocoding IPs is the answer. [20:34:33] halfak: now that milimetric has revamped geoeditors, I think we have the tools at hand to help [20:34:41] \o/ [20:34:51] Cool. I shall subscribe him :) [20:34:59] halfak: it'll still need work, but at least (some) data is present on the cluster [20:35:24] hm - actually halfak - We only store 90 days of those, given privacy policy restriction [20:35:33] But, it's on the cluster for those 90 days [20:35:33] Totally reasonable. [20:35:39] That's more than the recentchanges cluster [20:35:42] *table [20:35:51] should be similar [20:56:11] (still running...) [21:14:29] halfak: restrictions on our data are same than recenchanges cluster [21:14:37] halfak: we import and delete on same schedule [21:15:05] Oh so 30 days? [21:16:16] halfak: let me triple check cause there will be a buffer of time to recompute so it might be 60 [21:19:10] halfak: quite honestly besides geowiki we do not have any other data on that regard, isn't recentchanges data available on labs cluster? [21:21:43] halfak: we delete it every 90 days but what we have is basically the recentchanges stream, is that available on labs? [21:23:41] Woops. Looks like my living room gets bad wifi [21:25:19] halfak: I was about to comment on wiki but i really do not see why would this need access to prod data given labs new replicas , right? [21:25:41] You can't get IPs for registered editors on the labs replicas :) [21:32:27] halfak: I see, how about we give this user the geoeditors report for german wikipedia? I think that should be sufficient, no? (editor counts per wiki per country) [21:32:46] Seems like it might be. You should ask him. I'm just helping out :) [21:33:49] halfak: ok, i do not think we will grant access to data rather we can grant access to reports. [21:41:11] (03CR) 10Nuria: "Per our conversation:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [21:51:16] !log piwik maintenance completed [21:51:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:51:27] nuria_: o/ - piwik back in service if you want to check [21:51:30] (03CR) 10Nuria: Added pagecounts metric (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [21:51:41] elukey: will cehck! [21:52:42] it took "only" 8 hours to complete [21:53:18] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298#4320609 (10elukey) Piwik maintenance just completed, the website is back in service with the new 3.2.1 version. [21:54:26] elukey: all looks good [21:54:43] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: Sample code dialog has extraneous margin - https://phabricator.wikimedia.org/T198269#4320613 (10Krinkle) [21:55:52] * elukey afk again :) [22:12:24] (03CR) 10Nuria: [C: 031] "Looks good, let's make sure we do a test run of this before loading in cassandra. Please remember that aqs glue code needs to be deployed " (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/440145 (https://phabricator.wikimedia.org/T188928) (owner: 10Fdans) [22:16:15] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4320652 (10Krinkle) Hm.. well, the load.php url is still experiencing the same issue. I also see the same happen on CL... [22:33:54] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4320660 (10aaron) How is $cache defined? [22:41:20] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4320673 (10aaron) Maybe for certain keys, when the mc server used by nutcracker matches that of the mcrouter one, add(... [22:41:25] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Regression: Beta Cluster: EventLogging/RemoteSchema unable to obtain memcached lock - https://phabricator.wikimedia.org/T198280#4320674 (10Krinkle) ```lang=php $s = new RemoteSchema('NavigationTiming', 18156125); $s->cache ``` From [source code]... [22:44:17] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Sample code dialog has extraneous margin - https://phabricator.wikimedia.org/T198269#4320685 (10Krinkle) 05Open>03Resolved [23:05:03] so IntelliJ IDEA keeps giving me error that it cannot resolve `import org.wikimedia.analytics.schemas.TestTimestampSchema;` in refinery-camus/…/TestAvroMessageTimestamps.java. `mvn package` works fine but when I try to build project in IDEA I get that. Does anyone have any suggestions for how to proceed? [23:06:50] specifically the problem is that `org.wikimedia.analytics.schemas` does not exist in IDEA's eyes for some reason. has anyone run into this issue before? [23:14:39] bearloga: that is a classpath issue in INTELJ , sounds like you need your tests resources in classpath [23:35:13] nuria_: I added `refinery-camus/src/test/resources` (is that what you were referring to?) to dependencies in the refinery-camus module (based on https://stackoverflow.com/questions/854264/how-to-add-directory-to-classpath-in-an-application-run-profile-in-intellij-idea) but no success :( [23:37:14] it cannot resolve symbol 'schemas' [23:39:22] bearloga: yes, i would look at classpath to make sure the package itself is there and that test folder sis also marked as a source folder [23:40:35] nuria_: do you have a preferred/recommended method of importing the refinery source repo into intellij in a way that everything gets set correctly? [23:41:37] bearloga: i just import it as a new project , that's it [23:45:39] nuria_: thanks, I'll try that! I think that's how I imported before. [23:50:35] nuria_: ok, left all the options default when importing and still the same issue [23:51:22] bearloga: probably a default import does not import your test folders as source folders you might want to add that [23:58:33] nuria_: it doesn't have any problems with all the other test folders, though. and even other tests like TestAvroSchemaEvolution.java are fine, just cannot resolve in TestAvroSchemaTimestamps.java. [23:59:11] https://usercontent.irccloud-cdn.com/file/VboypP3C/Screen%20Shot%202018-06-27%20at%204.57.51%20PM.png [23:59:44] nuria_: I tried ^ too because that's where TestTimestampSchema is but no go