[00:08:29] 10Analytics: x1-analytics-slave hangs forever - https://phabricator.wikimedia.org/T157514#3007801 (10Mattflaschen-WMF) [00:12:56] 10Analytics: x1-analytics-slave hangs forever - https://phabricator.wikimedia.org/T157514#3007827 (10Mattflaschen-WMF) [00:21:02] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3007849 (10Reguyla) I really appreciate you folks considering my request. If I may suggest, a lot of apps including Flickrtocommons and Commons Helper among others use Mediawiki as the source wiki... [01:48:50] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008015 (10bd808) >>! In T157342#3007849, @Reguyla wrote: > I really appreciate you folks considering my request. If I may suggest, a lot of apps including Flickrtocommons and Commons Helper among... [01:49:32] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008016 (10bd808) [02:05:20] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3008064 (10bd808) Switching the wiki contacted for the OAuth handshake would really be a game of whack-a-mole. Today someone is affected by a meta ban, tomorrow it will be someone affected by a mw... [03:42:57] 10Analytics: Import 2001 wikipedia data - https://phabricator.wikimedia.org/T155014#3008155 (10Graham87) >>! In T155014#3005643, @Milimetric wrote: > Why is that, @Graham87? I mean I see why a big gap might be confusing, but it seems better than an even bigger gap without those imported edits. Is there another... [05:24:10] 10Analytics: Add "Damn Small XSS Scanner" (DSXS) to list of known bots - https://phabricator.wikimedia.org/T157528#3008221 (10Tbayer) [05:24:33] 10Analytics, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#2887617 (10Tbayer) [[https://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&start=2016-07... [05:25:14] 10Analytics, 06Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#3008240 (10Tbayer) [05:25:16] 10Analytics: Add "Damn Small XSS Scanner" (DSXS) to list of known bots - https://phabricator.wikimedia.org/T157528#3008221 (10Tbayer) [05:43:56] (03PS1) 10HaeB: Add DSXS (self-identified bot) to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/336575 (https://phabricator.wikimedia.org/T157528) [08:01:16] mmmm oozie the complainer is back [08:05:58] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: aqs1009-a.eqiad.wmnet/10.64.48.122:9042 (com.datastax.driver.core.TransportException: [aqs1009-a.eqiad.wmnet/10.64.48.122:9042] Cannot connect)) [08:06:11] the firewall [08:06:13] grrrrrrr [08:06:17] why only now?? [08:08:39] or maybe aqs1009-a was just bootstrapped and acting weirdly? [08:10:18] so I tried from an1046 to telnet to aqs1009-a:9042 and it hangs (as expected), meanwhile aqs1006-a does not [08:11:16] trying to re-run 2371-07 Feb 2017 18:00:00 to see if it picks up another host [08:16:19] nope again the same error [08:16:25] it seems that 1009 is now the preferred one [08:22:48] it seems that all the cassandra loading jobs are trying to do the same [08:22:51] sigh [08:29:31] so theoretically I know what to do, practically I want some review before applying the new ACLs to the routers [08:33:12] opening a phab task [08:45:03] 10Analytics, 10Analytics-Cluster: Cassandra loading job are causing Pageview stale data - https://phabricator.wikimedia.org/T157533#3008425 (10elukey) [08:45:10] 10Analytics, 10Analytics-Cluster: Cassandra loading job are causing Pageview stale data - https://phabricator.wikimedia.org/T157533#3008440 (10elukey) p:05Triage>03High [08:59:04] https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Barak_Obama/daily/2017020100/2017020800 does not show yesterday [08:59:13] so yeah I'll send an email [09:04:56] ok email sent [09:24:33] 10Analytics, 10Analytics-Cluster: Cassandra loading job are causing Pageview stale data - https://phabricator.wikimedia.org/T157533#3008509 (10elukey) [09:25:07] Hi elukey [09:25:14] Tkanks for fast comm on this [09:25:53] I noticed yesterday evening, an hourly job had failed - I restarted it assuming it would be a retry-solveed issue (as most of those days) [09:25:58] Thanks for having found the real cause [09:26:32] However is seems bizarre to me that the thing complains for 1009-a anr not before for 1007 and 8 ... Weird [09:30:16] joal: o/ [09:30:19] maybe it is due to https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-cassandra/src/main/java/org/wikimedia/analytics/refinery/cassandra/CqlRecordWriter.java#L54-L59 ? [09:30:41] 1009-a is, for some reason, the primary replica [09:31:14] anyhow, I am waiting for Mark to review my network rules :) [09:31:27] elukey: I don't know if it's cassandra or the java code that decides on which instance is the primary rep, but if it's cassandra, we've been lucky with the first 4 new instances :) [09:31:36] ahhahah yes [09:31:38] sure elukey - Thanks a lot again for fast reactivity ! [09:32:14] elukey: please let me know when ok, I'll take care of restarting the jobs [09:32:15] :) [09:32:28] I am going to start bootstrapping 1009-b [09:32:48] pageviews are a bit delayed, not a huuuuuuge deal for the moment [09:33:30] elukey: I'm not worried by the small delay so far [09:34:18] (last famous words from Luca and Joseph, then the meltdown) [09:36:51] (03CR) 10Joal: [C: 031] "LGTM, waiting for Nuria approval" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/336575 (https://phabricator.wikimedia.org/T157528) (owner: 10HaeB) [09:49:06] joal: aqs1009-b is bootstrapping! [09:49:11] yay :) [09:49:14] LAST ONE!! [09:49:26] then clean up, then, SERVING TRAFFIC ! [09:49:58] elukey: I'm surprised by how smooth this all thing went (only one issue so far, and not even that big) [09:50:16] joal: don't say these things before the completion of the job :P [09:50:28] Ok elukey, I won't: ) [09:52:09] jokes aside, the network ACLs are a nightmare for us [09:52:36] since we rely on them to contact prod, each change to the infrastructure not reflected to them will cause problems [09:52:46] like this one, webproxy, etc.. [09:53:38] elukey: :( Any idea on how to approach a solution? [09:59:05] one idea might be to rely only on Ferm and get rid of the ACLs [10:15:29] 10Analytics-Tech-community-metrics, 07Upstream: When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#3008604 (10Aklapper) Example from the last 9 days: `f94c87a3d35b3ab9d92d5c0d83426505d34f1820` should have been merge... [10:16:11] 10Analytics-Tech-community-metrics, 07Upstream: When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#3008618 (10Aklapper) Last weekend, Jesus mentioned that this should be possible already and that it might "just" be... [11:22:57] * elukey lunch! [12:07:39] hi team :] [12:07:43] wow lots of alarms [12:07:47] Hi A-team [12:07:57] hey joal [12:07:58] yes mforns, cassandra unhappy [12:09:02] joal, do you already know what the problem is? [12:09:11] elukey knows (as usual) [12:09:16] :D [12:09:53] mforns: this time it wasn't me! :D [12:10:00] huhuhu :D [12:10:08] hehe elukey [12:10:08] I sent an email to analytics@ and opened https://phabricator.wikimedia.org/T157533 [12:10:12] ok [12:10:21] basically the network rules [12:10:48] the current theory is that the datastax driver picks up aqs1009-a as primary replica to load data from hadoop [12:10:53] but that IP is not whitelisted [12:12:06] elukey, can I help in whitelisting the IP? It's my ops week [12:13:31] mforns: I am waiting for Mark or Faidon to add the ACL to the routers, I know how to do it but I want somebody to review the changes [12:13:41] ops duty :( [12:13:42] (actually, in this case, I'll be of total non help) [12:14:01] ok [12:28:13] 10Analytics, 10Analytics-Cluster: Cassandra loading job are causing Pageview stale data - https://phabricator.wikimedia.org/T157533#3008848 (10elukey) To be on the safe side, we are going to wait for the network operations experts before proceeding in changing the ACLs on the routers, since this is not a criti... [12:53:08] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3008872 (10JAllemandou) Recap before closing and creating new actions: - Analytics VLAN had to access new labsdb one - Thanks again @elukey and @MoritzMuehlenhoff. - After networ... [13:53:30] 06Analytics-Kanban, 13Patch-For-Review: Add "Damn Small XSS Scanner" (DSXS) to list of known bots - https://phabricator.wikimedia.org/T157528#3009109 (10JAllemandou) [14:20:06] heya, anybody know how I can tell if I can email a list without being a memeber? [14:20:13] i've got 4 lists i'm not sure about [14:20:20] ottomata: no idea :( [14:20:43] milimetric: ^^ ? [14:20:45] ottomata: by the way, would we spend some time revamping tranquility thing to match mforns prod jobs? [14:21:16] ottomata: I also would like a quick brain dump on big data sets :) [14:21:34] oh! yeah we shoudl do that [14:21:35] ottomata: whenever you have time in the middle of the anouncmeent :) [14:21:43] ok cool, yeah gimme a few... [14:21:53] ping me in like 20 or 30 mins if i forget... [14:22:17] suere ottomata [14:43:09] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3009305 (10Marostegui) >>! In T155658#3008872, @JAllemandou wrote: > > - Load on SQL machines seemed very acceptable (to be confirmed with @jcrespo and @Marostegui ) Do you happen... [14:44:52] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3009323 (10Reguyla) I totally understand I just offered that as a possibility but it makes sense that it was used for early adopters. I hadn't realized that. [14:45:09] joal: bad news is that both our network engineers might be out today :( [14:45:46] elukey: That is a pain [14:46:31] I know what to do but I don't like operating on the routers without backups :D [14:49:28] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3009326 (10chasemp) >>! In T155658#3004775, @JAllemandou wrote: > @chasemp / @yuvipanda : Another question for you guys - There seems to be no rev_text_id in the DBs (0 everywhere).... [14:53:14] joal: batcave? [14:53:26] sure ottomata [14:57:37] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Create orgs and enroll DB identities who are likely only activity from imported upstream repos (to make it visible how incorrect our data potentially is) - https://phabricator.wikimedia.org/T157569#3009336 (10Aklapper) [15:04:12] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3009382 (10chasemp) > > https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/templates/labs/db/views/maintain-views.yaml;7678fccb897fd6912a75e3a162d338e... [15:04:59] joal: hey man, responded to your rev_text_id things, teh tldr is I don't know exactly [15:05:20] my first thought was that view statement was just bad and never really noticed but I imagine there is some intent there for not revealing deleted revisions only? [15:05:57] jynus and I kicked it around briefly and the conclusion was the path is the same to change it so hopefully you can drive that? [15:21:50] chasemp: is the text accessible from labs? Like, if someone had a rev_text_id, could they pull the revision text out directly without going through the API? [15:22:08] I don't know the answer there but we can find out pretty quickly I think [15:22:24] taht seems like a reasonable place to start [15:22:42] ottomata: Let's go ! [15:23:07] right, if that's possible then it makes sense to blank them all in labs. In that case we'd have to import them round-about from prod [15:24:11] milimetric: mine and jynus's thinking was something like, not matter the answer to that question to get this changed we'll have to get someone from security to say "yes" as it's possible (though very unlikely) that teh API is doing a bad thing there instead of vice versa [15:24:24] that's just for instance [15:24:53] makes sense [15:25:04] sorry was with ottomata chasemp [15:25:09] thanks milimetric for covering :) [15:25:16] chasemp: Thanks for answers ! [15:25:30] no worries man, I wish I understood the current situation better :) [15:25:58] also chasemp, any doc on how logging fileds are nuulyfied based on content? [15:26:06] this is a thing that was converted from an old version that was converted from a process one person knew and who is no longer here [15:26:16] joal: you mean the trigger layer at sanitarium? [15:26:27] hm yeah there are not docs but it's easy to spit out the sql that does it [15:26:32] if I can recall where/how [15:26:36] huhu [15:26:53] chasemp: if this code is in gerrit for instance, just link me to it :)b [15:27:47] I don't see it, and I've only done this via jaime with him driving [15:27:56] (and only on teh rare occassion of a new wiki db) [15:28:00] I'll track it down [15:28:25] ooh, congrats on the official Cloud team [15:28:58] joal: let's go means restart tranquility? [15:29:12] I was wondering, instead of saying "can I get xyz from labsdb" would I now say "can I get xyz from our cloud"? [15:29:15] please ottomata, yes, it was unclear :) [15:29:18] k :) [15:29:26] milimetric: you have to do a rain dance now [15:29:34] joal: just added the new rules on the routers, restarted one job to see if it works :) [15:29:40] * milimetric looks up rain dancing [15:29:42] thanks a lot elukey [15:30:36] * milimetric decides the Thai rain dance fits his personality best [15:30:53] OoOok joal its going! [15:30:54] something about the cat parade [15:31:39] ottomata: data flowing ;) [15:31:43] ottomata: awesome [15:31:44] "The cats are then to be placed in baskets; only one cat should be put in a basket. You must then walk around in the village. When the “Cat Parade” goes through someone's house, that person needs to splash water on the cats. It is believed that rain would then fall after three to seven days." [15:32:57] I don't know enough to dispute that process for sure [15:35:31] Look, who am I to argue with history? It's been raining in Thailand for millions of years, they're doing something right. [15:35:35] !log restarted all the failed oozie cassandra load jobs [15:35:36] joal: milimetric check out https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/manifests/mariadb.pp;f8482375022ba1eee1aee518a58fb11e79a49299$705 which leads you to https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/manifests/mariadb.pp;f8482375022ba1eee1aee518a58fb11e79a49299$743 [15:35:37] so [15:35:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:35:42] modules/role/files/mariadb/redact_sanitarium.sh in puppet repo [15:35:51] elukey: , joal, ops sync, but we can skip today. i don't have any particular items [15:36:01] and iirc you basically run that (with the my.cnf file) and it spits out sql that you | into mysql to run on the wiki db [15:36:13] and it creates all of the safety triggers per wikidb on first creation [15:36:17] joining ottomata, we should have it for cassandra [15:36:22] elukey: --^ [15:36:25] joining [15:36:27] k [15:36:36] chasemp: got it [15:37:28] that + the views yaml is more or less the answer to how a thing came to be like it is in labsdb [15:37:39] the why of things is usually shrouded in mystery :D [15:37:51] chasemp: I'll look at that :) [15:37:56] chasemp: many thanks ! [15:37:57] yep, we had a cloudy grasp of the how, the why was more interesting [15:38:08] chasemp: you'll hear from me again I think ! [15:38:25] I suspect, it was only meant to old back deleted revision's? [15:38:29] I'm not sure [15:38:31] kk [15:49:16] 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#3009580 (10JAllemandou) >>! In T155658#3009305, @Marostegui wrote: >>>! In T155658#3008872, @JAllemandou wrote: >> >> - Load on SQL machines seemed very acceptable (to be confirmed... [15:51:14] joal: what's your mediawiki username? [15:51:19] mediawiki.org [15:51:19] ? [15:51:46] ottomata: JAllemandou_WMF I think [15:52:08] https://www.mediawiki.org/wiki/User:JAllemandou_WMF [15:52:09] not registered [15:52:44] https://www.mediawiki.org/wiki/User:JoAl_(WMF) [15:52:49] Fun ! [15:53:00] https://www.mediawiki.org/w/index.php?title=User:JAllemandou_(WMF)&redirect=no [15:53:18] ottomata: this one seems the good one --^ [15:55:19] ottomata: why? [15:55:19] danke [15:55:28] added you as auto reviewer for refinery stuff :) [15:55:36] and me :) [15:55:36] ottomata: awesome ! [15:55:38] :) [15:55:41] Thanks ! [15:57:39] thanks ottomata ! [16:01:31] milimetric: holaaa [16:02:18] 06Analytics-Kanban, 06Discovery, 06Discovery-Analysis (Current work), 13Patch-For-Review: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#3009593 (10Nuria) 05Resolved>03Open [16:04:15] 10Analytics, 10Analytics-Cluster: Cassandra loading job are causing Pageview stale data - https://phabricator.wikimedia.org/T157533#3009599 (10elukey) 05Open>03Resolved Network rules added, all jobs restarted and proceeding normally. [16:06:26] 06Analytics-Kanban, 06Discovery, 06Discovery-Analysis (Current work), 13Patch-For-Review: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#3009602 (10Nuria) Data from pivot is deleted. [16:06:34] 06Analytics-Kanban, 06Discovery, 06Discovery-Analysis (Current work), 13Patch-For-Review: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#3009603 (10Nuria) 05Open>03Resolved [16:06:40] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2577484 (10Fjalapeno) Just following up from T145829: In addition to wp10, the other revision models would also be usefu... [16:15:03] (03CR) 10Milimetric: "fix this then let's merge this thing." (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) (owner: 10Fdans) [16:17:25] (03CR) 10Fdans: Add map visualizer to Dashiki (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921) (owner: 10Fdans) [16:21:31] 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ got erased - https://phabricator.wikimedia.org/T156253#3009632 (10Aklapper) 05Open>03declined Thanks Luis for the explanation / how-to! I've decided that we should not spend time fixing this (will not re-set up the legacy korma.wmflabs.... [16:31:44] joal, projectcounts? [16:32:06] mforns: need to take a break, in 1h or so? [16:32:16] joal, sure! [16:32:22] thanks mforns :P [16:32:23] :) [16:32:27] np :] [16:33:05] (03CR) 10Nuria: [C: 032] Add DSXS (self-identified bot) to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/336575 (https://phabricator.wikimedia.org/T157528) (owner: 10HaeB) [16:33:23] (03CR) 10Nuria: [V: 032 C: 032] Add DSXS (self-identified bot) to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/336575 (https://phabricator.wikimedia.org/T157528) (owner: 10HaeB) [16:35:48] joal: mforns: nuria ¡Buen día! and bonjour! :) Can you think of a good starting point for data a discussion about banner activity data retention on Druid/Pivot? Maybe some existing comparable policy I could look at? Thanks in advance!!! [16:37:18] (03Merged) 10jenkins-bot: Add DSXS (self-identified bot) to bot regex [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/336575 (https://phabricator.wikimedia.org/T157528) (owner: 10HaeB) [16:49:57] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Create orgs and enroll DB identities who are likely only activity from imported upstream repos (to make it visible how incorrect our data potentially is) - https://phabricator.wikimedia.org/T157569#3009753 (10Aklapper) This will also ser... [16:50:24] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#3009755 (10Halfak) Just one quick note. It's essentially free to apply multiple models at the same time from ORES' point... [16:58:18] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Create orgs and enroll DB identities who are likely only activity from imported upstream repos (to make it visible how incorrect our data potentially is) - https://phabricator.wikimedia.org/T157569#3009336 (10Qgil) Very interesting appro... [17:00:28] joal, mforns : run tests on HaeB patch and took alook , it is ready to deploy today with everything else [17:00:48] nuria, thanks! was looking into that [17:01:05] nuria, but those changes were already merged no? [17:01:18] at least 10 minutes ago [17:01:31] mforns: yes, i did this 20 min ago [17:01:34] ok ok [17:01:37] thanks! [17:15:04] AndyRussG: policy does not depend on us but rtaher privacy policy of all wikimedia projects. The gist is that all PII information has to be removed after 90 days. [17:16:12] AndyRussG: so some datasets are not subjected to this policy at all, for example, our edit data -for the most part- is all public [17:16:38] AndyRussG: pageview data that is public and retained long term is normally about article or project counts [17:21:26] 10Analytics-Tech-community-metrics: korma: Mismatch between numbers for code merges per organization - https://phabricator.wikimedia.org/T129910#3009816 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https://phabricator.... [17:21:28] 10Analytics-Tech-community-metrics: korma: Panel for "Wiki revisions" on people.html does not provide 2016 data - https://phabricator.wikimedia.org/T141228#3009819 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https://p... [17:21:31] 10Analytics-Tech-community-metrics: korma: Time axis on repository.html only displays two months, repeated several items - https://phabricator.wikimedia.org/T115872#3009823 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[... [17:21:33] 10Analytics-Tech-community-metrics, 07JavaScript: korma: Syntax error, unrecognized expression on Korma profiles - https://phabricator.wikimedia.org/T126325#3009821 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https:... [17:21:35] 10Analytics-Tech-community-metrics: korma: Clicking "Age of open changesets by Affiliation" explanation link / legend goes to top of page - https://phabricator.wikimedia.org/T110874#3009827 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displa... [17:21:37] 10Analytics-Tech-community-metrics: korma: Font used for "Organizations" header on contributors.html looks a bit out of place - https://phabricator.wikimedia.org/T100569#3009825 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on stati... [17:21:39] 10Analytics-Tech-community-metrics: korma: GrimoireLib sometimes displays different names for same user ID; link does not display (existing) contributor data - https://phabricator.wikimedia.org/T140299#3009833 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | communi... [17:21:41] 10Analytics-Tech-community-metrics: korma: Illegible overlapping tables on narrow screens due to CSS - https://phabricator.wikimedia.org/T97115#3009829 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https://phabricator.w... [17:21:43] 10Analytics-Tech-community-metrics: korma: "Last 30 days" stats for specific mailing list display an account as one list item per username character - https://phabricator.wikimedia.org/T123927#3009835 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metric... [17:21:46] 10Analytics-Tech-community-metrics, 07JavaScript: korma: Failed to load resource: the server responded with a status of 404 (Not Found) - https://phabricator.wikimedia.org/T65061#3009831 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] display... [17:21:48] 10Analytics-Tech-community-metrics: korma: top-contributors.html displays comma as "Location" when a person has more than one affiliation - https://phabricator.wikimedia.org/T123926#3009837 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displa... [17:21:50] 10Analytics-Tech-community-metrics: korma: demographics.html: "Tickets participants" has "184 attracted" data for 1year03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics |... [17:21:52] 10Analytics-Tech-community-metrics: korma: Empty "subject" and "creator" fields for mailing list thread on mls.html - https://phabricator.wikimedia.org/T116284#3009841 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https... [17:21:54] 10Analytics-Tech-community-metrics: Update ITS related data from Bugzilla to Phabricator/Maniphest in project-info.json - https://phabricator.wikimedia.org/T114636#3009843 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ h... [17:21:56] 10Analytics-Tech-community-metrics: korma: Profile names in UTF-8 incorrectly displayed as ??? - https://phabricator.wikimedia.org/T119540#3009845 (10Aklapper) 05Open>03declined As the [[ https://www.mediawiki.org/wiki/Community_metrics | community metrics ]] displayed on static [[ https://phabricator.wikime... [17:25:10] nuria: cool thx!! Yeah basically I'm trying to get a general sense of what (if anything?) could or should be scrubbed in banner activity to keep it around longer term. Also coming up soon we should try to discuss the same for banner history (in EventLogging) [17:25:37] (03CR) 10Mforns: [C: 032] Add spark streaming job for banner impressions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/333655 (owner: 10Joal) [17:25:38] AndyRussG: most EL data is subjected to a 90 day retention [17:25:43] Yeah [17:25:54] (03CR) 10Mforns: [V: 032 C: 032] Add spark streaming job for banner impressions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/333655 (owner: 10Joal) [17:25:58] For the banner history EL I'd like to see if it can be scrubbed/processed somehow also for longer-term [17:26:25] mforns, joal: did you guys talked about data retention of the banner dataset? [17:26:45] nuria: Not that I know no [17:27:23] Maybe there's documentation of a previous discussion on how and what to retain of pageviews, that I could check out for reference/general principles to consider? [17:27:42] (np if not, just thought I'd ask :) ) [17:28:11] mforns, joal: we should have a plan for deletion for every dataset we productionize, in this case I think device would need to be removed and data agreggated on country level [17:28:26] AndyRussG: Have you read privacy policy? I would start there [17:28:55] nuria, I had a look into the dataset regarding privacy [17:29:10] nuria: yes agreed... no, haven't looked at in detail, nor very recently [17:30:06] the device dimension only contains 5 values: desktop, android, iphone, ipad, unknown [17:30:17] I think it's not sensitive [17:30:24] Ah yeah that's device as it's interpreted by CentralNotice [17:30:55] ottomata: can you provide me with a quick help on druid? [17:31:14] nuria, plus the dataset doesn't contain any field that indicates user consumption, like page titles [17:31:43] mforns: I see it is a big bucket for device but maybe long term what needs to be done is agreggate the android/iphone/ipad [17:31:58] joal: ya what's up? [17:32:24] ottomata: I've deleted POC datasources yesterday, but there is one I don't manage to :( [17:32:41] nuria, AndyRussG, the most sensitive bucket would be region+device, which for small countries/regions could be (remotely) potentially sensitive [17:33:06] mforns: still a iphone in san marino looking at a particular project for which there has been 1 edit that day it is likely private [17:33:23] mforns: so we probably need to aaggregate [17:33:24] ottomata: I wondered if there could have been anything hardcoded somwhare [17:34:06] nuria, but it's banners no? it has more to do with pageview than edits... [17:34:18] 10Analytics: Evaluate privacy of banner dataset for long term retention - https://phabricator.wikimedia.org/T157582#3009888 (10Nuria) [17:34:23] mforns: but it shares the same exploit [17:34:33] joal: hm [17:34:47] ottomata: culprit is named "pageviews" [17:34:51] it;s one of our old ones [17:34:53] hm [17:34:57] i don't know much about this, but looking [17:34:58] mforns: if you have a geolocated pageview and 1 edit by editor X you are (by proxy) disclosing the location of the editor [17:35:05] joal: how did you dleete the otherss? [17:35:27] mforns: nuria; I think it'd be totally fine to re-aggregate scrubbing device and region after 90 days [17:35:46] Could also scrub project [17:35:47] AndyRussG: ok, let's do that [17:35:52] ottomata: data got deleted from HDFS deep-storage without using druid endpoint - I think it might be the thing - But, I recreated some fake data this morning, and tried to delete again using druid - no luck [17:35:59] nuria, yes I understand [17:36:04] AndyRussG: not needed, if we can scrub device+ region we are fine [17:36:07] specially with minutely resolution [17:36:07] delete agin using druid...how? via json api? [17:36:21] ottomata: yes [17:36:31] ottomata: That's the proper way [17:36:32] Sounds great! [17:36:50] ok, so you did that to old 'pageviews' but it doesn't go away? [17:37:03] correct ottomata [17:37:22] In any case, sounds like the general approach is clear: re-aggregate after 90 days to remove potentially privacy-compromising dimensions [17:37:37] 10Analytics: Agreggate banner dataset for long term retention - https://phabricator.wikimedia.org/T157582#3009918 (10Nuria) [17:38:10] Really, so long as we keep country, campaign and banner, I think the data will be useful for longer-term analysis [17:38:11] nuria, AndyRussG, joal: OK so we have 1 month to write a job that scrubbs device and region from the data set, and maybe goes to hourly resolution [17:38:15] ottomata: in coord UI pageviews dataset is marked as disabled (that's what we expect), but I don't manage to get rid of it [17:38:15] joal, the datasource is the only one called "pageviews" [17:38:16] ? [17:38:17] AndyRussG: yes, having in mind that dataset could be cross checked with an existing one [17:38:28] mforns: yes, just about [17:38:35] k [17:38:37] correct ottomata [17:38:47] i'm seeing lots of logs in broker.log about that datasource [17:38:58] looks like queries? [17:39:05] joal [17:39:07] mforns: we can drop part of the dataset directly on druid if needed. [17:39:07] nuria: yeah... Also that's why I was thinking of removing project, too... Since there are some pretty small projects [17:39:09] tail -f /var/log/druid/broker.log [17:39:32] metrics for query/time ? [17:39:32] And I'm pretty sure that's one of the least important data points [17:40:22] ottomata: I don't get it :( [17:40:26] me neither [17:40:29] i don't know what those mean [17:40:38] is it possible something is actively inserting for that datasource? [17:40:50] I'll run this by fr-tech and stakeholders... I think it's all good :) [17:40:53] thx so much!!! [17:41:41] Amazing to see this producitonized, BTW... Mmmm just noticed a gap in impressions at the beginning of Feb, I assume that's known? [17:41:44] https://pivot.wikimedia.org/#banner_activity_minutely/line-chart/2/EQUQLgxg9AqgKgYWAGgN7APYAdgC5gQAWAhgJYB2KwApgB5YBO1Azs6RpbutnsEwGZVyxALbVeAfQlhSY4AF9kwYhBkc86FWs7AKVOoxZt1XTDnwAbUmGoNiFqgDd7AV3GnmYYgzC8ATAAMAIwA7AC0wWEAzAFwAQG48YkBAHTxAQBa+uQAJv7B4QF+YUGxSeWp6VmKwGAAnljuwHAAkgCyIBIASgCCAHIA4iAK8ooA2ugGTKzsnNzmfNSCSsJy+FIycjUARi4QANbUYD2qsxrKpya1stQAQnuHvko5LnbavAAKQQAiCkrMGB8J3epi0Z3wAJ8+no02M [17:41:46] czMvAEQlETQ2Nz+wBypCYlx0xGYEGouQoAHMRgBdZDkFwWCwrQEieykABe1ByEiYAEc3J4JBAMC5yE8xsByAymaz2VyeWA+QKhVRpSxZfzBb5KWNKdTaUA== [17:41:48] Aaaarg url [17:42:03] 06Analytics-Kanban, 13Patch-For-Review: Add "Damn Small XSS Scanner" (DSXS) to list of known bots - https://phabricator.wikimedia.org/T157528#3009927 (10Nuria) [17:42:06] 10Analytics, 06Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#3009926 (10Nuria) [17:42:14] joal: http://druid.io/docs/latest/operations/metrics.html [17:42:26] Try this instead: https://tinyurl.com/gllwbb7 [17:43:10] 10Analytics, 06Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#2393202 (10Nuria) @Tbayer: this ticket is for bot identification measures for bots that do not identify as such on user agent. [17:43:20] those metrics are about queries coming from thorium, so assuming pivot [17:43:28] joal: maybe datasource needs removed from pivot? [17:43:29] just guessing here [17:43:39] ottomata: completely possible ! [17:43:50] Let do some clean up in there as well :) [17:44:01] Thanks for the idea ottomata :) [17:45:06] 10Analytics, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#3009932 (10Nuria) @JAllemandou : sorry i should have looked into this closer, if this is a self-identified bot it shoul... [17:45:20] 10Analytics, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#3009934 (10Nuria) [17:45:23] 10Analytics, 06Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#3009933 (10Nuria) [17:45:40] joal: regarding this ticket: https://phabricator.wikimedia.org/T153699 [17:45:56] 10Analytics: Agreggate banner dataset for long term retention - https://phabricator.wikimedia.org/T157582#3009888 (10JAllemandou) Technical point: This can either be done when the data is originally created, or can be done afterward by reindexing same data based on Druid indexes. [17:46:03] yes nuria ? [17:46:40] Ah, your comment makes a lot of sense nuria [17:46:50] joal: my mistake cause i did not look at this closely enough but if this was a self-identified bot it should be added to the regex list, our purpose with agreggating [17:46:52] mforns: ^ ? (https://tinyurl.com/gllwbb7 , data gap at the beginning of Feb.?) Thx, sorry for the bother.... [17:46:58] items under this ticket: https://phabricator.wikimedia.org/T138207 [17:47:20] is to have an array of instances of bot traffic that is not self-reported as such. [17:47:42] makes sense nuria [17:47:45] 06Analytics-Kanban, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#3009966 (10Nuria) [17:47:53] AndyRussG, yes, don't worry, it's because the streaming job that populates the live events is running [17:47:54] joal: ok, moved ticket back to kanban [17:48:04] and the daily job still needs to be deployed [17:48:15] nuria: The thing being, there probably are self-identied bots that are not covered by our regexp ... [17:48:18] AndyRussG, BTW, I'll do the deploy today [17:48:35] and this should be fixed in a couple hours [17:49:45] mforns: ah K gotcha.... thx again!!!!!!:D [17:50:31] 10Analytics, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3009998 (10Ottomata) [17:50:44] ottomata / elukey : can we do a pivot restart, to see if it changes anything on requests (pageviews in not mentioned in pivot config) [17:51:14] 06Analytics-Kanban, 06Operations, 10netops: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3010000 (10elukey) Completed the AQS work due to T157533 (under Brandon's supervision). I am going to keep working on this task during the next days to fix the remaining items. Caveat:... [17:51:34] joal: i think so, i dunno what happens now that folks actually use it though :) [17:51:50] hm, hard cut I guess [17:51:51] oh, joal interseting [17:51:52] in pivot logs [17:51:53] Feb 08 17:51:41 thorium pivot[127298]: Error getting time for 'pageviews': No such datasource [17:51:56] as well as other ones [17:51:59] that you probably deleted [17:52:05] it'll be ok [17:52:06] ottomata: makes sense [17:52:07] !log restarting pivot [17:52:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:56:31] 10Analytics: Agreggate banner dataset for long term retention - https://phabricator.wikimedia.org/T157582#3009888 (10mforns) I think making the data set less granular over time might be enough. We can study how many buckets have request_count == 1 with hourly and daily resolution. [17:59:12] ottomata: didn't change a thing for me :( [18:00:38] hm [18:01:04] ot more precisely: some sources removed from Pivot, but pageview still not removable :( [18:01:42] which one joal ? Is it listed in the config? [18:01:50] I mean, the datacube [18:01:52] elukey: not listed in config [18:02:24] ottomata: I have disabled another source just after yuou restarted pivot, would you mind going for another round? [18:05:02] ok [18:05:07] !log restarting pivot [18:05:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:16:25] ottomata: I really don't get it :( [18:17:08] ottomata: Actually, I get it :) [18:21:40] ot at lesat I thought I did [18:21:56] ottomata: There still was indexing tasks for my test_banner datasource [18:22:00] I killed them [18:24:47] hmmm [18:24:48] ok [18:24:54] are there any for pageviews? [18:24:56] (I don't get it either) [18:27:36] * elukey goes afk! o/ [18:28:50] hey nuria, should we merge the action=submit patch before deploying? [18:29:44] mforns: we also need https://gerrit.wikimedia.org/r/#/c/335642/ to be merged for the full set of changes [18:30:02] looking [18:30:04] ottomata: nothing on pageviews - That doesn't makes sense :( [19:00:42] joal: i dunno :/ [19:00:55] ottomata: I don't know either :( [19:01:01] ottomata: that's a bit weird [19:01:42] ottomata: nothing major, no data on hadoop nor or druid, but datasource disabled, not delete [19:02:10] maybe druid just needs a restart too? [19:04:45] ottomata: possible :/ [19:05:03] ottomata: or as I said, having removed data manually (for pageviews), broke some stuff [19:05:15] ya [19:32:45] (03CR) 10Nuria: [V: 032 C: 032] Update jar/version in refinery and pageview jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335642 (https://phabricator.wikimedia.org/T156628) (owner: 10Joal) [19:41:37] 06Analytics-Kanban: Debian package for ua parser latest version - https://phabricator.wikimedia.org/T156821#3010575 (10Nuria) Re-opening. The version we have imported of ua-parser is missing a file: File "/usr/lib/python2.7/dist-packages/ua_parser/user_agent_parser.py", line 549, in from ._regexe... [19:41:45] 06Analytics-Kanban: Debian package for ua parser latest version - https://phabricator.wikimedia.org/T156821#3010576 (10Nuria) 05Resolved>03Open [19:41:52] joal: druid q for you! [19:41:56] sure ottomata [19:42:02] when does it decide to write to hdfs deep storage? [19:42:06] and. [19:42:19] ottomata: yt? [19:42:21] what currently loads data into druid automatically [19:42:25] ah sorry [19:42:28] just a hadoop/oozie job? [19:42:29] nuria: ya! [19:42:30] my irc wentkaput [19:42:30] hi [19:42:42] nuria_: can you please go for this one too : https://gerrit.wikimedia.org/r/#/c/335639/ ? [19:42:46] ottomata: see https://phabricator.wikimedia.org/T156821, the ua parser package is missing 1 file [19:43:24] joal: you can merge it sure [19:43:32] ottomata: I reopened ticket [19:43:41] ottomata: So, this is the process: hadoop is used to transform files into nicely formatted stuff for druid - Saved on HDFS [19:44:07] nuria: HM! [19:44:07] ok... [19:44:08] on it. [19:44:09] The overlard is responsible of creating / managing that task [19:44:49] once done successfully (at overlord level), overlord tells coordinator that new/refreshed data is availble for its historical nodes [19:45:12] coordinator tells the correct historical nodes to fetch data from HDFS [19:45:56] oh, ok, but that is not the deep storage, right? [19:45:58] ottomata: Loading rules are define in coordinator, which tells historical nodes (the computers) to drop segment if too old for instance [19:46:07] HDFS IS the deep-storage [19:46:10] ok [19:46:21] but it thought druid wrote old historical data to deep storage [19:46:25] you are talking about loading from hdfs? [19:46:46] Correct- You actually generate data onto deep-storage, and druid loads it [19:47:11] ah, and then it is there in deep storage for later [19:47:15] except for realtime tasks, where index files are generated on druid, then pushed to HDFS [19:47:16] if druid decides to drop stuff from in mem [19:47:18] but later someone queries? [19:47:41] When data is in druid, it doesn't mean in mem, can be on disk [19:48:05] But when data is not in druid anymore (loading for instance), it might still be available in deep-storage [19:48:17] makes sense? [19:48:22] ok, 'in druid then' [19:48:26] ok [19:48:27] ya [19:48:38] ok, i ask because i'm thinking about druid during hadoop downtime for upgrade [19:48:49] do we need to stop druid? [19:48:52] ottomata: should be ok [19:48:54] it sounds like druid will be ok [19:49:18] ottomata: Since no hadoop job, no indexing (I wonder more about real time, but this is not super important [19:49:25] I think so yes [19:50:03] nuria_: ping again on action-submit for pageview def - I think this needs to be merged before mforns deploys tomorrow (I'll +2 tomorrow if you don't today) [19:50:09] nuria_: https://gerrit.wikimedia.org/r/#/c/335639/ [19:50:14] Need to go for diner [19:50:22] Have a good evening a-team ! [19:50:25] joal: you can merge it please do, i just said that [19:50:38] joal: sorry, my irc is kind of kaput today [19:51:19] joal: no need to wait for my approval in any way, i though those two items were merged since we agreed edits should not be counted as pageviews [19:52:37] (03CR) 10Nuria: [C: 032] Update pageview definition to remove previews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335639 (https://phabricator.wikimedia.org/T156628) (owner: 10Joal) [19:53:54] joal: i think for the upgrade we should stop tranquility [19:53:59] it hsould start back from where it left off whne we restart it [19:54:17] (03CR) 10Nuria: [V: 032 C: 032] Update pageview definition to remove previews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335639 (https://phabricator.wikimedia.org/T156628) (owner: 10Joal) [20:45:02] tgr: Can i ask you a question that you probably know the answer to? [20:48:05] thanks nuria_ for merging, and sorry for double ping I didn't get your answer [20:48:31] joal:np, my irc issues are fixed now [20:48:51] mforns: We are all set for tomorrow deploy :) [20:49:10] joal, nuria, thanks :] [20:50:50] 06Analytics-Kanban, 13Patch-For-Review: Add "Damn Small XSS Scanner" (DSXS) to list of known bots - https://phabricator.wikimedia.org/T157528#3010974 (10JAllemandou) [20:50:53] 06Analytics-Kanban, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#3010976 (10JAllemandou) [21:30:00] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2796693 (10TBolliger) Hello. I would like access access to pivot, so I need LDAP permissions. I should be in group wmf I am a product manager at the WMF: https://meta.wikimedia.org/wiki/User:TBo... [21:32:40] 10Analytics: Investigate rise in IE views from Pakistan since 2015 - https://phabricator.wikimedia.org/T157404#3011082 (10Tbayer) Looks related to T148461 (I haven't looked at IE version numbers or other countries besides Pakistan though). [21:35:58] 10Analytics: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#2723684 (10Tbayer) See also {T157404}, excerpt from there: {F5523978, width=90%} [21:55:57] 10Analytics: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#3011175 (10Tbayer) Updating and extending Nuria's chart from above (global IE pageviews by version over time since mid 2015), it looks like this is still on the rise. Assuming that there is no reaso... [22:24:16] 06Analytics-Kanban, 13Patch-For-Review: Debian package for ua parser latest version - https://phabricator.wikimedia.org/T156821#3011264 (10Ottomata) OoOK! ``` In [5]: from ua_parser import user_agent_parser In [6]: ``` That was tricky. Upstream's setup.py [[ https://github.com/ua-parser/uap-python/blob/ma... [22:26:59] 06Analytics-Kanban, 13Patch-For-Review: Debian package for ua parser latest version - https://phabricator.wikimedia.org/T156821#3011291 (10Ottomata) This is the fix: https://gerrit.wikimedia.org/r/336722