[06:10:28] Analytics, Editing-Analysis: Create a chart showing percentage of new articles created each month that have not survived to the present - https://phabricator.wikimedia.org/T149049#2747570 (kaldari) If this is too complicated, just a chart comparing articles created per month with articles deleted per mon... [06:39:53] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2747588 (elukey) [08:37:59] Analytics, Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2747765 (HJiang-WMF) Analytics(@Milimetric, @mforns) and Editing(@HJiang-WMF) are working on this. We will start out by picking a single query or report vertical slice(for... [08:43:24] Analytics, Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2747774 (HJiang-WMF) In addition, ongoing conversations is underway to further understand old queries, their construction, and the rationale behind the decisions(parts of d... [09:11:22] joal: o/ [09:11:46] I am reviewing https://gerrit.wikimedia.org/r/#/c/316359/, that restricts access to the oozie server from stat100[24] [09:11:55] I don't see anything wrong with it [09:12:00] any concern? [09:12:05] just to double check [09:13:03] Hi elukey, reading [09:13:42] elukey: I think oozie needs to talk to every machine on the cluster, and nees to [09:14:15] and needs to be accessed from stat100[24] - After that I don't really know how this is done in code :) [09:14:38] (CR) Elukey: [C: 2] Replace Bugzilla links by Phabricator links [analytics/wikistats] - https://gerrit.wikimedia.org/r/315417 (owner: Aklapper) [09:15:02] For instancde, the CR you sent, I don't know if it only restrict access to the machine from stat100[24] or if it also blocks the server to access the cluster [09:15:19] * joal doesn't know about ferm :( [09:15:25] (CR) Elukey: "Thanks a lot! Now I need to figure out how this will get deployed :)" [analytics/wikistats] - https://gerrit.wikimedia.org/r/315417 (owner: Aklapper) [09:18:49] joal: see I am still sleepy, this is why I ask stuff to you :) [09:18:59] let me re-check [09:19:18] elukey: sure :) [09:21:47] joal: also analytics1027 runs hue [09:21:54] that needs access to the oozie server [09:21:54] good catch elukey ! [09:23:47] I am thinking if oozie needs to talk only with the master nodes or the whole cluster [09:35:59] elukey: I'm not sure ... [09:36:19] elukey: I don't know if oozie get's information from only resource manager of from application master [09:37:13] Actually elukey: given that oozie-managing- jobs are launched as mapreduce jobs, I suspect it talks to application masters and therefore any node [09:39:10] I put a -1 and then I'll follow up with Moritz [09:39:15] ;) [10:55:57] (PS3) Mforns: [WIP] Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) (owner: Milimetric) [11:43:28] * elukey lunch! [13:25:38] Analytics-Tech-community-metrics, Developer-Relations: Measuring Time To First Code Change (TTFCC) - https://phabricator.wikimedia.org/T137201#2748246 (Qgil) Would this task be a good topic for the #wikidev17 ? If so, the deadline to submit new proposals is next Monday, October 31: https://www.mediawiki.... [13:30:43] Analytics-Tech-community-metrics, Developer-Relations: Migration to new Bitergia's development dashboard - https://phabricator.wikimedia.org/T137997#2748257 (Qgil) Would this task be a good topic for the #wikidev17 ? If so, the deadline to submit new proposals is next Monday, October 31: https://www.medi... [13:44:08] mforns / nuria: it turns out we were never linting either :) [13:44:12] so I'm fixing that, might take a bit [13:52:48] milimetric: all the wmf users that we reviewed have access to pivot now [13:52:58] wmde are still in progress, working on it :) [13:53:04] great, thx [13:53:22] elukey: I was thinking zareen needs to be on that list too, she just started [13:53:34] (maybe she was onboarded properly though) [13:59:03] I'll add her if she does not get added [13:59:13] but for the moment I'd like to fix holes :) [13:59:52] I was trying to contact Daniel over IRC but can't find him online [14:01:13] elukey milimetric: sorry, my IRC was disconnected so I'm not sure what the list is for, but if it's related to onboarding, please add me :) [14:01:25] zareen: o/ [14:01:36] so we were talking about LDAP groups [14:01:44] precisely https://wikitech.wikimedia.org/wiki/LDAP_Groups [14:01:47] zareen: can you access https://pivot.wikimedia.org/ ? [14:03:03] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2748319 (Ottomata) + 1/2 to this idea. I'm all for renaming these boxes, not sure if misc element names is the way to go, but it might be! [14:03:15] milimetric: no, I can't log in [14:03:16] probably not, she is not in wmf [14:03:39] k, thx for checking zareen, and welcome, and we'll try and sort this out soon :) [14:04:58] zareen: if you have access to https://office.wikimedia.org/wiki/Contact_list, can you add yourself? [14:05:03] great, let me know if you need to me try it again later [14:07:02] elukey: I can't login to there either - "The supplied credentials could not be authenticated." [14:11:32] okok no rush, you'll get access to everything :) [14:47:46] (PS4) Milimetric: Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) [14:58:00] milimetric: linting? but that doesn't check out [14:58:09] milimetric: i have seen linting errors plenty times [14:58:51] milimetric: zaren would need a user and ssh keys so it might be a a bit [14:58:51] nuria: yeah, but a lot were hidden because the include paths were broken [14:58:55] I think they're fixed now [14:59:06] milimetric: but we shoudl not lint deps though [14:59:17] only our won code [14:59:17] *own [14:59:46] and that was happening before, as i have seen it fail many times [15:00:30] (PS5) Milimetric: Migrate from bower to npm instead of yarn [analytics/dashiki] - https://gerrit.wikimedia.org/r/316904 (https://phabricator.wikimedia.org/T147884) [15:00:47] a-team: standddduppppp [15:01:29] ACK [15:37:47] elukey: re: irc.wikimedia.org, I agree with or i's position [15:42:25] milimetric: oh yes I just wanted to bring up attention, that's it :) [15:43:09] Analytics, Dumps-Rewrite: Improve mediawiki data redaction - https://phabricator.wikimedia.org/T146444#2748578 (Milimetric) [15:46:20] INNTTERRNNEET Y U GOTTA BE LIKE THIS? [15:46:26] Analytics-Kanban: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#2748608 (Milimetric) a:Nuria [15:47:12] Analytics-Kanban, Discovery, Operations, Discovery-Analysis (Current work), and 2 others: Can't install R package Boom (& bsts) on stat1002 (but can on stat1003) - https://phabricator.wikimedia.org/T147682#2748617 (Milimetric) a:Ottomata [15:49:39] Analytics-Kanban: Find a strategy for integration testing - https://phabricator.wikimedia.org/T147442#2748683 (Milimetric) [15:58:25] milimetric: nuria, can you read y IRCs? [15:58:25] even hangout chat is lagging [15:58:25] sigh [16:02:54] Analytics-Kanban, Discovery, Operations, Discovery-Analysis (Current work), and 2 others: Can't install R package Boom (& bsts) on stat1002 (but can on stat1003) - https://phabricator.wikimedia.org/T147682#2700737 (Nuria) Turns out that sorting this is going to take a bit more than we though, we... [16:06:06] Analytics-Kanban: Change requests drop alarms to be more precise regarding data loss - https://phabricator.wikimedia.org/T148980#2748822 (Milimetric) p:Triage>Normal [16:10:59] Analytics-Kanban: Change requests drop alarms to be more precise regarding data loss - https://phabricator.wikimedia.org/T148980#2748833 (Milimetric) a:mforns [16:13:20] Analytics-Kanban: Change requests drop alarms to be more precise regarding data loss - https://phabricator.wikimedia.org/T148980#2748851 (Nuria) Quries: https://github.com/wikimedia/analytics-refinery/blob/master/hive/webrequest/select_missing_sequence_runs.hql [16:15:33] Analytics-Kanban, EventBus, Operations: setup/install/deploy kafka1003 (WMF4723) - https://phabricator.wikimedia.org/T148849#2748867 (Milimetric) [16:18:55] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Merge isolated accounts in korma data when email addresses match - https://phabricator.wikimedia.org/T149327#2748880 (Aklapper) [16:19:11] Analytics-Tech-community-metrics, Developer-Relations (Oct-Dec-2016): Merge isolated accounts in korma data when email addresses match - https://phabricator.wikimedia.org/T149327#2748880 (Aklapper) p:Triage>Low [16:19:44] elukey: for capacity planning for asq, ideally how many more nodes would we like to have to plan for better resiliency? [16:20:31] nuria: I'd say that doubling the cluster would be awesome [16:20:35] mforns: updated ticket this is teh sql that checks runs: https://github.com/wikimedia/analytics-refinery/blob/master/hive/webrequest/select_missing_sequence_runs.hql [16:20:37] so other 3 nodes [16:20:56] nuria, thanks! [16:24:22] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2748907 (ArielGlenn) >>! In T149228#2748319, @Ottomata wrote: > + 1/2 to this idea. I'm all for renaming these boxes, not sure if misc element names is the way to go, but it might be!... [16:29:09] team, is wikistats automagically deployed via puppet? [16:29:20] I am asking because of https://gerrit.wikimedia.org/r/#/c/315417/ [16:29:41] (it seemed super easy and I had +2ed it) [16:30:52] elukey: not sure? prob need to ask erik zachte. [16:37:29] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2748977 (Ottomata) - stat1001 - app/webserver, no analyst/ressearch access. - stat1003 - compute node, lots of storage, mostly used by researchers to connect to MySQL. - stat1002 - compu... [16:43:06] ottomata: so the file is updated on disk [16:43:11] but I am not sure if it needs to be ran [16:44:48] (CR) Elukey: "Aklapper: did you find any specific stat.w.o link related to this? I can see the file changed on stat1001 but not sure what webpage it ref" [analytics/wikistats] - https://gerrit.wikimedia.org/r/315417 (owner: Aklapper) [17:07:48] elukey ya it probably gets run when wikistats are generated, which happens i don't know how often [17:07:49] monthly? [17:27:24] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2749185 (AlexMonk-WMF) >>! In T149228#2748977, @Ottomata wrote: > - stat1002 - compute node, lots of storage, with private data and Analytics Cluster (Hadoop) access. You mention 'priva... [17:29:00] :/ [17:29:00] anyhow, going afk! [17:29:00] o/ [17:31:16] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2749206 (Ottomata) Perhaps so, but that data is not stored on stat1003. Those MySQL dbs are theoretically accessible from anywhere in the prod network, if you have the proper MySQL user... [17:51:42] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2749301 (AlexMonk-WMF) Okay. What sort of private data (beyond credentials to external systems) is stored on stat1002? [17:55:11] Analytics, Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2749313 (Ottomata) Most notably, (and historically), sampled webrequest logs in the udp2log format. [18:11:39] laters a-team, i'll be working a bit tomorrow [18:11:41] ttyt [18:30:49] PROBLEM - YARN NodeManager Node-State on analytics1040 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:31:40] RECOVERY - YARN NodeManager Node-State on analytics1040 is OK: OK: YARN NodeManager analytics1040.eqiad.wmnet:8041 Node-State: RUNNING [18:55:06] yarn had problems? [18:55:10] never seen that before... [19:50:37] bearloga: hello, could you point me to your clickthrough rates calculations? [19:50:50] bearloga: pivot issues should be resolved. let us know otherwise [19:54:10] nuria: howdy! pivot issues resolved (for me but idk about chelsyx, etc.). clickthrough calculations are done in multiple steps; is it okay to email you? [19:55:48] bearloga: ok, good they should be solved for everyone. Do use pivot to look at your spikes, looks very much like bot (or automated traffic , intentional or not) [19:56:08] bearloga: e-mail sounds great, thanks [19:59:34] nuria: I don't think we can use pivot for portal (yet?) should/could we change IsPageviewUDF/PageviewDefinition so that it starts to identify wikipedia.org portal pageviews so that we'd be able to use pivot for those? [19:59:57] bearloga: you can use pivot to see that your spikes appear elsewhere [20:00:02] bearloga: like the main page [20:00:09] bearloga: those are rarely isolated events [20:00:47] bearloga: there are more changes need to add the portal to the pageview definition than changing the code, it can be done but it requires a bit more work than that [20:24:55] Analytics-Kanban, EventBus, Operations: setup/install/deploy kafka1003 (WMF4723) - https://phabricator.wikimedia.org/T148849#2749952 (RobH) @Milimetric, I can note that I put LVM on it. Overall most servers should use LVM for a little breathing room on the disks in the event of a failed logrotate o... [20:44:36] Analytics-Kanban: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#2750021 (Nuria) After analyzing one hour of traffic requests are coming from mostly India/Iran/Pakistan/Afghanistan and they are all requests from Main_Page, this is again some kind of ping... [20:46:15] Analytics-Kanban: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#2750043 (Nuria) [20:46:17] Analytics, Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#2750042 (Nuria) [20:47:05] Analytics: Bot from an Azure cloud cluster is causing a false pageview spike (can we identify as bot?) - https://phabricator.wikimedia.org/T137454#2750045 (Nuria) [20:47:07] Analytics, Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#2393202 (Nuria) [20:49:35] Analytics-Kanban, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Spike: [Feed] Establish criteria for blacklisting likely bot-inflated most-read articles - https://phabricator.wikimedia.org/T143990#2750051 (Nuria) [20:49:37] Analytics, Research-and-Data-Backlog: Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#2750050 (Nuria) [22:24:02] Analytics, Editing-Analysis: Create a chart showing percentage of new articles created each month that have not survived to the present - https://phabricator.wikimedia.org/T149049#2750427 (Neil_P._Quinn_WMF) p:Triage>High [22:24:17] Analytics, Editing-Analysis: Determine: What percentage of new articles are created by non-autoconfirmed editors - https://phabricator.wikimedia.org/T149021#2750428 (Neil_P._Quinn_WMF) p:Triage>High [22:48:00] nuria: Huh, for some reason I thought you came from Yahoo!. [22:48:06] I made some tweaks to https://wikimediafoundation.org/wiki/User:NRuiz_(WMF)