[08:45:03] hi, I [08:45:19] hi, I'm setting up firewall rules for the eventlog* systems and I'm wonderin about ipython-notebook [08:46:40] what is it used for? it's currently listening on 8888 for external connections, is that needed, does it need to be granted in the access rules? [09:17:31] moritzm: hi! No idea, but probably mforns will be able to answer [09:20:46] ok [09:54:01] I guess something for debugging, but I am not sure.. [10:43:31] Analytics-Kanban, Wikimedia-Mailing-lists: Home page for the analytics mailing list should link to gmane [1 pts] - https://phabricator.wikimedia.org/T116740#2021913 (hashar) Awesome all good to me. Thank you @mforns! [12:04:17] brb lunch! [13:07:26] (CR) Qgil: "Now it's in the 6th position." [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/195895 (https://phabricator.wikimedia.org/T89788) (owner: Milimetric) [13:49:19] (CR) Milimetric: "Aah! Sorry! I should've abandoned it long ago. My fault." [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/195895 (https://phabricator.wikimedia.org/T89788) (owner: Milimetric) [13:49:28] (Abandoned) Milimetric: Analyze page size impact on editing [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/195895 (https://phabricator.wikimedia.org/T89788) (owner: Milimetric) [13:59:48] Analytics-Cluster, Analytics-Kanban, Reading-Admin, Easy, Patch-For-Review: PM sees reports on browsers (Weekly or Daily) {lama} [8 pts] - https://phabricator.wikimedia.org/T88504#2022303 (mforns) @dr0ptp4kt We will be implementing this and other improvements to the browser report format in short... [14:02:05] (PS3) Mforns: Update AppSession Metrics (typing and sorting) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/268639 (https://phabricator.wikimedia.org/T125960) (owner: Joal) [14:18:51] a-team: let me know if anybody wants to explain to me what is happening with oozie :) [14:19:24] I've read madhu's email but I'd need a bit more details :) [14:20:31] elukey so there's a whitelist for domains that we expect to see as we process requests [14:20:48] It's just a simple Hive table in the wmf db [14:21:11] And if something's not on that list, you see this warning [14:21:25] But, you also see this warning for a million other reasons [14:22:01] So I'd love to know how Madhu figured that out so I can also do it :) [14:23:55] milimetric: sorry to ask for the basics, but where is the wmf db? [14:24:52] first, it's ok to ask for basics, I was asking for basics for like two years, so you're way ahead right now [14:26:16] By the "wmf" db, I mean, if you run hive on stat1002 or one of the hadoop nodes, and go "use wmf;" and "show tables;" you'll see that table is in that db, I forget the exact table name [14:27:11] never done that, adding to my todo list :) [14:28:18] Oh cool, yeah, that's how all the analysis on the webrequest data happens. Well, that and spark streaming a bit [14:29:13] Lemme know if you want to go over the current pipeline (how the data goes from varnish through to this database and the various tables) [14:48:09] Joseph already told me in SF, I took a lot of notes, so I'll probably read them again and come back with questions! [15:05:02] milimetric: just as quick info, does it https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Access#ssh_tunnel.28s.29 work for you? [15:06:50] No, tunneling has never worked for me. I'll try again [15:07:29] how do the other guys check the status of the cluster? [15:07:45] I use hue, but that errors out a loooot [15:07:57] because the webserver on analytics1001 is listening on a 10.0.0.x, not localhost.. [15:08:10] but I am maybe missing something [15:08:24] and I don't have access to hue :( [15:08:33] not sure how to sync my ldap account with it [15:08:55] Oh, ask ottomata, he has to be able to add you [15:24:03] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/268639 (https://phabricator.wikimedia.org/T125960) (owner: Joal) [15:48:37] milimetric: I didn't know until yesterday either - but when the unexpected values error is raised, it also inserts the unexpected values into wmf.pageview_unexpected_values [15:49:01] So I looked it up for what was going into it yesterday [15:49:53] hello madhuvishy :) [15:50:19] Hello elukey :) [15:54:37] hello [15:54:51] madhuvishy: right, it comes from teh whitelist check right? [15:58:20] I am going to ask on research channel if we need to count those pageviews [16:00:38] nuria: yeah it does come from the whitelist check [16:00:47] yup I can patch it if we need to count [16:01:02] madhuvishy: I think we are going to need to add the new domain to whitelist [16:01:09] nuria: oh cool [16:01:13] let me [16:01:34] madhuvishy: only that i think only joal or otto can do it [16:01:42] madhuvishy: both of which are ahem ... out [16:01:45] nuria: no - i have hdfs root [16:01:50] madhuvishy: ah , ok [16:02:08] madhuvishy: good cause it is really easy [16:03:52] Analytics-Kanban: All memebers of analytics team need to have hdfs root on cluster - https://phabricator.wikimedia.org/T126752#2022587 (Nuria) p:Triage>High [16:06:09] hive (wmf)> SELECT * from pageview_unexpected_values WHERE year=2016 AND month=2 AND day=12 limit 10; [16:06:22] first hive query, I am proud of myself [16:06:23] :D [16:06:52] madhuvishy: I will submit patch and we can reload table. [16:07:10] nuria: ah - i was just doing it [16:07:20] madhuvishy: teh changeset? or teh update? [16:07:44] changeset [16:07:59] nuria, madhuvishy: would you mind to add me to the code reviews just as FYI? [16:08:06] elukey: sure :) [16:09:00] (PS1) Madhuvishy: Add ady.wikipedia to pageview hourly whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/270308 [16:09:08] nuria: ^ [16:09:13] madhuvishy: ok, go ahead, let me create ticket [16:10:36] Analytics: ady.wikipedia needs to be added to whitelist of pageview domains - https://phabricator.wikimedia.org/T126754#2022605 (Nuria) NEW [16:10:47] Analytics: ady.wikipedia needs to be added to whitelist of pageview domains - https://phabricator.wikimedia.org/T126754#2022612 (Nuria) a:madhuvishy [16:11:11] (PS2) Madhuvishy: Add ady.wikipedia to pageview hourly whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/270308 (https://phabricator.wikimedia.org/T126754) [16:12:19] Analytics, Patch-For-Review: ady.wikipedia needs to be added to whitelist of pageview domains - https://phabricator.wikimedia.org/T126754#2022617 (Nuria) Until this is done pageview jobs will fail as domain is not recognized. [16:14:17] nuria: pageview jobs fail? they don't fail - they run fully - and trigger this alert after if it found anything unexpected [16:15:01] madhuvishy: ahhh, i thought we set them to fail (which will be no so good) but i did not checked oozie [16:15:36] nuria: i'm sure they succeeded - that's what i understand from workflow and oozie - checking the hive tables [16:15:58] (CR) Nuria: [C: 2 V: 2] Add ady.wikipedia to pageview hourly whitelist [analytics/refinery] - https://gerrit.wikimedia.org/r/270308 (https://phabricator.wikimedia.org/T126754) (owner: Madhuvishy) [16:17:11] madhuvishy: ok, I think we cannot recreate table w/o stopping jobs so if we can wait for ottto or joal that would be best [16:17:40] madhuvishy: ahem... and both are out today [16:17:51] nuria: hmmmm [16:23:14] nuria: I ran an insert into manually [16:23:30] which should fix it for now I think - we can deploy the change on monday [16:23:33] madhuvishy: k [16:23:52] madhuvishy: thanks for the prompt response [16:24:51] nuria: also I think s1-analytics-slave is an alias to analytics-store [16:25:17] you can't connect to mysql using stat1002 - we do it from there but the host is s1-analytics-slave [16:25:47] madhuvishy:ah , will correct, either way my.cnf on personal dir will work, right? [16:26:15] Analytics-Kanban: All members of analytics team need to have hdfs root on cluster - https://phabricator.wikimedia.org/T126752#2022641 (Nuria) [16:26:48] if you want I can work with you and Andrew/Joseph to deploy the change on Monday, it would be useful to me [16:28:06] nuria: ya probably - but I have never tried it [16:29:23] elukey: this is not the best change to learn from because there is no jar deploy etc - so you'll only see one half of it. but sure! [16:31:57] madhuvishy: standup? [16:32:03] oopss coming [16:39:13] Analytics-Kanban, Patch-For-Review: ady.wikipedia needs to be added to whitelist of pageview domains - https://phabricator.wikimedia.org/T126754#2022681 (Nuria) [16:43:13] Analytics-Kanban, Patch-For-Review: Buurow Increase length of window to evaluate lag [1 pts] - https://phabricator.wikimedia.org/T125916#2022684 (Nuria) Open>Resolved [16:43:24] Analytics-Kanban, Patch-For-Review: Investigate adding piwik to transparency report {3 pts] - https://phabricator.wikimedia.org/T125175#2022686 (Nuria) Open>Resolved [16:50:34] Analytics-Kanban, DBA, Editing-Analysis, Patch-For-Review, and 2 others: Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} [8 pts] - https://phabricator.wikimedia.org/T124676#2022707 (jcrespo) This is running now (and during the weekend) on the analytics... [17:09:55] Analytics: Make last access data public - https://phabricator.wikimedia.org/T126767#2022824 (Nuria) NEW [17:10:40] Analytics: Make visualization of last access data - https://phabricator.wikimedia.org/T126768#2022851 (Nuria) [17:26:21] nuria, madhuvishy : Just read the patch [17:26:44] joal: k [17:27:08] indeed, with that change refinery should be deoplyed and job should be restarted [17:28:23] nuria: Shall it wait mondya ? [17:28:46] joal: you are taking a day off right? If so, yes, let's wait [17:28:57] nuria: yes I'm off today [17:29:13] I'll deploy refinery on monday, restarting both pageview for check and uniques [17:29:42] nuria: I'll delete existing data for uniques (not madhu's, the one current prod jobs have written) [17:29:50] nuria: And I'll start jobs from jan 1st [17:36:06] joal: we can wait until monday joseph, really [17:36:14] joal: the joys of working on tier-2 [17:36:32] nuria: Yeah, that's what I said: I'll do that on monday :) [17:36:53] joal: k, just confirming. see ya' then [17:36:58] Byyyyyye :) [17:37:06] Thanks for confirmation :) [17:38:34] Analytics-Cluster, Analytics-Kanban, Reading-Admin, Easy, Patch-For-Review: PM sees reports on browsers (Weekly or Daily) {lama} [8 pts] - https://phabricator.wikimedia.org/T88504#2023044 (dr0ptp4kt) Sweet, thanks! [17:42:21] joal: please ping me when you are going to deploy :) [17:56:48] logging off a-team, have a good weekend! [17:56:57] have a nice weekend Luca [17:57:02] bye elukey ! nice weekend [18:36:26] laters! [18:55:44] Hey folks. Looks like I can't access analytics-slave via stat3. Is this a known issue? [18:55:54] "Lost connection to MySQL server at 'reading authorization packet', system error: 0" [18:58:14] Looks like I *can* access the same database server from stat1002 [18:58:32] Looks like I mistyped earlier. I am trying to access "analytics-store.eqiad.wmnet" [18:58:48] ottomata, ^ [18:59:34] halfak: not a known issue [19:00:07] OK. Will file. [19:00:11] danke [19:00:16] cc jynus [19:00:28] you might just ask him in #ops halfak, dunno [19:00:44] seems like it's not the DB server [19:00:51] Since I can access it from stat1002 [19:02:49] Oh!!! Someone moved the cnf file! [19:02:56] It looks like it might have been a puppet change. [19:04:10] Wait... nope. still failing. [19:04:13] Maybe intermittent [19:05:53] FYI: https://phabricator.wikimedia.org/T126800 [19:07:47] halfak: dunno where you got that defaults-file arg from, should be /etc/mysql/conf.d/research-client.cnf, no? still doesn't work though. [19:08:26] Oh. Hmm... that's what I meant my changed the location of the cnf file, but I just forgot to look in conf.d [19:08:35] So yeah. my mistake there [19:09:42] still though doesn't work! [19:09:53] Indeed. [19:34:07] madhuvishy: how did it go with zombie bug? [19:34:55] nuria: andrew helped figure it out - no bug in EL [19:35:05] madhuvishy: k, where was it? [19:35:11] there was an alternate EL running from deployment-zookeeper01 [19:35:23] madhuvishy: whatatatatat [19:35:29] i dont know why it got deployment there [19:35:37] madhuvishy: ok, i was not expecting that [19:35:39] it was running some 8 processors or something [19:35:42] which is why [19:35:46] yes! [19:35:50] beta was getting those repeat events [19:35:52] so [19:35:55] there are duplicated events [19:35:57] right [19:35:57] we killed it [19:36:11] nice, moving another task to done then [19:36:33] I could have never found out - andrew found by listing all the processes from deployment-salt [19:36:54] Analytics-Kanban, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event [3 pts] - https://phabricator.wikimedia.org/T125423#2023755 (Nuria) There was a stranded process that was duplicating events into kafka. Fixed now. [19:38:21] madhuvishy: i spent like, at least half a day making sure EL beta was sound when i was looking into the duplicated events, could not think of anything else... [19:38:30] yaaa [19:38:34] madhuvishy: but great job, one less thing to worry about [19:46:33] milimetric: moved stuff arround: https://meta.wikimedia.org/wiki/Research:Unique_Devices [19:46:51] https://wikitech.wikimedia.org/wiki/Analytics/Unique_Devices [19:57:26] Analytics-Kanban, Analytics-Wikistats, Reading-Admin: {lama} Wikistats 2.0 - https://phabricator.wikimedia.org/T107175#2023909 (Nuria) [19:57:28] Analytics-Kanban, Reading-Admin, Patch-For-Review: Tabular layout on dashiki [8 pts] {lama} - https://phabricator.wikimedia.org/T118329#2023908 (Nuria) Open>Resolved [19:58:32] Analytics-Kanban: Fix mediawiki Avro camus import after last weeks deploy [8 pts] - https://phabricator.wikimedia.org/T126622#2023916 (Nuria) Open>Resolved [19:59:25] Analytics-Kanban, Patch-For-Review: Create a dedicated hive table with pageview API only requests for reporting [5 pts] {melc} - https://phabricator.wikimedia.org/T118938#2023922 (Nuria) Open>Resolved [20:03:19] nuria: I shared the lightning talk slides - still very WIP and feel free to comment [20:05:52] madhuvishy: will look [20:08:56] madhuvishy: didn't get your slides... [20:09:02] madhuvishy: are they on a google doc? [20:09:46] nuria: yeah - i shared it - thought it would have sent an email [20:11:44] nuria - sweet [20:22:16] Analytics, Analytics-EventLogging: Convert EventLogging to use extension registration - https://phabricator.wikimedia.org/T87912#2023958 (Legoktm) Go for it :) [20:22:43] Analytics-EventLogging, Analytics-Kanban: Some blacklist matching schemas are being consumed by Eventlogging {oryx} - https://phabricator.wikimedia.org/T126410#2023962 (madhuvishy) This was not an EL bug, just something happening in beta cluster. There was a stray eventlogging setup on deployment-zookeepe... [20:22:46] Analytics, Analytics-EventLogging: Convert EventLogging to use extension registration - https://phabricator.wikimedia.org/T87912#2023963 (Milimetric) I have no idea how to do it :) but is it urgent? [20:24:02] Analytics-Kanban, Patch-For-Review: ady.wikipedia needs to be added to whitelist of pageview domains {hawk} [3 pts] - https://phabricator.wikimedia.org/T126754#2023972 (madhuvishy) [20:38:28] Analytics, Dumps-Generation: Provide a way to check if a dump has been generated - https://phabricator.wikimedia.org/T126808#2023999 (MarkTraceur) NEW [20:40:10] Analytics, Dumps-Generation: Avoid duplication of effort when processing dumps somehow - https://phabricator.wikimedia.org/T126809#2024007 (MarkTraceur) NEW [21:41:06] Analytics, Security, Zero, operations, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/zero - https://phabricator.wikimedia.org/T92343#2024150 (Dzahn) [21:41:25] Analytics, Security, Zero, operations, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#2024152 (Dzahn) [21:41:34] Analytics, Security, Zero, operations, and 2 others: Purge > 90 days stat1002:/a/squid/archive/mobile - https://phabricator.wikimedia.org/T92341#2024153 (Dzahn) [21:41:46] Analytics, Security, operations, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2024154 (Dzahn) [21:42:06] Analytics, Security, operations, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#2024156 (Dzahn) [21:44:43] Analytics-EventLogging, operations, audits-data-retention: Delete vanadium:/srv/eventlogging - https://phabricator.wikimedia.org/T75084#2024173 (Dzahn)