[00:19:59] ottomata (in absentia): was in a meeting, sorry. Yes, I'm tbayer (just filled out the IRC nick field on phabricator, and one shiny day i'll have my IRC cloak updated too ;) [00:27:06] (PS3) Madhuvishy: Make DB and DB user creation different for prod [analytics/wikimetrics-deploy] - https://gerrit.wikimedia.org/r/264207 [00:31:13] milimetric: are you around? if not ping when back :) [00:44:10] (PS1) Madhuvishy: Fix config paths in database_migrations/env.py [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/264222 [00:44:32] madhuvishy: finally i think i have verified that my code calculates correctly the offsets per country [00:44:37] madhuvishy: pufffff [00:44:50] (CR) Madhuvishy: [C: 2 V: 2] "Self merging" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/264222 (owner: Madhuvishy) [00:44:55] nuria: ooh awesomeee [00:45:04] i have been deploying wikimetrics all day [00:45:09] madhuvishy: now i need to run it for a month and see but really it took me a while... [00:45:33] madhuvishy: good, i am looking forward having an environment where it is not scary to deploy [00:45:59] nuria: yeah, a bunch of things i'm discovering in first time deploy - hopefully it should all be good [00:46:02] madhuvishy: as in " will ...everything ....break... now when i do the update ?" [00:46:09] he he [00:53:15] * Jamesofur waves to analytics [00:53:39] madhuvishy: to verify, we keep 30 days or 60 days of data in cluster? [00:53:52] nuria: 62 days of refined data [00:53:59] 30 for raw i think [00:54:04] k [00:54:20] madhuvishy: so dec should be there entirely [00:54:33] Does anyone around have access to https://piwik.wikimedia.org/ ? [short answer is I want an account for the next week or so but if that's not possible I can bother people for numbers too] [00:55:16] nuria: yes [00:55:21] Jamesofur: we do , but thus far there is only 1 admin account and we would like to create more [00:55:36] Jamesofur: thus if you send us a ticket of what you want it will be best [00:55:54] nuria: I'm going to want things a lot... ongoing for the next week or so [00:56:12] I'm trying to help the comms team watch their banner clicks closely from 15.wikipedia [00:56:20] (from the main banners) [00:56:31] and need the views from the site there (with some variables being passed in) [00:56:37] Jamesofur: I am not sure we can do anything this week , we are in the midst of a crisis with EL that is going to take a bunch of our time [00:56:42] ugh [00:56:54] nuria: is it in hive at all? [00:57:13] it's essentially useless data for me after this week [00:57:20] I just realized it was using piwik, had assumed it would be in hive [00:57:28] (hence why I didn't ask earlier) [00:57:37] Jamesofur: pageviews are in hive yes. [00:57:47] Jamesofur: piwik just has a lot more info [00:58:18] Jamesofur: but neither one is updated real time, you know that right? [00:58:34] Jamesofur: piwik will add visist real time but will compute its info i think is daily [00:59:08] nuria: yup, but I can get the page view/visit data moderatly quickly. I have to get banner view data out of hive anyway (which means I have to wait for it to process [00:59:23] Jamesofur: but given that data started to come in today we have not verified the computations are going yet [00:59:34] Jamesofur: ok [00:59:51] yeah, would be useful to have access to the rest but most important to me is the page views so that I can see how many people are coming from the banners [00:59:55] Jamesofur: you can ping milimetric if I am not arround. We just need to create another piwik account [01:00:01] thanks [01:00:45] Jamesofur: do ping us tuesday, we should be doing a lot better with EL [01:01:27] nuria: will do, the banners will be going off at that point but we can do some analysis of the what happened, with the page view data in hive I'll at least be able to help them with the more immediate stuff (the banners) until then. [01:01:51] Jamesofur: let me see if piwik iunstance is getting real numbers, was website announced? [01:02:51] Jamesofur: I need to check who has permits to create users i think is only ops and thus the conflict with EL stuff [01:03:33] Jamesofur: ya,there is data now, about 2000 visits [01:03:36] nuria: yeah, there are banners going there now [01:03:39] at low levels [01:03:46] that will increase to 100% anon in the next couple hours [01:03:52] Jamesofur: half of them from the US [01:04:13] that makes sense [01:04:40] Jamesofur: let me see if i have permits to create a user cause i just saw a ticket about that [01:04:55] nuria: thanks! [01:07:05] Jamesofur: ahhahah [01:07:16] Jamesofur: dan already created you guys a user [01:07:26] Jamesofur: ok, we are good, I was afraid this required ops [01:07:30] Ha! Who is it? I'm happy to hit them over the head :) [01:07:32] nice [01:07:40] yeah that would be a pain [01:07:45] (PS4) Madhuvishy: Make DB and DB user creation different for prod [analytics/wikimetrics-deploy] - https://gerrit.wikimedia.org/r/264207 [01:08:01] Jamesofur: ping dan tomorrow for password (milimetric) and he can give it to you on a safe manner [01:08:07] will do [01:08:07] thanks [01:08:46] Jamesofur: but we are good, it is all set and we have permits to administer the UI so we are ok for now [01:08:55] perfect! [01:09:36] Jamesofur: I'll ping you with the user / pass, I assume you got past the first basic authentication prompt? [01:09:45] (I used my wikitech account for that) [01:09:49] I did yeah [01:09:59] I got excited at first that was all I needed ;) [01:10:01] Excellent, one less thing to worry about [01:13:34] madhuvishy: pong [01:13:38] milimetric: prod server is set up. let me know if you wanna restore dbs [01:14:01] https://tools.wmflabs.org/sal/wikimetrics [01:14:31] milimetric: can you also merge https://gerrit.wikimedia.org/r/#/c/264207/ - fixes some issues that came up along the way [01:15:09] (CR) Milimetric: [C: 2 V: 2] Make DB and DB user creation different for prod [analytics/wikimetrics-deploy] - https://gerrit.wikimedia.org/r/264207 (owner: Madhuvishy) [01:15:18] cool [01:18:27] madhuvishy: so what's done and what's left, infrastructure wise? [01:18:31] fabric creates the db now? [01:18:47] milimetric: yeah - its set up. alembic upgrade has run too [01:19:00] need to restore the data [01:19:06] hm, k, so i should be able to just do that with mysql dump [01:19:11] yes [01:19:13] and then copy the files to /static [01:19:26] cool, I've gotta do some chores but will be back in a bit and do that [01:20:09] use mysql -h tools.labsdb -u labsdbuser -p labsdbpassword from any instance in wikimetrics project [01:20:31] how do i get those credentials again? I don't even remember the user [01:21:03] well they should ideally appear if you clone wikimetrics-deploy with submodules. but i need yuvi to merge the patch [01:21:06] so until then [01:21:17] i'll pm gerrit patch [01:22:49] milimetric: i'm also wondering how the file paths are set up for the reports - i had to change them in a bunch of places [01:25:37] ok i think those are good [01:32:19] (PS1) Madhuvishy: More config path fixes [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/264235 [01:33:48] (PS2) Madhuvishy: More config path fixes [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/264235 [01:34:15] (CR) Madhuvishy: [C: 2 V: 2] "Self merging" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/264235 (owner: Madhuvishy) [01:43:19] Analytics: python-mwapi fails on import in Python 2.7 - https://phabricator.wikimedia.org/T123201#1936007 (ResMar) According to the [cross-filed issue](https://github.com/mediawiki-utilities/python-mwapi/issues/25) on GitHub this module was intended to be `3` only. This is a bit of an [issue](https://phabric... [01:43:38] Analytics: python-mwapi fails on import in Python 2.7 - https://phabricator.wikimedia.org/T123201#1936008 (ResMar) Open>Invalid [02:16:46] milimetric: just in case, you should be able to access secrets/wikimetrics by cloning wikimetrics-deploy now [02:32:05] Analytics-Wikistats: About the api format type to check the siteinfo - https://phabricator.wikimedia.org/T123702#1936082 (Cwek) NEW [06:11:40] Analytics, Analytics-Cluster: https://yarn.wikimedia.org/cluster/scheduler should be behind ldap - https://phabricator.wikimedia.org/T116192#1936207 (Tbayer) Yes, replacing bast1001.wikimedia.org with stat1002.eqiad.wmnet works for me: ``` ssh -N stat1002.eqiad.wmnet -L 8088:analytics1001.eqiad.wmnet:808... [07:06:05] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1936230 (faidon) Replag is ~50,000 seconds now, slowly decreasing since I killed that big query yesterday. This query is still running, for more than a day now:... [10:49:44] nuria: Hey, let me know when you're online, I have an idea to speed up you last-access- jobs [12:17:19] Analytics, operations: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#1936541 (MoritzMuehlenhoff) [13:05:17] Analytics, Analytics-Wikistats, DevRel-January-2016: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1936647 (ezachte) I reached out by mail to @Nemo_bis with comments on each open patch. [14:32:05] Analytics, Research consulting, Research-and-Data: Update official Wikimedia press kit with accurate numbers - https://phabricator.wikimedia.org/T117221#1936715 (ezachte) Since early Dec 2015 there is a new chart on 'Active wikis' which will help us to assess a good cut-off point. http://stats.wikime... [14:40:54] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1936731 (Ottomata) I talked to Neil p Quinn about it. Decided to just let it go, since I was able to find a work around to allow some EL tables to sync. Neil sa... [14:46:32] yellow there, is this html static, or the data is in any database? -> https://stats.wikimedia.org/EN/TablesWikipediaPT.htm [15:02:18] a-team, i'm away, will be back [15:11:33] (PS1) Mforns: Divide app session metrics job into global and split [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) [15:13:02] (CR) Mforns: Divide app session metrics job into global and split (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [15:25:18] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, Zero, operations, and 2 others: Request one server to suport piwik analytics - https://phabricator.wikimedia.org/T116312#1936788 (Cmjohnson) [15:25:22] Analytics, Ops-Access-Requests, operations, Patch-For-Review: add mforns, milimetric, nuria,ottomata, madhuvishy and joal to piwik-roots - https://phabricator.wikimedia.org/T122325#1936785 (Cmjohnson) Open>Resolved a:Cmjohnson The admin role has been added to and group added to piwki.yaml [15:27:58] Analytics-Wikistats: Provide a JSON output for wikistat tables - https://phabricator.wikimedia.org/T123742#1936789 (Alchimista) NEW [15:53:32] joal: hola [15:54:29] Alchimista: that data is only on that static page, it processed from db data into what you can see [15:59:12] Analytics, Analytics-Cluster: https://yarn.wikimedia.org/cluster/scheduler should be behind ldap - https://phabricator.wikimedia.org/T116192#1936906 (Nuria) >Thanks! Should we amend the documentation on Wikitech accordingly? Please add the workarround to docs, thank you [16:07:13] nuria, i think it wouldn't be too hard to change the script so that it can output json or similar schema, even a simple .csv. html scrapping is a nightmare :s [16:07:36] unfortunelly my perl skills aren't that good :s [16:08:28] Alchimista: we are working on replacing this infrastructure with newer data dumps but we are focusing on pageviews first , edit data will come later so we will not be getting to these changes in the near term [16:09:02] Alchimista: but ask in research channel, maybe you can work with any of the many other data dumps that there are to date [16:10:44] thanks. now with the wikipedia aniversary people is hungry with data, so at least i'll have 2 or 3 months to perform what i need :d [16:12:51] (PS2) Mforns: Divide app session metrics job into global and split [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) [16:17:03] (PS1) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [16:17:11] Analytics: Modify vagrant wikimetrics install to account for recent changes to puppet to get rid of the self-hostyed puppet master - https://phabricator.wikimedia.org/T123749#1936932 (Nuria) NEW [16:17:45] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [16:18:00] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [16:21:57] (CR) jenkins-bot: [V: -1] Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [16:23:59] mforns: yt? [16:24:03] nuria, yes [16:24:11] how are you? [16:24:16] mforns: where was the source of teh demo app for pageview api? [16:25:23] nuria, https://gist.github.com/marcelrf/49738d14116fd547fe6d [16:25:46] Analytics: Pageview API demo is broken in Firefox and does not show date selector arrows in Chrome - https://phabricator.wikimedia.org/T123584#1936959 (Nuria) Source at: https://gist.github.com/marcelrf/49738d14116fd547fe6d [16:46:03] * elukey needs to stop saying hello to halfak's figure in the office [16:46:13] * elukey needs probably more coffee [16:47:05] NO! Keep saying hello. And feed me coffee. :) [16:47:07] elukey: :) [16:48:44] halfak: your figure would appear like a zombie in no time if I do it [16:49:10] (for the coffee spilled not for the hellos) [16:49:27] OK. Maybe no coffee, but you could pick me up and put me in a fun place. [16:49:41] Extra points if you put me in a place where I can see myself during the next metric meeting. [16:49:46] :D [16:50:53] Analytics: Pageview API demo is broken in Firefox and does not show date selector arrows in Chrome - https://phabricator.wikimedia.org/T123584#1936986 (Nuria) As far as I can see demo works fine in the last version of FF. 40.0 on Mac os X [16:51:17] halfak: metric-meeting == 5th floor lounge right? (still trying to remember everything) [16:52:08] Yeah. That's right. [16:52:16] :DDD [16:52:32] * elukey goes to the 5th floor [16:52:39] \o/ [16:52:57] * halfak imagines the elevator conversation [16:53:04] Analytics: Pageview API demo is broken in Firefox and does not show date selector arrows in Chrome - https://phabricator.wikimedia.org/T123584#1936991 (Milimetric) The demo sometimes doesn't work in FF if you have security settings that prevent CORS requests. [16:53:22] "What are you doing with Aaron?" He wanted to go to the 5th floor. "Okaaaaay..." [16:57:24] Analytics: Pageview API demo is broken in Firefox and does not show date selector arrows in Chrome - https://phabricator.wikimedia.org/T123584#1936998 (Nuria) >The demo sometimes doesn't work in FF if you have security settings that prevent CORS requests. Since we cannot fix that I will just retitle this tas... [16:57:38] Analytics: Pageview API demo does not show date selector arrows in Chrome - https://phabricator.wikimedia.org/T123584#1937000 (Nuria) [16:59:23] halfak: I actually met somebody going to the 4th floor and after some smiles I had to say: "I know it is a bit weird, don't be scared" [16:59:28] (PS2) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [16:59:49] she asked if it was your birthday [16:59:52] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [17:00:25] Now the trick is to make sure it doesn't get moved out of view before the next meeting up there. [17:00:26] standddddupppp a-team [17:00:31] now you are nearby the presentation deck keeping company to a plant [17:00:37] :D [17:00:39] elukey, madhuvishy , mforns standduppp [17:00:46] o/ have fun standing [17:02:40] Analytics-Kanban: Gather phabricator metrics for quaterly review [3 pts] - https://phabricator.wikimedia.org/T122333#1937012 (Milimetric) a:Nuria [17:03:15] elukey: you can join me in this room if you like - diderot - one of the hallway rooms [17:03:29] if you are on 3rd [17:05:20] Analytics-Kanban: Create a set of celery tasks that can handle the global metric API input {kudu} [0 pts] - https://phabricator.wikimedia.org/T117288#1937026 (Milimetric) a:madhuvishy [17:05:31] Analytics-Kanban: Implement a simple public API to calculate global metrics {kudu} [0 pts] - https://phabricator.wikimedia.org/T117285#1937027 (Milimetric) a:madhuvishy [17:28:19] Analytics: Modify wikimetrics local install to account for recent changes to puppet to get rid of the self-hostyed puppet master - https://phabricator.wikimedia.org/T123749#1937073 (Nuria) [17:28:43] Analytics: Modify wikimetrics local install to account for recent changes to puppet to get rid of the self-hostyed puppet master - https://phabricator.wikimedia.org/T123749#1936932 (Nuria) Docker container might be an option. [17:30:17] milimetric: you said you had a question for me? [17:30:25] or was it for madhu? [17:30:26] ok, madhuvishy, so the new redis setup, it looks like it keeps the redis rdb file in /srv/redis/queue.rdb [17:30:30] oops [17:30:31] sorry madhuvishy [17:30:39] no, /srv/redis/queue/dump.rdb [17:30:40] ah I see, ok [17:30:54] right nuria, for madhu [17:31:44] ok, very well. [17:32:14] milimetric: yeah /srv/redis/queue sounds right [17:32:55] ok, madhuvishy so I'm going to just copy the current rdb over and rename it to dump, hopefully that just works [17:33:05] milimetric: alright [17:33:06] i guess i'll stop / start redis [17:36:36] darn... redis just wipes out that file when it starts again [17:46:21] madhuvishy, do you have 10 mins for questions on testing the app sessions job today? :] [17:55:11] milimetric: / [17:55:32] mforns: sure [17:55:52] madhuvishy, cool, let me know when is good for you [17:59:02] madhuvishy: I'm running a query on an EL table, it's taking much longer than it should. [17:59:18] lzia: EL tables are problematic atm [17:59:19] madhuvishy: I'm a bit worried that I bring down the server. [17:59:26] replication issues etc [17:59:29] I see. so maybe I should give it some days? [17:59:54] as in, wait for some days to run it or does the query run for days? [18:00:06] the columns I'm handling are indexed, so I was hoping that it can be done quickly, madhuvishy. [18:00:34] no, the query should run quickly. I'm wondering if I should wait for some days for things to go back to normal before running the query [18:00:50] lzia: that's fine - if it's a long running query I might wait - if not you should be fine even if it's slow i think [18:01:51] got it, madhuvishy. I'll let it run for another 30 min, if it doesn't end, I'll stop it. I don't expect it to need more than 15 min. It's been running for half an hour for now. [18:03:19] (PS3) Mforns: Divide app session metrics job into global and split [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) [18:03:39] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [18:06:56] mforns: batcave? [18:07:01] madhuvishy, sure! [18:11:34] https://www.irccloud.com/pastebin/ukFHVO1Y/ [18:11:40] mforns: ^ [18:13:46] Hey a-team, I'm back ! [18:14:22] o/ [18:16:23] lzia: okay :) our dba is on vacation so things should get better when he's back [18:16:52] also make sure to check if the latest data is available because there's been some lag, and not all tables have caught up [18:17:06] got it, madhuvishy. I'm going to stop the query now. It can just make things worse. [18:19:53] morniiing [18:19:57] wb, joal! [18:22:01] Jamesofur: FYI, that you will not see SEO traffic on piwik [18:22:14] Jamesofur: load was too high and we removed teh beacons for it [18:22:30] * Jamesofur nods [18:22:35] thanks for letting me know [18:22:48] Jamesofur: or stats from people not running JS [18:22:53] nuria, so I'm now seeing WikipediaPortal_14377354 completely up to date. Should I expect the same from other tables, or is this as a result of Otto giving it a kick, do you know? [18:22:56] there are worse issues [18:23:08] in general the site is getting a ton of traffic - so good news [18:23:13] Ironholds: only "small" tables will benefit from our ptach [18:23:15] *patch [18:23:16] but piwik may not be able to keep up with it, we'll see [18:23:43] Ironholds: or rather, tables with "small" updates [18:23:53] Ironholds: so, no, not all tables will be ok [18:24:09] Ironholds: if it's caught up and it doesn't get a huge number of events, it should be ok from now on. [18:24:24] but any table that gets more than 1000 events until the script gets back to it will lag [18:24:46] milimetric, gotcha! [18:24:48] nuria, thanks! [18:24:58] okay, so basically I should go through the tables we're dependent on and check each one [18:25:20] for now, yeah, hopefully Jaime will have more of an idea next week, we'll follow up with him [18:25:46] yay! [18:28:39] (CR) Joal: [C: -1] "Hey Marcel," [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [18:29:05] nuria: Hey M'dame [18:29:18] joal: yes [18:29:40] To have you job go faster, remove the reducers number parameter [18:29:49] joal: ok [18:29:52] It'll go significantly faster [18:30:01] joal: will do [18:30:15] This is the mistake I made for the monthly top as well [18:30:19] Analytics-Cluster, Analytics-Kanban, Reading-Admin, Easy, Patch-For-Review: PM sees reports on browsers (Weekly or Daily) {lama} [8 pts] - https://phabricator.wikimedia.org/T88504#1937265 (dr0ptp4kt) @mforns, yes, the distinct fields of OS, OS major, browser family, browser major should be in the... [18:30:24] joal: ok, will do [18:30:57] joal: will let actual job finish as it is more than half way there i think [18:31:02] np [18:31:04] joal: but will do next run w/o [18:31:16] nuria: piwik started ? [18:31:35] joal: piwik is chocking with traffic but , hey, there is our load test [18:31:41] :) [18:31:44] joal: we just removed bot traffiq [18:32:09] nuria: Do you need me to get involved for backup, or not yet ? [18:32:11] joal: it should catch up, we'll see tomorrow [18:32:29] If no, I'll finish my analysis on sanitization [18:32:31] joal: no, milimetric 's got the piwik problems , we just fixed 1 thing and deployed it [18:32:37] Awesome [18:32:39] joal: i think we are ok [18:32:56] milimetric: Thanks for users you 'll make happy :) [18:41:42] milimetric: were you able to restore the redis file? [18:42:35] Hey mforns: do you have a minute ? [18:42:56] joal, yes I've seen your review, currently doing changes [18:43:02] batcave? [18:43:28] mforns: wanted to ask about information entropy [18:43:32] sure, batcave ! [18:43:33] ok [18:43:56] I'll take advantage of that moment to ask what I have missed in standup ;) [18:55:02] milimetric: hi! i'm not sure if you're the right person to bother on this but i've noticed that /var/log/eventlogging/all-events.log has not been available on deployment-eventlogging03 for some time. additionally, new tables do not seem to be made in the database. today, i noticed that hardly anything is being logged in /var/log/upstart/*.log. am i [18:55:03] just on the wrong server? [18:56:16] madhuvishy: not able yet, but that piwik thing came up and now I'm having lunch [18:57:03] niedzielski: I'm not sure how that was all changed, but try deployment-eventlogging04 as well. [18:57:28] we recently had some trouble with production eventlogging not replicating properly and other folks reported the same problems you're reporting, but we haven't had time to look at them [19:00:57] milimetric: ah, same thing on 04. this is for beta stuff only [19:01:23] yes, beta, I meant production kept us busy and now we don't have otto around to take a look at beta [19:02:04] milimetric: ah ok. i'll open a phab then? [19:02:18] yeah, thanks niedzielski, that'd be best [19:02:21] tag it Analytics [19:02:27] milimetric: will do. thanks! [19:06:12] mforns: did we unbreak beta yet? [19:06:42] madhuvishy, yes, I reenabled puppet, and deployed a fixed version, so it should be OK [19:07:15] deployment-eventlogging03 right? [19:08:53] halfak: Hi sir [19:53:02] milimetric: okay [19:59:34] Analytics, Wikipedia-Android-App: Beta Event Logging no longer functional - https://phabricator.wikimedia.org/T123781#1937550 (Niedzielski) NEW [21:22:42] milimetric: let me know if you wanna do the restore together [21:22:45] madhuvishy: how'd you get stuff into https://tools.wmflabs.org/sal/wikimetrics as opposed to https://tools.wmflabs.org/sal/analytics [21:23:02] milimetric: I logged it on #wikimedia-labs [21:23:18] they have it set up such that if you do !log wikimetrics blah [21:23:27] wikimetrics is treated as a project [21:23:28] ok, cool, I'll do that [21:23:35] i figured out the redis thing, I think [21:23:40] oh awesome [21:23:42] it was weird and backwards from what everyone said online [21:23:48] ah [21:23:49] they all said to stop, copy the file, start [21:24:04] ya that's what i thought too [21:24:10] but that didn't work, so I tried just copying the file [21:24:16] but you should start, copy the file and be happy? [21:24:17] and it looks like it started just writing to it [21:24:23] right... :) [21:24:29] okayy [21:24:57] then I linked the files from wikimetrics/src/wikimetrics/static/public -> /srv/var/wikimetrics/public [21:25:03] so that should be good [21:25:20] now I'm trying to restore the db, figuring out what the mysqlimport or dump command is or whatever [21:25:32] and then I've gotta look into the python data aggregator thing and all the metrics we host on there [21:25:44] is that a symlink? should we set it up in some permanent way via puppet? [21:26:04] I'm going to turn off the scheduler on the prod instance actually, 'cause if I don't it'll keep running jobs [21:26:05] there's more things? [21:26:14] milimetric: yeah alright [21:26:17] yes, but i think they were all puppetized [21:26:23] i guess some of those things didn't carry over [21:26:41] but it's ok, that's why i wanted to start logging so I don't get lost [21:26:45] gonna go to labs [21:26:50] milimetric: which parts? https://github.com/wikimedia/operations-puppet-wikimetrics/ [21:26:59] ok [21:29:12] milimetric: can you help me understand why we need to symlink? [21:29:29] sure, wanna batcave? [21:32:26] milimetric: sure omw [21:51:38] milimetric: journalctl -u uwsgi* [21:53:35] todos: https://etherpad.wikimedia.org/p/analytics-todo [22:04:02] https://etherpad.wikimedia.org/p/analytics-todo [22:12:48] Analytics, operations: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#1938161 (Ottomata) [22:15:35] Analytics, Analytics-Cluster: https://yarn.wikimedia.org/cluster/scheduler should be behind ldap - https://phabricator.wikimedia.org/T116192#1938170 (Ottomata) Hm, yeah maybe should just change the docs. I betcha there is some new firewall rule in place that was keeping bast1001 from working. [23:00:03] (PS1) Milimetric: Remove comScore data from reportcard [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/264404 [23:00:13] (CR) Milimetric: [C: 2 V: 2] Remove comScore data from reportcard [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/264404 (owner: Milimetric) [23:03:54] madhuvishy: I found how the aggregator symlinks and stuff went missing [23:03:58] they were deleted from here: [23:03:59] https://github.com/wikimedia/operations-puppet/blob/32e0002f3dfa76caa10f98360a3b9aeae0946be0/manifests/role/wikimetrics.pp#L324 [23:04:10] (basically from that line to above the logster section at the bottom of that file) [23:07:08] (PS1) Milimetric: Link reason for stopping comScore updates [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/264406 [23:07:29] (CR) Milimetric: [C: 2 V: 2] Link reason for stopping comScore updates [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/264406 (owner: Milimetric) [23:08:46] milimetric: yeah i understood that - but should this really be in the wikimetrics instance? [23:09:00] i think i was confused when i first saw it and then forgot in the holidays [23:09:49] I have no opinion on whether it should or shouldn't be. But I also don't want to do the work to change it to something else. So it's up to you, but until we fix it, vital signs won't have the data it needs [23:10:58] (I didn't set it up this way) [23:11:43] madhuvishy: so you tell me, do you want to change it or keep it the same? [23:12:16] milimetric: i don't know enough about the ideas behind the way vital signs is set up either [23:12:38] it's extremely simple [23:12:50] vital signs has no idea about wikimetrics, the aggregator, or anything [23:12:59] it just displays some metrics [23:13:13] those metrics are configured to get their data from static files on metrics.wmflabs.org/... [23:14:06] it's slightly more complicated from the wikimetrics side but also simple [23:14:18] wikimetrics computes something like 6 metrics on a nightly basis [23:15:02] it puts the static file results in a place that gets served from static/public [23:15:24] so then I think when people added the pageview stats, they thought it would be symmetric to serve them from the same place [23:15:31] milimetric: i understand the metrics it gets from wikimetrics [23:15:50] so they just cloned the repos there and symlinked them from the same place the other metrics are [23:15:55] just not the pageview stats being served from this instance - then it become prod specific - not really needed [23:16:04] prod specific? [23:16:16] like, staging wikimetrics doesn't need it no [23:16:25] our test setup is completely unaware too [23:16:34] i had no idea i would break something else [23:16:41] when setting up this projec [23:16:43] project [23:17:01] it's ok, like 3 people look at vital signs anyway [23:17:43] I kind of agree with your point, but then again, just deleting puppet setup is likely to break something :) [23:17:57] (hiiii :) ) [23:18:20] hey :) [23:34:57] Analytics-Tech-community-metrics: SCM project 'core' contains Pywikibot and MediaWiki - https://phabricator.wikimedia.org/T123808#1938451 (jayvdb) NEW [23:54:14] milimetric: okay, i unbroke the CORS and static files serving parts [23:55:05] i will add the cloning to prod on puppet, but i would like that to go away - i don't know how yet - may be a separate vital signs role that handles it