[00:16:13] (03PS1) 10HaeB: Fix iOS app detection for monthly and daily uniques [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349140 (https://phabricator.wikimedia.org/T163403) [00:28:12] BTW why does https://analytics.wikimedia.org/dashboards/browsers/ have a "Mobile Site by Browser" tab but no corresponding "Desktop Site by Browser" tab? [03:43:33] (03Abandoned) 10Nuria: Bookmark for browser dashboard regarding graph and time [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) (owner: 10Nuria) [03:44:07] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3196485 (10LilyOfTheWest) >>! In T115042#3195557, @Nuria wrote: >>We are interested in getting some data to them to judge the effectiv... [03:50:56] 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Change userAgent field to user_agent_map in EventCapsule - https://phabricator.wikimedia.org/T153207#3196490 (10Nuria) @Tbayer: there are two things 1) what is persisted to DB (after an event is valid) for some schemas and 2) what is sent on... [03:53:41] HaeB: cause none has asked for it to date. [04:23:59] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3196508 (10Nuria) >Basically, because banners are heavily used by the community, it makes sense to empower the community to analyze th... [07:41:51] !log Restart mediacounts-archive-wf-2017-04-19 in Hue (Java Heap space issue) [07:41:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:42:04] hello people :) [08:29:39] Hi elukey :) [08:29:57] elukey: do we have anything in eqiad row-D (switch replacement question) [08:30:35] elukey: Thanks for restarting the mediacount-archive ;) [08:31:17] joal: yes we have, I mentioned it some days ago in the chat and during standup :) [08:31:33] elukey: sorry for not having paid enough attention :( [08:32:11] hahahaha I am jokiiingggg [08:32:43] I am rechecking Arzhel's email and this is the status afaics [08:32:47] While I kinda know, I still want to say I'm sorry :) [08:33:24] joal: please don't say that, if you'd ask me to repeat the status of mw history or anything else I'd be clueless [08:33:34] huhu :) [08:33:46] :) [08:34:01] soooo there are two kind of "maintenance" that will be done in that day [08:34:32] 1) extended for two racks only, that should meant ~minutes of downtime due to host being moved to another rack [08:34:54] 2) brief downtime just to reconnect hosts to a different switch (no rack moves) [08:35:04] so for 1) we have a bit of an issue with kafka nodes [08:35:15] because since we have two nodes per rack [08:35:39] we'd need to tear down 2 nodes in once [08:35:58] I don't understand 1 elukey [08:37:05] joal: let's batcave [08:37:11] do you have time? [08:37:14] I do [08:38:26] I da cave [09:01:21] 10Analytics-Tech-community-metrics, 10Phabricator, 06Developer-Relations (Jan-Mar-2017): Decide on wanted metrics for Maniphest in kibana - https://phabricator.wikimedia.org/T28#3196873 (10Aklapper) I'm happy to mention once basic data quality issues (T157898 and T161235, and to some extent T157709 and a bit... [09:34:14] joal: did you see https://phabricator.wikimedia.org/T159219#3193121 ? wdyt? [09:35:21] elukey: I had not follwoed in detail, except for the fact that it worked great :) [09:35:30] It does indeed reduce GC a lot [09:39:28] * joal bows to java master elukey [09:40:09] nono I am far from that, just wanted to know if in your opinion this was good/not-good :) [09:40:31] I like the reduced GC times but of course the heap is used more.. should be ok for us but I wanted your opinion :) [09:41:09] last step is to apply it to the Yarn Resource Manager [09:41:10] (348915) [09:41:14] elukey: I'm no java master - I assume less GC means RAM used in a more efficient way - But I can't be sure :) [09:41:45] and less pauses (more or less "violent" for the running app) [09:42:39] all right let's call it a win then :) [09:42:49] I really think it is :) [09:42:57] Thanks a lot for having gone through that ! [10:14:39] heads-up: just applied the Xms settings to the Resource Manager on 1002 [10:14:51] let's see how it goes [10:15:00] and tomorrow I'll apply it to 1001 [10:15:16] or maybe this afternoon that is better than Friday :) [10:16:09] elukey: +1 for no friday ;) [10:17:49] okok I'll do it today [10:18:01] we also have to upgrade Piwik today! [10:18:03] \o/ [10:18:17] elukey: /o\ [10:31:42] 06Analytics-Kanban: Investigate rise in IE views from Pakistan since 2015 - https://phabricator.wikimedia.org/T157404#3197131 (10fdans) The countries by use of anomalous UA string is the following: ``` SELECT geocoded_data['country_code'], count(*) ct FROM wmf.webrequest WHERE webrequest_source = 'text' AND y... [10:56:00] 06Analytics-Kanban: Investigate rise in IE views from Pakistan since 2015 - https://phabricator.wikimedia.org/T157404#3197198 (10fdans) More on the usage of that UA string, I ran the following to get how many unique ips are actually using it: ``` SELECT count(DISTINCT ip) unique_ips, count(*) total_reqs FROM wm... [11:03:34] dinner break! will be back in a bit :) [11:14:54] (03PS1) 10Joal: Bump jar version for mediawiki history denormalize [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349200 (https://phabricator.wikimedia.org/T157362) [11:15:13] (03CR) 10Joal: [V: 032 C: 032] "Self merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349200 (https://phabricator.wikimedia.org/T157362) (owner: 10Joal) [11:41:33] * elukey lunch! [12:16:02] restarted yarn rm on an1001 [12:16:10] so 1002 is temporary the active [12:16:15] I'll switch back in a few [12:23:01] taking a break a-team - later ! [12:30:26] 1001 back to active [12:46:53] 06Analytics-Kanban, 13Patch-For-Review, 15User-Elukey: Apply Xms Java Heap settings to all the Hadoop daemons - https://phabricator.wikimedia.org/T159219#3060783 (10elukey) Applied settings to the Yarn Resource Manager daemons. [12:59:27] elukey: around if you want to do the piwik thing [12:59:42] joseph: whenever you're back, you said we should work on a patch this morning [13:02:22] milimetric: \o/ [13:02:30] okok [13:02:47] so the plan would be to [13:03:09] 1) put piwik in maintenance mode and disable the tracker [13:03:18] 2) take the mysql dum [13:03:21] *dump [13:03:32] 3) install piwik [13:03:38] 4) run the upgrade script [13:04:05] 5) remove maintenance mode and check [13:04:12] milimetric: wdyt? [13:04:39] * milimetric looking up wdyt [13:04:52] what do you think [13:04:56] yeah, good plan [13:05:22] If you want to blame me for acronyms I'll deflect to ottomata :D [13:07:50] those who live in glass houses shouldn't throw stones, I'd never blame you, I'm so bad! :) [13:08:10] ok, elukey I looked at the data for every site and took screenshots just to have a quick idea of what a good restore will look like [13:08:48] I'm not worried about data loss, just as a sanity check that the restore goes ok [13:10:44] super [13:11:42] 𝘡𝘩𝘰𝘴𝘦 𝘸𝘩𝘰 𝘭π˜ͺ𝘷𝘦 π˜ͺ𝘯 𝘨𝘭𝘒𝘴𝘴 𝘩𝘰𝘢𝘴𝘦𝘴 𝘴𝘩𝘰𝘢𝘭π˜₯𝘯'𝘡 𝘡𝘩𝘳𝘰𝘸 𝘴𝘡𝘰𝘯𝘦𝘴 [13:11:53] never heard that before, I'm stealin it [13:14:31] heh, me and Obama have in common that we incorrectly think it comes from the bible [13:14:44] it's just an old proverb [13:16:09] milimetric: Piwik should be in mainteance mode now [13:16:16] cool [13:16:46] but for some reason I can login via ldap [13:16:48] mmm [13:18:14] ah no ok [13:18:17] mainteance good [13:18:25] backupping db [13:20:18] Hey all! [13:20:25] o/ [13:20:48] Whats the best mailing list to start up a conversation about dumping some wikibase related tables in hadoop to query? [13:21:01] I'm guessing analytics? [13:21:08] or would a phab ticket be a better approach? [13:24:50] addshore: mailing list or phab would be good [13:24:59] milimetric: one thing that I didn't check was the size of the db [13:25:15] it seems that it holds ~68GB of data on disk [13:25:23] and the mysqldump is already 10G [13:25:32] wow [13:25:39] that's a lot bigger than I'd think [13:26:35] and we have only 100G [13:27:10] so this is a bit of an issue [13:28:08] need to do it on a host via ssh [13:37:33] you could pipe it into zip if that's too slow, maybe [13:38:26] but also that's a little close for comfort in general, like we may run out of space. That's weird 'cause there shouldn't be that much data [13:38:53] yeah I am trying to send it to an1003 [13:44:16] milimetric: ok the dump is in progress [13:44:19] super hacky way [13:45:14] hey, good work, sorry I didn't realize the space was so tight earlier [13:45:24] 10Analytics-Cluster, 06Analytics-Kanban: Hadoop cluster expansion. Add Nodes - https://phabricator.wikimedia.org/T152713#3197616 (10Cmjohnson) [13:50:47] Dump completed on 2017-04-20 13:48:36 [13:51:23] milimetric: proceeding with the install [13:51:49] (03PS1) 10Milimetric: Update banner with prototype consultation [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/349217 [13:51:58] cool [14:00:04] so Executing ALTER TABLE `piwik_log_visit` MODIFY COLUMN `visit_entry_idaction_url` INTEGER(11) UNSIGNED NULL DEFAULT NULL... seems taking a long time :) [14:00:28] yeah, makes sense, lots more data than test environment [14:07:48] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3197679 (10Ottomata) @Halfak, @leila. We've got some quotes back from vendors, and have a choice between the [[ http://www.amd.com/en-us/products/graphics/workstatio... [14:10:27] milimetric: good news is that we have one backup of Piwik :D [14:11:15] backup! so fancy [14:13:50] ah milimetric did you see my updated for the Cookie/cache thing? [14:14:00] looking [14:16:19] elukey: weird that otto can't get it to pass no matter what and I always do on the direct file download [14:16:37] milimetric: did he try with the cookie header? [14:16:39] I'm gonna put that task in kanban though [14:16:51] not sure [14:17:10] i didn't try with the cookie header [14:17:13] i saw your response thoough [14:17:17] hiiiiiiiiiiiiiiiii [14:17:28] morning :) [14:17:31] is it possible to make an "always pass" rule? [14:18:10] we can craft a specific Cookie header for this job, but it might not resilient to future changes [14:18:20] or we could review our caching policy for those files [14:18:23] one of the two [14:19:20] you mentioned the cache::misc setting, not sure what that is, but it sounds like it was intended for the html part and not the datasets, can we prevent it from applying to that sub-folder? [14:21:31] milimetric: cache::misc is a section of the Varnish infra dedicated to our internal websites (more or less) [14:21:39] so phab, gerrit, analytics, etc.. [14:22:15] well we can force a pass with the header [14:22:24] but I saw also very high (~1day) Cache headers [14:22:37] are they returned by analytics.w.o ? [14:22:50] not sure if those will need to be tuned for the dataset files [14:23:06] lemme check apache configs [14:23:13] but I'm confused about something [14:23:45] I thought those apache cache settings were talking to the browser, not affecting varnish [14:24:05] hello team :] [14:24:21] so I tried to curl the file that we discussed yesterday without the header cookie, and I get the last result [14:24:29] hey mforns [14:24:34] I believe that Cache-Control: max-age=86400, public, must-revalidate is applied to browsers and caches in the middle [14:24:37] like varnish [14:24:51] oh ok, that's unexpected in my mind [14:24:58] if we can get varnish to ignore that, that'd be great [14:25:45] milimetric: what if we tune Cache-Control? Do the UI need very up to date data? [14:25:50] probably not [14:25:59] I mean, how frequent those files are changing? [14:26:01] elukey: any idea what server analytics.wikimedia.org lives on? I'll document at https://wikitech.wikimedia.org/wiki/Analytics/analytics.wikimedia.org [14:26:15] should be thorium [14:26:21] oh right [14:26:37] well, the UI needs to be able to clear the cache, that's the problem [14:26:42] so the files change at most once a day [14:26:59] but if the user wants to get a fresh copy, they should be able to do so reliabl [14:27:13] also, if two users clear cache, they should get the same results [14:27:15] elukey: would it be possible to PURGE a subpath? [14:27:22] we could build that into the file sync stuff [14:27:41] this could be an option yes, more or less how mediawiki does with edits no?? [14:27:46] i suppose [14:27:47] ! [14:27:48] :) [14:27:53] but i dunno about a whole subpath [14:27:55] but we'd need to ask to the traffic masters first :) [14:27:58] probably tha'ts for specific URIs [14:28:08] yeah I think so too [14:28:18] elukey: wanna do ops sync real quick? [14:28:22] sure! [14:29:17] in bc [14:32:42] we could do ETags to get updated content [14:35:08] 10Analytics: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3197778 (10Milimetric) For reference, cache policy on anaytics.wikimedia.org is set up here: ``` # Cache json, yaml, csv, and tsv files 1 day # (could be all files but wanted to be more restrictive to star... [14:35:23] 06Analytics-Kanban: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3197779 (10Milimetric) a:03Milimetric [14:50:35] milimetric: analytics.wikimedia.org is backed up by varnish [14:51:13] yeah, we've been looking at some bizarre caching behavior [14:51:13] milimetric: on the new-1001 , a deploy could clear cache but that is driven server side not client side [14:51:28] milimetric: varnish will keep old deploy hot [14:51:36] milimetric: because that is what we tell him to do [14:52:20] milimetric: i do not think we use etags with varnish nor do we probably need them [14:52:56] milimetric: we can talk about this after standup [14:53:04] well, in this case we need something, ETags seem like the easiest solution [14:53:41] nuria: yeah, we can talk post-standup, you can read the task and see what we saw so far [14:55:29] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3197831 (10leila) @Ottomata Bob and I reviewed and we are happy with your choice. [14:56:46] milimetric: today curl is retrieving the up to date content without any cookie so I'd say that Cache-Control was honored [14:57:01] (revalidate after 1 day) [14:57:03] makes sense, elukey, the cache expired [14:59:17] 06Analytics-Kanban: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3194051 (10Nuria) @milimetric: Our http caching is done by varnish , apache just sets the expire headers, that has effect on the client, not the server. [15:01:06] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3197846 (10Ottomata) Great, thanks! [15:01:30] elukey: ya, let's talk about this on standup cause I thinkwe are mixing two things, the server and client caching policy [15:05:07] (03PS3) 10Mforns: Improve annotations [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) [15:07:11] fdans, standup? [15:07:18] sorryyyyy [15:19:03] ottomata: stat1002 is VERY loaded - some R scripts - Should we suggest using nice or should I just forget about it ? [15:20:13] what the hell https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=stat1002&refresh=1m&orgId=1&from=now-6h&to=now [15:20:28] all system load, what is that thing doing? [15:20:28] R :) [15:20:34] I have no idea [15:20:48] well, from script names, computing some forecast [15:21:01] should we suggest to use nice and niceio ? [15:21:04] ionice sorry [15:21:07] bearloga: hello :) [15:21:25] it seems that R is overloading stat1002.. [15:23:38] joal: i also see ezachte perl [15:23:55] but ya, would be good to nice bearloga :) [15:24:05] ottomata: the moment I looked, R was 1200% CPU and perl 100% :) [15:25:21] joal elukey ottomata: hiya! [15:25:32] Hi bearloga :) [15:25:57] Sorry for bothering :) [15:26:36] we have a codebase of scripts that runs daily using reportupdater and computes metrics & forecasts [15:26:49] bearloga: stat1002 is very loaded when you run your R scripts - Would you mind nicing / ionicing? [15:26:50] https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=stat1002&refresh=1m&orgId=1&from=now-6h&to=now :) [15:28:22] joal: I don't know what that is, but if it would help and if it's possible to apply to reportupdater, I am all for it! [15:28:56] searching "nice" on wikitech is a useless endeavor :( [15:28:59] bearloga: I have not enough knowloedge on reportupdater <-- mforns, any help? [15:29:47] bearloga: https://en.wikipedia.org/wiki/Nice_%28Unix%29 [15:30:11] joal: alternatively, if the problem is mostly caused by the forecasting, then we can disable that temporarily until a solution is found. it's not high priority. [15:30:15] bearloga: just launching with nice prefix would already help a lot [15:30:19] oh [15:30:25] joal: I'll try that! thanks! [15:30:32] bearloga, we could surely execute reportupdater under a nice/ionice [15:30:36] bearloga: please do both nice and ionice ;) [15:30:56] bearloga: for boith CPU and swap ;) [15:31:04] Thanks bearloga ! [15:31:10] joal, bearloga we could change the command in puppet no? [15:31:17] for all reportupdater jobs [15:31:34] mforns: why not, I have no strong opinion [15:31:57] elukey: deploy of refinery failed because of an1027 <-- Reminds me from previous deploy [15:32:23] mforns joal: we're running reportupdater on our own and have talked to nuria about what we'd need to add our reports to analytics' puppet-based runs [15:32:48] bearloga, are you running reportupdater through a puppet job? or do you have your own cron job in stat1002? or do you run it manually? [15:33:01] mforns: crontab on my account [15:33:26] bearloga, OK OK, then yes, you can apply the nice/ionice in your cron job :] [15:33:33] joal: just `nice ionice ` or would you suggest I add some parameters like that on wiki article? [15:35:48] bearloga, the only parameter is -n which is a number between -20 and 19 [15:36:03] 10Analytics, 13Patch-For-Review: Fix iOS vs. Android detection in mobile apps uniques tables - https://phabricator.wikimedia.org/T163403#3196212 (10JAllemandou) Thanks for the patch Tilman ! Just made a quick check (see below), and so far we are fine with the current settings (Android as other), but I agree, w... [15:36:22] the lower the number the more favorable for your process, the higher the less favorable (in terms of cpu consumption) [15:36:33] the default is 10, so I think it's a good default [15:36:38] (03CR) 10Joal: [V: 032 C: 032] "LGTM ! Merging before deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349140 (https://phabricator.wikimedia.org/T163403) (owner: 10HaeB) [15:36:41] mforns: thanks! :) will do! [15:36:47] cool :] [15:38:23] 06Analytics-Kanban, 13Patch-For-Review: Fix iOS vs. Android detection in mobile apps uniques tables - https://phabricator.wikimedia.org/T163403#3197995 (10JAllemandou) [15:44:56] (03PS4) 10Mforns: Improve annotations [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) [15:47:23] (03CR) 10Mforns: "THX for the review" (034 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) (owner: 10Mforns) [15:47:50] milimetric: just restarted piwik since the alter was good [15:47:51] WARNING: /usr/share/piwik/core/SettingsPiwik.php(267): Notice - Undefined index: multi_server_environment - Piwik 2.17.1 - Please report this message in the Piwik forums: http://forum.piwik.org [15:47:56] in the UI... [15:47:59] awesome elukey [15:48:07] ooh... [15:48:18] I'll take a look in a moment [15:48:25] thanks D [15:48:26] :D [15:49:18] ah I might know why is that [15:49:47] mforns joal elukey: thanks for being patient and for helping out! I just niced/ioniced the commands (https://gerrit.wikimedia.org/r/#/c/349232/1/main.sh) and also rescheduled the cron job to run at like 10PM SF time :) please let me know if that doesn't help or if it doesn't help enough! [15:49:49] 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Change userAgent field to user_agent_map in EventCapsule - https://phabricator.wikimedia.org/T153207#3198065 (10Nuria) Updated: https://meta.wikimedia.org/wiki/Schema:EventCapsule [15:49:53] !log deployed Refinery [15:49:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:50:09] bearloga, thanks a lot! [15:50:33] thanks bearloga ! [15:52:05] (03CR) 10Milimetric: [V: 032 C: 032] "Looks great, Marcel. The smaller box looks good on my system too. And the multi-line annotations do show up on the compare sample, didn'" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) (owner: 10Mforns) [15:53:09] milimetric: fixed :) [15:53:35] yay (was it just dropped during the alter?) [15:54:19] nono I was using the old global.ini file [15:54:28] didn't override it during the install [15:54:35] so it was complaining about old settings [15:54:36] ottomata: this is the doc you commented on where you suggested modifying the pageviews pipeline, btw, in case you didn't link up the two [15:54:43] (the one we're reviewing now) [15:54:55] gotcha, elukey, k, cool, I'll double check the data now [15:55:27] 10Analytics, 10Analytics-Cluster, 06Operations, 10hardware-requests: EQIAD: stat1002 replacement - https://phabricator.wikimedia.org/T159838#3198093 (10Halfak) +1 [15:56:25] elukey: data is identical, no problems [15:57:03] and piwik js is being pulled correctly, and new events are going across correctly [15:57:06] also seems faster :) [15:57:08] thx!! [15:57:19] nice! [15:59:56] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3198110 (10elukey) 05Open>03Resolved Total downtime of two hours (just ended now) due to long backup for the mysql database and schema changes required (an alter table that last more than one hour was the primary issue). [15:59:58] 06Analytics-Kanban: Piwik improvements - https://phabricator.wikimedia.org/T163000#3198112 (10elukey) [16:00:22] 06Analytics-Kanban: Piwik improvements - https://phabricator.wikimedia.org/T163000#3182952 (10elukey) [16:00:23] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3198115 (10elukey) 05Resolved>03Open [16:09:40] 06Analytics-Kanban: Piwik improvements - https://phabricator.wikimedia.org/T163000#3198168 (10elukey) [16:09:42] 10Analytics, 06Operations: sync bohrium and apt.wikimedia.org piwik versions - https://phabricator.wikimedia.org/T149993#3198165 (10elukey) 05Open>03Resolved a:03elukey Just upgraded Piwik on bohrium and uploaded the new deb (retrieved from https://debian.piwik.org) to jessie-wikimedia main (as it was do... [16:11:53] removed some dust from Piwik :) [16:26:56] nic! [16:41:12] nice nice. I'm doing lunch will be back after to deploy [16:53:23] elukey: thank you!!!!! [16:57:08] HaeB: Hi ! [16:59:42] !log Restart mobile_apps-uniques-[daily|monthly]-coord [16:59:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:04:48] joal: cool thanks! [17:05:09] HaeB: May I let you update docs / make annotations as needed ?n [17:06:23] !log Restart wikidata-specialentitydata_metrics-coord and wikidata-articleplaceholder_metrics-coord [17:06:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:08:53] joal: do we actually have public documentation for this on wikitech? [17:09:20] (i made this recently for the sessions table: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/mobile_apps_session_metrics but for uniques i didnt see anything either) [17:10:16] HaeB: Correct, looked at TOC, no uniques for mobile page yet [17:12:41] * elukey off! [17:12:42] o/ [17:12:56] elukey: Got an error trying to edit Wikitech using visual editor (Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500. Would you like to retry?) [17:13:04] elukey: could be related to switchover? [17:13:12] oh, sorry elukey - Leave, I'll ask ottomata ! [17:13:16] yeah I've heard some problems this morning about it :) [17:13:20] k [17:26:13] !log Restart druid pageview loading [daily-monthly] [17:26:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:32:55] !log Start daily uniques druid loading job (from 2015-12-17) [17:32:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:41:45] joal: ok food's finally here, how can I help? [17:42:33] I see you're on the mediawiki_history one with the alter table [17:45:49] semi-random question... I'm trying to find web-traffic stats (ours, or anyone large) for specific CPUs. Specifically: to determine how many Athlon and Pentium 3 (and older) chips are still being used. Do we have that info, or does anyone have a quick link handy for external details? (They're ambiguous keywords to google for... :/ ) [17:47:32] quiddity: not sure i understand [17:47:45] quiddity: as in how many pageviews a CPU can process? [17:48:16] nuria, just how many people are still using specific very-old hardware. [17:48:18] Hey milimetric [17:48:26] Indeed I'm on that last one [17:48:43] quiddity: people as in "readers of wikipedia?" [17:48:49] milimetric: I think easiest would be to delete/recreate table, and rerun MWH job [17:48:51] yes, essentially [17:49:07] makes sense joal [17:49:28] quiddity: so you mean browser agent stats? cause that is the only piece of info we would receive from a reader that would be related to hardware [17:49:41] milimetric: only concern: people won't be able to run queries for the next day [17:50:03] milimetric: If we send an email to analytics-internal list, will it be enough? [17:50:06] joal: oh ok, you wanna run the job into a temp path? [17:50:11] quiddity: makes sense? [17:50:20] milimetric: feasible [17:50:32] nuria, I don't know what details are contained in browser-agent stats, beyond browser and OS? But, yes, if that contains any hints about hardware, it might help. [17:50:48] milimetric: will almost kill the downtime [17:50:52] joal: but one day of down-time is probably ok, actually let's check with neilpquinn [17:51:04] neilpquinn: are you ok if we take down mediawiki_history table for a day? [17:51:04] let's do it the other way, cleaner [17:51:22] ok, good to ask anyway, curious [17:51:23] quiddity: no, it only contains OS, what i am saying is that teh only piece of information we have from readers that is hardware related is the user-agent [17:51:31] quiddity: https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os/os-family-timeseries [17:51:41] quiddity: user agents sent OS and that's it [17:51:45] milimetric: ideally not today! I have some analysis I need to do :) [17:51:55] ok, thanks neilpquinn, good we asked [17:51:58] nod. I found that. Does anyone in the world have/track info about hardware? [17:52:03] indeed milimetric :) [17:52:09] joal: then it'll be done by tomorrow, if it's later tomorrow I can take care of swapping the new data in [17:52:10] milimetric: I appreciate you asking :) [17:52:15] ok milimetric, let's do it the correct way ;) [17:53:05] joal: I can launch the job with a new output folder, [17:53:07] quiddity: not likely because you are interacting via web browser [17:53:26] nuria, ok. Thanks for clarifying. :) [17:53:52] quiddity: and a web browser is sending you data using http , the only dat ait can send you is the one javscript apis give the brower access to ( besides the http defaults) [17:54:55] quiddity: some specific (MS) browsers might have browser apis for hardware components but those are not widely deployed [17:55:31] milimetric: as you wish, I'm on it as well [17:55:50] quiddity: you need plugings and such very taylored to that use case to get such an info [17:56:35] joal: it's my ops week, I can change mw_directory in coordinator.properties and then submit to oozie, right? [17:56:53] milimetric: correct, don't forget stop time ;) [17:57:15] stop_time same as start_time plus one second, right? [17:57:24] milimetric: -Dstart_time=2017-04-01T00:00Z -Dstop_time=2017-04-02T00:00Z [17:57:29] or second, same [17:57:44] k, and sudo -u hdfs when I submit to oozie, no -Duser parameter right? [17:57:57] and this makes me realize that the semantic we use for snapshot is different than the one we use for other datasets [17:58:28] wait what do you mean? [17:58:29] milimetric: you can go for -Duser hdfs, no bother [17:58:59] milimetric: currently, we make snapshot=2017-04 for data up to 2017-04-01 [17:59:00] I was gonna do mw_directory = ${name_node}/wmf/data/wmf/mediawiki_new [17:59:31] milimetric: in your user folder please - no prod-like stuff :) [17:59:49] ok, better [17:59:49] milimetric: do it with -D :) [18:00:06] oh, I never did it with -D, ok [18:00:08] milimetric: -Dmw_directory=hdfs://analytics-hadoop/user/milimetric/mw_backfill [18:00:10] yep [18:00:22] ok, doing [18:00:26] easier, no need to upload anything, prod job is good [18:02:37] sudo -u hdfs oozie job -Duser=hdfs -Dstart_time=2017-04-01T00:00Z -Dstop_time=2017-04-02T00:00Z -Dmw_directory=hdfs://analytics-hadoop/user/milimetric/wmf/data/wmf/mediawiki -submit -config oozie/mediawiki/history/denormalize/coordinator.properties [18:03:12] milimetric: no -submit: -run instead [18:03:18] k [18:03:23] -submit just submits it, doesn't start it [18:03:35] otherwise good? [18:03:50] I assume you're in /srv/deployment/analytics/refinery ? [18:03:56] yes [18:04:07] Oh, queues :) [18:04:17] -Dqueue_name=production -Doozie_launcher_queue_name=production [18:04:39] sudo -u hdfs oozie job -Duser=hdfs -Dstart_time=2017-04-01T00:00Z -Dstop_time=2017-04-02T00:00Z -Dmw_directory=hdfs://analytics-hadoop/user/milimetric/wmf/data/wmf/mediawiki -Dqueue_name=production -Doozie_launcher_queue_name=production -run -config oozie/mediawiki/history/denormalize/coordinator.properties [18:06:24] milimetric: looks good ! [18:06:30] oh no [18:06:38] oozie_url [18:06:44] --oozie $OOZIE_URL [18:06:48] k [18:06:56] when sudoing for hdfs, you need to give it [18:07:14] except from that, YES ! [18:09:51] k, running: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0070007-170228165458841-oozie-oozi-C/ [18:09:57] thanks joseph [18:10:02] I'll check on it periodically [18:14:38] Thanks milimetric !@ [18:15:18] milimetric: more detailed monitoring of the core job: https://yarn.wikimedia.org/proxy/application_1492691387549_0759/ [18:16:07] a-team: I also realized another job is failing a lot (silently, but not for long) [18:16:19] a-team: Providing a patch for it now [18:16:24] oh [18:17:58] happy to deploy it if you want [18:18:05] (the patch when it's ready) [18:18:09] milimetric: it can wait next week [18:18:13] ok [18:19:59] milimetric: Th [18:20:11] milimetric: (again) Thanks for offering nonetheless :) [18:21:34] (03PS1) 10Joal: Update oozie restbase to graphite spark job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349266 [18:27:19] 06Analytics-Kanban: Update restbase oozie job - https://phabricator.wikimedia.org/T163479#3198989 (10JAllemandou) [18:44:20] 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Change userAgent field to user_agent_map in EventCapsule - https://phabricator.wikimedia.org/T153207#3199093 (10Tbayer) >>! In T153207#3198065, @Nuria wrote: > Updated: https://meta.wikimedia.org/wiki/Schema:EventCapsule Looks great, thanks! [18:51:40] 06Analytics-Kanban: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3194051 (10Milimetric) p:05Triage>03Normal [18:53:34] (03CR) 10Milimetric: [V: 032 C: 032] "Looking good" [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/340720 (https://phabricator.wikimedia.org/T156694) (owner: 10Matthias Mullie) [18:54:27] (03CR) 10Milimetric: "Data for this should compute tonight and be available here: https://analytics.wikimedia.org/datasets/periodic/reports/metrics/multimedia-h" [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/340720 (https://phabricator.wikimedia.org/T156694) (owner: 10Matthias Mullie) [19:15:35] milimetric: how do I know which snapshots are available for the edit data? I did `select distinct snapshot from wmf.mediawiki_history where snapshot > 0` but that took 30 seconds and gave me no results. [19:16:20] hm, should be there in the list of partitions, one sec [19:17:00] ahh, `show partions`, I forgot about that. [19:17:04] neilpquinn: show partitions mediawiki_history; [19:17:54] neilpquinn: if you forget whether a column's a partition or not, desc mediawiki_history [19:18:42] milimetric: right, that works :) and so for each monthly snapshot, it contains all data through the previous month? So 2017-04 contains all the data through March? [19:19:27] neilpquinn: yep, snapshot is when we imported the dat [19:20:12] milimetric: cool! In the long run, it might be a bit more intuitive to name the snapshot based on the most recent data it contains, e.g. 2017-04 contains all the data through 2017-04, but not a big deal :) [19:21:01] neilpquinn: one caveat though, the _private ones, like 2017-01_private are the only ones with data from all wikis. The non-private ones are imported from labs and subject to whatever dbs they have in labs. Last I knew it was enwiki and a bunch of other small ones [19:21:06] but no like commons or wikidata [19:21:55] that makes sense, neilpquinn, we should do that [19:22:30] milimetric: oh...that's weird. So if going forward you're only importing from labs, there won't be data for Commons or Wikidata? [19:23:11] 10Analytics: Label mediawiki_history snapshots for the last month they include - https://phabricator.wikimedia.org/T163483#3199213 (10Milimetric) [19:23:54] neilpquinn: no, we're hoping labs will have all data soon, but until then we're happy to do on-demand snapshots from prod [19:24:00] it's just a parameter and a manual job run [19:55:46] milimetric: ah, okay, that makes sense. I don't need that data now but I'll ask if I do! [19:56:08] joal, you gone? [20:15:39] 10Analytics, 10ChangeProp, 10EventBus, 06Services (done): Make EventBus service support wildcards in schema definitions - https://phabricator.wikimedia.org/T157091#3199531 (10Pchelolo) The patch was merged, to be deployed on Monday [20:31:13] I am ottomata :) [20:31:16] tomorrow ;) [20:31:21] laters, np! [20:57:47] (03PS9) 10Ottomata: [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [20:59:25] (03PS10) 10Ottomata: [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [21:04:40] HOooOOOLY CHEEEEEEET I think this thing is working! :O