[02:51:26] (PS2) MaxSem: WIP: count page with geo tags [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) [05:12:09] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2767346 (MZMcBride) >>! In T130651#2732665, @Ottomata wrote: > At this time, we are moving forward with SSE. We can always revisit possible webso... [06:43:54] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 11 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2767405 (KartikMistry) ``` npm test ``` for cxserver is failing for me. Debugging further. [08:24:42] a-team: Varnish 4 migration in text started in codfw (two hosts for the moment) [08:24:59] I am keeping an eye on eventlogging-client-side and webrequest-text [09:19:51] k elukey, thanks for that [09:19:59] elukey: Let me know if there's any help I can provide [09:20:06] elukey: o/ as well :) [09:22:14] o/ [09:22:31] we could check EL and Oozie as always [09:22:46] kafkacat output for both topics is good [09:42:26] Analytics, ChangeProp, Citoid, ContentTranslation-CXserver, and 11 others: Node 6 upgrade planning - https://phabricator.wikimedia.org/T149331#2767686 (mobrovac) >>! In T149331#2767405, @KartikMistry wrote: > ``` > npm test > ``` > > for cxserver is failing for me. Debugging further. Remember t... [10:21:17] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching): Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2718379 (mobrovac) >>! In T148251#2766484, @mmodell wrote: > This seems to have gotten more noisy with #wmf-deploy-2016-11-01_1.29.0-wmf.1 That's strange... [10:57:24] * elukey afk for a bit, need to get back my laptop's charger -.- [11:52:09] In the meantime I recovered the charger and worked, now I am going to lunch :) [11:58:27] hi a-team :] [12:09:14] Hey mforns :) [13:36:12] o/ [13:36:40] I have the vk testing Docker container almost ready https://gerrit.wikimedia.org/r/#/c/319548/2 [13:37:04] does anybody know more or less what kind of copyright/licence we usually put for these things? [13:38:43] elukey: no idea :( [13:39:06] a-team: Lino is sick, going to the creche to bring him to the doctor [13:39:12] moorrrning [13:39:16] :( hope he feels better [13:39:17] heya ottomata [13:39:18] joal, ok, bye! [13:39:21] hi ottomata [13:39:27] and milimetric :] [13:39:33] feel bettter little lino! [13:39:44] I'll be back on;ine when my wife comes back home [13:40:23] elukey: our other stuff has apache 2 [13:40:50] hey mforns [13:40:56] ottomata: thanks! Will check [13:41:29] ottomata: still not finished but if you want to double check what I've done and give me feedback more than happy :) [13:41:53] I still need to tweak a bit the PHP test.php file to be a bit more random [13:42:04] but more or less I have automated myself [13:42:08] :D [13:42:13] sure [13:42:19] show meeeee [13:42:27] hey a-team, is the cluster super slow for you as well? [13:43:04] mforns: https://yarn.wikimedia.org/cluster/scheduler - 99.4% used :( [13:43:16] elukey, I see, thanks! [13:43:33] oozie is having a rave party [13:43:47] hehehe [13:44:46] i think jo a'ls EditHistoryRunner is having a rave party :) [13:45:09] its using 63% [13:45:12] :) [13:46:48] yeah, it does that [13:47:01] gonna have to figure out how to nice that thing [13:47:38] hah, yeah, i mean, prod jobs won't have a problem getting around it [13:47:45] users can submit jbos in the priority queue [13:47:56] which is higher priority than default, but less than production or essential [13:48:05] but, yeah, maybe we need one less than default [14:02:53] ottomata, how can I submit a hive query to the priority queue? (didn't find it in the docs). Is it: SET mapred.job.priority=...; ? [14:03:22] that might work, but i think it would be easier to do on CLI [14:03:29] -Dmapred.job.queue=priority [14:03:32] something like that, looking [14:03:35] ottomata, ok ok [14:04:07] -Dmapred.job.queue.name=priority [14:04:36] ottomata, thanks! [14:18:58] Analytics-Kanban, EventBus, Wikimedia-Stream, Services (watching), User-mobrovac: Public Event Streams - https://phabricator.wikimedia.org/T130651#2768403 (Ottomata) From some internal discussions, it seems likely that irc.wikimedia.org will be remain as is. We may rework the backend, but th... [14:57:23] (PS1) Mforns: [WIP] Improve oozie data loss alarms [analytics/refinery] - https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) [14:57:53] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery] - https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) (owner: Mforns) [15:00:20] hm, batcave not working? [15:01:21] ottomata, mforns : staddupppp [15:01:25] trying [15:01:29] i think batcave is not working [15:01:40] OH [15:01:41] hm [15:01:44] in! [15:05:09] Analytics: Inconsistant data in #all-sites-by-os-and-browser fot IE7 - https://phabricator.wikimedia.org/T148461#2768528 (Nuria) [15:32:24] milimetric: taskingggg? [15:34:20] ottomata: taskinggg???? [15:34:35] coming [16:00:55] Analytics: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2768657 (Nuria) What is the overall objective? Replace outdated hardware Add better resiliency for static domains. 1. Order new box 2. Move everything 3. Move out everything that can be on a large vm so websites have better resilie... [16:02:45] Analytics: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2768678 (Nuria) Tasks to copy everything: - rsync - apply puppet - put box in varnish - announcement - switch [16:03:07] Analytics-Kanban: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2752595 (Nuria) [16:05:55] Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2768695 (Nuria) [16:08:03] Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2768696 (Nuria) [16:11:03] Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2768706 (Nuria) [16:20:58] Analytics-Kanban, Operations, hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2768740 (Ottomata) [16:21:27] Analytics-Kanban, Operations, hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2768755 (Ottomata) [16:22:25] Analytics: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2768758 (Nuria) We will compute standard metrics with canned data (scoop updates are not happening recurrently) Process will be: - run metrics - vet data - CR Rinse and repe... [16:23:09] Analytics-Kanban: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2768760 (Nuria) [16:30:26] Analytics-Kanban: Missing raw pageview data for 7/29/2015 - 03:00 - https://phabricator.wikimedia.org/T147801#2768784 (Nuria) [16:30:28] Analytics-Kanban: Missing raw pageview data for 7/29/2015 - 03:00 - https://phabricator.wikimedia.org/T147801#2768785 (mforns) a:mforns [16:49:24] mforns, milimetric : echo 'SELECT COUNT() FROM joal.edit_history' | curl 'druid1001.eqiad.wmnet:8123/' --data-binary @- [16:56:41] https://stats.wikimedia.org/EN/TablesWikipediaEN.htm [17:02:26] https://stats.wikimedia.org/EN/ChartsWikipediaEN.htm [17:04:15] mforns: how is the query going? [17:04:44] elukey, it failed :[, but right now I'm brainbouncing with joseph and dan [17:05:15] ah okok :) [17:09:20] mforns, milimetric : https://gist.github.com/jobar/fdb992936dba7bc91f85b1e463151ead [17:09:28] joal, thx! [17:11:36] milimetric: /user/joal/wmf/data/wmf/edit_history/denormalized [17:57:50] * elukey going afk! [18:12:23] (PS3) MaxSem: Count pages with geo tags [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) [18:15:52] (PS4) MaxSem: Count pages with geo tags [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) [18:18:49] (CR) Yurik: [C: 1] "haven't tested, but looks ok" [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: MaxSem) [18:39:24] (CR) Nuria: "Are you sure that wherever this is deployed you would have a php environment? Python is used broadly for munching data on analytics machin" [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: MaxSem) [18:40:07] Logging off a-team, have a good end of day [18:40:18] bye joal see ya [18:40:20] laters [18:43:01] joal: ciao [19:09:06] (CR) MaxSem: "Yes, it has - another script is already running :) It was inspired by WMDE's stats that are also PHP, and PHP is more popular among WMF en" [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: MaxSem) [19:09:36] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2769525 (mmodell) I'm going to be monitoring this closely when I deploy #wmf-deploy-2016-11-01_1.29.0-wmf.1 to group2, if the frequency incre... [19:11:13] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2769526 (Ottomata) Unless the bug itself starts happening more often, this logging change shouldn't increase the number of logs. It may incr... [19:21:52] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2769571 (mmodell) @Ottomata no I'm not worried about what you've been doing. I'm only worried about the increase in frequency that I noticed... [19:24:15] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2769576 (Ottomata) Ah, yeah. @Pchelolo mayyybe can comment more, but this error is connected to the LinksUpdateComplete hook, which I think... [19:26:42] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2769585 (Pchelolo) We've recently deployed the `page-properties-change` event that's fired from the JobQueue. That happened around October 5... [19:52:03] (PS1) Addshore: Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319659 (https://phabricator.wikimedia.org/T146967) [19:54:59] (PS2) Addshore: Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319659 (https://phabricator.wikimedia.org/T146967) [20:05:30] (CR) Nuria: ">PHP is more popular among WMF engineers anyway" [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) (owner: MaxSem) [20:20:19] ottomata: what is the best way to look at cluster utilization? [20:22:57] (PS1) Addshore: Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319667 (https://phabricator.wikimedia.org/T146967) [20:23:26] nuria: hm, yarn is not bad [20:23:31] https://yarn.wikimedia.org/cluster/scheduler [20:23:49] 98.6% used [20:23:50] currnetly [20:23:56] 65.8% used in default queue [20:24:03] then, if i sort the active containers by queue name [20:24:19] or by allocated containers even [20:24:22] you can see what is using the most [20:24:30] EditHistoryRunner [20:24:37] has 215 running containers [20:24:43] each using a core [20:24:51] and 1132G RAM :o [20:25:00] total [20:26:50] ottomata: doesn't it seem that that is a problem of our user space? as in a job run by one user is taking all resources [20:29:01] nuria: yes. production and essential queue jobs will still ahve room to run [20:29:12] but, that's what we were talking about earlier with the need for a lower priority 'nicer' queue [20:29:28] pretty sure other jobs in default can't get easily atm [20:29:34] ottomata: right, let's create a ticket for that no? [20:29:36] jobs in the same queue can't preempt [20:29:39] you can [20:29:40] for now [20:29:45] put your job in the priority queue [20:29:51] and you should be able to get in somewhere [20:30:05] ottomata, yes I'm using that right now and works [20:30:20] -Dmapred.job.queue.name=priority [20:30:53] nuria: yeah, a lower priori queue would be good, but as joseph was saying, it might cause trouble for this job, depending on how we do it [20:31:00] (CR) Hoo man: [C: 2] Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319659 (https://phabricator.wikimedia.org/T146967) (owner: Addshore) [20:31:10] (Merged) jenkins-bot: Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319659 (https://phabricator.wikimedia.org/T146967) (owner: Addshore) [20:31:13] (CR) Addshore: [C: 2] Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319667 (https://phabricator.wikimedia.org/T146967) (owner: Addshore) [20:31:14] we aren't really sure, he's a little worried that if the spark job gets preempted, or somethign takes to long, that it might restart the entire job, not just a single task [20:31:18] but, its worth a try [20:31:22] (Merged) jenkins-bot: Fix apiLogScanner default run day selection [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/319667 (https://phabricator.wikimedia.org/T146967) (owner: Addshore) [20:52:34] Analytics, Research-and-Data: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2769800 (yuvipanda) [20:52:52] Analytics, Research-and-Data: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2769814 (yuvipanda) For analysts, this means they can get newer R versions pretty quickly. [20:53:07] (PS2) Mforns: Improve oozie data loss alarms [analytics/refinery] - https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) [20:54:17] bye a-team! cya tomorrow [20:54:35] ottomata, hey, know about kafka-event-bus.services.eqiad.wmflabs? [20:54:48] It uses a class called role::analytics::kafka::server which you deleted last year [20:57:19] Krenair: in services project? def can be deleted [20:57:55] wanna do that? I don't have projectadmin there [20:58:08] deleted. [20:58:47] heading out, back in 2 hours for my swat thing [21:02:21] Analytics, Research-and-Data: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2769867 (mpopov) This would be great! I use ```lang=bash sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> /etc/apt/sources.list' sudo apt-key ad... [21:10:55] (PS5) MaxSem: Count pages with geo tags [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319260 (https://phabricator.wikimedia.org/T149722) [21:32:06] Analytics, Research-and-Data: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2770103 (yuvipanda) Yup, that's pretty much what we'll do here. [21:44:11] Analytics, Research-and-Data, Patch-For-Review: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2770167 (yuvipanda) Note that this wouldn't upgrade any of the currently in use R packages - we'll set a date for that and do that manually. [21:55:51] Analytics, Research-and-Data: Upgrade R on stat* machines to latest (3.3.2) - https://phabricator.wikimedia.org/T149959#2770252 (yuvipanda) [22:37:21] Analytics, Research-and-Data: Upgrade R on stat* machines to latest (3.3.2) - https://phabricator.wikimedia.org/T149959#2770420 (mpopov) We're planning to make the upgrade on Nov 15th. Yuvi will handle the upgrade itself and I will notify folks in the appropriate places, along with the R code to re-insta... [23:43:10] Analytics-Kanban, EventBus, Patch-For-Review, Services (watching), and 3 others: Empty body in EventBus request - https://phabricator.wikimedia.org/T148251#2770562 (Ottomata) Hm so ok. Stuff has been SWATed. The 400 errors are no longer happening, but I would expect to see some more error infor... [23:54:29] (PS2) MaxSem: Count pages with geo tags [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/319262 (https://phabricator.wikimedia.org/T149722)