[02:10:57] 10Analytics, 06Operations: Install java 8 to stat1002 - https://phabricator.wikimedia.org/T151896#2833945 (10fgiunchedi) p:05Triage>03Normal [02:15:03] 10Analytics, 06Operations: Install java 8 to stat1002 - https://phabricator.wikimedia.org/T151896#2833946 (10fgiunchedi) IIRC the alternatives should already prefer java-7 if both are installed, I'm not sure to which puppet class/role to add the package though (cc @Ottomata @elukey ) [04:54:53] (03PS1) 10Gergő Tisza: Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) [04:55:13] (03CR) 10jenkins-bot: [V: 04-1] Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) (owner: 10Gergő Tisza) [04:58:03] (03PS2) 10Gergő Tisza: Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) [04:58:06] (03PS1) 10Gergő Tisza: Fix CI errors [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324380 [04:58:23] (03CR) 10jenkins-bot: [V: 04-1] Fix CI errors [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324380 (owner: 10Gergő Tisza) [04:58:26] (03CR) 10jenkins-bot: [V: 04-1] Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) (owner: 10Gergő Tisza) [05:01:27] (03PS2) 10Gergő Tisza: Fix CI errors [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324380 [05:01:30] (03PS3) 10Gergő Tisza: Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) [05:34:14] 10Analytics, 06Commons, 06Multimedia, 10Tabular-Data, and 3 others: Allow structured datasets on a central repository (CSV, TSV, JSON, GeoJSON, XML, ...) - https://phabricator.wikimedia.org/T120452#2834173 (10Yurik) @JEumerus sorry, did not see your response. The storage repository will be on commons as we... [06:04:06] (03PS1) 10Hjiang: Changed names and revised further [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324382 [06:06:20] (03PS1) 10Hjiang: --amend [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 [06:35:34] 10Analytics, 06Research-and-Data, 13Patch-For-Review: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2834203 (10yuvipanda) 05Open>03declined unfortunately, it looks like there are obstacles to this that I don't have any more time for :(. Looks like debian... [06:43:20] 10Analytics, 06Research-and-Data, 13Patch-For-Review: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2834207 (10yuvipanda) [06:43:22] 10Analytics, 06Research-and-Data: Upgrade R on stat* machines to latest (3.3.2) - https://phabricator.wikimedia.org/T149959#2834205 (10yuvipanda) 05Open>03declined Because of https://phabricator.wikimedia.org/T149949#2834203, I think I'm personally going to give up on this for now :( I found a way to get a... [06:55:05] 10Analytics, 06Research-and-Data, 13Patch-For-Review: Use R from upstream for stat* and notebook* machines - https://phabricator.wikimedia.org/T149949#2769800 (10Joe) As discussed with yuvi on irc: - Using external repositories is frowned upon in general for good (security) reasons - At least on jessie let'... [08:09:03] 10Analytics, 06Editing-Analysis: Move contents of ee-dashboards to edit-analysis.wmflabs.org - https://phabricator.wikimedia.org/T135174#2834283 (10HJiang-WMF) Thanks for the tip! Will add in the next commit [10:37:23] 10Analytics, 06Operations, 10netops: Thorium (new stat1001) needs to communicate with the Analytics VLAN - https://phabricator.wikimedia.org/T151990#2834430 (10elukey) [10:39:00] opened a task for thorium --^ [10:39:32] now the question is if thorium needs to be in the analytics vlan or just get some ad hoc ACLs to communicate with it [13:02:30] hi a-team :] [13:03:08] heya [13:03:14] o/ [13:03:28] Hey mforns [13:03:40] hellooo guys [13:04:30] I might have reached the point in which I know that refactoring varnishkafka is doable but probably not worth it [13:05:14] especially if I want to work on other things :D [13:09:08] yeah, that project seems like a black hole [13:21:35] mforns, milimetric: Having triple checked on fake-pageId places has paid: job has passed the previous failing point (but still running, not yet vicoty) [13:21:41] victory sorry [13:22:26] well... we're a step closer! :) [13:24:00] milimetric: I am happy because I understand a lot of bits that were obscure to me before, and this will ease my work for other fixes.. but I believe that the integration testing suite might be enough for the moment [13:53:39] the suspense is palpable joal [13:58:14] xD [14:18:30] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2834971 (10Urbanecm) [14:30:57] milimetric: do you have a min? [14:31:22] I am trying to figure out what /srv/geowiki/data-private-bare/ is on stat1001 [14:31:28] that is rsynced from stat1003 [14:32:15] looking [14:32:25] oh right [14:32:34] so that's part of geowiki [14:32:47] it's a bit complicated to explain, but first, why? Is it giving you trouble? [14:32:50] let me explain why I am asking [14:32:54] exactly :D [14:33:13] so thorium (the new stat1001) is not in the analytics VLAN at the moment, but in the main production one [14:33:21] so for example pivot can't contact druid [14:33:22] etc.. [14:33:26] aha [14:33:28] there are two options [14:33:39] 1) add thorium to the analytics VLAN [14:33:59] 2) leave thorium in prod and allow only specific traffic to flow [14:34:17] (03PS2) 10Mforns: Add queries and config for ee migration [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 (owner: 10Hjiang) [14:34:18] like TCP connections to port 8082 of druid* from thorium, etc.. [14:34:24] right [14:34:39] the main point is - do we have any sort of private data on thorium/stat1001 that we'd need to protect? [14:34:43] well, there's a bunch of rsyncs from stat100[2,3] to stat1001 [14:34:59] ah yes these go through ssh that is already allowed [14:35:12] yeah, I believe stat1001 serves some private data under an http password [14:35:50] (geowiki being an example) [14:35:53] (03PS3) 10Mforns: Add queries and config for ee migration [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 (owner: 10Hjiang) [14:36:38] ah reading https://wikitech.wikimedia.org/wiki/Analytics/Geowiki [14:36:48] I should have done it as first step sorry :( [14:36:50] hm, I'm not finding how that's served [14:37:00] nono, it's cool, that project's a bit arcane [14:37:52] it seems served by a stats.w.o page [14:38:00] elukey: that makes sense then [14:38:06] passwords are in htpasswd.stats-geowiki [14:38:31] yep /srv/stats.wikimedia.org/htdocs/geowiki-private [14:38:48] that are used by httpd for basic auth [14:38:49] that's mentioned in the wikistats apache virtual directory setup as special and protected by that password [14:39:07] but that still doesn't answer why that folder you mentioned is there, maybe a symlink (checking) [14:39:26] it is rsynced from stat1003 [14:39:55] so if the data is critical for us it might make sense to restrict thorium to the an VLAN [14:39:57] no, right, but I'm trying to trace it to what is served publicly [14:40:03] so ok, that folder symlinks to /srv/geowiki/data-private/datafiles [14:40:09] ah sorry [14:40:15] but not data-private-bare [14:41:04] I don't have permission to see inside data-private, but data-private-bare looks like the .git directory of a repo [14:41:40] let's see what mr ottomata thinks [14:41:45] thanks! [14:41:45] so yeah, it's needed somehow in this process, ends up being served via wikistats with a password [14:41:50] it is used, I know that much [14:42:00] folks like Asaf login once in a while [14:42:03] (03CR) 10Mforns: "It looks really good now!" (038 comments) [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 (owner: 10Hjiang) [14:42:07] and it's important for their work [14:42:43] oh yes no doubt, just wondering what level of protection we need [14:43:55] (03CR) 10Mforns: [C: 032 V: 032] "LGTM, let's merge." [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 (owner: 10Hjiang) [14:45:30] (03Abandoned) 10Mforns: Changed names and revised further [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324382 (owner: 10Hjiang) [14:45:37] (03Abandoned) 10Mforns: patched up some queries, changed names, branched out a new folder, and added the daily edits by anon [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 (owner: 10Hjiang) [14:45:46] (03Abandoned) 10Mforns: added several new modified sql queries, completed wiki_dbs, and made the config_all [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/323861 (owner: 10Hjiang) [14:45:52] (03Abandoned) 10Mforns: Add template for migration to reportupdater+dashiki [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/320433 (https://phabricator.wikimedia.org/T126358) (owner: 10Mforns) [14:47:00] (03Restored) 10Mforns: Add template for migration to reportupdater+dashiki [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/320433 (https://phabricator.wikimedia.org/T126358) (owner: 10Mforns) [14:47:05] (03Restored) 10Mforns: added several new modified sql queries, completed wiki_dbs, and made the config_all [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/323861 (owner: 10Hjiang) [14:47:09] (03Restored) 10Mforns: patched up some queries, changed names, branched out a new folder, and added the daily edits by anon [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 (owner: 10Hjiang) [14:47:14] (03Restored) 10Mforns: Changed names and revised further [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324382 (owner: 10Hjiang) [14:47:18] :P [14:52:14] I was wondering why you were abandoning those [14:52:18] :) [14:53:58] (03PS1) 10Milimetric: Remove glam_nara legacy TSV generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324458 (https://phabricator.wikimedia.org/T92340) [14:54:56] * milimetric would love to get a generic real-time traffic splitter / simple stream processor in place [14:55:20] milimetric: add it to the etherpad! [14:55:23] http://etherpad.wikimedia.org/p/stream-processing [14:55:23] that would make all these ongoing TSV jobs pointless [14:55:39] oh, milimetric that is kinda what kafkatee is, no? [14:55:47] (03PS1) 10Mforns: Add queries and config for ee migration [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324461 (https://phabricator.wikimedia.org/T126358) [14:55:58] (and what udp2log is) [14:55:59] :p [14:56:05] https://github.com/wikimedia/analytics-kafkatee [14:56:45] yeah, but we don't have the stream processing in place to deal with the tee-d off data [14:56:56] (03CR) 10Mforns: [C: 032 V: 032] "This patch has been reviewed and tested in another (abandoned) patch." [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324461 (https://phabricator.wikimedia.org/T126358) (owner: 10Mforns) [14:56:57] so we're doing this batch TSV thing instead [14:57:39] and these things are querying wmf_raw!! so it's even worse: https://github.com/wikimedia/analytics-refinery/blob/78d1e5870d397632a6a3832ed4994a8c9593ab6c/oozie/webrequest/legacy_tsvs/generate_glam_nara_tsv.hql [14:57:46] (03Abandoned) 10Mforns: Changed names and revised further [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324382 (owner: 10Hjiang) [14:57:51] (03Abandoned) 10Mforns: patched up some queries, changed names, branched out a new folder, and added the daily edits by anon [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324038 (owner: 10Hjiang) [14:57:57] (03Abandoned) 10Mforns: added several new modified sql queries, completed wiki_dbs, and made the config_all [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/323861 (owner: 10Hjiang) [14:58:03] (03Abandoned) 10Mforns: Add template for migration to reportupdater+dashiki [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/320433 (https://phabricator.wikimedia.org/T126358) (owner: 10Mforns) [14:58:10] (03Abandoned) 10Mforns: Add queries and config for ee migration [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324383 (owner: 10Hjiang) [14:58:26] sorry for the mess :/ [14:58:49] we could've used kafkatee except these things need to be bucketed, etc. so it makes more sense they get into hdfs first I guess [15:00:54] 10Analytics, 06Operations, 10netops: Thorium (new stat1001) needs to communicate with the Analytics VLAN - https://phabricator.wikimedia.org/T151990#2835067 (10Ottomata) Uh OH! This is SUPPOSED to be in the analytics VLAN! https://phabricator.wikimedia.org/T149911 Re-opening that ticket. Sorry, I shoulda... [15:01:26] 06Analytics-Kanban, 13Patch-For-Review: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2835070 (10Ottomata) [15:01:28] 06Analytics-Kanban, 06Operations, 10hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2835068 (10Ottomata) 05Resolved>03Open Uh OH! @RobH > This should be installed within the Analytics VLAN, but it does not matter which row. I think thorium may have had... [15:01:59] 06Analytics-Kanban, 06Operations, 10hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2835072 (10Ottomata) [15:02:01] 10Analytics, 06Operations, 10netops: Thorium (new stat1001) needs to communicate with the Analytics VLAN - https://phabricator.wikimedia.org/T151990#2835074 (10Ottomata) [15:03:26] ottomata: o/ [15:04:01] we can fix the vlan assignment quickly [15:04:11] the major question is if thorium needs to be there [15:04:12] or not [15:04:21] but I guess the answer is yes? :D [15:06:41] elukey: i would say yes, but i could be convinced no if folks think so strongly [15:09:44] ottomata: let's discuss it during the ops sync [15:09:51] but I am not stronly opinionated [15:10:01] just trying to get the use case since we are doing the switch :) [15:10:12] (potential cleaning task :P) [15:10:15] k [15:51:04] 10Analytics, 06Commons, 06Multimedia, 10Tabular-Data, and 3 others: Allow structured datasets on a central repository (CSV, TSV, JSON, GeoJSON, XML, ...) - https://phabricator.wikimedia.org/T120452#2835140 (10JEumerus) Ah, OK. [15:52:48] (03PS6) 10Milimetric: reportupdater queries for EventLogging [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/322007 (https://phabricator.wikimedia.org/T147034) (owner: 10MaxSem) [15:53:53] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Improve tests for KafkaSSE - https://phabricator.wikimedia.org/T150436#2835157 (10mforns) a:03mforns [16:00:01] mforns, milimetric : standdduppp [16:22:06] 06Analytics-Kanban, 06Operations, 10hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2835257 (10RobH) Correct, it was installed in the internal vlan, my bad! It'll need reinstallation, as well as the dns and network port being updated. [16:31:20] (03CR) 10Nuria: [C: 032] Remove glam_nara legacy TSV generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324458 (https://phabricator.wikimedia.org/T92340) (owner: 10Milimetric) [16:31:46] nuria: if you want we can merge and deploy that too now [16:31:48] it's approved [16:34:18] 06Analytics-Kanban, 06Operations, 10hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2835285 (10Ottomata) ​Ok, it can be reinstalled at will. The puppet that is in place is fine (it might fail on the first run). Let me know when it is back up and I will make s... [16:35:21] mforns: should I merge the 2 reportuptater puppet patches? [16:35:26] https://gerrit.wikimedia.org/r/#/c/324466/ [16:35:26] https://gerrit.wikimedia.org/r/#/c/322969/ [16:36:36] ottomata, https://gerrit.wikimedia.org/r/#/c/324466/ can be merged [16:37:06] ottomata, for the other one we still need to finish the change in reportupdater [16:37:45] milimetric, could you double check with dfoy just in case about stat1002 data? [16:37:53] aye ok [16:47:25] yurik: I cc-ed him yesterday when you said that [16:47:33] he hasn't responded [16:47:36] milimetric, ah, cool, thx :) [16:47:48] (sorry if I forgot to confirm that I did that) [16:48:40] no worries :) i wasn't sure you saw my response [16:51:26] (03PS1) 10Milimetric: Changelog v0.0.38 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/324492 [16:51:37] (03CR) 10Milimetric: [C: 032 V: 032] Changelog v0.0.38 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/324492 (owner: 10Milimetric) [16:56:27] 10Analytics, 10Analytics-Cluster: Kafka Security Features - https://phabricator.wikimedia.org/T152015#2835373 (10Ottomata) [16:56:44] 10Analytics, 10Analytics-Cluster, 06Operations, 10Traffic: Enable Kafka native TLS in 0.9 and secure the kafka traffic with it - https://phabricator.wikimedia.org/T121561#2835376 (10Ottomata) [16:56:46] 10Analytics, 10Analytics-Cluster: Kafka Security Features - https://phabricator.wikimedia.org/T152015#2835361 (10Ottomata) [17:02:42] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2835399 (10greg) >>! In T151570#2825908, @Krenair wrote: > These patches look good to me, let's schedule deployment after the trai... [17:05:05] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2835410 (10Krenair) It's not too late to do it at the suggested time, is it? [17:28:52] btw, to those waiting on the deploy, I got confused and now I'm going into meetings, will ping someone after [17:43:24] milimetric: k on meeting for 30 mins but can talk after [17:44:42] 10Analytics, 10Analytics-Cluster: Kafka Security Features - https://phabricator.wikimedia.org/T152015#2835611 (10Ottomata) Talked with Luca today, and with Analytics team about goals planning. We have a timeline for this. We need to replace the existing Kafka brokers, since they are out of warranty. We also... [18:01:17] nuria: I'm free for 30 minutes now [18:11:40] (03CR) 10Bartosz Dziewoński: [C: 031] Fix SQL queries [analytics/multimedia] - 10https://gerrit.wikimedia.org/r/324379 (https://phabricator.wikimedia.org/T98449) (owner: 10Gergő Tisza) [18:13:41] milimetric: on meeting can talk in abit [18:18:53] 10Analytics, 10Dumps-Generation, 05Security: Omit private data from being generated during dump runs - https://phabricator.wikimedia.org/T152021#2835735 (10ArielGlenn) [18:26:32] I'll be in scrum of scrums soon, we can talk after maybe? [18:29:46] milimetric: k [18:29:54] PROBLEM - YARN NodeManager Node-State on analytics1039 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:19] hey! [18:30:44] RECOVERY - YARN NodeManager Node-State on analytics1039 is OK: OK: YARN NodeManager analytics1039.eqiad.wmnet:8041 Node-State: RUNNING [18:35:48] ottomata: ani dea what was the hickup? [18:35:56] ottomata: overloaded by my edit job? [18:36:46] possibly, it came back so soon that I didn't bother looking :) [18:36:53] right :) [18:36:56] want me to look? [18:37:05] we might have checks a bit too strict [18:37:10] no no, just wondered if it was back after an action [18:37:16] * elukey afk! [18:38:27] joal: [18:38:33] 2016-11-30 18:33:46,494 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=19672,containerID=container_e37_1480065021448_13387_01_026148] is running beyond physical memory limits. Current usage: 9.0 GB of 9 GB physical memory used; 9.7 GB of 18.9 GB virtual memory used. Killing container. [18:38:49] but that's just the container... [18:38:57] ottomata: I know, some of my tasks died [18:39:33] ottomata: however there are bizarre things, since relaunching the containers makes the tasks succeed [18:39:47] ottomata: Possibly some cleanup or cahcing info not yet cleaned ... no sure [18:40:25] hm, actually the nodemanager proc didn't die [18:40:30] its been running since nov 21 [18:40:41] weird [19:01:35] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2835877 (10greg) If ya'll are ready, it's fine with me. [19:20:38] (03PS10) 10Milimetric: Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [19:23:46] (03CR) 10jenkins-bot: [V: 04-1] Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [19:29:58] 10Analytics, 06Operations: Install java 8 to stat1002 - https://phabricator.wikimedia.org/T151896#2835991 (10Ottomata) Hm hm. openjdk-7-jdk is required in a few classes that get included on stat1002. As long as the default alternative isn't updated as consequence of doing a `require_package('openjdk-8-jdk'0)... [19:34:46] (03CR) 10Milimetric: "I don't know what's up with jenkins but that test passes locally and it makes no sense that it would fail anywhere. So I'm merging this. " [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [19:34:51] (03CR) 10Milimetric: [C: 032 V: 032] Optionally record data to Graphite [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/322365 (https://phabricator.wikimedia.org/T150187) (owner: 10MaxSem) [19:37:24] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (watching): RecentChanges in Kafka - https://phabricator.wikimedia.org/T152030#2836021 (10Ottomata) [19:38:05] mforns: Notice: /Stage[main]/Role::Statistics::Cruncher/Reportupdater::Job[ee-migration]/Cron[reportupdater_limn-ee-data-ee-migration]/ensure: created [19:39:15] ottomata, thanks! [19:47:26] 06Analytics-Kanban: Wikistats 2.0. Edit Reports: Setting up a pipeline to source Historical Edit Data - https://phabricator.wikimedia.org/T130256#2836080 (10Nuria) [19:48:12] 10Analytics, 10Analytics-Wikistats: Visual Language for http://stats.wikimedia.org replacement - https://phabricator.wikimedia.org/T152033#2836084 (10Nuria) [19:55:24] nuria: ok, done with reportupdater for now, deploying again in a bit, are you going to lunch though/ [19:55:35] milimetric: will be here [19:57:58] 10Analytics: Calculate edit metrics from history reconmstruction in hadoop intermediate table - https://phabricator.wikimedia.org/T146489#2836109 (10Nuria) Closing this task as outdated. Metrics werre done here: https://phabricator.wikimedia.org/T150025 and here: https://phabricator.wikimedia.org/T150024 [19:58:06] 06Analytics-Kanban: Replacing standard edit metrics in dashiki with data from new edit data depot - https://phabricator.wikimedia.org/T143924#2836111 (10Nuria) [19:58:08] 10Analytics: Calculate edit metrics from history reconmstruction in hadoop intermediate table - https://phabricator.wikimedia.org/T146489#2836110 (10Nuria) 05Open>03Resolved [19:58:51] 06Analytics-Kanban: Create 1-off tsv files that dashiki would source with standard metrics from datalake - https://phabricator.wikimedia.org/T152034#2836112 (10Nuria) [19:59:13] ok, nuria, batcave or here? [19:59:30] milimetric: here [19:59:34] better [20:00:16] ok, so my main confusion so far is this [20:00:43] If pre-filled values are not correct, the simplest solution is to trigger a build [20:00:54] I'm not sure how to tell whether they're correct or not, not sure what they should be [20:01:38] for example, it should be v0.0.38, but it's v0.0.37 [20:01:57] it sounded like triggering a build would fix that, but it didn't, so did something go wrong or do I simply have to increment the version? [20:02:02] * milimetric is *very* literal [20:02:43] 06Analytics-Kanban: Productionize Edit History Reconstruction and Extraction - https://phabricator.wikimedia.org/T152035#2836132 (10Nuria) [20:02:49] nuria: ^ [20:03:42] 06Analytics-Kanban: Scale MySQL edit history reconstruction data extraction - https://phabricator.wikimedia.org/T134791#2836152 (10Nuria) [20:03:44] 06Analytics-Kanban, 13Patch-For-Review: Productionize edit history extraction for all wikis using Sqoop - https://phabricator.wikimedia.org/T141476#2836151 (10Nuria) [20:04:32] 06Analytics-Kanban: Productionize Edit History Reconstruction and Extraction - https://phabricator.wikimedia.org/T152035#2836132 (10Nuria) Productionize edit history extraction for all wikis using Sqoop. Productionize Scala code to do reconstruction [20:04:32] Hey milimetric [20:04:55] milimetric: let me see, triggering a build fixed it before [20:05:17] joal: go to sleep :) [20:05:21] :) [20:05:38] Need to fix some scala before getting ot bed, I might as well help on deploy ;) [20:05:59] but nothing's wrong, just me being very careful with something new [20:06:17] nuria: so I triggered the build and got this: https://integration.wikimedia.org/ci/job/analytics-refinery-release/43/ [20:06:37] after merging the new changelog: https://github.com/wikimedia/analytics-refinery-source/blob/master/changelog.md#v0038 [20:09:32] milimetric: were you logged in to jenkins when you triggered build? [20:09:40] yes [20:10:15] I didn't trigger a release, just a build [20:10:23] but yes, I was logged in [20:11:58] milimetric: let me trigger another build [20:12:19] nuria: what is it supposed to do, update the tags to 0.0.38? [20:12:38] how does it know it's not 0.1.0? [20:12:41] milimetric: i think so.. [20:13:46] milimetric: let's just change it here and see what happens: https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ [20:13:59] milimetric: to 0.38, this is just building jars [20:14:07] ok, I'll change them all to 38, but the -SNAPSHOT one should be 39?... [20:14:17] milimetric: nothing is going to break [20:14:30] I know it's to 0.0.38, I'm saying the automated system would have no way of knowing that, so it seems logical that we'd have to change it [20:14:30] milimetric: i do not think so, all .38 [20:14:33] but it's not in the instructions [20:14:39] ok [20:15:02] milimetric: nah, last time it was populated correctly i think, that is what the custom plugging code is for [20:15:25] but how would it know whether you wanted a major or minor release? [20:15:32] from the changelog.md? [20:15:35] milimetric: only special place to care is the SCM tag checkbox [20:15:35] milimetric: ah yes, no [20:15:55] milimetric: sorry, right. it doesn't know that [20:16:13] ok, I updated it manually and launched the release. Updating docs [20:16:43] milimetric: SCM checkbox is documented, right? [20:17:04] 06Analytics-Kanban, 13Patch-For-Review: Productionize edit history extraction for all wikis using Sqoop - https://phabricator.wikimedia.org/T141476#2836212 (10Nuria) [20:17:06] 06Analytics-Kanban: Productionize Edit History Reconstruction and Extraction - https://phabricator.wikimedia.org/T152035#2836211 (10Nuria) [20:17:11] yes, checkbox is documented [20:17:51] Cool :) [20:17:56] wasn't sure [20:19:47] milimetric: ok, let's keep going down the list and see if things work [20:20:01] I will, the build is still going, I'm updating docs [20:22:01] ottomata: ok, prettier README just waiting for Pchelolo to test https://github.com/wikimedia/node-rdkafka-statsd [20:27:38] ok, refinery jars updating https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/44/ [20:27:55] nuria: the refinery deployment that you mention, does that have updated references to the new 0.0.38 version? [20:28:32] milimetric: the oozie jobs? [20:28:51] milimetric: no that change needs to be done in oozie config [20:28:59] the properties files it looks like are then ones that hold the version [20:29:15] oozie config is checked into refinery though, right? [20:29:42] so which files need to change? [20:29:54] milimetric: these: [20:31:15] milimetric: ./oozie/webrequest/load/bundle.properties:refinery_jar_version [20:31:53] ok, so just the load [20:31:58] k, I'll change that and commit [20:32:19] 10Analytics, 06Reading-analysis, 06Research-and-Data, 10Research-consulting: Propose metrics along with qualifiers for the press kit - https://phabricator.wikimedia.org/T144639#2836277 (10leila) [20:32:40] nuria: you wanna deploy this while we're at it? https://gerrit.wikimedia.org/r/#/c/324458/ [20:32:57] (03CR) 10Nuria: [V: 032] Remove glam_nara legacy TSV generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324458 (https://phabricator.wikimedia.org/T92340) (owner: 10Milimetric) [20:32:59] or wait until next week to disable more of them? [20:33:04] cool nuria :) [20:33:20] milimetric: sure. [20:33:36] 10Analytics, 06Operations: Install java 8 to stat1002 - https://phabricator.wikimedia.org/T151896#2836283 (10EBernhardson) I am specifically doing some machine learning exploration using RankLib, a java implementation of various ML ranking algorithms (that could perhaps be integrated to an elasticsearch plugin... [20:34:54] (03PS1) 10Milimetric: Bump up refine job to version 38 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324553 [20:35:10] (03CR) 10Milimetric: [C: 032 V: 032] Bump up refine job to version 38 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324553 (owner: 10Milimetric) [20:35:54] ottomata: http question you might know [20:35:58] brb [20:36:07] ottomata: the "strict-transport-security" header [20:36:18] ottomata: has a "max-age" caching [20:37:59] ottomata: but we have no "cache-control" header [20:39:38] ottomata: nevermind, that is max-age for https connections no cache resources, ok that makes sense [20:42:05] 10Analytics-Dashiki, 06Analytics-Kanban: Improve initial load performance for dashiki dashboards - https://phabricator.wikimedia.org/T142395#2836294 (10Nuria) Looks to me resources served from analytics.wikimedia.org are fresh everytime. We do not seem to have cache headers: https://analytics.wikimedia.org/das... [20:44:38] bearloga: yt? [20:45:00] nuria: sorry, just saw your q [20:45:12] uhmmm, ha, i don't know anything about those headers! :) [20:45:28] 10Analytics-Dashiki, 06Analytics-Kanban: Improve initial load performance for dashiki dashboards - https://phabricator.wikimedia.org/T142395#2836302 (10Nuria) The requests to config files could have a 1hr ttl unless we are running in developer mode. Thus far we are requestingting them with TTL:0 An example of... [20:45:42] ottomata: ya, i know who might know [20:45:43] ottomata: here! wassup? [20:46:32] bearloga: i'm giving a pass at building your R packages on stat1002 [20:46:36] i think i got as far as you did [20:47:01] trying to understand why on stat1002 [20:47:01] 1. we need a ~/.R/Makevars at all (whne on stat1003 we don't) [20:47:02] and [20:47:41] 2. why the final shlib build loses the g++ command [20:47:59] i dunno about 1., but i betcha 2 is because there is some other make var that we need to set that whever the shlib command gets built, it uses [20:48:16] seeings as building the dependencies didn't work without setting some makevars [20:48:24] do you have any idea where the actual makefile for boom is? [20:48:25] ottomata: oh that was just to correct for a problem that should not have been happening. there really shouldn't be a need for ~/.R/Makevars at all :) [20:48:29] i don't see it in the tar.gz it dls [20:48:33] bearloga: yeah [20:48:43] i don't thikn the r-cran.mk is relevant [20:49:35] ottomata: Dirk (the author of that r-cran.mk) said it shouldn't be relevant in this case [20:49:39] hmm /usr/share/R/share/make/ [20:49:39] ? [20:50:07] ottomata: let me see if I can find the makefile for ya :) [20:52:02] !log Deployed refinery-source using jenkins [20:52:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:52:07] deploying refinery now [20:54:08] ottomata: "Generally, R packages should avoid a custom Makefile. Instead, use Makevars. Makevars is a make file that overrides the default make file generated by R (which is located at file.path(R.home('etc'), 'Makeconf'))" http://r-pkgs.had.co.nz/src.html#make [20:54:35] bearloga: yeah, it looks lke the actual makefiles are in /usr/share/R/share/make/ [20:54:40] ottomata: if you look in src/ in the tar.gz, you'll find the Makevars for Boom [20:54:47] i'm looking at /usr/share/R/share/make/shlib.mk now [20:55:28] wildly attempting to set SHLIB_LINK in ~/.R/Makevars to the same g++ command and flags that stat1003 used at that point [20:55:41] too bad it takes so long to compile this! [20:55:47] we only know if it fails at the end [20:56:51] hmmm ottomata the scap deploy finished successfully for refinery but /srv/deployment/analytics/refinery on stat1002 isn't updated [20:57:18] cc nuria ^ [20:58:13] milimetric: so.. for what hosts did it suceed? [20:58:22] milimetric: scap lists the hosts [20:59:29] it succeeded for all hosts [20:59:31] it's just not updated [20:59:44] like, git log on tin shows something other than git log on stat1002 [21:00:37] I guess... I'll deploy again? [21:00:40] * milimetric hates this [21:00:41] in 1002 i see: [21:00:43] milimetric: [21:00:47] https://www.irccloud.com/pastebin/5jr40Qwj/ [21:00:56] so your code updated fine [21:01:00] uh... that's not what I see [21:01:11] on git? [21:01:16] https://www.irccloud.com/pastebin/EvNvuk0E/ [21:01:17] milimetric: on git? [21:01:49] wtf... [21:01:50] milimetric: i would try again [21:01:56] oh I tried like 30 times [21:01:57] milimetric: it might have just flipped [21:02:03] https://www.irccloud.com/pastebin/PvT3NACM/ [21:02:11] yeah, something's very wrong [21:02:17] looking too [21:02:37] nuria: what's git status say for you? [21:02:43] I get head detached at 635a3ef [21:03:00] ottomata: good luck! :) sorry that this issue is being such a beavis & butthead [21:03:00] milimetric: [21:03:07] https://www.irccloud.com/pastebin/dPMzfmUl/ [21:03:07] i can't git status because of some permission error ??? [21:03:10] but ifi sudo [21:03:11] milimetric: i get [21:03:12] HEAD detached at 9cd8845 [21:03:27] k, well, batcave? [21:03:34] 'cause I'm ... lost [21:03:35] k [21:05:09] hahah, I AM AMAZING i fixed it all [21:05:10] bwhahah [21:05:15] (dan's problem, not the R one) [21:05:23] so clever [21:05:36] nuria: I was in the old directory, the symlink changed while I was inside it [21:05:42] updating docs :) [21:05:48] milimetric: ya, makes sense [21:09:02] milimetric: for me the step that actually like the least is having to manually update oozie job versions, both andrew and joseph think it should be that way though [21:09:27] oh I agree with that [21:09:33] that it should be done manually [21:09:44] it gives you control over what logic runs where [21:10:02] we could make tools that automatically update everything, but it's analogous to the dashiki deploys [21:10:17] we don't have an "udpate all" there, and that's a good thing in the normal case [21:12:35] milimetric: ok, last thing now would be to restart oozie jobs [21:14:07] nuria: the hdfs deploy is still going on [21:14:12] I'll log when it's done [21:14:33] milimetric: k, moved docs under deploy [21:14:40] so we can find them more easily [21:15:23] weird... my cat just ate ginger dressing. cats should hate that... [21:16:02] milimetric: jajaja [21:16:03] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Deploy/Refinery [21:16:46] oh, I found them ok before, this makes sense too I guess [21:16:49] all good [21:17:16] ok, all done [21:17:16] !log Deployed refinery using scap, then deployed onto hdfs [21:17:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:17:36] nuria: ok, I'll restart legacy_tsvs and load [21:20:15] hm... maybe just the load, since we'll restart legacy next week anyway [21:20:15] this is involved... [21:21:51] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (watching): RecentChanges in Kafka - https://phabricator.wikimedia.org/T152030#2836021 (10Krenair) It'll probably involve creating a class that implements the RCFeedEngine interface from MW core, registering that in wgRCEngines, then adding to our con... [21:23:57] milimetric: i just added an example [21:25:10] milimetric: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie/Administration#Example:_restart_of_refinery_load_pipeline [21:25:44] bearloga: meh [21:25:58] i got a tiny bit further, it seems to have run the exact command now that it did on stat1003 [21:26:22] thanks nuria, I just killed the load bundle because it finished all of 2016-11-30T19:00Z [21:26:26] but it fails and I get a different error on stat1002 [21:26:30] and I'm going to restart it now with 2016-11-30T20:00Z right? [21:26:44] /usr/bin/ld: Models/Bart/Bart.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC [21:26:44] Models/Bart/Bart.o: error adding symbols: Bad value [21:26:53] milimetric: i think so, this is the part i always mess up [21:26:55] nuria: based on the info I got from the bundle's coordinators here: https://hue.wikimedia.org/oozie/list_oozie_bundle/0000454-161121120201437-oozie-oozi-B [21:27:34] if I'm reading it correctly, it's saying to look at the oldest successful coordinator and add 1 increment for the new start_time [21:27:38] milimetric: that is what makes sense to me [21:27:39] all of them are at 19:00 [21:27:44] ok [21:27:56] I'll do that and I can get yelled at tomorrow if I did it wrong :) [21:29:42] milimetric: if this is not right we should just document it cause taht is what makes sense [21:29:57] nuria: your example has this in it, and mine that I copied from the instructions doesn't: "-Duser=$USER" [21:30:54] milimetric: ah ya, i am not sure whether it matters as the queue is production cc ottomata [21:32:10] ok, nuria maybe we can keep the example I added to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie/Administration#When_.properties_file_has_changed [21:32:42] milimetric: ya, totally edit mine away [21:32:45] !log restarted webrequest/load oozie bundle [21:32:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:33:53] ok, then release done [21:34:06] milimetric: let's talk tomorrow about this but I woudl like to add a TTL of 3600 to mediawiki storage [21:34:10] https://github.com/wikimedia/analytics-mediawiki-storage/tree/master/src [21:34:33] nuria: sure, but with the new dashiki extension those will be proper json files and we can fetch them as such [21:34:40] so we might have a bit of a different approach [21:34:48] right now we're kind of hacking json into the API [21:35:02] the -Duser=$USER doesn't really matter, the default is for user to be the user submitting the job [21:35:08] k, that works [21:35:11] so if you do sudo -u hdfs ... then -Duser=$USER doesn't matter [21:35:19] makes sense, thx [21:35:51] milimetric: but either way we need ttls if we fetch them from api [21:36:17] nuria: right, but we might not need mediawiki-storage [21:36:24] milimetric: or you mean fetching them from meta [21:36:28] yea [21:36:42] I'm not sure - have to maybe check with yuri but he was saying something along these lines [21:37:03] ? [21:37:30] milimetric: no, they will come from api [21:37:44] milimetric: they will be fetched just like EL does it correct? [21:37:50] milimetric: so same TTls [21:37:53] yurik: I can ask you later, not urgent [21:37:54] issues apply [21:37:59] * yurik is confused [21:38:33] yurik: ok, sorry, so if we deploy that dashiki extension I was showing you, and we have pages under Config:Dashiki:MyDashboard [21:39:09] then is there a way to fetch the json content in that page directly without parsing the API response? And if so, do those come with different TTLs for caching? [21:39:24] milimetric: yes, there is [21:39:26] https://meta.wikimedia.org/w/api.php?action=jsonschema&title=ExternalLinksChange&formatversion=2&revid=15716074 [21:39:34] or so I think [21:39:43] milimetric: but note it requires a revision id [21:39:53] milimetric: maybe there is a way to get latest [21:40:41] milimetric: and it comes with max-age=300 [21:40:55] which is 5 minutes? [21:40:58] why do we want more? [21:41:28] milimetric: that is fine, but we do not want to pass revision id for sure , there must be a way to get latests [21:42:26] milimetric, technically no - you still have to parse api response. BUT, https://commons.wikimedia.beta.wmflabs.org/w/api.php?action=jsondata&formatversion=2&format=jsonfm&title=Sample.tab makes it somewhat easier [21:43:06] but wait yurik (w/o knowing nothing about this) [21:43:26] so there's jsonconfig, jsondata, and jsonschema? ... [21:43:34] yurik: this returns json: https://meta.wikimedia.org/w/api.php?action=jsonschema&title=ExternalLinksChange&formatversion=2&revid=15716074 [21:43:34] as for TTL - i am not doing anything special about it, but IMO, api calls are not cached much, are they? [21:43:38] nuria: looks like there's also max-age that we can set [21:43:56] nuria, jsonschema is not part of jsonconfig [21:43:57] milimetric: ya, with maxAge=3600 is how we do it elsewhere [21:44:30] yurik: that i guess means "not easy to do " in mediawiki speak [21:44:42] nuria, ?? not sure what you mean [21:45:14] :) I think I understand, it's ok, yurik, thanks for explaining [21:45:27] yurik: i guess that jsonschema is just eventlogging specific , i though it was mediawiki [21:45:48] yurik: ya, i think i understand too, thnaks [21:45:50] *thanks [21:45:51] nuria, jsondata is not enabled anywhere yet, but we could enable it anywhere we want [21:47:09] btw, yurik, can't you just use json instead of jsonfm in your example? https://commons.wikimedia.beta.wmflabs.org/w/api.php?action=jsondata&formatversion=2&format=json&title=Sample.tab [21:47:31] milimetric, of course, jsonfm is just for demo [21:47:56] right, so that's cool! We can use that when it's available, anything I should do to help get it deployed? [21:48:16] milimetric, ask debt on interactive channel when it gets deployed :) [21:48:17] we wanted to deploy our extension and everything that's needed for it, and we're not afraid to cry whine and fight :) [21:48:35] sure, but anything we can do to advocate for it? [21:50:21] milimetric, technically, we could enable action=jsondata without the tabular data - i think there is a flag for taht [21:52:10] hm... now I'm not sure if we should store our configs just as tabular data if that's available soon [21:52:27] that'd be cool... or maybe the subnamespace would be useful too, to group them together [21:55:11] milimetric, not a good idea - tabular data will be deployed only on Commons' data [21:55:26] namespace, whereas you want a custom config for your system [21:56:25] k, makes sense, we'll just stick with the original. So jsondata can be used to get JsonConfig pages? [22:09:09] sorry, connection issues [22:11:26] milimetric, if $wgJsonConfigEnableLuaSupport is true, the Lua and API support is enabled [22:12:04] coolness [22:25:26] (03PS1) 10Alex Monk: Whitelist fiwikivoyage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324616 (https://phabricator.wikimedia.org/T151570) [22:27:39] (03Abandoned) 10Alex Monk: Whitelist fiwikivoyage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/324616 (https://phabricator.wikimedia.org/T151570) (owner: 10Alex Monk) [22:28:09] (03CR) 10Alex Monk: "Ping. Wiki is now live." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/323699 (https://phabricator.wikimedia.org/T151570) (owner: 10MarcoAurelio) [22:47:29] 06Analytics-Kanban, 10Wikimedia-Site-requests, 13Patch-For-Review, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570#2836702 (10Krenair) Stuff to do: * RB restart * Parsoid deploy - @ssastry? * Labs replicas - @chasemp or @andrew need to run somet... [23:51:20] 06Analytics-Kanban, 06Operations, 10hardware-requests: stat1001 replacement box in eqiad - https://phabricator.wikimedia.org/T149911#2836823 (10RobH) 05Open>03Resolved reinstalled, puppet and salt keys accepted. it has some puppet failures, but since those are service related, i'll leave them to you to... [23:51:22] 06Analytics-Kanban, 13Patch-For-Review: Replace stat1001 - https://phabricator.wikimedia.org/T149438#2836825 (10RobH)