[08:56:01] 10Analytics, 10Continuous-Integration-Config: Add CI to all analytics/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180301#3753824 (10hashar) [10:57:19] 10Analytics, 10Wikimedia-Stream, 10Services (watching): Update EventStreams to service-template-node v0.5.3 - https://phabricator.wikimedia.org/T171011#3754281 (10mobrovac) [11:10:57] Hue seems returning 500s... [11:11:12] I can't log in [11:11:38] now this is really not great [11:11:39] WebHdfsException: HTTPConnectionPool(host='analytics1001.eqiad.wmnet', port=14000): Max retries exceeded with url [11:12:01] we have blocked the daemon a long time ago [11:14:17] now it works [11:14:18] whaaaat [11:15:46] !log suspend pageview-druid-hourly-coord to allow an easier druid daemon reload (new prometheus jvm agent) [11:15:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:20:02] all right restarting all the druid daemons to get the jvm prometheus exporters working [11:20:06] Cc: joal --^ [11:53:24] joal: I'd need some help if you are around [12:24:23] ok solved, there was an issue with the coordinator restarts [12:29:12] all right going to lunch and then commute to home [12:29:25] it is showing in here!! [12:37:37] Hi elukey - I'm here now [12:38:31] elukey: but too late from what I reada :( [13:05:29] !log Rerun webrequest-druid-hourly-wf-2017-11-13-11 [13:05:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:08:50] !log Suspend webrequest druid loading waiting for elukey [13:08:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:20:15] 10Analytics: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#3754629 (10JAllemandou) [13:20:17] 10Analytics, 10Analytics-Data-Quality: wmf.mediawiki_history: page_is_redirect/page_is_redirect_latest - https://phabricator.wikimedia.org/T180310#3754631 (10JAllemandou) [13:20:38] Hig GoranSM - Just merged your new task in the already exisisting one on hat matter: ) [13:43:44] 10Analytics: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#3754693 (10GoranSMilovanovic) @JAllemandou Thanks! [13:55:35] (03PS1) 10GoranSMilovanovic: Structure change for production 13 Nov 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391000 [13:56:10] (03CR) 10Addshore: [C: 032] Structure change for production 13 Nov 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391000 (owner: 10GoranSMilovanovic) [13:56:17] (03Merged) 10jenkins-bot: Structure change for production 13 Nov 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391000 (owner: 10GoranSMilovanovic) [13:59:55] joal: o/ [14:00:11] it took me a ton of time to reach home, snow everywhere /o\ [14:00:13] Heya elukey [14:00:31] now worries elukey - I'm sorry to have late started today :S [14:00:53] nono it was a weird thing that happened with the coordinator [14:01:05] I pinged in case you were around to get some ideas :) [14:01:14] elukey: looks like it;s still broken (I suspended webrequest hourly because of issues) [14:01:37] joal: it shouldn't, what do you see? [14:01:53] error at indexing tiem - didn't investigate, just waited for you [14:02:03] also elukey, I think pageview hourly jobs are still suspended :) [14:02:36] ah snap I forgot to stop webrequest load [14:02:38] ufff [14:02:40] no big deal [14:02:50] yeah I still need to complete the restarts [14:02:52] so, how do you want us to proceed? [14:02:59] I've only done druid100[123] [14:03:22] elukey: those are the hard ones - real-time jobs, hourly jobs etc [14:03:37] elukey: 456 are prod, but almost nothing uses them [14:03:43] (yet) [14:03:59] ok so going to restart them [14:05:48] joal: so the coordinator issue was due to errors connecting to the an1003's database [14:06:06] elukey: Weirdohh [14:06:11] when we split the cluster in two we created two schemas on the db [14:06:20] druid and druid_public [14:07:21] the db user is 'druid' for both [14:07:37] but in puppet private it was listed with two passwords [14:08:02] I think that when the public cluster was created we modified the druid's user password [14:08:24] but since the coordinators still have long lived conns with the db, nothing happened for druid 'analytics' [14:08:31] then when Luca restarted them [14:08:38] a new connection attempt was created [14:08:47] ending up in "access denied" [14:08:55] I temporarily fixed the issue [14:09:02] but we might want to have two separate users [14:09:19] joal: so in theory now the druid analytics cluster should be ok [14:09:50] elukey: woaw ... Makes sense - good catch debbugging that 1 [14:10:04] elukey: I still don't understand why jobs fail [14:10:17] elukey: I'm gonna try to restart pageview and see [14:10:20] all right [14:10:42] !log Unsuspend pageview-druid-hourly-coord [14:10:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:10:53] will it push data to druid100[456] ? [14:11:10] elukey: nope, we only have jobs pushing data to private cluster [14:11:28] public cluster is not productionise in term of loading [14:13:36] * elukey is too ignorant [14:14:56] (03PS1) 10Fdans: Adds metric schema checking and improves display of number units [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/391003 (https://phabricator.wikimedia.org/T178084) [14:16:15] Right elukey - Something's broken on the cluster - pageview jobs failed as well [14:17:33] * joal is going into druid investigation mode - With gold-serp and some herbs [14:18:14] !log Suspending pageview-druid-hourly-coord again trying to fix druid loadin [14:18:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:19:33] elukey - redirects seem the issue [14:19:46] https://hue.wikimedia.org/jobbrowser/jobs/job_1504006918778_296040/single_logs [14:19:47] Caused by: java.sql.SQLException: Access denied for user 'druid'@'10.64.36.102' (using password: YES) [14:19:49] elukey: --^ [14:20:10] elukey: Ah - maybe [14:20:20] that one is the overlord [14:21:53] ahhh joal! Now I got it! [14:22:00] lemme restart the overlord [14:22:02] (s) [14:22:06] sure elukey [14:22:08] they still have the old password! [14:22:17] elukey: you didn't restart them ? [14:23:16] only once [14:23:26] then I discovered the coordinator issue (it was throwing alarms) [14:23:37] so I restart only the coordinators a second time [14:23:39] not the overlords [14:24:31] joal: can you retry now? [14:25:44] sure elukey [14:31:57] reporting for duty [14:34:14] milimetric: o/ [14:34:31] hi elukey [14:34:53] Hi milimetric [14:35:01] hey joal [14:36:26] (03PS1) 10GoranSMilovanovic: Initial [analytics/wmde/WDCM-Overview-Dashboard] - 10https://gerrit.wikimedia.org/r/391008 [14:37:01] (03CR) 10GoranSMilovanovic: [C: 032] Initial [analytics/wmde/WDCM-Overview-Dashboard] - 10https://gerrit.wikimedia.org/r/391008 (owner: 10GoranSMilovanovic) [14:37:07] (03Merged) 10jenkins-bot: Initial [analytics/wmde/WDCM-Overview-Dashboard] - 10https://gerrit.wikimedia.org/r/391008 (owner: 10GoranSMilovanovic) [14:37:34] joal: all good with indexing? [14:39:29] o/ [14:39:41] morning! [14:39:53] fyi, am about to do el deployment :) [14:40:00] elukey: still waiting [14:41:46] joal: ack, whenever you have time can we write down a guide to restart properly the druid clusters? Thinks to keep in mind, etc.. [14:42:12] (03PS1) 10GoranSMilovanovic: Init [analytics/wmde/WDCM-Usage-Dashboard] - 10https://gerrit.wikimedia.org/r/391010 [14:43:37] (03CR) 10Addshore: [C: 032] Init [analytics/wmde/WDCM-Usage-Dashboard] - 10https://gerrit.wikimedia.org/r/391010 (owner: 10GoranSMilovanovic) [14:43:43] (03Merged) 10jenkins-bot: Init [analytics/wmde/WDCM-Usage-Dashboard] - 10https://gerrit.wikimedia.org/r/391010 (owner: 10GoranSMilovanovic) [14:43:48] elukey: Let's DO IT ! [14:44:12] !log Resuming all druid loading jobs after fixing restart issues [14:44:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:44:16] joal: now? [14:44:23] elukey: when you wish :) [14:44:35] (03PS1) 10GoranSMilovanovic: Init [analytics/wmde/WDCM-Semantics-Dashboard] - 10https://gerrit.wikimedia.org/r/391012 [14:44:39] elukey: problem solved - Thanks for having fixed the thing ! [14:44:48] joal: sorry for the trouble :( [14:45:10] elukey: no trouble - only moar learning :) [14:45:29] (03CR) 10Addshore: [C: 032] Init [analytics/wmde/WDCM-Semantics-Dashboard] - 10https://gerrit.wikimedia.org/r/391012 (owner: 10GoranSMilovanovic) [14:45:35] (03Merged) 10jenkins-bot: Init [analytics/wmde/WDCM-Semantics-Dashboard] - 10https://gerrit.wikimedia.org/r/391012 (owner: 10GoranSMilovanovic) [14:45:58] joal: if you have time I'll go to the cave :) [14:46:08] elukey: OMW ! [14:46:24] (03PS1) 10GoranSMilovanovic: Init [analytics/wmde/WDCM-ShinyServerFrontPage] - 10https://gerrit.wikimedia.org/r/391013 [14:47:05] (03CR) 10Addshore: [C: 032] Init [analytics/wmde/WDCM-ShinyServerFrontPage] - 10https://gerrit.wikimedia.org/r/391013 (owner: 10GoranSMilovanovic) [14:47:11] (03Merged) 10jenkins-bot: Init [analytics/wmde/WDCM-ShinyServerFrontPage] - 10https://gerrit.wikimedia.org/r/391013 (owner: 10GoranSMilovanovic) [15:01:16] elukey: can you create gerrit repos? [15:01:36] addshore: o/ [15:01:39] yep [15:01:41] ooooh :D [15:01:51] Could you create a analytics/wmde/WDCM-packages for me? [15:02:06] inherit from analytics/wmde ? [15:02:10] it will contain a commited / in git verison of the R packages needed by the WDCM scripts [15:02:12] yes please! [15:02:52] done! [15:02:55] With that I think I should be able to make a finalish version of the puppet patches for this stuff. [15:02:57] thanks!!!! [15:06:07] (03PS1) 10Addshore: .gitreview [analytics/wmde/WDCM-packages] - 10https://gerrit.wikimedia.org/r/391016 [15:06:19] (03CR) 10Addshore: [V: 032 C: 032] .gitreview [analytics/wmde/WDCM-packages] - 10https://gerrit.wikimedia.org/r/391016 (owner: 10Addshore) [15:08:29] joal: http://druid.io/docs/0.9.2/operations/metrics.html -> Real Time doesn't seem to be super useful [15:08:52] ah no wait maybe "Ingestion Metrics" ? [15:11:55] elukey: real-time + ingestion seems actually interesting [15:14:07] joal: from a first glance I'd do it in a second iteration of the druid_exporter [15:14:17] to establish a bit better what we need [15:14:20] but it looks super cool [15:14:20] works for me elukey [15:21:08] joal, elukey, the pageview and webrequest druid alerts are triggered because of your task right now? [15:21:20] BTW, helloooo :] [15:21:40] mforns: o/ - yes my fault sorry [15:21:46] if not, I will look into them, ops week [15:21:52] np ok [15:24:23] mforns: https://gerrit.wikimedia.org/r/#/c/391020 [15:29:17] ah mforns, I restarted the cleaning on db1108 [15:34:34] elukey, ok! [15:34:57] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3755154 (10jcrespo) Proposal C: * Maintenance of datasets is puppetized (so it is both the source and... [15:43:52] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resolve EventCapsule / MySQL / Hive schema discrepancies - https://phabricator.wikimedia.org/T179625#3755242 (10Ottomata) Alright, EL code changes are running fine in production. Here's the steps to make them take affect, with respect to the... [15:58:47] 10Analytics, 10Pageviews-API: Provide weekly top pageviews stats - https://phabricator.wikimedia.org/T133575#2236232 (10Trizek-WMF) Requested 5 minutes ago on IRC wikipedia-fr. The goal is to easily update the weekly [[ https://fr.wikipedia.org/wiki/Wikipédia:Wikimag | on-wiki magazine ]]. [16:11:46] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3755413 (10Ottomata) [16:17:41] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3755449 (10bd808) >>! In T173511#3755154, @jcrespo wrote: > Proposal C: > > * Maintenance of datasets... [16:21:07] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3755465 (10jcrespo) Yes, although not sure it is really a separate proposal; it would just be a small... [16:24:54] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3755480 (10Aklapper) [16:24:56] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820#3755479 (10Aklapper) [16:25:09] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820#3737130 (10Aklapper) Yeah, good point [16:26:38] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2: The granularity selector does not work for tops metrics - https://phabricator.wikimedia.org/T180266#3755484 (10JAllemandou) [16:26:45] 10Analytics-Kanban: Wikistats2: The granularity selector does not work for tops metrics - https://phabricator.wikimedia.org/T180266#3751801 (10JAllemandou) [16:35:47] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3755502 (10bd808) I agree that Proposal C is not really opposing A or B. It is instead proposing an im... [16:38:12] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3755509 (10jcrespo) As a pro- I have run many times into people doing the certain summary queries many... [16:47:14] (03CR) 10Ottomata: [C: 031] Add clickstream oozie job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [16:50:30] (03CR) 10Joal: [C: 04-1] "I'd like to add a renaming script for files - They don't look good for public consumption as is." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [16:52:57] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resolve EventCapsule / MySQL / Hive schema discrepancies - https://phabricator.wikimedia.org/T179625#3755571 (10Ottomata) @Tbayer I'd really like to move on the above plan asap, seeing as I will be out the ends of the next two weeks, and I'd m... [16:55:34] ;; ANSWER SECTION: [16:55:38] analytics-slave.eqiad.wmnet. 300 IN CNAME db1108.eqiad.wmnet. [17:54:47] bearloga: around? :D [18:11:47] HaeB: hiyaaa, would love to do this this afternoon, does this plan sound ok to you? https://phabricator.wikimedia.org/T179625#3755571 [18:11:51] re. popups table [18:11:53] in hive [18:24:22] joal: o/ [19:06:42] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Possibly faulty BBU on analytics1029 - https://phabricator.wikimedia.org/T178742#3756252 (10elukey) Chris swapped the battery (two times) and it seems that the second one is ok! Will keep an eye on it and close the task tomorrow if everything is good. [19:07:08] elukey: \o [19:07:30] hey joal! I had to kill a spark job on an1029 due to maitenance [19:07:48] I think it was bannerimpression, but not the app master [19:07:55] elukey: let me check [19:08:27] indeed elukey: 20:07:30 < elukey> hey joal! I had to kill a spark job on an1029 due to maitenance [19:08:32] arf [19:08:36] copy paste error again [19:08:44] elukey: https://yarn.wikimedia.org/proxy/application_1504006918778_285237/executors/ [19:09:03] elukey: Therefore no bother - spark managed ;) [19:09:11] Thanks for letting me know though :) [19:09:34] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resolve EventCapsule / MySQL / Hive schema discrepancies - https://phabricator.wikimedia.org/T179625#3756267 (10Tbayer) Could this be held off two more days, when the data collection for this one ends (T178500)? Having to join two tables with... [19:09:59] oookk!!! :) [19:10:03] * elukey off! [19:10:08] bye elukey [19:15:59] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Resolve EventCapsule / MySQL / Hive schema discrepancies - https://phabricator.wikimedia.org/T179625#3756298 (10Ottomata) Ah great! Didn't realize it was ending so soon. That's fine, we can wait. > Having to join two tables with incompatibl... [19:17:06] ottomata: replied there [19:18:00] oh and you replied again too ;) my IRC wasn't updating for some reason [19:25:17] :) [19:25:56] HaeB: , if we wait, do you mind/care if I don't go through the steps to make event.popups contain all the data in tbayer.popups? [19:26:04] then tbayer.popups can remain as is [19:26:11] we'll just stop refining data to it [19:26:17] then event.popups will only have new data [19:27:18] i think that should be ok [19:27:56] BTW there's a possibility we will reactivate the schema soon to collect data again for a new question (https://phabricator.wikimedia.org/T178500#3756363 )... [19:28:07] ..for that, the new table will be fine [19:28:24] but i'm curious, how does schema versioning work for the Hive refined EL? [19:28:51] ok great [19:29:02] HaeB: pretty much, you can't make incompatible changes to the schemas [19:29:08] you can add fields [19:29:10] but you can't change types [19:29:50] its actually more important that the refinement process never sees records with different types [19:30:02] so, full jsonschema (with union types) is not supported [19:30:16] the EventLogging extension won't actually let you declare union types [19:30:18] but it will let you say [19:30:24] "type": "any" [19:30:29] as it is, ^ won't break refinement [19:30:29] but [19:30:43] if any two records ever have different types for the same field [19:30:44] it will break [19:33:03] in this case it will just be a new field [19:33:23] for the general future setup this sounds like another thing worth documenting and discussing with data ausers ;) [19:37:24] (03Draft1) 10Addshore: WDCM libs [analytics/wmde/WDCM-packages] - 10https://gerrit.wikimedia.org/r/391069 [19:40:32] ottomata: if you have a few moments could I get you to read and comment on https://phabricator.wikimedia.org/T171258#3756418 ? :D [19:50:05] (03PS1) 10GoranSMilovanovic: _installProduction_analytics-wmde.R puppetization [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391071 [19:50:29] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] _installProduction_analytics-wmde.R puppetization [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391071 (owner: 10GoranSMilovanovic) [19:50:35] (03Merged) 10jenkins-bot: _installProduction_analytics-wmde.R puppetization [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391071 (owner: 10GoranSMilovanovic) [19:57:31] HaeB ya for sure [20:00:56] thanks ottomata! [20:08:03] leila [20:08:23] sorry :) wrong chat! [20:15:17] ottomata: hey there! who's the right person to ask Data Lake questions? There's no October snapshot yet and I'm wondering when I should expect it. [20:15:31] joal: or milimetric i think [20:15:41] gotcha, thanks! [20:16:09] (October mediawiki_history snapshot, to be specific) [20:29:15] neilpquinn: took me a moment, this month was a bit different [20:29:33] we were improving some things and Joseph killed the normal job so he could start it when the changes are ready [20:30:06] neilpquinn: we found discrepancies in how revisions joined to pages and stuff like that, when we compared more closely with Wikistats 1.0 [20:30:23] neilpquinn: so he's working on the fixes now. [20:30:42] neilpquinn: if you'd rather the old logic be run just so you have updated data, let us know [20:30:59] milimetric: thanks for the update, I appreciate the work on data quality :) [20:31:40] yeah, we thought it would be a quick fix, but we should announce this so others aren't confused. Sending an email [20:31:40] milimetric: it would be helpful to have the old logic run. I made new editor retention part of our monthly metrics, so now they depend on the data lake. [20:32:02] neilpquinn: gotcha, ok, will make sure Joseph knows [20:32:24] On my part, I should've let you know I had built something dependent on the data lake :) [20:33:21] milimetric: do you think the run can be done today or tomorrow? if not, no worries—I'll just know to leave retention out this month. [20:34:02] neilpquinn: ah, it might not be fast enough unless we start now... ooof, I'm sorry :( [20:34:28] I'll see if I can start it now and we can replace it later worst-case. [20:35:00] milimetric: these are generaed once a month, and we are halfway throough nov. maybe just do oct with old logic, and then get joseph's changes on nov data? [20:35:18] milimetric: it's all right, growing pains happen :) do your best, but don't kill yourself trying to get it started. not a huge deal if we don't have it this month. [20:35:19] ottomata: yeah, that's what I'm gonna try to launch now [20:42:29] addshore: merged. [20:42:37] Notice: /Stage[main]/Statistics::Wmde::Wdcm/File[/srv/analytics-wmde/wdcm]/ensure: created [20:43:07] ottomata: awesome :) [20:43:22] yep, looks like it is all there I might have another patch in a second adding a little more [20:51:30] (03PS1) 10GoranSMilovanovic: install script knows where it lives [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391080 [20:52:35] (03CR) 10Addshore: [C: 04-2] "# - setup" [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391080 (owner: 10GoranSMilovanovic) [20:54:16] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] install script knows where it lives [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391080 (owner: 10GoranSMilovanovic) [20:58:09] (03PS1) 10GoranSMilovanovic: minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391083 [20:58:26] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391083 (owner: 10GoranSMilovanovic) [21:09:26] (03PS1) 10GoranSMilovanovic: Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 [21:09:51] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:11:19] (03Abandoned) 10Addshore: install script knows where it lives [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391080 (owner: 10GoranSMilovanovic) [21:13:40] (03CR) 10Addshore: [V: 032 C: 032] "Looks good and is running for me" [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:14:46] (03PS2) 10Addshore: Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:14:53] (03CR) 10jerkins-bot: [V: 04-1] Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:14:59] (03Abandoned) 10GoranSMilovanovic: minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391083 (owner: 10GoranSMilovanovic) [21:15:03] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3756679 (10bd808) Do we need a full blown [[https://en.wikipedia.org/wiki/Extract,_transform,_load|ETL... [21:16:07] (03PS3) 10Addshore: Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:16:32] (03CR) 10Addshore: [C: 032] Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:16:38] (03Merged) 10jenkins-bot: Installation script mkdirs r-library [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391086 (owner: 10GoranSMilovanovic) [21:19:20] ottomata: is !log a thing in this channel? [21:20:27] !log test [21:20:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:20:33] ooh yes,! [21:20:35] !log addshore@stat1005:/srv/analytics-wmde/wdcm/src$ sudo -u analytics-wmde Rscript ./_installProduction_analytics-wmde.R [21:20:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:24:43] (03PS1) 10Addshore: Add r-library dir to .gitignore [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391089 [21:25:02] (03CR) 10Addshore: [C: 032] Add r-library dir to .gitignore [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391089 (owner: 10Addshore) [21:25:08] (03Merged) 10jenkins-bot: Add r-library dir to .gitignore [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/391089 (owner: 10Addshore) [21:32:28] ottomata: Was it you that said something about deploying new code with scap? [21:32:39] If so, do you have a link to something that I can read about setting that up? [21:34:09] addshore: https://wikitech.wikimedia.org/wiki/Scap#Other_Software_Deployments [21:34:09] ? [21:35:21] maybe *reads* [21:36:59] Yeh, so it looks like it is possible to use that to basically just pull a git repo [21:37:30] I wonder if there is any point or to just keep pulling manually, or switch to the way I do the wmde-scripts directory where there is a "production" branch that is pulled to "latest" in puppet [22:56:22] 10Analytics-Kanban, 10Analytics-Wikistats: Replace any debouncing with Vue.nextTick - https://phabricator.wikimedia.org/T180412#3756944 (10Milimetric)