[02:19:34] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10Reedy) [07:31:49] good morning :) [07:47:15] Heya [07:49:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) Status: from my tests everything seems working fine, but this delicate change should probably be applied before all hands to avoid headaches :) Next steps: R... [07:49:42] elukey: shall we deploy this morning? --^ [07:52:25] joal: I would love it but tomorrow I'll be on a plane and I don't want to leave headaches to the team if anything breaks ;) [07:52:34] so probably after all hands [07:52:39] what do you think? [07:52:41] meh - We'll have all day to realize if it breaks :) [07:53:04] yes but say that it breaks in weird ways only once every x jobs etc.. [07:53:44] hm [07:53:54] It is not super urgent so I think that we can leave it getting a bit of dust [07:54:16] elukey: only thing that broke was pyspark - we can easily try that, and oozie will run some job (or I can force it to test)_ [07:54:19] As you prefer :) [07:55:01] joal: I also got some failures with the spark-shell yesterday, very rare but could happen.. also we don't know if the heisenbug will not re-appear in prod :( [07:55:08] or another incarnation of it [07:55:08] ok [07:55:30] I re-tested this morning queries in Hadoop Test and everything looks good [07:55:40] oozie should already be ready to go [07:55:46] is there anything missing? [07:55:54] I'll also talk about this with Erik [07:56:03] I pinged him on the task but will try to triple check [07:56:21] it all seems good on my side [07:58:14] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10JAllemandou) @Reedy : kerberos ticket needs to be created in the notebook environment, using a notebook terminal page. See https://wikitech.wikimedia.org/wiki/SWAP#Kerberos. [07:58:41] also elukey, here we are about bz2: https://phabricator.wikimedia.org/T243241 [08:01:47] nice! [08:01:57] also kudos for the upstream patch! [08:29:58] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Some xml-dumps files don't follow BZ2 'correct' definition - https://phabricator.wikimedia.org/T243241 (10ArielGlenn) Thanks for the heads up. This is probably a side effect of using lbzip2 as the compressor for these files. I'll be monitoring the progress... [08:38:12] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Some xml-dumps files don't follow BZ2 'correct' definition - https://phabricator.wikimedia.org/T243241 (10JAllemandou) > In the meantime, might you be able to use bzip2 to decompress and recompress the problem file(s) so that you can get your import into h... [08:38:57] creche just called, Naé's turn to have the gastoenteritis - I'll be flaky on schedule today [08:42:38] hi joal, yt? [08:49:56] hola dsaez [08:50:08] joal: ouch ok :( [09:07:56] dsaez: just created your kerberos account, you should now see an email in your inbox with istructions [09:16:04] elukey, great, I've received. Let me see if this solves the problem (I'll lose my connection now [flying], and reconnect in around 30 mins) thx [10:07:53] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10elukey) @MarcoAurelio thanks a lot for the feedback! I am trying to find use cases for the IRC channel, we don't want to cause disruptions to i... [10:23:57] 10Analytics, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) I have created https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue#Hive_query_errors_with_Kerberos to help users finding the workaround. [10:24:16] 10Analytics, 10Analytics-Kanban, 10User-Elukey: No queries run in Hue - https://phabricator.wikimedia.org/T242306 (10elukey) [11:58:32] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Some xml-dumps files don't follow BZ2 'correct' definition - https://phabricator.wikimedia.org/T243241 (10ArielGlenn) From chat on irc: we're waiting on a release to come from the upstream patch; then folks here will see if they can tweak the local hadoop... [12:00:45] * elukey lunch! [12:45:02] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10dr0ptp4kt) @JAllemandou I noticed you at-mentioned @Reedy here. Does @Reedy need to add some kind of permissions? For the sake of completeness from the Jupyter terminal I did a `kdestroy` and fresh `kinit`, but t... [12:56:16] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10JAllemandou) @dr0ptp4kt Thanks for making me seing things correctly :) I think you need to `kdestroy` from ssh session to clean this session, then `kdestroy` and `kinit` again in notebook terminal to start fresh.... [13:34:16] \o/ commons-compress patch merged upstream :) [13:38:09] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Some xml-dumps files don't follow BZ2 'correct' definition - https://phabricator.wikimedia.org/T243241 (10JAllemandou) Patch merged upstream, will be released as part of commons-compress 1.20 Will follow up to try to force use that version once published. [13:38:22] joal: woooooww nice! [13:38:42] very responsive and nice people :) [13:53:58] (03PS3) 10Joal: Update wikitext oozie job adding current [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565558 (https://phabricator.wikimedia.org/T238858) [13:57:18] o/ [13:57:26] hi milimetric [13:57:36] oof wishing you the best with stomach bug, that's the worst [13:58:07] milimetric: it's short this year (one day) - Not too bad [13:58:30] But that's the thing with those: one of us catch it, then the whole family gets it it, with more or less delay [13:58:30] hm, a not-too-bad plague, interesting :) [13:58:36] :) [13:59:01] yeah, it's awful when you realize with certainty that it wasn't some bad food that just one person ate, but indeed something that you will definitely go through very soon [13:59:15] like a rollercoaster with a 2-day climb to the top [14:00:02] if anybody has time, after reviewing https://phabricator.wikimedia.org/T220575 I came to the conclusion that I can remove all the old home dirs [14:00:11] can you double check? [14:00:27] Nuria should have saved all IIUC, but better safe when deleting :D [14:04:05] * joal is afraid of data deletion - looking away [14:13:51] heh, wonder what youtubeconspiracies_result is [14:17:24] +1 elukey [14:17:31] reviewed everything and seems good [14:18:16] well, elukey I don't have access to that doc where nuria says to follow for updates, so if someone else can double check that, I guess that'd be more thorough [14:18:44] milimetric: I checked, it seems that all is saved, but lemme give you access [14:18:47] if I can [14:19:26] milimetric: done [14:24:01] ok, elukey -1, no good [14:24:21] milimetric: is there anything not copied? [14:24:24] there are several things marked as SAVE (SECURE) that are only backed up in nuria's home folder on the notebooks [14:24:29] those should be moved to HDFS [14:24:45] and there are several things marked as SAVE on notebook1003 that have no notes [14:24:56] (some of them look like they need neil's attention but not all) [14:25:11] good point about the SECURE, I'll wait for Nuria then [14:25:16] thanks for the review :) [14:26:08] elukey: also same for stat1004. I would argue these should all be saved in HDFS in a folder where we'll save any former coworker's work so they can all be found in one single place [14:26:38] like /wmf/data/archive/former-contributors/tbayer or something like that [14:26:40] could be an interesting idea [14:41:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10Ottomata) > this delicate change should probably be applied before all hands to avoid headaches You mean after all hands? :) [14:44:07] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Enable encryption in Spark 2.4 by default - https://phabricator.wikimedia.org/T240934 (10elukey) >>! In T240934#5819475, @Ottomata wrote: >> this delicate change should probably be applied before all hands to avoid headaches > You mean after all hands? :)... [14:45:13] bbiab [14:52:25] bibimbap? [14:53:10] fdans is in Korean mode :) [14:54:06] milimetric: the annoying timerange was the fact that date-fn doesn't know about utc, so I had to add a lil preparation for it [14:54:14] can you check that the patch is good now? [14:55:28] Will look [15:01:06] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) > But first, I think a big source of confusion in our patches is the conflation of the word 'service'. I think you are r... [15:22:37] 10Analytics, 10Product-Analytics: Check home leftovers of dfoy - https://phabricator.wikimedia.org/T239571 (10elukey) @Milimetric did you get any response? [15:23:27] 10Analytics, 10Product-Analytics: Check home leftovers of dfoy - https://phabricator.wikimedia.org/T239571 (10Milimetric) Yes, Yael said it was fine to delete. [15:28:09] milimetric: +1 to remove dfoy's homes then? [15:30:53] * milimetric takes a deep breath [15:36:35] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915 (10Milimetric) I'm looking at this again, and I think @sahil505 was exactly right - best to define the variables in some root file and use them everywhere. Side note for myself to look... [15:48:27] (03CR) 10Milimetric: [C: 04-1] "Nope, all messed up now, most widgets show October -> October for the last 24 months instead of December -> December" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [15:49:25] milimetric: wat, I was testing with nuria's timezone and it was all good Dec-Dec [15:49:36] #abolishTimezones [15:50:03] seriously, did you see the article luca shared? [15:50:43] fdans: I donno, sorry I don't have time to help debug, but it's a new issue since last patch so most likely it's not a fluke [15:50:55] (I didn't do any dev on that repo since I last tested) [15:51:07] milimetric: nono thank you, I'm just confused [15:51:26] I'll take another look [15:54:26] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > My kneejerk reaction to this is "instance of what"? of eventgate? K cool, let's figure out a different name. I like ser... [15:56:46] milimetric: where are you seeing this problem manifest itself? [16:04:48] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10dr0ptp4kt) That worked, @JAllemandou . Thanks! I updated https://wikitech.wikimedia.org/wiki/SWAP#Kerberos. [16:16:52] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) I had a very nice chat with the BigTop committers, here's some reference: https://lists.apache.org/thread.html/r9b588c1c9f693bd78549e7f3251004bc114c7... [16:19:16] 10Analytics: Unable to access Hive from notebook1003 - https://phabricator.wikimedia.org/T243239 (10elukey) 05Open→03Resolved a:03JAllemandou [16:46:24] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey) Summary of current thoughts: * From https://lists.apache.org/thread.html/r9b588c1c9f693bd78549e7f3251004bc114c754b8d16f4edd796b828%40%3Cuser.bigtop.... [16:52:58] fdans: sorry it's on the homepage the dashboard widgets [16:53:07] which were fine before the latest patch [16:53:28] sorry I thought I was unambiguous with "widget" but realized that's not true [16:53:41] hmmmm [16:53:50] milimetric: I don't see it but let me do a full reload [16:54:53] that's weird... [17:48:22] (03PS3) 10Fdans: Add language selection functionality to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/564047 (https://phabricator.wikimedia.org/T238752) [18:07:43] fdans / ottomata: strategy for the v2 deploy: [18:08:03] * make new repo or push to a branch (I prefer new wikistats-deploy repo) [18:08:21] * update the puppet change, test v1 links [18:08:25] * deploy [18:08:36] so let's make the repo/branch decision together now [18:08:39] and go from there [18:10:41] milimetric: I'd rather have the same repo because dealing with cross repo is annoying [18:11:14] fdans: but we had the same repo like you were saying and decided against it, some of those merges were bad, and caused problems [18:11:27] (remember where we were duplicating that code in main.js I think it was?) [18:12:29] milimetric: hmmmyea but I don't see how that would be fixed by having it in a separate repo [18:12:50] the other repo would be literally just the target of the build step, so we build, push, that's it [18:13:06] no merge or anything, the build deletes old stuff in the clean step [18:13:53] milimetric: ok, we gotta create the repo tho [18:15:12] 10Analytics: Easier Dashboards. Superset can authenticate to presto via kerberos - https://phabricator.wikimedia.org/T241189 (10Nuria) Anyone on LDAP user groups can access now (via presto) anything that is on hive using presto. This conflates somewhat data access opened via LDAP with the privacy of some datas... [18:16:01] fdans: that's easy, question is if there are other downsides, you think it'd be annoying, does using it as the build target solve that or are there other reasons? [18:16:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Nuria) @cchen they need to be added to the data source on superset, you can just ed... [18:16:49] milimetric: I don't know, both approaches are more annoying than the current [18:17:09] I think the new repo one uses less extraneous things [18:17:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Milimetric) >>! In T232659#5813403, @Nuria wrote: > As far i can see month, day_of_... [18:17:29] like, it doesn't make us use a subtree branch that only contains the dist contents [18:19:19] fdans: hm, we could make a v1 folder in the source, and put it in .gitignore. Then we would never be able to accidentally overwrite the v1 folder in the htdocs [18:20:00] that would let us keep everything as is and protect v1 (which should be backed up anyway) [18:20:47] and we could make /v1 be a symlink that's ensured by puppet actually, so it's extra safe [18:20:52] yeah, we'd be good then [18:21:02] oh wait [18:21:12] nvm, I'll listen and we can talk after [18:23:41] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) [18:25:32] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) [18:25:41] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) Ideas: * VPN * Single Sign On [18:26:18] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) 05duplicate→03Open [18:30:48] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) [18:30:48] 10Analytics, 10Security-Team, 10User-Elukey: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10Nuria) [18:32:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10cchen) @Nuria I tried to add new columns 'language' in datasource page for edit_hou... [18:36:19] 10Analytics, 10Product-Analytics: Add Presto to Analytics' stack - https://phabricator.wikimedia.org/T243309 (10Nuria) [18:36:36] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) [18:36:37] 10Analytics, 10Product-Analytics: Add Presto to Analytics' stack - https://phabricator.wikimedia.org/T243309 (10Nuria) [18:37:13] 10Analytics: Defining a better authentication scheme for Druid and Presto - https://phabricator.wikimedia.org/T241189 (10Nuria) [18:37:14] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Kerberize Superset to allow Presto queries - https://phabricator.wikimedia.org/T239903 (10Nuria) [18:38:01] 10Analytics, 10Product-Analytics: Add Presto to Analytics' stack - https://phabricator.wikimedia.org/T243309 (10Nuria) [18:38:30] 10Analytics, 10Product-Analytics: Enable shell access to presto from jupyter/stats machines - https://phabricator.wikimedia.org/T243312 (10Nuria) [18:41:05] elukey: did you sync jiawang's hue ldap account? [18:49:01] ottomata: I did yes, there was a task about it [18:49:47] great, she emailed us too! was wondering why it was alreday there :) [18:50:49] yeah I was about to answer! [18:50:55] also https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html contains what we discussed [18:51:18] (it is for hadoop 3 but didn't change from 2) [18:51:47] my main question now is about the druid and presto users [18:52:10] since in theory they can fetch any hdfs data, even without doing the runAs stuff [18:52:18] mmmm no wait [18:52:29] so druid can read its own indexed things, right [18:52:37] in its own hdfs dirs [18:52:57] but what is presto doing now when queried from superset? [18:59:48] I mean, 'presto' shouldn't be able to read privatedata by itself [19:02:17] ah ok [19:02:19] "Presto can impersonate the end user who is running a query. In the case of a user running a query from the command line interface, the end user is the username associated with the Presto CLI process or argument to the optional --user option. Impersonating the end user can provide additional security when accessing HDFS if HDFS permissions or ACLs are used. [19:02:48] right makes sense, i think hive is doing the same thing [19:03:03] oh hm [19:04:34] elukey: https://github.com/apache/incubator-superset/pull/3404 [19:05:33] maybe there is some way to put the username in the db connection string template [19:05:55] ah [19:05:56] {username} [19:05:56] hmmm [19:05:59] let me try that! [19:06:33] need to go afk for a bit! [19:09:17] elukey i think it works, not sure how to tell if the impersonation is being done right [19:09:18] but [19:09:36] try the presto_analytics_hive_usertest db [19:09:38] in superset [19:09:54] also, i should fix the presto CLI wrapper I made ( on an-presto1001) to use the shell username when connecting [19:32:19] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10dr0ptp4kt) [19:33:05] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10dr0ptp4kt) [19:35:26] milimetric: yoohoo [19:35:35] job task talk? [19:35:44] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10dr0ptp4kt) [19:37:38] ottomata: goin to the cave [19:52:59] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10Bstorm) We changed the exports. The mounts need to be unmounted and remounted. I can do that. [20:03:51] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10dr0ptp4kt) Thanks @Bstorm. Please advise when I should re-check. [20:04:04] ottomata: will try during the next days, logging off! [20:04:05] o/ [20:04:31] ok bye luca! [20:54:27] ok team - I have a working example of dask on the cluster :) [20:55:06] ottomata: I also think I have an idea of your problem for skein (I have seen your post on github) [20:58:19] 10Analytics: Spike. Try to ML models distributted in jupyter notebooks with dask - https://phabricator.wikimedia.org/T243089 (10JAllemandou) First step done: Having dask running on our cluster. Python environment setup: ` https_proxy=http://webproxy.eqiad.wmnet:8080 virtualenv -p /usr/bin/python3 dask_yarn cd d... [20:59:16] ottomata: I feel VINDICATED [20:59:31] ottomata: the rerun button is not on my hue ui, i must be missing some permits [20:59:41] no way nuria ... [20:59:43] batcave? [20:59:55] joal: just bc with marcel [21:00:00] ahahah :) [21:00:01] ok :) [21:00:03] joal: really [21:00:23] joal: i did not even know (sadness) that there were "other" buttons [21:00:38] joal: that is why i used the oozie cmd line for everything [21:00:44] meh [21:00:53] joal: but teh rerurns look kind of different when you do them that way [21:01:02] hm [21:01:05] joal: ALL this time thinking it was ME [21:01:20] nuria: just checked hue, you have the same permit we do (at least from what I can see) [21:01:41] nuria: https://hue.wikimedia.org/useradmin/groups [21:02:06] joal: i do not know aht to tell you, i though it was the privacy badger [21:02:11] joal: but ia hve it disabled [21:02:28] crazy [21:02:30] joal: I do not have permits to see that screen: https://hue.wikimedia.org/useradmin/groups [21:02:35] joal: ALL this time! [21:02:44] Ah! here we are getting somewhere [21:03:29] I think I found it nuria [21:03:35] nuria: try again looking at users [21:03:35] joal: WOW [21:03:50] joal: OMG [21:03:53] The button should be there I guess [21:03:55] \o/ [21:04:07] that setting is well hidden [21:04:17] joal: NOW it works [21:04:21] joal: man... [21:04:39] :( [21:04:40] joal: well, this issue has helped me in learning to use oozie in teh cmd line real well [21:04:48] We should have had this convo a long time ago [21:04:59] I can imagine yes [21:19:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Product-Analytics (Kanban): Add dimensions for Project type & language to Edits_hourly, Pageviews_hourly, Pageviews_daily - https://phabricator.wikimedia.org/T232659 (10Nuria) mmmm, you are right. Adding columns does not work, i just "refreshed metadat... [21:30:38] 10Analytics: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10Bstorm) I see the puppetization needs updating, I see. [21:34:40] 10Analytics, 10Analytics-Kanban, 10Product-Analytics (Kanban): Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10cchen) @Nuria can we add namespace field in this task as well? related ticket: https://phabricator.wikimedia.org/T237053 [21:40:54] 10Analytics, 10Analytics-Kanban, 10Product-Analytics (Kanban): Add new dimensions to virtual_pageview_hourly and pageview_hourly - https://phabricator.wikimedia.org/T243090 (10Nuria) @cchen: yes, it will come from the namespace provided on the schema: https://meta.wikimedia.org/wiki/Schema:VirtualPageView [21:53:13] milimetric: FYI we currently do use puppet git::clone to checkout the dist/ directory explicitly [21:53:32] HMM wait... [21:53:48] no we don't sorry [21:54:10] it is a symlink [21:54:11] sorry [22:00:09] milimetric: i like luca's most recent suggestion, I think it is easiest to reason about [22:00:16] https://phabricator.wikimedia.org/T237752#5785128 [22:00:27] have both v1 and v2 [22:00:28] dirs [22:00:32] move all of wikistats1 into v1 [22:00:53] have the v2 dir symlink to the dist/ dir of the wikistats 2 repo [22:00:56] (like it is now) [22:00:59] I think he liked saper's suggestion better than that, and the redirects were getting complicated [22:01:03] hm [22:01:19] and the team seemed to prefer the simpler approach of just putting it all in one folder [22:01:37] all in one folder? [22:01:50] reading more [22:08:24] i see, that is a 3rd solution [22:08:32] and that is better because everything is explicit [22:08:38] that seems a bit crazy though [22:08:43] sure the config is simple [22:08:49] but now the repos and files are all over [22:09:05] all the wikistats v1 stuff will be untracked files [22:09:07] if we use git anyway [22:09:17] we could use scap and post deploy rsync dist/ into htdocs [22:09:21] that'll make a big mess [22:09:23] but would work [22:10:04] 10Analytics, 10Patch-For-Review: "stale file handle" error on notebook1003 when trying to access /mnt/data - https://phabricator.wikimedia.org/T243328 (10Bstorm) 05Open→03Resolved a:03Bstorm `lang=shell-session [bstorm@notebook1003]:~ $ ls -al /mnt/data/xmldatadumps/public/enwiki/20200101/enwiki-20200101... [22:10:25] milimetric: my brain is on this and i'm not going anywhere atm, lemme know if you want more batcave time with otto [22:10:27] :) [22:11:25] ok ottomata let's cave for a bit [22:50:39] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10DerHexer) #cvn-unifications is intensively used by stewards to detect abusive user names. It's the easiest (if not only useful) source of globa... [23:04:12] (03CR) 10Nuria: "Also the prod build has issues:" (035 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [23:09:12] (03CR) 10Nuria: "This looks good and it is going to be very helpful going forward, have we tested the jobs?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565558 (https://phabricator.wikimedia.org/T238858) (owner: 10Joal) [23:15:55] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Schniggendiller) As a steward, I very much rely on `#cvn-unifications`. The SUL watcher bots report all new account creations there, with a li...