[07:11:01] 10Analytics, 10Operations, 10ops-eqiad: analytics1057's BBU is faulty - https://phabricator.wikimedia.org/T239045 (10elukey) [08:10:24] Good morning :) [08:17:00] 10Analytics: Check home leftovers of maxsem - https://phabricator.wikimedia.org/T239047 (10MoritzMuehlenhoff) [08:19:10] o/ [08:28:40] 10Analytics: Check home leftovers of maxsem - https://phabricator.wikimedia.org/T239047 (10elukey) ` ====== stat1004 ====== total 0 ls: cannot access '/var/userarchive/maxsem.tar.bz2': No such file or directory ====== stat1005 ====== total 0 ls: cannot access '/var/userarchive/maxsem.tar.bz2': No such file or d... [08:39:59] 10Analytics: Check home leftovers of maxsem - https://phabricator.wikimedia.org/T239047 (10elukey) @aezell hi! Hope to reach out to the right person.. Max left some files (summary above) on the Analytics stat hosts, and usually before deleting anything we ask to somebody a quick review to see if something needs... [08:41:55] joal: bonjour! I found the original task for HDFS where they added AES encryption for the datanode protocol - https://issues.apache.org/jira/browse/HDFS-6606 [08:41:58] very interesting [08:45:13] Nice elukey [09:47:24] Still trying to work on snakebite but I hit another problem, namely the puresasl lib doesn't support encryption for digest-md5, and I am afraid it is required [09:47:49] tried to make python-sasl work (that is a cloudera wrapper around libsasl) but it doesn't work even with the namenode protocol [09:47:58] some gssapi failure that I cannot explain [09:47:59] sigh [10:04:44] joal: did you see my comments for https://phabricator.wikimedia.org/T238432 ? [10:05:33] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) [10:11:06] elukey: I'm getting ready to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/544989, last chance to object [10:11:34] elukey: in particular, we are repurposing an-airflow1001.eqiad.wmnet (which was a spare) [10:11:41] I did not follow that conversation much [10:17:32] hexmode: o/ - an-airflow is a vm that I created in the analytics vlan, is waiting for the patch, please proceed :) [10:17:39] ouch sorry hexmode [10:17:42] gehel: --^ [10:18:05] there is also a task for perms: https://phabricator.wikimedia.org/T238905 [10:18:08] elukey: ok, thanks! [10:18:19] when you have time let's discuss if you like the idea or not [10:18:30] Daniel proposed a plan that I think makes sense [10:18:42] I'd only renamed the group to airflow-search-admins [10:18:53] * gehel is reading [10:22:09] and rebase it on top of that main airflow patch [10:22:37] yep [10:23:41] gehel: if you like the idea I can amend Erik's patch and ask to the SRE team this evening if it is ok to merge [10:23:51] so you guys will not have to wait another week [10:23:58] sounds like a good plan! thanks! [10:24:05] super will do :) [10:45:38] 10Analytics, 10Analytics-EventLogging, 10QuickSurveys, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), and 2 others: QuickSurveys EventLogging missing ~10% of interactions - https://phabricator.wikimedia.org/T220627 (10ovasileva) 05Open→03Resolved Resolving this. Thanks to everyone that spent time analyzi... [11:28:58] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) Note: this is only a proposal after the first reading of the puppet code, only meant to gather feedback, nothing stated in stone. Comme... [11:38:58] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) >>! In T234826#5688880, @elukey wrote: > Note: this is only a proposal after the first reading of the puppet code, only meant to ga... [11:44:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) >>! In T234826#5688919, @Marostegui wrote: >>>! In T234826#5688880, @elukey wrote: >> Note: this is only a proposal after the first rea... [11:52:25] * elukey lunch! [13:27:37] !log set global read_only=1 on db1108's log database [13:27:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:38:31] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10Patch-For-Review, and 2 others: Track WDQS updater UA in wikidata-special-entitydata grafana dashboard - https://phabricator.wikimedia.org/T218998 (10Lydia_Pintscher) \o/ Good from my side. Leaving for @Addshore to close. [13:45:06] 10Analytics, 10Event-Platform, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Type collisions in log events causing indexing failures in ELK Elasticsearch - https://phabricator.wikimedia.org/T150106 (10fgiunchedi) We're now alerting when logstash index failures happen (T236343) thus I'm boldl... [13:45:45] 10Analytics, 10Event-Platform, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Type collisions in log events causing indexing failures in ELK Elasticsearch - https://phabricator.wikimedia.org/T150106 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi [14:01:33] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10elukey) @Krinkle thanks for the links, I have a better picture now. I can see three layers of technical debt in here: 1) There are super useful bots, once written by community... [14:16:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) >>! In T234826#5688951, @elukey wrote: >>>! In T234826#5688919, @Marostegui wrote: >>>>! In T234826#5688880, @elukey wrote: >>> Not... [14:20:56] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10Patch-For-Review, and 2 others: Track WDQS updater UA in wikidata-special-entitydata grafana dashboard - https://phabricator.wikimedia.org/T218998 (10Addshore) 05Open→03Resolved [14:27:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) >>! In T234826#5689742, @Marostegui wrote: >>>! In T234826#5688951, @elukey wrote: >>>>! In T234826#5688919, @Marostegui wrote: >>>>>!... [14:44:59] I am restarting our zookeeper cluster for jvm upgrades [15:10:10] helloooo teammm [15:32:46] mforns: hola! [15:32:53] elukey, hey :] [15:52:16] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10elukey) [16:01:23] ottomata: ping standup :) [16:19:04] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Jdforrester-WMF) [16:28:28] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Ottomata) > new software component will be created, to bridge the current bot interface (IRC) with a more recent and simple one (EventStreams). +1! I don't know if using 'Event... [16:29:03] ping ottomata groskin [16:29:51] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Wikimedia-Stream: Eliminate usage of mocha-eslint from eventstreams - https://phabricator.wikimedia.org/T238937 (10Ottomata) [16:35:20] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10Anomie) Apparently to keep using the access I was approved for in {T155143}, I need to make some sort of request here? Username is still `anomie`. >>! At https://wikitech.wikimedia.org/wiki/Ana... [17:04:38] 10Analytics: Check home leftovers of maxsem - https://phabricator.wikimedia.org/T239047 (10mforns) p:05Triage→03High [17:05:00] 10Analytics, 10Operations, 10ops-eqiad: analytics1057's BBU is faulty - https://phabricator.wikimedia.org/T239045 (10mforns) p:05Triage→03High [17:05:23] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Nuria) Seems that this might be using m.track() from the client which sends data straight to statsd? [17:06:44] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Wikimedia-Stream: Eliminate usage of mocha-eslint from eventstreams - https://phabricator.wikimedia.org/T238937 (10mforns) p:05Triage→03Normal [17:08:16] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Jdforrester-WMF) Yes, this isn't an EventLogging stats system, it's straight into statsd (or statsv? I forget which is which). [17:08:48] 10Analytics: Legend for 12 months is not correct in graphs of analytics - https://phabricator.wikimedia.org/T238894 (10mforns) [17:08:50] 10Analytics: Wikistats2 time related bugs - https://phabricator.wikimedia.org/T231248 (10mforns) [17:09:01] 10Analytics: Legend for 12 months is not correct in graphs of analytics - https://phabricator.wikimedia.org/T238894 (10mforns) p:05Triage→03Normal [17:12:38] 10Analytics: Update wikitext-processing on hadoop various aspects - https://phabricator.wikimedia.org/T238858 (10mforns) p:05Triage→03Normal [17:16:45] 10Analytics: Import slots table through scoop - https://phabricator.wikimedia.org/T239127 (10Nuria) [17:17:56] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Nuria) loading times look a bit worrisome: https://grafana.wikimedia.org/d/000000094/visualeditor-load-save?orgId=1&from=now-7d&to=now-3m&panelId=14&fullscreen [17:19:07] hip: hello! did the code patches for error logging went in, if so can you subscribe me to those changes? [17:28:11] 10Analytics: Superset getting slower as usage increases - https://phabricator.wikimedia.org/T239130 (10Nuria) [17:29:11] 10Analytics: [Spike] Spark job for digests-only mediawiki-history-reduced - https://phabricator.wikimedia.org/T212928 (10mforns) [17:30:04] 10Analytics, 10Analytics-Kanban: mediawiki-history-wikitext-coord job fails every month - https://phabricator.wikimedia.org/T228883 (10mforns) 05Open→03Resolved Seems this job is working lately! Let's resolve this task and re-open later if needed. [17:33:10] 10Analytics: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10mforns) [17:33:35] 10Analytics: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10mforns) So, it seems Hue doesn't run in python3. What's the next step here? [17:35:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add MaxMind DB files on an-coord1001 for hive local-jobs using UDF to succeed - https://phabricator.wikimedia.org/T238432 (10mforns) a:05elukey→03JAllemandou [17:38:53] 10Analytics, 10Analytics-Kanban: Add Sakizaya Wikipedia to analytics setup - https://phabricator.wikimedia.org/T237378 (10mforns) [17:39:28] 10Analytics, 10Cloud-Services, 10Discovery, 10Discovery-Search, 10Elasticsearch: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135 (10Pchelolo) [17:40:01] ottomata: elukey hi! would you mind creating a couple of topics with musltiple partitions in kafka when you have time? T239135 [17:40:01] T239135: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135 [17:40:21] Pchelolo: o/ [17:40:28] kafka main you mean? [17:40:31] yeah [17:40:38] ah okok! [17:40:52] I could do it myself, but I figured better ask the professional :) [17:40:54] Andrew is probably already out, I can try to create them in a bit [17:40:57] hahahaha [17:41:18] if you can do it please just add the commands in the task, I can quickly code review them and unblock you [17:41:22] what do you think? [17:41:40] 10Analytics: Revise wiki scoop list from labs once a quarter - https://phabricator.wikimedia.org/T239136 (10Nuria) [17:42:03] 10Analytics, 10Analytics-Kanban: Add Sakizaya Wikipedia to analytics setup - https://phabricator.wikimedia.org/T237378 (10mforns) 05Open→03Resolved a:03mforns Part of this task is done. The other part will be tackled by another task: T239136 [17:42:41] 10Analytics, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Ottomata) Qs: - Do we need to support full IRC spec? I see in [[ https://gist.github.com/paravoid/3419e0b5ae1f24b6ea21906a142f2f47 | Faidon's PoC ]] he's attempting to at leas... [17:43:13] elukey: oki [17:45:25] 10Analytics: Archive docs for old Wikistats and update links to Wikistats 2 - https://phabricator.wikimedia.org/T238001 (10mforns) [17:47:28] 10Analytics, 10Cloud-Services, 10Discovery, 10Discovery-Search, 10Elasticsearch: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135 (10Pchelolo) I believe these should do it? On eqiad: ` kafka topics --create --partitions 3 --replication-factor 3 eqiad.cpjobqu... [17:58:11] 10Analytics, 10Cloud-Services, 10Discovery, 10Discovery-Search, 10Elasticsearch: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135 (10elukey) >>! In T239135#5690582, @Pchelolo wrote: > I believe these should do it? > > On eqiad: > ` > kafka topics --create --... [17:58:38] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Jdforrester-WMF) >>! In T239121#5690306, @Nuria wrote: > loading times look a bit worrisome: https://grafana.wikimedia.org/d/000000094/visualeditor-load-save?orgId=1&from=... [18:00:09] 10Analytics, 10Cloud-Services, 10Discovery, 10Discovery-Search, 10Elasticsearch: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135 (10Pchelolo) > DISCLAIMER: I assume that the .partitioned. name was discussed and decided, don't want to get into the naming bike... [18:02:31] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10elukey) >>! In T237605#5685440, @Pcoombe wrote: > Hello, requesting Kerberos credentials for Hadoop access on stat100x and notebook100x. My username is `pcoombe`. Thanks! Done! Please check you... [18:04:32] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10elukey) >>! In T237605#5690138, @Anomie wrote: > Apparently to keep using the access I was approved for in {T155143}, I need to make some sort of request here? Username is still `anomie`. > >>>... [18:05:24] Pchelolo: all good, as I wrote in the task I assume that the name of the topics was already decided etc.. [18:05:57] yeag man, bikeshedding train has sailed [18:06:09] oh. trains don't sail.. [18:06:16] you know what I mean :) [18:08:15] ahhaha [18:08:19] sure sure [18:20:03] ebernhardson: o/ - do you have a bit of time for airflow? [18:20:14] if you want we can start merging some stuff [18:20:52] elukey: sure [18:21:19] I am merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552613/ [18:21:26] that creates airflow-search-admins [18:21:44] (got approved today by the SRE meeting) [18:21:50] excellent [18:23:49] once jenkins merges i'll update the other patch [18:24:00] ebernhardson: first question is about https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/544996/ [18:24:24] since it has your /home/etc.. stuff inside [18:24:37] can it be integrated in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/544989/? [18:24:42] (brb) [18:25:07] elukey: the thing is i want the git history to include what was changed from upstream, it's been tedious in other cases like elasticsearch.yml to upgrade and not know what exactly we changed from upstream [18:25:25] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check if a GPU fits in any of the remaining stat or notebook hosts - https://phabricator.wikimedia.org/T220698 (10RobH) So, I advise against any guesswork on this. If you want to know if these 5 servers will hold a GPU, each purchase group needs to be... [18:25:26] elukey: and for whatever reason, airflow "auto-magics" a config in place, which is where it gets my home dir [18:26:26] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check if a GPU fits in any of the remaining stat or notebook hosts - https://phabricator.wikimedia.org/T220698 (10RobH) Please note we likely need to schedule downtime for each of those 4 hosts to shutdown and check them. @elukey: Can you advise how m... [18:29:52] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check if a GPU fits in any of the remaining stat or notebook hosts - https://phabricator.wikimedia.org/T220698 (10elukey) a:05Cmjohnson→03RobH @RobH if we could go one at the time I think that a day before the maintenance is sufficient, I'll take c... [18:30:57] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Check if a GPU fits in any of the remaining stat or notebook hosts - https://phabricator.wikimedia.org/T220698 (10RobH) a:05RobH→03Cmjohnson Please note that Chris will still be performing this, it needs to stay assigned to him. He will be coordin... [18:36:02] ebernhardson: ack, waiting for ci and then I'll merge the first two [18:45:24] ebernhardson: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552872/ [18:47:31] ah more missing [18:54:06] <3 [18:55:49] elukey: sorry was just finishing up a meeting, looking [18:57:05] ebernhardson: np! [18:57:12] I found more weird bugs [18:57:18] in other places (not your code) [18:57:26] I am running puppet now [18:58:43] will take a bit (all hadoop packages) [19:09:32] updated that patch, i think all it needs to do now is: a) allow login by airflow-search-admins b) fix typo in sudo that had airflow-scheduler twice instead of airflow-scheduler and airflow-websrver [19:10:27] yes I saw that sorry [19:10:32] still fixing some bugs [19:10:42] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552878 [19:10:52] I can also see [19:10:52] 19:05:27 ERROR - deploy-local failed: [Errno 2] Config file not found: 'deploy1001.eqiad.wmnet/search/airflow/.git/DEPLOY_HEAD' [19:11:05] is the repo there? [19:11:10] doh, for some reason i thought puppet magic'd file and perm requires into place [19:11:29] elukey: oh, this is the first deploy of that repo. So it probably needs puppet run on deploy1001 [19:11:36] makes sense [19:15:25] so scap deploy --init fails even on deploy1001 [19:16:09] ebernhardson: I don't recall what needs to be done when you init a repo [19:16:12] there is some magic [19:17:12] elukey: hmm, i don't remember any magic but lemme check, it's been most of a year since i last deployed a new repo.. [19:17:31] https://wikitech.wikimedia.org/wiki/Scap3/Migration_Guide#First_Deployment seems fine [19:17:46] mmmm [19:18:38] hmm, it created the dir but no git repo, which it seems to expect. hmm [19:19:58] is the scap config ok in the repo? [19:20:03] (thinking out loud) [19:20:34] elukey: so ... turns out scap.cfg still says analytics/airflow instead of search/airflow [19:20:37] maybe that will fix it, sec [19:20:44] yeah [19:21:39] analytics_dsh_targets will need to add an-airflow1001 as well, or a new dsh group [19:21:54] (although that probably isn't current error, but will come up soon) [19:24:15] hmm, probably should be new dsh group...care about the naming? search-airflow is probably reasonable? [19:25:28] yeah I think so [19:25:41] but wait, will it need new keys for keyholder? [19:26:05] hope not, don't remember now the user that we are using [19:26:31] ah deploy-service ok [19:26:38] it was using deploy-service, yea [19:29:27] ebernhardson: just to be clear, are you about to send a code change or should I do something? [19:29:48] elukey: just sent em [19:29:56] elukey: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552885 for dsh [19:30:02] and i just merged the scap.cfg fix to search/airflow [19:30:54] for ref, scap.cfg fix is https://gerrit.wikimedia.org/r/#/c/search/airflow/+/552883/ [19:32:15] not sure that fixes it, since the repo didn't even clone on deploy1001, but maybe [19:34:55] yeah still broken [19:37:03] hmm [19:37:55] seems puppet logs are root only and not in logstash, so not sure what to go off of :S [19:38:25] the error ok deplo1001 is still [19:38:26] OSError: [Errno 2] No such file or directory: '/srv/deployment/search/airflow/.git/config-files' [19:38:29] 19:34:19 ERROR - deploy failed: [Errno 2] No such file or directory: '/srv/deployment/search/airflow/.git/config-files' [19:38:46] hmm, there must be something that fails to clone earlier [19:39:15] maybe it's gated on directory existing, and something created /srv/deployment/search/airflow/scap/log, checking [19:41:08] I am wondering if we need to clone the repo [19:43:10] seems to end up at scap_source resource defined in ruby, i'm not particularly familiar with ruby in puppet though so still figuring out what it does [19:46:19] afaict the scap_source provider should work with dir exists but no clone, it checks that condition explicitly in exsits? [19:47:57] ebernhardson: rm -rf + puppet agent -tv worked [19:48:10] elukey: ahh, ok. I was about to link this which suggests the same: https://github.com/wikimedia/puppet/blob/production/modules/scap/lib/puppet/provider/scap_source/default.rb#L163 [19:48:16] crazy [19:48:39] they have a separate exists check which probably gates if create tries to run [19:51:58] now it says fatal: repository 'http://deploy1001.eqiad.wmnet/search/airflow/.git/' not found [19:52:01] sigh [19:53:00] http? sigh ... lemme check [19:53:18] the dir is there [19:54:03] so wget http://deploy1001.eqiad.wmnet/search/airflow/.git works [19:54:23] huh, i thought that was all via ssh. hmm [19:56:16] mmmmmmmm what if it is related to git proxy [19:57:41] doesn't seem so [19:59:22] ebernhardson: if not super urgent I'll keep working on it tomorrow morning, seems that it will take a bit [20:00:13] elukey: sure, i'll look over this. I think i found this specific problem (per git "dumb http" it must respond to .git/info/refs, but that doesn't exist here [20:00:39] ah ok! Please write in here so I'll read tomorrow :) [20:00:41] thanks! [20:00:46] * elukey off to dinner :) [20:05:16] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10nettrom_WMF) Like so many others, I'd like to request my credentials for access on stat100x and notebook100x. My username is `nettrom`. I'll keep an eye out on my Gmail spam folder as well, chee... [20:08:42] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10EBernhardson) I'd like to request two credentials for hadoop access: ebernhardson: typical personal usage, ebernhardson@wikimedia.org discovery-analytics: We run analytics jobs (submit to ooz... [20:26:51] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10AndyRussG) @elukey @Nuria Thanks so much!!!!!!!!!!!!!!!!!!!!!!!! Thanks also @Dstrine, @mepps, @EYener , @Ejegg, @spatton! :) [20:45:31] 10Analytics, 10Product-Analytics, 10SDC General, 10Wikidata: Data about how many file pages on Commons contain at least one structured data element - https://phabricator.wikimedia.org/T238878 (10matthiasmullie) I don't really know much about how `mediawiki_history` gets populated, but `wbcreateclaim` & `wb... [21:04:40] 10Analytics, 10Editing-team, 10Performance-Team: VE edit data stopped at ~2019-11-24Z01:25 - https://phabricator.wikimedia.org/T239121 (10Krinkle) >>! In T239121#5690306, @Nuria wrote: > loading times look a bit worrisome: https://grafana.wikimedia.org/d/000000094/visualeditor-load-save?orgId=1&from=now-7d&t... [21:09:48] elukey: tl/dr: `scap deploy init` on deploy1001 creates that file and clone should owrk now [21:21:55] 10Analytics, 10Analytics-Kanban: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Nuria) For all who use the analytics cluster: Please get your hadoop credentials: https://phabricator.wikimedia.org/T237605 [21:49:54] 10Analytics, 10Operations, 10ops-eqiad: analytics1057's BBU is faulty - https://phabricator.wikimedia.org/T239045 (10wiki_willy) a:03Jclark-ctr [22:08:07] 10Analytics, 10Operations, 10ops-eqiad: analytics1057's BBU is faulty - https://phabricator.wikimedia.org/T239045 (10Jclark-ctr) @elukey No spare bbu around [22:12:14] 10Analytics, 10Analytics-Kanban: Create kerberos principals for users - https://phabricator.wikimedia.org/T237605 (10Anomie) >>! In T237605#5690652, @elukey wrote: > I will not comment the friendly introduction to ask for the credentials. If that was meant to criticize my show of minor annoyance at the necess...