[00:53:55] (03CR) 10Urbanecm: [C: 03+1] Add nia.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650218 (https://phabricator.wikimedia.org/T270409) (owner: 10Gerrit maintenance bot) [00:54:00] (03CR) 10Urbanecm: [C: 03+1] Add nia.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650221 (https://phabricator.wikimedia.org/T270408) (owner: 10Gerrit maintenance bot) [00:54:10] (03CR) 10Urbanecm: [C: 03+1] Add diq.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/649869 (https://phabricator.wikimedia.org/T270275) (owner: 10Gerrit maintenance bot) [05:07:34] 10Analytics, 10Event-Platform: Schema compatibility check for changing event schemas fails when adding to the middle of an array - https://phabricator.wikimedia.org/T270470 (10Tgr) [07:27:15] Good morning [07:28:50] bonjour [07:38:22] 10Analytics: Switch off skipTrash for some data purging - https://phabricator.wikimedia.org/T270431 (10JAllemandou) [07:42:30] 10Analytics: Switch off skipTrash for some data purging - https://phabricator.wikimedia.org/T270431 (10JAllemandou) I think that jobs where skipTrash should be removed are the one applying purge to multiple sub-folders at once (events for instance). Jobs applying to well defined folder/datasets should probably k... [07:45:06] (03CR) 10Joal: [V: 03+2 C: 03+2] "merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650218 (https://phabricator.wikimedia.org/T270409) (owner: 10Gerrit maintenance bot) [07:45:55] (03PS2) 10Joal: Add diq.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/649869 (https://phabricator.wikimedia.org/T270275) (owner: 10Gerrit maintenance bot) [07:46:15] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/649869 (https://phabricator.wikimedia.org/T270275) (owner: 10Gerrit maintenance bot) [07:49:27] (03PS2) 10Joal: Add nia.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650221 (https://phabricator.wikimedia.org/T270408) (owner: 10Gerrit maintenance bot) [07:49:53] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650221 (https://phabricator.wikimedia.org/T270408) (owner: 10Gerrit maintenance bot) [08:12:49] joal: ok so big issue, /mnt/hdfs doesn't work with the credentials in /run/user/etc.. [08:13:01] aouch :( [08:15:45] ah no wait now it works [08:15:46] whattt [08:15:58] MEH? [08:17:02] false alarm then :) [08:17:12] I found some sneaky corner cases for kerberos-run-command though [08:17:28] nothing huge but I'd need to have a chat with Moritz on Monday about them [08:40:45] https://doordash.engineering/2020/11/19/building-a-gigascale-ml-feature-store-with-redis/ [08:40:49] interesting :) [08:41:53] " one of our high volume use cases, store ranking, makes more than one million predictions per second and uses dozens of features per prediction. " [09:33:51] all right we are refining webrequest etc.. on test again [09:33:54] (with CDH) [09:40:29] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Kerberos credential cache location - https://phabricator.wikimedia.org/T255262 (10elukey) [10:57:37] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Kerberos credential cache location - https://phabricator.wikimedia.org/T255262 (10elukey) The current state of the deployment is the following: * stat1004 is the only host with the new credential cache location under `/run/user/$uid/krbcc` * kerbe... [11:01:34] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Kerberos credential cache location - https://phabricator.wikimedia.org/T255262 (10elukey) As follow up, what I'd do is something like the following: 1) create a dir (via puppet) called /run/kerberos (on tmpfs) 2) set the krb5.conf default ccache... [11:22:41] http://airflow.apache.org/blog/airflow-two-point-oh-is-here/ [11:22:43] joal: --^ [11:40:45] * elukey lunch! [12:14:15] Yeah elukey! I saw that yesterday :) [13:53:28] isaacj: o/ [13:53:40] I fear that I have to rollback the settings on stat1004 [13:53:47] the solution is not that great [14:01:14] elukey: no worries, so expect to kinit on swap and normal ssh separately again? [14:01:33] isaacj: yeah I'd also need to stop your notebook [14:01:44] i'll go in and do that right now [14:02:00] but I should have all use cases now to test in hadoop-test env, before re-trying on stat1004 [14:02:10] thanks a lot [14:04:12] actually weirdly i'm not seeing any notebooks as running in the SWAP interface on stat1004 so maybe just kill it for me if there's one [14:05:11] i think the only one was on stat1008 and i just killed that one [14:06:36] also elukey it felt like the kinit was expiring much more quickly than i expected (i'd have to re kinit each morning even when i had done it only the day before). does it automatically expire before the 2 days if there aren't notebooks running or if i kinit on another server or something like that? [14:09:14] isaacj: interesting, this is a really great feedback! So the new location of the credential cache for kerberos is under /run/user/$uid/etc.., that is managed by systemd.. I am pretty sure that is auto-cleaned up, so this is probably why you see this behavior [14:09:21] I'll take it into account to find a better solution [14:09:39] I know it seems straightforward but finding something that works for all the tools etc.. is not easy :( [14:09:43] sorry for the noise [14:10:57] !log restore stat1004 to its previous settings for kerberos credential cache [14:11:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:15:31] (03PS6) 10Awight: Aggregate TemplateWizard metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/649351 (https://phabricator.wikimedia.org/T262209) [14:17:57] (03CR) 10Awight: "PS 6:" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/649351 (https://phabricator.wikimedia.org/T262209) (owner: 10Awight) [14:18:06] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Kerberos credential cache location - https://phabricator.wikimedia.org/T255262 (10elukey) I just rolledback the state on stat1004, kept the settings only for an-test-client1001. The `kerberos-run-command` script has also been reverted to its origi... [14:31:09] no problem elukey and thanks for working on all these upgrades! [14:57:39] o/ [14:57:44] hi everyone [15:00:03] good morning :) [15:09:42] hello! [15:10:41] gooood morning :) [15:12:20] ottomata: so to complete the list of things with sad_trombone.wav, I found some weird use cases for the shared kerberos credential cache :( [15:12:36] but we are getting closer [15:13:46] 10Analytics-Clusters, 10Operations, 10ops-eqiad: an-presto1004 shows only the NIC in the boot list - https://phabricator.wikimedia.org/T268951 (10Cmjohnson) The part did not arrive in time on Monday for the tech to get here and then a snow/ice storm delayed the tech. We rescheduled this for this coming Mond... [15:32:11] 10Analytics: Switch off skipTrash for some data purging - https://phabricator.wikimedia.org/T270431 (10Ottomata) Agree! I think this would then be: ` /wmf/data/raw/eventlogging /wmf/data/raw/event /wmf/data/event /wmf/data/event_sanitized [15:36:27] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/649351 (https://phabricator.wikimedia.org/T262209) (owner: 10Awight) [15:43:11] razzi: o/ good morning [15:43:23] Hi elukey o/ [15:43:38] one qs - did we deploy the new version of superset with the memcache dep? If not, how does it work on staging? [15:44:47] yeah, I deployed https://gerrit.wikimedia.org/r/c/analytics/superset/deploy/+/647387 to staging [15:45:01] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/649861 (https://phabricator.wikimedia.org/T270246) (owner: 10Andrew-WMDE) [15:46:03] razzi: ahh okok then good :) Remember to add an-tool1010 to the scap config in superset too [15:46:14] so you'll be able to deploy to an-tool1010 if needed [15:52:43] (03PS3) 10Razzi: Install pylibmc and update wheels for superset [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 [15:53:28] (03CR) 10Mforns: [C: 03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/650034 (https://phabricator.wikimedia.org/T257412) (owner: 10Joal) [15:57:18] 10Analytics: Presto error in Superest - only when grouping - https://phabricator.wikimedia.org/T270503 (10EYener) [15:57:38] (03PS4) 10Razzi: Install pylibmc and update wheels for superset [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) [15:58:21] (03CR) 10Ottomata: [C: 03+1] Install pylibmc and update wheels for superset [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:03:15] mforns: o/ [16:03:23] going to run the ru job on launcher ok? [16:05:01] 10Analytics-Clusters, 10Patch-For-Review: Move Superset and Turnilo to an-tool1010 - https://phabricator.wikimedia.org/T268219 (10razzi) For superset, the following 3 patches should be all we need to move traffic over with a short window of downtime: - Update deployment settings for superset to run on an-tool... [16:11:35] (03CR) 10Elukey: [C: 04-1] "There is a new "target" file, it should be into the scap dir in theory.." [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:14:22] (03CR) 10Elukey: [C: 04-1] "Also let's keep the scap changes separate if possible, in another separate review (so it is easier to check etc..). Given the number of fi" [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:16:02] 10Analytics: Switch off skipTrash for some data purging - https://phabricator.wikimedia.org/T270431 (10mforns) Also agree! [16:28:04] (03CR) 10Razzi: "> Patch Set 4: Code-Review-1" [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:29:12] (03PS5) 10Razzi: Install pylibmc and update wheels for superset [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) [16:30:45] (03CR) 10Elukey: [C: 03+1] Install pylibmc and update wheels for superset (031 comment) [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:32:47] (03PS1) 10Razzi: Add an-tool1010.eqiad.wmnet to scap/targets [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/650526 (https://phabricator.wikimedia.org/T268219) [16:37:41] hiyaaa mforns [16:37:54] hey ottomata :] [16:38:00] on my EL patch to add a warningbox to migrated pages, Ori suggested we just edit protect the page on metawiki instead [16:38:15] aha [16:38:19] makes sense [16:38:21] i thought that was a pretty good and simple idea, but to do that we need admin rights on metawiki [16:38:27] i just submitted a request for that at [16:38:27] O.o [16:38:28] https://meta.wikimedia.org/wiki/Meta:Requests_for_adminship#Requests_for_limited_adminship [16:38:33] you should probably do the same [16:38:33] ok :] [16:38:46] I requested limited adminship for 1 year [16:38:47] https://meta.wikimedia.org/wiki/Meta:Requests_for_limited_adminship/Ottomata [16:42:56] (03CR) 10Razzi: Install pylibmc and update wheels for superset (031 comment) [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) (owner: 10Razzi) [16:44:53] (03PS6) 10Razzi: Install pylibmc and update wheels for superset [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/647387 (https://phabricator.wikimedia.org/T268219) [16:48:19] (03CR) 10Joal: [V: 04-1 C: 04-1] "Woops - This actually fails testing - I shouldn't have +1ed this my bad" (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/649884 (https://phabricator.wikimedia.org/T268809) (owner: 10Fdans) [16:48:50] o/ hey everyone, I am done for this year. Have great holidays (quiet or busy, whatever you prefer), and see ya in 2021. [16:48:58] bye tobias see you flipside! :) [16:56:51] ah mforns [16:56:55] perhaps https://office.wikimedia.org/wiki/WMF_Staff_userrights_policy is the proper process [16:58:25] ok, will send an email [17:01:05] mforns: [17:01:09] i'll send one for the both of us [17:01:13] * elukey afk for ~30 mins! [17:01:22] ok ottomata, thanks! [17:04:00] mforns: what is your 'work username'? [17:04:05] Mforns_WMF [17:04:07] ? [17:04:22] ottomata: for meta it's: Mforns (WMF) [17:04:36] with a space in between [17:04:55] and parentheses [17:10:18] great, sent [17:11:16] thanks! [17:15:08] (03CR) 10Mforns: [C: 03+1] "@joal This has 2 +1s, it can be merged no?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638040 (owner: 10Joal) [17:15:27] joal: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/638040 <- ok to merge? [17:15:35] looking mforns [17:16:53] mforns: Ok theoretically - I have not tested any job with the new jars so... The most tricky one in term of dependencies IIRC is Refine - I should test at least that one before deployingn [17:17:40] joal: ok, don't need to do that now, I was just checking for hanging Gerrit changes! But if you want me to deploy that, just let me know [17:18:01] Ack mforns - Thanks for pointing :) [17:18:40] np! [17:19:25] btw ottomata, next week, should I take the corresponding schema migrations? [17:58:52] (03CR) 10Ladsgroup: [C: 03+2] "> Patch Set 1:" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/649676 (owner: 10Lucas Werkmeister (WMDE)) [17:59:56] (03Merged) 10jenkins-bot: Send accurate timestamp with lexeme statistics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/649676 (owner: 10Lucas Werkmeister (WMDE)) [18:01:41] (03CR) 10Ladsgroup: Reduce duplicate code in lexeme statistics script (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/649710 (owner: 10Lucas Werkmeister (WMDE)) [18:02:02] (03PS1) 10Ladsgroup: Send accurate timestamp with lexeme statistics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/650362 [18:04:29] 10Analytics-Clusters, 10Product-Analytics: Configure superset cache - https://phabricator.wikimedia.org/T268784 (10razzi) Quick poll: what should the default caching timeout be? I'm thinking 12 hours, since it seems most charts have daily granularity, so viewing a chart one day and then the next day will show... [18:08:27] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFirstRun Event Platform Migration - https://phabricator.wikimedia.org/T267346 (10nshahquinn-wmf) >>! In T267346#6606752, @Ottomata wrote: > @nshahquinn-wmf > Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as par... [18:09:15] 10Analytics, 10Event-Platform, 10Inuka-Team: InukaPageView Event Platform Migration - https://phabricator.wikimedia.org/T267344 (10nshahquinn-wmf) >>! In T267344#6606750, @Ottomata wrote: > @nshahquinn-wmf > Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as part... [18:10:06] 10Analytics-Clusters, 10Operations, 10ops-eqiad: an-presto1004 shows only the NIC in the boot list - https://phabricator.wikimedia.org/T268951 (10Cmjohnson) 05Open→03Resolved a:03Cmjohnson Dell tech arrived today, swapped the raid controller. All disks are now online. resolving [18:11:08] 10Analytics, 10Event-Platform, 10Inuka-Team: KaiOSAppFeedback Event Platform Migration - https://phabricator.wikimedia.org/T267345 (10nshahquinn-wmf) >>! In T267345#6606751, @Ottomata wrote: > @nshahquinn-wmf > Let us know if this schema needs client IP and/or geocoded data? If not, it will be removed as par... [18:11:17] 10Analytics, 10Event-Platform, 10Language-analytics: UniversalLanguageSelector Event Platform Migration - https://phabricator.wikimedia.org/T267352 (10nshahquinn-wmf) >>! In T267352#6606759, @Ottomata wrote: > @nshahquinn-wmf > Let us know if this schema needs client IP and/or geocoded data? If not, it will... [18:11:49] wow an-presto1004 up again! [18:15:01] ottomata: the inuka schemas can be migrated? do the apps clients support EventGate? Does the KaiOS app? [18:15:32] (03CR) 10Ladsgroup: [C: 03+2] Send accurate timestamp with lexeme statistics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/650362 (owner: 10Ladsgroup) [18:16:29] 10Analytics, 10Product-Analytics, 10Inuka-Team (Kanban): Set up preview counting for KaiOS app - https://phabricator.wikimedia.org/T244548 (10nshahquinn-wmf) >>! In T244548#6699011, @Ottomata wrote: > Oh, I see this is for VirtualPageView, which makes what I said more complicated. :) Because VirtualPageView... [18:16:56] (03Merged) 10jenkins-bot: Send accurate timestamp with lexeme statistics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/650362 (owner: 10Ladsgroup) [18:17:17] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) @Ottomata when I tested topicmappr before, I uploaded the binary directly onto the host; when we do this in production, will it make sense to debianize ht... [19:45:19] 10Analytics, 10Event-Platform: Schema compatibility check for changing event schemas fails when adding to the middle of an array - https://phabricator.wikimedia.org/T270470 (10Tgr) [20:01:37] Gone for tonight team - see you next week! [20:16:49] 10Analytics-Clusters, 10Analytics-Kanban: Deprecate the 'researchers' posix group - https://phabricator.wikimedia.org/T268801 (10matmarex) I still need access, please move me to 'analytics-privatedata-users'. [21:03:16] 10Analytics-Clusters, 10Product-Analytics: Configure superset cache - https://phabricator.wikimedia.org/T268784 (10Ottomata) 12 hours sounds like a good start, lets try it! [21:06:51] 10Analytics, 10Product-Analytics, 10Inuka-Team (Kanban): Set up preview counting for KaiOS app - https://phabricator.wikimedia.org/T244548 (10Ottomata) It has special case behavior (it doesn't use the EventLogging extension to send the events), so it will be one of the last schemas migrated. It may be pos... [21:08:05] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10Ottomata) Hm, topicmappr just helps in generating the reassignment.json file, right? I think we can use it as a one off tool to generate the reassignment.json f... [23:59:57] (03CR) 10Fdans: [C: 03+2] Add Catalan and Greek to Wikistats languages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/650259 (owner: 10Fdans)