[00:17:09] 10Analytics-Clusters: Reduce manual kinit frequency on stat100x hosts - https://phabricator.wikimedia.org/T268985 (10nettrom_WMF) This would be awesome! Is there a way to do this for Jupyter as well, since `kinit` within Jupyter is distinct from outside of it? [06:08:49] PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:19:33] RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:55:06] Good morning [07:56:30] bonjour [07:58:59] I have rebuilt hive with the patch in https://github.com/apache/hive/commit/84e107b37e4989eb4131f8ca179dcd407aa8718d [07:59:35] created a table in my database, but I can't add subcol to the struct with a simple hive session [07:59:48] basically I tried [07:59:49] set hive.metastore.disallow.incompatible.col.type.changes=false; [08:00:03] alter table yada yada and at the end a new "test" param [08:00:07] err col sorry [08:03:41] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) > So, please let us know if you're OK with reducing to 60 or you'd rather keep the 90. OK! > Would you guys be wil... [08:08:04] Good morning to elukey only [08:09:01] Amir1: good morning! :D [08:11:03] elukey: this is sad :) [08:11:10] :( sorry (wrong parenthesis) [08:11:47] joal: I am surely missing some patch or something else, will keep working on it :) [08:12:12] Ok elukey - don't sweat too much on it - having it set by default would be very ok :) [08:26:55] joal: I was tempted to restart hive to pick up the DBTokenStore stuff but it is the first of the month sooooo [08:27:03] I'll dedicate time to something else :D [08:27:07] :) [08:27:11] Thanks for that [08:30:34] Amir1: deployed and checked that it was a no-op everywhere, thanks :) [08:31:03] elukey: thanks. The hadoop modules seem to be used everywhere [08:31:11] there are lots of it left [08:31:54] yep I know we have a task under "Tech debt" to fix it, and you are helpin [08:32:01] *helping a lot, so I am happy :) [08:33:45] ^^ [09:03:16] !log clean up old hive metastore/server old logs on an-coord1001 to free space [09:03:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:56:43] Meh - Sqoop of image table for commons failed due to int/long type error [09:57:43] I managed to make it work manually, I'll send a patch later today for next run, and I hope this error will be the only one [10:03:21] joal: do we need to do any follow up? [10:03:28] (other than the one that you did) [10:19:33] 10Analytics, 10Product-Analytics, 10Inuka-Team (Kanban): Set up preview counting for KaiOS app - https://phabricator.wikimedia.org/T244548 (10hueitan) `Inuka Team` progress PR is here https://github.com/wikimedia/wikipedia-kaios/pull/286 [10:34:12] joal: ok if I restart oozie to pick up log4j settings? [10:43:00] elukey@an-test-coord1001:/var/log/oozie$ du -hs oozie.log-2020-12-01-07 [10:43:03] 268K oozie.log-2020-12-01-07 [10:43:06] elukey@an-test-coord1001:/var/log/oozie$ du -sh oozie.log-2020-12-01-08.gz [10:43:09] 20K oozie.log-2020-12-01-08.gz [10:43:18] so now we have good compression, and it reduces the size of logs a lot :D [10:43:29] (there was a warning about space consumption on an-coord1001 earlier on) [10:43:38] and hive is the next, we keep too many logs, not compressed [10:46:15] ack elukey - I'm afraid of restarts while big jobs are running but it seems everything went fine :) [10:49:02] Fun activity of the morning: figuring out why I have no Internet connection. [10:49:36] joal: but only sqoop is running, the mw history ones are not yet kicked off no? [10:49:44] klausman: fun indeed! [10:50:04] true elukey, but some other big jobs run (uniques etc) - no big deal [10:50:11] Turns out dhcpcd recently decided to segfault on start. So I rebuilt it (fun side project: how do you download a source tarball when your Internet is out?) Recompile fixed it. This whole technology thing is bullshit. [10:50:34] joal: ack :) [10:50:44] !log restart oozie to pick up new logging settings [10:50:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:01:59] brb [11:34:50] so https://github.com/apache/hive/commit/84e107b37e4989eb4131f8ca179dcd407aa8718d is the commit that I am working on [11:34:58] from the tests it is clear that it is what we need [11:35:07] and it is the patch for the branch-2 (so hive 2.x versions) [11:35:12] but I cannot alter [11:35:13] mmmmm [12:04:59] heya joal and elukey :] will you have 30 mins today for the netflow migration? [12:05:24] sure! [12:05:37] :] [12:05:53] I was about to go to lunch, is it something that we can do later on? [12:05:58] of course [12:06:05] the best time for you guys [12:06:09] ack! [12:06:14] I also haven't started working yet [12:06:25] let's hear from Joseph, I don't have any time issues :) [12:06:30] * elukey lunch [12:06:32] ok [12:08:37] Heya mforns and elukey [12:09:04] hey joal [12:09:05] ;] [12:09:37] If we can do between 3pm ad 4pm that'd be great - otherwise you can do without me between 4pm and 6pm, or after meetings :) [12:10:13] joal: 3-4pm is good for me! and I think elukey is fine with that IIUC [12:10:22] that would be perfect [12:10:35] ok, let's meet at 3pm elukey and joal :] thanks! [12:12:03] oh, joal, at 15h there's the mission/vision office hours [12:12:09] is that a problem? [12:13:00] mforns: I hadn't planned to go - but we can make it later, either after or post-standup [12:24:54] I also hadn't planned to go joal, 15h is ok [12:25:21] let's confirm with Luca as well mforns - We need him for that operation :) [12:25:27] yes [12:30:28] 10Analytics: Investigate oozie banner monthly job timeouts - https://phabricator.wikimedia.org/T264358 (10JAllemandou) > Are you just saying that the job needs a longer timeout? Actually there is no timeout defined for this coordinator (see https://github.com/wikimedia/analytics-refinery/blob/master/oozie/bann... [12:31:31] 10Analytics: mediawiki-wikitext-history-2020-10 failed - https://phabricator.wikimedia.org/T269032 (10JAllemandou) hm - This is not cool. Let's see if by reducing the number of cores to 3 for the same amount of RAM the job works. I'll also grow a bit the number of executors as the cluster is more powerful than i... [12:37:26] (03PS1) 10Joal: Grow mediawiki-wikitext-history spark job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644510 (https://phabricator.wikimedia.org/T269032) [13:01:28] fine for me :) [13:20:28] (03PS3) 10Fdans: Add historical_raw job to load data from pagecounts_raw [analytics/refinery] - 10https://gerrit.wikimedia.org/r/640146 (https://phabricator.wikimedia.org/T251777) [13:25:54] joal: can you explain https://gerrit.wikimedia.org/r/c/analytics/refinery/+/644510/1/oozie/mediawiki/wikitext/history/coordinator.properties ? [13:25:59] I didn't get the calculation [13:29:38] elukey: Reducing the number of cores per executor means more RAM per core [13:30:35] joal: sure I wanted to get the approach, I didn't also get the 2GB reference [13:31:18] ah ok I see that executors are already beefy [13:31:53] hm [13:32:04] It's actually me not getting the 2Gb ref :) [13:33:20] Ah actually I get it now - The commit message - It's actually 2Tb for the whole job [13:33:25] Correcting now [13:33:46] ok - And we have an issue with the image table for commonswiki :( [13:34:16] ahhhh [13:34:30] (03PS2) 10Joal: Grow mediawiki-wikitext-history spark job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644510 (https://phabricator.wikimedia.org/T269032) [13:34:34] elukey: --^ [13:34:48] (03CR) 10Elukey: [C: 03+1] Grow mediawiki-wikitext-history spark job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644510 (https://phabricator.wikimedia.org/T269032) (owner: 10Joal) [13:34:54] yes ok it makes sense [13:40:51] * elukey coffee [13:56:06] hey elukey I thought that we could stop the netflow druid jobs before the end of the hour, so that the new hour is not triggered... [13:56:16] possible? [13:56:39] yep sure [13:56:43] puppet is disabled [13:56:46] thanks!!! [13:56:47] do you want me to stop? [13:56:51] will stop them, no no [13:57:44] mforns: what do you mean? Do you need them stopped later? I stopped them [13:57:47] sudo systemctl stop *netflow* [13:57:48] very quick [13:57:49] :) [13:57:55] I just stopped them! [13:58:00] also verified with sudo systemctl list-timers | grep netflow [13:58:10] mforns: which ones did you stop? [13:58:17] the netflow druid ones [13:58:26] systemctl stop eventlogging_to_druid_netflow_hourly.timer [13:58:26] systemctl stop eventlogging_to_druid_netflow_daily.timer [13:58:26] systemctl stop eventlogging_to_druid_netflow-sanitization_daily.timer [13:58:47] yep sure they were already stopped, you can use the command above next time [13:58:51] (quicker) [13:59:01] did you stop the netflow refine ones too? [13:59:20] yep [13:59:28] if it contained "netflow" it is stopped [13:59:32] ok, but that does not kill an ongoing service right? [13:59:32] see systemctl list-timers | grep netflo [14:00:08] it could yes, I should've probably used *netflow*.timer [14:00:30] I can check [14:00:32] I don't think there will be problems [14:00:38] wanna bc? [14:01:00] the refine netflow was scheduled for 15 mins ago [14:01:00] mforns, elukey - Actually fdans mentioned to me that our new data-VP will be in office hour [14:01:09] I'll join there and can also answerr here [14:01:14] ok ok, no problemo [14:01:28] mforns: yep I checked from the scheduled time and we are good :) [14:01:37] awesome [14:01:39] I just saw that in the list of attendees so take it with a grain of salt :) [14:02:38] elukey: wait, https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/643748 is not deployed yet! [14:02:56] somehow I assume it would be... :[ [14:03:04] *assumed [14:03:34] but today is tuesday, bad calculation from my part [14:03:48] Oh my - That's my bad mforns - I didn't realize the patch was not merged :( [14:04:11] Ah - Actually it's refinery-source - I didn't deploy that [14:04:15] I don't think this was ready the last tuesday... [14:04:47] ah ok so I guess we can do it tomorrow? [14:04:52] yea... [14:04:53] we'll do a train later on [14:04:57] sorry guys [14:04:58] ack, re-enabling all then :) [14:05:01] ok [14:05:03] thanks elukey [14:05:04] nah it is fine! Good check :) [14:06:09] Heya dsaez - There is CTO office hour right now [14:06:15] dsaez: Can we reshcedule please? [14:14:25] (03CR) 10Mforns: [V: 03+2 C: 03+2] Add datasource argument to HiveToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/643748 (https://phabricator.wikimedia.org/T231339) (owner: 10Mforns) [14:14:38] heya elukey should 'Move oozie's hive2 actions to analytics-hive.eqiad.wmnet' be in ops week? [14:14:43] https://phabricator.wikimedia.org/T268028 [14:14:45] probably not, right? [14:15:39] (03CR) 10Ottomata: [C: 03+1] Grow mediawiki-wikitext-history spark job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644510 (https://phabricator.wikimedia.org/T269032) (owner: 10Joal) [14:15:42] ottomata: hello! Nono I still need to figure out hive setting for the metastore, thanks :) [14:15:46] ah, it is in ops week [14:15:50] that's why razzi was adding patches [14:15:55] we were both looking at it yesterday [14:16:10] ah ok ok but I saw a patch for puppet [14:16:20] anyway, I am going to move it to ops excellence [14:16:34] right, we were talking about refinery, dunno why he did puppet but i htink he was just trying to do ops week tasks [14:16:39] let's move that out of ops week then, right? [14:16:48] i'll put in ops excellence [14:17:05] oh, or you just did :P? [14:17:12] ty :) [14:17:18] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move oozie's hive2 actions to analytics-hive.eqiad.wmnet - https://phabricator.wikimedia.org/T268028 (10elukey) Moved from ops week to ops excellence :) [14:17:40] yes yes now I get the confusion, I didn't see it was there [14:23:38] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I can't say much about the code. I'm not familiar enough with Scala. Since this is not production, it's probably the best idea to just mer" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/620000 (https://phabricator.wikimedia.org/T246439) (owner: 10Awight) [14:34:26] 10Analytics: Investigate oozie banner monthly job timeouts - https://phabricator.wikimedia.org/T264358 (10Ottomata) I wonder if the -1 timeout in coordinator.xml conflicts with the Oozie property [[ https://oozie.apache.org/docs/3.1.3-incubating/oozie-default.xml | `oozie.service.coord.default.max.timeout` ]], w... [14:36:47] 10Analytics, 10Analytics-Kanban: Alter table for navigation timing errors out in Hadoop test - https://phabricator.wikimedia.org/T268733 (10elukey) I rebuild the packages, deployed them on both an-test-coord1001 and an-test-client1001, restarted but the alter doesn't work. I left a note in https://issues.apach... [14:38:47] * elukey errands for a bit [15:04:48] ottomata: Ah yes right we discussed how the changes should have been for refinery, but I forgot and made a patch for puppet. Sorry for the confusion. [15:07:45] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data, 10Patch-For-Review: MEP Client MediaWiki PHP - https://phabricator.wikimedia.org/T253121 (10Ottomata) Hiya, I think we need to expedite https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/623459. We need... [15:08:51] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data, 10Patch-For-Review: MEP Client MediaWiki PHP - https://phabricator.wikimedia.org/T253121 (10Ottomata) [15:09:58] no prob razzi! :) [15:10:06] ottomata, razzi, can you deploy today the thing that I just pasted to the deployment etherpad, please? :] [15:11:16] ok mforns that just goes out with new refinery source version then, right? [15:11:21] nohthing else to do? [15:11:34] no, just refinery-source [15:11:40] ok cool [15:11:46] thanks!! [15:16:49] I just realized that my probationary period is over as of today :D [15:33:50] klausman: nice! [15:35:41] (03PS1) 10Razzi: Switch an-coord1001 to analytics-hive [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644541 (https://phabricator.wikimedia.org/T268028) [15:36:23] razzi: hello! I moved the task to ops excellence don't worry, I still need to verify some hive settings before moving on [15:36:32] (for the metastore) [15:36:36] ok cool [15:37:14] (03Abandoned) 10Razzi: Switch an-coord1001 to analytics-hive [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644541 (https://phabricator.wikimedia.org/T268028) (owner: 10Razzi) [15:37:22] and I can't test them now since mediawiki history is running and Joseph would probably not talk with me anymore if I restart hive now :D [16:06:12] 10Analytics-Clusters, 10Cassandra: Set up a testing environment for the AQS Cassandra 3 migration - https://phabricator.wikimedia.org/T257572 (10elukey) This is pending the racking of T264336 [16:20:59] 10Analytics, 10Analytics-Kanban: Alter table for navigation timing errors out in Hadoop test - https://phabricator.wikimedia.org/T268733 (10elukey) I am reasonably sure that the patch gets applied, I did the following: - Checked what jar contained the class modified (/usr/lib/hive/lib/hive-common.jar, from th... [16:35:48] (03PS1) 10Mforns: Add dsaez to receive traffica anomaly alerts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644567 (https://phabricator.wikimedia.org/T267356) [16:36:21] (03PS2) 10Mforns: Add dsaez to receive traffic anomaly alerts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644567 (https://phabricator.wikimedia.org/T267356) [16:36:44] ottomata: would it be ok in your opinion to deprecate 'analytics-users' too? [16:36:44] hey elukey, if you have a second, want to go over the wikistats deploy? [16:37:04] milimetric: sure! [16:38:07] gimme 5 [16:38:25] np, I'm in the cave whenever [16:38:25] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging this, as it's trivial." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644567 (https://phabricator.wikimedia.org/T267356) (owner: 10Mforns) [16:41:25] elukey: hm [16:41:38] don't we need that? [16:43:19] ottomata: atm very few users, some of them are also in 'researchers', I suspect it is only another confusing access.. we could establish that the baseline is analytics-privatedata-users (access to GPUs + wikireplicas) without kerberos, and then +kerberos if needed [16:50:45] 10Analytics, 10Patch-For-Review: [Data quality stats] Add dsaez to receive traffic anomaly alarms - https://phabricator.wikimedia.org/T267356 (10mforns) I did a cheat-deploy of the modified Oozie bundle files, so @diego should receive traffic alarms from now on :] [16:51:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [Data quality stats] Add dsaez to receive traffic anomaly alarms - https://phabricator.wikimedia.org/T267356 (10mforns) [16:56:13] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) Cool! Thanks :] Will do. I'll let @JAllemandou coordinate with you on a good date and time for the team presentation! [17:01:03] fdans: standup! :) [17:40:33] 10Analytics: Deprecate the anaytics-users POSIX group - https://phabricator.wikimedia.org/T269150 (10elukey) [17:45:27] elukey: razzi retro! [18:14:16] * milimetric lunching [18:16:14] razzi: do you have specific doubts about the decom stuff? [18:16:33] (also, Riccardo added more logging to the cookbook, it should be more clear why it fails now) [18:17:35] mforns: ok to merge the netflow retention reduction to 60d? [18:22:41] 10Analytics-Radar, 10Product-Analytics (Kanban): Content for analytics.wikimedia.org - https://phabricator.wikimedia.org/T267254 (10mpopov) [18:23:41] mforns: wohooo [18:23:41] https://stream-beta.wmflabs.org/v2/ui/#/ [18:23:50] 10Analytics-Radar, 10Product-Analytics (Kanban): Content for analytics.wikimedia.org - https://phabricator.wikimedia.org/T267254 (10mpopov) [18:25:11] 10Analytics, 10Patch-For-Review: Deprecate the anaytics-users POSIX group - https://phabricator.wikimedia.org/T269150 (10elukey) @brion @Mhurd @SantoshiWiki Hi! My team is trying to consolidate POSIX groups, and you are the last ones remaining in `analytics-users`. The group basically grants only ssh access to... [18:26:09] it wooOoOrks! [18:29:33] elukey: yeah, about decommissioning: how can I check if netbox is in a consistent state? [18:32:07] mforns: this is so COOOOOL [18:32:16] the easiest way to see it working is to stream eventlogging_SearchSatisfaction [18:32:37] and then do searches on https://meta.wikimedia.beta.wmflabs.org/wiki/ [18:33:26] razzi: on cumin you can run 'sudo cookbook -d sre.dns.netbox "test"' that basically prints what's wrong, without asking you to commit.. it is good to check if the netbox dns step fails for example [18:40:47] elukey: cool, it ran clean this time, so I'm going to continue with decommissioning [18:41:07] PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [18:43:40] elukey: yes! [18:43:47] merge please if good [18:44:26] ottomata: :D [18:44:36] glad you made it work! [18:47:54] mforns: deployed [18:48:05] elukey: thanksss!! [18:48:27] elukey: during the next couple days will backfill the 90 to 60 day gap [18:57:35] ack :) [18:57:39] ottomata: I modified https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Access_Groups [18:57:45] also razzi --^ [18:58:00] I'll try to simplify it even further [18:58:44] what I'd like to do is to add a disclaimer for users that says "you are accessing a host with PII data, please be careful about bla bla) [18:58:47] " [18:59:27] at the end I'll also send an email to SRE to explain the new process, that will hopefully be very straightforward [19:02:18] +1 [19:02:29] razzi: ready to do train whenever you are! [19:02:44] ottomata: Ok! gimme 5 [19:02:49] k [19:03:34] * elukey afk! [19:05:10] ok razzi before we jump in hangout [19:05:11] read [19:05:15] (or skim) [19:05:16] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery-source [19:05:17] and then [19:05:21] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery [19:05:47] or actually, you've done this once before with marcel, right? [19:05:53] if so nm, let's go! :) [19:12:21] ottomata: yup. Ready to go in batcave [19:18:29] (03PS1) 10Razzi: Update changelog.md for version v0.0.140 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/644598 [19:18:50] (03CR) 10Razzi: [C: 03+2] Update changelog.md for version v0.0.140 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/644598 (owner: 10Razzi) [19:18:56] (03CR) 10Razzi: [V: 03+2 C: 03+2] Update changelog.md for version v0.0.140 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/644598 (owner: 10Razzi) [19:19:43] dumb question, which ACL do I need to be able to query the wmf.webrequest table in Hive? is that analytics-privatedata-users ? [19:24:42] cdanis: that's correct. You need that and kerberos. Would you like help setting up access? [19:30:05] razzi: do I need any approvals to join the group? I'm happy to send the puppet match and merge it after review :) [19:31:29] cdanis: i approve! [19:31:35] cdanis: Go ahead and make a patch :) [19:32:01] cdanis: you'll also have to do https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos#Create_a_principal_for_a_real_user [19:32:15] yeah that part I did :) done it before for others as part of clinic duty [19:32:48] https://gerrit.wikimedia.org/r/c/operations/puppet/+/644600 [19:33:01] oh nice [19:33:44] thanks! [19:40:19] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.0.140 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644601 [19:41:36] (03CR) 10Razzi: [V: 03+2 C: 03+2] Add refinery-source jars for v0.0.140 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/644601 (owner: 10Maven-release-user) [19:43:51] !log deploy refinery with refinery-source v0.0.140 [19:43:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:44:00] mforns: woo hoo and in prod too [19:44:00] https://stream.wikimedia.org/v2/ui [19:44:03] (with fewer streams exposed) [19:44:16] mforns: feature request: maybe the scroll back should be backwards, with most recent up top [19:44:30] in my browser, the sticky scroll doesn't happen unless i manually scroll down [19:46:13] sorry ottomata was in a meeting [19:46:21] proood!! :D [19:46:43] ottomata: I considered this when I designed the ui [19:47:12] ya? [19:47:26] ottomata: yes, the sticky scroll only works if you explicitly scroll to the bottom once [19:47:34] yes, wanna bc, I can explain [19:47:56] in bc with razzi now but i think we are almost done [19:48:00] ok [19:48:02] joining [19:48:34] 10Analytics, 10Event-Platform, 10EventStreams: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) [19:49:24] 10Analytics, 10Event-Platform, 10EventStreams: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) [19:50:33] 10Analytics, 10Event-Platform, 10EventStreams: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) [19:51:21] 10Analytics, 10Event-Platform, 10EventStreams: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) [20:31:20] 10Analytics, 10Event-Platform, 10EventStreams, 10Operations, and 4 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata)