[01:51:12] PROBLEM - Check the last execution of monitor_refine_event_failure_flags on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:14:28] !log force re-run of monitor_refine_event_failure_flags after fixing a refine failed hour [05:14:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [05:17:08] RECOVERY - Check the last execution of monitor_refine_event_failure_flags on an-launcher1001 is OK: OK: Status of the systemd unit monitor_refine_event_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:45:15] 10Analytics, 10Analytics-Kanban: check leftovers of jmorgan - https://phabricator.wikimedia.org/T251600 (10elukey) 05Open→03Resolved Cleaned up stat/notebook hosts! [05:50:24] 10Analytics: Check home leftovers of lexnasser - https://phabricator.wikimedia.org/T252363 (10elukey) [05:52:42] morning, looking at alarms [05:54:00] ah, just say elukey's message, out of memory issue [05:54:52] yep :) [05:55:59] good morning fdans [05:58:26] PROBLEM - Check the last execution of wikimedia-discovery-golden on stat1007 is CRITICAL: CRITICAL: Status of the systemd unit wikimedia-discovery-golden https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:03:52] 10Analytics, 10Discovery-Analysis, 10Product-Analytics: Wikimedia Discovery Golden timer on stat1007 keeps logging errors - https://phabricator.wikimedia.org/T252365 (10elukey) [06:06:33] !log restart wikimedia-discovery-golden on stat1007 - apparenlty killed by no memory left to allocate on the system [06:06:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:06:43] also opened a task, the are failures logged :( [06:09:18] RECOVERY - Check the last execution of wikimedia-discovery-golden on stat1007 is OK: OK: Status of the systemd unit wikimedia-discovery-golden https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:54:55] (03PS24) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [07:59:17] 10Analytics, 10Operations, 10Traffic, 10Patch-For-Review: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 (10ema) >>! In T237993#6111924, @fgiunchedi wrote: > Rephrasing to make sure I understand: the major problem is making sure that the mapping from `struct kafka.Stats` to... [08:45:43] (03PS25) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [10:13:17] mmm webrequest load text failed [10:16:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Investigate sporadic failures in oozie hive actions due to Kerberos auth - https://phabricator.wikimedia.org/T241650 (10elukey) Happened today again. This is very sporadic, at this point I hope it will be fixed in Hive 2.x.. [10:17:04] !log re-run webrequest-load-wf-text-2020-5-11-9 [10:17:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:30:32] 10Analytics: Refresh stat1004 with a new host and GPU - https://phabricator.wikimedia.org/T241187 (10elukey) 05Stalled→03Resolved [10:32:41] 10Analytics, 10Analytics-Cluster, 10Operations: notebook1004 - /srv is full - https://phabricator.wikimedia.org/T232068 (10elukey) 05Open→03Resolved We added SWAP to all stat100x hosts, and set a deprecation of the notebooks for June 2020. In theory we shouldn't receive any more alarms, closing. [10:46:19] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10elukey) [10:46:28] 10Analytics, 10Analytics-Kanban: Analytics Ops Technical Debt - https://phabricator.wikimedia.org/T240437 (10elukey) [10:47:00] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10elukey) [10:56:51] * elukey lunch! [11:25:52] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10JAllemandou) I like the feature but I'm afraid of the potential errors it can lead to in case of update-error or people misunderstanding. More precisely, I dislike having an non-explicit thing (here, the partition... [11:43:35] (03CR) 10Joal: "Implementation details: Given wikidata-dump generation artifact, the full range of needed page-move events should be fully available at le" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric) [12:15:30] 10Analytics, 10Browser-Support-Opera, 10good first task: Split opera mini in proxy or turbo mode - https://phabricator.wikimedia.org/T138505 (10AMuigai) [13:48:54] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10Ottomata) Perhaps a different name would be less confusing? We don't actually care which data is being queried, just that it points to some relatively recent partition that has data. - probe - sample - example... [13:53:27] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10elukey) [13:53:31] 10Analytics, 10Analytics-Kanban: Analytics Ops Technical Debt - https://phabricator.wikimedia.org/T240437 (10elukey) [13:53:35] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Remove references to m4-master - https://phabricator.wikimedia.org/T245238 (10elukey) 05Open→03Resolved a:03elukey [14:05:42] good point joal, I’ll implement the dataset [14:06:22] Hi milimetric - Please take some rest :) [14:07:00] milimetric: I'll gladly continue the talk around trade-off between correctness and speed though - That case not being the best is all :) [14:07:37] hey team! [14:10:02] 10Analytics: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006 - https://phabricator.wikimedia.org/T252203 (10Ottomata) Just launched a backfill for mediawiki_cirrussearch_request and mediawiki_api_request for any failed or modified raw data hours since... [14:16:05] ottomata: shall we spend some brain-cycles on trying to fix that --^ ? [14:18:03] joal: yes! but i would like to make some progress on some mep stuff today [14:18:06] so maybe not today? [14:19:10] np ottomata - let's just not forget! [14:19:13] might be really hard to repro [14:19:18] mhmm [14:19:21] indeed [14:19:46] i should have made a copy of the entire camus history when it happened. [14:19:53] oh [14:19:53] actualy [14:20:04] i think i do have a repro history diur [14:20:13] that i was using to test when i was writing out sequence etlkey files [14:20:25] hm, but the offsets are expired [14:20:28] by now [14:21:09] ottomata: Shall we prepare and have a camus version with a new kafka-client ready? [14:21:21] ot is there another strategy ottomata ? [14:21:34] joal: we shoiuld do that anyway [14:21:36] since luca wants TLS [14:21:42] makes sesne [14:21:53] ok let's prioritie that then [14:21:57] yeah prob a good idea [14:22:03] its on lucas todo but i betcha he wouldn [14:22:18] 't mind if you had some cycles to do it! :D [14:22:31] I guess so [14:22:34] tee hee [14:22:40] but ya let's talk post standup today [14:35:37] I have started today joal [14:35:45] if you want I can hand it over to you :) [14:36:19] what I want to do it now is to check a simple java code that uses TLS without horrible hacks, separated from Camus [14:36:36] just to double check that the functionality is already built it [14:36:46] or if we have to add it in some way [14:37:00] then afterwards patching camus should be painful but probably not that much [14:37:07] elukey: hmmmm - You mean, run a simple java code using kafka new connector to ensure TLS is supported (just rephrasing to be sure) [14:37:17] joal: yes exactly [14:37:23] ack [14:37:28] rather than testing on camus directly [14:37:36] makes sense [14:37:54] elukey: I can help if you wish :) [14:38:27] joal: if you have time feel free to do it, you have more experience than me in Java, I am doing it since nobody has time :) [14:38:38] (and it blocks kafka tls basically) [14:38:42] elukey: it's about priorities :) [14:39:06] elukey: will try - do you have a kafka-connector version in mind? [14:39:54] kafka-connector? [14:40:00] do you mean client? [14:40:09] yessir - sorry, wrong terminology [14:40:18] I am using kafka-clients 0.11.0.3 now [14:40:23] in my pom [14:40:33] elukey: you can prob use whatever version of kafka broker we have [14:40:35] righht? [14:40:41] or later, but might be best to start with that [14:40:46] should we tighten that to our kafka version? [14:40:46] yeah I thought the same [14:40:49] ah yes [14:41:06] And we have kafka 0.11 I guess :) [14:41:09] ok [14:42:07] should I keep doing my tests or should I start doing something else ? [14:43:05] elukey: I think the apprach is great [14:43:33] elukey: given the load you have, I do the test if you prefer [14:44:09] joal: atm I am not super loaded, if the ops queue doesn't explode of course for some unknown hadoop issue :D [14:44:23] :) [14:44:59] elukey: I'm not sure to provide value given you know TLS better than me - And I'm not sure about my java skills being better than yours [14:45:20] elukey: But I'm willing to provide cycles! pair-code, or review, or debugging, or doing it fully if you prefer :) [14:46:47] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) [14:47:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Release-Engineering-Team: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) [14:51:21] joal: all right I'll finish my hack and then report in the task :) [14:51:50] elukey: ok - if ops-load raises, please pass over! [14:52:03] I am thinking that eventually it would be awesome to finally use only port 9093 on kafka jumbo [14:52:10] :) [14:52:31] so TLS by default, so people don't forget to encrypt PII data, and then lock down topics with auth + checks [14:52:49] the last bit is not trivial to maintain [14:53:00] buuut always TLS is a starter [14:53:08] :) [15:23:07] 10Analytics: SQL query failed on superset SQL lab - https://phabricator.wikimedia.org/T252225 (10Ottomata) Hi @jwang, this doesn't answer your SQL lab question, but could the event.mediawiki_user_blocks change help you here? ` presto> use analytics_hive.event; USE presto:event> show create table event.mediawiki... [15:23:21] 10Analytics: SQL query failed on superset SQL lab - https://phabricator.wikimedia.org/T252225 (10Ottomata) >>! In T252225#6125157, @Ottomata wrote: > Hi @jwang, this doesn't answer your SQL lab question, but could the event.mediawiki_user_blocks_change table help you here? > > ` > presto> use analytics_hive.eve... [15:24:33] 10Analytics, 10Analytics-Kanban: Request a Kerberos identity for jrobson - https://phabricator.wikimedia.org/T252222 (10Ottomata) a:03Ottomata [15:27:21] 10Analytics, 10Analytics-Kanban: Bump up SLA of pageview jobs after deploying bots check - https://phabricator.wikimedia.org/T252220 (10mforns) [15:27:23] 10Analytics: Study whether we need to increase pageview-hourly SLA after adding automated tag - https://phabricator.wikimedia.org/T252211 (10mforns) [15:39:45] 10Analytics: Idea: Add 'top X bigger than Y' sanitization method to EL-to-Druid - https://phabricator.wikimedia.org/T251145 (10Ottomata) p:05Triage→03Low [15:55:20] 10Analytics, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on analytics1055 - https://phabricator.wikimedia.org/T252070 (10Ottomata) 05Open→03Declined [15:56:08] 10Analytics, 10Operations, 10observability: systemd::syslog conf should use :programname equals instead of startswith - https://phabricator.wikimedia.org/T251606 (10Ottomata) a:03Ottomata [15:56:18] 10Analytics, 10Analytics-Kanban, 10Operations, 10observability: systemd::syslog conf should use :programname equals instead of startswith - https://phabricator.wikimedia.org/T251606 (10Ottomata) [16:09:33] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` kafka-jumbo1007.eqiad.wmnet ` The log can be found... [16:11:25] a-team i just read in #wikimedia-discovery that elukey takes full responsibility for the existance of kerberos and its implementation and use in hadoop [16:11:47] perhaps we should not have made that t-shirt of him and the 3 headed dog so friendly looking [16:12:19] haahhahaha [16:12:24] :) [16:12:59] I don't want people to swear against the whole analytics team so I try to take the blame where possible :) [16:18:47] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10elukey) Today I did a quick test with the following code on stat1005, that worked nicely: ` package org.wikimedia.kafkatest; import org.apache.kafka.cli... [16:47:04] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1007.eqiad.wmnet'] ` Of which those **FAILED**: ` ['kafka-jumbo1007.eqiad.wmnet'] ` [16:57:42] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10elukey) I had to abort the wmf reimage script because it wasn't getting to the point of running puppet, then I accepted manually the new puppet cert and ran... [17:01:10] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` kafka-jumbo1008.eqiad.wmnet ` The log can be found... [17:01:46] * elukey afk for a bit [17:23:24] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1008.eqiad.wmnet'] ` and were **ALL** successful. [17:24:26] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) Hm, IIRC Java keytool will now allow generation of .jks files without a password. If no password is used, cergen will not generate .jks files.... [17:25:21] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) Or if we could find some other way to generate passwordless .jks files, that would work too. I wouldn't mind managing a puppet CA only truststo... [17:36:17] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` kafka-jumbo1009.eqiad.wmnet ` The log can be found... [17:37:06] ottomata: new jumbo nodes almost ready! --^ [17:40:25] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10Isaac) Whatever is decided is fine with me, but I'm also of two minds about using `latest` vs. some other term: * On one hand, I completely understand where @JAllemandou is coming from. Silent errors suck and the... [17:42:58] nuria - Can you take another look at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventLogging/+/595005/ ? [17:47:27] oh boy! [17:56:21] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1009.eqiad.wmnet'] ` and were **ALL** successful. [17:59:09] joal: elukey i'm currently blocked on eventlogging evengate stuff. can/should I help with camus kafka client? I betcha I could get a working patch up this afternoon if i dug in...just guessing [18:00:40] ottomata: it is ok for me, I left a comment in the task about distributing the keystore [18:01:09] works for me ottomata [18:01:09] i saw, and responed :) [18:04:22] ottomata: so I'd prefer to not manage the password if not needed, but no idea if possible with keytool [18:05:13] another hacky solution would be to grab the puppet ca public cert on every node and create a jks with keytool on the fly if not present [18:05:21] but not really a clean solution [18:07:51] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10elukey) 05Open→03Resolved [18:07:58] 3 new brokers ready! --^ [18:09:05] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) Timo suggested I just add routing to these backends in ATS via horizon hiera in deployment... [18:16:28] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10JAllemandou) @Isaac you commented faster than I did :) After some talk with the team at standup today, we agreed on suggesting adding a `latest` partition for tables handling snapshots (meaning full-data, and not... [18:23:43] 10Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10kaldari) [18:24:07] * elukey off! [18:30:29] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10Isaac) > After some talk with the team at standup today, we agreed on suggesting adding a latest partition for tables handling snapshots (meaning full-data, and not released too often). Ahh excellent, this is a co... [18:33:52] elukey: the problem is that java keytool will not LET you create a .jks file without a password [18:33:54] iirc [18:34:01] oh you are off [18:34:04] byyeee! [18:34:39] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) Oh boy. Just realized that Camus uses the Old Kafka consumer interface which does not support TLS. I dunno if this will be so easy after all. [18:34:54] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) https://kafka.apache.org/090/documentation.html#oldconsumerconfigs [18:37:26] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10nettrom_WMF) [18:37:49] 10Analytics: SQL query failed on superset SQL lab - https://phabricator.wikimedia.org/T252225 (10JAllemandou) Some more info on querying `mediawiki_*` tables snaphsots in `wmf_raw` database from Presto: - They are stored in `avro` file format. This format is not columnar, therefore a lot less performant when qu... [18:41:32] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) Yeah, I take it all back. I don't think this task is so easy after all. This isn't just a client upgrade, it is a library change. `kafka-clie... [19:06:29] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10kaldari) My opinion is that EventLogging should not be affected by DNT. Here are my reasons: * DNT is a failed experiment and has been superseded by ad blockers, browser-based tracking prot... [19:09:04] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10kaldari) [19:29:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Request a Kerberos identity for jrobson - https://phabricator.wikimedia.org/T252222 (10Ottomata) Hiya! Done. You should receive an email at jrobson@wikimedia.org with instructions on how to log in and set your password. [19:29:29] elukey: did you get a chance to update the jenkins/archiva password? [19:33:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Allow MediaWiki client side JS to POST to EventGate webproxy in beta - https://phabricator.wikimedia.org/T252417 (10Ottomata) 05Open→03Resolved a:03Ottomata [19:33:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) [19:36:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) Woohoo! I just logged a Test event to eventgate in beta via `mw.eventLog.logEvent("Test", {"O... [19:40:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) I think we are ready to move here. The annoying part is that our switch is de... [19:45:02] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) Ah! Yes we can! Timo just pointed out: https://gerrit.wikimedia.org/g/media... [20:14:41] 10Analytics, 10Product-Analytics: Update mediawiki_user_blocks_change to log partial block parameters - https://phabricator.wikimedia.org/T252455 (10nettrom_WMF) [20:16:59] 10Analytics: SQL query failed on superset SQL lab - https://phabricator.wikimedia.org/T252225 (10nettrom_WMF) >>! In T252225#6125157, @Ottomata wrote: > could the `event.mediawiki_user_blocks_change` help you here? @jwang and I've talked about using that table, but one of the issues is that it's not logging blo... [20:46:43] (03PS2) 10Mforns: Make anomaly detection correctly handle holes in time-series [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) [20:49:46] (03CR) 10Mforns: [V: 03+2] "I added the sparsity threshold and a couple tests." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns) [20:50:05] * mforns off! [21:06:32] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10Krinkle) [21:06:34] 10Analytics, 10Patch-For-Review, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) [21:08:05] 10Analytics, 10Patch-For-Review, 10User-Elukey: Redesign architecture of irc-recentchanges on top of Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) [21:08:26] 10Analytics, 10Patch-For-Review, 10User-Elukey: Port architecture of irc-recentchanges to Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) [21:08:40] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10Krinkle) [21:08:42] 10Analytics, 10Patch-For-Review, 10User-Elukey: Port architecture of irc-recentchanges to Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) [21:08:58] 10Analytics, 10Patch-For-Review, 10User-Elukey: Port architecture of irc-recentchanges to Kafka - https://phabricator.wikimedia.org/T234234 (10Krinkle) [21:09:02] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10Krinkle) [21:09:05] 10Analytics, 10User-Elukey: Port IRCRecentChanges to Kafka - https://phabricator.wikimedia.org/T232483 (10Krinkle) [21:09:38] 10Analytics, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server, 10Code-Stewardship-Reviews: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319 (10Krinkle) [21:39:25] 10Analytics, 10EventStreams: EventStreams socket stays connected without any traffic incoming - https://phabricator.wikimedia.org/T250912 (10stjn) Sorry for not answering this sooner. I am running [[ https://github.com/stjohann/DiscordWikiBot | a Discord bot ]] for Wikimedia Discord servers that is written in... [22:06:16] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10jwang) @nuria, @Bmueller As we discussed, I have calculated bot editors % per wiki projects and added to [[ https://docs.google.com/s...