[00:04:35] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 3 others: EventLogging MEP Upgrade Phase 3 (Sampling) - https://phabricator.wikimedia.org/T234594 (10jlinehan)
[01:19:35] <wikibugs>	 10Analytics, 10Product-Analytics: 'namespace_is_content' column from pageview_hourly and pageview_daily returns 1, 0 and NULL as booleans in Superset/Turnilo - https://phabricator.wikimedia.org/T255222 (10cchen)
[01:21:14] <wikibugs>	 10Analytics, 10Product-Analytics: 'namespace_is_content' column in pageview data returns 1, 0 and NULL as booleans in Superset/Turnilo - https://phabricator.wikimedia.org/T255222 (10cchen)
[09:30:21] <wikibugs>	 (03PS1) 10Elukey: oozie: move cassandra config to refinery-cassandra-0.0.115 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605167 (https://phabricator.wikimedia.org/T254849)
[09:30:23] <wikibugs>	 (03PS1) 10Elukey: oozie: move cassandra config to use refinery-hive-0.0.115.jar [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605168 (https://phabricator.wikimedia.org/T254849)
[09:30:25] <wikibugs>	 (03PS1) 10Elukey: oozie: use refinery-hive-0.0.115.jar in cassandra hql files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605169 (https://phabricator.wikimedia.org/T254849)
[09:30:27] <wikibugs>	 (03PS1) 10Elukey: oozie: update refinery-job jars for mediawiki history coordinators [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605170 (https://phabricator.wikimedia.org/T254849)
[09:30:29] <wikibugs>	 (03PS1) 10Elukey: oozie: update refinery-hive jar for unique_devices per project fam coordinators [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605171 (https://phabricator.wikimedia.org/T254849)
[09:30:31] <wikibugs>	 (03PS1) 10Elukey: oozie: update refinery-hive jar in insert_hourly_mediacounts.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605172 (https://phabricator.wikimedia.org/T254849)
[09:30:33] <wikibugs>	 (03PS1) 10Elukey: oozie: move remaining coordinator properties to use refinery-job 0.0.115 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605173 (https://phabricator.wikimedia.org/T254849)
[09:30:35] <wikibugs>	 (03PS1) 10Elukey: Remove artifacts that are not needed anymore. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605174 (https://phabricator.wikimedia.org/T254849)
[09:39:50] <elukey>	 brb
[10:07:06] <moritzm>	 it's great to see how AMD explicitly pitches the open source nature of ROCM: https://www.youtube.com/watch?v=CnlKvgdJU9Q
[10:11:17] <elukey>	 I thought it was sarcastic and I was worried to watch the video but no, it is great :)
[10:11:53] <elukey>	 I particularly like all the work that they are doing on HIP
[11:04:28] <elukey>	 https://archiva-new.wikimedia.org/#welcome
[11:04:31] <elukey>	 ready!
[11:05:43] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move Archiva to Debian Buster - https://phabricator.wikimedia.org/T252767 (10elukey) https://archiva-new.wikimedia.org/ ready for testing!
[13:18:20] <wikibugs>	 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Gilles) 05Open→03Resolved Thanks. Looking at the last few hours of data, I don't see the huge values reoccurring and the Firefox 78 records we receive look fine.  Sometimes the browsers se...
[13:20:42] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Purge old files on Archiva to free some space - https://phabricator.wikimedia.org/T254849 (10elukey) The above patches are only an idea/proposal if we want to unblock option 1), but no strong opinion :)
[13:26:25] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Purge old files on Archiva to free some space - https://phabricator.wikimedia.org/T254849 (10JAllemandou) >>! In T254849#6219059, @elukey wrote: > The above patches are only an idea/proposal if we want to unblock option 1), but no strong opinion :)  If we...
[13:27:09] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Purge old files on Archiva to free some space - https://phabricator.wikimedia.org/T254849 (10elukey) >>! In T254849#6219086, @JAllemandou wrote: >>>! In T254849#6219059, @elukey wrote: >> The above patches are only an idea/proposal if we want to unblock op...
[13:38:45] <wikibugs>	 10Analytics, 10Analytics-Cluster: Kerberos credential cache location - https://phabricator.wikimedia.org/T255262 (10elukey)
[13:39:23] <elukey>	 ottomata: o/ all firewall rules updated
[13:39:34] <ottomata>	 Thank you!
[13:39:41] <ottomata>	 i looked at camus logs and it looks like the topic is importing now
[13:39:43] <joal>	 ack elukey - Let's see if camus downloads new stuff :)
[13:39:54] <joal>	 ok - too late I am :)
[13:40:06] <ottomata>	 elukey:  btw this might be handy for ya
[13:40:06] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/605219/5/modules/admin/files/home/otto/.bashrc
[13:40:12] * joal get's back to reviewing java code
[13:40:20] <ottomata>	 getting tired of forgetting!
[13:40:38] <elukey>	 ottomata: I saw it! Nice!
[13:40:45] <elukey>	 I loved the commit msg "otto - etc.."
[13:40:54] <elukey>	 :D
[13:41:16] <ottomata>	 also, i am trying out a vertical mouse  
[13:42:56] <joal>	 ottomata: in a few days you'll wonder how you've been doing without for so long :)
[13:43:31] <ottomata>	 very not used to it yet
[13:43:42] <joal>	 it took me 2/3 days
[13:44:07] <joal>	 but now my (not big) pains in forarms are gone
[13:51:34] <elukey>	 ottomata: when you have a moment https://gerrit.wikimedia.org/r/#/c/operations/dns/+/605225/
[13:51:52] <ottomata>	 joal:  i dunno i can't wear this wrist brace i've been wearing and use the mouse at the same time
[13:52:14] <joal>	 Ah! I use no jowelery - can't speak
[13:52:16] <ottomata>	 also, i just noticed that whenever i used a mouse before (which i've only started to due to wrist issues), i don't really move my arm around
[13:52:28] <ottomata>	 i can move the mouse up and doesn enough just by extending my fingers
[13:52:31] <ottomata>	 but i can't do that with vertical mouse
[13:52:36] <ottomata>	 have to move arm to move cursor up
[13:53:52] <joal>	 ottomata: feels like you need to set it up (speed) so that you can move with enough distinace without having to move the arm - I also use a mouse-pas with a little cushion - My wrist actually doesn't move from the cushion
[13:54:37] <ottomata>	 heh ye\ah i can turn speed up.  the way this thing moves or accelerrates is so different!  gotta find the right spot
[13:54:39] <ottomata>	 will keep trying !
[13:54:49] <wikibugs>	 10Analytics: Create a small VM to host jobs/system timers currently running on stat1007 - https://phabricator.wikimedia.org/T255265 (10elukey)
[13:54:55] <joal>	 ottomata: it's worth the learning IMO
[13:55:33] <wikibugs>	 (03CR) 10Joal: "Plenty comments - On main code only, I have not reviewed tests" (0315 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[13:55:53] <joal>	 ottomata: a lot of stuff to discuss --^ !! Let me know if I'm too nitpicky :)
[13:55:56] <wikibugs>	 10Analytics: Separate Report Updater Jobs to dedicated VM - https://phabricator.wikimedia.org/T255266 (10elukey)
[13:58:32] <ottomata>	 joal:  thank you!  will read now.  i'm sure there iis LOTS to pick at in that patch
[13:58:55] <joal>	 ottomata: patch is big, I went for big stuff first )
[14:01:32] <wikibugs>	 10Analytics, 10CAS-SSO, 10User-Elukey: Secure Hue/Superset/Turnilo/Yarn/Piwik with CAS (and possibly 2FA) - https://phabricator.wikimedia.org/T159584 (10elukey)
[14:03:15] <elukey>	 very interesting CPU usage pattern https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=an-launcher1001&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-7d&to=now
[14:04:18] <elukey>	 I bet that it is RU 
[14:04:20] <joal>	 wow elukey - this is fun!
[14:04:58] <joal>	 I'd be interested to understand elukey - Also to note - Network usage at the longer spike (today)
[14:05:04] <joal>	 elukey: Why on Friday???
[14:06:19] <elukey>	 it is a pity that I noticed it by chance only now, two hours ago would have got us more info
[14:06:30] <joal>	 I understand the little bump  at midnight every day, but how come today?
[14:06:51] <joal>	 elukey: also when looking at longer time, spikes have happened at non-regular times
[14:08:02] <elukey>	 joal: ah wait that might be camus
[14:08:10] <joal>	 hm
[14:08:25] <elukey>	 this morning I enabled firewall rules for jumbo1007-9
[14:08:38] <elukey>	 and ottomata was saying yesterday that camus was not running correctly due to that
[14:08:57] <joal>	 this is weird - camus launches executors on hadoop, small local footprint for launcher
[14:09:25] <joal>	 elukey: spike started at ~1:30am UTC - too early for camus
[14:09:28] <elukey>	 in theory yes, could may be only the network part then
[14:09:53] <elukey>	 ah yes right it was early morning
[14:09:56] <elukey>	 nevermind then, not camus
[14:10:43] <elukey>	 well then I don't have anything out of the ordinary in mind
[14:13:57] <joal>	 elukey: I get it I think :) hdfs dfs -ls /wmf/data/raw/mediawiki/dumps/pages_meta_history/20200601 | grep 2020-06-12 | wc -l
[14:14:53] <joal>	 elukey: looks like dumps of small wikis got released tonight :)
[14:17:43] <elukey>	 joal: so was it hdfs-rsync? If not I am not sure how those get imported
[14:17:54] <joal>	 elukey: I think it was indeed
[14:18:15] <elukey>	 ack then we know now :)
[14:22:03] <joal>	 elukey: I'm not suprised of the duration of the spike, however the CPU usage for hdfs-rsync feels high!
[14:48:02] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (0311 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[14:48:56] <ottomata>	 joal:  responded to most of them!  let me know what you think about those responses
[14:57:55] <milimetric>	 hi
[15:03:52] <nuria>	 elukey: do you have time for some k8 /druid questions?
[15:06:30] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] "looks good, Let's please remember to add this to the train when merged as the job needs to be re-started" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605167 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:07:37] <elukey>	 nuria: hola! I do but k8s is not really my thing :D
[15:08:06] <nuria>	 elukey: when k8 is set up for druid, will users authnticate similar to how is it done for presto?
[15:08:30] <nuria>	 elukey: so it is using teh user k8 credentials 
[15:08:35] <joal>	 Nice :) k8 is usually used for kubernetes nuria, not kerberos :)
[15:08:49] <nuria>	 elukey: ah yes, sorry, kerberos
[15:08:56] <nuria>	 elukey: me dummy
[15:09:11] <elukey>	 ahhhhh
[15:09:24] <joal>	 I guess Kerberos we could shorten to k3 :)
[15:09:44] <elukey>	 nuria: in theory yes
[15:09:55] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] oozie: move cassandra config to use refinery-hive-0.0.115.jar [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605168 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:10:03] <elukey>	 but I didn't check in dept how druid manages it
[15:10:27] <nuria>	 elukey: this means that druid dashboards are going to stop working for all PMs and similar that do not have a kerberos user
[15:10:31] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "Confirmed no changes since 0.0.107" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605167 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:10:42] <nuria>	 elukey: so we will need to create those ourselves and manage those by hand 
[15:11:00] <nuria>	 elukey: right? as we cannot ask those users to log in via ssh 
[15:11:23] <elukey>	 nuria: I haven't had a chat with Moritz about it, but I was thinking of adding posix users to puppet without requiring ssh access or similar
[15:11:41] <elukey>	 so they'll be valid users deployed, without the full access req
[15:11:49] <elukey>	 we do something similar for ldap-only users
[15:12:26] <nuria>	 elukey: i see, taht sounds good
[15:12:50] <nuria>	 *that
[15:14:27] <elukey>	 nuria: I have the use case in mind, will not forget it when trying to come up with some proposal for securing druid I promise :)
[15:14:28] <wikibugs>	 (03CR) 10Nuria: [C: 03+2] "Merging, since these are doc changes" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605169 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:14:30] <wikibugs>	 (03CR) 10Nuria: [V: 03+2 C: 03+2] oozie: use refinery-hive-0.0.115.jar in cassandra hql files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605169 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:15:05] <nuria>	 elukey: ok, i told jeff that the kerberos auth for druid will happen after we upgrade druid, let me know if that is not correct
[15:15:36] <elukey>	 nuria: yes that is the plan, hopefully by the end of next Q
[15:16:04] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] oozie: update refinery-job jars for mediawiki history coordinators [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605170 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:16:21] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] oozie: update refinery-hive jar for unique_devices per project fam coordinators [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605171 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:16:47] <nuria>	 elukey: this quarter or next quarter?
[15:17:52] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] oozie: move remaining coordinator properties to use refinery-job 0.0.115 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605173 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:18:12] <wikibugs>	 (03CR) 10Nuria: [C: 03+1] Remove artifacts that are not needed anymore. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/605174 (https://phabricator.wikimedia.org/T254849) (owner: 10Elukey)
[15:19:07] <elukey>	 nuria: sorry I meant upgrade druid by the end of this Q (optimistic deadline, I'll prioritize bigtop testing) and druid+kerberos next Q
[15:19:20] <elukey>	 worst case scenario both next Q
[15:24:32] <mforns>	 pfffffffff...... pyarrow works with my adaptation of LocalExecutor... so no able to reproduce the error :-(
[15:26:15] <nuria>	 mforns: ahrm.. isn't that a GOOD thing
[15:26:17] <nuria>	 ?
[15:26:32] <mforns>	 no, I could not reproduce the error
[15:26:49] <mforns>	 meaning I still don't know why airflow gets stuck with pyarrow
[15:34:34] <wikibugs>	 (03CR) 10Joal: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (039 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[15:34:45] <joal>	 ottomata: a bunch of responses here --^
[15:36:01] <ottomata>	 OH my canary domain comment didn't stick..hm
[15:36:14] <joal>	 arf
[15:36:15] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[16:03:50] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[16:29:57] <wikibugs>	 (03CR) 10Nuria: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[16:36:48] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[16:56:54] * elukey off!
[16:56:58] <elukey>	 have a nice weekend people :)
[17:01:27] <wikibugs>	 (03CR) 10Nuria: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[18:02:15] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[22:52:10] <wikibugs>	 10Analytics-Data-Quality, 10QuickSurveys, 10Research, 10WMDE-Technical-Wishes-Team, and 4 others: Remove Do Not Track support for QuickSurveys - https://phabricator.wikimedia.org/T254224 (10leila) >>! In T254224#6194100, @dr0ptp4kt wrote: > As I recall (@leila please chime in if you have contrary info), in...