[00:09:39] <wikibugs>	 10Analytics, 10Analytics-Kanban: logging level of cassandra should be warning or error but not debug - https://phabricator.wikimedia.org/T236698 (10Nuria) Maybe, maybe  On the job that is executing the ReducerToCasandra the bindings choosen for slf4j are:  ` SLF4J: Class path contains multiple SLF4J bindings....
[00:55:38] <wikibugs>	 10Analytics, 10Product-Analytics: Null username in ServerSideAccountCreation table - https://phabricator.wikimedia.org/T238393 (10Mayakp.wiki)
[00:56:53] <wikibugs>	 10Analytics, 10Product-Analytics: Null username in ServerSideAccountCreation table - https://phabricator.wikimedia.org/T238393 (10Mayakp.wiki)
[01:04:12] <wikibugs>	 10Analytics, 10Product-Analytics: Null username in ServerSideAccountCreation table - https://phabricator.wikimedia.org/T238393 (10Nuria) If this is the schema: https://meta.wikimedia.org/wiki/Schema:ServerSideAccountCreation  Cause :  ` select  event.userId, event.userName from serversideaccountcreation where...
[02:31:59] <wikibugs>	 10Analytics, 10Analytics-Kanban: logging level of cassandra should be warning or error but not debug - https://phabricator.wikimedia.org/T236698 (10Nuria) Ok, tried gazillion exclusion schemes in pom with no sucess
[04:53:23] <wikibugs>	 10Analytics, 10Analytics-Kanban: logging level of cassandra should be warning or error but not debug - https://phabricator.wikimedia.org/T236698 (10Nuria) What is happening here: cassandra loading job is using ch.qos.logback.classic.util.ContextSelectorStaticBinder as logger, we do not have a specific config f...
[06:50:32] <wikibugs>	 10Analytics, 10Pageviews-Anomaly: Manipulation of pageview statistics - https://phabricator.wikimedia.org/T232992 (10Superbass) >>! In T232992#5597291, @MusikAnimal wrote: > Just as with T236121 I think we've done all we can do here. We've identified obvious false traffic and removed it from Topviews. The long...
[07:57:38] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Upgrade to Superset 0.35 - https://phabricator.wikimedia.org/T236690 (10elukey) https://pypi.org/project/apache-superset/ is the new name of the Pypi package, that still shows 0.34.1. It would be really great to start using the pyp...
[08:04:22] * elukey reviews https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/550945/
[08:08:58] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10jcrespo) If you don't plan to recover the data, and it is for archival purposes, that is ok. However I strongly suggest to use...
[08:09:23] <joal>	 Good look with the review elukey :S
[08:14:50] <joal>	 fdans: cron still failing for backfilling :(
[08:24:23] <wikibugs>	 (03CR) 10Elukey: "Thanks a lot for this Joseph!" (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269) (owner: 10Joal)
[08:24:52] <elukey>	 joal: thanks a ton for this work, I know it was super tedious
[08:24:54] <elukey>	 :(
[08:25:06] <elukey>	 I found a couple of nits and I think one issue in webrequest refine
[08:25:15] <elukey>	 but the rest looks good
[08:25:23] <joal>	 elukey: I tried to be concentrated all the time, but probably missed some stuff - Thanks for the review :)
[08:25:39] <elukey>	 I'll do a second pass later on, so easy to miss things
[08:26:00] <joal>	 indeed - a lot of almost exactly the same ... 
[08:26:15] <joal>	 elukey: names and syntax make sense, no need for big change?
[08:28:49] <elukey>	 joal: sorry didn't get the question, what do you mean with big change?
[08:29:06] <joal>	 like if a parameter name would not be good enough
[08:29:31] <elukey>	 nono I think everything looks good
[08:29:44] <elukey>	 and also we tested it in labs
[08:29:50] <elukey>	 err in the test environment
[08:29:57] <joal>	 yes - I replicated all that from labs
[08:30:05] <elukey>	 all good then :)
[08:30:06] <joal>	 yes - test-env
[08:30:16] <elukey>	 doorbell brb!
[08:30:53] <joal>	 elukey: I'm actually gonna do the double-security: update test env with the patch, and start some jobs
[08:32:05] <wikibugs>	 (03CR) 10Joal: "Thanks elukey for the review :)" (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269) (owner: 10Joal)
[08:32:24] <wikibugs>	 (03PS2) 10Joal: Update hive and spark oozie jobs for kerberos [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269)
[08:59:12] <fdans>	 joal: ugh, I changed the $OOZIE_URL but not the $USER. I've changed that too now :(
[08:59:32] <fdans>	 joal: is it ok for me to launch the coord now or you think it's too late in the day?
[09:00:17] <joal>	 you can go for it fdans 
[09:01:48] <fdans>	 thanks joal :)
[09:04:19] <elukey>	 joal: doing a second pass!
[09:10:08] <joal>	 Thanks elukey :)
[09:18:03] <wikibugs>	 (03CR) 10Elukey: Update hive and spark oozie jobs for kerberos (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269) (owner: 10Joal)
[09:30:15] <elukey>	 ok checked with a horrible set of greps, the above should be the last --^
[09:30:24] <elukey>	 both spark and hive2 
[09:30:47] <elukey>	 (I checked for spark and hive2 actions without a above line with an an action + cred)
[09:33:16] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Prepare the Hadoop Analytics cluster for Kerberos - https://phabricator.wikimedia.org/T237269 (10elukey)
[09:35:33] <wikibugs>	 (03CR) 10Joal: Update hive and spark oozie jobs for kerberos (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269) (owner: 10Joal)
[09:36:03] <wikibugs>	 (03PS3) 10Joal: Update hive and spark oozie jobs for kerberos [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269)
[09:53:24] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] Update hive and spark oozie jobs for kerberos [analytics/refinery] - 10https://gerrit.wikimedia.org/r/550945 (https://phabricator.wikimedia.org/T237269) (owner: 10Joal)
[10:04:38] <wikibugs>	 10Analytics: Evaluate (again) Gobblin as possible replacement for Camus - https://phabricator.wikimedia.org/T238400 (10elukey)
[10:05:55] <elukey>	 joal: --^ :)
[10:06:28] <joal>	 Yes elukey :)
[10:52:28] <elukey>	 a-team: there is a friend in town from Dublin, I am going to have lunch with him, will take a longer break today
[11:22:18] <wikibugs>	 10Analytics: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Erik_Zachte) I have just reapplied for server access with John Bond I was supposed to add the new public key myself at https://phabricator.wikimedia.org/T215790, but I can't even view that ticket as Erik_Zachte (ez...
[12:09:34] <wikibugs>	 10Analytics, 10Wikimedia-Stream: EventStreams butcher up some Unicode characters - https://phabricator.wikimedia.org/T198994 (10stjn) In `�ир` missing character was `м`, in `запро�` it was `с`, and in `Хакур�й` it is `е`. The article in the linked diff was called `Миру — мир! (скульптура)`, EventStreams emitte...
[13:11:40] <fdans>	 elukey: when you're back we can take a look at the mod_rewrite for wikistats 2
[13:12:38] <fdans>	 (I mean, if so your heart desires)
[13:41:19] <elukey>	 fdans: I am back!
[13:41:27] <elukey>	 took me a bit, traffic sigh
[13:41:54] <fdans>	 elukey: no rush! we can pair on this whenever you want
[13:43:12] <elukey>	 lemme check a sec the httpd conf
[13:52:08] <elukey>	 so my understanding is that
[13:52:26] <elukey>	 /srv/stats.wikimedia.org/htdocs is the doc root, and it contains an index.html
[13:52:47] <elukey>	 since the httpd allows index in the dir, it is picked up when we hit stats.w.o
[13:55:10] <elukey>	 htdocs also sets FollowSymlink
[13:55:36] <elukey>	 now I am wondering if a simple solution like symlink index.html -> v2/index.html could work
[13:57:37] <elukey>	 fdans: am I crazy?
[13:58:32] <elukey>	 now in theory to prove my thesis, I could just add a elukey.html symlink to /v2/index.html
[13:58:36] <fdans>	 elukey: hmmmmmmmm that should work
[13:58:37] <elukey>	 in htdocs
[14:01:17] <elukey>	 https://stats.wikimedia.org/elukey.html
[14:01:30] <elukey>	 doesn't work, of course it doesn't find css etc..
[14:03:01] <elukey>	 but maybe an Alias?
[14:07:58] <elukey>	 fdans: https://stats.wikimedia.org/elukey.html still blank, but cannot see anymore the 404 for the assets
[14:08:44] <elukey>	 yeah but it seems not really working sorry
[14:10:12] <fdans>	 elukey: yeah it wouldn't find the assets just doing thta
[14:16:27] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Ottomata) Ah!  Good to know.  Will do next time.  Or...I could dump again, the DBs are still up.  Do you think I should?
[14:17:45] <wikibugs>	 10Analytics: Evaluate (again) Gobblin as possible replacement for Camus - https://phabricator.wikimedia.org/T238400 (10Ottomata) +1, we should consider Gobblin again.  I was looking at it the other day a bit too.  I'm not giving up on Kafka Connect yet, but I'm not hopeful.
[14:18:25] <wikibugs>	 10Analytics: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Ottomata) Ah hi Erik!  Ok thank you!
[14:26:01] <wikibugs>	 10Analytics, 10Wikimedia-Stream: EventStreams butcher up some Unicode characters - https://phabricator.wikimedia.org/T198994 (10Ottomata) Hm, right, but what I mean is, the missing `м` looks like it is actually missing from the saved content:  `  curl -s 'https://ru.wikipedia.org/w/api.php?action=compare&fromr...
[14:26:20] <wikibugs>	 (03PS1) 10Fdans: Add top mediarequests metric to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/551187 (https://phabricator.wikimedia.org/T234589)
[14:27:11] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add top mediarequests metric to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/551187 (https://phabricator.wikimedia.org/T234589) (owner: 10Fdans)
[14:28:23] <elukey>	 fdans: we can bc if you have time
[14:28:37] <fdans>	 elukey: omw!
[14:39:06] <wikibugs>	 10Analytics, 10Pageviews-Anomaly: Manipulation of pageview statistics - https://phabricator.wikimedia.org/T232992 (10Steinsplitter) >>! In T232992#5632739, @Superbass wrote: > I would like to add that we now regularly receive complaints in OTRS about this manipulation. It's a bit embarrassing that nothing has...
[14:39:14] <wikibugs>	 10Analytics, 10Wikimedia-Stream: EventStreams butcher up some Unicode characters - https://phabricator.wikimedia.org/T198994 (10stjn) I think you misunderstood what the purpose of that diff was. The bot in the diff consumes EventStreams data and posts the result. These kinds of omissions were present in EventS...
[14:39:23] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Create new eventgate-logging deployment in k8s with helmfile - https://phabricator.wikimedia.org/T236386 (10fgiunchedi) >>! In T236386#5664885, @Ottomata wrote: > Yeah.  Public endpoint!  Which...
[14:44:59] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Create new eventgate-logging deployment in k8s with helmfile - https://phabricator.wikimedia.org/T236386 (10Ottomata) We have to answer this question for eventgate-analytics-external too.  If we...
[15:02:17] <wikibugs>	 10Analytics, 10Pageviews-Anomaly: Manipulation of pageview statistics - https://phabricator.wikimedia.org/T232992 (10Der_Keks) See also https://de.wikipedia.org/wiki/Wikipedia:Umfragen/Umgang_mit_der_Anzeige_von_beliebten_Artikeln
[15:19:13] <wikibugs>	 (03PS2) 10Fdans: Add top mediarequests metric to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/551187 (https://phabricator.wikimedia.org/T234589)
[15:20:11] <wikibugs>	 10Analytics, 10Wikimedia-Stream: EventStreams butcher up some Unicode characters - https://phabricator.wikimedia.org/T198994 (10Ottomata) Ah!  I did misunderstand that, thank you.  Ok I think I get it now.   So the bot consumed an event for revision 93764184 create (right?), which is for a page with the title...
[15:35:07] <wikibugs>	 10Analytics: Rerun pingback reports to categorize software versions correctly. - https://phabricator.wikimedia.org/T238389 (10mforns) Thanks for creating this task @CCicalese_WMF! I completely forgot about the reruns. I already deleted the data that needs rerun. Reportupdater will start computing the missing dat...
[15:54:16] <icinga-wm>	 PROBLEM - Hadoop NodeManager on analytics1077 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration
[15:54:46] <icinga-wm>	 PROBLEM - Hadoop DataNode on analytics1077 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration
[15:55:33] <elukey>	 analytics1077 is probably kaput
[15:56:17] <elukey>	 nope, the oom killer acted
[15:56:19] <elukey>	 this is a first
[15:57:41] <fdans>	 elukey: did I break the things?
[15:57:57] <elukey>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=analytics1077&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics
[15:58:13] <elukey>	 fdans: what do you mean? :D
[15:58:19] <elukey>	 anything that you launched?
[15:58:34] <fdans>	 elukey: I launched monthly top mediarequests backfilling
[15:59:08] <fdans>	 elukey: at around 4pm
[15:59:34] <elukey>	 fdans: like an hour ago?
[15:59:38] <elukey>	 ah ok the cluster is full
[16:00:08] <elukey>	 allocated memory 2031616MB
[16:00:09] <elukey>	 lol
[16:00:32] <fdans>	 elukey: yea the monthly top is pretty savage
[16:00:37] <fdans>	 yikes
[16:00:43] <elukey>	 can we stop it?
[16:01:34] <elukey>	 fdans: --^
[16:02:05] <fdans>	 elukey: hmmmm
[16:02:09] <fdans>	 one second
[16:03:17] <elukey>	 I mean it seems that 2TB of ram requested for a job is a little too much
[16:05:02] <fdans>	 elukey: seems to have gone down now no?
[16:05:11] <fdans>	 (looking at grafana)
[16:07:41] <elukey>	 fdans: https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&panelId=50&fullscreen
[16:07:47] <elukey>	 used memory
[16:08:59] <elukey>	 I guess that the jobs are pulling too much data no?
[16:09:11] <fdans>	 elukey: the thing is
[16:09:24] <fdans>	 it's a huge job that takes about 1:30 to complete
[16:09:53] <fdans>	 elukey: right now it's at 100% map, 46% reduce
[16:10:12] <fdans>	 I don't want to stop it because it doesn't have that much left
[16:10:13] <elukey>	 fdans: let's bc
[16:10:20] <fdans>	 omw
[16:20:33] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata)
[16:21:27] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) Heard some preferences from SREs to use a separate domain.  I'd like to use the same doma...
[16:39:52] <icinga-wm>	 RECOVERY - Check the last execution of hdfs-cleaner on an-coord1001 is OK: OK: Status of the systemd unit hdfs-cleaner https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[16:40:03] <wikibugs>	 10Analytics: Rerun pingback reports to categorize software versions correctly. - https://phabricator.wikimedia.org/T238389 (10CCicalese_WMF) @mforns Thank you!!
[16:42:02] <wikibugs>	 10Analytics, 10Product-Analytics: Null username in ServerSideAccountCreation table - https://phabricator.wikimedia.org/T238393 (10Mayakp.wiki) Hi Nuria, we are using data from event_sanitized database and observed this issue in event_sanitized.serversideaccountcreation
[16:56:56] <joal>	 Heya elukey and fdans
[16:57:12] <joal>	 Have you killed that job finally?
[16:59:12] <fdans>	 joal: we let the thing run but I'm killing the rest of months
[16:59:57] <joal>	 makes sense fdans - I have monitored the thing during the first hour, and it takes a lot of resources but not for too long
[17:03:22] <elukey>	 joal: sometimes it gets to 4TB of ram used :D
[17:03:30] <joal>	 of course it is :)
[17:03:40] <joal>	 hive takes while available
[17:03:45] <elukey>	 I didn't like the oom killer kicking in on one worker node
[17:03:51] <elukey>	 it is a first
[17:03:56] <joal>	 yup, not cool
[17:04:05] <joal>	 I was using the cluster at the same time
[17:05:55] <elukey>	 !log restart hdfs-cleaner (failed due to tmp hive files not present when deleting)
[17:05:56] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:10:21] <wikibugs>	 10Analytics, 10Wikimedia-Stream: EventStreams butcher up some Unicode characters - https://phabricator.wikimedia.org/T198994 (10Iluvatar) Bug was not in specific symbols. Periodically (not always!) EventStream did a defect in random characters of different writing (not only in Cyrillic). One time an error is i...
[17:42:03] <joal>	 nuria !!! I waited for that cassandra explanation :)
[17:42:12] <nuria>	 joal: the DEAL
[17:42:24] <nuria>	 joal: due to the dependency mess that login is 
[17:42:42] <nuria>	 joal: the reducer that loads cassandra  ends up using this logger
[17:43:02] <nuria>	 joal: http://logback.qos.ch/manual/configuration.html
[17:43:06] <nuria>	 joal: "logback"
[17:43:23] <joal>	 ok
[17:43:31] <nuria>	 joal: because we do not have any specific config for logback on our stack (makes sense)
[17:43:33] <joal>	 Bizarre
[17:43:37] <nuria>	 joal: it ends up loging debug
[17:43:41] <joal>	 mwarf
[17:43:42] <joal>	 ok
[17:43:46] <nuria>	 joal: ya, lovely
[17:44:10] <joal>	 nuria: would explicit logging-framework dependency on the cassandra-jar help?
[17:44:23] <nuria>	 joal: I tried to remove that dep (it comes via casandra-all) but it creates a casacading mess on dependency hell
[17:44:51] <joal>	 I can imagine - But explicitely adding a new one, and adding a userJarFirst param to the job?
[17:44:52] <nuria>	 joal: not sure if we are saying same thing, i tried <exclusion>
[17:45:10] <nuria>	 joal: for ALL login from refinery-cassandra
[17:45:16] <joal>	 yup
[17:45:20] <nuria>	 joal: and that gives a guava erroro
[17:45:36] <nuria>	 joal: so the why i think i know, the how to fix it not yet
[17:45:53] <joal>	 nuria: ok - What a mess ...
[17:46:02] <nuria>	 joal: ya , cassandra mess
[17:46:09] <nuria>	 joal: it is fixable for sure
[17:46:19] <nuria>	 joal: i just have not figured out how to fix it
[17:49:07] <joal>	 nuria: as I suggested above, I suggest explicitely adding a logging fraemwork to our jar deps, and run the job with userJarFirst parameter (or userClassPathFirst, can't recall)
[17:54:47] <joal>	 mforns: if you;re nearby
[17:55:04] <mforns>	 joal, in meeting, ping you after!
[17:55:22] <joal>	 might be gone - letting you know I think I have a working solution for you
[17:55:30] <joal>	 mforns: for notebooks and hive
[17:59:29] <joal>	 And I am also experiencing a very weird thing :( Using the geocoded_data function on random webrequest data has failed for me :(
[17:59:56] <wikibugs>	 10Analytics, 10Analytics-Kanban: Make stats.wikimedia.org point to wikistats2 by default - https://phabricator.wikimedia.org/T237752 (10elukey) First of all, inside the Virtual host there are Directory entries that are probably super stale:  `     # Allow CGI scripts for this site     <Directory "/srv/stats.wi...
[18:05:20] <wikibugs>	 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Pamputt) This task has been marked as resolved, so now what about T188859? Will this information be available on https://stats.wikimedia.org/v2/#/all-pr...
[18:05:50] <wikibugs>	 10Analytics, 10Analytics-Kanban: Make stats.wikimedia.org point to wikistats2 by default - https://phabricator.wikimedia.org/T237752 (10elukey) The other idea that me and Fran had about moving on with the stats.w.o -> v2 transition is the following:  1) Add something like the following to the httpd config:  `...
[18:06:04] <elukey>	 fdans: --^
[18:07:29] <elukey>	 check https://stats.wikimedia.org/elukey.html :)
[18:07:33] <elukey>	 I am testing it now
[18:07:52] <joal>	 Ok I got it - we have a funny edge case elukey :)
[18:08:01] <elukey>	 elukey@thorium:/srv/stats.wikimedia.org/htdocs$ ls -l elukey.html
[18:08:02] <elukey>	 lrwxrwxrwx 1 elukey wikidev 13 Nov 15 15:24 elukey.html -> v2/index.html
[18:08:15] <elukey>	 joal: wikistats or elsewhere?
[18:08:26] <joal>	 elukey: let me know when you have enough bandwidth for something else :)
[18:08:32] <joal>	 nono no wikistats
[18:08:47] <elukey>	 yes please let me know
[18:09:32] <elukey>	 joal: --^
[18:09:34] <joal>	 elukey: running the geocoded_data UDF at small scale as an example with HiveServer2 fails
[18:09:55] <joal>	 Other UDFs work (AFAICT), but that one fails
[18:10:06] <joal>	 I can predict the ISP one will fail as well
[18:10:56] <joal>	 root-cause: for queries on a very small number of rows ( for instance SELECT geocoded_data(ip) FROM wmf.webrequest WHERE webrequest_source = 'text' and year=2019 and month=11 and day=1 and hour=0 LIMIT 10)
[18:11:45] <joal>	 A Job-local is chosen - And I think we don't have the needed DB replicated on an-coord1001 
[18:11:59] <joal>	 The maxmind files I mean elukey 
[18:12:27] <elukey>	 ah ok so something solvable right?
[18:12:49] <joal>	 Very much I think: If we install the maxmind DBs on an-coord1001 :)
[18:12:50] <elukey>	 (restored wikistats config fdans, but it worked :)
[18:13:09] <elukey>	 makes sense, shouldn't be an issue
[18:13:17] <elukey>	 let's do it on Monday 
[18:13:20] <joal>	 That would be great elukey :)
[18:13:27] <joal>	 Thanks a lot! Creating a task
[18:14:37] <elukey>	 +1
[18:14:59] <elukey>	 for a moment I thought that you found a blocker for kerberos
[18:14:59] <elukey>	 :D
[18:15:50] <wikibugs>	 10Analytics, 10Product-Analytics: Null username in ServerSideAccountCreation table - https://phabricator.wikimedia.org/T238393 (10Mayakp.wiki) 05Open→03Resolved a:03Mayakp.wiki We were able to identify the root cause : as a part of T222101 Morten changed the whitelisting for tracking isApi and removed us...
[18:16:27] <elukey>	 going afk! o/
[18:16:33] <elukey>	 have a good weekend :)
[18:17:39] <joal>	 Same for me - gone :) See you on monday team
[18:18:04] <wikibugs>	 10Analytics, 10Analytics-Kanban: Add MaxMind DB files on an-coord1001 for hive local-jobs using UDF to succeed - https://phabricator.wikimedia.org/T238432 (10JAllemandou)
[18:24:40] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Research-Backlog, 10Product-Analytics (Kanban): Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10LGoto)
[18:27:07] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Product-Analytics, 10Research-Backlog: Identify next steps needed for Product Analytics to approve switching mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T238434 (10kzimmerman)
[18:29:50] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Product-Analytics, 10Research-Backlog: Identify next steps needed for Product Analytics to approve switching mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T238434 (10kzimmerman) p:05Triage→03Normal
[18:34:51] <wikibugs>	 10Analytics: Add editors per country data to AQS API - https://phabricator.wikimedia.org/T238365 (10Nuria)
[18:34:53] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: Add statistics for the geographical origin of the contributors - https://phabricator.wikimedia.org/T188859 (10Nuria)
[18:35:36] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) - stream-intake.wikimedia.org  ?
[18:35:44] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256 (10Nuria)
[18:35:46] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: Add statistics for the geographical origin of the contributors - https://phabricator.wikimedia.org/T188859 (10Nuria) 05duplicate→03Open
[18:37:09] <wikibugs>	 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Nuria) @pamputt at some point yes. Now, that update will not happen in the next couple of quarters so for now the data will only exist is public files....
[18:38:59] <wikibugs>	 10Analytics, 10Analytics-Kanban: Make stats.wikimedia.org point to wikistats2 by default - https://phabricator.wikimedia.org/T237752 (10Nuria) +1 to dropping those configs. Geowiki data was archived a whileback: https://wikitech.wikimedia.org/wiki/Analytics/Archive/Geowiki#On_the_hadoop_cluster_(foundation_int...
[18:55:19] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) More brain bouncing with Jason.  I think we like:  - stream.wikimedia.org/produce/$instan...
[18:56:56] <joal>	 mforns: example of what I have working on adding jar on a notebook: https://gist.github.com/jobar/d697f3fa54fa9e85131e4bbf1e83ce5a
[19:02:26] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10jlinehan) I think this is great because it makes the API actually readable in a meaningful way that...
[19:03:50] <mforns>	 thanks joal will look into it after dinner :]
[19:20:20] <wikibugs>	 10Analytics, 10Event-Platform, 10Operations, 10Wikimedia-Logstash, 10observability: Move eventgate logs to new logging infrastructure - https://phabricator.wikimedia.org/T225129 (10Ottomata) Ah, I think the deployment-chart for eventgate already has this done.  We just need to deploy.  Will do soon enoug...
[20:28:13] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Research-Backlog, 10Product-Analytics (Kanban): Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10Nuria) I think (if #performance-team agrees) this work can be done by @jlinehan as part of th...
[20:45:49] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Research-Backlog, 10Product-Analytics (Kanban): Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10jlinehan) >>! In T223931#5668049, @Nuria wrote: > I think (if #performance-team agrees) this...
[20:48:26] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) Ok, TLS everywhere, right?  Got some new patches up.  I'd like to start merging these nex...
[20:49:03] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, and 4 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) Hmmm...@fgiunchedi do your various Kafka logging producers use Kafka TLS?  We haven't don...
[21:19:14] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Research-Backlog, 10Product-Analytics (Kanban): Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10Nuria) >it seems like HTML5 localStorage (link), which does not expire on tab closure, and is...
[21:23:35] <wikibugs>	 (03CR) 10Nuria: "Let's hold on deploying these changes until we have the data loaded in cassandra, otherwise the treads for data are not going to help with" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/551187 (https://phabricator.wikimedia.org/T234589) (owner: 10Fdans)
[21:35:22] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Research-Backlog, 10Product-Analytics (Kanban): Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10jlinehan) >>! In T223931#5668164, @Nuria wrote: > Problem is that localStorage does not work...
[22:23:03] <wikibugs>	 10Analytics, 10Pageviews-Anomaly: Manipulation of pageview statistics - https://phabricator.wikimedia.org/T232992 (10MusikAnimal) >>! In T232992#5666267, @Superbass wrote: >>>! In T232992#5597291, @MusikAnimal wrote: >> Just as with T236121 I think we've done all we can do here. We've identified obvious false...