[05:07:04] <wikibugs>	 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Cirdan) >>! In T203826#4572491, @Milimetric wrote: > Ultimately we want to have a lot more data available via this and similar APIs, and as we evolve towar...
[07:37:27] <wikibugs>	 10Analytics, 10Analytics-Cluster: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey)
[07:52:16] <elukey>	 https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_os_requirements.html#cdh_cm_supported_os
[07:52:19] <elukey>	 lovely
[07:52:36] <elukey>	 it seems that CDH6 has no official support for Debian, but only ubuntu Xenial
[07:54:03] <wikibugs>	 10Analytics, 10Analytics-Cluster: Update to CDH 6 or other up-to-date Hadoop distribution - https://phabricator.wikimedia.org/T203693 (10elukey)
[08:02:13] <joal>	 nice
[08:04:24] <elukey>	 yeah :D
[08:04:26] <elukey>	 morniinggg
[08:04:30] <addshore>	 MORNING
[08:04:38] <elukey>	 YO ADAM
[08:05:10] <addshore>	 Yo LUCA!
[08:05:36] <joal>	 Yo YO ?
[08:57:28] <elukey>	 joal: any concern if I move the cron to drop webrequest sampled data from hdfs to a timer?
[09:10:34] <joal>	 elukey: no prob - I thought we didn't have sampled data anymore\/
[09:10:37] <joal>	 ?
[09:13:00] <elukey>	 joal: in druid? How so?
[09:13:10] <joal>	 Ah !
[09:13:14] <joal>	 In druid - My bad
[09:13:25] <elukey>	 ah snap I missed to mention it
[09:13:34] <elukey>	 sorry I am going through the list of puppet crons
[09:15:52] <joal>	 no prob, sounds good
[09:39:37] <elukey>	 deployed! Looks working fine
[09:42:29] <joal>	 \o/
[10:00:44] <elukey>	 joal: if you want to check, you can do sudo journalctl -u refinery-drop-webrequest-sampled-druid.service
[10:00:47] <elukey>	 on analytics1003
[10:00:56] <elukey>	 there is not much since we use --quiet
[10:01:10] <elukey>	 but you can see the logs (you should be able to sudo journalctl now)
[10:01:30] <joal>	 elukey: indeed ! Tells me it has started :)
[10:18:29] <elukey>	 super :)
[10:19:46] <elukey>	 joal: you should also be able to do systemctl status refinery-drop-webrequest-sampled-druid.service
[10:20:12] <elukey>	 that should say that the unit is inactive (now) but
[10:20:13] <elukey>	 Process: 25228 ExecStart=/srv/deployment/analytics/refinery/bin/refinery-drop-druid-deep-storage-data -d 60 webrequest_sampled_128 (code=exited, status=0/SUCCESS)
[10:20:43] <elukey>	 then we have
[10:20:45] <elukey>	 elukey@analytics1003:~$ systemctl list-timers
[10:20:45] <elukey>	 NEXT                         LEFT          LAST                         PASSED    UNIT                                         ACTIVATES
[10:20:48] <elukey>	 Tue 2018-09-11 14:51:15 UTC  4h 30min left Mon 2018-09-10 14:51:15 UTC  19h ago   systemd-tmpfiles-clean.timer                 systemd-tmpfiles-clean.service
[10:20:51] <elukey>	 Wed 2018-09-12 05:15:00 UTC  18h left      n/a                          n/a       refinery-drop-webrequest-sampled-druid.timer refinery-drop-webrequest-sampled-druid.service
[10:20:55] <elukey>	 Wed 2018-09-12 10:00:00 UTC  23h left      Tue 2018-09-11 10:00:00 UTC  20min ago check_webrequest_partitions.timer            check_webrequest_partitions.service
[10:20:58] <elukey>	 Wed 2018-09-12 10:10:00 UTC  23h left      Tue 2018-09-11 10:10:00 UTC  10min ago check_pageviews_partitions.timer             check_pageviews_partitions.service
[10:21:29] <elukey>	 so the refinery-drop-webrequest-sampled-druid timer has not been started yet (I've run it manually with systemctl start refinery-drop-webrequest-sampled-druid to test)
[10:21:49] <elukey>	 meanwhile the other two have been executed 20 mins ago
[10:21:57] <joal>	 that's really great!!
[10:22:13] <elukey>	 looks better right?? 
[10:22:26] <joal>	 elukey: My non-knowledge of systemd asks for some wikidocs :(
[10:22:41] <elukey>	 for sure, I am going to write some this afternoon
[10:22:52] <joal>	 Many many thanks :)
[10:22:57] <joal>	 This will really help a lot
[10:23:19] <elukey>	 I just wanted to show some things to get some feedback ("this thing is awful! This looks fine!" eetc.. :)
[10:25:34] <elukey>	 and this script seems to have all the exit codes set correctly
[10:29:59] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Negative total number of bytes for German Wikipedia in 2001? - https://phabricator.wikimedia.org/T203906 (10Habitator_terrae) You understand my question in the right way. I wanted to make a graph of the data in the de.wikipedia.org content namespace and I was very confused....
[10:37:22] <elukey>	 lunch break!
[10:39:08] <wikibugs>	 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban, 10CX-analytics, 10Language-2018-July-September: Setup Config:Dashiki:CX2Translations as a public chart and update the Dashiki documentation accordingly - https://phabricator.wikimedia.org/T203516 (10Amire80) Oh, thanks! It looks good, indeed.  My last...
[12:03:59] <wikibugs>	 (03PS1) 10Joal: Update python/refinery/utils/HdfsUtils [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489)
[12:05:15] <wikibugs>	 (03PS3) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489)
[12:09:31] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Negative total number of bytes for German Wikipedia in 2001? - https://phabricator.wikimedia.org/T203906 (10JAllemandou) The closest to total number of bytes we have is the metric you have checked. As explained previously, the negative aspect of it is due to historical chang...
[12:29:58] <wikibugs>	 (03CR) 10Mforns: "Left a couple annoying inline comments :]" (037 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/458784 (https://phabricator.wikimedia.org/T203180) (owner: 10Fdans)
[13:01:35] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥1.13 or ≥2.1 - https://phabricator.wikimedia.org/T203498 (10mpopov) >>! In T203498#4571941, @Neil_P._Quinn_WMF wrote: > @mpopov, I'm actually confused now. I'm looking at [Hive downloads page](https://hive.apache.org/downloads.html), which has the best version...
[13:14:40] <wikibugs>	 10Analytics, 10Analytics-Kanban: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Nuria) Super fun and all, Ijust saying that  think all this info is on the MySQL consumer log, as we have those for the past month.
[13:15:01] <wikibugs>	 (03CR) 10Nuria: "Thanks for taking care of this" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/459637 (owner: 10Milimetric)
[13:16:40] <wikibugs>	 (03CR) 10Nuria: [C: 032] Fix mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459496 (owner: 10Joal)
[13:17:04] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] Add proxy to namespace-downloader script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/458860 (owner: 10Joal)
[13:18:53] <wikibugs>	 (03CR) 10Mforns: [C: 031] "Looks good to me! Didn't find any typo in the renames. The only things I didn't find a matching code snippet in the new code, are the impl" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata)
[13:25:11] <wikibugs>	 (03PS11) 10Ottomata: Add ConfigHelper trait to auto load config files and CLI opts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415174 (https://phabricator.wikimedia.org/T203804)
[13:25:48] <wikibugs>	 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10Nuria) >Completely agree for the reasons you mention! Right know however we a...
[13:27:09] <ottomata>	 joal addressed your final comment about tuple val names :)
[13:32:51] <ottomata>	 wow elukey i think i didn't realize the datacenter switch is toda
[13:32:51] <ottomata>	 y
[13:32:55] <ottomata>	 is that true?
[13:39:27] <elukey>	 ottomata: only services, tomorrow is mediawiki/traffic/etc..
[13:41:34] <elukey>	 morninggg
[13:42:22] <ottomata>	 cool, so we need to make sure we watch codfw eventbus, etc.
[13:42:25] <ottomata>	 mirror maker
[13:42:33] <wikibugs>	 (03CR) 10Mforns: [C: 031] "LGTM! But didn't test..." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal)
[13:49:47] <wikibugs>	 (03CR) 10Ottomata: "Comments inline." (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal)
[13:50:50] <wikibugs>	 (03CR) 10Ottomata: "Hm, I just left comments on this file in https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/456654 :D" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal)
[13:51:43] <wikibugs>	 (03CR) 10Ottomata: "Ya, implicits are in https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/415174/11/refinery-core/src/main/scala/org/wikimedia/a" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata)
[13:56:42] <ottomata>	 heya milimetric yt?
[13:56:52] <milimetric>	 hey ottomata yep
[13:57:04] <ottomata>	 q about rfc stuff
[13:57:10] <milimetric>	 yea
[13:57:12] <ottomata>	 it is very unclear how long one has to wait to get anything done
[13:57:22] <ottomata>	 i submitted the RFC for event intake weeks ago
[13:57:27] <ottomata>	 but have no idea what the status is
[13:57:34] <ottomata>	 and i just saw that it isn't scheduled for the RFC meeting this week
[13:57:41] <milimetric>	 lemme look at the board
[13:57:49] <ottomata>	 our goals for this quarter are kinda to mostly settle all these RFCs, i'm getting a little worried
[13:57:52] <milimetric>	 https://phabricator.wikimedia.org/project/board/52/
[13:57:54] <ottomata>	 the schema registry one is just sitting aroudn too
[13:58:00] <ottomata>	 its unclear what happens next
[13:58:10] <milimetric>	 so it's in the request IRC meeting along with 4 others
[13:58:19] <milimetric>	 basically they'll run through them as fast as they can
[13:58:38] <ottomata>	 ok
[13:58:45] <ottomata>	 but there can only be one a week?
[13:58:48] <milimetric>	 I've missed the last couple of meetings in jury duty, but don't look at this as a blocker
[13:59:07] <milimetric>	 yeah, it's like 2am in Australia or something, so they only do 1 hour after the Tech Com meeting
[13:59:36] <milimetric>	 but don't look at this as a blocker
[14:00:09] <milimetric>	 we can start making a project plan and even make some coding progress without a decision either way, maybe?
[14:00:13] <wikibugs>	 10Analytics, 10Analytics-Kanban: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Ottomata) That's true it would be! :o  It would also be in the eventlogging-valid-mixed files too.
[14:02:49] <wikibugs>	 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10Ottomata) > EL is really not the best tool to do error logging  Eh?  Why not?...
[14:03:08] <wikibugs>	 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Milimetric) >>! In T203826#4573076, @Cirdan wrote: >>>! In T203826#4572491, @Milimetric wrote: >> Ultimately we want to have a lot more data available via...
[14:14:23] <wikibugs>	 10Analytics, 10Analytics-Kanban: Test role::analytics_cluster::coordinator on Debian Stretch - https://phabricator.wikimedia.org/T204060 (10elukey) p:05Triage>03Normal
[14:17:30] <wikibugs>	 10Analytics, 10Analytics-Kanban: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Ottomata) Looking in the consumer logs as you suggest was very easy (but less fun :p).  Between Aug 26 06:50:58 (our oldest consumer log) and now,...
[14:19:56] <wikibugs>	 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban, 10CX-analytics, 10Language-2018-July-September: Setup Config:Dashiki:CX2Translations as a public chart and update the Dashiki documentation accordingly - https://phabricator.wikimedia.org/T203516 (10fdans) @Amire80 it seems then that you've been skipp...
[14:32:45] <elukey>	 joal: is there any reason why we have the webrequest misc coordinator still doing things?
[14:40:14] <elukey>	 ottomata: 
[14:40:15] <elukey>	  Switching services parsoid, restbase, restbase-async, mobileapps, apertium, citoid, cxserver, eventstreams, graphoid, mathoid, proton, pdfrender,
[14:40:18] <elukey>	                    recommendation-api, zotero, eventbus, ores, wdqs, wdqs-internal: eqiad => codfw (akosiaris@sarin)
[14:40:23] <elukey>	 (from #operations)
[14:43:09] <enick_847>	 Hello! Are beta cluster EventLogging events available in Grafana? For example, on prod I can see https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=VirtualPageView. I was trying to duplicate the dashboard (just hardcoding VirtualPageView for now) but have been unsuccessful and am uncertain if the data is even connected there. Here's my dummy board:
[14:43:09] <enick_847>	 https://grafana-labs.wikimedia.org/dashboard/db/niedzielski-eventlogging-schema?var-schema=VirtualPageView
[14:44:06] <ottomata>	 elukey:  thanks.  
[14:44:31] <ottomata>	 elukey:  also, there is no reason i know of, except we didn't turn it off: https://phabricator.wikimedia.org/T200822
[14:45:07] <elukey>	 ottomata: ah okok nice, just wanted to know if you guys decided something that I wasn't aware of during my holiday
[14:45:26] <ottomata>	 enick_847:  your dash is in labs
[14:45:34] <ottomata>	 there are events in labs
[14:45:40] <ottomata>	 but the kafka cluster names will be different
[14:45:43] <ottomata>	 and teh counts will be different too
[14:45:49] <ottomata>	 its a totally different environment
[14:46:13] <enick_847>	 ottomata: is there a beta cluster dashboard?
[14:47:00] <ottomata>	 enick_847:  no we don't maintain dashboards in both places
[14:49:20] <wikibugs>	 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban, 10CX-analytics, 10Language-2018-July-September: Setup Config:Dashiki:CX2Translations as a public chart and update the Dashiki documentation accordingly - https://phabricator.wikimedia.org/T203516 (10Amire80) 05Open>03Resolved //Alá vai o meu heroe...
[14:49:33] <enick_847>	 ottomata: is the beta cluster data source available in the prod grafana? i saw a "beta prometheus" source on the labs instance.
[14:51:31] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: "Total Article Count" Wikistats  metric (per project and overall) - https://phabricator.wikimedia.org/T198425 (10fdans)
[14:52:23] <wikibugs>	 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10MusikAnimal) For the record, XTools allows users to [[ https://xtools.readthedocs.io/en/stable/opt-in.html#how-to-opt-in | opt-in to restricted statistics...
[14:52:53] <milimetric>	 mforns: got a few minutes to jump in the cave early?
[14:53:00] <mforns>	 milimetric, sure!
[14:53:01] <mforns>	 omw
[14:54:44] <ottomata>	 enick_847:  not sure i understand, but i thin what you want should be
[14:55:00] <ottomata>	 the kafka cluster name will be different, but the same metrics should be in deployment-prep beta as in prod
[14:55:03] <ottomata>	 just from different clusters
[14:55:26] <ottomata>	 i believe the name of the kafka cluster in beta is 'main-deployment-prep'
[14:55:29] <ottomata>	 OH
[14:55:31] <ottomata>	 no eventlogging
[14:55:32] <ottomata>	 hmmm
[14:55:33] <ottomata>	 maybe
[14:55:36] <ottomata>	 jumbo-deployment-prep
[14:56:49] <wikibugs>	 (03CR) 10Joal: [C: 031] "LGTM ! Nice patch :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415174 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata)
[14:57:49] <enick_847>	 ottomata: this is in the prod grafana?
[14:58:46] <ottomata>	 ?
[14:58:48] <ottomata>	 enick_847:  no
[14:59:07] <ottomata>	 grafana in labs uses labs datasources
[14:59:11] <ottomata>	 grafana in prod uses prod datasources
[14:59:39] <ottomata>	 in labs in deployment-prep, there are a couple of kafka clusters with different names
[14:59:43] <joal>	 elukey: indeed no good reason for misc still being here - However it looks like there still is data flowing in 
[14:59:53] <ottomata>	 the deployment-prep/beta eventlogging instance uses the kafka cluster called 'jumbo-deployment-prep'
[15:00:12] <ottomata>	 it should be in the deployment-prep (or labs?) prometheus datasource  not sure if there is one or not but I'd assume there is
[15:00:26] <ottomata>	 in prod, it is in the ops eqiad datasource in a kafka cluster named 'jumbo-eqiad'
[15:00:49] <enick_847>	 ottomata: and the beta cluster is considered part of labs?
[15:01:27] <elukey>	 joal: mmmm very weird, the misc cluster is gone
[15:01:33] <joal>	 :-S
[15:01:39] <ottomata>	 enick_847:  yes
[15:01:44] <joal>	 elukey: when?
[15:01:55] <ottomata>	 i don't know if the prometheus/grafana stuff in labs is deployment-prep( aka beta) specific, or for all of labs
[15:02:00] <elukey>	 joal: a while ago, lemme pull the task
[15:02:44] <milimetric>	 ottomata: reign of terror?
[15:03:17] <elukey>	 joal: https://phabricator.wikimedia.org/T164609
[15:03:18] <ottomata>	 OH MY
[15:03:19] <milimetric>	 (standup ottomata / elukey )
[15:03:23] <elukey>	 comingggg sorry
[15:03:40] <elukey>	 joal: will follow up with e*ma on this
[15:04:23] <joal>	 Thanks elukey - Will be good to remove the misc joib :)
[15:36:11] <enick_847>	 ottomata: so i have "Tools Prometheus" with jumbo-deployment-prep kafka and I don't see any events. I don't want to take up too much of your time but is there something else I should check?
[15:38:42] <ottomata>	 Not sure what Tools is, but i wouldn't be surprised if it was separate from deployment prep (or regular labs?) prometheus
[15:42:03] <enick_847>	 Yeah, my options appear to be limited to: Beta Prometheus, default, Labs Graphite, Labs JSON tool, Prod Graphite, Tools Prometheus, -- Mixed --, and -- Grafana --.
[15:43:41] <ottomata>	 Beta Prometheus sounds good!
[15:43:49] <ottomata>	 that should be a prometheus instance in beta / deployment-prep
[15:46:19] <ottomata>	 enick_847:  sorry i'm pretty unfamiliar with the beta/labs monitoring setup, they might be able to help you figure that out more in #wikimedia-cloud, or maybe even #wikimedia-operations (ping godog there probably)
[15:46:33] <ottomata>	 i don't seem to be able to log in or create dashboards at grafana-labs
[15:46:45] <ottomata>	 there should be a prometheus web interface you could explore, but id ont' know the machine:
[15:46:48] <ottomata>	 https://wikitech.wikimedia.org/wiki/Prometheus#Access_Prometheus_web_interface
[15:46:53] <ottomata>	 or if you or I have access
[15:46:53] <enick_847>	 Well, that was one I tried first but I don't even get graph axes drawn when I select it so I think that indicates that something is wrong with the data source and query combo.
[15:47:20] <enick_847>	 well you gotta go to https://grafana-labs-admin.wikimedia.org to make changes it seems
[15:47:31] <ottomata>	 OHHH RIGHT
[15:48:00] <ottomata>	 enick_847:  do I sign in wth my wikitech login?
[15:48:16] <wikibugs>	 (03CR) 10Ottomata: [C: 032] Add ConfigHelper trait to auto load config files and CLI opts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415174 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata)
[15:48:30] <enick_847>	 i think so but i'm unsure
[15:49:34] <enick_847>	 ottomata: I think I've about popped the lid on my own timebox so I'm going to move on for the time being. Thanks for your help though!
[15:52:52] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10Cmjohnson)
[16:01:50] <mforns>	 ping joal?
[16:03:50] <joal>	 yes
[16:06:47] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: Decommission Ganeti vm meitnerium.wikimedia.org (old Archiva host) - https://phabricator.wikimedia.org/T203087 (10Krenair) No switch port disabling step for VMs either
[16:07:08] <ottomata>	 elukey:  is the camus checker email alert i just got you testing things?
[16:07:11] <ottomata>	 OH!
[16:07:11] <ottomata>	 not!
[16:07:16] <ottomata>	 that is datacenter switch!
[16:07:17] <ottomata>	 COOL!
[16:07:25] <ottomata>	 it works! :o
[16:08:20] <joal>	 \o/
[16:17:43] <elukey>	 ottomata: can I ask you a question about https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/459807/ ?
[16:18:40] <elukey>	 I don't get why we use a regex instead of a simple if schema_name in  etc..
[16:18:52] <elukey>	 (if I got the change correctly)
[16:23:52] <elukey>	 Riccardo already commented in the code review (he was the one asking for it in the first place)
[16:26:51] <wikibugs>	 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10pmiazga) IMHO we should enable this only to users without gadgets (maybe anon...
[16:29:50] <ottomata>	 elukey:  commented there
[16:29:55] <ottomata>	 but, mainly because we used to have that feature
[16:29:59] <ottomata>	 some of the blacklists were regexes
[16:30:00] <ottomata>	 wanted to keep  it
[16:30:07] <ottomata>	 guess since it is all in this file though
[16:30:12] <ottomata>	 we can change the code whenever we need
[16:30:20] <ottomata>	 could siwtch in array contains if you think that is better
[16:33:14] <elukey>	 ottomata: no strong opinion, from the code point of view it looks clearer to simply check into a list, but please do as you prefer :)
[16:33:31] <elukey>	 no real opposition, just wanted to understand the code flow if I have to modify it :)
[16:35:53] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10Ottomata)
[16:38:18] <wikibugs>	 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10ovasileva)
[16:46:23] <joal>	 mforns: may i delay debrief by 1/2hour?
[17:16:16] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999)
[17:16:19] <wikibugs>	 10Analytics, 10DBA: mariadb::service and managed services don't play well on Stretch - https://phabricator.wikimedia.org/T204074 (10elukey) p:05Triage>03Normal
[17:17:18] <elukey>	 ottomata: --^
[17:17:55] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999)
[17:18:09] <elukey>	 joal: I am using kafkacat -b kafka-jumbo1001.eqiad.wmnet:9092 -t webrequest_misc from stat1004, I can only see health checks in there
[17:18:41] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999)
[17:19:20] <wikibugs>	 10Analytics: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete - https://phabricator.wikimedia.org/T200822 (10elukey) After checking with `kafkacat -b kafka-jumbo1001.eqiad.wmnet:9092 -t webrequest_misc` I can only see health checks flowing from Kafka.  @ema just t...
[17:23:28] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999)
[17:26:37] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999)
[17:26:53] <wikibugs>	 (03PS1) 10Elukey: Remove cache misc from Refinery [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459827 (https://phabricator.wikimedia.org/T200822)
[17:30:06] <wikibugs>	 10Analytics: Better redirect handling for pageview API - https://phabricator.wikimedia.org/T121912 (10bd808) One very concrete use case for this is {T185202}
[17:31:04] <elukey>	 all right logging off!
[17:31:08] <elukey>	 o/
[17:46:43] <ottomata>	 bye!
[17:48:48] <wikibugs>	 (03CR) 10Joal: "LGTM - Didn't double check for other instances." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459827 (https://phabricator.wikimedia.org/T200822) (owner: 10Elukey)
[17:49:16] <wikibugs>	 10Analytics, 10Community-consensus-needed: Decide whether enable per-editor edits stats (community decision) - https://phabricator.wikimedia.org/T203826 (10Cirdan) >>! In T203826#4574147, @Milimetric wrote: > The reason is because this kind of thing already happens and the data is sometimes really wrong.  So m...
[18:16:37] <wikibugs>	 (03PS5) 10Ottomata: Use ConfigHelper for RefineMonitor instead of scopt [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804)
[18:17:45] <wikibugs>	 (03PS6) 10Ottomata: Use ConfigHelper for RefineMonitor instead of scopt [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804)
[18:26:00] <wikibugs>	 (03CR) 10Joal: [C: 031] "Good for me :) Did not test though" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804) (owner: 10Ottomata)
[18:37:33] <ottomata>	 joal:  i have an issue with that sorta...i think we should/need to support required properties
[18:37:37] <ottomata>	 trying to figure that out nicely
[18:37:55] <ottomata>	 i can take the defaults out of the case class fine, and an exception will be thrown
[18:37:57] <ottomata>	 but it isn't a nice one
[18:38:05] <ottomata>	 e.g.
[18:38:05] <ottomata>	 Caused by: DecodingFailure(Attempt to decode value on failed cursor, List(DownField(input_path)))
[18:38:18] <ottomata>	 with a large stack trace and context of everywhere Profig might have loaded stuff from
[18:39:28] <joal>	 ottomata: making sure I understand- We want to fail in case some properties are NOT passed to profig (whether throuhg file or param)
[18:39:49] <joal>	 ottomata: meaning, enforcing certain properties to be set
[18:40:14] <ottomata>	 yes
[18:40:27] <ottomata>	 and hopefully print nicely if missing
[18:41:18] <joal>	 right
[18:43:10] <joal>	 ottomata: catching the exception?
[18:43:20] <joal>	 ottomata: I can't really think of something better
[18:43:21] <ottomata>	 yeah i'm trying that now
[18:43:33] <ottomata>	 getting the field name is deep in the Circe stuff, and I can probalby do it for our simple use case
[18:44:04] <joal>	 ottomata: would be nice to PR upstram!
[18:48:20] <ottomata>	 joal:  what's a nice way to pattern match RuntimeException(_, DecodingFailure(_, List(DownField(fieldName))) ? :p
[18:48:35] <mforns>	 milimetric, I'm about to review annotations, is there an existing annotation page that I can plug in to test?
[18:48:36] <ottomata>	 RuntimeException .getCause() is a DecodingFailure
[18:48:37] <joal>	 wow
[18:48:41] <ottomata>	 which has a message and a history
[18:48:46] <ottomata>	 the history is from circe
[18:48:58] <ottomata>	 which in the case I want is a List(DownField(fieldName))
[18:49:21] <ottomata>	 i can do multi level match
[18:49:25] <ottomata>	 would be nice to do in one line :p
[18:50:18] <joal>	 ottomata: I'm assuming you've tried pattern matching as above :)
[18:50:56] <ottomata>	 yeah, i think maybe RuntimeException doesn't have unapply?
[18:51:25] <ottomata>	 oh because its from java?
[18:51:51] <joal>	 sounds a correct answer
[18:52:03] <ottomata>	 haha ok thought maybe you'd know better :p
[18:54:23] <joal>	 Nope - I don't use exception :)
[18:54:38] <joal>	 Except for exceptions
[19:10:38] <joal>	 ottomata: I have a questiomn on my rsyn
[19:10:47] <joal>	 if you have a minute of brainpowah
[19:12:31] <ottomata>	 sure!
[19:12:36] <ottomata>	 joal bc or irc?
[19:12:43] <joal>	 ottomata: can be here :)
[19:12:51] <ottomata>	 either is fine!
[19:13:11] <joal>	 I'm about to add a parameter: fix_corrupted, using should_delete for destinatio
[19:13:27] <joal>	 destination cleaning, and fix_corrupted for well, corruption fix :)
[19:13:33] <joal>	 Good for you?
[19:15:18] <ottomata>	 ahhh, before fix did both?
[19:15:27] <joal>	 correct
[19:15:34] <ottomata>	 in this case corrupted means what?  just different?
[19:15:36] <ottomata>	 different sizes?
[19:15:53] <joal>	 different-types or different sizes
[19:16:08] <joal>	 (sizes for files only, I don't monitor sub-folders)_
[19:16:20] <ottomata>	 hmm, i woudln't call that 
[19:16:22] <ottomata>	 corruption
[19:16:45] <ottomata>	 joal:  that should be the default behavior, no?  unless dry_run is true?
[19:17:11] <joal>	 ottomata: rsync is build with the idea of incremental imports
[19:17:21] <joal>	 ottomata: in my case, not really :)
[19:17:31] <joal>	 I can make it default behavior for sure
[19:18:35] <joal>	 ottomata: you think it's better not to have a flag?
[19:18:38] <joal>	 very feasible
[19:19:56] <ottomata>	 joal:  it isn't the default behavior in your case?
[19:19:59] <ottomata>	 oh in your case its corruption
[19:20:29] <ottomata>	 joal:  i guess it could have a flag, but if it were behaving like rsync, i would expect it to overwrite files with differences
[19:20:39] <ottomata>	 but, i'd make the flag the other way around if you wanted it
[19:20:42] <joal>	 ok makes sense ottomata - will do that
[19:20:53] <ottomata>	 should_overwrite=True
[19:20:54] <ottomata>	 perhaps
[19:20:54] <joal>	 (remove flag - easier
[19:20:58] <ottomata>	 just as the default
[19:21:18] <ottomata>	 remove flag is fine too
[19:28:21] <wikibugs>	 (03PS2) 10Joal: Update python/refinery/utils/HdfsUtils [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489)
[19:28:54] <wikibugs>	 (03CR) 10Joal: "> Patch Set 1:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal)
[19:35:54] <wikibugs>	 (03PS4) 10Joal: Add python script importing xml dumps onto hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/456654 (https://phabricator.wikimedia.org/T202489)
[19:36:19] <joal>	 Ok - done for tonight team - See you tomorrow evening :)
[19:36:35] <ottomata>	 laaterrrs
[19:40:46] <wikibugs>	 (03PS1) 10Ottomata: Throw usable exception when missing required field in ConfigHelper configure [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/459845 (https://phabricator.wikimedia.org/T203804)
[19:43:45] <wikibugs>	 (03PS7) 10Ottomata: Use ConfigHelper for RefineMonitor instead of scopt [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/458862 (https://phabricator.wikimedia.org/T203804)
[19:44:56] <wikibugs>	 10Analytics, 10Analytics-Wikimetrics, 10Epic: EPIC: Productionizing Wikimetrics {dove} - https://phabricator.wikimedia.org/T76726 (10Reedy)
[19:44:58] <wikibugs>	 10Analytics, 10Analytics-Wikimetrics, 10Security-Reviews: security review of Wikimetrics {dove} - https://phabricator.wikimedia.org/T76782 (10Reedy) 05Open>03stalled
[19:50:10] <wikibugs>	 10Analytics, 10Analytics-Wikimetrics, 10Security-Reviews: security review of Wikimetrics {dove} - https://phabricator.wikimedia.org/T76782 (10charlotteportero) a:03charlotteportero
[19:54:50] <wikibugs>	 (03CR) 10Ottomata: [C: 031] "Nits, but +1 :D" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/459780 (https://phabricator.wikimedia.org/T202489) (owner: 10Joal)
[21:05:08] <jdlrobson>	 hey all. I'm having trouble viewing EventLogging schema traffic in beta cluster. Krinkle says its possible Prometheus might be down
[21:05:30] <jdlrobson>	 In particular I'm looking for traffic to WebClientError schema.. anyone able to help?
[21:11:02] <ottomata>	 jdlrobson:  i'm here for a few more mins
[21:11:04] <ottomata>	 i can tyr to help
[21:11:51] <ottomata>	 what's up?
[21:12:57] <ottomata>	 there were at least some events that made it through!
[21:13:06] <jdlrobson>	 hey ottomata 
[21:13:10] <jdlrobson>	 basically trying to setup a dashboard
[21:13:15] <ottomata>	 oh in grafana?
[21:13:22] <jdlrobson>	 https://grafana-labs-admin.wikimedia.org/dashboard/db/niedzielski-eventlogging-schema?var-schema=VirtualPageView&panelId=1&fullscreen&edit&orgId=1
[21:13:29] <ottomata>	 enick_847:  was here earlier trying to do this too... hah
[21:13:33] <jdlrobson>	 yeh... can't seem to get it working for different schemas though
[21:13:44] <jdlrobson>	 https://grafana-labs-admin.wikimedia.org/dashboard/db/niedzielski-eventlogging-schema?var-schema=WebClientError&panelId=1&fullscreen&edit&orgId=1
[21:13:51] <ottomata>	 jdlrobson:  we need to figure out how to access the prometheus web interface
[21:13:59] <ottomata>	 in beta
[21:14:02] <jdlrobson>	 so either we're doing it wrong or there's something not piping these through
[21:14:05] <ottomata>	 https://wikitech.wikimedia.org/wiki/Prometheus#Access_Prometheus_web_interface
[21:14:08] <jdlrobson>	 ah so this is not working yet?
[21:14:13] <ottomata>	 i dunnno
[21:14:16] <ottomata>	 i thin it should be working
[21:14:22] <jdlrobson>	 mm
[21:14:35] <jdlrobson>	 kafka_cluster="jumbo-deployment-prep" < is that the right cluster?
[21:14:51] <ottomata>	 yes
[21:15:35] <jdlrobson>	 there should be a few data points in that graph for today but so far no good
[21:16:48] <ottomata>	 looking
[21:16:56] <ottomata>	 OH
[21:17:05] <ottomata>	 hmm
[21:18:58] <ottomata>	 yeah, its not enabled in labs
[21:19:06] <ottomata>	 the prometheus exporters need to be explicitly declared
[21:19:15] <jdlrobson>	 is that easy to fix?
[21:19:19] <ottomata>	 i'm not sure...
[21:19:27] <ottomata>	 stil lloking
[21:20:07] <ottomata>	 no
[21:20:08] <ottomata>	 can't fix.
[21:20:12] <ottomata>	 it uses exported resources
[21:20:17] <ottomata>	 which aren't availabe for labs puppet
[21:22:11] <ottomata>	 jdlrobson:  yeah, dunno this is not an easy one.  if you want a fix, you probably need a ticket with ops, tag filippo
[21:22:31] <ottomata>	 puppet prometheus exporters dont' work in labs because exported puppet resources are not queryable
[21:22:44] <ottomata>	 in labs, because of security reasons (cross project puppet stuff i think)
[21:24:49] <ottomata>	 ok i gotta run, sorry i couldn't help more than that
[21:24:51] <jdlrobson>	 :( ok will do! will add you to help tweak the wording
[21:24:59] <jdlrobson>	 day after tomorrow problem!
[22:12:16] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥1.13 or ≥2.1 - https://phabricator.wikimedia.org/T203498 (10Neil_P._Quinn_WMF) >>! In T203498#4573966, @mpopov wrote: > I'm confused too! I was going off https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF where `substring_index` (one of the UD...
[22:13:46] <wikibugs>	 10Analytics, 10Readers-Web-Backlog, 10Wikimedia-Site-requests, 10MobileFrontend (MobileFrontend.js), 10Patch-For-Review: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError) - https://phabricator.wikimedia.org/T203814 (10Tbayer) >>! In T203814#4574066, @Nuria wrote: >>Completely agree for the reas...
[22:14:06] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥ 2.0 - https://phabricator.wikimedia.org/T203498 (10Neil_P._Quinn_WMF)
[22:18:32] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥ 2.0 - https://phabricator.wikimedia.org/T203498 (10Neil_P._Quinn_WMF)
[22:39:06] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Hive to ≥ 2.0 - https://phabricator.wikimedia.org/T203498 (10Tbayer) Let's also not forget the potential speedups. In particular, the LLAP thing introduced in Hive 2.0 ("Hive Interactive Query") sounds interesting:   https://medium.com/sqooba/hive-llap-brings-interacti...