[01:26:21] <wikibugs>	 (03CR) 10Milimetric: "some simple naming observations, sorry to be late, happy to do the ones you agree with" (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T206883) (owner: 10Joal)
[01:32:39] <wikibugs>	 (03CR) 10Milimetric: "generally +2 except waiting to finalize the phab discussion for the field names.  If the latest names get accepted, then just rename revis" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492304 (https://phabricator.wikimedia.org/T178587) (owner: 10Joal)
[01:51:25] <wikibugs>	 (03CR) 10Milimetric: "mainly the observation Tilman made, that logs can have tags" (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492320 (owner: 10Joal)
[02:03:56] <wikibugs>	 (03CR) 10Milimetric: Update mediawiki-reconstruction with log info (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493012 (owner: 10Joal)
[02:45:49] <wikibugs>	 (03CR) 10Milimetric: Refactor mediawiki-page-history computation (037 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 (https://phabricator.wikimedia.org/T190434) (owner: 10Joal)
[04:19:04] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529 (10Ottomata) > Are you sure this is actually the case  Oh I guess not! @JAllemandou I thought we were using Hive timestamps?
[07:41:01] <wikibugs>	 10Analytics, 10EventBus, 10Operations, 10Prod-Kubernetes, and 2 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10akosiaris) I can not rule out a networking issue but it seems improbable, as after all the logs did make it to logstash. Al...
[08:02:11] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529 (10JAllemandou) Hi Folks, I'll tr to provide more infoon Hive Timestamps and related formats. There are (at least) two considerations when dealing with Hive data...
[08:39:19] <ema>	 hello analytics! The varnish frontend on cp2002 has been routed through ATS backends in codfw since yesterday: https://grafana.wikimedia.org/d/wI0nURqiz/ats-cluster-view?orgId=1&from=1552555324302&to=1552638973378
[08:39:48] <ema>	 please let me know if you find anything strange in webrequest and similar when it comes to such host (or more in general, codfw upload)
[08:52:45] <elukey>	 ema: o/
[08:52:49] <joal>	 Hi ema
[08:52:54] <joal>	 you beat me again elukey :)
[08:53:05] <elukey>	 the only thing that we noticed were a bit of errors on the 12th (12-14 UTC)
[08:53:08] <elukey>	 nothing more
[08:53:12] <elukey>	 (for upload)
[08:53:20] <elukey>	 bonjour joal :)
[08:53:24] <joal>	 ema: We'll keep an eye, but if nothing alerts, well we'll consider it works :)
[08:53:28] <elukey>	 I am battling against hive that hates me
[08:53:30] <joal>	 Bonjour elukey :)
[08:53:31] <elukey>	 as always
[08:53:38] <joal>	 elukey: May I help?
[08:53:59] <elukey>	 nah it is TLS related, I enabled the encrypted shuffle but of course it doesn't work :D
[08:54:22] <joal>	 right - Please let me know if there is anything I could do :)
[08:54:27] <elukey>	 sure!
[08:55:31] <ema>	 elukey: oh, errors on the 12th must be unrelated (no ATS test that day)
[08:59:54] <elukey>	 ack!
[09:57:55] <wikibugs>	 (03PS10) 10Joal: Refactor mediawiki-page-history computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493390 (https://phabricator.wikimedia.org/T190434)
[10:45:31] <elukey>	 ok so I am definitely puzzled
[10:45:45] <elukey>	 namenode to journalnodes seems to be ok with TLS
[10:46:02] <elukey>	 also yarn daemons are up with tls settings
[10:46:25] <elukey>	 but, when I enable the encrypted shuffle, the node managers log exceptions like
[10:46:41] <elukey>	 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: 0xad298e02, /10.64.53.15:56384 => /10.64.36.133:13562] EXCEPTION: org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record
[10:46:57] <elukey>	 and with tcpdump I can indeed see plain HTTP get requests
[10:47:04] <elukey>	 I guess from the reducers
[10:47:12] <elukey>	 (the ips in the error log are hadoop workers)
[10:47:37] <elukey>	 I have no idea why the reducers are not using TLS to get data from the shuffle service
[10:48:39] <joal>	 hm
[10:54:03] <elukey>	 I am wondering if hadoop.rpc.protection=privacy is needed
[10:54:13] <elukey>	 in theory no, since it regulates RPC calls
[10:54:16] <elukey>	 not HTTP ones
[10:54:24] <elukey>	 in practice, documentation is not great
[10:57:49] <joal>	 :(
[10:59:59] <joal>	 elukey: I'm assuming you've followed https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html
[11:03:22] <joal>	 elukey: need to go for errand - will be back in ~2h
[11:05:12] <elukey>	 yes also the cloudera specific one, that uses deprecated settings -.-
[11:07:25] <elukey>	 that is https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cdh_sg_encryption_enable.html
[11:07:37] <elukey>	 hadoop.ssl.enabled is considered deprecated 
[11:09:36] <elukey>	 one thing that I didn't do is set the other properties in core-site.xml
[11:09:42] <elukey>	 since their defaults are already ok
[11:10:03] <elukey>	 I am now wondering if there is a stupid code that checks for their presence
[11:10:13] <elukey>	 disregarding the defaults
[11:31:31] <elukey>	 lunch! Will try later on :)
[14:38:08] <milimetric>	 hi joal, I'm reading over your comments now
[14:38:20] <joal>	 Hi milimetric :)
[14:38:47] <milimetric>	 oh, no comments, just new patches
[14:38:55] <milimetric>	 wanna chat in cave?
[14:38:57] <joal>	 milimetric: I need to go get the kids at the hour - Do we talk now?
[14:38:59] <joal>	 Yeah :0
[15:11:03] <milimetric>	 you know, we should just do all these changes in a follow-up patch, they're just cosmetic
[16:01:13] <nuria>	 ping joal mfournier 
[16:03:20] <joal>	 sorry for the ping mfournier - nuria was without doubt pinging mforns
[16:44:30] <wikibugs>	 (03CR) 10Milimetric: "ok, all addressed, we can think about adding the log change tags later, similar to how we do log_params" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492320 (owner: 10Joal)
[16:44:34] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Add change_tags to mediawiki_history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/492320 (owner: 10Joal)
[16:46:01] <dsaez>	 hey milimetric, are you around?
[16:52:30] <wikibugs>	 (03CR) 10Milimetric: Update mediawiki-reconstruction with log info (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493012 (owner: 10Joal)
[17:03:28] <joal>	 Hi dsaez - He's with me, I'll release him soon (maybe)
[17:04:03] <dsaez>	 joal, hi, it's ok,  I've already found what I was looking for
[17:04:12] <joal>	 Ok :)
[17:13:29] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10leila) @RyanSteinberg Lauren and I talked yesterday and based on the latest analysis, the effect of this particular bug is evenly d...
[18:27:33] <nuria>	 neilpquinn: yt? 
[18:28:02] <nuria>	 neilpquinn: there was agoogle doc that went with this data we wanted to load into turnilo: https://phabricator.wikimedia.org/T211173
[18:28:18] <nuria>	 neilpquinn: but i cannot find it, would you be so kind to add it to ticket?
[18:31:24] * elukey off!
[18:31:27] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10ops-eqiad: confirm gpu form factor in stat1005 - https://phabricator.wikimedia.org/T216528 (10Cmjohnson) @elukey please see attached jpg . {F28393516}
[18:37:45] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10bmansurov) @Ottomata a heads up that we'll be collecting citation data starting [[ https://wikitech.wikimedia.org/w/index.php?title...
[18:49:21] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10ops-eqiad: confirm gpu form factor in stat1005 - https://phabricator.wikimedia.org/T216528 (10EBernhardson) Thanks chris! Based on this a standard dual-slot card will fit in this configuration. There will not be room for a second GPU.  My suggestion is to mo...
[18:52:12] <wikibugs>	 (03PS6) 10Joal: Update refinery sqoop to use dedicated labsdb host [analytics/refinery] - 10https://gerrit.wikimedia.org/r/495266 (https://phabricator.wikimedia.org/T215550)
[18:52:47] <joal>	 Current test without fetch-size is positive - Making the parameter optional --^
[18:53:07] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10EBernhardson) https://phabricator.wikimedia.org/T216528#5027798 includes a picture of the rear of the case. This confirms a dual-slot card will fit into one side of the chasis. There will b...
[19:01:53] <elukey>	 ebernhardson: o/ - I was about to ping you for the GPU and you anticipated me :)
[19:02:03] <elukey>	 I agree with the https://www.neweggbusiness.com/product/product.aspx?item=9b-14-105-087
[19:02:08] <elukey>	 let's buy it
[19:03:28] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10bmansurov) @leila ^.
[19:06:12] <elukey>	 a-team: habemus GPU to buy
[19:06:34] <milimetric>	 habemus?
[19:07:08] <milimetric>	 oh! lol
[19:07:11] <elukey>	 yeah like https://en.wikipedia.org/wiki/Habemus_papam
[19:07:13] <elukey>	 :D
[19:07:33] <milimetric>	 you need to send smoke up to land that joke
[19:07:38] <joal>	 \o/! Let's celebrate a mess for Saint Erik of the Search, patrons of SIMD :)
[19:08:07] <elukey>	 milimetric: hahah yes
[19:08:21] <HaeB>	 the smoke part should be achievable by blocking the GPU's fans, right?
[19:08:31] <joal>	 :D
[19:08:35] <milimetric>	 lol
[19:08:39] <milimetric>	 nice HaeB 
[19:08:48] <elukey>	 I've read about a AMD bug that disabled the fan by mistake
[19:08:58] <elukey>	 so no need for any manual intervention
[19:09:00] <elukey>	 :D
[19:13:27] <HaeB>	 milimetric: btw thanks for your observation at https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/492320/ - we are in the process of adding tags to many more log actions (https://phabricator.wikimedia.org/T215675 etc.) so that will become more important soon
[19:14:30] <milimetric>	 well, thank you for pointing it out, HaeB.  But right now that comment would just mean we have them available in the algorithm, we wouldn't be surfacing them yet.  We still have to figure out where to put them when they come from events instead of revisions
[19:21:17] <wikibugs>	 (03CR) 10Nuria: [C: 04-1] Update refinery sqoop to use dedicated labsdb host (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/495266 (https://phabricator.wikimedia.org/T215550) (owner: 10Joal)
[19:38:11] <wikibugs>	 (03CR) 10Milimetric: "thoughts on what Nuria was saying, mostly agreed" (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/495266 (https://phabricator.wikimedia.org/T215550) (owner: 10Joal)
[19:52:46] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10leila) @bmansurov thanks! How is the staging done? In the second round, we gradually increased data collection from 1-15% per [[ ht...
[20:03:23] <wikibugs>	 (03PS8) 10Joal: Update mediawiki-reconstruction with log info [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/493012
[20:21:07] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10bmansurov) @leila we did staging because we wanted to make sure that the back end can handle the load. Now that we know it can, we...
[20:49:30] <wikibugs>	 (03PS1) 10Bmansurov: WIP: Add workflow for article-recommender [analytics/refinery] - 10https://gerrit.wikimedia.org/r/496885 (https://phabricator.wikimedia.org/T210844)
[20:53:06] <wikibugs>	 (03CR) 10Bmansurov: "Andrew, Joseph, in addition to the inline comments, is the general approach acceptable?" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/496885 (https://phabricator.wikimedia.org/T210844) (owner: 10Bmansurov)
[21:27:41] <wikibugs>	 (03PS4) 10Milimetric: Add concept of metric groups, rotate in dashboard [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494241 (https://phabricator.wikimedia.org/T187806) (owner: 10Fdans)
[22:07:03] <wikibugs>	 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Nuria) Ping @fgiunchedi about putting this as a commong goal next quarter
[22:24:07] <wikibugs>	 10Analytics, 10MobileFrontend, 10Performance-Team (Radar), 10Readers-Web-Backlog (Tracking), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381 (10pmiazga) I confirm MobileFrontend has it's own X-Analytics header handling for regular requests (not API) doesn't use t...
[22:24:48] <wikibugs>	 (03CR) 10Nuria: WIP: Add workflow for article-recommender (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/496885 (https://phabricator.wikimedia.org/T210844) (owner: 10Bmansurov)
[22:30:08] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 2 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10Jdlrobson) What is the recommendation here? Should we decline this t...
[22:44:13] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Beta: Provide easier mapping between Wikistats1 metrics and Wikistats2 metrics (example: "active editors") - https://phabricator.wikimedia.org/T187806 (10Nuria) {F28395080}  {F28395085}  {F28395101}  Overall idea looks good. Adding...
[22:52:09] <wikibugs>	 (03CR) 10Nuria: "Idea looks good, I think Ux still needs a bit of work, I have uploaded couple screenshots to https://phabricator.wikimedia.org/T187806" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494241 (https://phabricator.wikimedia.org/T187806) (owner: 10Fdans)
[23:09:07] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 2 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10Nuria)  I am sorry I cannot advice of  a better place for this code...
[23:23:07] <wikibugs>	 10Analytics, 10Pageviews-API: No results for Special:BlankPage or Special:BlankPage/RTRC - https://phabricator.wikimedia.org/T151363 (10Krinkle) 05Open→03Declined My gadget is terribly abusing how special pages happen to work in MediaWiki, and is not something I think the Analytics team should try to accom...