[16:52:15] <Nettrom>	 halfak: is there a guideline on where to build data pipelines like the one were looking at? E.g. are there particular database servers or something?
[16:52:57] <Nettrom>	 I’m not aiming to design it ready for production, but having something that’s not half bad would be nice.
[16:54:45] <halfak>	 Nettrom, you should ask that question in -analytics :) 
[16:54:58] <halfak>	 joal and milimetric will have opinions. 
[16:55:00] <Nettrom>	 halfak: alright, will do, thanks! :)
[16:55:06] <halfak>	 Oh they are here '
[16:55:10] <halfak>	 So you can ask them here too :D 
[16:55:40] <Nettrom>	 I think we just did? :D
[16:57:35] <halfak>	 reminder for joal and milimetric: we're looking at extracting the "organic" link graph from wikitext and to aggregate "pageview rates" and other general stats from request logs. 
[16:58:56] <Nettrom>	 so far I’ve figured out that getting pageview rates most likely means running some queries on Hive, and I’m thinking that the results go into some kind of intermediary table
[17:29:50] <joal>	 Hi Nettrom and halfak  -- I'll be in meeting soon, will catch up with you after
[20:58:00] <ragesoss>	 halfak: "working on api.php abuse" from scrum of scrums... know what that entails? Some of my tests against ORES just started breaking.
[20:58:25] <halfak>	 ragesoss, not you.  I just did a deployment and something is up. 
[20:59:04] <ragesoss>	 ah, I see, following along in -ai...
[21:10:33] <halfak>	 ragesoss, can you try what you're doing against ores.wmflabs.org and tell me how it goes?
[21:16:40] <ragesoss>	 halfak: tried a few times, and the tests were passing each time. failure is intermittent on production. Still slower than usual on wmflabs though, I think.
[21:17:04] <halfak>	 Yeah.  Also seeing intermittent here. 
[21:17:16] <halfak>	 The problem is only in codfw -- not in eqiad 
[21:17:28] <halfak>	 (different datacenters -- should be same code and config)
[21:17:31] <halfak>	 still working on it. 
[21:18:14] <ragesoss>	 that data center naming convention is so terrible.
[21:20:12] <ragesoss>	 {
[21:20:12] <ragesoss>	   "scores": {
[21:20:12] <ragesoss>	     "enwiki": {
[21:20:12] <ragesoss>	       "wp10": {
[21:20:12] <ragesoss>	         "scores": {
[21:20:14] <ragesoss>	           "675892696": {
[21:20:16] <ragesoss>	             "error": {
[21:20:18] <ragesoss>	               "message": "Timed out after 15 seconds.",
[21:20:20] <ragesoss>	               "type": "TimeoutError"
[21:20:23] <ragesoss>	             }
[21:20:25] <ragesoss>	           }
[21:20:27] <ragesoss>	         },
[21:20:29] <ragesoss>	         "version": "0.5.0"
[21:20:31] <ragesoss>	       }
[21:20:34] <ragesoss>	     }
[21:20:36] <ragesoss>	   }
[21:20:38] <ragesoss>	 }
[21:20:42] <ragesoss>	 interesting place to hide the error message!
[21:21:14] <ragesoss>	 hadn't seen one before, but I wouldn't have expected so much nesting.
[21:21:39] <halfak>	 Yeah.  It takes the place of the score. 
[21:21:50] <ragesoss>	 I'll have to add some checks for that.
[21:21:51] <halfak>	 Because you can score multiple revisions and only some error. 
[21:23:16] <Emufarmers>	 ragesoss: https://4.bp.blogspot.com/-wTejjZemfws/Vm3fv97XotI/AAAAAAAACe0/LwWQmvjyMwo/s1600/haduuken.jpg
[21:23:45] <ragesoss>	 :-)
[21:30:32] <ragesoss>	 oh, I see, it was just a quirck of my test forcing a float value that make it come up as 0.0 rather than nil when the timeout errors occurred.
[21:30:48] <ragesoss>	 I was very confused about how my code was resulting in scores of 0.0.
[21:56:13] <halfak>	 ragesoss, looks like we're in the clear now. 
[21:56:25] <halfak>	 Switched traffic to point back at eqiad and everything is happy again. 
[21:56:49] * halfak writes up and incident report. 
[21:59:56] <ragesoss>	 sweet, thanks! it broke some tests, but looks like the production dashboard suffered no ill effects. so that's nice to know.