[11:34:39] <wikibugs>	 06Analytics-Kanban, 15User-Elukey: Improve purging  for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3254707 (10elukey) Some notes taken while researching:  * running the purge script and `eventlogging_syncy.sh` on the same host might generate contention while INSERTING/DELE...
[11:41:19] <joal>	 taking a break lads
[11:41:23] <elukey>	 joal: o/
[11:42:01] <elukey>	 fdans: hola amigo
[12:22:15] <fdans>	 elukey: holaaaa sorry I was at lunch
[12:35:25] <wikibugs>	 10Analytics, 10EventBus, 13Patch-For-Review, 06Services (watching): Check eventbus Kafka cluster settings for reliability - https://phabricator.wikimedia.org/T144637#3254899 (10elukey) I think I already asked this question to @Ottomata but I forgot the answer :D, so I am going to ask again: any reason to k...
[12:36:18] <elukey>	 fdans: o/
[12:41:24] <elukey>	 fdans: I think that in beta everything is good, I applied your patch as cherry pick and I didn't end up in detached head
[12:43:44] <elukey>	 and before on tin the branch was in detached head
[12:43:57] <elukey>	 (el analytics repo I mean)
[12:52:57] <fdans>	 elukey: wonderful!! let's deploy then?
[12:53:17] <fdans>	 oh I see so the key is that HEAD isn't in a detached state
[12:53:20] <fdans>	 that makes sense
[12:55:17] <elukey>	 fdans: do you know about the cherry pick automagic from the gerrit UI?
[12:55:54] <fdans>	 elukey: nope! is that the piece of the puzzle that I'm missing that will make me looooove gerrit? :D
[12:56:06] <elukey>	 https://gerrit.wikimedia.org/r/#/c/352579/ --> top right corner 'Download' --> select anonymous http in the bottom --> Cherry pick
[12:56:37] <elukey>	 then you apply that one on deployment-tin
[12:58:18] <elukey>	 about the deployment: since https://gerrit.wikimedia.org/r/#/c/352579 has no +1 in there and Andrew is on vacation and we are going to Prague soon, let's postpone the deployment to when we'll be back?
[12:58:40] <elukey>	 unless it is super urgent
[12:59:01] <elukey>	 I like Joseph's idea of "deployment freeze" for analytics a couple of days before the offsie
[12:59:04] <elukey>	 *offsite
[13:03:16] <fdans>	 yeah that makes sense
[13:08:44] <mforns>	 hey team :]
[13:08:44] <nuria_>	 elukey: +1
[13:09:07] <nuria_>	 to deployment freeze from today until we are all in Prague
[13:18:37] <nuria_>	 mforns: are the tests for refinery-core broken for you in master? 
[13:18:58] <mforns>	 nuria_, /me checks
[13:24:58] <mforns>	 nuria_, refinery-core passes for me...
[13:25:09] <nuria_>	 mforns: master?
[13:25:17] <mforns>	 nuria_, but refinery-jobs fails!
[13:25:19] <mforns>	 yes, master
[13:25:33] <nuria_>	 mforns: ok, for me it fails, with some dependencies not present 
[13:25:54] <mforns>	 for refinery-job I get: java.io.FileNotFoundException: File /tmp/unittest/refinery-source/refinery-job/TestSubgraphPartitioner/38de5826-c854-4dbc-97f4-a022b28db32e does not exist
[13:29:35] <wikibugs>	 (03PS1) 10Nuria: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021)
[13:30:29] <nuria_>	 mforns: for me master is all broken, ok, we are going to have to look at it in more detail, fdans, do tests for master in refinery work for you?
[13:31:04] <mforns>	 refinery_source no?
[13:31:24] <fdans>	 nuria_: checking
[13:33:09] <wikibugs>	 (03PS2) 10Nuria: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021)
[13:34:08] <nuria_>	 mforns: right , refinery-core actually
[13:35:01] <fdans>	 refinery core seems fine to me after pulling nuria_ 
[13:35:18] <nuria_>	 fdans: so tests pass
[13:35:23] <fdans>	 yes
[13:35:59] <nuria_>	 fdans: ok, let me blow up all deps then, i wonder if i need a new vs of scala
[13:36:43] <fdans>	 nuria_: I'm running them with mvn test, that's the right way?
[13:37:03] <nuria_>	 fdans: yaya totally, i must have something not right on my env
[13:39:48] <nuria_>	 fdans: ok, refinery-job ?
[13:39:53] <nuria_>	 fdans: i think that one fails
[13:40:02] <fdans>	 checking
[13:41:11] <fdans>	 nuria_: which error are you getting?
[13:41:30] <nuria_>	 fdans, mforns all ok now but refinery-job
[13:41:39] <nuria_>	 there are several tests that fail
[13:42:34] <mforns>	 refinery-job fails for me too
[13:43:22] <fdans>	 +1
[13:43:28] <fdans>	 refinery-job failing
[13:49:08] <joal>	 hey guys - mvn clean package works for me on master
[13:51:05] <joal>	 :/
[13:59:21] <nuria_>	 joal: it is an issue of deps: no snappyjava in java.library.path
[14:01:17] <nuria_>	 elukey: hola! standup
[14:05:54] <joal>	 just retested on stat1004 after having removed ~/.m2 folder (force cleaning mvn deps) --> works for me :(
[14:06:09] <joal>	 refinery-source --^
[14:44:05] <halfak>	 https://phabricator.wikimedia.org/T164377#3235864
[14:56:29] <joal>	 nuria_: redirect codes we should look at( sorry lost the chat when closing hangout): 301, 302, 304, 307 0 anything else?n
[15:04:37] <wikibugs>	 10Analytics, 06Scoring-platform-team, 10rsaas-articlequality , 07Spike: [Spike] Store article quality data inside hadoop and make AQS outputs a public API - https://phabricator.wikimedia.org/T164377#3255338 (10Nuria) >Is there a public hadoop task that we could make this task block on? Our plans for next y...
[15:04:51] <nuria_>	 304 is pageview
[15:04:56] <nuria_>	 ~^joal
[15:05:01] <nuria_>	 so you alredy have that one
[15:05:05] <nuria_>	 *already
[15:05:14] <nuria_>	 the others are 307, 301 and 302
[15:05:22] <nuria_>	 halfak: answered on ticket
[15:11:35] <joal>	 thanks nuria_ 
[15:12:23] <wikibugs>	 06Analytics-Kanban: Cleaning scheme for banner data _SUCCESS files - https://phabricator.wikimedia.org/T164497#3255364 (10mforns) a:03mforns
[15:14:25] <wikibugs>	 (03PS1) 10Mforns: Add script to delete banner activity _SUCCESS files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/353309 (https://phabricator.wikimedia.org/T164497)
[15:15:45] <wikibugs>	 (03PS1) 10Joal: Provide RedirectToPageview function and UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353310 (https://phabricator.wikimedia.org/T143928)
[15:53:22] <joal>	 nuria_: There must be something else :(
[15:54:31] <joal>	 there is a HUGE artefact on m.wikidata.org for last-acess-global-set and last-access-not-set
[15:54:38] <joal>	 and redirects don't cover
[15:54:45] <joal>	 There must be something else
[16:14:05] <nuria_>	 joal: did you look at http codes
[16:15:44] <joal>	 I'm not sure what you mean
[16:16:46] <joal>	 nuria_: was expecting that, adding new redirects to set, we would find a size-equivalent uniques set for global as the ones on m.wikidata.org 
[16:16:55] <joal>	 nuria_: But there is not :(
[16:17:13] <nuria_>	 joal: what about if you do not exclude any http code?
[16:17:24] <joal>	 hm
[16:17:27] <joal>	 will try
[16:17:34] <nuria_>	 joal: sorry, mi initial reply was not clear 
[16:25:37] <joal>	 nuria_: mismo :(
[16:26:10] <joal>	 nuria_: numbers are bigger on some other specific cases, but the artifact is still here
[16:26:15] <nuria_>	 joal: the number increases when looking at redirects right? (even if there is a disparity)
[16:27:02] <joal>	 nuria_: total number of uniques found increases, yes - but the artifact between global and not on L-A not set while L-A-G set is still here
[16:27:33] <nuria_>	 joal: ok, we are going to have to do more thinking
[16:27:45] <joal>	 Looks like so :(
[16:27:55] <joal>	 I was so happy we had found a good candidate
[16:28:51] <elukey>	 people I am going afk! o/
[16:29:04] <joal>	 bye elukey 
[17:21:32] <wikibugs>	 06Analytics-Kanban: Create purging script for mediawiki-history data - https://phabricator.wikimedia.org/T162034#3256023 (10mforns) @JAllemandou   > Can we drop partitions that contain subpartitions, or do we need to iterate over every subpartition?  Thanks for the heads-up! In my ignorance, though, I don't unde...
[17:22:32] <mforns>	 joal, I commented on the mediawiki snapshot task, about your last comment, when you have time can you have a look, please? :]
[17:56:30] <joal>	 Hi mforns - commenting :)
[17:56:37] <mforns>	 :] thx joal 
[17:56:38] <joal>	 sorry for the delay mforns 
[17:56:41] <mforns>	 np
[18:07:17] <wikibugs>	 06Analytics-Kanban: Create purging script for mediawiki-history data - https://phabricator.wikimedia.org/T162034#3256238 (10JAllemandou)  > In my ignorance, though, I don't understand what could be the potential problem of deleting the whole snapshot directory (with sub-partitions in it)? You think Hadoop won't...
[18:10:08] <mforns>	 joal, thanks for the comments
[18:19:02] <joal>	 mforns: np ! Does it make sense ?
[18:19:44] <mforns>	 joal, yes! I was thinking of a low tech alternative which was to execute the msck repair table after all deletions, but your solution is better
[18:20:02] <mforns>	 even... repair table might not work at all...
[18:20:18] <joal>	 mforns: I think repair only adds, doesn't delete unfortunately
[18:20:23] <mforns>	 I see
[18:20:30] <joal>	 mforns: not sure though
[18:21:02] <mforns>	 anyway, the drop partition command both deletes data AND meta-data right?
[18:21:32] <joal>	 nope mforns, external tables, so only meta-data when droping partitions
[18:21:32] <mforns>	 joal, ^
[18:21:37] <mforns>	 oh, ok
[18:22:02] <mforns>	 K, I think I can finish the script now, thanks a lot!
[18:22:15] <joal>	 no prob mforns, thanks a lot for doing that !
[18:22:23] <mforns>	 o/
[18:54:00] <wikibugs>	 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: analytics1030 stuck in console while booting - https://phabricator.wikimedia.org/T162046#3256414 (10Cmjohnson) The new system board has been ordered and I will be contacted by a Dell tech to visit the cage and replace.    In regards to Service Tag –...
[20:19:16] <wikibugs>	 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3256732 (10AndyRussG) >>! In T115042#3252540, @mforns wrote: > 1) We spoke about the target segment of a campaign, and gave as an exam...
[20:55:10] <wikibugs>	 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3256808 (10Niharika) The bot is currently using promises for fetching redirects and it's...lightning fast somehow...Examples below -  Before (~20 hours): ``` 20...
[20:57:21] <wikibugs>	 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3256813 (10kaldari) @Niharika: Since using promises provides such a dramatic speed improvement, I would hate for us to throw that away for the majority of WikiP...
[21:30:02] <wikibugs>	 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3256908 (10MusikAnimal) >>! In T164178#3256808, @Niharika wrote: > The bot is currently using promises for fetching redirects and it's...lightning fast somehow....
[21:38:13] <wikibugs>	 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3256959 (10Niharika) >>! In T164178#3256813, @kaldari wrote: > @Niharika: Since using promises provides such a dramatic speed improvement, I would hate for us t...