[09:44:19] <Amir1>	 akosiaris: hey, I will send you lots of questions soon. I just woke up.
[09:44:32] <Amir1>	 Let me get to the office
[09:45:24] <akosiaris>	 Amir1: sure thing, take your time, no rush
[10:57:13] <Amir1>	 akosiaris: hey, just got to the office
[10:57:19] <Amir1>	 first of all, happy Friday
[10:57:46] <Amir1>	 second: I'm working on pool counter support
[10:58:33] <Amir1>	 I already reached to a point that it works locally and slows down requests and sends time out after a while
[10:58:50] <Amir1>	 joe said we should spin up a new pool counter node, do you think we should do it?
[10:59:10] <Amir1>	 I think so too but is it VM, bare metal?
[11:02:40] <akosiaris>	 Amir1: it is going to be a VM (the mediawiki ones are already), but does this really matter to you ?
[11:03:07] <akosiaris>	 what's more important would be to have support for multiple instances (mediawiki does)
[11:03:33] <akosiaris>	 so that in case instance A goes down for whatever reason, ORES will keeping using instance B and not die horribly
[11:03:44] <Amir1>	 akosiaris: basically because if it takes really long time or not
[11:03:54] <Amir1>	 I guess real hardware is harder
[11:04:21] <akosiaris>	 yeah it's going to be VM and should be < 30 mins to create it 
[11:04:55] <Amir1>	 akosiaris: hmm, the code just ignores if it can't connect
[11:05:13] <Amir1>	 I know it's not good but I wanted to start with this first
[11:05:56] <akosiaris>	 yeah from what I gather it's still a WIP, which is why I pointed that out now and not later on when it would be harder to change
[11:06:34] <Amir1>	 okay noted, I will work on it
[11:06:59] <akosiaris>	 do you have like an idea of when you 'd want to deploy the support for poolcounter ?
[11:07:13] <akosiaris>	 so that I prioritize getting the VM and all ready
[11:07:36] <Amir1>	 I guess next week I will start working on it labs setup and beta cluster
[11:07:52] <Amir1>	 the week after next is the most likely case
[11:08:07] <Amir1>	 or end of the next week
[11:10:50] <akosiaris>	 ok cool
[11:12:57] <Amir1>	 akosiaris: the other thing is about moving to k8s
[11:13:45] <Amir1>	 it's not dockerized, what is the plans about that. How I can push this to move forward?
[11:15:04] <akosiaris>	 no need to dockerize anything. The plan is for everything to be build via the CI/CD pipeline
[11:15:30] <akosiaris>	 the "dockerizing" part is done by blubber, which has moved along quite a bit and how has some python support
[11:15:31] <Amir1>	 blubber?
[11:15:35] <akosiaris>	 lemme give you the link
[11:15:51] <akosiaris>	 https://wikitech.wikimedia.org/wiki/Blubber
[11:16:34] <akosiaris>	 it's in charge of build the dockerfile, and the pipeline does the actual building of the container
[11:16:57] <Amir1>	 I see, the only thing that needs to happen is making the blubber yaml file
[11:17:25] <akosiaris>	 yup
[11:17:34] <akosiaris>	 there are some possible complications specific to ORES
[11:17:52] <akosiaris>	 things like maybe creating different images for the web and the worker
[11:17:56] <wikibugs>	 (03PS2) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/436523
[11:18:02] <wikibugs>	 (03CR) 10Hashar: "check experimental" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/436523 (owner: 10Hashar)
[11:18:14] <akosiaris>	 but even with a shared one it would be fine for starting with it
[11:18:33] <akosiaris>	 unfortunately we haven't managed to really schedule any k8s work this quarter due to the switchover and all
[11:18:52] <akosiaris>	 the idea is to do a big push next quarter however and move multiple services
[11:19:38] <Amir1>	 Nice, I keep this on the radar until the next Q
[11:20:50] <akosiaris>	 keep in mind blubber is in a beta-ish state right now, things might change 
[11:20:59] <Amir1>	 okay, noted
[11:21:04] <Amir1>	 another thing: https://gerrit.wikimedia.org/r/c/operations/puppet/+/454546
[11:21:42] <Amir1>	 I'm trying to make wikilabels more puppetized so I can throw away nodes and use another one
[11:21:42] <akosiaris>	 zlib1g-dev ?
[11:21:54] <akosiaris>	 that's header files, isn't it ?
[11:21:56] <Amir1>	 without it, pyllibmc can't be installed 
[11:22:09] <akosiaris>	 ah
[11:22:47] <akosiaris>	 and what installs pylibmc ?
[11:22:58] <Amir1>	 I have no clue TBH
[11:23:02] <akosiaris>	 hahahahaha
[11:23:08] <akosiaris>	 :D
[11:23:28] <akosiaris>	 anyway, this is labs specific, I am fine with that
[11:23:30] <akosiaris>	 lemme merge it
[11:23:31] <Amir1>	 the python dependency graphs can be sometimes, complicated 
[11:23:39] <Amir1>	 yeah
[11:23:45] <Amir1>	 the only thing here is the database and OAuth credentials
[11:24:02] <Amir1>	 I wish we had a private repo in labs as well
[11:24:45] <akosiaris>	 you kind of do. It's the per project hiera. IIRC it's only visible to the people belonging to the project
[11:24:54] <akosiaris>	 you set it in the horizon interface
[11:25:09] <akosiaris>	 it's not great ofc 
[11:25:40] <Amir1>	 aha, it used to be in wikitech
[11:25:44] <Amir1>	 that's good to know
[11:26:15] <akosiaris>	 yeah it's the same thing. albeit a tad more structured in the interface
[11:28:29] <Amir1>	 Still it's better than a public place :D
[11:29:46] <Amir1>	 one last thing is about the ores setup in labs. It has been unloved for a while now So the precaching is completely broken and I don't want to bring it back online as I don't see any real use for that
[11:30:06] <Amir1>	 the only fun thing here is that the ores in labs is being hit 200K times a day
[11:30:26] <Amir1>	 Tracking down who is hitting us is not easy though 
[11:31:28] <Amir1>	 https://phabricator.wikimedia.org/T202655
[11:32:04] <Amir1>	 I downsized the project a little 
[11:34:17] <akosiaris>	 heh
[11:34:37] <akosiaris>	 yeah the emit an error code telling them to reach you will probably work
[11:49:23] <Amir1>	 akosiaris: Just to make sure: the multi-instance option for pool counter is just to have the second one as fallback and that's all?
[11:49:32] <Amir1>	 Also it probably needs to be set up per dc
[11:54:23] <akosiaris>	 Amir1: that's up to you. It can be a fallback or you can distribute the locks if you want (you will have to do some consistent hashing though based on some key). Both approaches are fine cause they are locks, losing them means nothing. mediawiki does the latter btw
[11:55:01] <akosiaris>	 yes we are going to create 4 instances, 2 per dc (same way as mediawiki)
[11:55:31] <akosiaris>	 fwiw we could reuse the mediawiki instances (thumbor already uses them too) but it's probably more prudent to split them
[11:55:53] <Amir1>	 okay. Noted
[12:09:42] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Operations: Spin up a new poolcounter node for ores - https://phabricator.wikimedia.org/T201824 (10akosiaris) Yes, it does make sense indeed to not share the infrastructure with mediawiki. I 'll file a task for creating 4 VMs (2 poolcounter instances per DC) for this
[12:36:01] <travis-ci>	 wikimedia/wikilabels#408 (fix_form - 133a75b : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/wikilabels/builds/420097972
[13:01:36] <Amir1>	 akosiaris: one other thing: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455157 :D
[13:12:37] <akosiaris>	 Amir1: that's the metapackage that would also bring the postgresql database server and install it
[13:12:45] <akosiaris>	 I am thinking this is not what you want
[13:12:52] <akosiaris>	 you only need to psql binary, right ?
[13:13:15] <akosiaris>	 change it to postgresql-client
[13:16:36] <Amir1>	 akosiaris: yeah, let's do that
[15:12:02] <wikibugs>	 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Turn article quality javascript into a gadget - https://phabricator.wikimedia.org/T202744 (10Ladsgroup)
[15:13:42] <wikibugs>	 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Build a gadget to give list of quality of articles in categories - https://phabricator.wikimedia.org/T202745 (10Ladsgroup)
[15:54:56] <Amir1>	 I go eat something, will be back soon
[17:02:17] <notconfusing>	 awight, Amir1, is there a public endpoint for the staging wikilabels? I know there are two wikilabels repos 1) for testing 2) for deployment. Aaron merged my PR into the non-deploy repo, so where might that be viewable. Is that the serve you gave me access to awight?
[17:07:29] <Amir1>	 notconfusing: I just got back
[17:07:34] <Amir1>	 we actually have three
[17:07:53] <Amir1>	 labels-staging.wmflabs.org, labels-experiment.wmflabs.org, labels.wmflabs.org
[17:08:04] <Amir1>	 in any of them you need to deploy
[17:08:22] <Amir1>	 and fun fact, I just deployed all changes of wikilabels to production
[17:09:18] <Platonides>	 you should rename labels.wmflabs.org to labels-nolabel.wmflabs.org :P
[17:12:33] <Amir1>	 :)))
[17:12:38] <Amir1>	 Thanks but no :D
[17:33:55] <notconfusing>	 Amir1, cool. So how would I load a data for labels-experiment.wmflabs.org ?  ssh into wikilabels-01.wikilabels.eqiad.wmflabs ?
[17:34:54] <Amir1>	 notconfusing: well, 1- I'm phasing out wikilabels-01, the traffic for prod is actually on wikilabels-02 2- you should not use prod, use staging wikilabels-staging.eqiad.wmflabs
[17:35:13] <notconfusing>	 ok
[17:35:29] <Amir1>	 I'm going to drop the whole wikilabels-01 after making sure no one uses it for whatever reason
[17:35:40] <Amir1>	 notconfusing: https://wikitech.wikimedia.org/wiki/Wikilabels
[17:35:45] <Amir1>	 This might be useful 
[17:36:23] <notconfusing>	 ok, great info.
[17:37:59] <notconfusing>	 this guides says to make a new labelling campaign, use wikilabels-02,  but i should try it on wikilabels-experiment first, right?
[17:39:25] <Amir1>	 notconfusing: yeah, or staging, they both point to the same database
[17:39:44] <Amir1>	 I was too lazy to make experiment and staging separate databases 
[17:39:56] <notconfusing>	 cool, thanks, I'll try this out today and hope i don't mess anything up
[19:30:47] <Amir1>	 I'm around in case you have any questions, or need help for anything
[19:42:13] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Migrate gadgets that use ores labs instance to prod - https://phabricator.wikimedia.org/T202630 (10Ladsgroup) Thank you all very much!
[20:05:04] <wikibugs>	 10Scoring-platform-team, 10Wikilabels: Migrate wikilabels action=compare to support MCR - https://phabricator.wikimedia.org/T202767 (10Ladsgroup)