[00:31:59] 10Scoring-platform-team (Current), 10Operations, 10Scap, 10Patch-For-Review, and 2 others: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10greg) [00:32:55] 10Scoring-platform-team, 10GitHub-Mirrors: GitHub IRC integration is being deprecated by October 1st, 2018 - https://phabricator.wikimedia.org/T194070 (10greg) [10:55:56] o/ [10:56:17] Amir1: o/ [10:56:22] Should I merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474694/1/modules/ores/manifests/web.pp ? [10:56:37] akosiaris: hey, let's do it! [10:56:44] thanks [11:08:50] ores1001 done [11:09:05] things look fine [11:11:23] I 'll slowly proceed with the entirety of codfw then [11:11:54] should take ~45-50mins (most of which will be idle time) [11:15:03] akosiaris: Awesome. Thank you! [11:33:56] Amir1: I take it back! there's an isue [11:34:01] redis.exceptions.ResponseError: NOAUTH Authentication required. [11:34:20] thankfully I 've stopped it soon enough [11:35:18] * akosiaris fixing [11:36:07] let's see what's going on [11:36:11] it worked on beta [11:36:28] the password [11:36:43] it's still in the old format [11:38:20] akosiaris: I can make a follow up [11:38:24] are you on it? [11:38:26] already done [11:38:29] submitting it now [11:38:30] Thanks [13:00:46] akosiaris: My next to-do item list is redis-sentinel. Should we start working on it? We need a new node, we need to puppetize sentinel configs and services and then roll it out in beta cluster or ores in labs (how? we don't have three redis nodes there :/) then roll in production and dance. Does it sound good to you? [13:01:42] hmm, don't worry about three node thing. For testing we can have different redis servers on different ports [13:01:56] but we need another node in prod [13:18:56] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Travis is failing on master of ORES - https://phabricator.wikimedia.org/T209852 (10Ladsgroup) https://github.com/celery/celery/issues/5175 [13:20:56] Amir1: so what's your ETA for production ? keep in mind: a) my availability timewise is a bit limited for the next 30 days or so b) train freezes for holidays and all hands. [13:22:36] as far as the third node goes, I 'll have to read up first a bit and understand the exact requirement (does it need a quorum or not and so on). I 'll try and do so tomorrow [13:25:06] akosiaris: I just looked at the Q goals and it's not for this Q but it will be definitely for the next quarter, so it needs to be done before end of March. Would that work? [13:25:37] Amir1: yeah that sounds pretty fine [13:25:39] Noting that I want moving to k8s to be done in that quarter if possible [13:25:59] so it might be lots of work for the next Q [13:26:11] bugs aside in the sentinel implementation and having to learn about it I don't expect this to consume too much of my time [13:26:17] on your time on the other hand... [13:26:21] I have no estimation [13:27:30] I can work on it, that's basically the only high-priority task left in maintenance of ORES [13:31:57] wikimedia/ores#1150 (fix_redis - 62ba709 : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/ores/builds/457940373 [13:33:22] akosiaris: do we have sort of k8s cluster in cloud. I was wondering as a warm-up work I migrate wikilabels to blubber and helm and then deploy it to labs [13:34:00] paws or tools ? [13:34:07] that's the 2 envs I remember [13:34:27] tools will probably not work, mostly because of the process to build the image and the registry it uses [13:34:39] I have no idea about paws though [13:34:50] that being said, minikube would also work [13:35:17] so if you have a machine with enough memory you can use minikube to get a local k8s cluster and play around with it [13:35:47] installing helm in it is like 10secs of work. download helm and do helm --init [13:36:09] akosiaris: That sounds good, Do you have some documentation on how to do it handy? [13:38:47] Amir1: for minikube: https://kubernetes.io/docs/tasks/tools/install-minikube/. Note you will need virtualbox (or some other hypervisor). For helm https://docs.helm.sh/using_helm/#installing-helm [13:39:06] those 2 should be enough to provide you with a limited but fully functional kubernetes environment [13:39:17] limited as in resources, not functionality [13:39:48] adding our own charts repo is easy. $ helm repo add wmf https://releases.wikimedia.org/charts/ [13:40:07] but it's not really required for what you want to do [13:40:18] I see [13:40:19] thanks [13:41:17] minikube has it's own built-in docker and blubber is released by releng https://people.wikimedia.org/~thcipriani/blubber/ [13:41:47] you can also use directly blubber as a service per https://wikitech.wikimedia.org/wiki/Blubber#Stateless [13:42:10] I am expecting though blubber to require some development for it to be able to support ORES [13:42:45] Niice [13:42:56] I hope I can help with those work [13:53:31] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for Galician Wikipedia - https://phabricator.wikimedia.org/T201146 (10Elisardojm) The labeling campaign is finished! :) [13:58:51] wikimedia/ores#1151 (fix_redis - dc2f589 : Amir Sarabadani): The build was fixed. https://travis-ci.org/wikimedia/ores/builds/457951386 [14:09:36] halAFK: Hey, one quick review: https://github.com/wikimedia/ores/pull/292 [14:09:49] o/ [14:10:08] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Travis is failing on master of ORES - https://phabricator.wikimedia.org/T209852 (10Ladsgroup) https://github.com/wikimedia/ores/pull/292 [14:13:02] {{merged}} [14:13:02] 10[1] 04https://meta.wikimedia.org/wiki/Template:merged [14:13:16] Amir1, ^ [14:13:38] Thanks! \o/ [14:16:08] wikimedia/ores#1153 (master - c42d090 : Aaron Halfaker): The build was fixed. https://travis-ci.org/wikimedia/ores/builds/457958940 [14:22:01] 10Scoring-platform-team (Current): Respond to ORES question from Mark Wang - https://phabricator.wikimedia.org/T210086 (10Halfak) [14:47:56] Looks like we have article quality data from glwiki. [14:50:14] afk for lunch [15:02:21] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:40:37] Amir1, for when you get back: https://github.com/wikimedia/revscoring/pull/415 [15:40:47] Should be straightforward. :) [15:41:29] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: FeatureScalar appears in features list rather than something meaningful - https://phabricator.wikimedia.org/T209869 (10Halfak) [15:41:39] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: FeatureScalar appears in features list rather than something meaningful - https://phabricator.wikimedia.org/T209869 (10Halfak) a:03Halfak [15:44:15] wikimedia/revscoring#1552 (feature_meta_names - 4ef1dbb : halfak): The build passed. https://travis-ci.org/wikimedia/revscoring/builds/458000832 [15:45:54] back now [15:48:25] Thanks for the merge :) [15:48:51] yw [15:49:07] Last call for SoS reporting [15:49:15] you have half an hour [15:49:24] Amir1, do you think you'll have cycles to include a batching strategy in https://phabricator.wikimedia.org/T201691 ? [15:49:45] o/ notconfusing [15:50:01] o/ halfak [15:50:23] Just pinged you about a data request for goodfaith newcomers on Meta :) [15:50:30] halfak: I'm not actively working on the patch as it's been sitting in review for months now [15:50:56] Amir1, gotcha. I can only review the JS components effectively. Shall I take another look? [15:53:10] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:55:49] yeah, sure, just it's the basic baseline, I'm not going to do anything fancy with the patch [16:00:13] ok, i'll check out meta [16:16:08] OMG emails. [16:16:20] I eat up so much of the day just digging through them. [16:51:28] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Implement JS ORES client in mw-ORES extension - https://phabricator.wikimedia.org/T201691 (10Halfak) OK so new idea: separate queues. One queue per unique set of models. Each requested score is given a... [16:57:58] halfak: thanks for sending this person to me, my hope is that I can rope them in to push on labelling more. [17:00:38] \o/ [17:00:43] Yesss... Perfect [17:41:52] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10kchapman) Reminder that there is an IRC meeting today (Wednesday November 21st) at 11pm PST(November 22nd... [18:00:35] halfak: Amir1: I'm chatting with the Freie Universitat team via email, and suggesting to them that we can whitelist their IP to allow a short period of highly parallel requests while we monitor the servers. [18:00:40] Does that sound sane? [18:02:00] awight: I think it's fine but I would recommand waiting until the hadoop stuff is finished and produce dumps for researchers [18:02:38] Yeah that would be much nicer. I'm still trying to get clarity about what they're doing with the scores, if they need raw features, etc. [18:02:57] awight, +1 [18:03:24] Oh wait. we need to have them sign an MOU :| [18:03:32] And publish their research open access. [18:03:39] WMF policy, I'm afraid. [18:03:52] If we're going through that level of trouble to support their work. [18:04:02] But! That's not too hard. [18:05:31] hmm okay I'll mention and CC you [18:06:50] kk [18:06:58] BS time? [18:08:07] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Implement JS ORES client in mw-ORES extension - https://phabricator.wikimedia.org/T201691 (10awight) Considering the enterprise-y level of machinery going into this library, I suggest making it a CommonJS... [18:08:33] here now [18:08:47] will hop back in a minute [18:24:27] Oh something I forgot to mention that was brought up in the SoS was that php7 stuff is going to be rolled out in prod appservers and they will start serving traffic from the next week [18:34:03] Great to hear! [18:34:18] Amir1: Question about the poolcounter. Do we have the ability to whitelist? [18:34:36] awight: yup, It doesn't work with ranges though [18:34:46] I wanted to implement that but fell off the radar [18:34:54] I didn't see that code in the lock_manager package... [18:35:09] by range I mean CIDR range [18:35:19] it's there I think even you merged it :P [18:35:24] ah okay it's in scoring_system [18:35:36] +1 CIDR, I would need that for the use case I'm looking at [18:36:04] I can add [18:36:44] https://github.com/wikimedia/ores/blob/master/ores/scoring_systems/scoring_system.py#L33 [18:36:53] +1 thanks [18:37:13] awight: FWIW, There is a built-in library to make things really easy for handling CIDR [18:37:44] basically you can say if IP('something') in IP('something/20') [18:38:54] * halfak --> lunch [18:39:37] https://wikitech.wikimedia.org/wiki/Blubber#User_Guide [18:39:41] Disclaimer! This is the beautiful and perfect and absolutely correct and error-free documentation for Blubber. If reality disagrees, reality is unambiguously wrong and needs to be replaced. (Not really. Please update and fix this wiki page instead.) [18:40:07] O_O [18:43:32] 10ORES, 10Scoring-platform-team: Support CIDR range whitelist for ORES throttling - https://phabricator.wikimedia.org/T210103 (10awight) [18:52:28] Someone has a sense of humor [19:01:33] gotta relocate, back in ~30min [19:50:24] I was able to dockerize wikilabels and run pytest using blubber [19:50:46] that was fun but I'm too tired to finish it off [19:53:33] Nice. Sounds like a good time to call it a day then [19:56:42] oh I just started looking at helm [19:56:45] :D [20:00:08] halfak: https://github.com/wikimedia/ores/pull/291/files This might help (+ http://docs.celeryproject.org/en/v4.1.0/userguide/configuration.html#new-lowercase-settings) [20:02:56] Amir1, just finished leaving notes. [20:03:40] halfak: I closed that PR, it was done using #291 [20:04:43] Oh! I wonder how I came across that. [20:08:27] the hardest part was doing it via puppet because some of them are in puppet and you can't mix old and new configs [20:08:34] that was fun :/ [20:09:50] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Implement JS ORES client in mw-ORES extension - https://phabricator.wikimedia.org/T201691 (10Halfak) I don't see how to fit this into a general library. I'm not sure what you mean by "tie the list of mod... [20:10:31] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Implement JS ORES client in mw-ORES extension - https://phabricator.wikimedia.org/T201691 (10Halfak) I should mention that ORES has this constraint for a good reason -- different models need different data... [20:20:14] 10ORES, 10Scoring-platform-team: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Halfak) 05Invalid>03Open I was just working on a big batch job that got hung at score 300k. When I ^C'd the process, I got this: ` Traceback (most recent call last): File "/srv/hom... [20:20:23] Very WIP: https://github.com/wikimedia/wikilabels/pull/251/files The tests is being ran [20:21:53] wikimedia/wikilabels#435 (blubber - 2ad6987 : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/wikilabels/builds/458121174 [20:28:34] 10ORES, 10Scoring-platform-team: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Halfak) https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result suggests that our "result()" call should raise an exception and that would cause the scrip... [20:30:30] awight: I finally watched the gender mag presentation [20:39:01] 10Scoring-platform-team (Current), 10Operations, 10User-Ladsgroup: Spec out migrating ORES to kubernetes - https://phabricator.wikimedia.org/T210109 (10Ladsgroup) [20:39:33] Okay I'm calling it a day. See you all tomorrow [21:16:13] o/ have a good one [21:20:51] halfak: where's the etherpad for the offsite? [21:21:13] https://etherpad.wikimedia.org/p/scoring_fy19Q2_offsite [21:21:47] Thank you [21:57:05] https://meta.wikimedia.org/wiki/User:Halfak_(WMF)/CSCW_2018 [21:57:07] I did it! [21:57:17] One event down, 4 more to go. [21:57:34] Now for my last meeting and then I'm hitting the road. [21:57:41] So I'll just wave now. [21:57:42] o/ [21:57:59] Have a good holiday harej & awight [21:58:14] And anyone else that is doing this Thanksgiving thing :) [22:13:50] whew... 3hours later, sorry. [22:14:53] harej: Right, did GenderMag bring to mind anything we can use for Jade? [22:23:00] *Right on [22:23:12] space bar failureis killing me [22:30:34] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/475186 (owner: 10L10n-bot) [22:51:08] 10ORES, 10Scoring-platform-team: Support CIDR range whitelist for ORES throttling - https://phabricator.wikimedia.org/T210103 (10awight) @Ladsgroup There were a few things I'm thinking of doing with the lock manager, but I wanted to check with you first, since some might go against the design discussions we've... [23:05:39] I’ll have to think more about it, but I think at some point we will want to do the walkthrough that they recommend [23:07:35] nice. Maybe our personas can be merged with theirs, too? [23:09:32] Amir1: lmk if I should not mess upyour lock_manager, I can also do a minimal patch, it'sjust annoying cos it will add to debt [23:10:27] spaa ce bar pleasecome back [23:12:40] halfak: I see config defaults in ORES code, maybe we can take advantage of all the yaml merging and provide a default_config.yaml in thesource repo? [23:21:43] actually, I'm not sure it's so easy. Thedefaults would need to applyto repeated subtrees, for example. [23:59:23] relocating self