[07:47:07] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2388018 (10mobrovac) >>! In T137370#2385142, @Halfak wrote: > Do you have suggestions on what steps we might go through to make use of it? I think the first and foremost is to set-up a mee... [08:14:38] 06Revision-Scoring-As-A-Service, 10ORES: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2388053 (10akosiaris) >>! In T137819#2386783, @Ladsgroup wrote: >>>! In T137819#2386716, @Ladsgroup wrote: >> That's an issue with scap::source since they can't use diffusion repos, https://pha... [08:59:52] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: More robust way of handling errors in Cache.php - https://phabricator.wikimedia.org/T137880#2388192 (10Ladsgroup) [11:28:27] (03CR) 10Daniel Kinzler: Add addQuotes to $threshold (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294740 (owner: 10Ladsgroup) [14:29:32] 06Revision-Scoring-As-A-Service, 06Operations, 06Services, 07service-deployment-requests: New Service Request: ORES - https://phabricator.wikimedia.org/T117560#2388657 (10akosiaris) 05Open>03Resolved a:03akosiaris Resolving since ORES has been in production for the past 2 weeks [14:33:43] o/ [14:33:58] Having networking issues today. Should be connected shortly [14:35:58] halfak|Mobile: hey [14:36:07] I'm still in WMDE office [14:36:34] working on stuff. [14:37:32] Cool. Say, I have a pull request in ORES. Could you have a look? [14:37:43] schana: commented on it [14:37:53] Do maybe he wants to follow up [14:38:00] So* [14:41:22] halfak|Mobile: I checked and saw schana is reviewing so I didn't review. Do you need my review too [15:08:45] No worries re that review, Amir1. Would be good to have schana finish it [15:18:30] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2388759 (10DarTar) @halfak do we also have an MOU on file for @Ladsgroup and if so, can we add him [[ https://www.mediawiki.org/wiki/Wikimedia... [15:19:23] hello folks [15:19:35] _o/ [15:19:46] DarTar: hey [15:19:51] I left this ^ on Phab, I should have probably pinged you here instead [15:19:55] hey both [15:21:03] Amir1: congrats on the Wikidata deployment, I haven’t talked to you since it happened [15:21:20] Thanks DarTar :) we are using in both fawiki and wikidata [15:21:45] it would be great to have some sort of announcement (to attract more collaborators for ORES) [15:21:55] my name is in R:Revscoring [15:22:22] but not in that place you mentioned. I think halfak|Mobile will do it :) [15:22:29] halfak now :D [15:22:52] o/ Amir1 [15:22:54] I'm online! [15:22:56] WOOO [15:23:08] yay. I have so many thing to tell to you [15:23:30] I just checked out the scrollback [15:23:40] Can you link me to the phab task that DarTar was referring to [15:23:44] awk to go between phone and PC [15:23:59] https://phabricator.wikimedia.org/T134651#2388759 [15:25:08] o/ DarTar [15:25:19] wasn't sure that an MOU made sense since Amir1 is WMDE staff. [15:25:22] hey halfak [15:25:22] sorry the kids attacked me and I went offline :-/ [15:25:28] but happy to list him as a collaborator [15:25:37] :P Excuses excuses [15:25:51] I think it makes sense, the NDA doesn’t come with a tacit MOU attached [15:26:05] OK. Will draft one. [15:26:12] I realized I'm not staff for WMDE. I'm what they call "freelancer contractor" [15:26:20] :P "staff" [15:26:23] that's why my name is not in the staff list [15:26:28] awesome (you saw from the other email thread we’re having trouble tracking them centrally) [15:26:52] Amir1: :( [15:27:10] don’t know what their contractor policy is [15:27:21] no worries :) [15:27:24] maybe there’s a minimum contract duration for contractors to be listed? [15:28:07] IDK my contract was for six months and got renewed for another six months (which will end in 13 days) [15:29:41] oh weird [15:30:19] we’ll list you as a formal collaborator once halfak gives you the very last piece of paper to sign :D [15:30:34] Will be incoming shortluy [15:30:46] :D [15:30:51] thanks :) [15:31:27] DarTar, do you sign the MOUs these days? [15:31:40] Or do I still need a "C-level" [15:32:08] what do you think of announcement? should we do it? when it would be okay to do? other departments make a big announcements for much less :D [15:32:41] Amir1, we need to draft it. We want comms to help us with timing. [15:32:44] I' [15:32:54] no, I don’t have delegation for that [15:32:54] I'll ping Jeff and CC you and DarTar [15:32:55] yeah, agreed [15:32:57] it’s still Wes [15:33:26] kk Thanks DarTar [15:33:35] sounds good re: Comms, it’s probably going to be super hard to get hold of them in the next days b/c of Wikimania [15:33:43] Amir, I'll be getting you this MOU thingie after I get a signature from Wes. [15:33:49] Amir I might see you there btw [15:33:58] trying to sneak in, undercover, for one day [15:34:20] DarTar, understood. I want to give Jeff the chance to say "Hold on! Don't announce until we are ready." or "Go ahead, we'll do a bigger thing later." [15:34:22] I contacted the organizers to see if they’ll block me at the gates and I need to bribe somebody [15:34:37] lol [15:34:40] halfak: yeah makes sense [15:34:51] :D [15:35:33] DarTar: I'm there the whole wikimania [15:51:57] Amir1: cool, if they let me in I’ll come and find you :D [15:52:07] when’s the ORES session scheduled? [15:52:29] June 24 and June 23 [15:52:52] I’ll try and come in on Friday 24 [15:55:06] DarTar: https://wikimania2016.wikimedia.org/wiki/Programme [15:55:16] yep just saw it [15:55:55] awesome [15:56:01] it would be great [15:56:50] brb. nephew is here [16:09:43] Amir1, between now and the rest of the deployments, I want to do some house cleaning. [16:10:16] Can you help me record tech debt that we have around deployments, docs, unknowns, etc. for ores.wikimedia.org and the extension? [16:10:36] I'm thinking that we should just file a bunch of missing tasks and then loop back to sort them and prioritize [16:11:34] e.g. setting up the separate repos for ores-deploy and ores-wmflabs [16:11:57] And getting precached worked out [16:12:09] Theres' some circularity to the docs too [16:12:11] I'll get that filed [16:13:14] halfak: I was afk for coffee [16:13:25] (Mate actually) [16:13:30] reading [16:14:00] halfak: I tried to do that before we move on [16:14:29] 10Revision-Scoring-As-A-Service-Backlog: Fix circular linking in ORES docs - https://phabricator.wikimedia.org/T138089#2388853 (10Halfak) [16:14:42] so one of the things I think we should get it done is moving GUI [16:14:43] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Documentation: Fix circular linking in ORES docs - https://phabricator.wikimedia.org/T138089#2388866 (10Halfak) [16:14:48] +1 [16:14:55] That one is important [16:15:23] Andrew Otto was here today and I showed him how ORES works and I told him about our current issues (specially swagger being in the main repo) [16:15:36] he suggested that we ship it in wheels [16:16:06] and we can make it work via symlinks [16:16:34] all of our stuff like swagger, wikimedia UI and bootstrap and some other stuff we use [16:16:40] halfak: what do you think of this [16:17:24] making repos untangled is another thing I'm working on atm [16:19:04] +1 for moving to wheels. [16:19:15] Not sure how that will work yet, but I like the idea of packaging it up. [16:20:07] 10Revision-Scoring-As-A-Service-Backlog: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2388883 (10Halfak) [16:20:25] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2366332 (10Halfak) OK. It looks like this is a whole separate task. I've created T138090 so that we can continue the discussion there. [16:21:08] we make a symlink from /srv/deployments/ores/deploy/submodules/wheels/swagger to /srv/deployments/ores/deploy/ores/wsgi/static/swagger [16:21:31] But then it's just a wheel [16:21:34] We need a folder [16:21:42] but since scap makes a separate folder time, it gets confusing [16:21:54] no, we put nltk data in wheels repo [16:22:03] all of wheels are not wheels [16:22:07] Oh. We did? [16:22:10] Great [16:22:12] Yeah [16:22:16] *wheels data [16:22:24] But wait... ORES should be able to host a swagger without ores-deploy [16:22:30] So that wouldn't quite work. [16:22:32] yeah, that was your idea actually [16:23:04] Maybe we can create a pypi project that includes swagger UI elements and wrap that up in a wheel. [16:23:17] I like that better because then ORES could just depend on it. [16:23:31] hmm [16:23:56] 10Revision-Scoring-As-A-Service-Backlog: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2388904 (10Halfak) Right now, we send events to ORES for every edit that is saved. We are currently looking into filtering out bot edits from the precaching system (... [16:33:34] Amir1, will look into it a little bit today and file tasks based on what I work out. [16:33:41] * halfak --> lunch [16:33:49] nice [16:33:58] thanks halfak|Lunch :) [17:05:50] 06Revision-Scoring-As-A-Service, 10ORES: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2389017 (10mmodell) @akosiaris: Sure I can take care of it. [17:36:34] Amir1, so these new diffusion repos mirror github or the other way around? [17:36:41] woops [17:36:44] forgot to rename [17:37:14] are mirrored from github [17:37:21] that's what you wanted right? [17:38:25] Yes [17:38:26] Wooo! [17:41:10] Amir1, just to confirm, we can do the whole PR dance on github now rather than in gerrit. Is that right? [17:41:36] Should we move any of the process to diffusion? [17:42:58] halfak: per what I'm doing right now we shouldn't except for the gerrit repo (deploy repo that we've got diverged) [17:43:31] so for updating submodules, changes in scap, we need to make gerrit patches [17:43:55] but for ores, wikiclass, and editquality everything should be fine [17:44:05] OK great. [17:44:22] Should we consider wiki-ai/ores-wikimedia-config fully diverged? [17:44:40] I would [17:46:30] halfak: also please ship wikimediaUI (which has bootstrap in it) with swagger. [17:46:33] I want to rename that repo "ores-wmflabs-config" and start working with the new ores "applicatiosn" [17:46:35] It would fix the GUI issue [17:46:46] Yeah. I'm working on that first [17:46:52] nice, thanks [17:46:56] Looking into available pypi libraries and how they handle it now. [17:47:10] halfak: agreed [17:47:28] Cool. [18:02:30] 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2389162 (10Ladsgroup) a:05Ladsgroup>03None [18:27:06] OK. I think we can do this [18:27:07] https://gist.github.com/halfak/7e847efab5cbdebc54b7970b2936f814 [18:27:26] I'm going to register a secondary blueprint in flask [18:28:34] I'll have to do this for both Swagger and WikimediaUI [18:29:56] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2380089 (10Paladox) Repo's have been created. [19:08:39] halfak, I got my Tool on Tool Labs up and running (Its named sonitool) [19:08:53] Great! Let me know how the queries go. [19:09:07] I'm running my query on the DB, but wanted to know how I can store them somewhere [19:09:24] Amir1, just finishing up a project for wrapping up swagger UI static assets. It works pretty sell. I'll be working on WikimediaUI next [19:09:27] So we dont have to run a 30 min query everytime we need to access to data :P [19:09:53] SoniWP, "cat myquery.sql | sql enwiki_p > my_output_file.tsv" [19:10:20] I've got to change locations. I should be back online in ~45 mins. [19:36:29] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Untangle deployment repositories - https://phabricator.wikimedia.org/T137819#2390589 (10mmodell) @akosiaris The repository mirroring is broken because of the system gitconfig file on iridium. I've submitted a patch https://gerrit.wikimedia.org/r/#/... [19:45:05] And back! [19:45:06] Woo! 34 minutes [19:45:08] Not bad [19:46:10] halfak, Just in time. [19:46:58] My first query trying to replicate https://quarry.wmflabs.org/query/10293 just finished [19:47:36] great! [19:47:38] The query unexpectedly stopped working at about 30 minuted. [19:47:46] Oh... wait [19:47:54] So it failed? [19:48:07] (I was using the cat method, so I have no way of really knowing if the Query was killed or not. [19:48:20] But just about 30 mins in... Whoosh. [19:49:34] Weirdly though, I think I might have let it run for just over 30 mins the first time I was trying to run it. I was directly running it on MySQL then (via the MariaDB) [19:49:54] (Accidentally ended up Interrupting the first run so had to start over) [19:51:17] *SQL, not MySQL [19:55:01] halfak, If you want another test run for it, I have time for just another run before I need to leave [19:55:29] Since I'm not 100% confident yet that the 30 minute limit is server imposed [19:56:59] yuvipanda, ^ what's the pattern for running long running queries in toollabs? It seems like we're running into trouble and I didn't remember needing to do anything fancy. [19:57:09] Looking at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database and not seeing something obvious [20:06:59] ^ I just ran it from the tool named "sonitool", if it helps [20:09:18] I'm completely swamped, can't really help right now, sorry! [20:12:16] No worries [20:12:37] SoniWP, if you ping the Q again in #wikimedia-labs, you'll likely find someone who might know what's up. [20:13:04] halfak, You mean like right now or later? [20:13:12] Either or [20:13:22] Maybe give it another try so you're sure it's getting killed [20:13:23] I'd reccomend filing a bug instead [20:13:49] ^ follow yuvipanda advice [20:13:53] I recommend that generally :) [20:14:04] Ugh. I think I'm about out of time so I cant try another run now. [20:14:09] OK. Next time :) [20:14:17] Thanks for hacking today :) [20:14:45] Cool. Or you can try running it once if you dont mind/are around for at least 30-40 mins [20:17:13] Sorry. Working on other things. :/ [20:17:39] But the good news is that I'm just about done with the wheelification of swagger. [20:17:48] I found an issue with my past work and fixed it :) [20:18:27] I have no idea what that means, but cool :) [20:20:02] https://github.com/wiki-ai/flask-swaggerui [20:20:11] ^ Still probably not going to help you know what it means [20:20:14] :D [20:24:42] Nope :P [20:35:25] * halfak uploads flask-swaggerui==0.0.1 to pypi [20:41:34] 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2390678 (10Halfak) I started some work to see if we could wrap these assets in a wheel. I worked out that wrapping assets in a python project and then providing access to them via whee... [20:41:53] 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2390679 (10Halfak) a:03Halfak [20:50:04] So... for the wikimediaUI assets, I think that we want to just add a static path via blueprint and that is it. [20:50:14] So! Let's do that! [20:52:39] halfak, Leaving now [20:52:47] The bug was filed at https://phabricator.wikimedia.org/T138111 [20:52:48] :) [20:52:48] o/ SoniWP [20:52:51] ttyl! [20:52:52] \o [20:52:57] Woot for phab tasks :) [20:53:10] Ping me on Hangouts if needed [20:53:12] See ya [20:56:15] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Explore] Spam and Vandalism new page creation - https://phabricator.wikimedia.org/T135644#2390703 (10Soni) First Two Queries attempted to test Tool Labs (tool name - "Sonitool") === QUERY 1 === cat query1.sql | sql enwiki_p > query1_op.tsv QUERY ST... [20:58:47] 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2390708 (10Halfak) I filed a bug at WikimediaUI to ask about how to build from the repo. See https://github.com/munmay/WikimediaUI/issues/2 @Ladsgroup, do you know a good way to do this?