[06:42:34] halfak: what does "Amir1, we need this precached to be as close to the web nodes as possible" mean ? [06:43:48] * akosiaris still not sold on the precached being on the scb clusters btw. If we are indeed going to use change propagation for that functionality, why can it not be in labs until then ? [07:37:43] 10Revision-Scoring-As-A-Service-Backlog: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2398199 (10mobrovac) Would this describe the usage: //for every new revision not made by a bot, make a request to ORES// ? If so, what does the request look like? Can... [07:38:08] 10Revision-Scoring-As-A-Service-Backlog, 10EventBus, 06Services, 15User-mobrovac: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2398201 (10mobrovac) [08:50:46] hey halfak! [09:37:19] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2398397 (10schana) @halfak, to clarify, this task is referencing the labs instance, correct? How has the infrastructure changed for the production d... [10:25:24] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2254643 (10Johan) There are [[ https://meta.wikimedia.org/wiki/User_talk:Shanmugamp7#ORES | comments on this on Meta ]]. [10:25:25] 10[2] 04https://meta.wikimedia.org/wiki/https://meta.wikimedia.org/wiki/User_talk:Shanmugamp7%23ORES [12:24:21] Amir1 (IRC): merged [14:16:12] o/ Amir1 [14:16:18] I see you doin stuff :) [14:16:30] hey, it's hackathon [14:16:33] :D [14:16:43] mostly talking and fixing things [14:16:52] right now I'm going to deploy to wikilabels [14:16:57] (first, staging [14:17:11] Great! [14:17:15] We needed that [14:17:39] I'm working on getting your docs together for the workshop [14:17:59] nice, thank you :) [14:18:14] 06Revision-Scoring-As-A-Service, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2399241 (10Johan) [[ https://sv.wikipedia.org/wiki/Wikipedia:Etiketter | Yes ]]. There was a discussion on the Swedish Village Pump and I had hoped someone else would take char... [14:18:15] 10[3] 04https://meta.wikimedia.org/wiki/https://sv.wikipedia.org/wiki/Wikipedia:Etiketter [14:18:47] staging looks okay, we are deploying two main changes (first, changing assigning system, second fixing ltr thingy) [14:19:33] Amir1, have you tested out the gadget on staging? [14:19:46] You can actually load the loader.js on staging now and it'll work as expected. [14:19:46] yes, did it right now [14:19:54] As a user-script that is. [14:19:55] (I just deployed it into staging) [14:20:15] https://meta.wikimedia.org/wiki/User:Wiki_labeler/global.js [14:21:38] yeah [14:23:47] everything works just as expected [14:25:06] halfak: last part, Can you update CSS and js for wikilabels? [14:25:17] Sure! [14:25:21] Ready now? [14:26:26] Amir1, ^ [14:27:01] yeah [14:27:07] sorry, went afk for Amir [14:28:50] OK no diff to CSS, but there were minor differences to JS and some new lang stuff. [14:31:30] are you sure about CSS [14:31:41] I thought we made some changes there [14:38:33] Amir1, yeah. Just double-checked. [14:38:46] Has the deployment made it to labels.wmflabs.org yet? [14:39:24] yeah, I think he didn't remove that part [14:39:29] he is fixing it atm [14:40:11] OK [14:44:36] halfak: I'm making some operational work (mostly making the beta cluster more similar to prod cluster than the labs setup) [14:44:50] Sounds good. [14:45:03] Should I expect the CSS for wikilabels to get updated soon? [14:45:21] yup [14:46:52] two things left 1- celery is dead for beta setup looking to understand why, 2- I need to fix DNS proxies [14:55:10] https://github.com/wiki-ai/ores [14:55:15] 62% python [14:55:17] yessss [14:59:31] :DDD [14:59:47] Amir1, if celery is dead and the workers are alive, check redis [15:00:10] worker service is dead [15:00:16] probably bad config reading [15:00:42] I will get to it, let's first fix the wikilabels, I forgot to add commits to deploy branch [15:03:50] Oh! Yeah. That's something I run into sometimes too. [15:05:58] okay, the issue with it is that sca01 doesn't have enough RAM [15:06:06] let's try something else [15:06:14] maybe building sca03 [15:10:30] o/ schana [15:10:30] Oh! akosiaris, sorry I missed your question earlier re. precached being close to the web nodes. [15:10:30] Essentially, we need the precached utility to be faster than any bots or other tools that might be requesting scores using the RCStream [15:10:30] Proximity gives us a few ms advantage. [15:10:30] And that can make a big difference for making sure that we don't generate duplicate scores. [15:10:30] Regretfully (or happily depending on how you look at it), we don't use a lock to register that a score is being generated so we have a race condition under which a score might be generated twice rather than generated once and returned twice. [15:20:29] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2399533 (10Halfak) Yes. This references the labs install. Re. production, I don't know. @akosiaris? [15:43:00] (03PS1) 10Ladsgroup: Better performance in recentchanges query [extensions/ORES] - 10https://gerrit.wikimedia.org/r/295528 [15:51:51] Amir1, https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service/BWDS_review [15:52:38] 06Revision-Scoring-As-A-Service, 10ORES, 10bwds, 10revscoring: Create documentation for BWDS review process - https://phabricator.wikimedia.org/T138419#2399611 (10Halfak) [15:52:43] 06Revision-Scoring-As-A-Service, 10ORES, 10bwds, 10revscoring: Create documentation for BWDS review process - https://phabricator.wikimedia.org/T138419#2399625 (10Halfak) https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service/BWDS_review [15:54:52] (03PS2) 10Ladsgroup: Better performance in changeslist query by using rc_timestamp index [extensions/ORES] - 10https://gerrit.wikimedia.org/r/295528 [15:55:22] halfak: https://gerrit.wikimedia.org/r/#/c/295528/ [15:55:29] I was working on this with hoo [15:55:39] making performance of the extension much better [15:55:59] Nice! [15:56:55] halfak: one thing that people always tell me [15:57:15] they say words like shit is okay to use in article about shit [15:57:32] and I say it's okay, since we check proportion of words added [15:57:49] I think we should include that in docs too so people know [15:58:01] Amir1: o/ [15:58:07] Krinkle: o/ [15:58:46] o/ Krinkle [15:58:47] :D [15:59:36] hi halfak [15:59:46] missed your ping from earlier [16:00:37] Hey! was wondering what your plans were for the hackathon [16:01:26] halfak: no plans, playing it by ear [16:02:17] (03CR) 10Hoo man: [C: 04-1] Better performance in changeslist query by using rc_timestamp index (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/295528 (owner: 10Ladsgroup) [16:02:18] kk If you don't find something otherwise exciting maybe you could work with the local opsen on getting our health checks fixed. [16:02:29] I'm very nervous with us having sub-par health checks. [16:02:37] We've paid for it with long downtime events recently. [16:05:28] OK Amir1, I think I have the full set of documentation available for your workshop [16:08:23] halfak: nice! thank you! [16:12:20] 06Revision-Scoring-As-A-Service, 10ORES: Document safe method to deploy ORES in prod - https://phabricator.wikimedia.org/T138234#2394194 (10Ladsgroup) https://wikitech.wikimedia.org/wiki/Ores/Deployment#Production_cluster_.28ores.wikimedia.org.29 [16:13:33] (03PS3) 10Ladsgroup: Better performance in changeslist query by using rc_timestamp index [extensions/ORES] - 10https://gerrit.wikimedia.org/r/295528 [16:52:56] halfak, Just realised something [16:53:00] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Steps_to_create_a_user_database_on_the_replica_servers [16:53:06] Other than the tables that obviously contain private user data, the text table is also redacted in labs database replicas. The reason is that it contains deleted revisions which are too hard to remove. Users can use the API, or dumps to access page contents. [16:53:19] Indeed. [16:53:51] According to this, we will not be able to run our queries to extract information from the replica DBs [16:53:58] 06Revision-Scoring-As-A-Service, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2399904 (10Ladsgroup) I just loaded the campaign into the wikilabels, tested, fixed some minor issues and it's ready now! https://sv.wikipedia.org/wiki/Wikipedia:Etiketter [16:54:12] 06Revision-Scoring-As-A-Service, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2399905 (10Ladsgroup) a:03Ladsgroup [16:54:19] Oh, got it. So we'll be switching back again to the main DB once we have extracted the list of pages [16:54:23] Is that correct? [16:55:01] halfak: Have you sent the announcement? [16:55:06] we need it :) [16:56:54] Amir1, uh oh. I haven't [16:56:57] That'll be up next [16:57:21] halfak: btw. svwiki edit quality campaign is just started [16:57:32] I loaded the revs in wikilabels [16:57:43] Great! [16:58:07] we should leave this room, they are closing [16:58:28] first, I need to something and then we are good to go [16:59:13] :P [16:59:19] I'm staying here [17:02:52] :D [17:02:57] afk for dinner [17:03:00] be back soon [17:08:22] halfak, Any way to quickly find out what % of currently existing Qikipedia articles have been created in the last 12 months? [17:54:57] halfak, You around? [18:00:09] Just got back [18:00:11] Was at lunch [18:00:12] Sorry [18:00:19] Cool. [18:00:28] OK. So yeah... finding page creation dates... [18:00:30] * halfak thinks [18:00:40] if only recentchanges went back that far. [18:00:41] I was playing around with the pages data [18:00:41] Hmm [18:00:54] And made a rather interesting realisation [18:01:08] Both rev_id and page_id are chronological in order [18:07:40] halfak: just came back from dinner [18:07:43] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2400081 (10akosiaris) lb rebalance does >>! In T134782#2399533, @Halfak wrote: > Yes. This references the labs install. > > Re. production, I d... [18:07:50] going for another deployment of wikilabels [18:07:59] which should be super straightforward [18:08:55] Amir1, announcement just went out [18:09:04] yes yes yes [18:09:06] thanks [18:09:22] SoniWP, yes. So we can sort by rev_id to get the first revision of many pages. [18:09:31] then I start working on stuff for presentation both for tomorrow and the day after [18:11:32] SoniWP, this will get you the page_ids: select rev_page as page_id, MIN(rev_timestamp) as first_revision from revision GROUP BY page_id HAVING MIN(rev_timestamp) BETWEEN "20150621" and "20160621"; [18:11:47] It's efficient because it uses the compound index on rev_page and rev_timestamp. [18:12:09] This will not be limited to articles [18:12:23] So I'd run this query, dump it into a temp table and then join that to page. [18:12:27] Actually... hmm [18:13:23] SoniWP, here this will do what you want: SELECT page_id, page_title FROM page WHERE page_id IN (select rev_page as page_id from revision GROUP BY page_id HAVING MIN(rev_timestamp) BETWEEN "20150621" and "20160621") and page_namespace = 0; [18:13:39] * halfak runs a test [18:14:25] https://usercontent.irccloud-cdn.com/file/mnUowXob/ [18:14:34] nice RTL support now [18:14:46] thanks to Amir (not me) [18:17:02] Amir1, ready for css/js updates on meta? [18:17:10] halfak: yup [18:17:12] kk [18:17:15] merged into deploy? [18:19:08] Amir1, OK meta pages updates [18:19:15] Ctrl+F5 and enjoy! [18:19:48] yeah, [18:20:00] thanks :) I load from loader.js from the labels.wikimedia.org [18:20:09] which lots of people still do that [18:20:25] halfak, I had a much better idea actually. [18:20:31] Sorry got afk for a while [18:20:50] Basically even the page_id are sorted in chronological order [18:21:20] And for our purposes, we only care about pages that are strictly made in our time frame [18:21:44] Ahh yes. You can find the page_id of the first page created in your time-bound [18:21:58] I think if we just get the page_id of the first page in our time bound, and the last page in our timebound [18:22:23] We can cosntruct a much better query by simply having page_id between this minimum and maximum [18:23:29] Right now, I'm just thinking if there are any quick or easy ways to extract strictly the min pageid in our timeframe [18:23:52] halfak: precaching for prod is down, Can you restart it? or put it somewhere I can control (that would be much better since we can test stuff here) [18:24:22] Gross. OK. [18:24:26] Not sure how you could control it. [18:24:45] I guess I can make a shared user on ores-compute [18:25:30] Back online [18:25:41] Amir1, one option is to feel free to kill the process on ores-compute-01 [18:25:59] If you ever want to start your own, that's fine with me [18:26:03] halfak: is it possible for me to reach out to your screens? [18:26:12] we can run it as a root [18:26:15] Amir1, I don't think so [18:26:26] Sounds like something to avoid [18:26:31] Could run it as www-data [18:26:46] yeah, that's good too [18:27:05] but we can't do "sudo -u www-data -i" in ores-web-03 [18:27:15] I tried it before, didn't work [18:27:43] "This account is currently not available." [18:27:47] Hmm... I'm a fan of just running this as-is for now and letting you sudo kill it if you need to [18:28:10] halfak, I need to be leaving now. Will ping you later with an update on things, if any [18:28:19] I don't want to spend any more time making this hacky crap work better :/ [18:29:29] \o [18:29:52] yeah, I see [18:30:30] See you SoniWP o/ [18:34:51] halfak: https://grafana.wikimedia.org/dashboard/db/ores [18:35:10] you can see when is precaching is down, there are still a very few requests [18:35:11] Blech [18:35:27] See "Scoring Requests" for non-precaching requests. [18:35:28] these requests comes from the extension setups here and there [18:35:49] Oh I see [18:35:58] The precaching requests from extension [18:35:59] :) [18:36:05] no, I mean in the precaching part since the extension sends requests with precaching too [18:36:10] yeah [18:36:27] Yeah. I wonder if we should capture that differently. [18:36:34] E.g. precaching="extension" [18:36:43] and precaching="change_propagation" [18:40:19] yeah, that would be nice [18:40:49] you build the ores service part and I will do the extension and CP part (since that's easier D:) [18:41:10] \o/ sounds good. [20:02:38] 10Revision-Scoring-As-A-Service-Backlog, 10EventBus, 06Services, 15User-mobrovac, 05Wikimania-Hackathon-2016: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2400316 (10mobrovac) We have discussed this today at the Esino Lario hackathon with @Lads... [20:56:55] halfak: http://ores-beta.wmflabs.org/scores/wikidatawiki/reverted/56789/ [20:57:28] ores-beta is up now with the exact configs as production cluster [20:57:38] living in sc03 [20:57:41] *sca03 [20:58:54] https://gerrit.wikimedia.org/r/295586 [20:58:57] Amir1, ^ [20:59:05] Awesome work on beta! [20:59:12] I've got to step away for a little while [20:59:20] telegram will make it to me. [20:59:41] {{merged}} [20:59:41] 10[4] 04https://meta.wikimedia.org/wiki/Template:merged [21:00:26] 06Revision-Scoring-As-A-Service: Develop training session material for Wikimania Hackathon - https://phabricator.wikimedia.org/T134785#2400472 (10Ladsgroup) a:05Ladsgroup>03Halfak [21:03:44] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Make hidenondamaging=1 faster - https://phabricator.wikimedia.org/T138444#2400491 (10Ladsgroup) [21:06:10] for the first time every, we don't have anything in review [21:06:15] woooot [21:06:56] *ever [21:13:28] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2400514 (10Ladsgroup) That's a little bit strange to post something in English in your home wiki when you are not native English speak... [21:13:56] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2400515 (10Ladsgroup) [21:42:45] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#2400541 (10Ladsgroup) [21:43:10] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#2400557 (10Ladsgroup) ores-beta.wmflabs.org [21:46:19] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES, 13Patch-For-Review, 05Wikimania-Hackathon-2016: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#2400566 (10Ladsgroup) [21:50:05] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 05Wikimania-Hackathon-2016: Make hidenondamaging=1 faster - https://phabricator.wikimedia.org/T138444#2400574 (10Ladsgroup) [21:50:21] 06Revision-Scoring-As-A-Service, 10Wikilabels, 05Wikimania-Hackathon-2016: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2400575 (10Ladsgroup) [21:50:29] 06Revision-Scoring-As-A-Service, 10ORES, 05Wikimania-Hackathon-2016: ORES /etc/ores does not get populated on worker nodes - https://phabricator.wikimedia.org/T137813#2400576 (10Ladsgroup) [22:15:09] 06Revision-Scoring-As-A-Service, 10Wikilabels, 05Wikimania-Hackathon-2016: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2400616 (10Johan) Excellent. I'll see if I can engage some of the Swedes who are at the Wikimania hackathon tomorrow.