[15:17:15] o/ Amir1 [15:17:31] It looks like our periodic task cleanup script isn't working for Wikilabels. [15:17:37] Do you remember how it worked? [15:18:00] halfak: hey [15:18:07] We have https://github.com/wiki-ai/wikilabels/blob/master/wikilabels/utilities/remove_expired_tasks.py [15:18:16] Do we just have a cronjob to run it. [15:18:20] It works, I have seen it several times in the past week [15:18:23] Or did we puppetize it somehow? [15:18:29] yes, it's a cronjob [15:18:38] we should puppetize it but I'm not sure how [15:18:39] Weird. Looks like the item quality pilot is jammed. Let me double-check. [15:19:21] I definitely have tasks that are not getting removed after 48 hours. [15:19:25] https://usercontent.irccloud-cdn.com/file/LjmnHvwN/ [15:19:44] It closed my workset for fawiki which is around two days old [15:19:59] Hmm... yeah. nevermind that looks good. [15:31:49] Amir1, just confirmed that we have many worksets that are supposed to expire several days ago. [15:32:19] halfak: I have a hypotheses, do you know what type they are? [15:32:20] Maybe our cleanup query is failing. [15:32:37] Campaign 49 in wikidata. [15:32:39] It might be that it only cleans up the damaging campaigns [15:32:44] Hmm [15:33:08] The damaging campaigns have the property that they only need one label per task. [15:33:16] Whereas this campaign needs two. [15:33:35] FYI: https://gist.github.com/halfak/9ee3f7c2bd0e275b6addc3cdaf9b75f4 [15:34:10] I'll look into this. [15:34:37] * halfak looks around for awight [15:35:45] halfak: 0 0 * * * /srv/wikilabels/venv/bin/python /srv/wikilabels/config/submodules/wikilabels/utility remove_expired_tasks --config=/srv/wikilabels/config/config/ [15:35:52] Thanks [15:35:52] this is the cronjob [15:36:00] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is not cleaning up expired tasks for Wikidata item quality campaign - https://phabricator.wikimedia.org/T161521#3134150 (10Halfak) [15:36:34] halfak: (you can see / edit it using "crontab -e" in wikilabels-01) [15:36:47] is it the root crontab? [15:42:40] Amir1, I was right about the labels_per_task affecting the cleanup script. [15:42:46] I'll get a PR together soon :) [15:42:51] glorian_wd, ^ [15:42:58] FYI this is a blocker for the pilot finishing. [15:43:22] halfak: it's in my username. Can you see it? [15:43:58] Amir1, I might be able to. I think that's all good though [15:44:12] Awesome [15:44:12] It looks like the deletion query is what is wrong here. [15:44:19] And I can fix it easily. [15:44:33] I think I might give it a run before submitting the pull request so I can unblock [15:45:02] 4218 rows! [15:45:31] * halfak takes a backup just in case. [15:51:12] https://phabricator.wikimedia.org/T161525 [15:51:13] :D [15:51:48] lol [15:53:25] wow. Deleting these tasks takes a *looong time* [15:53:48] Strangely, querying get the tasks is really fast [15:53:51] wooo! it worked! [15:54:45] Uh oh. I think I broke something [15:55:25] Not on prod [15:55:28] On my local box [15:55:49] Yup. Somehow deleted all of workset_task [15:58:00] 06Revision-Scoring-As-A-Service, 10ORES: Implement ScoreRequest/ScoreResponse pattern in ORES - https://phabricator.wikimedia.org/T161526#3134359 (10Halfak) [15:58:06] 😂 [15:58:07] 06Revision-Scoring-As-A-Service, 10ORES: Implement ScoreRequest/ScoreResponse pattern in ORES - https://phabricator.wikimedia.org/T161526#3134374 (10Halfak) a:03Halfak [15:58:28] 06Revision-Scoring-As-A-Service, 10ORES: Exclude precaching requests from cache_miss/cache_hit metrics - https://phabricator.wikimedia.org/T159502#3069692 (10Halfak) [15:58:30] 06Revision-Scoring-As-A-Service, 10ORES: Implement ScoreRequest/ScoreResponse pattern in ORES - https://phabricator.wikimedia.org/T161526#3134359 (10Halfak) [15:59:39] 06Revision-Scoring-As-A-Service, 10ORES: Implement ScoreRequest/ScoreResponse pattern in ORES - https://phabricator.wikimedia.org/T161526#3134359 (10Halfak) https://github.com/wiki-ai/ores/pull/191 [16:00:14] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is not cleaning up expired tasks for Wikidata item quality campaign - https://phabricator.wikimedia.org/T161521#3134396 (10Halfak) https://gist.github.com/halfak/9ee3f7c2bd0e275b6addc3cdaf9b75f4 [16:04:43] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3134403 (10Halfak) [16:55:01] 06Revision-Scoring-As-A-Service, 10Wikilabels, 15User-Ladsgroup: Fix RTL issues in Wikilabels after OOjs UI upgrade - https://phabricator.wikimedia.org/T161533#3134553 (10Halfak) [16:55:08] 06Revision-Scoring-As-A-Service, 10Wikilabels, 15User-Ladsgroup: Fix RTL issues in Wikilabels after OOjs UI upgrade - https://phabricator.wikimedia.org/T161533#3134571 (10Halfak) https://github.com/wiki-ai/wikilabels/pull/169 [16:55:20] 06Revision-Scoring-As-A-Service, 10Wikilabels, 15User-Ladsgroup: Fix RTL issues in Wikilabels after OOjs UI upgrade - https://phabricator.wikimedia.org/T161533#3134574 (10Halfak) 05Open>03Resolved p:05Triage>03High [16:58:19] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikimedia-Logstash, 13Patch-For-Review, 15User-Ladsgroup: Send ORES logs to logstash - https://phabricator.wikimedia.org/T149010#3134607 (10Halfak) a:05Ladsgroup>03None [18:06:26] o/ halfak , Amir1 [18:07:11] So I am wondering, has the problem pertaining to WikiLabels already solved? [18:09:07] not yet [18:09:14] Meetings are blocking me :\ [18:09:17] oh okay [18:09:19] I saw an increase in the number of items that have been labeled. [18:09:21] But I know the problem and the solution [18:09:28] So, I thought the issue has fixed [18:09:49] halfak: okay. Let me know once you're done with it. I might want to pull some people again :) [18:10:30] sure [19:23:21] glorian_wd, done [19:23:28] okay [19:23:33] halfak: thanks. [19:24:07] halfak, Amir1: would be cool, if you can label some sets too :P. I plan to post on project chat to remind people that we have handful of sets left [20:04:24] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is not cleaning up expired tasks for Wikidata item quality campaign - https://phabricator.wikimedia.org/T161521#3134150 (10Halfak) https://github.com/wiki-ai/wikilabels/pull/170 [20:04:46] glorian_wd, I'm rushing to do other work. Also, i'm trying to enforce my wikibreak from Wikidata [20:05:12] In the next two hours I have to train/test 3 novel models and incorporate a bunch of language assets [20:58:07] wiki-ai/revscoring#909 (names_and_normalization - 02a86fc : halfak): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/215694499 [20:58:27] what [21:14:20] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3135387 (10Halfak) https://gerrit.wikimedia.org/r/345016 -- "Adds hunspell-ko to ores:base [puppet]" [21:21:57] Building hewiki models [21:26:37] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: ORES logs not being saved to logstash - https://phabricator.wikimedia.org/T161563#3135423 (10EBernhardson) [21:30:53] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: ORES logs not being saved to logstash - https://phabricator.wikimedia.org/T161563#3135442 (10EBernhardson) In production this error has been logged 704,874 times between 2017-03-27T06:25:20 and 2017-03-27T21:28:06.41, or just under 800 times per minute. The full... [21:32:46] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10Wikimedia-Logstash: ORES logs not being saved to logstash - https://phabricator.wikimedia.org/T161563#3135445 (10EBernhardson) [21:34:58] wiki-ai/revscoring#909 (names_and_normalization - 02a86fc : halfak): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/215694499 [22:15:19] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3135516 (10Halfak) {P5144} [22:25:39] wiki-ai/revscoring#911 (korean - a8bc4b4 : halfak): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/215722312 [22:29:47] wiki-ai/revscoring#913 (korean - ed9ba77 : halfak): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/215724063 [22:32:02] wiki-ai/revscoring#915 (names_and_normalization - 3681c30 : halfak): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/215724731 [22:55:24] * halfak trains all the models. [22:55:35] It's very tempting to take a guess at the right parameters for a model. [22:56:04] It would be cool if the report built up progressively. [23:08:42] hewiki models are building. [23:09:06] Amir1, are you still around? [23:09:24] halfak: I just woke up [23:09:47] I slept right after after the meeting [23:10:18] Oh man. That doesn't seem like enough. I should let you sleep :/ [23:10:53] But since you're up, I just wanted to check on your schedule tomorrow and if you could spare some time for some review around 1400 UTC [23:10:58] nope, I myself woke up like ten minutes ago and couldn't sleep [23:11:10] I probably take a nap during the day [23:11:21] OK :) [23:11:39] If I get up early, I'll have the deploy in order by 1400 UTC and that seems like an hour you're usually around. :) [23:12:05] Looking at my calendar it looks like it's right after the weekly pm meeting for wikidata [23:12:15] I'm training the hewiki models now and I think I might even get the kowiki model building by EOD [23:12:26] derp. [23:12:29] I won't participate at the second PM meeting [23:12:31] I should have just looked at that. [23:12:35] Gotcha. perfect :) [23:12:47] That time looks okay to me [23:13:16] Great! That'll give me enough time to address any derps :) [23:13:37] halfak: The services deployment window is 17 UTC [23:13:38] https://wikitech.wikimedia.org/wiki/Deployments [23:13:46] One hour after that [23:14:07] nope, three hours :D [23:14:13] :P [23:15:27] halfak: I'm just loading up, so what do you need me to do in 1400 UTC? [23:16:13] Just be around to help me get a few changes merged. [23:16:24] In that case sure! [23:16:25] Prepare for review :) [23:16:27] \o/ [23:20:45] halfak: Have you tested this in your localhost? [23:20:46] https://github.com/wiki-ai/wikilabels/pull/170 [23:21:33] Amir1, yes. Also in prod O.O [23:21:36] I took a backup. [23:21:38] :D [23:22:45] Check it out: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [23:23:45] That's worrying :D