[06:34:12] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2451368 (10Legoktm) p:05Triage>03Unbreak! This is also spamming exception.log [07:44:40] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2451599 (10Ladsgroup) a:03Ladsgroup [07:51:29] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2451649 (10Ladsgroup) Reverted works just fine: https://ores.wikimedia.org/scores/nlwiki/?models=reverted&revids=47071016%7C... [08:08:23] Krinkle: I'm working on it. You have them fixed by today afternoon :) [08:08:31] thanks for pointing out [10:44:52] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2452082 (10Ladsgroup) This one would fix it: https://github.com/wiki-ai/editquality/commit/e3d5a1d9a2ba06f66a608fea328685e8a... [11:14:34] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 07Spike: How to deal with large abstract feature vectors? - https://phabricator.wikimedia.org/T139978#2452113 (10Danny_B) [11:14:36] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2452115 (10Ladsgroup) https://ores-beta.wmflabs.org/scores/nlwiki/?models=damaging&revids=47071016 In beta works now. It's... [11:17:18] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10Wikimedia-Site-requests, 07Community-consensus-needed: Deploy ORES review tool in Turkish Wikipedia - https://phabricator.wikimedia.org/T139992#2452137 (10Danny_B) [11:19:01] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10Wikimedia-Site-requests, 07Community-consensus-needed: Deploy ORES review tool in English Wikipedia - https://phabricator.wikimedia.org/T140003#2452140 (10Danny_B) [11:19:17] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10Wikimedia-Site-requests, 07Community-consensus-needed: Deploy ORES review tool in Polish Wikipedia - https://phabricator.wikimedia.org/T140005#2452142 (10Danny_B) [11:30:23] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2452192 (10Ladsgroup) ores in labs is okay now. [12:51:44] halfak: o/ tell me when you are around [15:43:34] halfak, https://etherpad.wikimedia.org/p/samples_summary [15:44:37] o/ GhassanMas [15:44:43] how do I read "False True"? [15:45:04] afk for coffee, plwiki reverted will reduce 17 MB, the system is working on wikidatawiki reverted [15:45:17] :) [15:45:24] I forgot I had coffee! [15:45:25] WOOO [15:46:36] o/ [15:46:58] False True are edits which were flagged as true but should have been false [15:49:27] Amir1, just saw your notes re. issues with the damaging nlwiki model [15:50:15] Amir1, we need to add CaughtDependencyError to "expected_errors" so that it doesn't fill up the log. [15:50:18] I'll look into that [15:50:55] GhassanMas, just had another look at the etherpad. So "False False" is a correctly predicted non-damage? [15:51:35] False false are prediction that were false but should have been true [15:52:47] Hmm... [15:52:51] This scheme is confusing. [15:53:22] Maybe we could use the standard terms of "False Positive", "False Negative", "True Positive" and "True Negative" [15:56:37] yeah! you are right . "false positive" is the common scheme [16:01:36] halfak: with the new models, it works just fine [16:01:54] We probably just need to rebuild those models. [16:02:12] Just to be clear, the problem seems to happen with the "damaging" model -- and that model is new, right? [16:02:23] Newish [16:04:36] exactly, I just rebuilt those [16:04:56] halfak: https://github.com/wiki-ai/editquality/commit/e3d5a1d9a2ba06f66a608fea328685e8a2d4c7cd [16:05:09] (I pushed the new model to the master :( ) [16:05:27] It's OK. [16:05:42] This is a Fix NOW situation and I don't expect the rebuilding of a model to be contentious. [16:11:18] halfak: I pushed a new commit to add wikidata and plwiki models but it's not there yet: https://github.com/wiki-ai/editquality/pull/40 [16:11:21] https://status.github.com/ [16:15:29] halfak, all set [16:16:19] Thanks GhassanMas [16:16:49] NOP , halfak it should have been like that in first place (: [16:18:45] GhassanMas, do you see any of those "uncommon words" in these lists? https://github.com/wiki-ai/revscoring/blob/master/revscoring/languages/arabic.py [16:19:23] schana, I see you commenting on the refactor PR. Thanks! [16:19:29] I'll try to get to your comments today. [16:19:54] Also +1 for nit-picking the docs :) [16:20:35] Hmm... We should be getting CaughtDependencyError in the logs [16:23:37] halfak, no I didn't find them [16:23:53] Thanks for looking. [16:23:55] Hmm... [16:24:24] GhassanMas, are those "Fact Updates" something that really need human review? E.g. changing a number or something like that? [16:25:42] halfak: it's up-to-date now [16:25:42] https://github.com/wiki-ai/editquality/pull/40 [16:26:38] updating: the new king of Saudi Arabia when , a football player have transferred to a different team [16:27:15] GhassanMas, gotcha. Hmm... So this might be fine as far as false-positives go. It's good that someone reviews them. [16:27:30] It would be better if ORES was prescient though :D [16:27:40] Amir1, looks good. Test stats are good too? [16:27:49] yeah [16:28:05] I tested it in staging :D [16:28:33] {{merged}} [16:28:33] 10[3] 04https://meta.wikimedia.org/wiki/Template:merged [16:28:40] thanks halfak [16:28:46] I make the patches for prod [16:29:31] Still looking into this error situation. I don't think the CaughtDependencyErrors should show up in the logs since DependencyError is an expected super class. [16:29:40] I wonder if it is showing up during feature extraction. [16:29:47] But even then, it looks like it is handled. [16:29:48] Hmm [16:29:49] yeah, it's hard to get it precises regarding facts, considering that many of them being updated by unregistered user [16:29:53] I need to look at these logs. [16:30:00] I pinged legoktm asking for help in -tech [16:30:02] halfak: oh, btw. While testing I realized our docs are out-dated: https://github.com/wiki-ai/revscoring/pull/283 [16:30:48] halfak: you can search for log in logstash.wikimedia.org (NDA LDAP required) [16:30:51] GhassanMas, OK. So it looks like the False positives are mostly OK then. The false negatives we might be able to do something about. [16:31:06] GhassanMas, Do you think there is a pattern with vandals removing markup? [16:31:26] Gotcha. Will try that [16:31:30] Amir1, ^ thanks [16:31:45] :) [16:33:03] Amir1, Oh! I think the error is filling up from the Extension's side! [16:33:14] So the extension logs errors it gets while populating tables. [16:33:29] That's probably a good thing. We should leave this as is, right? [16:33:32] Amir1, ^ [16:33:47] yeah [16:33:51] halfak: I agree [16:33:56] OK cool. [16:34:00] {{done}} [16:34:00] You rule, halfak! [16:34:01] lol [16:34:38] I'm going to run away for Lunch. When I get back I'll look into how we capture features for markup removal and then start addressing schana's notes re the refactor [17:02:36] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2450695 (10Ladsgroup) With the deployment of f472f65 in prod. It's resolved now :) \o/ [17:02:57] deployed in prod [17:03:03] it's just fine [17:03:16] halfak|Lunch: for when you're back: https://grafana.wikimedia.org/dashboard/db/ores [17:04:11] rough day... [17:07:32] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2454182 (10Ladsgroup) 05Open>03Resolved [17:07:45] 06Revision-Scoring-As-A-Service, 10ORES, 07Wikimedia-log-errors: ORES Failed to process datasource.lower ("object has no attribute 'lower'") - https://phabricator.wikimedia.org/T140038#2450695 (10Ladsgroup) I will re-open this before the meeting :) [17:21:20] afk to commute [19:04:51] halfak|Meeting, I have proposed some notes regarding messing with the markup at https://etherpad.wikimedia.org/p/samples_summary [19:05:31] Thanks GhassanMas. This is really helpful. Would you be willing to post this on the discussion page at https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service ? [19:05:51] * halfak wants to keep follow-ups there for reference. [19:05:58] yeah for sure, halfak [19:11:49] I have to go home now, I will post it at the discussion once I have the chance to [19:11:59] Great. Thanks GhassanMas. [19:12:09] This is really important work. Thanks for taking the initiative! [19:12:10] :) [19:12:17] any time (: [19:18:37] halfak: I'm monitoring the ores in prod. Everything works as expected (still look like a memory leak happening too): https://grafana.wikimedia.org/dashboard/db/ores [19:19:08] Amir1, OK. Will see if I can get some focus on that before I pick up the PR again. [19:19:17] two good things: 1- number of errored requests getting really low [19:19:59] 2- we can deploy ores review tool for plwiki too :) [19:21:01] halfak: no rush at all, given the current situation and the pace of reducing memory we have at least a week [19:21:09] until we hit 10% [19:21:40] ha! I'm going to be on vacation next week. [19:21:49] !!! I forgot to bring that up in the sync meeting [19:21:57] I'll be AFK for the entire week. [19:22:00] GOing camping :) [19:22:18] have fun halfak [19:22:20] I didn't know [19:22:28] let me check your calendar :D [19:22:31] :( My bad. [19:22:56] It's okay [19:23:10] I think we can get lots of stuff done by then [19:23:18] I have a monster of a week now [19:29:40] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Make user-centered documentation for review tool - https://phabricator.wikimedia.org/T140150#2454899 (10Halfak) [19:32:37] halfak: tomorrow I will be working on per web node monitoring in prod and deploying ores review tool in trwiki (we postponed it several times) [19:33:01] is there anything else important enough to be taken care of for tomorrow? [19:33:18] (I'm going to hit the bed right now) [19:33:59] Just started another meeting. Will catch up again tomorrow. OK? [19:34:06] (otherwise looks good) [19:34:25] that would be later [19:34:47] send me an email by end of today if you got anything [19:34:51] *late [19:34:55] Sounds good. [19:35:20] I try to see if I can get anything working for memory leak too [19:35:46] OK. i should have notes on that by EOD [19:35:50] Make sure you check the phab card [19:35:56] thanks [19:36:02] see you :) [19:36:04] o/ [20:56:45] 06Revision-Scoring-As-A-Service, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2455266 (10Capt_Swing) Done. Provided @halfak with verbal feedback, and he decided he didn't need anything more formal ATM. [21:08:45] 06Revision-Scoring-As-A-Service, 10ORES: Explore growing memory usage of web workers on scb machines - https://phabricator.wikimedia.org/T140020#2455300 (10Halfak) Reviewing the status of scb1001. We have 16 celery and 72 uwsgi workers running. Each process takes between 2.9 and 3.5% of memory. Obviously... [21:18:46] 06Revision-Scoring-As-A-Service, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2455366 (10Capt_Swing) 05Open>03Resolved [21:20:44] 06Revision-Scoring-As-A-Service, 10ORES: Explore growing memory usage of web workers on scb machines - https://phabricator.wikimedia.org/T140020#2455395 (10Halfak) I've been tracking memory usage using `ps`. Here's two plots show the distribution of *resident* memory usage over time. https://commons.wikim... [21:23:10] 06Revision-Scoring-As-A-Service, 10ORES: Explore growing memory usage of web workers on scb machines - https://phabricator.wikimedia.org/T140020#2455417 (10Halfak) Better, these two plots show how memory usage changes per-process: https://commons.wikimedia.org/wiki/File:Ores.per_process.celery_memory_usage_ov... [21:24:28] 06Revision-Scoring-As-A-Service, 10ORES: Explore growing memory usage of web workers on scb machines - https://phabricator.wikimedia.org/T140020#2455422 (10Halfak) I'll update these graphs in the next hour. [21:25:38] 06Revision-Scoring-As-A-Service, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2455437 (10Halfak) Notes are here: https://etherpad.wikimedia.org/p/ores_review_tool_heuristic_review Followup is T140150 [22:12:11] 06Revision-Scoring-As-A-Service, 10ORES: Explore growing memory usage of web workers on scb machines - https://phabricator.wikimedia.org/T140020#2455654 (10Halfak) Graphs updated. I'm going to call it a night, but if someone else could run the following commands on scb1001 before I get back and record the res...