[01:18:36] wikimedia/articlequality#132 (bug/T214091 - c59b80d : Marius Hoch): The build passed. https://travis-ci.org/wikimedia/articlequality/builds/486197928 [01:47:30] 10Jade, 10Scoring-platform-team (Current): Jade workshop position paper for HumBL - https://phabricator.wikimedia.org/T214960 (10awight) [02:05:15] 10Jade, 10Scoring-platform-team (Current): Jade workshop position paper for HumBL - https://phabricator.wikimedia.org/T214960 (10awight) [09:56:29] 10ORES, 10Scoring-platform-team, 10Operations, 10vm-requests: New node request: oresrdb[12]003 - https://phabricator.wikimedia.org/T210582 (10Ladsgroup) [09:56:29] El búfer 12 está vacío. [09:56:45] 10ORES, 10Scoring-platform-team, 10Operations: Investigate memory usage of ORES in kubernetes - https://phabricator.wikimedia.org/T210264 (10Ladsgroup) [09:57:10] 10Scoring-platform-team (Current): Write team status update for November 2018 - https://phabricator.wikimedia.org/T208541 (10Ladsgroup) Three months have passed, we should decline this. [11:43:46] 10Scoring-platform-team, 10Wikidata, 10Wikidata-Campsite, 10editquality-modeling, and 2 others: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737 (10Lydia_Pintscher) @Ladsgroup Can you add the list of existing features as we discussed? [11:48:39] 10Scoring-platform-team, 10Wikidata, 10Wikidata-Campsite, 10editquality-modeling, and 2 others: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737 (10Ladsgroup) Sure: ` is_client_move, is_client_delete, is_merge_into, is_merge_from, is_re... [12:39:34] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Sebastian_Berlin-WMSE) What's the next step for this? Is there any more information needed? As part of #wmse-dev-support-2019-q... [14:37:08] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Salgo60) No thoughts of gathering feedback from readers of an article? Both logged in and anonymous users Example: - 1-5 fou... [15:02:40] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @amir1 & @Thiemo_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:23:07] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Johan) @Salgo60: It has been tried. https://en.wikipedia.org/wiki/Wikipedia:Article_Feedback_Tool [15:52:40] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @Thiemo_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:06:15] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10WMSE-Development-Support-2019 (Automatic article quality assessment), 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Sebastian_Berlin-WMSE) [17:07:07] afk for lunch [18:04:30] Amir1: there is an interesting drop in scores in https://grafana.wikimedia.org/d/000000255/ores?orgId=1&from=1548867939860&to=1548870584688 [18:04:31] fyi [18:04:54] that looks pretty wrong. [18:04:57] * halfak digs. [18:05:50] No errors or overloads. But it looks like codfw disappears. Could it be changeprop in CODFW? [18:06:17] back now [18:07:08] akosiaris: it seems eqiad went half, codfw went to zero, it seems something with precache :D [18:07:30] it's very likely change prop [18:08:05] Yeah. No errors showing up. All signs seem to point to a change in requests. [18:08:49] akosiaris: tell me when you're around for the deploy [18:09:17] Amir1: I am [18:09:57] let's do it [18:20:51] 10ORES, 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: feature.len() has already been added to {wikibase.revision.diff}. Could be overwritten? - https://phabricator.wikimedia.org/T214979 (10Ladsgroup) [18:24:56] hey halfak, when are we talking about embeddings ? [18:25:47] Good Q! Soon. I had a chat with Search folks and they have been doing a bit of experimenting but they aren't quite sure what to do with the tech yet. They are very interested in making good use of embeddings though. [18:26:03] I'm not sure when we'll have time to sync up. How about lunch? [18:26:06] dsaez, ^ [18:27:47] 10ORES, 10Scoring-platform-team, 10translatewiki.net, 10Patch-For-Review, 10Security: New ORES model relies on translatewiki.net API, which is not hosted on WMF production - https://phabricator.wikimedia.org/T213131 (10Ladsgroup) As a security measure, I disabled translatewiki on prod. It's an obvious ca... [18:28:39] halfak, lunch is good [18:31:11] afk to run an errand [18:32:11] PROBLEM - ores grafana alert on icinga1001 is CRITICAL: CRITICAL: ORES ( https://grafana.wikimedia.org/d/000000255/ores ) is alerting: Scores processed (not cached) alert. [18:33:29] RECOVERY - ores grafana alert on icinga1001 is OK: OK: ORES ( https://grafana.wikimedia.org/d/000000255/ores ) is not alerting. [18:52:51] 10ORES, 10Scoring-platform-team, 10ChangeProp, 10Wikimedia-Incident: Investigate precaching blip (ChangeProp?) during Jan 30th ORES deployment - https://phabricator.wikimedia.org/T214982 (10Halfak) [18:53:36] 10ORES, 10Scoring-platform-team, 10ChangeProp, 10Wikimedia-Incident: Investigate precaching blip (ChangeProp?) during Jan 30th ORES deployment - https://phabricator.wikimedia.org/T214982 (10Halfak) Related: {T213116} [18:54:31] 10ORES, 10Scoring-platform-team, 10Scap: Investigate: why are we getting overload errors during ORES deployments? - https://phabricator.wikimedia.org/T213116 (10Halfak) [18:54:34] 10ORES, 10Scoring-platform-team, 10ChangeProp, 10Wikimedia-Incident: Investigate precaching blip (ChangeProp?) during Jan 30th ORES deployment - https://phabricator.wikimedia.org/T214982 (10Halfak) [18:55:12] akosiaris, ^ [18:55:20] I documented what I could so we can look again later. [18:55:56] I'd like to understand what leads to overload blips in general but for this task, I want to focus on what happened with ChangeProp. [18:56:05] Anyway, back to allhands mode :) [18:56:27] * halfak closes laptop [19:05:05] 10ORES, 10Scoring-platform-team, 10translatewiki.net, 10Patch-For-Review, 10Security: New ORES model relies on translatewiki.net API, which is not hosted on WMF production - https://phabricator.wikimedia.org/T213131 (10Joe) [19:36:06] halfak: around? [19:36:14] Hi hoo! [19:36:18] I'm 50% here :) [19:36:28] Am sitting in an allhands event, but I could answer a question. [19:36:31] :D [19:36:59] So I'm done adding words to watch to articlequality (https://github.com/wikimedia/articlequality/pull/74) [19:37:22] I can continue w/ editquality, but I would collide with Amir1's rebuild of all models there [19:37:29] https://github.com/wikimedia/editquality/pull/183 [19:37:51] hoo: don't worry about mine, I need to discard it [19:38:07] It doesn't work, I need to change and rebuild anyway [19:38:19] good to know [19:38:27] in that case I'll tackle editquality next [19:39:54] 10ORES, 10Scoring-platform-team, 10Scap: Investigate: why are we getting overload errors during ORES deployments? - https://phabricator.wikimedia.org/T213116 (10Ladsgroup) >>! In T213116#4860787, @awight wrote: > Maybe this is because scap is restarting the services in serial, but not accounting for the long... [19:43:01] 10Scoring-platform-team (Current), 10draftquality-modeling, 10artificial-intelligence: Implement and test words_to_watch features for draftquality - https://phabricator.wikimedia.org/T214091 (10hoo) For articlequality: https://github.com/wikimedia/articlequality/pull/74 [19:44:29] Amir1: halfak: Do you have anything good to pick up next? [19:45:14] hoo, we should have a chat about how you like this kind of work and if you would like to continue similar work or branch out. [19:45:51] If similar work, I'd recommend implementing the embedding-based features in editquality and draftquality. We use them in drafttopic and I think they have potential for other modeling work. :) [19:45:59] This is actually quite fun (if you count out the troubles I had with the various fetch/extract utilities :P) [19:46:37] Heh. I hear you there. It's nice to have your view of new model dev to help prioritize what we fix :) [19:46:55] OK. I have a task for implementing the embeddings-based features. Let me find it. [19:47:07] It's not fully spec'd yet, but I can work on that with you next week. [19:47:23] Sounds interesting [19:47:32] https://phabricator.wikimedia.org/T197007 [19:48:19] * halfak digs for a code example. [19:48:46] https://github.com/wikimedia/drafttopic/blob/master/drafttopic/feature_lists/wordvectors.py [19:49:18] Here, a "word2vec" "FeatureVector" is engineered. It results in 300 feature-values, but revscoring handles that somewhat transparently. [19:50:46] There will be some cleverness involved in figuring out how to use the vectors for editquality. But I think draftquality can use a similar strategy to drafttopic [19:51:13] I need to run away again in 10 minutes. Photos + lunch. [19:51:56] halfak: So you take all (non stop) words, make a vector from that, take the mean and that is the signal? [19:52:00] dsaez, It looks like there is a tech PM lunch that was organized and I forgot about. Can we meet up for lunch tomorrow? [19:52:11] hoo, right. [19:52:17] halfak, ok np [19:52:29] Great. Sorry for the confusion [19:54:21] 10ORES, 10Scoring-platform-team, 10ChangeProp, 10Wikimedia-Incident: Investigate precaching blip (ChangeProp?) during Jan 30th ORES deployment - https://phabricator.wikimedia.org/T214982 (10Ladsgroup) There are similar blips in CP dashboard: https://grafana.wikimedia.org/d/000000201/eventbus?orgId=1&var-si... [19:55:52] 10ORES, 10Scoring-platform-team, 10ChangeProp, 10Services, 10Wikimedia-Incident: Investigate precaching blip (ChangeProp?) during Jan 30th ORES deployment - https://phabricator.wikimedia.org/T214982 (10Pchelolo) [19:56:59] halfak: Ok… that's for next week… anything quick-ish I can pick up today/ tomorrow [19:57:01] Amir1: ^ [19:57:24] hoo, the editquality work with words-to-watch sounds good to me. [19:57:55] Off I go! Back in 1.5 hours-ish [19:58:24] hoo: if you want to, https://phabricator.wikimedia.org/T214979 I just found it in logs [19:58:42] https://phabricator.wikimedia.org/T214841 and this [20:08:02] Amir1: I'll check them out, thanks [21:33:29] o/ [21:33:31] hoo, just got back [22:04:43] halfak: I think this is wrong: https://github.com/wikimedia/revscoring/pull/419/commits/e4827fbbda6514515d02f08df744051cfb76127c#diff-f0f51e0e990e3daa4ad6c9381ea709f9R62 [22:05:24] Amir1, hey! What's up? [22:05:42] It actually turns the registration datasource to a normal datasource and doesn't determine how to get it from its dependency [22:05:57] My knowledge in this part of oress codebase is pretty basic though [22:06:30] halfak: I think it should be self.timestamp = Timestamp(self.timestamp_str, name + ".timestamp") [22:06:33] Tell me more about the problem you're running into and what this isn't doing. [22:07:01] The whole thing doesn't work (installing and rebuilding the models didn't fix the json issue) [22:07:12] so I'm looking back to see what's wrong [22:07:27] I'm sure there's something specific that is going wrong. I'm not even sure what you think is wrong with this chunk of code. [22:08:14] What error are you running into? [22:08:21] Yeah, The part I have trouble is with this type there isn't a way to do process on the datasource [22:08:23] Is something not JSON-able in the root datasources? [22:08:32] yup, the timestamp [22:08:43] There is. That's what the "or_none()" function call does. [22:08:49] it passes around the timestamp as a rootdatasource [22:08:53] oh I see [22:09:13] Maybe you are seeing an error when ORES tries to JSON-ify something? [22:09:55] no, I made it print the root datasources in ores-staging' [22:09:59] and got it from log [22:10:15] it really tries to pass timestamp [22:11:10] it might be that it didn't pick up the revscoring update. Maybe it needs a release (I installed it in venv when building the models + installed it in venv of ores-staging both from master) [22:11:30] maybe I should vim the files and see if it has the new updates [22:11:57] It's possible that it doesn't discover that "timestamp_str" is the root. Could be a version issue or maybe "dig()" isn't getting to it for some reason. [22:12:20] * halfak tries to dig() [22:12:25] yeah, that's my only explanation so far [22:14:00] halfak: found it, the server doesn't have it [22:14:12] Gotcha. :) [22:14:15] let me check if my venv in stat1007 have it too [22:14:26] we should release a new version anyway [22:14:48] Can I do a release quickly? [22:16:07] https://phabricator.wikimedia.org/P8049 [22:16:13] Oh sure. I'll kick one out. [22:17:19] OK 2.3.1 is in pypi [22:18:13] halfak: https://github.com/wikimedia/revscoring/pull/426 [22:18:30] oops already done [22:19:05] Yeah. Since it's just a version change, I usually just push to master. I'm certainly open to discussion, but it seems safe to me. [22:19:20] It's fine for me [22:19:36] Maybe for major changes to write release notes [22:19:41] but beside that. meh [22:20:50] wikimedia/revscoring#1589 (release_json_to_the_world - 605dab7 : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/revscoring/builds/486649153 [22:24:21] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/487233 (owner: 10L10n-bot) [22:47:07] wikimedia/editquality#461 (json_datasource_models_try_ii - 3bcb637 : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/editquality/builds/486660443 [23:16:35] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 (10Ladsgroup) Okay, It seems I made some progress: ` Traceback (most recent call last): File "./ores/wsgi/routes/v3/util.py"... [23:25:55] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 (10Ladsgroup) https://github.com/wikimedia/revscoring/pull/427 [23:26:08] https://github.com/wikimedia/revscoring/pull/427 [23:26:14] and https://github.com/wikimedia/revscoring/pull/425 [23:26:25] I need to catch the last train [23:26:29] see you tomorrow [23:59:29] wikimedia/revscoring#1593 (json2 - e46c3fc : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/revscoring/builds/486677121