[00:00:37] Ethanol is 0.44nm as a vapor... [00:00:50] +1 awight [00:00:54] It wouldn't taste as good [00:00:59] re. the task [00:01:10] * halAFK learns how to make hard liquor from beer. [00:01:11] hehe not the helmet-mounted distillery, then [03:27:15] (03CR) 10jenkins-bot: build: Updating mediawiki/mediawiki-codesniffer to 23.0.0 [extensions/JADE] - 10https://gerrit.wikimedia.org/r/474009 (owner: 10Libraryupgrader) [05:06:34] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Spanish Wikiversity - https://phabricator.wikimedia.org/T209670 (10Lsanabria) [05:08:33] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Spanish Wikiversity - https://phabricator.wikimedia.org/T209670 (10Lsanabria) The translation for "Edit quality (20k sample)" is "Editar calidad (20k muestra aleatoria)" [05:17:26] 10Scoring-platform-team: Read through teahouse literature to find exact outcome metric. - https://phabricator.wikimedia.org/T209652 (10notconfusing) [05:18:30] 10Scoring-platform-team: Read through teahouse literature to find exact outcome metric. - https://phabricator.wikimedia.org/T209652 (10notconfusing) [06:31:28] (03CR) 10jenkins-bot: build: Updating mediawiki/mediawiki-codesniffer to 23.0.0 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/474064 (owner: 10Libraryupgrader) [08:29:15] 10ORES, 10Scoring-platform-team, 10monitoring, 10Wikimedia-Incident: Create Grafana graph to show number of ORES API requests per user-agent - https://phabricator.wikimedia.org/T182222 (10fgiunchedi) My two cents: given the big cardinality of `user-agent` header I think doing this with grafana (and thus gr... [09:00:45] 10ORES, 10Scoring-platform-team, 10monitoring, 10Wikimedia-Incident: Create Grafana graph to show number of ORES API requests per user-agent - https://phabricator.wikimedia.org/T182222 (10akosiaris) FWIW, I 'll echo @Ladsgroup and @fgiunchedi. Having the data is obviously useful. Representing them in grafa... [10:30:01] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Spanish-Sites, 10artificial-intelligence: Edit quality campaign for Spanish Wikiversity - https://phabricator.wikimedia.org/T209670 (10MarcoAurelio) [10:43:53] o/ [11:26:58] akosiaris: hey, do you have a minute to merge this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/474157 [11:27:14] It's very unlikely to cause anything as the renamed happened really long time ago [12:06:37] Amir1: yeah, merging [16:47:44] 10JADE, 10Scoring-platform-team (Current), 10Documentation: Write glossary of JADE concepts - https://phabricator.wikimedia.org/T204905 (10awight) [16:59:57] what is the meeting link? [17:00:10] oops! [17:00:15] nvm found it [17:00:22] https://meet.google.com/jzp-npps-khm [17:03:50] Amir1, meeting! [17:03:59] It's a makeup meeting for Monday [17:05:21] Amir1: answered to your email (quite a bit of work ;-). In case it wasn't clear I 've also restarted uwsgi and celery workers earlier today to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/474157 (although it was a noop, just to be on the safe side) [17:05:39] note btw that we got bit for a different thing by https://phabricator.wikimedia.org/T209628 [17:05:56] I 'll be changing things in the redis ores dbs to avoid this biting us in the future [17:43:52] halfak: hey, I said I might not be able to participate (wrote it yesterday here and put a maybe in the invite) [17:45:12] akosiaris: Thanks! regarding that part, it's a good idea but if we want to do it, we need to have two replicas (according to the docs), we might need to get another node per dc [17:49:08] notconfusing: I'm trying to catch up with your project so I can help with code review and stuff... Do you have regular hours that I should know about? [17:53:07] i work during business hours (mexico city time). but on which days I dedicate to CivilServant and which to WMF i don't have a pattern, maybe i should. Maybe thurday-fri wikimedia is better. [17:53:43] the things i could use help with right now are modelling verifications, and later i would need code review in creating the package [17:53:56] the newcomerquality package [17:54:17] so maybe we can setup a pair-programming work date for next week to grind on that? [17:55:16] to work on packaging? can do [17:56:02] at some point i need to convert from ipython noteboook to python-package, so yes that would probably be good for next week [17:56:54] notconfusing: Which days you work is your business IMO, I think irregular days are equally valuable to us. However, if you will know ahead of time then maybe we should set up some lightweight communication about it, like giving you access to post your "on" days to the scoring platform calendar? [17:57:18] Sure I might have thoughts about the packaging, let's do that. [17:57:33] The plan is still to build a standalone thing, outside of ORES right? [17:57:49] maybe with a dependency on ORES, of course. [17:57:53] yes, the plan is to create a package that hits the ORES api [17:58:41] we tried building it into ORES but it was a big architecural undertaking to have the datamodel accept a list of edits rather than a single edit [17:59:21] +1 [18:27:57] wikimedia/ores#1140 (celery4_config - 35fc735 : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/ores/builds/456099304 [18:51:49] I'm done for the day, have a nice weekend [18:53:29] bye! [20:08:34] Hey. So my calendar just filled up with meetings so it was sort of a lie [20:09:09] But I'll stay connected via my phone until I fly in a couple hours [20:53:37] checkout this Precision Recall Population Graph. We can acheive if we select the best 3rd we can still acheive 95% accuracy [20:53:39] https://imgur.com/xRO3GQY [21:11:01] notconfusing: What's the x-axis about? Is this the learning curve? [21:11:52] Also, which outcome are P-R from, "session is damaging"? [21:12:07] so in these graphs, what happens is you order the predictions by confidence. so x=0.3 means we are considering the top 30% most confident predictions, then you see what the precision and recall are in those locations [21:12:23] "session is goodfaith" [21:12:41] because i want to send goodfaith editors to the teahouse, even if they make a damaging mistake [21:13:21] Oh, that's a great type of graph! [21:16:51] notconfusing: I'm curious about the statistics of that most confident third, e.g. whether they're longer or shorter sessions... [21:18:26] well, that's a good point, actually, gradient-boosting is slightly outperforming logistic regression, but if we use logistic regression its really easy to say (this person was recommended because they had a "long session length" or any other criteria [21:18:39] let me rerun this thing with just the non-singletons and see how it fairs [21:18:43] LIME can linearize :) [21:19:13] I'm sure you have example code, but in case not; https://github.com/adamwight/ores-lime/blob/master/Explain%20edit%20quality.ipynb [21:21:46] But I'm sidetracking--this end of the model outputs is already good enough to use for your HostBot experiment, then? [21:22:41] yes, i think it's sufficient [21:22:45] what's LIME? [21:23:21] ok, so you get feature_importance from the gradientboost? What's the technique? [21:24:05] i only know feature_importance for LogRegression and RandomForrest, maybe you use the fact that its an ensemble? [21:24:12] Oh I'm just catching up with a leap you probably made: yeah it's unfair to think we have "precision" for the singletons, right? [21:25:38] But wouldn't that also mean that we should subtract some binomial thing from e.g. 2-edit sessions, to account for how the baseline is a function of ORES's own precision? [21:26:04] LIME is https://homes.cs.washington.edu/~marcotcr/blog/lime/ [21:26:59] The ores-lime notebook I sent above is a visualization of gradients around a single prediction, created by training a linear model to approximate the real model's gradients. [21:32:53] interesting. [21:33:18] so without the singletons, the characteristic of the curve is a bit different [21:33:20] https://i.imgur.com/p2Mq6mZ.png [21:34:10] so maybe what I should do, is not consider the singletons, and then label/train more non-singletons and then do model selection and then we're ready? [21:38:16] 10Scoring-platform-team (Current): Qualitative Analysis of Session-Edit mismatches. - https://phabricator.wikimedia.org/T208362 (10notconfusing) Done, for results see presentation. https://docs.google.com/presentation/d/1OfZ04LRm0ZrU0q4pZZmZfpHZXeAIewu9mKp_MqNORa4/edit#slide=id.g471478cce6_0_568 Two new featu... [21:42:10] 10Scoring-platform-team (Current): Qualitative Analysis of Session-Edit mismatches. - https://phabricator.wikimedia.org/T208362 (10notconfusing) {F27245336} the data and analysis of gtm [21:44:06] 10Scoring-platform-team (Current): Label 100 more non-singleton sessions and repeat model selection. - https://phabricator.wikimedia.org/T209728 (10notconfusing) [21:49:21] apergos: late Friday question. We're trying to stuff a complete set of all ORES scores into HDFS, and have it kept up-to-date. [21:49:50] it's almost mdnight and 'm watching a movie.... [21:50:00] :) Nothing on fire here [21:50:15] ask and i'll read/comment later [21:50:27] <3 Thanks [21:51:09] What I'm thinking is that per-wiki dump archives of all ORES scores would be valuable to many people. [21:52:03] Seems to fit into the "analytics datasets", how are those produced? [22:32:29] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10awight) [22:32:51] harej: Do we already have dump files for ORES scores? [22:33:13] awight: I don’t think so [22:33:17] Just some one offs [22:33:39] halfak|Mobile: o/ thanks. Yeah it's feeling useful, I'll write a task and justify there. [22:33:40] E.g. the article quality prediction dumps [22:33:52] ^ where's that? [22:34:00] We'll want different dump strategies for different models [22:34:41] https://figshare.com/articles/Monthly_Wikipedia_article_quality_predictions/3859800 [22:34:56] ty [22:36:04] halfak|Mobile: Say more about dump strategies. I was thinking of a path like /wmf/data/ores/score/wiki=enwiki/model=editquality/... [22:37:17] No strong opinion on location. But re. Dumps it would be useful to have every revision for edit quality but we only need a revision per month or so for article quality. [22:37:35] Similarly a dunno if most recent is useful for article quality but not edit quality [22:37:47] *dunno --> dump [22:37:57] halfak|Mobile: nice [22:38:00] Sorry. Phone is hard [22:38:17] thumb is big [22:39:39] Right [22:46:22] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) [22:47:08] Do they actually have an "=" in the path? [22:47:18] Cause that's kind of amazing. [22:53:04] halfak|Mobile: What do you think about including the feature cache in hadoop / dumps? [22:54:05] I'm rereading a letter from fu-berlin and need to clarify with them, but starting to think they might want to capture *all* features for all of wikidata. [23:00:35] +1 if space isn't an issue [23:15:18] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:16:12] That's an unfortunate shorthand. [23:16:31] PROBLEM - puppet on ORES-worker02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:18:47] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:23:40] 10ORES, 10Scoring-platform-team, 10User-Ladsgroup: Rewrite ORES "reference" UI using React - https://phabricator.wikimedia.org/T195274 (10Jdlrobson) Forgot to post this :-): > FYI: I moved the React bindings for jQuery.i18n into it's own library: > https://www.npmjs.com/package/@wikimedia/react.i18n I've b... [23:24:08] 10ORES, 10Scoring-platform-team, 10Analytics: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) [23:24:31] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) [23:25:33] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:27:16] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:27:49] 10ORES, 10Scoring-platform-team, 10Analytics: Include feature values in ORES changeprop stream - https://phabricator.wikimedia.org/T209734 (10awight) [23:28:43] 10ORES, 10Scoring-platform-team, 10Analytics: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) [23:28:46] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10awight) [23:30:52] "accidents" happen. https://farm3.static.flickr.com/2033/2334805994_0bb26e1e0f_b.jpg [23:33:35] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10awight) [23:34:56] 10ORES, 10Scoring-platform-team, 10User-Ladsgroup: Rewrite ORES "reference" UI using React - https://phabricator.wikimedia.org/T195274 (10dbarratt) >>! In T195274#4754880, @Jdlrobson wrote: > I've been keeping jQuery out of the React components library to allow others to use it. > Given all strings can be pa... [23:39:30] 10ORES, 10Scoring-platform-team, 10Analytics: Backfill ORES Hadoop scores with historical data - https://phabricator.wikimedia.org/T209737 (10awight) [23:43:18] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:41] 10ORES, 10Scoring-platform-team, 10Analytics: Backfill ORES Hadoop scores with historical data - https://phabricator.wikimedia.org/T209737 (10awight) [23:44:31] RECOVERY - puppet on ORES-worker02.experimental is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:46:47] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:47:17] 10ORES, 10Scoring-platform-team, 10Analytics: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) @awight FYI that events need to abide to a schema that can be persisted to sql: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Schema_Guidelines and tha... [23:50:55] 10ORES, 10Scoring-platform-team, 10Analytics: Choose HDFS paths and partitioning for ORES scores - https://phabricator.wikimedia.org/T209731 (10Nuria) It is worth looking at already existing event data, if we want to reuse the logic that reads events and persists those to hive partitions cannot be schema dep... [23:53:33] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:57:09] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation: Produce dump files for ORES scores - https://phabricator.wikimedia.org/T209739 (10awight) [23:57:16] RECOVERY - puppet on ORES-worker01.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:58:20] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation: Produce dump files for ORES scores - https://phabricator.wikimedia.org/T209739 (10awight) Pointers from @Nuria: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie https://github.com/wikimedia/analytics-refinery/tree/master/o... [23:58:43] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation: Produce dump files for ORES scores - https://phabricator.wikimedia.org/T209739 (10awight)