[00:00:15] (03CR) 10Catrope: Add special page with model statistics (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [00:01:36] (03CR) 10jerkins-bot: [V: 04-1] Add special page with model statistics [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [00:02:14] (03CR) 10Catrope: Add special page with model statistics (033 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [00:03:09] (03CR) 10jerkins-bot: [V: 04-1] Add special page with model statistics [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [00:11:34] (03PS17) 10Catrope: Add special page with model statistics [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 [00:11:51] (03CR) 10Catrope: Add special page with model statistics (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [01:11:11] traipsing off into the sunset. [01:20:33] oh! i had a question for you [01:20:42] but I was waiting for you to further write your thoughts on the above [01:20:47] but that hasn't happened yet [04:12:53] harej: ah sorry [04:13:02] I shifted my rambling into a local doc [04:13:10] pretty incoherent mess so far. [08:15:11] o/ [08:27:26] (03CR) 10Ladsgroup: [C: 032] "It looks ace. Thank you!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [08:37:07] (03Merged) 10jenkins-bot: Add special page with model statistics [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [08:45:12] (03CR) 10jenkins-bot: Add special page with model statistics [extensions/ORES] - 10https://gerrit.wikimedia.org/r/433956 (owner: 10Catrope) [09:15:58] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Ladsgroup) It's already there: https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists/gl [09:47:59] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10ORES-Support-Checklist, 10Patch-For-Review, 10User-Ladsgroup: Change mentions of wp10 to articlequality in products - https://phabricator.wikimedia.org/T203080 (10Ladsgroup) >>! In T203080#4557074, @Catrope wrote: > Wouldn't articlequality... [09:49:17] afk for lunch [09:49:20] will be back soon [12:02:27] back now [12:48:34] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Ladsgroup) We should note that hive is behind NDA and production access which only most staff and handful... [13:30:39] (03PS1) 10Acamicamacaraca: Add special page alias for Serbian [extensions/ORES] - 10https://gerrit.wikimedia.org/r/458173 [13:33:31] (03PS2) 10Acamicamacaraca: Add special page alias for Serbian [extensions/ORES] - 10https://gerrit.wikimedia.org/r/458173 [13:44:08] https://en.wikipedia.beta.wmflabs.org/wiki/Special:ORESModels [13:44:09] Fancy [13:48:41] (03PS3) 10Acamicamacaraca: Add special page alias for Serbian [extensions/ORES] - 10https://gerrit.wikimedia.org/r/458173 [14:02:42] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Tim_WMDE & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:04:20] o/ [14:04:57] halfak: hey, do you have anything in wikilabels-01? [14:05:27] I'm throwing it out (already migrated traffic to wikilabels-02 for two weeks now) [14:05:46] copied the db and oauth creds [14:06:49] yes. Lots of old datasets I'd like to save. [14:06:53] Let me grab them. [14:07:29] we should have a general purpose storage node [14:07:41] maybe put them into analytics datasets? [14:08:55] Amir1, agreed. They aren't critical, but it's nice to have some records match up to a collection of files. [14:09:24] we already have labels-dumps.wmflabs.org but that's not very old [14:09:43] http://wikilabels-dumps.wmflabs.org [14:10:09] let me see what is the right URL [14:10:43] OK. Data copied. [14:11:06] Amir1, I have nothing left on the machine of value. [14:11:30] cool [14:11:39] just deleted this three year old instance [14:11:48] that was the oldest instance of us [14:12:00] (already renewed most ores nodes as well) [14:12:14] I hope you didn't have any data on ores-lb-02 :D [14:12:15] Oh damn. I should have grabbed my bash history. Oh well. [14:12:23] Nope. I never log into that. [14:12:59] hmm, I put most of my useful commands in https://wikitech.wikimedia.org/wiki/Wikilabels [14:13:16] specially regarding new campaigns [14:21:07] oh this: http://wikilabels-dumps.wmflabs.org/psql_dumps/ [14:22:33] halfak: I was worried that this pool counter thing might affect labs as they share IPs but checked and it's not one IP, I was able to pinpoint ten different labs IP that hit ores.wikimedia.org more than 1K times yesterday [14:22:47] the data is in hadoop, it's great we have the data [14:24:49] and the top one hit us 16K times a day which is not much [14:36:13] Nice! [14:36:46] I'm sure it'll come in handy to use this in hadoop. Can you add some docs on where the data lives and a couple of the queries you have been running? [14:36:47] Amir1, ^ [14:37:12] yeah [14:37:18] something like this: select ip, count(*) as hitcount from wmf.webrequest where uri_host = 'ores.wikimedia.org' and year = 2018 and month = '09' and day = '04' group by ip order by hitcount desc limit 5000; [14:37:32] let me find a good place, where do you think is good? [14:37:52] wikitech:ORES? [14:38:07] Yeah probably wikitech. [14:38:29] Maybe under something like ORES/DOS or maybe ORES/Metrics [14:40:57] aand I made a really quick home page for wikilabels back up node: http://wikilabels-dumps.wmflabs.org/ [14:41:05] we should puppetize it too [14:41:28] oh we have https://wikitech.wikimedia.org/wiki/ORES/Metrics [14:42:17] Nice. [14:48:17] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Implement PoolCounter support in ORES - https://phabricator.wikimedia.org/T201823 (10akosiaris) [14:48:21] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review, 10User-Ladsgroup: Use poolcounter to limit number of connections to ores uwsgi - https://phabricator.wikimedia.org/T160692 (10akosiaris) [14:48:23] 10Scoring-platform-team, 10ORES, 10Operations, 10vm-requests, 10Patch-For-Review: Site: 4 VM request for ORES poolcounter - https://phabricator.wikimedia.org/T203465 (10akosiaris) [14:52:10] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Tim_WMDE & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:02:33] Ugh. CSCW publications have an open-access fee of $1300 [15:02:34] WTF [15:02:43] It used to be $350 [15:02:55] * halfak checks what we have budgeted. [15:03:22] We have $1050 budgeted. [15:06:13] we should have a general purpose storage node << out of curiosity, what is the nature of this data? Are these dump files, or something else? [15:22:23] Looks like Amir1 is going a technical advice IRC thingie. [15:25:24] yup, [15:25:31] I will be on SoS [15:25:46] kk thank you [15:29:51] last chance to give me thing to report to SoS [15:42:37] o/ [15:43:15] harej: To what you were saying yesterday (or Monday?), maybe we're overloading JADE with too many use cases. [15:43:42] Specifically, maybe we can drop support for write-only workflows... [15:44:11] AIUI, we're interested in group judgment rather than just the mean average of individual judgments. [15:44:51] Maybe we wouldn't want to touch the individual stuff unless someone comes through and synthesizes anyway. [15:44:57] If we drop write-only workflows it limits our integration options by a lot. [15:45:02] heck yeah [15:45:04] :-) [15:45:12] Possibly a feature and not a bug. [15:45:31] It even makes WikiLabels integration inappropriate. [15:47:02] I've been poking at the literature on group judgment, and it seems that it has many better properties than taking the mean of individual judgments. [15:47:55] halfak also reminded me of a requirement yesterday, that we don't want to train on e.g. Huggle data, where there would be a tendency towards positive feedback with ORES. [15:48:06] I'm not sure direct wikilabels makes sense under the current formulation of the JADE schema anyway. [15:48:20] Right on. [15:48:55] Origin an the labeler themselves are important data for making use of JADE. [15:49:12] hehe I was just about to say something similar but opposite. [15:49:49] In theory, judgment is judgment. In practice, humans and contexts have weird patterns of bias. [15:50:48] Tracking provenance is obviously a big problem, since it gets so messy > 1 author, so maybe it's best to just look at the group judgment and take the authorship as opaque for most of our analyses. [15:52:45] I'm still just playing with this idea, but looking for an appropriate content schema that gives editors free reign to express lots of different types of group judgment which are still machine-readable. [15:53:11] Dropping write-only workflows seems like a nice way to narrow scope. [15:53:46] awight, I don't think it gets messy. [15:54:05] E.g. give me all judgements that are endorsed by at least one Wikilabeler. [15:55:01] Endorsement seems really weird if it's coming from a write-only client like WikiLabels, though. [15:55:33] Yeah. Fair point. I think write-only clients don't make clear sense in any schema. [15:55:39] The alternative is that we use JADE to identify borderline cases, then feed them into WikiLabels as a separate system. [15:55:51] Except maybe "bucket of judgement" which doesn't really make sense otherwise. [15:56:03] hehe I want that to become part of our lexicon either way. [15:56:28] "you want the apron with that?" [15:56:59] tl;dr, I'm going to try to do away with "bucket of judgment" today. [15:57:37] The fundamental question that should be guiding this is "who are we building this for" [15:57:54] Append-only workflows make sense if you view JADE as a data gathering mechanism for researchers. [15:58:00] Or as a general information sharing tool. [15:58:06] From the 30k-foot-view, it seems fine to not integrate with patrolling workflows until they're able to read existing JADE judgments and support a collaborative pattern. [15:58:10] harej, it doesn't make sense if you expect researchers to actually use that data. [15:58:15] harej, ^ [15:58:43] Append-only interestingly doesn't require wiki pages, also... [15:59:51] but I have no ideas yet for how to capture that data stream into anything different than JADE, and what value it might have. [16:00:16] oh. [16:00:23] You can append to the talk page. [16:01:00] That also comes with an endorsement-style authorship record. [16:01:36] Someone synthesizing content can point to the talk page trail as part of their justification for a judgment, perhaps. [16:01:47] oops, meeting [16:02:23] awight, harej: meeting! [16:02:31] woops. [16:59:51] I'm glad we had this discussion about cutting out append-only use cases, but it actually wasn't what I was talking about when I was concerned about too many use cases [17:00:08] My second proposal is to have the initial deployment only focus on the damaging and goodfaith concepts. [17:33:59] J-Mo: u frozen to us [17:37:16] leila: https://arxiv.org/abs/1702.03859 [17:43:36] * halfak --> lunch [17:51:11] ups, wrong channel [18:32:36] food [18:37:57] back now, was in break after 8 hours [18:38:07] will work for one or two hours [18:39:50] sounds like you should go home after working a full day :p [18:42:33] o/ [18:50:33] awight: yup but I need to fix the mirroring thing and then test the pool conter in beta [19:05:23] awight: halfak https://github.com/wikimedia/ores/pull/261 [19:05:49] Looks like this won't be run in case of precache [19:05:52] Is that intentional? [19:06:05] Don't we *want it* to run when precaching? [19:06:19] I can push it to labs when it gets merged and then check the metrics [19:06:20] Oh! i see that it is intended to measure the log time specifically. [19:06:28] yeah [19:06:31] What about the other lock operations? [19:06:45] it's basically isolate the timing [19:06:54] There are other lock operations though. [19:07:06] Also, seems like having an all-encompasing timing would be very valuable. [19:07:07] yeah, the only one is lock release [19:07:26] do you mean anything beside lock release? [19:07:31] Why not just set start = time.time() right before this *if* and compare against it there. [19:08:26] I would be okay with that but I also want to know about impact of pool counter lock on response time specificity too [19:08:40] *specificly [19:09:00] I just noted that you need to use the MetricsCollector interface too. [19:09:11] Can't assume graphite. [19:09:22] oh, I thought it's there [19:09:36] we should makr them with _send_timing_event [19:09:40] "send_timing_event" should probably be renamed to "_send_timing_event [19:09:40] but anyway [19:09:43] yup [19:15:15] halfak: what is the most important literature about Wikimedia and AI that you can think of? (I'm assuming you have some standard list you reference) [19:16:24] Oh... Hmm. selfishly, I'd like to point you to our ORES paper preprint. [19:16:31] It's not highly cited yet [19:16:37] So it depends what you mean by "important" [19:16:37] halfak: hmm, since there is nothing in the existing methods in the interface, do you think I should add send_timing_event to interface and all of its implementions or make the sending time for poolcounter happen only if metric_collector is statsd (I hate this solution it violates LSP and DIP) [19:17:19] Amir1, no. Add a specific action. Like "lock_acquired" [19:17:37] yeah, that sounds good [19:18:58] halfak: one last question, should we add metric for lock release response time? [19:19:12] there isn't any other lock action as far I can tell [19:19:13] halfak: is it good enough for me to circulate among several wmf product managers? [19:19:31] Amir1, I think so. And wrap the whole thing in a single big timing event. [19:19:38] Yup. [19:19:49] https://commons.wikimedia.org/wiki/File:ORES_-_Facilitating_re-mediation_of_Wikipedia%27s_socio-technical_problems.pdf [19:21:38] wikimedia/ores#988 (timing_poolcounter - 7cfe63e : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/ores/builds/424941611 [19:23:42] halfak: [19:24:15] awight, that one is the pre-FATML cut that I think we should just send directly to a journal. [19:24:25] BTW, the FATML one got desk rejected. [19:24:43] I wasn't even aware u had submitted a thing [19:24:47] We had a latex artifact that made the paper spill over the 10th page. They wouldn't consider it because it was over-size :( [19:24:52] NO [19:25:00] that's despicable [19:25:19] Yeah. :( I tried to reason with them but they wouldn't respond. [19:25:27] Oh well. I guess we'll send it somewhere else. [19:25:54] What is this, a reality elimination lynching... [19:27:45] J-Mo: This is the paper I quoted from the abstract of, https://www.sciencedirect.com/science/article/pii/0749597889900551 [19:30:10] Hey folks! Fun story. [19:30:20] I think I just got us some free fitness for all of our damage detection models! [19:30:34] The bad side of this is that we had a weird bug in our diff algorithm! [19:30:43] The good side is, fixing it should help a lot! [19:31:09] holy donut holes! Do share [19:31:10] I've just uploaded deltas 0.4.7 to pypi and I'm going to experiment with rebuilding some damage detection models with it. [19:31:18] https://github.com/halfak/deltas/commit/aa314c89b1034263cdd010b976d9d9bce1baff35 [19:31:38] Basically, our sentence and paragraph segmenter did a great job of ignoring tables. [19:31:54] And that causes a weird offset in how some added-content diff operations are flagged. [19:32:15] Essentially, some vandalism, if it occurs near a table, will not get caught in the diff. [19:32:25] Oh wait. We use term frequency measures and not diff. [19:32:26] I think. [19:32:29] * halfak digs. [19:32:52] So this test currently fails? [19:33:04] Well the test failed, but the fix makes it pass. [19:33:16] halfak: the new patch is up :D [19:33:46] halfak: awight btw this will go live next week https://en.wikipedia.beta.wmflabs.org/wiki/Special:ORESModels [19:33:58] Roan did all the work I just reviewed it [19:34:04] halfak: don't mind me, I hadn't scrolled past the fold to see the fix [19:34:16] Amir1: Yesss [19:35:20] \o/ [19:35:25] Amir1, one note. Found a typo [19:35:29] Otherwise looks good. [19:37:25] OK. Looks like we don't use the diff algorithm after all [19:37:27] ha [19:37:38] So this fix will help other stuff but not revscoring/editquality [19:37:48] Term frequency is more robust anyway :) [19:40:40] halfak: thanks, fixed now [19:41:35] https://github.com/wikimedia/ores/blob/d8ccba5512ed8db57cd992c37f55b327eb5762a9/ores/metrics_collectors/tests/test_statsd.py [19:41:45] We should probably "test" it. [19:41:52] Sorry I didn't notice that earlier. [19:42:03] See also: https://github.com/wikimedia/ores/blob/d8ccba5512ed8db57cd992c37f55b327eb5762a9/ores/metrics_collectors/tests/test_logger.py [19:46:34] yeah [19:46:37] let me do it [19:46:39] btw. https://wikitech.wikimedia.org/wiki/ORES/Metrics [19:46:44] (Added views) [19:57:43] The test for statsd is a "seam". This is first time I've seen a seam. I love it [20:02:26] halfak: tests added [20:04:38] Looks good. [20:04:50] :D [20:04:58] * halfak is not familiar with the term "seam" [20:05:46] {{merged}} [20:05:47] 10[2] 04https://meta.wikimedia.org/wiki/Template:merged [20:06:06] that's one of inventions of Martin Fowler :D [20:06:23] Thank you! [20:13:27] J-Mo: This one is fun, https://www.researchgate.net/publication/6653857/download [20:14:04] ~ Having some dissent is important to decision-making, mostly because it spurs more conversation. [20:14:04] thanks for both of the papers, awight [20:14:06] I have a late meeting today, so I'm going to go AFK for a bit to do some chores. I'll be back in about an hour. [20:14:44] halAFK: good luck! [20:18:01] wikimedia/ores#994 (timing_poolcounter - cfc6121 : Amir Sarabadani): The build was broken. https://travis-ci.org/wikimedia/ores/builds/424964905 [20:18:57] random failure ^ [20:18:59] restarting [20:20:42] I need to leave [20:20:48] see you tomorrow [20:21:30] wikimedia/ores#994 (timing_poolcounter - cfc6121 : Amir Sarabadani): The build was broken. https://travis-ci.org/wikimedia/ores/builds/424964905 [20:23:09] Amir1: o/! [21:36:41] * halfak fought with his bike wheels for far too long [21:36:49] I'm going to have sore thumbs for a week [21:37:06] Continental tires, eh? [21:37:50] I can't stand that. Loose-fit tires ftw [21:38:27] 10Scoring-platform-team (Current): Fix diffing issue (mistaken added text) - https://phabricator.wikimedia.org/T203610 (10Halfak) [21:38:37] 10Scoring-platform-team (Current): Fix diffing issue (mistaken added text) - https://phabricator.wikimedia.org/T203610 (10Halfak) a:03Halfak [21:38:59] awight, was setting up my tubeless road tires (schwalbe almotion) [21:39:18] Regretfully, my offroad tan-walls were affixed to the rims with a thin layer of sealant. [21:39:38] I had to push on the tire with all my might to get it to un-seat. [21:40:01] Usually, it's getting the tire over the rim that is the problem and the tire will unseat itself when you deflate it. [21:41:33] Hardcore. I've only heard of tubular, not tubeless [21:42:35] Tubelss is a big thing for offroad. You seal your rim, and then put a tire on it without a tube and a little sealant floating around. [21:42:45] It makes it so that minor punctures get sealed with sealant. [21:43:09] That's really critical for me since I go over some rough terrain really fast. [21:45:22] What's funny is that I've had several orders of magnitude more flats in the city than offroad [21:45:27] Even riding road bikes off-road [21:45:38] cursed glass! [21:46:14] & staples, potholes, underinflated tires, lack of rim tape... [21:46:18] PTSD cenral [21:46:18] Yeah. I hear that! So I run tubeless everywhere :D [21:46:21] *central [21:46:28] lack of rim tape :P [21:46:32] /o\ [21:46:47] A lot of bike problems come down to "lack of $10". [21:48:44] 10Scoring-platform-team (Current): Fix diffing issue (mistaken added text) - https://phabricator.wikimedia.org/T203610 (10Halfak) https://github.com/halfak/deltas/commit/aa314c89b1034263cdd010b976d9d9bce1baff35 Got it. I released deltas 0.4.7. It turns out we don't use this diff code for edit quality stuff... [21:48:53] 10Scoring-platform-team (Current): Fix diffing issue (mistaken added text) - https://phabricator.wikimedia.org/T203610 (10Halfak) 05Open>03Resolved [21:56:00] I hear that. [21:56:18] I spend an absurd amount of money on my race bike just to make sure that I don't have issues while racing. [21:56:28] E.g. I replace tires when they are about half-worn. [21:56:43] The old tires stay in my garage waiting to be mounted in the offseason. [21:56:50] So I guess it isn't super wasteful. [21:57:05] Anyone want some half-used semi-offroad 700x42mm tires? [21:57:09] Tubeless read ;) [21:57:12] *ready [22:01:53] I'm having an IRC issue. [22:02:04] I'm trying to send newbies to this channel, but they can't join. [22:02:20] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @RoanKattouw & @Niharika - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [22:02:22] It might be because we don't allow people who have not registered an account. [22:05:55] halfak: I recently got a car from a friend, and he had put second-hand racing tires on all wheels---of four different sizes [22:06:05] halfak: this channel imports all bans from #wikimedia-bans [22:06:19] halfak: +1 yeah Freenode is on perpetual lockdown now [22:06:23] and #wikimedia-bans has a ban on all non-registered accounts [22:07:09] so, in order for newbies to join here, you need to break that inheritance [22:07:33] Gotcha, OK [22:08:08] I can tell BanBot to join here [22:08:23] it's been being quite efficient with the last spams [22:52:15] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @RoanKattouw & @Niharika - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [23:15:06] Meetings done! [23:15:09] * halfak runs away [23:15:10] o/ [23:22:43] oh yeah, that's why I haven't read it yet. No access. [23:23:18] halAFK: Can borrow "Accuracy and confidence in group judgment" Sniezek et al? https://www.sciencedirect.com/science/article/pii/0749597889900551