[10:14:56] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718772 (10ema) >>! In T179156#3717895, @BBlack wrote: > My best hypothesis for the "unreasonable" behavior that would break un... [11:29:03] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718957 (10Lucas_Werkmeister_WMDE) > The only live polling feature I can think of that was recently introduced is for the live... [11:48:47] halfak: for when you're online: https://github.com/wiki-ai/wikilabels/pull/208 [11:48:49] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719057 (10BBlack) >>! In T179156#3718772, @ema wrote: > There's a timeout limiting the total amount of time varnish is allowed... [11:51:38] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719064 (10hoo) [11:56:35] halfak: another one: https://github.com/wiki-ai/ores/pull/227 [12:19:58] and https://github.com/wiki-ai/wikiclass/pull/52 [12:23:47] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719145 (10Lucas_Werkmeister_WMDE) >>! In T179156#3719057, @BBlack wrote: >could other services on text-lb be making these kind... [13:17:59] 10Scoring-platform-team (Current), 10Beta-Cluster-Infrastructure, 10ORES, 10User-Ladsgroup: ORESFetchScoreJob: RuntimeException No model available for [goodfaith] - https://phabricator.wikimedia.org/T178792#3719327 (10Ladsgroup) a:03Ladsgroup [14:09:51] o/ [14:09:55] 10Scoring-platform-team (Current), 10Beta-Cluster-Infrastructure, 10ORES, 10Patch-For-Review, 10User-Ladsgroup: ORESFetchScoreJob: RuntimeException No model available for [goodfaith] - https://phabricator.wikimedia.org/T178792#3719557 (10hashar) 05Open>03Resolved [14:23:14] o/ [14:23:20] * halfak gets to his actual desk [14:23:43] I have a buddy over today. He just got back from a big bike tour. Surprise to me that he showed up late last night. [14:27:20] Halfak i plan on mentoring some scoring platform documentation tasks for GCI got any tasks you want done for it? [14:30:14] o/ Zppix [14:30:25] Hmm.. Let me think. [14:30:46] What sort of documentation do you have in mind? [14:31:44] Any thing that needs to be done, and would help aid future developers in developing scoring platform projects [14:32:32] Sure! I'm just wondering you have something specific you'd like to direct work on. E.g. we have [[:mw:ORES/FAQ]] [14:32:32] 10[2] 04https://meta.wikimedia.org/wiki/:mw:ORES/FAQ [14:32:51] Or we have things like https://ores.wikimedia.org/v3/ [14:33:19] Do we have open tasks for these? [14:34:16] btw we should have ores interwiki map so [[ores:v3]] would work [14:34:17] 10[3] 04https://meta.wikimedia.org/wiki/ores:v3 [14:34:21] that would be fun [14:34:42] ^ im mentoring mostly doc tasks but agreed [14:35:08] Amir1, interesting concept. We have that for phab :) [14:35:30] yeah [14:35:34] Zppix, no open tasks, but there's work to do. I think mentoring would involved spec'ing out the tasks themselves. [14:35:56] I can help if needed [14:36:25] halfak: I have some other PRs waiting for you :D [14:36:25] Ok [14:37:31] btw. I enabled ORES in wikidata, fixed beta cluster (ores extension there), my next plan is to get wikilabels deployed [14:37:34] Amir1, gimme links and I'll rapid fire them. [14:38:04] Nice. I'm hoping to get things cleaned up on the workboard and do reviewing today. Today is just a big long string of meetings. [14:38:20] Check out my calendar if you are curious. I blocked off Lunch because it was the *only* open time I had! [14:38:24] halfak: https://github.com/wiki-ai/ores/pull/227 [14:38:55] halfak: Travis would fail for this because we didn't have it running :/ https://github.com/wiki-ai/wikiclass/pull/52 [14:39:04] I will fix that in other PRs [14:40:35] I'm going through this and cleaning up stuff: https://github.com/search?q=org%3Awiki-ai+nose&type=Code [14:40:44] Where would you like the documentation to be halfak for restapi for ores? [14:41:31] Zppix, https://ores.wikimedia.org/v3/ & https://github.com/wiki-ai/ores/blob/master/ores/wsgi/templates/v3_swagger.json [14:41:45] Zppix, see also https://swagger.io/ [14:42:10] One thing that would be very cool is https://phabricator.wikimedia.org/T140364 [14:42:39] Essentially, I want to be able to render https://www.mediawiki.org/wiki/ORES/Support_table on ORES [14:48:08] halfak: https://github.com/wiki-ai/flask-swaggerui/pull/1 [14:48:34] Amir1, keep track of these on the workboard somehow OK? [14:48:39] Like maybe one big task. [14:48:54] hmm okay [14:53:21] 10Scoring-platform-team (Current), 10ORES, 10Wikilabels, 10revscoring, and 2 others: Remove usages of nosetest and replace it with pytest - https://phabricator.wikimedia.org/T179296#3719709 (10Ladsgroup) [14:55:04] https://phabricator.wikimedia.org/T179296 [14:55:06] 10Scoring-platform-team (Current), 10ORES, 10Wikilabels, 10revscoring, and 2 others: Remove usages of nosetest and replace it with pytest - https://phabricator.wikimedia.org/T179296#3719709 (10Ladsgroup) - https://github.com/wiki-ai/flask-swaggerui/pull/1 - https://github.com/wiki-ai/wikiclass/pull/52 -... [14:55:48] 10Scoring-platform-team (Current), 10JADE, 10Documentation: Clean up JADE homepage - https://phabricator.wikimedia.org/T179297#3719729 (10Halfak) [14:56:40] 10Scoring-platform-team (Current), 10JADE, 10Documentation: Clean up JADE homepage - https://phabricator.wikimedia.org/T179297#3719745 (10Halfak) 05Open>03Resolved a:03Halfak https://www.mediawiki.org/w/index.php?title=JADE&type=revision&diff=2598998&oldid=2586036 [14:57:55] 10Scoring-platform-team (Current): Post about splitting "revision" and "edit" conceptually - https://phabricator.wikimedia.org/T179298#3719751 (10Halfak) [14:58:12] 10Scoring-platform-team (Current), 10JADE, 10Documentation: Post about splitting "revision" and "edit" conceptually - https://phabricator.wikimedia.org/T179298#3719765 (10Halfak) [14:58:18] 10Scoring-platform-team (Current), 10JADE, 10Documentation: Post about splitting "revision" and "edit" conceptually - https://phabricator.wikimedia.org/T179298#3719751 (10Halfak) https://www.mediawiki.org/wiki/Topic:Tzygtjcyrclilp8u [15:00:31] https://github.com/wiki-ai/flask-wikimediaui/pull/1 [15:07:13] 10Scoring-platform-team, 10Documentation: Mock JADE discussion page - https://phabricator.wikimedia.org/T179301#3719822 (10Halfak) [15:15:55] 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3719840 (10Marostegui) [15:20:32] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715032 (10daniel) @BBlack wrote: > something that's doing a legitimate request->response cycle, but trickling out the bytes o... [15:21:55] https://commons.wikimedia.org/wiki/File:JADE_state_validator.svg [15:24:38] sync up is today or wednesday? [15:25:44] codezee, today :) [15:25:52] 10Scoring-platform-team (Current), 10ORES, 10Wikilabels, 10revscoring, and 2 others: Remove usages of nosetest and replace it with pytest - https://phabricator.wikimedia.org/T179296#3719886 (10Ladsgroup) - https://github.com/wiki-ai/flask-wikimediaui/pull/1 - https://github.com/wiki-ai/editquality/pull/104 [15:26:01] BS time in 5 mins if you want to join and talk about non-work stuff. [15:26:50] gotta catch dinner :P [15:32:07] Amir1: halfak: social time? [15:32:13] Woops. Yes! [15:34:47] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719914 (10BBlack) Trickled-in POST on the client side would be something else. Varnish's timeout_idle, which is set to 5s on... [15:39:15] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719928 (10daniel) > In any case, this would consume front-edge client connections, but wouldn't trigger anything deeper into... [15:54:37] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3719995 (10BBlack) >>! In T179156#3719928, @daniel wrote: >> In any case, this would consume front-edge client connections, bu... [16:27:52] 10Scoring-platform-team (Current), 10ORES, 10Wikilabels, 10revscoring, and 2 others: Remove usages of nosetest and replace it with pytest - https://phabricator.wikimedia.org/T179296#3720103 (10Ladsgroup) - https://github.com/wiki-ai/flask-oojsui/pull/2 The repos left: - drafttopic - wb-vandalism (needed... [16:28:33] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3720106 (10BBlack) p:05Unbreak!>03High Reducing this from UBN->High, because current best-working-theory is this problem is... [16:48:29] 10Scoring-platform-team: Build mid-level WikiProject category training set - https://phabricator.wikimedia.org/T172321#3720194 (10Sumit) [16:48:31] 10Scoring-platform-team: Efficient method for mapping a WikiProject template to the WikiProject Directory - https://phabricator.wikimedia.org/T172325#3720192 (10Sumit) 05Open>03Resolved a:03Sumit [16:49:08] 10Scoring-platform-team: Build mid-level WikiProject category training set - https://phabricator.wikimedia.org/T172321#3494777 (10Sumit) https://github.com/wiki-ai/drafttopic/pull/5 [16:54:21] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3720214 (10Halfak) a:03Halfak [16:55:27] 10Scoring-platform-team: Generate mid-level WikiProject categories - https://phabricator.wikimedia.org/T179311#3720220 (10Sumit) [16:55:34] 10Scoring-platform-team: Generate mid-level WikiProject categories - https://phabricator.wikimedia.org/T179311#3720220 (10Sumit) https://github.com/wiki-ai/drafttopic/pull/5 [16:55:47] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3720235 (10Halfak) a:05Halfak>03awight [16:56:40] 10Scoring-platform-team (Current): Generate mid-level WikiProject categories - https://phabricator.wikimedia.org/T179311#3720236 (10Sumit) [17:04:32] halfak: Amir1: I just realized, it probably makes more sense to deploy revscoring 2 to current production before continuing with new-cluster stress tests. [17:05:43] awight, +1 to that [17:05:54] * halfak works on codezees PR [17:06:01] Darn. I can cause timeouts with https://ores-beta.wmflabs.org/v3/scores/enwiki/123456?features= [17:06:06] checking whether that’s a regression... [17:07:24] It’s a regression. [17:07:44] weird that we get features extracted but not a prediction [17:07:47] Any chance it’s something that will only happen on labs? [17:08:00] https://ores-beta.wmflabs.org/v3/scores/enwiki/12345?features= works [17:08:27] https://ores-beta.wmflabs.org/v3/scores/enwiki/1237?features= works. [17:08:32] Looks like it's just that one revision [17:08:53] No, it’s not timing out 100% of the time. I can get 12345 to time out also. [17:10:05] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3720290 (10daniel) > Because they're POST they'd be handled as an immediate pass through the varnish layers, so I don't think t... [17:11:16] awight, weird. not sure what to think about that. [17:11:30] Gotta go do lunch ASAP or I'll never be able to eat [17:11:34] please do. [17:12:11] I’ll look at which MW APIs are being hit and see if they’re actually timing out, then will look at the Celery subtask thing and whether it might be timing out there. [17:14:47] 10Scoring-platform-team, 10ORES, 10Google-Code-in-2017, 10User-Zppix: Document ORES's restAPI functions - https://phabricator.wikimedia.org/T179314#3720316 (10Zppix) [17:15:14] 10Scoring-platform-team, 10ORES, 10Google-Code-in-2017, 10User-Zppix: Document ORES's restAPI functions - https://phabricator.wikimedia.org/T179314#3720330 (10Zppix) Ill mentor this [17:15:34] ^ halfak could update this task with more info please? [17:18:38] halfak: Well that’s interesting. I can’t reproduce the timeout from my virtualbox, in the wilderness. I’m scapegoating labs for now. [17:19:14] I’ll go back to my plan to (ab)use ores* as a mega-beta, to smoke test this bug some more. [17:24:22] 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3720347 (10madhuvishy) The 1001 reboot is all done. Notes from my planning etherpad: labsdb1001 (Planned for Oct 30 2017 14:3... [17:36:41] halAFK: https://github.com/wiki-ai/flask-oojsui/pull/2 and https://github.com/wiki-ai/editquality/pull/104 [17:36:48] I'm leaving for today [17:36:51] o/ [17:39:59] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3720392 (10BBlack) >>! In T179156#3719995, @BBlack wrote: > We have an obvious case of normal slow chunked uploads of large fil... [17:41:03] Amir1: \o! [17:47:55] o/ Do any of you know of research looking at anomalies in Wikipedia vandalism? Most of the research on WP vandalism look at the feasibility of detecting it, not particularly at anomalies in it (or as a result of the detection). [18:05:56] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3720468 (10Halfak) [18:05:58] 10Scoring-platform-team, 10ORES, 10Operations: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3720469 (10Halfak) [18:16:55] Nettrom, nothing I know of. [18:17:01] * halfak in meeting [18:43:32] There are several methods that can be used for anomaly detection, but especially comparing the possible vandalism against similar texts. If the possible vandalism is too unlikely, then it is an anomaly in this context, and can be rejected. [18:44:01] o/ Nettrom ^ [18:44:35] Also, I'm not sure I'm aware of any relevant work. I'm trying to think of any studies of the patterns and dynamics of vandalism as a phenomena. [18:44:38] halfak: thanks, I didn’t know of any either, and jeblad was curious, great to see he stopped by here as well :) [18:44:54] o/ jeblad :) [18:45:30] I know that some of A.G. West's work looked at the temporal patterns in vandalism, but I think it was all limited to detection. [18:45:36] Not exploring the actual patterns. [18:46:44] awight, I'm really confused about the timeout behavior in ores-beta [18:46:53] The thing is, there usually are no pattern and that is the clue to detect them. ;) [18:47:01] awight, Feature extraction is what takes the longest in generating a score [18:47:51] jeblad, right. I agree. Not all anomalies would be damage, but all damage is an anomaly of some sort [18:49:06] halfak: It’s concerning. If the features are present, then we succeeded in hitting the MW API, right? So would the timeout be in communication between the celery master and worker? [18:49:34] awight, nope. The features get computed in the worker too. [18:49:37] * halfak looks through code [18:49:53] ack. I wish I could reproduce it locally, to instrument better. [19:10:40] awight: is the data collection for the draft topic repeatable? [19:11:05] like, a Quarry job or something similar? [19:11:13] sorry, meant to ask halfak [19:11:31] codezee, ^ [19:11:31] it would be super useful for the wikicredit outreachy project [19:20:15] If anyone is interested, mail me. I've been doing neural nets since early -90. Mostly signal processing in analog neural nets. [19:20:24] tgr: Hi, I was away, all the progress for drafttopic is here - https://github.com/wiki-ai/drafttopic currently it can generate WikiProjects hierarchy in json [19:20:51] cherio, I'm out, must read some stuff [19:20:57] tgr: Mid-level category to WikiProjects mapping soon to come in - https://github.com/wiki-ai/drafttopic/pulls [19:21:32] tgr: and I'm working on getting random page instances tagged with all WikiProjects which is currently private [19:22:15] tgr: but yes to answer your question once I'm done with it, it'll all be reproducible like other wiki-ai stuff as wikiclass or editquality through just one line commands in Makefile [19:22:52] tgr: and I'm Sumit, with this nick here... in case of any confusion :P [19:24:11] tgr: whats the wikicredit stuff btw? also you can post that info here - https://phabricator.wikimedia.org/T123327 if you'd like [19:24:15] so that it remains on the radar [19:30:15] halfak: ^ [19:33:17] codezee: https://phabricator.wikimedia.org/T174528 / https://meta.wikimedia.org/wiki/Research:WikiCredit [19:34:06] basically, an editcounter-ish tool that focuses on displaying a user's contributions in a form that would fit into a CV or a report to their boss [19:34:34] and figuring out which contributions are relevant to their area of work is a big part of that [19:40:39] tgr: it seems interesting, do you have any plans on "figuring out which contributions are relevant to their area of work"? [19:41:52] maybe a similarity score of the text they added with the representative text of the topic to which they added their content? [19:42:24] nothing specific, categories, wikiprojects and wikidata properties are tha raw data that seems useful [19:43:07] say i edited a stub biography article and my edit highly correlates with the signal in biography related topics, then i'm likely adding value to that article [19:44:26] due to time and skill contraints the Outreachy project will probably be limited to using existing APIs and whatever fits into a few SQL / SPARQL queries [19:44:58] yes, i understand that, this is a very open ended research question, and can't be specifically tackled within outreachy timeframe [19:45:26] tgr: but i'd be happy to help out with any stuff related to data gathering or similar should you need it, feel free to reach out :) [19:45:57] but I'm certainly hoping but it can be followed up in some way with more focus on how to categorize contributions and assign value to them [19:46:09] thanks! will do [19:49:38] yes, i saw that the task was more about getting the right tools in place to help that research, which should be beneficial later [19:52:20] Halfak sorry i had to go afk did you see my last ping? [19:53:01] Zppix, just saw it in the scrollback. [19:53:29] What sort of extensions to the current docs did you have in mind? [19:54:05] Any i just need you to put more info to as what you would like to see [20:01:37] Zppix, hmm... Maybe a formal review of the API docs that could lead to some recommendations [20:46:25] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3721074 (10Zppix) [20:46:38] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3721075 (10thcipriani) A workaround over the short-term may be to use `git_upstream_submodules: True` in the `scap.cfg` file. This would cause a fetch of the s... [20:54:21] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3721109 (10thcipriani) hrm. I was able to clone this locally on tin FWIW: ``` [thcipriani@tin ~]$ git clone http://tin.eqiad.wmnet/ores/deploy/.git/modules/su... [21:19:53] https://commons.wikimedia.org/wiki/File:ORES_architecture.svg [21:20:37] :) nice diagram halfak [21:23:07] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3721216 (10awight) Just a point of information, we have three large repos, which add up to c. 3GB and will only grow. Our deployment cluster has 9 machines, s... [21:23:28] halfak: do you have a min to review my two PRs? [21:23:32] https://github.com/wiki-ai/editquality/pull/104 [21:23:37] https://github.com/wiki-ai/flask-oojsui/pull/2 [21:23:38] clicking [21:23:46] then I'll go to drafttopic [21:24:12] Can you enable travis for editquality? [21:24:24] Or maybe just confirm to me that the tests still pass. :) [21:24:26] For now [21:24:30] halfak: i can :) [21:29:09] err it wont let me for some reasom [21:29:57] halfak: I will after that [21:30:10] kk [21:30:14] Amir1, confirmed the tests run? [21:30:20] yeah [21:33:14] Thanks [21:36:34] wiki-ai/wb-vandalism#169 (master - d940cea : halfak): The build has errored. https://travis-ci.org/wiki-ai/wb-vandalism/builds/295070054 [21:36:47] Woo [21:36:51] We should kill that repo [21:36:55] or hide it or something [21:37:34] I can kill it [21:37:39] On travis [21:38:17] Its killed on travis [21:39:01] Halfak do we need this repo or could it just be deleted? [21:39:03] cool [21:39:08] let's keep it for npow [21:39:19] Ill find a way to hide it [21:40:31] I cannot hide it unless github org is upgraded [21:42:56] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3721358 (10thcipriani) In some fiddling I realized this error message is coming from phab and not tin. Found via `GIT_TRACE=1` directly on the ores1002 server... [21:46:18] That repo is now never going to alert us here again halfak [22:08:43] \o/