[05:46:07] 10Scoring-platform-team (Current), 10Wikilabels, 10Easy, 10User-Ladsgroup: Allow Wiki Labels API to list inactive campaigns - https://phabricator.wikimedia.org/T171768#3807822 (10Ladsgroup) https://github.com/wiki-ai/wikilabels/pull/215 [05:47:38] wiki-ai/wikilabels#238 (inactive_campaigns - 7cc6254 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/wikilabels/builds/311143057 [05:49:09] Hey Hey, [09:51:50] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Spanish-Sites, and 2 others: Edit quality campaign for es.wikiquote - https://phabricator.wikimedia.org/T177762#3808065 (10Ladsgroup) While I'm here I made reverted model as well: ``` Model Information: - type: GradientBoosting... [10:01:45] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Spanish-Sites, and 2 others: Edit quality campaign for es.wikiquote - https://phabricator.wikimedia.org/T177762#3808107 (10Ladsgroup) https://github.com/wiki-ai/editquality/pull/108 [10:05:43] wiki-ai/editquality#18 (eswikiquote - 399cedf : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/311222837 [13:05:51] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Release-Engineering-Team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3808791 (10akosiaris) >>! In T181661#3804344, @awight wrote: > I just ran scap with `-l "ores1001.*" and deployment went smoothly.... [14:50:27] halfak: yt [14:50:39] in meeting. [14:50:53] halfak: let me know when your out of it please [15:32:48] Halfak: please contact me when you can [15:44:06] awight: check T181894 [15:44:06] T181894: Github irc notifications are not working - https://phabricator.wikimedia.org/T181894 [15:47:25] Zppix: o/ thanks for the note! [16:03:28] o/ Adotchar [16:03:33] o/ Zppix [16:03:39] Just got out of the meeting [16:04:52] Halfakkkk [16:07:41] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Complete fawiki (v2) edit quality campaign - https://phabricator.wikimedia.org/T129703#3809422 (10Halfak) [16:14:40] halfak: I have lots of things to review :D [16:16:04] Amir1, Cool. I'll focus on that before lunch [16:16:12] send me the links :D [16:16:16] halfak btw, we need to do https://phabricator.wikimedia.org/T181795 as well [16:16:31] I don't have the full picture and timelines I guess we need adam as well [16:17:05] Thanks for the ping akosiaris. I think awight will be best suited too. I'll make some time to work on it with him. [16:17:16] I'll add what I have. [16:17:24] ok, thanks! [16:17:37] I 'll include what I have as well [16:18:47] Thanks akosiaris :) [16:22:26] brb [16:27:29] I'm calling it a day, see you around soon [16:27:30] o/ [16:29:34] 10Scoring-platform-team, 10ChangeProp, 10ORES, 10Services (doing): Change ORES rules to send all events to new "/precache" endpoint - https://phabricator.wikimedia.org/T158437#3809532 (10Pchelolo) @Ladsgroup Oh indeed we've dropped the ball on this. I'll prepare a PR over the next couple of days [16:44:35] Back [16:44:39] Sorry forgot to say [16:44:57] * halfak does emails. [16:45:31] I’m gonna deploy the more verbose celery logging, it looks good on Beta. [16:45:37] +1 [16:49:01] 10Scoring-platform-team, 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10monitoring: Make an ORES service log dashboard for logstash-beta - https://phabricator.wikimedia.org/T182005#3809625 (10awight) [16:54:53] 10Scoring-platform-team, 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10monitoring: Make an ORES service log dashboard for logstash-beta - https://phabricator.wikimedia.org/T182005#3809669 (10awight) Currently, apache loglines are tagged with type "ores", e.g. > [pid: 10481] 10.68.21.68 (-) {34 var... [16:55:52] Ah. I should wait for our deployment window to do this. [16:56:21] :) [16:56:45] halfak: Just a thought: want to switch to gitlab repos before LFS'ing? [16:56:51] Or is it already the future over there? [16:57:34] awight, not sure what you mean [16:57:49] That we should try to split up the migration if possible [16:57:57] and it makes most sense to split it so that LFS comes last [16:58:57] wiki-ai/editquality#20 (simplewiki - 85d6176 : halfak): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/311409553 [16:59:01] 1) We switch to developing against gitlab, 2) WMF repos are switched to track those rather than GitHub, 3) Migrate repos to LFS [16:59:11] awight, hmm... I'm not sure this is something we need to be so cautious about. We'll waste a lit of time and bandwidth [16:59:34] We'll essentially need to fully upload the repo history to gitlab twice rather than once. [17:00:06] But that's really the only drawback I see. [17:00:24] The ancestry won’t change, so observers should in theory not pull any extra data. [17:00:35] You’re just talking about the step to push repos upstream? [17:01:00] If you’re already LFS on gitlab, maybe you could rename the repos, temporarily. [17:01:36] The thing I’m anticipating is that there are a set of problems stemming from just the gitlab migration, so we could work those out separately in this scenario. [17:02:19] awight, there's no good theoretical reason for gitlab issues since it's just a git pull that is happening. [17:02:34] eh firewall, “other" machinery [17:02:40] no_justification made this argument to me. [17:02:43] + surprises [17:02:46] and paladox [17:03:04] lol well I’ll leave it alone, it’s minor [17:03:09] Another hole in the firewall was made for git.legoktm.org [17:03:11] ok [17:03:17] o/ paladox [17:03:23] heh :) [17:05:23] awight, I agree that we might find problems though. Hmm... We could do it. [17:05:30] Right now, renaming repos doesn't make sens.e [17:05:39] But creating separate repos with a different name does. [17:06:13] 13.7% of edits are reverted in simplewiki [17:06:14] wow [17:06:16] that's a lot [17:06:17] lol [17:06:51] When you rename a repo, it puts a redirect in its place with no option [17:07:39] yuck [17:07:59] halfak: nvm, we can deal with the +10% complexity [17:08:13] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Release-Engineering-Team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3809754 (10mmodell) >>! In T181661#3808791, @akosiaris wrote: > > So, scap closed the connection, 1m, 6s after the login. scap c... [17:08:34] halfak: So when you made gitlab#wiki-ai, did you add me as a github user, or shall I go ahead and make an unassociated account? [17:09:07] awight, go ahead and make an account. :) [17:09:10] I did that [17:09:13] I don't want to overlap [17:10:06] +1 good [17:10:28] hmm gitlab not getting a passing grade yet: > Minimum length is characters. [17:11:33] ? [17:11:46] My password was >> 8 characters, but it let me continue before getting the ReCaptha response [17:11:56] once that happened, the form was borked in exciting ways. [17:12:00] Bloody edge [17:12:13] Hmm. Yeah. That's not great [17:14:20] oh dear. I registered, then signed in. Landed in a 404 [17:14:32] hahaha https://gitlab.com/users is 404 if you’re already logged in [17:14:34] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, 10artificial-intelligence: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3809820 (10Halfak) An important distinction. ORES does not "mark something as V... [17:15:14] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, 10artificial-intelligence: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3809822 (10Halfak) @awight, could you look at https://gerrit.wikimedia.org/r/394... [17:15:18] LOL but the “sign in” page is still available, and doesn’t indicate that you’re already logged in. This is disturbing [17:15:51] anyway, at least it’s git so we’re not vendor-locked-in [17:16:42] Right [17:16:46] And it's open source [17:16:51] So further non-vendor-locking [17:17:01] gitlab's software that is. [17:17:08] :) [17:17:09] It's sort of like what github ought to have been. [17:17:20] Well I’m excited to file my bug reports :p [17:17:35] k, I have an account “adamwight” feel free to add me when you find the time. [17:18:32] 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3809870 (10Halfak) [17:20:05] I don’t understand gitlab.com vs gitlab.org yet. [17:20:24] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, 10artificial-intelligence: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3804284 (10Halfak) [17:20:26] 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3809895 (10Halfak) [17:20:33] gitlab.org is about the OS software [17:20:48] gitlab.com is where they sell server time/bandwidth on their local install of gitlab. [17:20:59] Open source projects get all the perks. [17:21:06] Private/controlled projects pay money. [17:22:26] When I login at gitlab.org, I’m sent to gitlab.com. I guess that makes sense? [17:22:55] right [17:23:12] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3809916 (10akosiaris) [17:23:14] 10Scoring-platform-team, 10ORES, 10Operations: Problem with Redis server configuration on new ORES cluster - https://phabricator.wikimedia.org/T181806#3809914 (10akosiaris) 05Open>03Resolved So the boxes were rebooted some 18 days ago, and given the stresstest was supposed to be not lasting that long, th... [17:23:37] Anyway, I think I see what you’re saying. gitlab.org is a site for the software used to run gitlab.com [17:24:33] 10Scoring-platform-team (Current), 10editquality-modeling, 10Spanish-Sites, 10User-Ladsgroup, 10artificial-intelligence: Complete Spanish Wikibooks edit quality campaign - https://phabricator.wikimedia.org/T145408#3809920 (10Halfak) [17:31:24] 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES, 10User-notice-collaboration: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3809937 (10Trizek-WMF) [17:34:13] Duly filed bugs against https://gitlab.com/gitlab-org/gitlab-ce/ :) [17:40:58] (03CR) 10Awight: [V: 032 C: 032] "I guess there are no test statistics yet because we have no labeled data? This promised to be interesting!" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/394759 (https://phabricator.wikimedia.org/T181848) (owner: 10Halfak) [17:41:15] Pushing to beta ^ [17:41:49] halfak: i just needed you to read the task i sent awigt [17:42:26] In another meeting ^_^ [17:42:35] Zppix: On the bright side, we’re planning to kill GitHub shortly! [17:42:46] c.f. http://gitlab.com/wiki-ai/ [17:43:14] awight: is the org setup completely [17:43:52] Zppix: no, there’s plenty left to do [17:44:19] Ok [17:46:22] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap, 10Release-Engineering-Team (Watching / External): scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3809968 (10demon) > What will happen if we try to checkout a project with git-lfs-enabled submodules on tin? Scap does not speak g... [17:47:50] 10Scoring-platform-team, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3809969 (10mmodell) 05Open>03Resolved [17:49:46] 10Scoring-platform-team: Use "cache_revs" scap config to limit us to 1 or 2 revisions for rollback - https://phabricator.wikimedia.org/T182013#3809974 (10awight) [17:51:35] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap, 10Release-Engineering-Team (Watching / External): scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3809993 (10awight) >> And will scap be able to fetch and checkout on deployment targets? > > See above. The only caveat is that I'... [17:51:44] FYI beta is down [17:52:34] deployment problem [17:56:09] (03PS1) 10Awight: Limit to no more than 3 cached revisions [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/395048 (https://phabricator.wikimedia.org/T182013) [18:01:07] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap, 10Release-Engineering-Team (Watching / External): scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810031 (10demon) Phab's already behind Varnish, Gerrit is not yet (cf some bug I don't have in front of me). But with objects this... [18:04:44] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap, 10Release-Engineering-Team (Watching / External): scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810042 (10awight) Good point, that won't work at all. On the bright side, we know that the git-lfs load is actually smaller than... [18:05:00] beta is back up. [18:06:05] halfak: https://ores-beta.wmflabs.org/v3/scores/simplewiki/12345 [18:06:15] \o/ awight [18:06:16] :) [18:07:32] I think we also need a config change to enable the precaching and UI features, should I go ahead and do that today? [18:09:33] halfak: for an anti-vandal bot (on simple) i need some randomized vandalism and good edits without bias how could i go about that? [18:11:11] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, 10artificial-intelligence: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3810064 (10awight) This change is deployed to the beta service, e.g. https://ore... [18:13:32] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap, 10Release-Engineering-Team (Watching / External): scap support for git-lfs - https://phabricator.wikimedia.org/T181855#3810070 (10demon) Yep I think we're on the same page here. [18:15:09] * awight squints really hard at PHP lint [18:15:53] lol I copied a whitespace indicator due to .vimrc [18:32:12] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for simple.wikipedia.org - https://phabricator.wikimedia.org/T181849#3810141 (10Halfak) http://labels.wmflabs.org/ui/simplewiki/ [18:34:44] done! [18:34:46] woo [18:34:47] lunch [18:34:48] o/ [18:35:17] Zppix, see T181849 [18:35:26] T181849: Edit quality campaign for simple.wikipedia.org - https://phabricator.wikimedia.org/T181849 [18:44:52] halfak|Lunch: is data for that public? If so can i steal it when some data is generated [18:47:01] Amir1: halfak|Lunch: Question, why is the wp10 model disabled for enwiki in wgOresModels? [19:14:02] Strange, I’m not seeing any highlighting: https://simple.wikipedia.beta.wmflabs.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&damaging__likelygood_color=c5&damaging__maybebad_color=c4&damaging__likelybad_color=c3&damaging__verylikelybad_color=c2&urlversion=2 [19:14:21] thresholds requests are returning good values and precache requests are going back and forth. [19:18:31] I think the scores aren’t being cached into MediaWiki? [19:21:38] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, 10artificial-intelligence: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3810357 (10awight) This is on the beta wiki, but I'm not going to proceed furthe... [19:22:12] Amir1: We need a database migration to enable ORES on new wikis? Probably a good process to document so I can figure it out next time… [19:30:23] awight, right. That's not something I've managed before. We'll need to wait for Amir1 for instructions. [19:30:46] Glad to find the roadblocks... [19:31:30] I'm feeling a little bit under the weather. [19:31:38] I'm going to try laying down for a little bit. [19:31:42] btw I need to take Mari out again, probably will be able to work in 30 min then for 2hr [19:31:44] no worries [19:31:51] I'll check back in after about an hour. [19:32:06] I have some pretty safe deployments for this afternoon, I’ll be dogfooding the new monitoring instructions :) [19:32:08] kk thanks for the info. That means you'll be around in an hour to catch up, right? [19:32:16] awight, cool [19:32:44] I’ll probably be able to work 20:00-22:00 UTC [19:32:47] Ahh damn, I have an interview in 30 minutes. [19:32:57] cool makes sense. I should be around. [20:06:22] back. [20:06:29] But just about to get on a call. [20:06:33] hehe [20:06:43] was there something specific you wanted to catch up on> [20:06:45] ? [20:07:03] na. Just to be available if you needed me :) [20:07:08] :) [20:07:31] I’m a fan of the long haul, just get better for the rest of the week :o [20:08:20] I'm OK. I think I just pedaled too hard during my lunch time ride. [20:08:28] Feeling better after laying down for a bit. [20:09:28] lol [20:09:51] on that note, I’m very partial to naming our vision page “own goals" [20:10:05] I'd be OK with that. [20:10:12] I think for some reason I thought you were against that [20:12:44] LOL it’s the single greatest idea yet [20:13:04] Hey so what are the consumers for wp10? [20:13:06] Please feel free to move :) [20:24:13] wat, ores-beta is "overloaded" [20:24:50] ooh maybe it really is, there’s a lot of traffic. [20:25:23] halfak: How do I reset overload counters? [20:26:54] no worries, I’ll just deal with this later. We’re close to functional on simplewiki-beta [20:57:40] tgr: I don’t think I’ve seen the bug you’re describing—if the special cases were only pylru and redis, there are a lot of packages which would have been missing. [20:58:31] pylru/redis are special cases in that they are not defined as dependencies at all [20:58:51] but the properly defined dependencies were not getting installed either [20:59:07] that's why vagrant fetches the requirements.txt and feeds it to pip [20:59:28] which is a nasty hack but only way I could make it work [21:00:19] oh I missed that! I’ll check it out. [21:01:08] Also, interesting point wrt. optional requirements, we could include an optional-requirements.txt, but I’m not sure if that makes sense to do when there could be alternative optional configurations. [21:01:27] there is optional dependency support in setup.py [21:02:14] you can set ores[lru] as a dependency, and then some hook in setup.py receives 'lru' and can modify the dependencies based on that [21:02:36] ah https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies [21:02:43] extras_require [21:03:08] e.g. https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/sentry/manifests/init.pp#L84-L92 [21:03:35] a lot nicer because you can decouple the exact dependency list from puppet [21:04:06] not sure if that would work with editable packages but worth a try IMO [21:04:39] tgr: btw I’ve been thinking about this much heavier solution: we could base the vagrant install on our production repo, which includes all the wheels, and the consuming modules which implement concrete models: https://github.com/wikimedia/mediawiki-services-ores-deploy/ [21:05:15] nice example, thanks! [21:07:44] there are two goals for vagrant: make it easy to write patches for all the repos which are typically involved in new feature development for ORES, and make the setup close enough to production that you can meaningfully test config changes [21:08:10] the second is more of an afterthought, most of the time it's not fully feasible [21:08:35] That’s helpful—both points suggest that we should try to adapt the production repo. [21:08:57] AIUI using wheels would make the first hard (say you need to write a patch for ORES and an accompanying one for revscoring, how would that work?) [21:09:23] good point, we would have to prune the wheels a bit. [21:09:25] I don't know too much about Python dependency managament though so might just misunderstand the setup [21:10:14] for example in PHP we don't use the mediawiki/vendor repo in vagrant [21:10:43] revscoring is the only package included as a wheel, that we would want to override and make locally developable. [21:10:49] that would make it easier to test for production, but pretty much impossible to write patches for the upstreams of mediawiki/vendor, which is a much more common task [21:11:18] That shouldn’t be too hard, we would either filter the wheel installs to remove it, or pip install -e after the wheels. [21:11:46] well, revscoring pulls in deltas, mwapi etc [21:11:53] what if you need to change those? [21:12:26] if it's a wheel they are packaged in a way that you can't really write patches for them, right? [21:12:55] or are dependencies of a wheel managed outside of it? [21:16:30] wheels aren’t too restrictive, it’s just a shortcut for the install step. Once that step is completed, it’s easy to override in the virtualenv. [21:22:48] pip install takes it out of the packaging, and once installed it looks the same as any other python module, so you can “pip install -e” right over it. [21:23:18] I would prefer to filter the initial install though, so there’s less chance of accidentally using the wheel rather than a check-out source repo. [21:34:59] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Release-Engineering-Team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3810860 (10mmodell) [21:37:22] halfak: Looks like I have to go, there’s a tense moment on the trampoline. [21:37:51] huh? [21:38:05] I can explain tomorrow, circus class drama. [21:38:14] Deployment is healthy, we have verbose logging and simplewiki server-side support. [21:38:23] \o/ [21:38:25] Wooo [21:38:27] Thanks! [21:58:11] 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES, 10User-notice-collaboration: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3810891 (10Halfak) Confirmed that this now deployed to the ORES service. https://ores.wikimedia.org/v3... [22:33:06] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for simple.wikipedia.org - https://phabricator.wikimedia.org/T181849#3810961 (10Halfak) a:03Halfak