[07:49:03] 10Scoring-platform-team (Current), 10DBA, 10MediaWiki-extensions-ORES, 10Blocked-on-schema-change, 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3780295 (10Marostegui) [07:50:36] 10Scoring-platform-team (Current), 10DBA, 10MediaWiki-extensions-ORES, 10Blocked-on-schema-change, 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3780297 (10Marostegui) [07:57:02] 10Scoring-platform-team (Current), 10DBA, 10MediaWiki-extensions-ORES, 10Blocked-on-schema-change, 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3780300 (10Marostegui) [10:51:29] 10Scoring-platform-team (Current), 10DBA, 10MediaWiki-extensions-ORES, 10Blocked-on-schema-change, 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3780746 (10Marostegui) [12:07:38] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3780900 (10MaxBioHazard) When ORES will be reenabled? [12:13:53] (03PS1) 10Prtksxna: Update colors of betafeatures icon [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392827 (https://phabricator.wikimedia.org/T180427) [12:21:47] (03CR) 10Ladsgroup: [C: 032] Update colors of betafeatures icon [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392827 (https://phabricator.wikimedia.org/T180427) (owner: 10Prtksxna) [12:23:25] (03Merged) 10jenkins-bot: Update colors of betafeatures icon [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392827 (https://phabricator.wikimedia.org/T180427) (owner: 10Prtksxna) [12:52:43] (03PS5) 10Petar.petkovic: [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [12:53:42] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [12:56:19] (03PS6) 10Petar.petkovic: Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [13:08:09] (03CR) 10Petar.petkovic: "For position in RC section, the ticket wants "Above Pending changes section, if possible.". Both prefs are added via Hooks and preferences" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [13:37:34] hey [14:17:24] (03PS1) 10Ladsgroup: Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) [14:18:29] (03CR) 10jerkins-bot: [V: 04-1] Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [14:20:45] (03PS2) 10Ladsgroup: Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) [14:54:15] o/ [14:54:58] halfak: hey [14:57:13] halfak: awight I made a patch in the extension to "ease" the issue a little but we need a more robust solution here [14:57:30] Amir1: cool, looking forward to hearing about it. [15:01:56] Amir1: halfak: haha surprise new meeting time [15:02:12] surprise? Requested! [15:02:12] was looking for a place :D [15:31:47] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: Migrate ORES extension threshold config from old to new syntax - https://phabricator.wikimedia.org/T181159#3781494 (10awight) [16:06:54] Hey! [16:07:02] So I'm going to look at the status of beta. [16:07:17] I want to get all of the versions cleaned up there and see if I can break it >) [16:07:20] *>:) [16:11:05] O/ [16:12:31] Awight users are asking when/ if ORES will be reenabled on ruwiki and frwiki, any idea? [16:13:00] We're dealing with fallout now. I think the best we can guess is next week -- after the US holiday weekend [16:13:14] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Revscoring tune does not recognize a set of labels as target - https://phabricator.wikimedia.org/T181163#3781612 (10Sumit) [16:13:18] halfak: alright [16:13:43] Zppix: Nice to hear we’re popular! [16:13:54] if for the wrong reasons... [16:14:17] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Revscoring tune does not recognize a set of labels as target - https://phabricator.wikimedia.org/T181163#3781612 (10Sumit) [16:15:06] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3781629 (10Zppix) >>! In T181006#3780900, @MaxBioHazard wrote: > When ORES will be reenabled? U... [16:16:05] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3781630 (10Halfak) @MaxBioHazard it seems like #release-engineering-team would like us to wait u... [16:20:32] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3781656 (10MaxBioHazard) OK, but can you explain - why you can't just undo the change, that caus... [16:20:38] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Revscoring: Statistic for multilabel classification - https://phabricator.wikimedia.org/T181166#3781670 (10Sumit) [16:21:00] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Revscoring: Statistic for multilabel classification - https://phabricator.wikimedia.org/T181166#3781670 (10Sumit) [16:21:02] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Train/test draft topic model (new article routing AI) - https://phabricator.wikimedia.org/T123327#1926288 (10Sumit) [16:21:04] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Revscoring tune does not recognize a set of labels as target - https://phabricator.wikimedia.org/T181163#3781683 (10Sumit) [16:23:12] 10Scoring-platform-team (Current), 10Research Ideas, 10artificial-intelligence: Revscoring tune does not recognize a set of labels as target - https://phabricator.wikimedia.org/T181163#3781686 (10Sumit) [16:23:33] 10Scoring-platform-team (Current), 10Research Ideas, 10artificial-intelligence: Revscoring: Statistic for multilabel classification - https://phabricator.wikimedia.org/T181166#3781688 (10Sumit) [16:28:44] halfak: Personally, I’d rather not ask for ORES to be reenabled on ruwiki [16:29:01] This is effectively Friday cos of the holiday [16:29:02] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3781696 (10Halfak) The machine predictor for Russian is intact. It's an incompatibility with Me... [16:29:07] awight, technically, politically, both? [16:29:17] awight, right. [16:29:21] Politically / respect for coworkers relaxation window [16:29:33] But it's better to not say "no" unless I know the answer is "no" :) [16:29:33] technically I think we’d be fine :D [16:29:49] The best I can do is confirm for MaxBio [16:29:55] Sure, it’s up to you—I’d rather not even ask though. So good thing you’re doing the asking :) [16:35:02] OK I'm going to work on beta now. [16:35:04] awight, ^ [16:35:13] +1 [16:35:41] halfak: You’ll want to downgrade beta to match production. [16:35:44] 5084251 [16:35:55] tree-ish for ores-prod-deploy ^ [16:36:07] awight, actually, I want to upgrade beta to see what the failure was like. [16:36:11] oh [16:36:13] Right now, it is working [16:36:20] So I suspect it's *not* running recent code. [16:36:25] good—cos I accidentally pasted you the *future* tree-ish [16:36:26] lolol [16:36:36] Reading my SAL pastes backwards. [16:36:52] 5084251 is the revision that I deployed which broke everything. [16:37:01] 82a13ae should be the current revision on production. [16:37:57] Confirmed 5084251 [16:38:21] I think I'll attempt the changes we made to scap though. that'll bring us up to 1eee380 [16:38:31] meh... first I'll replicate [16:38:34] Then test scap changes. [16:39:48] Nice, thanks for taking that task [16:39:52] 10Scoring-platform-team (Current), 10ORES: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781727 (10Halfak) [17:02:38] awight, is there something I need to do in order to tell beta wikis to update thresholds? [17:03:04] 10Scoring-platform-team (Current), 10ORES: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781823 (10Halfak) I deployed https://phabricator.wikimedia.org/source/ores-deploy/ @ 5084251 Problem does not exist in Beta at https://ru.wikipedia.beta.wmflabs.org/wiki/Служебная:Свежие_пр... [17:03:38] halfak: You’ll probably need to purge the keys, since it’s caching thresholds for 24hr [17:03:48] e.g. [17:03:49] awight@tin:~$ mwscript eval.php --wiki frwiki [17:03:50] $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache(); [17:03:51] $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 ); [17:03:52] $cache->delete($key); [17:04:22] awight, can you do that in a paste for me? <3 [17:04:30] * halfak doesn't want to f--- it up [17:09:05] totes [17:09:31] Do it from deployment-tin.eqiad.wmflabs, obviously [17:09:48] https://phabricator.wikimedia.org/P6366 [17:10:01] Has to be done for each {wiki} x {model} [17:10:51] Crtl-D at the end? [17:11:12] Also, I'm running this from deployment-tin, right? [17:12:05] *labs* pls [17:12:10] yes to the rest of that. [17:12:21] Right. labs [17:13:13] PHP Fatal error: no version entry for `frwiki`. [17:13:57] codezee, do you have a WikiProject directory dataset ready to share? [17:14:07] I want to share it with a researcher at the WMF [17:14:19] awight, ^ getting that PHP error above. Any idea what's going on? [17:14:24] nuts [17:14:46] I'm back [17:15:17] halfak: Is there a beta frwiki? [17:15:24] Yesa [17:15:29] https://fr.wikipedia.beta.wmflabs.org/wiki/Special:RecentChanges [17:15:35] seems that no [17:16:04] Woops [17:16:08] I should be doing ru [17:17:14] Done. Still can't replicate. [17:17:36] 10Scoring-platform-team (Current), 10ORES: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781881 (10Halfak) I might need to clear thresholds. I'm running P6366 for "damaging" and "goodfaith" on ruwiki. Still seems to all be working. [17:17:56] halfak: Wait I thought you said earlier you could replicate. [17:18:05] Nope. We could *try* [17:18:11] But never succeeded. [17:18:14] dafuq [17:18:17] Right [17:18:30] Are you seeing loglines for requesting the threshold? [17:18:52] 10Scoring-platform-team (Current), 10ORES: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781888 (10Halfak) FYI, here's what I ran: ``` $ mwscript eval.php --wiki ruwiki > $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache(); > $key = $cache->makeKey( 'OR... [17:19:06] awight, not sure where to look for those log lines. [17:20:44] halfak: https://logstash-beta.wmflabs.org/app/kibana#/dashboard/default?_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue%3A0)%2Ctime%3A(from%3Anow-1h%2Cmode%3Aquick%2Cto%3Anow)) [17:20:51] grr that can’t be right [17:21:10] * awight paste into incognito [17:21:44] halfak: OK this one will work, https://logstash-beta.wmflabs.org/goto/bf3b7bfbab2a6c24b537d10210b57e87 [17:22:09] BTW — no python application found, check your startup logs for errors --- [17:23:29] oh. Huh [17:23:37] ORES service is totally dead :) [17:23:41] In beta [17:24:06] lawl [17:24:06] the first sentence without the punch line can give people heart attack [17:24:19] long live the ORES service! [17:25:00] right. I realized my mistake sending that right away :) [17:26:03] "ImportError: No module named 'ores'" [17:26:05] WTF [17:26:13] WTFTFTFFYCK [17:26:20] that sounds like the virtualenv squirreliness [17:26:31] check /srv/deployment/ores/venv/lib/python3.4/site-packages [17:26:35] populated? [17:26:44] awight, we don't install an ORES wheel [17:26:52] it’s… [17:26:56] It should get imported from CWD vis symlink [17:26:58] yes [17:27:37] submodules is empty in the deployment [17:27:43] I bet we ran out of space on tin again [17:27:46] that it. [17:27:52] not tin, on sca03 [17:28:10] you have to f***ing manually remove some versions from deploy/rev-cache [17:28:11] oh yeah. Great :) [17:28:19] deploy-cache/revs [17:28:20] UGH [17:28:23] Can I do that or do we need releng? [17:28:28] sudo is yours [17:28:51] * halfak looks at hashes and chooses who lives and who dies [17:28:56] /o\ [17:28:58] Any problem with me clearing out all of them? [17:29:03] doit [17:29:10] rm -rf * [17:29:11] ls -t is your friend [17:29:11] ! [17:29:23] I use that [17:29:30] ls -lt [17:29:32] :P [17:30:17] * halfak lights sca03 on fire with deletions [17:30:24] 10Scoring-platform-team, 10Scap: Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3781945 (10awight) [17:30:25] rm -rf /* [17:30:44] um [17:30:48] no you didn't [17:30:52] no :) [17:30:55] :D [17:30:59] O:) [17:36:37] halfak: regarding the draftquality issue, I thought we dropped the index and it's was still not working, I checked and it seems they haven't dropped it in enwiki yet [17:36:37] https://phabricator.wikimedia.org/T180045 [17:36:41] that was my bad [17:36:49] we have to wait a little bit more [17:36:54] gotcha. [17:39:06] Damn. ORES-beta is still dead after cleaning up and re-deploying. [17:39:08] * halfak digs [17:40:38] halfak: oh on beta you can also sudo less /srv/log/ores/main.log [17:40:49] on the worker node [17:41:33] Did you redeploy? I don’t see the server restart. [17:43:53] You might have to scap deploy --force so it doesn’t abort after seeing that the revision hasn’t changed. [17:43:59] halfak: ^ [17:48:08] awight, Oh! thanks [17:48:14] That's probably why nothing happened. [17:48:18] Was trying to figure that out. [17:48:26] Sooo many pitfalls... [17:48:33] Sorry you’re getting to experience them first-hand. [17:49:29] I'm using ores-misc atm [17:50:05] halfak: yeah I had this --force thing too, I should've documented that [17:50:41] ORES-beta lives! [17:50:59] \o/ [17:52:28] OK ores is alive, I've cleared the keys. Still can't reproduce the error. [17:52:31] https://ru.wikipedia.beta.wmflabs.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=30&urlversion=2 [17:53:43] halfak: well, it's reproduced [17:53:50] to some degrees [17:53:58] ores is disabled [17:54:11] which is because it's disabled in prod [17:54:13] of course [17:54:16] Amir1, oh! that's a thing. Can we re-enable it here? [17:54:24] let me make a change and deploy it [17:54:35] it requires weird deployment process [17:54:56] * awight reports entire team to S.I. Mueller [17:55:21] oh “S.C.” [17:57:13] Amir1, thanks [17:57:18] Please tag https://phabricator.wikimedia.org/T181168 [17:58:09] 10Scoring-platform-team (Current), 10ORES: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3782050 (10Halfak) With the help of @awight and @ladsgroup, I figured out that the deploy failed. The deploy has now succeeded and we're working on ruwiki's configuration so that it will eve... [18:00:28] when rebuilding the models I get this error, ImportError: numpy.core.multiarray failed to import [18:00:41] which numpy version is needed? it's 1.10.4 [18:01:57] halfak: The patch is being merged and I will let you know when it's live (It's continuous deployment and might take one hour, I can't do it manually without too much hassle) [18:02:12] Amir1: take numpy from submodules/wheels [18:02:14] I also need to rebase the config repo in prod because of sync reasons [18:02:19] Amir1, https://github.com/wiki-ai/revscoring/blob/master/requirements.txt#L10 [18:02:50] Looks like we expect 1.10.4 to work [18:02:59] I bet the problem is sklearn [18:03:08] or maybe scipy version mismatch [18:05:19] halfak: ladsgroup@deployment-tin:~$ mwscript eval.php --wiki=ruwiki [18:05:19] > echo $wmgUseORES; [18:05:19] 1 [18:05:22] enabled [18:05:29] Amir1, cool thanks! [18:05:40] Damn it! No error still [18:06:04] Halfak maybe varnish cache? [18:06:41] Na. I'm logged in. [18:07:27] IIRC, varnish passes everything in beta for debug reasons [18:08:02] halfak: strangely it's not enabled yet [18:08:02] Can I check to see what values a key has? [18:08:10] I see it. [18:08:14] $cache->get [18:08:15] https://ru.wikipedia.beta.wmflabs.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?damaging=maybebad&hidebots=1&hidecategorization=1&hideWikibase=1&hidelog=1&limit=50&days=30&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&urlversion=2&uselang=en [18:08:18] Load that up [18:08:24] you should see it in the filters. [18:08:47] yeah [18:08:50] had to reload [18:08:54] * Amir1 facepalms [18:09:11] halfak: Do you have the thresholds URL being requested? [18:09:22] no. Not sure how to look that up [18:09:26] Maybe in ores logs [18:09:29] we can find it in logstash beta [18:09:40] It’s that URL we were using earlier... [18:09:48] * awight gets off lazy butt [18:10:14] https://ores-beta.wmflabs.org/v3/scores/ruwiki/?models=damaging&model_info=statistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.99%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.45%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.75%22%7Cstatistics.thresholds.true.%22maximum+filter_rate+%40+recall+%3E%3D+0.9%22&format=json [18:10:21] no null. [18:10:32] oh hold up there was only one model nulling... [18:10:32] https://ores-beta.wmflabs.org/v3/scores/ruwiki/?models=damaging&model_info=statistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.99%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.45%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.75%22%7Cstatistics.thresholds.true.%22maximum+filter_rate+%40+recall+%3E%3D+0.9%22&format=json [18:10:34] Right [18:10:43] https://logstash-beta.wmflabs.org/goto/face6a3469fcda5a72577f39a729130e [18:11:05] Pretty sure it was goodfaith [18:11:10] and there’s no threshold request for that [18:11:16] halfak: did you purge both? [18:11:20] Yes [18:11:25] where is the... [18:11:30] I'll do it again [18:12:09] {{done}} [18:12:09] You rule, halfak! [18:12:42] See https://phabricator.wikimedia.org/T181168#3781888 [18:12:49] {{procrastinated}} [18:12:50] 10[1] 04https://meta.wikimedia.org/wiki/Template:procrastinated [18:12:59] maybe it should be "makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 0 );" [18:13:05] Instead of "makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 );" [18:13:12] no version is set to 1 [18:13:15] kk [18:13:23] "OresCacheVersion": 1 [18:13:39] Does that key ever get populated? [18:14:02] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3782119 (10Halfak) Confirmed that ORES is showing up in Special:Recentchanges. [18:14:43] ruwiki does have goodfaith thresholds set in config, at least. [18:14:59] They include formulas that require the API call [18:15:03] I get nothing for $cache->get($key) for either damaging or goodfaith. [18:15:21] halfak: goodfaith model is disabled on labs! [18:15:34] wmf-config/InitialiseSettings-labs.php -> wgOresModels [18:15:41] that is why [18:15:49] WTF [18:15:57] Don’t know what’s up with the damaging key not getting populated though [18:16:05] :))) [18:16:09] I made another patch [18:16:14] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3782127 (10Halfak) Checking on key values: ``` halfak@deployment-tin:/srv/deployment/ores/deploy$ mwscript eval.php --wiki ruwiki > $cache = MediaWiki\MediaWikiServices... [18:16:14] Not the first nor last time we’ll see halWTF [18:16:18] *make [18:16:24] Amir1: ty! [18:16:42] hehe it’s enabled for enwiki only [18:16:54] * halWTF is in WTF mode and is trying to recover. [18:17:11] awight, please post your discovery here: https://phabricator.wikimedia.org/T181168 [18:17:16] halWTF: Why bother when it will be reenabled again in moments [18:18:29] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3782148 (10awight) We noticed that the `$wgOresModels['default']['goodfaith']` model is disabled on the beta cluster for all wikis but enwiki. @Ladsgroup is patching `I... [18:19:32] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Improvements to ORES deployment documentation and process - https://phabricator.wikimedia.org/T181183#3782151 (10awight) [18:21:14] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Improvements to ORES deployment documentation and process - https://phabricator.wikimedia.org/T181183#3782151 (10awight) [18:22:25] Amir1: halWTF: Want to review https://wikitech.wikimedia.org/wiki/Incident_documentation/20171120-Ext:ORES before I email? [18:22:33] codezee, in the mid-level dataset, I see "Geography.Americas": [],' [18:22:36] That can't be right. [18:22:40] We could add more actionables, but those are the only ones that seem “actionable” [18:22:42] awight, yes. [18:22:47] ty [18:23:06] Let's settle on why this didn't appear in beta first. [18:23:08] awight, ^ [18:23:18] 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3782175 (10awight) [18:23:25] halWTF: K, I’m fine waiting. Seems like you’re close. [18:23:42] :) [18:23:45] I mean—it didn’t appear in beta cos *I didn’t check ruwiki:S:RC*, I won’t lie [18:24:15] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review, 10Wikimedia-Incident: Parallelize scap for ORES - https://phabricator.wikimedia.org/T181067#3782183 (10awight) [18:26:07] Even if there is an element of misconfiguration. [18:26:54] awight, sure, but maybe someone else would have noticed. [18:27:10] We should have code sitting in beta for a while before deployments most of the time [18:27:38] I think we'll be adding a todo to make sure Beta's configuration looks like prod. [18:27:52] None of this "goodfaith in enwiki only" nonsense. [18:28:16] It def was on beta for a good amount of time—let me spelunk exactly how long. [18:29:06] +1 the TODO to make the config the same [18:29:58] Amir1: Is it correct that we can just delete config keys from InitialSettings-labs.php and they fall back to production values, or does the beta cluster *only* use the -labs config? [18:30:22] awight: it falls back to default in -labs and then falls back to prod [18:30:38] oh nice, ty. So we can just delete config then... [18:30:43] there is some sort of documentation in top of -labs file [18:30:48] yup [18:33:26] halfak: Darn, I only deployed to beta at 20:17, so that’s 1h20 before pushing to production. [18:33:41] Woops. [18:33:43] Pretty much no chance anyone would have randomly gone to the pages in question, even if config was identical. [18:33:46] Not terrible, but not great [18:33:57] Well, it might have set off someone else's alarms. E [18:33:58] halfak: awight It's not deployed to labs yet, you can test it easily [18:33:58] The follies of youth [18:33:59] g. collab [18:34:26] Amir1: sorry, what’s not deployed yet? [18:34:37] enabling goodfaith in ruwiki [18:34:37] SoS time... [18:34:43] merged but not deployed [18:34:43] \o/ [18:34:56] second dinner time [18:35:06] be back in one hours or so [18:35:08] o/ [18:35:49] o/ [18:36:25] "[ba10d7bf] PHP Fatal Error: Call to a member function makeKey() on null" [18:36:28] https://logstash-beta.wmflabs.org/app/kibana#/dashboard/default?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(('$$hashKey':'object:2642','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:channel,negate:!t,value:MessageCache),query:(match:(channel:(query:MessageCache,type:phrase)))),('$$hashKey':'object:88','$state':(store:appState),meta:(alias:!n,disabled:!f,index [18:36:28] :'logstash-*',key:channel,negate:!t,value:CentralAuthVerbose),query:(match:(channel:(query:CentralAuthVerbose,type:phrase))))),options:(darkTheme:!f),panels:!((col:1,id:Dashboards,panelIndex:1,row:1,size_x:12,size_y:2,type:visualization),(col:1,id:Events-Over-Time,panelIndex:2,row:3,size_x:12,size_y:2,type:visualization),(col:1,id:Event-Types,panelIndex:3,row:5,size_x:9,size_y:2,type:visualization),(col:10,id:Event-Level,panelIndex:4,row:5,size_ [18:36:30] x:3,size_y:2,type:visualization),(col:1,columns:!(type,level,wiki,host,message),id:Default-Events-List,panelIndex:5,row:7,size_x:12,size_y:25,sort:!('@timestamp',desc),type:search)),query:(query_string:(analyze_wildcard:!t,query:ruwiki)),title:default,uiState:(P-2:(vis:(legendOpen:!f)),P-3:(vis:(legendOpen:!f)),P-4:(vis:(legendOpen:!f)))) [18:36:34] lolwat [18:36:49] https://logstash-beta.wmflabs.org/goto/3ae6f66074d90335858f59ddb1b9abc8 [18:38:10] wtf is that [18:38:15] No idea. [18:38:20] * awight considers going full-on WTF mode [18:38:21] It happens right after the call to ORES [18:38:33] No nulls returned by ORES [18:38:39] https://ores-beta.wmflabs.org/v3/scores/ruwiki/?models=damaging&model_info=statistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.99%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.45%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.75%22%7Cstatistics.thresholds.true.%22maximum+filter_rate+%40+recall+%3E%3D+0.9%22&format=json [18:41:17] * halfak refreshes waiting for deploy of ruwiki ores [18:41:24] (To beta) [18:42:30] Actually, this looks like a good time for a break. [18:47:26] halfak|Lunch: That means that the $cache wasn’t initialized—we’re using very standard factories to get that though. [18:51:17] [1e9c32e7637ff25a621b4b73] [no req] ErrorException from line 1 of /srv/mediawiki-staging/php-master/maintenance/eval.php(78) : eval()'d code: PHP Notice: Undefined variable: cache [18:52:23] halfak|Lunch: I don’t see evidence that it’s “right after” any call to ORES though, looking at timestamps. [18:52:55] Nothing for at least a minute in either direction. Oh—and it’s from tin, so that’s your commandline. [18:55:36] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): ORES beta cluster config should be as close to production as possible - https://phabricator.wikimedia.org/T181187#3782311 (10awight) [18:58:30] halfak|Lunch: Amir1: there it is. https://ru.wikipedia.beta.wmflabs.org/wiki/Служебная:Свежие_правки?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=30&urlversion=2 [19:01:27] (03CR) 10Awight: [C: 04-1] "This is a good incremental fix--can you also catch an unparseable threshold (RuntimeException) and apply the same degradation?" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [19:03:48] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES, 10Release-Engineering-Team (Watching / External): Make ORES-consuming pages more robust to ORES errors - https://phabricator.wikimedia.org/T181191#3782397 (10awight) [19:17:55] (03CR) 10Awight: [C: 04-1] "This can be tagged with T181191, too." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [19:24:09] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES, 10Release-Engineering-Team (Watching / External): Make ORES-consuming pages more robust to ORES errors - https://phabricator.wikimedia.org/T181191#3782477 (10awight) [19:24:40] halfak|Lunch I looked into Americas being blank, it turns out https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Council/Directory/Geographical links to the article of Americas twice, once under "Cities in the United States" and other under "Countries" which seems ok [19:25:05] but since "Cities in the united states" comes first it gets parsed under that heading and is not processed further [19:25:18] * I mean not processed again under countries [19:26:14] so all the valid entries( with active:yes ) are appearing under "Cities of the United States" [19:26:26] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Make ORES-consuming pages more robust to ORES errors - https://phabricator.wikimedia.org/T181191#3782486 (10awight) [19:26:33] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: ORES beta cluster config should be as close to production as possible - https://phabricator.wikimedia.org/T181187#3782487 (10awight) [19:27:09] i think catching these kinds of errors isn't that difficult but someone will have to sit down and sift through the 2000 line mid-level dataset to make sure that the wikiprojects under each heading make sense [19:27:28] and make note of these anomalies which can be fixed [19:29:17] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3782495 (10awight) @hoo Wondering if you wrote an incident report, that I can add to with an explanation of ORES's involvement? [19:35:18] relocating. [19:36:31] Yay! Error! [19:36:36] https://ru.wikipedia.beta.wmflabs.org/wiki/Служебная:Свежие_правки [19:36:37] haha [19:36:41] yep [19:37:03] codezee, gotcha. [19:37:12] i'll try to take a pass at it myslef this weekend [19:37:16] codezee, we should harej review. [19:37:27] sounds good :) [19:37:33] he did a lot of work on the directory. [19:37:51] he would be able to pinpoint problems quickly [19:38:11] Right. Might even fix the directory. [19:38:12] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3782508 (10BBlack) No, we never made an incident rep on this one, and I don't think it would be fair at this time to implicate... [19:38:18] I'm considering an edit. [19:38:22] and once we flag these errors, fixing them is quite easy, just going to the relevant directory page and removing the erroneous entry [19:40:57] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3782516 (10awight) @BBlack Thanks for the detailed notes! All I was going to add was my understanding of how Ext:ORES has the... [19:42:14] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715229 (10Zoranzoki21) Does it made problem with high sleep times in pywiki? [19:42:26] +1 codezee [19:42:59] The Americas section doesn't make any sense. [19:45:53] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3782522 (10demon) >>! In T179156#3782516, @awight wrote: > @BBlack Thanks for the detailed notes! All I was going to add was m... [19:46:10] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3782523 (10Halfak) Got it! From https://ru.wikipedia.beta.wmflabs.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5... [19:49:11] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781727 (10Halfak) a:03Halfak [19:50:25] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3782540 (10Halfak) [19:50:28] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781727 (10Halfak) [19:54:11] we are largely saved of these inconsistencies by taking only the mid-level categories.... [19:55:29] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3782546 (10Halfak) In T181168, I capture the error message we wanted to see in Beta. In order t... [19:56:11] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Replicate RC/WL failures in Beta - https://phabricator.wikimedia.org/T181168#3781727 (10Halfak) BTW, here is the query that returns a null: https://ores-beta.wmflabs.org/v3/scores/ruwiki/?models=goodfaith&model_info=statistics.thresholds.false.%2... [19:59:48] OK I think I've finished my documentation task here. I think the Incident report is ready for email. [19:59:56] Will ping awight when he gets back. [20:00:01] I'm going back to JADE land :) [20:07:57] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3782558 (10greg) >>! In T181006#3781696, @Halfak wrote: > I've just asked in #wikimedia-releng w... [20:18:16] We wait until midnight to deploy ;-) [20:19:09] halfak: I’m not there there’s anything to add the incident report wrt your beta cluster investigation, aside from the TODO about resetting the beta config? [20:19:24] The wiki page is ready for another review pass… [20:19:41] awight, oh I think it's ready. Said so when you were offline briefly :) [20:19:46] +2! [20:19:55] Kruto. [20:20:15] https://en.wiktionary.org/wiki/круто [20:20:22] Great work Adam :) [20:20:57] Hmm, I thought the literal meaning was “hard-boiled”. [20:21:19] Zppix: Thanks, I’ve been defining that work all morning :D [20:21:22] *word [20:27:31] halfak: i was wondering if we could do like a semi-annual satisfacton/usage survey for ORES to send to users? [20:27:43] Be interesting to see results? [20:28:08] Zppix, hmm maybe. I think that JADE is sort of an ongoing infra for doing exactly that. [20:28:12] *all the time [20:28:23] Amir1, would you be interested in looking at https://phabricator.wikimedia.org/T171119 ? [20:28:29] I'm getting some pings about it recently. [20:28:49] Might go nicely in the space you have carved out for looking at article quality in fawiki [20:28:50] awight: btw, thanks for filing those #-incident follow-up tasks! [20:29:58] greg-g: Thanks again for the suggestion, and all the onwiki docs that let me know how to proceed! [20:30:35] greg-g: oh here’s a weird one—at the time I think you were saying something about ORES 500’ing the site down every other week... [20:30:38] greg-g, ! Thanks for your support in this and for responding to my unreasonable question :) [20:30:52] halfak: :P anytime [20:30:58] No worries if that was just something that was said, but I wanted to know if this is actually a perception? [20:31:15] wait waaa.. I don't think ORES has that many problems. We just talk openly about them a lot. [20:31:31] Halfak ORES is the problem child xD [20:31:37] awight: I think it was a "recent issues over taking perception" issue. When was the last 500 spike due to ORES? 3-4 weeks ago? [20:31:45] greg-g: I know the Oct 28 thing looked like it had our fingerprints all over it [20:31:46] ah okay [20:32:16] greg-g, that wasn't ORES at all :\ [20:32:27] greg-g: bblack had some nice words to say about that in summary, https://phabricator.wikimedia.org/T179156#3782508 [20:32:39] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: ORES beta cluster config should be as close to production as possible - https://phabricator.wikimedia.org/T181187#3782311 (10Addshore) @awight I saw in the incident report that "Not all wikis are availa... [20:33:07] greg-g: Totally not offended that it *looked* like we were taking the site down, I just wanted to either remember better or clarify what happened. [20:33:09] Also I'm remembering the days when ORES was blamed for memory usage on SCB nodes and we've recently learned that ORES hardly uses any memory at all per worker :) [20:33:31] Thanks to the new dedicated cluster, we know we can boost workers by a factor of 10 in less memory. [20:34:24] lol /me buys a used car from halfak [20:34:40] :) [20:35:17] * halfak sells the best used cars. F U Baltimore! [20:35:33] halfak: i love the new cluster i was actually testing with ores's webui for scoring and i only one issue.. my internet adaptor died... finally, [20:35:36] Related (strong language): https://www.youtube.com/watch?v=U1dUk8x2XkA [20:35:44] (I was waiting to put in the new one) [20:36:12] * awight immediately clicks [20:36:19] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3782602 (10greg) [20:36:46] Zppix: The new cluster we’re talking about isn’t accessible from the Internet, so please do let us know if you figure out how to hit it. [20:36:57] Its not? [20:37:07] It feels like it was [20:37:09] nope. Still testing. :D [20:37:12] It was faster then normal [20:37:15] Weird [20:37:20] Maybe i was lucky xD [20:37:31] We could boost capacity so much on this cluster that we discovered a bunch of scaling issues [20:38:37] halfak: few months from now youll be saying need moar xD [20:38:53] How many wikis is ores enabled on? :) [20:39:03] addshore: alot [20:39:06] addshore: around ten [20:39:17] * addshore thinks we can defintly make beta sites for all of those! [20:39:31] addshore: just export the files into xml and import? [20:39:32] halfak: around? I just got back from dinner [20:39:34] Not hard? [20:39:39] what can I do now [20:39:54] * addshore goes back to eating dinner [20:40:00] Amir1: I left a comment on your Ext:ORES patch, fwiw [20:40:14] (03PS1) 10Legoktm: Avoid unnecessary SpecialPage::getContext() call [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392908 [20:40:17] (03PS1) 10Legoktm: Range: Avoid some unnecessary function calls [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392909 [20:40:21] (03PS1) 10Legoktm: Make it a bit easier to figure out that Range::combineWith() is used [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392910 [20:40:27] Spam incomming [20:40:31] Amir1, help fix Ext:ORES :) [20:40:49] Then next, I want to talk about either eu article quality or JADE :D [20:40:49] awight: let me check [20:41:32] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3782606 (10Halfak) Thanks for chiming in @greg. The good news for @MaxBioHazard is that we've n... [20:42:03] I'm still trying to wrap my head around a bunch of the new code [20:42:13] (03CR) 10jerkins-bot: [V: 04-1] Range: Avoid some unnecessary function calls [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392909 (owner: 10Legoktm) [20:42:38] (03CR) 10Legoktm: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392909 (owner: 10Legoktm) [20:42:44] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: ORES beta cluster config should be as close to production as possible - https://phabricator.wikimedia.org/T181187#3782611 (10awight) @Addshore good idea! Here's the matrix of wikis that either have ORE... [20:43:04] (03CR) 10Ladsgroup: "I disagree, when the boundaries can't be parsed it mean something is horribly wrong (server is down, returns 500, etc.) and it properly sh" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [20:43:15] answered there awight [20:43:23] back to legoktm's patches [20:43:30] halfak: T81183's actionable for monitoring client side stuff is we NEED decent grafana alerts in here! We need red loud nuclear alarms... im not 100% sure how grafana backend is with regards to monitoring software but iirc ops has some of that [20:43:43] T181183 * [20:43:43] T181183: Improvements to ORES deployment documentation and process - https://phabricator.wikimedia.org/T181183 [20:44:35] (03CR) 10Awight: [C: 04-1] "It certainly should scream, and log like crazy. But we can't crash, we need to return empty set gracefully." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [20:46:52] halfak: whats this about ai list being dropped? [20:48:05] Oh nm [20:48:07] Lol [20:49:02] (03CR) 10Ladsgroup: [C: 032] Avoid unnecessary SpecialPage::getContext() call [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392908 (owner: 10Legoktm) [20:49:36] (03CR) 10Ladsgroup: [C: 032] Range: Avoid some unnecessary function calls [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392909 (owner: 10Legoktm) [20:51:57] (03Merged) 10jenkins-bot: Avoid unnecessary SpecialPage::getContext() call [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392908 (owner: 10Legoktm) [20:53:14] (03Merged) 10jenkins-bot: Range: Avoid some unnecessary function calls [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392909 (owner: 10Legoktm) [20:53:59] legoktm: Amir1: If it sounds fun, we really need to cut API parsing (processRevision) out of Cache [20:56:12] TBH, The extension needs lots of love [20:56:26] I'm responsible for most of the ugliness [20:56:49] Amir1: Hey I did my share of damage! [20:57:24] Pretty sure I even put that API function I mentioned, in the wrong place. [20:57:57] Your hooks cleanup from Monday was a great move... [21:00:43] Amir1, awight: extension love --> goal for next quarter? [21:00:56] == start working on it this quarter? [21:01:17] halfak: well, it's hard to make it a goal [21:01:33] because the maintenance is hard to measure [21:03:01] halfak: question, I'm trying to drop the ratio --pop-rate "true=0.9998525516181488" in wikidata models [21:03:12] but revscoring.errors.ModelConsistencyError: ModelConsistencyError: Labels {False, True} not in list of expected labels set() [21:03:16] which is obvious [21:03:32] should I make it --pop-rate "true=0.5" ? [21:04:05] Amir1, you can do --labels [21:04:32] thanks [21:04:35] e.g. --labels true,false [21:04:36] :) [21:04:37] np [21:06:41] halfak: extension love is a great idea, though I agree with Amir1 that we need a concrete goal. [21:07:06] awight, fair point. Just because we can't make it a goal doesn't mean it's not good work to plan out. [21:07:08] Maybe we can somehow use JADE as an excuse? Like, we need. a safe place to add a few more MediaWiki hooks? [21:07:18] IMO down with goals when they get in the way of doing good work. [21:07:31] oh I didn’t mean to say it can’t be concrete, but that we should choose a small bite of that as a goal. [21:07:49] Like, 100% test coverage or something. [21:08:11] 100% will probably be impossible. [21:08:18] What is hard because of the mess that is the ORES ext? [21:08:21] :) yeah [21:09:13] well, the coverage is already good (I think I spent half a month writing those after some stupid outages) [21:09:20] It’s not that much of a mess IMO, but tracing a call can be confusing because of the hook-heaviness and the slightly jumbled layers [21:09:37] Amir1: nice! Sorry I hadn’t bothered to check that before speaking. [21:09:48] tracing any call requires less than N steps? [21:09:57] but in order to get wp10 support there, I'm pretty sure need to rethink and redesign the onSave hook [21:10:17] Could make getting wp10 support in there a goal. [21:14:54] That’s a good way to do it. We did a thing that sort of worked in Fundraising, we had an agreement to stealth-couple maintenance with new features, so the plan is to do a new feature, and we all understand that it’s going to take longer than expected due to internal fixing. [21:15:08] * awight feels like I just gave away the plot to kill Batman [21:15:32] This is broken but what else can we do. [21:15:54] ggellerman_, ^ a case for your studies about how quarterly goals are broken [21:16:03] lol [21:16:12] ggellerman_ was there for it, IIRC [21:19:57] Proposal: You can't submit judgements to JADE if you are blocked on any wiki. [21:20:01] What say? [21:20:18] Global blocks are extreme and rare [21:20:35] I did some time managment resources for free a while back to help me with develping my fun but worthless projects [21:21:05] halfak: i say yes to that [21:21:24] Cool. We can change our minds later too. [21:21:34] E.g. if you are blocked on enwiki, no judging things in enwiki [21:21:52] Luckily this works great: https://www.mediawiki.org/w/api.php?action=help&modules=query%2Bglobaluserinfo [21:21:54] Thanks Amir1 :) [21:21:56] I say if blocked on mediawikiwiki or meta your blocked on all? [21:22:05] guiid= came from you, right? [21:22:21] :D [21:22:24] yes [21:22:29] I wrote that part [21:22:35] :D [21:24:24] Hey halfak theres an AI summit in New York NY https://theaisummit.com/ looks interesting [21:24:54] Boo. no more travel for me for a while. [21:25:05] I think its annual [21:25:30] I would go but im broke and i strongly hate asking for money [21:26:16] hi everyone, can anyone help me find the QPS limit of the ORES api? [21:26:32] victor, two parallel threads. [21:26:44] You'll get better throughput if you request in 50 revision batches. [21:26:51] Next Q, what are you querying for? [21:26:59] We might have a dataset for you :) [21:27:02] halfak: Just to be clear, if someone is blocked on a wiki then they’re only disallowed from judging on that wiki, right? [21:27:11] awight: correct [21:27:12] We are evaluating the ORES's precision [21:27:21] awight, my proposal is to have them blocked from judging everywhere. [21:27:30] awight: but IMO i think meta and mediawikiwiki blocks should be considered global [21:27:31] So we need to get a list of 6000~revisions and get ORES damaging model scores of them [21:27:42] halfak: Why would we do that, instead of just the wikis they’re blocked on? [21:28:03] awight, most vandals only get blocked on one wiki. [21:28:14] victor, Gotcha! So, I recommend using the ores score utility. I'll get a link. [21:28:17] awight: halfak what if we treat mediawiki and meta blocks as global and then other project indiviual [21:28:17] one sec. [21:28:18] I think that goes against the current security model. Some people are blocked on one wiki and considered productive on others. [21:28:27] Thank you [21:28:48] awight, right. Most aren't though. And I think that's for bureaucratic reasons [21:29:00] Zppix, there exists a global block [21:29:02] And also, I am already batching in 50 rev each request, by doing '|'.join(list-of-revid) [21:29:15] victor, great. That gets you far. [21:29:20] halfak: i meant global i meant global. For jade [21:29:30] Sure, but are you making this case to work around a technical obstacle? [21:29:35] halfak: ^ [21:29:46] Right, but given there are 6400~ revids to evaluate, we still will end up having about 130~ requests. [21:29:47] Looks like we're due for a new version of ORES. [21:29:50] I'm going to cut it. [21:29:56] O_O [21:30:12] The first a few are latency of 0.5 seconds, but after the 5th request, it become as long as 12 seconds [21:30:33] I guess that's caused by a QPS limit [21:30:36] victor, right. That's expected. [21:30:36] but not sure [21:30:59] @halfak, what would you suggest in this case [21:31:06] victor, getting link [21:32:05] Looks like our docs are in disrepair. I'll describe how to get the utility docs in a sec. [21:33:05] victor, install the most recent version of ores "pip install ores==1.1.0" [21:33:08] Amir1: Do you still disagree with this? I’m okay with merging your change as-is and adding more handling later, but figured this was a good opportunity to straighten the whole thing out. > It certainly should scream, and log like crazy. But we can't crash, we need to return empty set gracefully. [21:33:18] victor, then run "ores score_revisions -h" [21:34:11] thanks +halfak [21:34:12] doing it [21:34:35] by the way, this is my first time on IRC, so bear with me for a super dump question: what's your IRC client in recommendation? [21:34:49] No worries! IRC is crazy old tech. [21:34:59] I use hexchat, but most newcomers are better off with IRCCloud. [21:35:16] hexchat is an old-school client based on XChat. [21:35:25] Ok, halfak, looks much better on IRCCloud [21:35:39] IRCCloud is a modern web UI that does more than hexchat, but lives in your browser. [21:35:43] right :) [21:36:05] Looks like we set the default parallel requests to 4. [21:36:37] That'll be OK [21:38:23] awight: No, I just need a little bit of time to make it happen [21:38:36] we can merge this as it is and I make a follow up [21:39:06] victor, I'm stoked you're auditing ORES :) [21:39:44] hi [21:40:04] I am on IRCCloud now [21:41:07] :) and it's working [21:41:22] but I can't use my old name [21:41:28] I have to add __ as postfix. LOL [21:41:36] Back to ORES QPS limit problem [21:42:10] Amir1: gotcha! [21:42:13] Tried your command, `pip install ores==1.1.0` doesn't work [21:42:39] Not sure if it's some PIP end point I need to configure, is 1.1.0 version already released? [21:42:49] https://gist.github.com/halfak/a145512ed714cf3f526bd00b69ce1727 [21:42:53] That's what I get. [21:42:57] Already released, yeah [21:43:11] (03CR) 10Awight: [C: 032] "Talked through it a bit on IRC. IMO this is a great first step, don't crash on expected data. But we should still take the next step bef" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [21:44:33] +halfak, is there something like `pip install git+https://github.com/some-user/some-project` kind of specific github endpoint we can use [21:44:44] (03Merged) 10jenkins-bot: Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [21:44:58] victor__, there is but it should be totally unnecessary [21:45:05] Amir1: I’ll mess with that exception-catching... [21:45:11] victor__, What error do you get? [21:45:31] let me paste to you [21:46:17] run `pip install ores==1.1.0` and end up having error https://www.irccloud.com/pastebin/qjxcm02c/pip%20failure [21:46:51] victor__: Are you using python2? [21:46:59] my `pip --version` yields `pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)` [21:47:00] Try pip3 install [21:47:13] I’d recommend a virtualenv also, but that’s your call [21:47:17] +awight, is ores using pip2 or pip3? [21:47:24] python 3 [21:47:25] victor__: ores is py3 only [21:47:28] OK [21:47:49] * halfak prepares a rant for python 2 being deprecated 10 years ago [21:47:51] * halfak feels old [21:58:11] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Technical-Debt: Cleanup: Move processRevision out of Cache - https://phabricator.wikimedia.org/T181198#3782749 (10awight) [21:58:21] https://www.irccloud.com/pastebin/BZf2Hpkf/ [21:58:48] +awight and halfak this is the failure when running pip install ores==1.1.0 under virtualenv python 3 [21:58:51] victor__, try "pip install pip --upgrade" [21:58:56] Then "pip install wheel" [21:59:04] Then finally, try the ores install again. [21:59:10] Fyi mediawiki php requirement rfc in #wikimedia-office in few mins (https://phabricator.wikimedia.org/T172165) [21:59:13] ^pip3 [21:59:15] Pip went through some changes that should make this way easier. [21:59:29] yes. "pip3 install pip --upgrade" [21:59:30] oops, I just read the scrollback [22:00:02] samething [22:00:04] with pip3 [22:00:09] or with install wheel [22:00:21] victor__: I think you need to: apt python-dev [22:01:02] You mean `sudo apt-get install python-dev`? [22:01:40] victor__: on second thought, https://github.com/wiki-ai/revscoring [22:01:49] sudo apt-get install python3-dev g++ gfortran liblapack-dev libopenblas-dev [22:02:12] I'm surprised. I thought you could get away without the headers if you installed the wheel. [22:02:14] awight: kk [22:02:20] I guess it will reference the .so files. [22:03:54] Good, after doing sudo apt-get install python3-dev it works [22:03:59] `sudo apt-get install python3-dev` [22:04:09] halfak: I think you’re right about the wheels, maybe the wheel wasn’t available [22:04:20] Might be. That'd make sense. [22:04:42] +halfak, you want me t o call ORES directly, right? [22:04:46] what's the next step? [22:05:04] halfak: yup no wheel: https://pypi.python.org/pypi/mmh3/2.5.1 [22:05:06] Right, if you use that utility I talked about "ores score_revisions -h" [22:05:34] It'll ask for an ores-host URL and you can give it https://ores.wikimedia.org [22:05:37] literally `ores score_revisions -h` or something like `ores 567890123 -h` ? [22:05:46] literally [22:06:33] It takes a dataset of revisions as JSON blobs '{"rev_id": 345678}\n{"rev_id": 2374623}\n...' [22:06:39] And will score them as fast as it can. [22:06:52] Can I suggest `pip install ores==1.1.0` to stop installing whenever it finds the client is not running python3? [22:07:18] cat my_revisions.json | ores score_revisions https://ores.wikimedia.org ... > scored_revisions.json [22:07:53] Or if you don't like piping, then you can do something like this: [22:07:59] victor__: Good idea, I’ll see if there’s a way to specify !=py2 compatibility [22:08:07] ores score_revisions https://ores.wikimedia.org --input my_revisions.json --output scored_revisions.json [22:08:16] awight, I think we might already do that [22:08:22] awight: i think its possible with pip [22:08:31] awight: i cant remember now [22:08:35] https://github.com/wiki-ai/revscoring/blob/master/setup.py#L37 [22:08:48] OK it seems like we have a category but not an clear requirement [22:09:37] I can look into that and do it potientally it sounds easy enough halfak [22:09:39] https://github.com/pypa/pip/issues/4327 [22:09:44] ugh https://pypi.python.org/pypi/require-python-3/1 [22:09:44] Looks like it doesn't exist [22:09:47] We can do it manually. [22:09:55] lol [22:10:08] baahaha and it 404s at github [22:10:25] Well at least the docs specify what was in the code :) [22:11:07] I can work on that? Just do the if statement if the version isnt 3 then fail the setup.py? [22:11:11] Halfak ^ [22:11:46] Zppix: Cool, that’s a big help! [22:11:50] \o/ [22:12:53] Is there a task? [22:13:15] halfak: is stderr perferred over print? [22:13:19] Can you elaborate this part [22:13:19] `` [22:13:20] cat my_revisions.json | ores score_revisions https://ores.wikimedia.org ... > scored_revisions.json [22:13:20] 2:07 PM Or if you don't like piping, then you can do something like this: [22:13:20] `` [22:13:57] I can update the mediawiki.org/wiki/ORES page for this instruction so people who want to query ORES in a batch can run their own instance [22:14:17] victor__: that would helpful :) [22:14:58] +Zppix, thank you [22:15:06] How do you "at"/"mention" someone on IRC? [22:15:08] I [22:15:19] Just say the nick [22:15:25] Zippix [22:15:39] Eh close enough xD mispelt it [22:15:43] But it doesn't get highlighted by irccloud.com client, nor do I think the IRC system recognzie it [22:15:58] Zppix ok [22:16:03] Still no effect [22:16:32] victor__: go to settings and set the highlight words [22:17:26] 10Scoring-platform-team, 10articlequality-modeling, 10Easy, 10Google-Code-in-2017, 10artificial-intelligence: Implement feature for detecting clumps of text that lack references - https://phabricator.wikimedia.org/T174384#3782792 (10Ladsgroup) @Aklapper: [[ https://github.com/wiki-ai/wikiclass/blob/maste... [22:20:03] https://www.irccloud.com/pastebin/6xQujEr4/ [22:21:04] Hi victor__. So I was giving a unix command-line example for how to use the "score_revisions" utility [22:21:47] is the three dots in the `cat my_revisions.json | ores score_revisions https://ores.wikimedia.org ... > scored_revisions.json` also part of the command? [22:22:01] Nope. [22:22:30] the ... would probable be something like "enwiki damaging --verbose" [22:24:01] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3782801 (10Ladsgroup) @Aklapper That guideline is for writing qunit tests in mediawiki which is indeed not easy. In wikilabels, you have control to install npm and do... [22:24:05] halfak: https://ores.wikimedia.org/ui/ shows 2 results whats the difference between them for ex i did rev id 123332 on frwiki goodfaith [22:24:24] victor__, Here's an example run: https://gist.github.com/halfak/c7cec45d55a26891e535cb62225bc4d7 [22:25:14] Zppix, one is True, the other is False [22:25:23] True + False score == 1 [22:25:25] Why? [22:26:08] https://ores.wikimedia.org/v2/scores/frwiki/damaging/123332 [22:26:15] Does that help? [22:27:02] Oh i see [22:27:19] Its showing what the probability of true false that makes more sense [22:27:21] Cool [22:27:30] and that can be unlimited? +halfak [22:27:33] QPS wise? [22:27:36] I wonder if its possible to make the clear on /ui/? [22:27:39] It's pure local, right? [22:28:33] victor__, not pure local. This will do the rate limiting for you. I think you'll find that it's very fast. [22:28:52] That utility is designed to go as fast as possible while respecting ORES usage. [22:29:00] Zppix, I think we have a task for that [22:29:34] Ok, cool [22:29:37] That would be great [22:29:51] Zppix, https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Archived#ORES_UI_visual_JSON_representation [22:29:56] no task yet. Want to file it? [22:29:56] FYI, I updated the mediawiki.org/wiki/ORES [22:29:59] https://usercontent.irccloud-cdn.com/file/oxMaIzPp/image.png [22:30:16] I can current or backlog halfak? [22:30:25] Zppix, backlog [22:30:28] Ok [22:30:39] subscribe me and post some ideas of how you want to implement it :) [22:30:48] +halfak, +awight hope you don't mind I update the local usage to the Wiki page of ORES [22:30:52] victor__, +1 nice job [22:30:57] It's good :) [22:31:00] BOLD! [22:31:28] NOOOO globaluserinfo doesn't give me the local user groups [22:31:30] ARG! [22:31:34] Thank you! By the way. on Wikipedia universal id, I am @xinbenlv [22:32:07] https://en.wikipedia.org/wiki/User:Xinbenlv ? [22:32:11] halfak: iirc match the uid in globalinfo to that in user id and then get usergroups from that userid [22:32:20] Zppix, right. [22:32:21] let me change to xinbenlv [22:32:22] Yes [22:32:40] Zppix, I don't want to run 50 queries for every user though :/ [22:32:52] Halfak dont, make all one [22:33:03] Amir1, still awake? I know it's getting late there. I was wondering if I could ask you something about globaluserinfo [22:33:05] Its only using globalinfo and user table [22:33:21] Zppix, need to look at every wiki DB. [22:33:29] halfak: yup [22:33:36] halfak: oh yeah [22:33:41] halfak: derp [22:33:49] I'm working on cleaning up stuff to do for ORES [22:33:54] Amir1, globaluserinfo doesn't tell me the local groups of a user. Do you think it would be crazy to include that in the result? [22:33:58] Halfak would api work better than a db query? [22:34:08] Amir1, I know you just did guiid stuff but I figured that you'd have some insights [22:34:23] Is there any documentation how to use the ORES local instance programmably, like in a Python program [22:34:39] halfak: hmm, it can get global rights [22:34:51] but local rights, I know it's possible but will be messy [22:35:06] I think another API request for that would be okay [22:35:10] Amir1, damn. I see it is getting local edit counts so I figured that maybe it would be a matter of another join [22:35:21] so we can separate caching too [22:35:23] It's going to be N api requests -- one for each wiki they are a part of. [22:35:30] Halfak what is the expected outcome from https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Archived#ORES_UI_visual_JSON_representation so i file task correctly [22:35:47] xinbenlv, when you are doing scoring locally, you are probably looking at http://pythonhosted.org/revscoring/ [22:36:26] Zppix, IMO, it would be to have the /ui/ interface produce output that looks like the image I made in that thread. [22:36:27] well, no, If I'm checking stats of a campaign in enwiki, I think only two things are important: global rights, rights in enwiki [22:36:41] everything else is uneeded [22:37:05] Amir1, in this case I'm working from JADE. I want to be able to check on a users' status across all wikis [22:37:30] Maybe I could use oauth to do this. [22:37:30] oh in that case [22:38:04] it'll be difficult. I know it's technically possible (see [[Special:CentralAuth/Dexbot]]) [22:38:04] Good, thank you. Followup Question: there seem to be multiple models inside of "damaging" such as `linear_svc`, which one should I use to reproduce official ORES api result? [22:38:05] 10[2] 04https://meta.wikimedia.org/wiki/Special:CentralAuth/Dexbot [22:38:35] xinbenlv, only one model in enwiki damaging [22:38:49] Can you point me to where you see "linear_svc"? [22:39:10] Halfak k [22:40:36] halfak: it's not possible via API [22:40:48] halfak: note that users have different rights in an OAuth request than in a normal request [22:40:55] writing things like that is also a little bit complex and might be too expensive for prod [22:41:02] or might, depends on which API you use [22:41:03] tgr, not sure what that means. [22:41:15] OAuth is about limiting rights [22:41:20] sure [22:41:35] Our oauth has "identify" [22:41:40] you make an OAuth request as a user -> you have less rights than you would when you make a web request as that user [22:41:51] some APIs will be affected by that [22:42:19] specifically, I think meta=userinfo will be and list=users will not, but would have to check to be sure [22:42:23] tgr, that's not really relevant here. We're just using OAuth to identify users. I'll have a totally uncredentialled user check their status. [22:42:45] oh, sorry, I misunderstood you then [22:43:12] Halfak working on task now [22:43:24] I wonder if we should check someone's rights every time they try to do something that requires rights and not cache anything. [22:43:32] That would be the safest. [22:43:44] In which case, I don't care about querying the local wiki. [22:43:47] halfak: or use short living cache [22:43:55] halfak: 6h? [22:43:57] https://usercontent.irccloud-cdn.com/file/ABXPzkG5/image.png [22:44:02] yeah... was thinking that too. ' [22:44:02] http://pythonhosted.org/revscoring/ [22:44:08] +halfak [22:44:38] xinbenlv, ahh yeah. That's not an example you can run. If you are looking for models to run, check out https://github.com/wiki-ai/editquality/tree/master/models [22:44:56] Regretfully, it makes little sense to keep a working model in the revscoring repo. [22:45:05] You'll note there that there's only one enwiki.damaging model. [22:45:22] halfak: you should check the rights [22:45:32] Platonides, every time? [22:45:36] Yeah, I think you're right. [22:45:44] if someone removes a right from a vandal, and he is able to still continue doing the X bad thing… [22:45:51] well, you will get a lot of complaints ;) [22:45:55] Was just thinking about the same thing. [22:48:45] I'll incorporate that. [22:52:42] OK notes left for future me. I'm going to log off shortly. [22:52:51] halfak: awight: thanks for thinking of me ;) [22:53:03] ggellerman_, :D of course [22:53:48] xinbenlv, good luck with your work. I'll be offline for the US holiday tomorrow and Friday. But I'll be back on Saturday for my "office hours" between 1700 and 2000 UTC [22:53:56] 11AM and 2PM CST (my timezone) [22:54:43] Take care folks. Happy Thanksgiving to those who lean towards turkey holidays. And happy quiet work time for those who don't :) [22:54:44] o/ [22:56:43] You too [22:57:48] Thank you halfak [22:57:55] Glad to know yyou [22:58:03] happy thanksgiving! [23:04:11] fun fact, my phone is about to die because I brought my beard trimmer charger instead of the phone charger [23:04:55] 10Scoring-platform-team, 10ORES, 10User-Zppix: Make ORES' webui when getting results easier to understand - https://phabricator.wikimedia.org/T181201#3782878 (10Zppix) [23:06:03] Lol [23:07:02] 10Scoring-platform-team, 10ORES, 10User-Zppix: Make ORES' webui when getting results easier to understand - https://phabricator.wikimedia.org/T181201#3782878 (10Ladsgroup) This needs at least a fast UI/UX designer look, it doesn't look user friendly. [23:16:15] // Filter out incompatible types; log actions and external rows are not scorable [23:16:15] $conds[] = 'rc_type NOT IN (' . $dbr->makeList( [ RC_LOG, RC_EXTERNAL ] ) . ')'; [23:16:35] is there a reason this isn't rc_type IN ( RC_EDIT, RC_NEW ) ? [23:16:46] whitelist vs blacklist? [23:17:55] I guess it doesn't make a difference [23:43:02] halfak: I spec'ed out the work I need to do to refactor the extension and specially making it more robust in matter of onSave hook to make it usable for all usecases (TM)