[08:18:31] In the future development of AI might we have to follow the path that c created unix and create the programming language in order to create the operating system? [08:19:19] Perhaps a true AI would be a entire os? [11:44:33] hello [12:10:00] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: Scream properly when threshold stats fetch is failing - https://phabricator.wikimedia.org/T180587#3762568 (10Ladsgroup) [12:10:03] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: Scream properly when threshold stats fetch is failing - https://phabricator.wikimedia.org/T180589#3762596 (10Ladsgroup) [12:10:14] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: Scream properly when threshold stats fetch is failing - https://phabricator.wikimedia.org/T180589#3762611 (10Ladsgroup) 05Open>03Invalid [13:02:34] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Add link to FAQ to ORES homepage - https://phabricator.wikimedia.org/T179629#3762794 (10Ladsgroup) a:03Ladsgroup [13:03:43] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Add link to FAQ to ORES homepage - https://phabricator.wikimedia.org/T179629#3762804 (10Ladsgroup) https://github.com/wiki-ai/ores/pull/234 [13:05:00] wiki-ai/ores#835 (faq - 4eb7af1 : Amir Sarabadani): The build failed. https://travis-ci.org/wiki-ai/ores/builds/302470214 [13:29:15] 10Scoring-platform-team (Current), 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Article quality campaign for Persian Wikipedia - https://phabricator.wikimedia.org/T174684#3762881 (10Ladsgroup) I looked into the good and featured articles in Persian Wikipedia and the wiki has [[ht... [14:59:41] (03PS1) 10Awight: Put all the wheel tools into the Makefile [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) [15:26:41] awight: one thing i came up with for our deploy was shipping all the wheels to archiva (archiva.wikimedia.org) and using git-fat, instead of commiting the binaries to the repo. Removes the need for an extra wheels repo. Not sure if you would want to do that in ores too [15:27:07] otherwise our deploy ended up looking pretty similar to ores. thanks for the pointers! [15:27:24] very nice! We’re blocked on git-fat unfortunately, cos we need to do both production and labs deployments using the same repos. [15:27:30] Is that necessary for archiva? [15:27:56] awight: you can pull from archiva from labs (or locally, or whatever) it's just http [15:27:59] Cos pushing wheels of every version and letting pip pick the right one according to a frozen requirements list would be a big win [15:28:07] neat. [15:28:18] Have a link to what you did? [15:28:45] awight: the wheels still have to be "commited" to the repo, its not like a pypi repo, but the commit is just a sha1 hash that it uses to fetch the binary. yup lemme find how to link to a file in our repos that arn't on github yet [15:29:01] haha no rush [15:29:06] I’ll just make a placeholder task... [15:29:31] awight: well, i guess the gerrit link works. This is the script for uploading: https://gerrit.wikimedia.org/r/#/c/388345/10/upload_wheels.py [15:30:05] awight: it does require an archiva password though, which is sadly shared (otto has some multi-year old TODO about getting archiva+ldap working) [15:30:18] hahaha [15:30:24] that might not fly on labs, either [15:30:31] Unless that’s just for upload? [15:30:43] awight: that script is just for the upload. download requires no password [15:30:50] excellent [15:31:23] 10Scoring-platform-team: Investigate using archiva.wmo to host our wheels - https://phabricator.wikimedia.org/T180606#3763224 (10awight) [16:44:49] RoanKattouw: Can you remind me what magic URL param I use to get the new RC filters? [16:47:30] I thought it was the default now... [16:51:47] Amir1: ^ Do you happen to know? [16:56:09] All good, I got it to show up using the $wg vars [17:02:14] 10Scoring-platform-team: Switch to a non-WMF hangout for the staff meetings so we don't have access granting foolishness - https://phabricator.wikimedia.org/T180613#3763513 (10awight) [17:27:37] awight: the patch for vagrant is +2'd [17:28:00] Amir1: oops—this one is required first, https://github.com/wiki-ai/ores/tree/revid_stats [17:28:09] https://github.com/wiki-ai/ores/pull/232 [17:31:00] awight: two notes there :D [17:31:26] Argh, that branch is messy [17:31:29] DNMs all over [18:02:05] Amir1: The branch is ready for re-review FYI [18:27:44] awight: ?rcfilters=1 generally works [18:28:04] RoanKattouw: thanks for that! I’ll put it somewhere safe this time. [18:28:50] I was able to get the new interface using config vars, btw. [18:31:13] Cool [18:55:58] wiki-ai/ores#841 (master - 23ec45a : Amir Sarabadani): The build was fixed. https://travis-ci.org/wiki-ai/ores/builds/302638615 [19:04:44] Nice! Relocating, biab. [20:09:09] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Gender-Support, 10I18n: “Would you revert this edit?” in Revision Scoring should support GENDER - https://phabricator.wikimedia.org/T132204#3764190 (10MarcoAurelio) [20:12:38] awight: You said something today in SoS about thresholds on beta right? [20:13:00] awight: Elena is asking me why ORES filters don't appear on RCFilters in beta labs, and usually that's because the E:ORES code fails to fetch thresholds [20:13:05] RoanKattouw: Revscoring 2 thresholds have been available on beta for a while [20:13:09] Hum [20:13:22] Sorry, I’ve seen that bug go by but hadn’t dug in yet. [20:13:44] What I mentioned today was that we’ve merged patches that make the thresholds available in mw-vagrant. [20:13:52] and they will be used by Ext:PRES [20:13:54] *ORES [20:14:45] Oh right [20:16:46] hah here we go: [20:16:49] > $stats = ORES\Stats::newFromGlobalState(); [20:16:50] > var_dump($stats->getThresholds('damaging')); [20:16:50] array(0) { [20:16:50] } [20:17:13] O_o [20:17:16] I'll find the task and put it on there for posterity [20:17:19] TY [20:17:29] after this Research Showcase I’ll check that out [20:28:12] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764583 (10Catrope) [20:28:43] awight: I couldn't find the/an existing task, so I created a new one --^^ please dupe if there is a pre-existing task that I missed [21:00:45] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764712 (10awight) p:05Triage>03High [21:00:58] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764583 (10awight) [21:20:29] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764583 (10awight) > Cannot process your request because the server is overloaded. Try again in afew minute... [21:25:21] PROBLEM - puppet on ores-worker-08 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 11 minutes ago with 0 failures [21:25:40] PROBLEM - puppet on ores-worker-05 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 22 minutes ago with 0 failures [21:26:21] PROBLEM - puppet on ores-web-05 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 1 minute ago with 0 failures [21:26:29] PROBLEM - puppet on ores-worker-07 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 7 minutes ago with 0 failures [21:26:39] PROBLEM - puppet on ores-worker-09 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 19 minutes ago with 0 failures [21:27:24] PROBLEM - puppet on ores-web-03 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 10 minutes ago with 0 failures [21:27:38] PROBLEM - puppet on ores-lb-02 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 3 minutes ago with 0 failures [21:28:24] PROBLEM - puppet on ores-redis-01 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 24 minutes ago with 0 failures [21:28:34] PROBLEM - puppet on ores-worker-06 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 16 minutes ago with 0 failures [21:29:11] PROBLEM - puppet on ores-worker-10 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 24 minutes ago with 0 failures [21:29:20] PROBLEM - puppet on ores-redis-02 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 27 minutes ago with 0 failures [21:29:32] wassup [21:33:00] awight [21:21:09] !log Disabling puppet across cloud VPS through cumin on labpuppetmaster1001 [21:33:55] hey, yes - i'm doing some maintenance [21:34:08] :D thanks! [21:34:38] will let you know when i'm done :) (it should automatically come back with OK) [21:35:25] madhuvishy: Don’t worry about notifying, it looks harmless! [21:59:25] awight: around? [21:59:36] Amir1: hi! [21:59:46] I’m on for another hour, fwiw [21:59:59] awight: okay, how can I help (I got afk for dinner) [22:00:09] for anything related to the ores [22:00:30] I’m looking at https://phabricator.wikimedia.org/T180633 [22:02:02] Amir1: Feel like CR’ing T179602 ? [22:02:03] T179602: Rewind revscoring 1/2 compatibility hacks - https://phabricator.wikimedia.org/T179602 [22:02:07] awight: I don't want to step on your toes [22:03:01] There’s also debugging T179830 now that we can do thresholds from mw-vagrant [22:03:01] T179830: Notice: Undefined property: stdClass::$ores_damaging_threshold in /srv/mediawiki/php-1.31.0-wmf.6/extensions/ORES/includes/Hooks.php on line 602 - https://phabricator.wikimedia.org/T179830 [22:03:13] (03CR) 10Ladsgroup: [C: 032] Revert "Fallback to old thresholds API as necessary (take 2)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383667 (https://phabricator.wikimedia.org/T179602) (owner: 10Awight) [22:03:28] {{merged}} that was easy [22:03:29] 10[1] 04https://meta.wikimedia.org/wiki/Template:merged [22:03:36] lol that wasn’t enuf to keep you busy I see [22:03:48] It needs smoke testing... [22:03:57] or we can just let it hit beta, I guess [22:04:09] regarding the other one, RoanKattouw made the patch and I added comments there [22:04:33] I'm not sure if I can/should take over [22:04:58] awight: yeah, will test it in beta also we have two weeks (next week train won't go) [22:05:22] PROBLEM - puppet on ores-worker-08 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 50 minutes ago with 0 failures [22:05:31] (03Merged) 10jenkins-bot: Revert "Fallback to old thresholds API as necessary (take 2)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/383667 (https://phabricator.wikimedia.org/T179602) (owner: 10Awight) [22:05:42] PROBLEM - puppet on ores-worker-05 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 1 hour ago with 0 failures [22:06:22] PROBLEM - puppet on ores-web-05 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 41 minutes ago with 0 failures [22:06:24] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764923 (10awight) I restarted the servers, and see that something is making requests for the old-style stat... [22:06:32] PROBLEM - puppet on ores-worker-07 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 46 minutes ago with 0 failures [22:06:42] PROBLEM - puppet on ores-worker-09 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 59 minutes ago with 0 failures [22:07:24] Amir1: If you want a bigger project, I would be happy to hand off the Celery 4 branch for as long as you’re interested. [22:07:27] PROBLEM - puppet on ores-web-03 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 50 minutes ago with 0 failures [22:07:42] PROBLEM - puppet on ores-lb-02 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 42 minutes ago with 0 failures [22:08:07] It gives timeouts locally 100% of the time now. I can catch you up on what I’ve done so far, but it’s pretty clear from reading the ores #CELERY_4 branch. [22:08:10] well it looks scary but I would be happy to give it a try, it helps me understand the magic behind celery (I hope) [22:08:18] :D [22:08:27] PROBLEM - puppet on ores-redis-01 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 1 hour ago with 0 failures [22:08:37] PROBLEM - puppet on ores-worker-06 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 55 minutes ago with 0 failures [22:08:49] awight: how I can reproduce it? the PR of celery 4 in ores? [22:09:12] PROBLEM - puppet on ores-worker-10 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 1 hour ago with 0 failures [22:09:20] Pull ores-prod-deploy#CELERY_4 and update submodules [22:09:22] PROBLEM - puppet on ores-redis-02 is WARNING: WARNING: Puppet is currently disabled, message: Testing Deny user auth if /etc/block-ldap-key-lookup exists patch, last run 1 hour ago with 0 failures [22:09:25] run that locally... [22:09:34] *if* you want to make the same mistakes I have been :) [22:10:20] I never checked out that repo, it'll be huge I guess [22:10:26] let me try [22:10:36] also I have this for you https://github.com/wiki-ai/ores/pull/234 [22:10:36] hmm it’s small on its own [22:10:51] you can clone the submodules from your local checkouts if you wanna save bandwidth [22:11:00] Amir1: cool, looking now. [22:11:03] RECOVERY - puppet on ores-worker-08 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:11:06] yeah [22:11:59] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:12:03] RECOVERY - puppet on ores-web-05 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:12:10] RECOVERY - puppet on ores-worker-05 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:12:10] RECOVERY - puppet on ores-worker-09 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:12:36] Amir1: If you see a “correlation-id” error, then you haven’t installed the Celery 4 wheel yet... [22:13:03] RECOVERY - puppet on ores-web-03 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:13:08] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:13:54] RECOVERY - puppet on ores-redis-01 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:14:10] awight: thanks [22:14:41] RECOVERY - puppet on ores-worker-10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:14:50] RECOVERY - puppet on ores-redis-02 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:15:05] RECOVERY - puppet on ores-worker-06 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:16:06] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764946 (10awight) Now that we're caching for 1 day again, I'm afraid the extension isn't gonna re-request.... [22:21:18] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta labs, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764949 (10awight) So, now that I've restarted services, fetching thresholds is currently working, but the e... [22:26:00] I’m signing out. Amir1 you good? [22:26:15] * awight glances suspiciously at a world clock :p [22:26:17] awight: yeah, trying to see how I can get it working [22:26:46] Do you have halfak’s mystical 99-local.yaml? [22:27:05] nope [22:27:39] https://gist.github.com/adamwight/56393d2f2340594cc53d59f6ef0270e4 [22:27:51] it goes in ores-prod-deploy/config/ [22:34:10] Once you have that, Redis running, and a virttualenv with the submodules/wheels installed, all it should take is two shells with “python ores_celery.py” and “python ores_wsgi.py" [22:35:21] okay [22:35:31] I will spend some time on it tomorrow morning [22:35:36] I probably go to bed now [22:39:14] bye! [23:49:09] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta cluster, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3765152 (10greg) [23:49:59] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Beta-Cluster-reproducible: ORES RC filters missing in beta cluster, fetching thresholds fails - https://phabricator.wikimedia.org/T180633#3764583 (10greg) [offtopic https://wikitech.wikimedia.org/wiki/Labs_labs_labs ]