[13:37:34] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3952439 (10Bodhisattwa) @awight, all strings are translated now. [13:49:23] o/ awight Amir1 [13:49:31] bonjour [13:49:38] I'm out sick this morning. [13:49:42] hey [13:49:52] I'm finishing the templating fawiki [13:49:59] :( [13:50:09] Do you want to do staff meeting without me or delay until tomorrow? [13:50:19] Sorry to hear it [13:50:37] Thanks. It sucks. I haven't been this sick in a while [13:51:23] 10Scoring-platform-team (Current), 10editquality-modeling, 10Epic, 10artificial-intelligence: [Epic] Edit quality models (damaging/goodfaith) - https://phabricator.wikimedia.org/T130213#3952452 (10Bodhisattwa) [13:51:25] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3952451 (10Bodhisattwa) [13:51:33] You had a pretty bad one a month ago! [13:53:06] I'd take the flu it over this crap. Ugh. [13:53:21] Anyway. What do y'all think about the meeting? [13:55:21] I’m fine with postponing it, if you think you’ll be back in action tomorrow. Usually we have to shelve half the items when you’re absent. [13:58:27] Ok I'll move it. Thanks. [13:59:08] Thanks for thinking of it! [14:04:19] halfak_: akosiaris reimaged ores*, I could go ahead with deploying our wares and testing [14:04:32] Now I will stop pinging. [14:04:42] awight: no, not right now [14:04:57] I am deploying ORES on ores* currently [14:05:08] akosiaris: :D okay thanks for the warning [14:17:14] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3952480 (10Bodhisattwa) [14:17:39] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3952492 (10Bodhisattwa) [14:20:32] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3952499 (10Bodhisattwa) [14:21:11] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3952502 (10Bodhisattwa) [14:24:47] 10Scoring-platform-team (Current), 10editquality-modeling, 10Epic, 10artificial-intelligence: [Epic] Edit quality models (damaging/goodfaith) - https://phabricator.wikimedia.org/T130213#3952504 (10Bodhisattwa) [14:24:50] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3952503 (10Bodhisattwa) [14:26:06] 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3952506 (10awight) [14:39:20] Amir1: I’m still trying to run your selenium patch. Which instructions did you follow to provision mw-vagrant? https://www.mediawiki.org/wiki/Browser_testing/Setup_instructions, or maybe https://www.mediawiki.org/wiki/Selenium/Node.js/Inside_MediaWiki-Vagrant ? [14:40:03] awight: I ran it on my localhost (without vagrant) [14:40:07] aha [14:40:15] :P [14:42:13] wiki-ai/editquality#108 (fawiki_templating - 24c6565 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/338526437 [14:50:25] 10Scoring-platform-team (Current), 10JADE, 10MediaWiki-Vagrant, 10Patch-For-Review: Vagrant role for JADE - https://phabricator.wikimedia.org/T182055#3811422 (10awight) a:03awight [15:03:32] Amir1: I thought I saw a patch go by which swapped Makefile.automated with Makefile [15:13:09] awight: yup [15:13:30] now Makefile is automated and Makefile.manual is ... manual [15:14:10] MarkupSafe-1.0-cp34-cp34m-linux_x86_64.whl is not a supported wheel on this platform. [15:14:14] awight: ^ [15:14:24] and this is blocking the reimage :-( [15:14:40] well the enabling of the ores* hosts, not the reimage itself [15:15:06] I 'll create a task for better tracking [15:15:25] I 've got a number of other problems as well with scap, I 'll upload a patch and document them [15:16:06] I’ll get right on that [15:19:58] thanks [15:21:07] 10Scoring-platform-team, 10ORES, 10Epic: Migrate Scoring services to Debian Stretch - https://phabricator.wikimedia.org/T186238#3952791 (10akosiaris) [15:21:09] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3952792 (10akosiaris) [15:21:12] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3952793 (10akosiaris) [15:21:14] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Make sure ORES is compatible with stretch - https://phabricator.wikimedia.org/T182799#3952787 (10akosiaris) 05Resolved>03Open Reopening since ORES is not currently stretch compatible. For more info see T184135 [15:21:45] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Rebuild ORES wheels on Stretch - https://phabricator.wikimedia.org/T184135#3873517 (10akosiaris) Just tried using the wheel on stretch in the new ORES cluster and I get ``` MarkupSafe-1.0-cp34-cp34m-linux_x86_64.whl is not a supported wheel on... [15:24:13] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501#3726795 (10akosiaris) ORES dsh groups populated via conftool alongside 2 canaries is in https://gerrit.wikimedia.org/r/#/c/408799/. Some changes are required... [15:26:07] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3952800 (10akosiaris) ORES deploy has failed (see T182799#3952787 and T184135#3873517) but the topic of this task, which is to reimage the hosts t... [15:27:05] Looks like we’re just missing a submodule bump. [15:28:22] akosiaris: Can you deploy the ORES repo on branch “stretch_conversion” [15:28:29] halfak_: awight https://etherpad.wikimedia.org/p/JADE_extension [15:28:31] That’s where we’re hiding the updated wheels, etc. [15:28:51] my 2c for the jade extension (stand-alone extension) [15:29:39] fyi the skeleton should be mirrored over to gerrit soon, https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests [15:31:53] Amir1: +1 to everything in that list, except the VE editing interface [15:32:14] Would that work on an arbitrary JSON blob? [15:32:25] Also, I don’t think we need to provide editing. The built-in JSON editor is fine. [15:33:22] I was wondering about the extra patrolling load (patrolling patrollers), and it might be okay as long as there is just one entry per judgment, and they’re being added manually. [15:33:36] When judgments are inserted by automated tools, we’ll have a problem. [15:33:56] I’d like your opinion on Ext:JADE’s role in this diagram, when you have the time: https://docs.google.com/drawings/d/1Lagl0BJWVWHNvHLy5y6RNNKvl0C1tdVrE5YniwgqFJY/edit [15:36:53] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Rebuild ORES wheels on Stretch - https://phabricator.wikimedia.org/T184135#3952838 (10awight) @akosiaris Our Stretch patches are available on the `stretch_conversion` branch. That includes Python 3.5 wheels... [16:01:32] PROBLEM - puppet on ORES-worker03.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:30:58] RECOVERY - puppet on ORES-worker03.experimental is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:44:39] (03CR) 10Awight: [C: 032] "Merging without local smoke testing. The tests are running and succeeding under Jenkins,." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [16:48:12] (03CR) 10jerkins-bot: [V: 04-1] Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [16:50:52] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:23] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [16:51:52] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 848 bytes in 7.538 second response time [16:51:56] That sounds bad [16:52:58] akosiaris: Any recent changes that would have increased the ORES worker count in production? [16:54:42] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 872 bytes in 0.041 second response time [17:03:56] (03CR) 10Awight: [C: 032] "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [17:04:52] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 835 bytes in 3.528 second response time [17:06:37] (03Merged) 10jenkins-bot: Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [17:07:26] (03CR) 10jenkins-bot: Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [17:13:32] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [17:21:03] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 873 bytes in 0.310 second response time [17:23:33] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [17:26:12] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 835 bytes in 3.528 second response time [17:29:22] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:13] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 834 bytes in 4.036 second response time [17:45:27] I’ve reduced the number of workers everywhere. Not ideal, but it seems to be holding steady. [17:47:42] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [17:53:42] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:42] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 849 bytes in 6.545 second response time [18:06:54] 10Scoring-platform-team, 10ORES, 10Operations: Clean up redundant ORES celery_workers defaults - https://phabricator.wikimedia.org/T186734#3953455 (10awight) [18:54:39] (03PS1) 10Ladsgroup: Move Hooks::onRecentChange_save to a dedicated class [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408849 (https://phabricator.wikimedia.org/T181335) [19:19:39] https://ca.wikipedia.org/w/index.php?title=Tema:U69a9ixx05okuizq&topic_showPostId=u721aia2tlktjtfk&fromnotif=1#flow-post-u721aia2tlktjtfk [19:31:04] Amir1: I had some ‘net issues, donno if you saw my responses to your ext-JADE notes? [19:31:30] awight: Hey, I've been looking for you! [19:31:35] :) [19:32:44] 1- I saw them, hadn't time to fully respond but I totally agree with you, I think we can start with not supporting VE and/or if needed, we come up with a plan [19:33:10] 2- I made a patch against the extension, please take a look :D [19:33:20] 3- https://ca.wikipedia.org/w/index.php?title=Tema:U69a9ixx05okuizq&topic_showPostId=u721aia2tlktjtfk&fromnotif=1#flow-post-u721aia2tlktjtfk \o/ [19:36:36] Hehe, is there something I should do for (3)? [19:37:18] Where do I find the patch for (2) [19:46:37] Amir1: ^ [20:30:59] awight: sorry, lied down a little [20:31:14] Good time for that [20:31:35] awight: 3- yup, we should make a phab card (if it's not there) and deploy damaging model [20:31:37] BTW, I’m working on https://www.mediawiki.org/wiki/Wikimedia_Scoring_Platform_team/FY2019, jump in if you have any thoughts. [20:32:26] 2- it's in https://gerrit.wikimedia.org/r/#/c/408849/ [20:32:36] awight: ^ also I take a look at it soon [20:35:55] (03CR) 10Awight: [C: 032] "I like it." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408849 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [20:38:35] (03Merged) 10jenkins-bot: Move Hooks::onRecentChange_save to a dedicated class [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408849 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [20:38:37] 10Scoring-platform-team, 10ORES: Train and test damaging model for Catalan - https://phabricator.wikimedia.org/T186749#3954028 (10awight) [20:39:52] (03CR) 10jenkins-bot: Move Hooks::onRecentChange_save to a dedicated class [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408849 (https://phabricator.wikimedia.org/T181335) (owner: 10Ladsgroup) [20:40:03] Thanks awight ! [20:42:12] Way to knock out our tech debt! [20:52:11] (03PS1) 10Ladsgroup: Add tests for ORES\ORESService::getWikiID [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408872 [20:52:58] awight: ^ :D super fast [20:53:30] (03CR) 10Awight: [C: 032] Add tests for ORES\ORESService::getWikiID [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408872 (owner: 10Ladsgroup) [20:55:45] you rock Adam [20:56:17] (03Merged) 10jenkins-bot: Add tests for ORES\ORESService::getWikiID [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408872 (owner: 10Ladsgroup) [20:57:32] (03CR) 10jenkins-bot: Add tests for ORES\ORESService::getWikiID [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408872 (owner: 10Ladsgroup) [21:33:02] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408909 (owner: 10L10n-bot) [21:43:53] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3954233 (10awight) [21:45:25] gtg [21:51:23] Aargh awight JUST left [21:51:32] Amir1: Do you have any idea why https://gerrit.wikimedia.org/r/#/c/406597/ is breaking? [21:51:39] (03PS3) 10Catrope: Enable ORES filters on RecentChangesLinked [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) (owner: 10Sbisson) [21:52:13] RoanKattouw: I will take a look ASAP [21:52:30] Thanks. I've also rebased it just to check if it's fixed in master already [22:00:14] (03PS1) 10Ladsgroup: Add tests for PopulatedSqlModelLookup [extensions/ORES] - 10https://gerrit.wikimedia.org/r/408935 [22:00:52] RoanKattouw: it passes now, selenium is the browser test which randomly fails [22:04:19] RoanKattouw: btw. It would be great if you write tests for ChangesListHookHandlers: https://doc.wikimedia.org/cover-extensions/ORES/includes/Hooks/ChangesListHooksHandler.php.html [22:13:07] (03CR) 10Catrope: [C: 032] Enable ORES filters on RecentChangesLinked [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) (owner: 10Sbisson) [22:16:04] (03Merged) 10jenkins-bot: Enable ORES filters on RecentChangesLinked [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) (owner: 10Sbisson) [22:17:00] (03CR) 10jenkins-bot: Enable ORES filters on RecentChangesLinked [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) (owner: 10Sbisson)