[06:31:15] PROBLEM - check load on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:32:36] PROBLEM - check disk on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:32:38] PROBLEM - check users on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:34:56] PROBLEM - ssh on ORES-web01.Experimental is CRITICAL: Server answer [06:35:40] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:36:52] RECOVERY - ssh on ORES-web01.Experimental is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [07:24:37] RECOVERY - check disk on ORES-web01.Experimental is OK: DISK OK [07:24:38] RECOVERY - check users on ORES-web01.Experimental is OK: USERS OK - 1 users currently logged in [07:24:53] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:16] RECOVERY - check load on ORES-web01.Experimental is OK: OK - load average: 0.04, 0.07, 0.19 [09:47:55] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:49:19] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.196 second response time https://wikitech.wikimedia.org/wiki/ORES [14:55:35] 10ORES, 10Scoring-platform-team: [Discuss] Future ORES architecture - https://phabricator.wikimedia.org/T226193 (10Halfak) @ottomata & @akosiaris, @Acraze and I had a conversation about future ORES architecture that brought us toward Kafka and [Faust](https://faust.readthedocs.io/en/latest/playbooks/vscelery.h... [15:29:10] 10ORES, 10Scoring-platform-team: [Discuss] Future ORES architecture - https://phabricator.wikimedia.org/T226193 (10Ottomata) For sure! Faust, cool! Hadn't heard of that. Looks a bit like Kafka Streams, but for Python. Would be happy to talk. I know @Nuria has some thoughts about future ML pipeline stuff.... [15:48:56] Thanks ottomata! [15:48:57] :) [15:51:08] :) [15:59:39] Standup time! [15:59:40] Y: Worked on some issues with getting our commits into place for the ORES deploy. I forgot that I had a paper review due today, so I got that squared yesterday. Otherwise, I've been doing some organizing work around the Future of ORES Architecture and attending meetings for Tech Management and Mentorship at the Wikimania Hackathon. [15:59:40] T: I'm going to get back to work on Jade stuff and the opportunity fund request for contracting money. I want the Jade epic task in good shape and I want a draft of the opportunity fund request in Erika's inbox for review. Anything else I should be aware of? Maybe support of the upcoming ORES deploy? [15:59:44] accraze, ^ [16:00:24] cool, my update -- [16:02:05] Y: mostly attempting to get ORES on beta. I think I finally got it up on beta at the end of the day, although not sure since grafana beta cluster shows nothing: https://grafana-labs.wikimedia.org/dashboard/db/ores-beta-cluster?refresh=10s&orgId=1 [16:02:36] http://ores-beta.wmflabs.org/v3/scores/enwiki/?model_info [16:02:40] "Internal server error" [16:02:43] So something went wrong. [16:02:46] also I'm thinking I might not have access to logstash beta (none of my credentials work) [16:04:28] and also can't seem to ssh into the beta deployment either, getting "getpeername failed: Bad file descriptor" [16:05:47] but yeah, today my goal is to get all this figured out in hopes to deploy tomorrow [16:06:22] "[2019-08-06T06:25:42] FileNotFoundError: [Errno 2] No such file or directory: 'submodules/articlequality/models/svwiki.wp10.gradient_boosting.model'" [16:06:32] hmmm [16:07:03] ahhh i might not have properly initialized the submodules on deployment-prep [16:07:32] Aha! [16:07:46] Looks like you didn't run "git submodule update --init" [16:07:54] yup that must have been it [16:08:06] Also, there's a weird thing that has happened. [16:08:06] can I just redeploy? [16:08:13] It appears that we're getting some errors from sklearn. [16:08:20] ohhh interesting [16:08:30] Want to get on a call quick? I'll show you what I mean. [16:08:35] lets do it [16:08:39] kk call when ready [16:23:30] https://github.com/wikimedia/revscoring/pull/446 [16:24:30] accraze, ^ when you are ready [16:31:32] wikimedia/revscoring#1675 (halfak-patch-1 - 6ecff50 : Aaron Halfaker): The build has errored. https://travis-ci.org/wikimedia/revscoring/builds/568452432 [16:41:23] https://github.com/wikimedia/articlequality/pull/90 [16:56:17] T229848 [16:56:18] T229848: ORES deployment, Early August 2019 - https://phabricator.wikimedia.org/T229848 [16:56:46] (03PS1) 10Accraze: Fix revscoring and sklearn dep wheels [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/528518 (https://phabricator.wikimedia.org/T229848) [16:57:27] (03CR) 10Halfak: [C: 03+2] Fix revscoring and sklearn dep wheels [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/528518 (https://phabricator.wikimedia.org/T229848) (owner: 10Accraze) [17:05:43] (03PS1) 10Accraze: Updating articlequality and wheels submodules for revscoring v2.5.2 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/528520 (https://phabricator.wikimedia.org/T229848) [17:06:22] (03CR) 10Halfak: [V: 03+2 C: 03+2] Updating articlequality and wheels submodules for revscoring v2.5.2 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/528520 (https://phabricator.wikimedia.org/T229848) (owner: 10Accraze) [18:14:50] 10Scoring-platform-team, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Update RC Filters for new ORES capacities (July, 2019) - https://phabricator.wikimedia.org/T227094 (10Catrope) It sounds to me like we just need to do the usual threshold configuration, like in the commit Aaron linke... [18:22:32] 10ORES, 10Scoring-platform-team, 10WMF-JobQueue, 10Growth-Team (Current Sprint), and 2 others: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10JTannerWMF) Hey @Etonkovidova can you check to see if this is sti... [18:25:05] 10ORES, 10Scoring-platform-team, 10WMF-JobQueue, 10Growth-Team (Current Sprint), and 2 others: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Catrope) We should check if this still happens in production. The... [18:35:06] o/ accraze [18:35:22] just got back from lunch (actually lunch chores). Anything on fire? [18:36:51] so far things seem good [18:41:34] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, 10Chinese-Sites, and 2 others: Deploy ORES filters for zhwiki - https://phabricator.wikimedia.org/T225562 (10SBisson) ` vagrant@vagrant:/vagrant/mediawiki$ mwscript extensions/ORES/maintenance/ConfigureThreshold... [19:08:24] * accraze -> lunch [19:16:58] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, 10Growth-Team (Current Sprint), 10artificial-intelligence: Update ORES thresholds for nlwiki - https://phabricator.wikimedia.org/T225561 (10SBisson) @Halfak can you confirm the nlwiki models have been updated?... [19:25:15] 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10Growth-Team (Current Sprint), 10Patch-For-Review: Non-overlapping threshholds in ORESModels on lvwiki - https://phabricator.wikimedia.org/T221871 (10SBisson) a:03SBisson [19:35:10] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, 10Growth-Team (Current Sprint), 10artificial-intelligence: Update ORES thresholds for nlwiki - https://phabricator.wikimedia.org/T225561 (10Halfak) Confirmed. The change for nlwiki was minor. [19:50:56] 10Jade, 10Scoring-platform-team, 10Operations, 10TechCom, and 4 others: Deploy Jade extension MVP to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [19:54:44] 10Jade, 10Scoring-platform-team: Implement a fluent API for Jade - https://phabricator.wikimedia.org/T201651 (10Halfak) [19:55:17] 10Jade, 10Scoring-platform-team, 10Documentation: Write full specification for the Jade API - https://phabricator.wikimedia.org/T199834 (10Halfak) [19:55:23] 10Jade, 10Scoring-platform-team (Current), 10Patch-For-Review: Write API contract for Jade - https://phabricator.wikimedia.org/T217904 (10Halfak) [19:55:56] 10Jade, 10Scoring-platform-team: Implement fluent API for Jade entity JSON - https://phabricator.wikimedia.org/T201651 (10Halfak) [19:57:18] 10Jade, 10Scoring-platform-team, 10Documentation: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10Halfak) [19:59:58] 10Jade, 10Scoring-platform-team, 10Epic, 10Goal: Complete Jade documentation - https://phabricator.wikimedia.org/T229967 (10Halfak) [20:01:06] 10Jade, 10Scoring-platform-team, 10Documentation, 10Epic, 10Goal: Write Jade extension documentation - https://phabricator.wikimedia.org/T229968 (10Halfak) [20:01:10] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, 10Growth-Team (Current Sprint), 10artificial-intelligence: Update ORES thresholds for nlwiki - https://phabricator.wikimedia.org/T225561 (10SBisson) 05Open→03Resolved a:03SBisson The thresholds were veri... [20:01:12] 10Scoring-platform-team, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Update RC Filters for new ORES capacities (July, 2019) - https://phabricator.wikimedia.org/T227094 (10SBisson) [20:02:49] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe Jade use-cases - https://phabricator.wikimedia.org/T229969 (10Halfak) [20:03:26] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe Jade use-cases - https://phabricator.wikimedia.org/T229969 (10Halfak) https://www.mediawiki.org/wiki/JADE/Use_cases [20:04:09] 10Jade, 10Scoring-platform-team, 10Documentation: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10Halfak) [20:04:11] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe Jade use-cases - https://phabricator.wikimedia.org/T229969 (10Halfak) [20:07:06] 10Jade, 10Scoring-platform-team, 10Design: Implement Jade Entity UI - https://phabricator.wikimedia.org/T208819 (10Halfak) [20:07:38] 10Jade, 10Scoring-platform-team, 10Design: Implement Jade Entity UI - https://phabricator.wikimedia.org/T208819 (10Halfak) [20:07:41] 10Jade: Jade Implementation: Entry edit mode - https://phabricator.wikimedia.org/T212385 (10Halfak) [20:08:21] 10Jade, 10Scoring-platform-team: Jade Mockups: Entry view mode - https://phabricator.wikimedia.org/T212370 (10Halfak) [20:08:23] 10Jade, 10Scoring-platform-team, 10Design: Jade Mockups: Entry edit mode - https://phabricator.wikimedia.org/T199128 (10Halfak) [20:09:49] 10Jade, 10Scoring-platform-team: Design Jade entity UI - https://phabricator.wikimedia.org/T212370 (10Halfak) [20:10:27] 10Jade, 10Scoring-platform-team: Design Jade entity UI - https://phabricator.wikimedia.org/T212370 (10Halfak) Copied from T199128: I'm not a designer, but I figured that it would help jump-start our design conversation if I took a stab at some layouts. Below are three wireframes that I have worked out to dem... [20:10:55] 10Jade, 10Scoring-platform-team: Design Jade entity UI - https://phabricator.wikimedia.org/T212370 (10Halfak) [20:10:58] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity edit mode - https://phabricator.wikimedia.org/T212374 (10Halfak) [20:12:14] 10Jade, 10Scoring-platform-team: Design Jade entity UI - https://phabricator.wikimedia.org/T212370 (10Halfak) Copied from: T212374 https://docs.google.com/drawings/d/1fP-4Tmn4bDDLWKD_Uu_W0xQNM-QyRc4LDKQTkKNp_fc/edit covers a few different types of edits someone might do. Menu items allow "edit", "details", "... [20:13:53] 10Jade, 10Scoring-platform-team: Investigate MCR support gap for Jade purposes - https://phabricator.wikimedia.org/T204303 (10Halfak) 05Open→03Declined We decided almost a year ago that this wasn't relevant. I'm just doing cleanup now. [20:14:29] 10Jade, 10Scoring-platform-team: Design Jade entity UI - https://phabricator.wikimedia.org/T212370 (10Halfak) [20:14:31] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Wireframes: Entity view mode - https://phabricator.wikimedia.org/T212379 (10Halfak) [20:22:10] 10Jade: Complete Jade user testing - https://phabricator.wikimedia.org/T212373 (10Halfak) 05Open→03Declined Declining for lack of resources. [20:22:21] 10Jade, 10Scoring-platform-team, 10Operations, 10TechCom, and 4 others: Deploy Jade extension MVP to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [20:23:08] 10Jade, 10Scoring-platform-team, 10Design: Implement Jade Entity UI - https://phabricator.wikimedia.org/T208819 (10Halfak) [20:23:10] 10Jade, 10Scoring-platform-team, 10User-Testing, 10Epic: [EPIC] Jade Mockups: Prepare mockups of Jade initial feature set for user testing - https://phabricator.wikimedia.org/T210535 (10Halfak) [20:31:46] 10Jade, 10Scoring-platform-team, 10Epic, 10Goal: Complete Jade documentation - https://phabricator.wikimedia.org/T229967 (10Halfak) [20:31:48] 10Jade, 10Scoring-platform-team, 10Documentation: Develop "local" documentation for wikis where Jade will be deployed - https://phabricator.wikimedia.org/T199519 (10Halfak) [20:32:59] 10Jade, 10Scoring-platform-team, 10Documentation: Write user-facing documentation for Jade - https://phabricator.wikimedia.org/T199519 (10Halfak) [20:33:31] 10Jade, 10Scoring-platform-team, 10Documentation: Write user-facing documentation for Jade - https://phabricator.wikimedia.org/T199519 (10Halfak) [20:33:33] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Write glossary of JADE concepts - https://phabricator.wikimedia.org/T204905 (10Halfak) [20:33:56] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Write glossary of JADE concepts - https://phabricator.wikimedia.org/T204905 (10Halfak) a:05awight→03Halfak [20:34:48] 10Jade, 10Scoring-platform-team, 10Documentation: Write user-facing documentation for Jade - https://phabricator.wikimedia.org/T199519 (10Halfak) [20:35:24] 10Jade, 10Scoring-platform-team, 10Documentation, 10Epic, 10Goal: Write Jade extension documentation - https://phabricator.wikimedia.org/T229968 (10Halfak) [20:36:25] 10Jade, 10Scoring-platform-team, 10Documentation: Write user-facing documentation for Jade - https://phabricator.wikimedia.org/T199519 (10Halfak) [20:36:29] 10Jade, 10Scoring-platform-team, 10Documentation, 10Epic: [Epic] Write user-facing documentation for Jade - https://phabricator.wikimedia.org/T199069 (10Halfak) [20:38:55] 10Jade, 10Scoring-platform-team, 10Advanced-Search, 10Discovery-Search, and 2 others: Implement search indexing for Jade entity pages - https://phabricator.wikimedia.org/T206352 (10Halfak) [20:40:42] 10Jade, 10Scoring-platform-team, 10Epic: Implement Jade Entity pages - https://phabricator.wikimedia.org/T229973 (10Halfak) [20:40:59] 10Jade, 10Scoring-platform-team, 10Epic: Implement Jade Entity pages - https://phabricator.wikimedia.org/T229973 (10Halfak) [20:41:01] 10Jade, 10Scoring-platform-team, 10Design: Implement Jade Entity UI - https://phabricator.wikimedia.org/T208819 (10Halfak) [20:41:17] 10Jade, 10Scoring-platform-team, 10Epic: Implement Jade Entity pages - https://phabricator.wikimedia.org/T229973 (10Halfak) [20:41:19] 10Jade, 10Scoring-platform-team, 10Advanced-Search, 10Discovery-Search, and 2 others: Implement search indexing for Jade entity pages - https://phabricator.wikimedia.org/T206352 (10Halfak) [20:42:06] 10Jade, 10Scoring-platform-team, 10Epic: Implement Jade Entity pages - https://phabricator.wikimedia.org/T229973 (10Halfak) [20:42:08] 10Jade, 10Scoring-platform-team: Simplify and extend JADE content schema (based on API spec insights) - https://phabricator.wikimedia.org/T229355 (10Halfak) [20:43:41] 10Jade, 10Scoring-platform-team: [Epic] Align terminology and user experience in Jade code with Jade pitch - https://phabricator.wikimedia.org/T212178 (10Halfak) 05Open→03Resolved a:03Halfak [20:43:43] 10Jade, 10Scoring-platform-team (Current): Explore alternative names for Jade data - https://phabricator.wikimedia.org/T200365 (10Halfak) [20:43:48] 10Jade, 10Scoring-platform-team, 10Operations, 10TechCom, and 4 others: Deploy Jade extension MVP to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [20:45:09] 10Jade, 10Scoring-platform-team, 10Design: Jade Implementation: Watchlist integration - https://phabricator.wikimedia.org/T201361 (10Halfak) 05Open→03Declined It's not clear that this feature is important at this time. [20:45:14] 10Jade, 10Scoring-platform-team, 10Operations, 10TechCom, and 4 others: Deploy Jade extension MVP to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [20:45:56] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Watchlist integration - https://phabricator.wikimedia.org/T210560 (10Halfak) 05Open→03Declined See parent task [20:45:58] 10Jade, 10Scoring-platform-team, 10User-Testing, 10Epic: [EPIC] Jade Mockups: Prepare mockups of Jade initial feature set for user testing - https://phabricator.wikimedia.org/T210535 (10Halfak) [20:46:01] 10Jade, 10Scoring-platform-team, 10Design: Jade Implementation: Watchlist integration - https://phabricator.wikimedia.org/T201361 (10Halfak) [20:46:34] 10Jade, 10Scoring-platform-team, 10User-Testing, 10Design: Jade Wireframes: Watchlist integration - https://phabricator.wikimedia.org/T212378 (10Halfak) 05Open→03Declined [20:46:37] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Watchlist integration - https://phabricator.wikimedia.org/T210560 (10Halfak) [20:47:03] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe Jade use-cases - https://phabricator.wikimedia.org/T229969 (10Halfak) 05Open→03Resolved [20:47:05] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe "catchall" endpoint for Jade - https://phabricator.wikimedia.org/T229362 (10Halfak) [20:47:07] 10Jade, 10Scoring-platform-team, 10Documentation: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10Halfak) [20:47:55] 10Jade, 10Scoring-platform-team, 10User-Testing: Create test fixtures for Jade user testing - https://phabricator.wikimedia.org/T210562 (10Halfak) 05Open→03Declined We'll be testing this with real data. [20:47:57] 10Jade, 10Scoring-platform-team, 10User-Testing, 10Epic: [EPIC] Jade Mockups: Prepare mockups of Jade initial feature set for user testing - https://phabricator.wikimedia.org/T210535 (10Halfak) [20:50:55] 10Jade, 10Scoring-platform-team, 10Epic: Impement secondary Jade Integrations - https://phabricator.wikimedia.org/T229974 (10Halfak) [20:53:33] 10Jade, 10Scoring-platform-team, 10Epic: Implement New Filters integration (RecentChanges/Watchlist) - https://phabricator.wikimedia.org/T229975 (10Halfak) [20:53:54] 10Jade, 10Scoring-platform-team, 10Epic: Design New Filters controls for Jade - https://phabricator.wikimedia.org/T229976 (10Halfak) [20:54:47] 10Jade, 10Scoring-platform-team, 10Epic: Implement New Filters integration (RecentChanges/Watchlist) - https://phabricator.wikimedia.org/T229975 (10Halfak) [20:54:50] 10Jade, 10Scoring-platform-team, 10Epic: Design New Filters controls for Jade - https://phabricator.wikimedia.org/T229976 (10Halfak) [20:58:15] 10Jade, 10Scoring-platform-team, 10Epic: Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak) [20:59:06] 10Jade, 10Scoring-platform-team, 10Epic: Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak) See https://phabricator.wikimedia.org/search/query/zFcopzzmzOG./#R for a bunch of commits representing work that had already been done to develop t... [20:59:40] 10Jade, 10Scoring-platform-team, 10Epic: Implement New Filters integration (RecentChanges/Watchlist) - https://phabricator.wikimedia.org/T229975 (10Halfak) [20:59:42] 10Jade, 10Scoring-platform-team, 10Epic: Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak) [20:59:53] 10Jade, 10Scoring-platform-team, 10Epic: Impement secondary Jade Integrations - https://phabricator.wikimedia.org/T229974 (10Halfak) [20:59:55] 10Jade, 10Scoring-platform-team, 10Epic: Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak)