[00:57:26] 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Have ORES-web[01,02] been removed? - https://phabricator.wikimedia.org/T232921 (10Zppix) 05Open→03Resolved a:03Zppix [06:25:29] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:58:50] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) [08:59:33] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) [09:00:07] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Advanced-Search, 10CirrusSearch, and 8 others: Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days - https://phabricator.wikimedia.org/T232608 (10zeljkofilipin) 05Open→03Resolved selenium-daily-beta-CirrusSearch is... [11:13:15] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10Puppet: Require git-lfs in ORES hosts - https://phabricator.wikimedia.org/T232494 (10akosiaris) @Halfak, I 've uploaded a different proposal in a new PS. Already makes jenkins happier. [11:30:06] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:27:05] o/ [13:32:25] Aha! I need to silence ^ [13:46:45] Just submitted a patchset for updating our list of nodes. [13:52:53] halfak: it appears the new web-* checks on icinga2 are working as expected :) lmk if i need to add any more hosts [13:53:21] Thanks Zppix. Hopefully, we'll be calm for a little while :) [13:53:52] halfak: I hope you guys figure out how to fix you guys getting hit hard [13:54:05] I think we're in a pretty good place right now. [13:54:50] halfak: I hear you can download ram so maybe that would help :P jk [13:55:20] https://downloadmoreram.com/ [13:56:19] the fact someone went through the trouble to code that website is what i love about the internet halfak [13:56:31] Agreed. [14:52:33] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review: Investigate intermittent delay for basic uwsgi requests. - https://phabricator.wikimedia.org/T232228 (10Halfak) 05Open→03Resolved It looks like the solution here is to reduce memory pressure on the machines. We've started up 3 new VMs with m... [14:53:30] 10Scoring-platform-team (Current): Add more capacity to ORES wmflabs cluster - https://phabricator.wikimedia.org/T233112 (10Halfak) [14:53:59] 10Scoring-platform-team (Current): Add more capacity to ORES wmflabs cluster - https://phabricator.wikimedia.org/T233112 (10Halfak) Started up ores-worker-03 and ores-worker-04. They seem to be working as intended. Load is more reasonable now. [14:54:08] 10Scoring-platform-team (Current): Add more capacity to ORES wmflabs cluster - https://phabricator.wikimedia.org/T233112 (10Halfak) 05Open→03Resolved [14:54:22] 10ORES, 10Scoring-platform-team (Current): Add more capacity to ORES wmflabs cluster - https://phabricator.wikimedia.org/T233112 (10Halfak) [14:55:15] 10ORES, 10Scoring-platform-team (Current): Address high wait state on ores-redis-02 - https://phabricator.wikimedia.org/T233113 (10Halfak) [14:56:02] 10ORES, 10Scoring-platform-team (Current): Address high wait state on ores-redis-02 - https://phabricator.wikimedia.org/T233113 (10Halfak) a:03Halfak I worked with @Andrew to move the VM to a new host. It looks like the old host has another VM that is taking up a ton of resources. The issue seems to be res... [14:56:11] 10ORES, 10Scoring-platform-team (Current): Address high wait state on ores-redis-02 - https://phabricator.wikimedia.org/T233113 (10Halfak) 05Open→03Resolved [14:56:31] SPAM TIME [14:56:38] 10ORES, 10Scoring-platform-team (Current): ORES deploy early Sept. 2019 - https://phabricator.wikimedia.org/T232660 (10Halfak) 05Open→03Resolved [14:56:40] 10ORES, 10Scoring-platform-team (Current): POST method not allowed for some ORES endpoints - https://phabricator.wikimedia.org/T232500 (10Halfak) 05Open→03Resolved [14:56:42] 10ORES, 10Scoring-platform-team (Current): ORES deploy early Sept. 2019 - https://phabricator.wikimedia.org/T232660 (10Halfak) [14:56:44] 10Jade, 10Scoring-platform-team (Current): Fix jade role in mediawiki vagrant - https://phabricator.wikimedia.org/T231497 (10Halfak) 05Open→03Resolved [14:56:46] 10ORES, 10Scoring-platform-team (Current): ORES is creating a lot of metrics. This is due to revid count-based metrics - https://phabricator.wikimedia.org/T232164 (10Halfak) 05Open→03Resolved [14:56:48] 10ORES, 10Scoring-platform-team (Current): ORES deploy early Sept. 2019 - https://phabricator.wikimedia.org/T232660 (10Halfak) [14:56:50] 10ORES, 10Scoring-platform-team (Current): Feature injection doesn't work when using "?revids=" param - https://phabricator.wikimedia.org/T232143 (10Halfak) 05Open→03Resolved [14:56:52] 10ORES, 10Scoring-platform-team (Current): ORES deploy early Sept. 2019 - https://phabricator.wikimedia.org/T232660 (10Halfak) [14:56:54] 10Scoring-platform-team (Current): Automate docs build for model repos - https://phabricator.wikimedia.org/T230517 (10Halfak) 05Open→03Resolved [14:56:56] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Generate dataset for MSFT explainability experiements - https://phabricator.wikimedia.org/T230737 (10Halfak) 05Open→03Resolved a:03Halfak [14:57:00] 10Scoring-platform-team (Current): Develop automated release strategy from travis CI - https://phabricator.wikimedia.org/T229850 (10Halfak) 05Open→03Resolved [14:57:03] 10Scoring-platform-team (Current), 10articlequality-modeling, 10draftquality-modeling, 10drafttopic-modeling, and 2 others: Implement versioning in the model repos - https://phabricator.wikimedia.org/T229248 (10Halfak) [14:57:05] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Describe "catchall" endpoint for Jade - https://phabricator.wikimedia.org/T229362 (10Halfak) 05Open→03Resolved [14:57:09] 10Scoring-platform-team (Current), 10articlequality-modeling, 10draftquality-modeling, 10drafttopic-modeling, and 2 others: Implement versioning in the model repos - https://phabricator.wikimedia.org/T229248 (10Halfak) 05Open→03Resolved [14:57:11] 10Jade, 10Scoring-platform-team (Current), 10Documentation, 10Patch-For-Review: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10Halfak) [14:58:25] 10ORES, 10Scoring-platform-team (Current), 10User-Zppix: Disable check_load on ORES wmflabs celery workers - https://phabricator.wikimedia.org/T233114 (10Halfak) [14:58:28] 10ORES, 10Scoring-platform-team (Current), 10User-Zppix: Disable check_load on ORES wmflabs celery workers - https://phabricator.wikimedia.org/T233114 (10Halfak) a:03Zppix [15:12:22] halfak: {{done}} [15:12:22] How cool, Zppix! [15:12:30] \o/ awesome. Thank you. [15:12:41] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Zppix: Disable check_load on ORES wmflabs celery workers - https://phabricator.wikimedia.org/T233114 (10Zppix) 05Open→03Resolved [15:12:43] np [15:56:13] wikimedia/revscoring#1718 (session_orientation - 6f6092c : halfak): The build was fixed. https://travis-ci.org/wikimedia/revscoring/builds/586128216 [15:56:19] Damn right [16:00:48] Async time! [16:10:01] Ok Here's my update: [16:10:02] Y: Mostly Jade stuff, finished UpdateProposal and DeleteProposal. Fixed some weird bugs with the title helper. Also appened formatversion=2 to all example urls. [16:10:03] T: Work on all Jade endorsement actions (update/delete/move/endorse). Maybe look at revscoring docs pr again. [16:13:55] Y: Worked on no-op warnings for Jade product docs. Finished off the majority of the ores-labs overhaul. Seems to be running well now. Reached out to Kevin who has accepted our offer :). Did a follow-up review for ToSC. Emailed researchers about using feature injection with deleted text. Reviewed most Docs CI PRs. Iterated on git::lfs puppet code. [16:13:55] T: Working on cleaning up ORES-labs icinga warnings. Cleaning up session orientation PR. Attending a ton of meetings about OKRs and product strategy. Organizing travel for an invited talk at the University of Washington in November. [16:29:38] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Address NLTK vulnerability in revscoring - https://phabricator.wikimedia.org/T233131 (10Halfak) [16:29:49] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Address NLTK vulnerability in revscoring - https://phabricator.wikimedia.org/T233131 (10Halfak) a:03Halfak https://github.com/wikimedia/revscoring/pull/453 [16:32:37] wikimedia/revscoring#1720 (bump_nltk - 5ec545b : halfak): The build passed. https://travis-ci.org/wikimedia/revscoring/builds/586142859 [16:34:10] ^ merged it halfak [17:02:17] brb gonna take a quick run before the weather gets bad [17:21:34] Thanks accraze|afk [17:21:41] Forgot to mark myself as at lunch [17:27:14] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Address NLTK vulnerability in revscoring - https://phabricator.wikimedia.org/T233131 (10Halfak) 05Open→03Resolved [21:10:08] 10ORES, 10Scoring-platform-team, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-GoranSMilovanovic: track quality of all/top 10000 Wikidata items over time - https://phabricator.wikimedia.org/T195702 (10GoranSMilovanovic) Status: - working on analytics/visualizations now; - next steps: dashboard. [22:21:05] o/ [23:17:05] so when there isn't a specific threshhold specified for a given wiki in InitialiseSettings.php, I see comments that suggest that threshholds used will be default, but I don't know where the default thershholds are set [23:17:09] e.g. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/444015