[00:18:37] (03PS8) 10Sbisson: Hide RC/WL related preferences as appropriate [extensions/ORES] - 10https://gerrit.wikimedia.org/r/377377 (https://phabricator.wikimedia.org/T175611) (owner: 10Catrope) [00:26:09] (03CR) 10Sbisson: [C: 032] Hide RC/WL related preferences as appropriate [extensions/ORES] - 10https://gerrit.wikimedia.org/r/377377 (https://phabricator.wikimedia.org/T175611) (owner: 10Catrope) [00:27:44] (03Merged) 10jenkins-bot: Hide RC/WL related preferences as appropriate [extensions/ORES] - 10https://gerrit.wikimedia.org/r/377377 (https://phabricator.wikimedia.org/T175611) (owner: 10Catrope) [00:28:53] Zppix: That looks buggy. [00:32:11] Web search isn’t helping much, either. [00:33:04] #wikimedia-dev seems to be the official help channel for mw-vagrant, maybe someone there will have an idea? [00:33:30] Or have you solved it already? [00:33:39] It’s something… worse than PATH problems. [00:33:50] *.exe\.. is not a thing [00:34:05] What is your system path? [00:34:31] Did the installer prompt you with any custom directory questions? [02:35:07] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017), 10Schema-change: Index on oresc_probability, temporarily or permanently - https://phabricator.wikimedia.org/T175778#3602993 (10Mattflaschen-WMF) [02:40:04] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017), 10Schema-change: Index on oresc_probability, temporarily or permanently - https://phabricator.wikimedia.org/T175778#3603010 (10jcrespo) Not directly related... [04:06:03] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Store wp10 predictions in page_props table. - https://phabricator.wikimedia.org/T175757#3602360 (10Legoktm) Page props are generally things that are extracted out of wikitext/parsing process. I don't think things generated by ORES fit th... [04:08:22] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Store wp10 predictions in page_props table. - https://phabricator.wikimedia.org/T175757#3603265 (10Legoktm) More specifically, page properties are set by calling `ParserOutput::setProperty()` during the parse process. I'm not sure how we... [14:03:31] 10Scoring-platform-team (Current), 10ORES, 10Operations: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604437 (10Joe) I would suggest AGAINST giving access to all logs. We should have a `tail-ores` command that specifically tails the ores logs, like we d... [15:57:02] 10Scoring-platform-team (Current), 10ORES, 10Operations: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604828 (10awight) @Joe Great, thanks for the pointer! I can put together that tail wrapper shortly, but in the meantime I'm working on T169586 which w... [16:01:35] Amir1: lmk what you think about the staff meeting, I’m fine having or not [16:02:17] awight: I don't have any updates, I think we should skip it and do something else [16:02:53] Amir1: cool, ty [16:03:03] You won’t have to listen to me slurp breakfast ;-) [16:08:38] :D [16:15:59] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604853 (10akosiaris) The above gerrit change should make the main.log files readable by everyone and should fix the issue while a... [16:19:19] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604877 (10awight) @akosiaris Could you point me to the config that propagates those log messages to logstash? I was looking at e... [16:32:37] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604901 (10akosiaris) That would be https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/service/m... [16:36:27] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604913 (10awight) @akosiaris Thanks, I see the messages in logstash now! I'll tweak the ORES dashboard to include them. [16:38:54] 10Scoring-platform-team, 10ORES, 10Wikimedia-Logstash, 10Patch-For-Review: Send celery logs and events to logstash - https://phabricator.wikimedia.org/T169586#3604923 (10awight) Looks like we already have messages going to logstash, per T175736#3604901. Confirm that celery logs are included, before closin... [16:47:07] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Give ores admins read access to /srv/log/ores/main.log* - https://phabricator.wikimedia.org/T175736#3604957 (10awight) Correct to my last comment—the logs are already in logstash. [17:07:37] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3391572 (10awight) Random tool which seems to give a good estimate of RSS not including copy-on-write pages from the parent process:... [17:31:29] awight: i cannot setup vagrant i was up til like 4am trying to [17:31:44] i have to do it w/o vagrant [17:33:38] Zppix: What went wrong? 64-bit problems? [17:33:48] that issue i said last night [17:33:57] i legit tried everything [17:34:04] Do what you gotta do, but it’s probably not going to be easier on “bare metal”. [17:34:12] sorry, which issue? [17:34:21] awight: '"C:\HashiCorp\Vagrant\bin\vagrant.exe\..\..\embedded\bin\ruby.exe"' is not recognized as an internal or external command, operable program or batch file. [17:34:44] for the install i did everything by default [17:34:49] didnt custom anything [17:36:01] Too bad! [17:36:18] You could create a WMF labs instance if you want to try a VPS [17:36:53] Being the first one to provision ORES and Wiki Labels on Windows is not gonna be fun, however [17:37:35] I guess i could ill talk to cloud and ask if i could, then ill proceed from there... [17:41:05] awight: https://phabricator.wikimedia.org/T175846 [17:53:08] Zppix: This is old news, but thread suggests a “repair install”, http://vagrant.1086180.n5.nabble.com/Vagrant-Windows-Since-1-2-3-ruby-exe-is-missing-from-embedded-bin-td1337.html [17:54:30] awight: meh ill just go vps [17:55:00] +1, I’m doing vagrant on MacOS but it’s not the most satisfying arrangement [19:32:56] Zppix: Still in a VM, but I just succeeded in installing ORES and Wiki Labels without the vagrant image. [19:33:28] If you decide to go that way, I would follow the instructions for installing Flask first, to make sure you have those dependencies squared up: http://flask.pocoo.org/docs/0.12/installation/ [19:34:12] Once deps are installed, you can start a dev server, https://github.com/wiki-ai/wikilabels#starting-the-dev-server [19:35:04] FWIW, I like to have both vagrant and handmade installs of software I’m working on [19:37:52] The vagrant serves as a reference implementation, and a setup I can readily recreate from scratch for clean testing. [19:38:31] Hand-provisioned code is nice cos you have more flexibility, and as a backup in case the vagrant needs maintenance [19:43:46] I hear other devs’ use of vagrant ranges widely, though. Looking forward to hearing about when and how vagrant works out for you, Zppix. [19:44:06] (and *if) [19:57:59] 10Scoring-platform-team: Tasks should not be marked permanently revoked due to high system load - https://phabricator.wikimedia.org/T175860#3605722 (10awight) [22:13:53] 10Scoring-platform-team, 10Wikilabels, 10User-Zppix: Error messages should not contain relative paths or error codes - https://phabricator.wikimedia.org/T175726#3606164 (10Zppix) 05Open>03stalled Blocked on T175846 [22:14:40] awight: i hope thats okay ^^ [22:15:34] Zppix: Not quite, since your personal setup doesn’t block the task itself, only the you doing part [22:16:13] awight: (technically it blocks the task cause i cannot work on it til i get the resources to do it no? [22:16:57] Zppix: Perhaps you can make a “stalled” column on https://phabricator.wikimedia.org/project/board/2745/ [22:17:24] hmm [22:17:35] 10Scoring-platform-team, 10Wikilabels, 10User-Zppix: Error messages should not contain relative paths or error codes - https://phabricator.wikimedia.org/T175726#3606207 (10Zppix) 05stalled>03Open [22:20:39] 10Scoring-platform-team: Tasks should not be marked permanently revoked due to high system load - https://phabricator.wikimedia.org/T175860#3606244 (10awight) The cause seems to be the scoring_systems.timeout config, which is set to 15 seconds. If it takes longer than that to serve a request, it'll automaticall... [22:34:14] 10Scoring-platform-team: Tasks should not be marked permanently revoked due to high system load - https://phabricator.wikimedia.org/T175860#3605722 (10Zppix) maybe limit the tasks that cause ORES to overload the server to certain time/size/query time/etc [22:50:28] 10Scoring-platform-team: Tasks should not be marked permanently revoked due to high system load - https://phabricator.wikimedia.org/T175860#3606303 (10awight) I did an experiment in which I set the apply_async expires=timeout+1, so the signal would fire before celery times out. IMO we should do this, and the lo... [23:09:26] 10Scoring-platform-team: [Investigate] Does alarm timeout break Celery? - https://phabricator.wikimedia.org/T175875#3606349 (10awight)