[02:26:28] wiki-ai/editquality#284 (awight-huwiki - b68b6f7 : Adam Wight): The build was fixed. https://travis-ci.org/wiki-ai/editquality/builds/361451036 [13:39:21] o/ [13:41:54] 10Scoring-platform-team, 10Research, 10Wikilabels, 10Research-2017-18-Q3, 10Research-2017-18-Q4: Design a data collection pilot using WikiLabels platform (mining reasons) - https://phabricator.wikimedia.org/T186351#4100535 (10Miriam) > Wiki database id, for example fawiki, de... [14:43:38] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [14:51:38] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [14:52:46] OK with that out of the way. Now it's time to shovel -- again >:( [15:10:49] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [15:19:31] Well. That's frustrating. I wish April would get its act together. [15:19:34] Amir1, ^ [15:19:46] Should we be concerned about these icinga alerts? [15:25:50] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [15:38:50] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [15:46:04] 10Scoring-platform-team (Current), 10Research-Backlog, 10Research-outreach, 10Epic, 10Paper: [Epic] Write paper about ORES as a socio-technical probe - https://phabricator.wikimedia.org/T121719#4100997 (10Halfak) I've re-invite @Capt_Swing, @staeiou, @Ladsgroup, and @JtsMN to work on this again. [15:55:50] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [16:11:59] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [16:19:59] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [16:39:18] 10Scoring-platform-team, 10Scap, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4101205 (10awight) Is there anything I should be doing to support this? [16:43:07] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [16:50:08] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [16:55:54] What’s that all about? [16:56:51] The service itself looks reasonably healthy [16:59:37] I think so too. I had a short look and pinged Amir1 via telegram. [16:59:39] No response. [16:59:42] awight, ^ [16:59:52] kk [17:09:46] (03PS1) 10Awight: Build venv into deployed source dir (take 2) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/423740 (https://phabricator.wikimedia.org/T181071) [17:10:52] akosiaris: Are you available to push a puppet change, some time before 18:00 UTC? [17:11:13] I’m ready to deploy the virtualenv patch… [17:15:28] (03CR) 10Awight: [V: 032 C: 032] "Self-merging already reviewed change." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/423740 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [17:15:59] halfak: FYI, I’m giving the virtualenv hack a shot [17:16:26] Cool. How do you plan to test? [17:17:26] I’ve tried it on beta, where it at least deploys the files correctly. Unfortunately, there’s a piece we haven’t tested yet, that the init scripts will pull python packages from the right directory. [17:17:39] That’s a one-liner though, so easy to rollback if needed. [17:18:43] The state we’re about to be in is to have two virtualenvs, one in the normal location and one in the versioned location. [17:19:28] that will work regardless of which application code is active. [17:21:35] OK gotcha. [17:22:00] So the plan is to do a normal deployment and see what happens? [17:22:55] yep! here goes [17:26:32] akosiaris: Why would the production ORES deployment suddenly try to use scb1002 as the canary again? [17:26:59] cat scap/ores-canary [17:26:59] scb1002.eqiad.wmnet [17:27:00] Check the scap config [17:27:09] bad scap config [17:27:11] kk [17:27:27] * halfak puts scap config in the kennel for a time out [17:28:42] 10Scoring-platform-team, 10Operations: Remove deprecated hosts from ORES scap config - https://phabricator.wikimedia.org/T191321#4101428 (10awight) [17:28:50] oh boy [17:28:51] Timeout, server ores1002.eqiad.wmnet not responding. [17:29:42] I can ssh from my local box [17:39:37] SSH is timing out for me [17:39:42] awight, ^ [17:43:37] I can still get in [17:43:53] We’re looking at some funkiness with dsh, I think [17:50:32] (03PS1) 10Awight: Try to switch to the dsh host manifest [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/423752 (https://phabricator.wikimedia.org/T191321) [17:50:40] 10Scoring-platform-team, 10MediaWiki-Interface, 10MediaWiki-extensions-ORES, 10ORES, and 2 others: Add ORES editquality highlighting/filtering to article's history - https://phabricator.wikimedia.org/T190768#4101511 (10kaldari) [17:50:52] halfak: Feel like reviewing my dsh patch? [17:51:02] linky [17:51:10] https://gerrit.wikimedia.org/r/423752 [17:51:48] They say that scap will discover /etc/dsh/groups/ores if I don’t give it a hosts manifest in the repo [17:52:38] Hmm. I don't know how that works. [17:53:01] Not sure i can review effectively. [17:53:16] But it I can confirm that you seem to have done what you set out to do. [17:53:20] lol [17:53:52] I’m shooting from the hip, too. Here’s the dsh fallback code, https://github.com/wikimedia/scap/blob/master/scap/targets.py#L226 [17:54:34] Why rename ores-worker-labs? [17:54:47] Seems like a better name would be ores-beta maybe? [17:54:49] only cos I got rid of the worker vs web targets [17:55:09] Ahh. [17:55:28] Who *would* be a good reviewer for this? akosiaris? [17:55:59] (03PS2) 10Awight: Try to switch to the dsh host manifest [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/423752 (https://phabricator.wikimedia.org/T191321) [17:56:08] I’ll add a few reviewers [17:57:05] Meanwhile, I’m out of runway for the day. Will try again tomorrow. [18:15:25] ! looks like ORES Extension stopped processing jobs a few hours ago [18:15:25] 04Error: Command “looks” not recognized. Please review and correct what you’ve written. [18:15:28] Amir1, ^ [18:15:45] hargh [18:16:30] awight, do you know how we could more effectively check on this? [18:16:58] I can do it. I was distracted by the other deployment but back on duty now. [18:17:29] Generally, I don’t know of any effective ways to debug the extension. I’m starting with logstash. [18:18:22] Responses are fine, https://ores.wikimedia.org/v3/scores/wikidatawiki/?models=damaging%7Cgoodfaith&revids=660582788&precache=1&format=json [18:22:06] I just got Amir1 on gchat. [18:22:17] "I would definitely bring this up to ops or releng [18:22:17] because this shows there is an underlying problem" [18:54:39] wiki-ai/editquality#285 (model_info_dir - 26c57ba : Adam Wight): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/361781716 [19:15:07] halfak: When you have a minute, can you evaluate model health before and after the merge_labels change? before: https://phabricator.wikimedia.org/P6932 after: https://phabricator.wikimedia.org/P6931 [19:16:18] Looks food to me. Did you see anything that surprised you that I should look at specifically? [19:18:04] good faith model_info: before: https://phabricator.wikimedia.org/P6933 after: https://phabricator.wikimedia.org/P6934 [19:18:48] We're not very good at making this prediction. Interesting. [19:18:59] It's clear that we're getting better with the new merge though. [19:19:14] I wonder if this is because the new merge is more honest and less likely to make assumptions. [19:19:50] My theory was that we were overfitting due to the repeated observations [19:20:19] But the fit gets better after [19:20:47] Yeah… I forget which data I was testing on, now :-/ [19:20:49] In the case of duplicates, they were likely to show up in the test set and skew stats. [19:20:52] ha [19:21:11] It’s been a long week! [19:23:18] I can imagine :| [19:24:30] Ah, another thing that might be improving the goodfaith prediction is that we default to goodfaith=!damaging when the label is missing. [19:24:54] previously, we would have defaulted to goodfaith=true [19:26:46] pchelolo fixed our graph! [19:28:31] awight, right. I think that's better. [19:28:33] \o/ [19:31:12] 10Scoring-platform-team (Current), 10Operations, 10Patch-For-Review: Remove deprecated hosts from ORES scap config - https://phabricator.wikimedia.org/T191321#4101950 (10awight) [19:40:27] halfak: Maybe I misunderstood you, or just failed to write the task clearly: T190747 [19:40:30] T190747: Report revscoring utilities errors in a summary at the end of the run, rather than inline warnings - https://phabricator.wikimedia.org/T190747 [19:40:57] I thought I was supposed to take the random loglines that appear during e.g. extraction, and instead increment tallies, which are reported at the end. [19:41:00] Ahh yeah. I think I see what you're saying now. The task did not help me when I was grooming the backlog. [19:41:09] +1 for that [19:41:22] I think we need an example. Also probably don't need to do it for all of the utilities. [19:41:34] Maybe just the ones that run really fast like merge_labels. [19:42:54] hum, okay I’ll rewrite and leave it for now [19:44:35] https://phabricator.wikimedia.org/T190747#4101978 [19:44:36] 10Scoring-platform-team: Report revscoring utilities errors in a summary at the end of the run, rather than inline warnings - https://phabricator.wikimedia.org/T190747#4101978 (10Halfak) I'm thinking we should print a table at the end of merge_labels that looks like this: ``` ======================... [19:47:09] 10Scoring-platform-team: Some revscoring data utilities should report warnings at the end, rather than with console loglines - https://phabricator.wikimedia.org/T190747#4101989 (10awight) [19:47:55] Cool, thanks for the ideas [20:21:56] I think there's an intersection between the terms Hearing to Speech and Preserving the Margins. [20:22:24] This intersection was first pointed out in Sorting Things Out by Bowker and Star. [20:22:55] This sounds like important procrastination [20:22:59] wiki-ai/editquality#288 (no_diff - 1e6d0a1 : Adam Wight): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/361819417 [20:23:24] Na. I'm developing an argument for what ORES is from a socio-technical point of view [20:23:48] Assuming that tool development is an ecology, then there's dynamics of conversation/adaptation/innovation. [20:24:27] By making it easier for non-mainstream technologies to progress and gather attention/relevance, we're "preserving the margins"(Mugar, 2017) [20:24:41] Or "hearing to speech"(Morton, 1985) [20:25:17] Looks like Morton uses the term in 1977 also, “Beloved Image" [20:25:59] Gotcha. I'm getting this reference secondarily via Bowker/Star who sort of brought the idea to HCI research. [20:27:24] I need to figure out what ORES *is* though. All I'm really getting at is what ORES does. [20:27:47] Is it a boundary generator? A normalizer? A socio-technical safety net? [20:27:50] ORES is dehydrated curators [20:27:57] ha [20:28:12] It's just a box of ghosts [20:28:29] Whatever ORES is, ToolForge is it too [20:28:55] The whole purpose of ToolForge is to make tool development and maintenance easier so that we can get more-better tools. [20:29:28] The whole purpose of ORES is make IUI-tool development and maintenance easier so that we can more-better AI-based tools. [20:30:15] http://knowyourmeme.com/photos/624000-creepypasta [20:31:04] I usually try to explain ORES as just a container for AIs [20:31:37] That's not a good way to publish papers :P [20:32:21] lol [20:36:38] LOL: http://www.oracle.com [20:37:05] ? [20:37:35] I was hoping to get steered to wikipedia, instead I found Oracle’s homepage 404'ing [20:41:15] It didn't 404 for me [20:44:49] Maybe they’re trying to deliver an overly fancy ad to WMF staff [20:46:51] I'm very confused how we came to be talking about oracle. [20:49:12] A complete aside. I got there by reading about how the term is used in machine learning. [20:54:49] ORES is a standardization initiative to pull together homebrewed AIs, and a central place to learn and apply insights [20:56:33] a repository for sharing ML models [21:03:06] Container is a pretty good analogy, though. It does something similar to the democratization of VPSes, where a small company or an individual can host servers without needing to invest in the infrastructure, and via ORES a smaller-language wiki can have models without needing to build the tool itself. [21:03:30] That's an interesting observation. [21:03:45] In a lot of ways, I think the whole internet did this to ideas and publishing. [21:04:12] It didn't save the world, but it certainly was a hearing-to-speech [21:06:14] 10Scoring-platform-team, 10JADE: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381#4102255 (10awight) [21:06:51] 10Scoring-platform-team, 10JADE, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Extension review for JADE - https://phabricator.wikimedia.org/T188307#4102259 (10awight) [21:07:10] 10Scoring-platform-team, 10JADE: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381#3851603 (10awight) [21:17:52] 10Scoring-platform-team, 10JADE: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381#4102332 (10awight) [21:18:01] 10Scoring-platform-team, 10JADE: Build consumable dumps of JADE - https://phabricator.wikimedia.org/T183204#4102330 (10awight) 05Open>03Invalid We've built JADE to store data in on-wiki namespaces, so separate dumps are no longer needed. [21:18:39] 10Scoring-platform-team, 10JADE: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381#3851603 (10awight) [21:18:48] 10Scoring-platform-team, 10JADE, 10Design: Design conceptual prototype of JADE integration with MediaWiki - https://phabricator.wikimedia.org/T182829#4102337 (10awight) [21:19:00] 10Scoring-platform-team, 10JADE, 10Design: Design curation/suppression integration with MediaWiki (for JADE) - https://phabricator.wikimedia.org/T183276#4102334 (10awight) 05Open>03Invalid Suppression is handled by normal wiki mechanisms. [21:19:12] 10Scoring-platform-team, 10JADE: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381#3851603 (10awight) [21:19:20] 10Scoring-platform-team, 10JADE, 10Design: Design conceptual prototype of JADE integration with MediaWiki - https://phabricator.wikimedia.org/T182829#3835906 (10awight) [21:22:49] 10Scoring-platform-team, 10JADE, 10Wikimedia-Extension-setup, 10Design, 10Wikimedia-extension-review-queue: Design review for JADE - https://phabricator.wikimedia.org/T190690#4102376 (10awight) [21:27:52] 10Scoring-platform-team, 10Operations, 10Ops-Access-Requests: Request for "administrator" rights on beta cluster - https://phabricator.wikimedia.org/T191356#4102382 (10awight) [21:52:47] ^ <3 awight [21:53:03] just cleaning up a bit [23:30:24] time for pedALING [23:30:38] It's assessment day. I hate assessment day. [23:30:57] Oh! But first I need to shovel AGAIN >:(