[00:27:54] CUSTOM - Host Ores-Compute-01 is UP: PING OK - Packet loss = 0%, RTA = 1.24 ms test [16:07:36] o/ [16:07:55] Hey folks. Catching up on some email and then I'll be looking into the regex issues from yesterday's downtime. [16:07:58] o/ Amir1 [16:08:15] glorian_wd, I saw you sent me a message in another channel. [16:08:47] halfak: hey! yeah. Wanted to ask you if you work today (apparently you do) [16:09:09] wanna give you some updates regarding to GetSuggestions patch and wikiclass PR [16:09:38] So, I've updated the PR. I've removed all dependentsets and streamlined the algorithms. [16:16:30] OK so it seems you'd like me to look at this today. [16:16:36] halfak: On the other hand, for the GetSuggestions patch, I have the impression that it's gonna take another month to discuss, which I couldn't afford. See https://gerrit.wikimedia.org/r/#/c/356043/20. TL;DR: it seems that Daniel wants to include classifying properties for "all_suggestions" mode. But this is not possible because the current data is not set up to do so. In the existing data in wbs_propertypairs, there [16:16:36] are no co-occurence of *common* property-value pair which suggest classifying properties. The discussion is still on going though [16:16:36] So, I'd like to ask you if we can leave this feature, and just use the existing feature for the time being. [16:16:52] However we had a major downtime event yesterday and I don't think it'd be prudent to ignore that. [16:17:01] Later, once I finish my thesis, I can help to working on the new feature [16:17:31] i.e. the getsuggestion [16:17:41] glorian_wd, I'll leave it up to you what you need for your thesis work. [16:19:10] halfak: Re. I'll leave it up to you what you need for your thesis work. Ok. Then I just need to wait you for merging the PR right? you have said that you can't do it today [16:48:50] 10Scoring-platform-team, 10ORES, 10Wikimedia-Incident: Respond to ORES downtime (2017-06-23) - https://phabricator.wikimedia.org/T168773#3376387 (10Halfak) [16:49:51] 10Scoring-platform-team, 10ORES, 10Wikimedia-Incident: Respond to ORES downtime (2017-06-23) - https://phabricator.wikimedia.org/T168773#3376401 (10Halfak) [17:10:06] 10Scoring-platform-team, 10ORES, 10Wikimedia-Incident: Respond to ORES downtime (2017-06-23) - https://phabricator.wikimedia.org/T168773#3376405 (10Halfak) I have a bit of a scattered understanding of what happened because it was a travel day for me. Here's what I can put together: https://grafana.wikimedi... [17:12:12] OK This looks good. It helped me quite a bit to go through the chat logs and figure out what was going on. It looks like subbu already submitted a PR to fix the problematic regex and I merged it yesterday. So we'll want to get revscoring updated in production ASAP. [17:12:23] This will require a model rebuild for everything that uses badwords/informals. [17:12:46] I also want to figure out why the heck our timeouts didn't just deal with this. It's exactly why we have them. [17:22:19] 10Scoring-platform-team, 10Wikilabels: [Discuss] Wikilabels routes refactor - https://phabricator.wikimedia.org/T165046#3376409 (10Halfak) Right on! So, a Campaign is a chunk of work for a group if people to do over a period of time. A Workset is a chunk of work from a Campaign for me to do in one sitting.... [17:23:49] 10Scoring-platform-team-Backlog, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3376412 (10Halfak) I think the goal here is to totally switch from the SCB* nodes to the new ORES* nodes. [17:26:42] * halfak tries to get through his email backlog [17:30:18] * paladox finally got icinga2 working properly on stretch :) [17:58:36] \o/ [18:12:27] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Albanian - https://phabricator.wikimedia.org/T168369#3376427 (10Halfak) Currently, @Ladsgroup has a nice framework for running the #bad-words-detection-system. I have an old pull reques... [18:14:30] 10Scoring-platform-team, 10Patch-For-Review, 10User-Zppix: Create memory checks for instances - https://phabricator.wikimedia.org/T167602#3376428 (10Halfak) Gotcha. I think @akosiaris is right. Let's skip the memory checks. Really we just want to know when ORES isn't working -- not when it has a short-ter... [18:14:34] 10Scoring-platform-team, 10Patch-For-Review, 10User-Zppix: Create memory checks for instances - https://phabricator.wikimedia.org/T167602#3376429 (10Halfak) 05Open>03declined [18:17:09] 10Scoring-platform-team-Backlog, 10Wikilabels: Complete Romanian Wikipedia edit quality campaign - https://phabricator.wikimedia.org/T156517#3376430 (10Halfak) Great news! Thanks for the update. We'll get to work on advanced models ASAP. [18:17:22] 10Scoring-platform-team-Backlog, 10Wikilabels: Complete Romanian Wikipedia edit quality campaign - https://phabricator.wikimedia.org/T156517#3376431 (10Halfak) [18:17:32] 10Scoring-platform-team, 10Wikilabels: Complete Romanian Wikipedia edit quality campaign - https://phabricator.wikimedia.org/T156517#2977464 (10Halfak) [18:17:58] 10Scoring-platform-team, 10editquality-modeling, 10revscoring, 10artificial-intelligence: Build damaging/goodfaith models for Romanian Wikipedia - https://phabricator.wikimedia.org/T156503#3376434 (10Halfak) [18:18:11] 10Scoring-platform-team, 10ORES, 10Wikimedia-Incident: Respond to ORES downtime (2017-06-23) - https://phabricator.wikimedia.org/T168773#3376436 (10Halfak) a:03Halfak [18:19:20] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Implement wp10 model for trwiki - https://phabricator.wikimedia.org/T164671#3241520 (10Halfak) https://github.com/wiki-ai/wikiclass/pull/42 [18:19:36] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Implement wp10 model for trwiki - https://phabricator.wikimedia.org/T164671#3376440 (10Halfak) @Nettrom or @Ladsgroup ^ :) [18:21:57] 10Scoring-platform-team, 10Operations, 10Ops-Access-Requests, 10User-Zppix: Graphite access for Zppix - https://phabricator.wikimedia.org/T168014#3376441 (10Halfak) I talked to @Zppix and @RStallman-legalteam. We're putting this on hold for now. Sorry for any extra work this caused. Please feel free to... [18:26:26] 10Scoring-platform-team, 10Operations, 10Ops-Access-Requests, 10User-Zppix: Graphite access for Zppix - https://phabricator.wikimedia.org/T168014#3353061 (10Dereckson) @Halfak A workaround could be to prepare public graphite graphs/dashboard useful for ORES, and so Zppix or any other can use them. [18:28:40] 10Scoring-platform-team, 10Operations, 10Ops-Access-Requests, 10User-Zppix: Graphite access for Zppix - https://phabricator.wikimedia.org/T168014#3376444 (10Halfak) Agreed. I was hoping to have @zppix do all of the work to get this set up (because I have like 4 jobs and waiting for me to do something is b... [19:21:46] halfak: I just got back [19:21:50] hope you're around [19:22:12] Amir1, yup. here for a bit longer. [19:22:28] Just finished 1/2 of my inboxes [19:22:29] sorry, was afk to meet a very old friend [19:22:33] :))) [19:22:38] I'm around to do some work [19:22:40] no worries. I hope your very old friend is aging well ;) [19:22:49] * halfak makes dad jokes [19:22:50] sorry [19:22:55] :)))))) [19:23:03] You are not that old for dad jokes [19:23:10] looool [19:23:17] it'll only get worse from here on out [19:24:09] :D [19:24:31] I want to work on ORES in preferences [19:24:40] already cleaned up some codes to make it easier [19:24:44] Can I ask you for a couple of merges first? [19:24:52] halfak: btw. the new highlighting is there [19:24:54] suree [19:24:56] https://github.com/wiki-ai/revscoring/pull/327 [19:24:59] nice! [19:25:10] https://github.com/wiki-ai/wikiclass/pull/42 [19:25:24] 327 needs manual rebase [19:25:30] Woops. Gotcha. [19:25:48] Woops. That was a different one. [19:25:51] I can look at that. [19:25:59] https://github.com/wiki-ai/revscoring/pull/298 [19:26:01] Amir1, ^ [19:26:55] okay, [19:27:07] I can't merge, the javascript seems broken [19:27:10] let me see [19:29:28] \o/ [19:29:50] I'll hopefully get them a basic revert detection model this coming week [19:30:29] \o/ [19:30:42] And we can get that deployed with the revscoring regex fixes :) [19:30:50] Regretfully I didn't get any of that tested today. [19:31:03] But I did get a description of the outage in phab. [19:31:20] I'll plan on getting a description in wikitech incident report on Monday if no one beats me to it. [19:31:42] nice [19:49:01] OK time for me to head out. Have a good one folks. [20:33:14] PROBLEM - check load on Ores-Compute-01 is WARNING: WARNING - load average: 8.31, 7.64, 5.31 [20:33:29] that's me rebuilding a model [20:44:13] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Albanian - https://phabricator.wikimedia.org/T168369#3376498 (10Ladsgroup) Started, ping me if it's not there after 24 hours. [21:03:15] PROBLEM - check load on Ores-Compute-01 is WARNING: WARNING - load average: 3.04, 5.44, 6.23 [21:07:41] RECOVERY - check load on Ores-Compute-01 is OK: OK - load average: 0.14, 2.43, 4.74