[14:07:46] o/ [14:15:27] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Add indonesian stemmer - https://phabricator.wikimedia.org/T141441#2498823 (10Halfak) [14:16:05] o/ schana [14:16:14] hey halfak [14:16:17] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Add indonesian stemmer - https://phabricator.wikimedia.org/T141441#2498839 (10Halfak) p:05Triage>03Low [14:16:42] How's hacking? :) [14:17:52] slow going - I'm trying to free up some space to set up a VM [14:20:46] Gotcha. what's the eta on a laptop replacement? [14:22:02] waiting for approval to purchase locally [14:22:38] over here it's more expensive to repair than buying a new one with US pricing [14:22:59] That sounds problematic. Seems like this process should be faster. It's been over a week. [14:23:42] the first week was waiting for the local repair center to diagnose it [14:23:51] OIT has moved pretty fast [14:25:00] I guess, but it's expensive to have all of this downtime. [15:38:08] halfak: hey there, I just woke up (and had breakfast) but I need to go and buy some stuff before they get closed. I'll be back in one hour. I want to know if there is anything urgent [15:38:45] Nothing urgent. Thanks for checking in :) [15:39:10] okay [15:39:14] o/ [17:39:44] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:54] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Add a new grafana pane for timeout error metric - https://phabricator.wikimedia.org/T140363#2499538 (10Ladsgroup) Okay, Several days have passed and it should give some timeout errors by now but nothing in graphite yet. [17:44:02] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 353 bytes in 9.565 second response time [18:05:43] Amir1, looks like plwiki damaging and goodfaith models are crazy [18:05:50] Always badfaith damaging for every edit [18:06:57] yeah [18:07:06] I didn't know but I'm not surprsied [18:07:14] OK. I'll have a look at it quick. [18:22:01] For some reason, I'm struggling to generate the rev_reverted dataset for plwiki [18:25:12] This is weird. [18:29:55] OMG. I needed to put a darn "rev_id" header on the file [18:29:58] * halfak grumbles. [18:33:18] and the revert detector churns. [18:33:23] * halfak twiddles [19:08:48] Amir1, sad news. I didn't get the models updated yet and I must run away for a doctor appointment. I'll pick this back up again in 1.5 hours. [19:09:00] I might just do the deploy to wmflabs tomorrow. [19:09:04] We'll see. [19:09:07] o/ [19:10:38] halfak: o/ [20:32:01] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:11] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 354 bytes in 6.909 second response time [21:41:48] back [21:43:40] halfak: welcome back [21:44:08] I couldn't work :((( I'm so excited that I can't work :|||| [21:44:14] * Amir1 clams down [21:58:15] Why are you too excited to work? [22:00:03] halfak: https://www.facebook.com/groups/wikipediaweekly/permalink/1054123404635490/? [22:00:16] I wrote the 500,001th article, that was sooo close [22:01:04] Nice! I saw that notification while I was in the waiting room, but them my internet connection dropped so I couldn't actually see *what* milestone had been crossed. [22:01:41] * halfak is finally retraining the damaging model for plwiki [22:01:51] it is actually about my 10th birthday in wikipedia [22:02:19] Nice! I'm only 9 in Wiki years. [22:02:31] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:34] My 10th will be Sept. 2017. [22:02:40] -05 seems to be struggling recently [22:08:31] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 337 bytes in 0.619 second response time [22:20:51] halfak: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/Deployment [22:20:56] what do you think of this? [22:26:22] Amir1, shall I read and take an editing pass? [22:26:43] sure [22:26:45] thanks [22:44:15] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 05Wikimedia-Incident: [Discussion] Move wikilabels js to another place? - https://phabricator.wikimedia.org/T137626#2500539 (10greg) [22:44:24] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 05Wikimedia-Incident: [Spec] CI tests for Wikilabels - https://phabricator.wikimedia.org/T137625#2500540 (10greg) [22:54:16] (03PS1) 10Ladsgroup: Add revision_id to log for errors [extensions/ORES] (wmf/1.28.0-wmf.12) - 10https://gerrit.wikimedia.org/r/301516 (https://phabricator.wikimedia.org/T141368) [23:00:23] OK models look much better now. [23:00:35] I'm going to submit a PR for 'em then take an editing pass. [23:03:31] nice [23:06:39] (03CR) 10Dereckson: [C: 032] Add revision_id to log for errors [extensions/ORES] (wmf/1.28.0-wmf.12) - 10https://gerrit.wikimedia.org/r/301516 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [23:06:51] (03CR) 10Dereckson: "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.12) - 10https://gerrit.wikimedia.org/r/301516 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [23:07:31] (03Merged) 10jenkins-bot: Add revision_id to log for errors [extensions/ORES] (wmf/1.28.0-wmf.12) - 10https://gerrit.wikimedia.org/r/301516 (https://phabricator.wikimedia.org/T141368) (owner: 10Ladsgroup) [23:07:33] Amir1, https://github.com/wiki-ai/editquality/pull/41 [23:09:05] halfak: {{merged}} [23:09:17] Have you changed the settings or anything? [23:09:23] Makefile hasn't been changed at all [23:09:45] Indeed. All I did was make sure that the latest wheels were installed and rebuilt the features and models. [23:12:19] oh, I see [23:17:15] Hmm... So the damaging model still seems to be crazy [23:17:18] looking into it. [23:19:00] Oh! Seems to only be crazy on my ubuntu install. [23:19:05] It is sane on Jessie. [23:19:12] OK! Let's give it a try in staging! [23:20:06] https://github.com/wiki-ai/ores-wmflabs-deploy/pull/67 [23:20:09] Amir1, ^ [23:20:46] halfak: {{done}} [23:21:08] I'm waiting on SWAT to get better logging for time out error [23:21:12] *errors [23:30:05] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Set up grafana dashboard for ORES extension - https://phabricator.wikimedia.org/T141169#2489242 (10Krinkle) Few tips: * Use the "Dashboards" link feature instead of hardcoding hyperlinks. That way users will stay on the current Grafana domain (e.g.... [23:37:00] Hmm... Looks like things aren't working on staging. [23:38:22] Yeah. Can't seem to get celery to score anything [23:39:13] Oh! I have to fix the weirdo staging config [23:44:14] IT WORKS [23:44:18] MWAHAHAHA [23:45:37] :)))) [23:48:28] halfak: https://phabricator.wikimedia.org/T141368 [23:48:35] click on the link to the logstash [23:48:42] "Model contains an error for 360033467: The operation timed out." [23:48:45] for wikidata [23:48:52] https://www.wikidata.org/w/index.php?diff=360033467 [23:48:54] that's the edit [23:49:03] and that's the score: https://ores.wikimedia.org/v2/scores/wikidatawiki/?models=damaging&revids=360033467 [23:49:11] works just fine but probably cached [23:49:44] This is a way to get around the cache: https://ores.wmflabs.org/v2/scores/wikidatawiki/damaging/360033467?features [23:49:53] Still fast. [23:50:04] I wonder if there was a slow API call in there. [23:50:12] That could really prevent a score from returning fast. [23:50:36] OK. So I think I want to declare victory for today in deploying the new ORES to staging on wmflabs. [23:50:50] I think tomorrow morning (utc-6), I'll deploy to ores.wmflabs.org [23:51:14] I want to do an analysis of the memory behavior on wmflabs.org before we move to prod. [23:51:26] We could deploy to prod though. [23:53:26] *beta [23:53:28] derp [23:53:34] OK. Now I'm off to go to chores. [23:53:35] o/ [23:55:25] :D