[00:17:21] 10ORES, 10Scoring-platform-team (Current), 10Performance, 10User-Ladsgroup: ores sends duplicate page text over the wire - https://phabricator.wikimedia.org/T214841 (10Halfak) The PR didn't quite do what it needed to to so I pushed a followup commit and cleaned up some of the tests. [00:21:21] OK good enough for today. Have a good one, folks! [00:21:22] o/ [00:52:06] wikimedia/ores#1301 (T214841 - e0c1506 : halfak): The build was broken. https://travis-ci.org/wikimedia/ores/builds/498469755 [00:56:32] wikimedia/ores#1303 (master - e1d2bc1 : Aaron Halfaker): The build was broken. https://travis-ci.org/wikimedia/ores/builds/498471325 [05:10:03] PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1080 MB (5% inode=92%); [06:26:03] RECOVERY - check disk on ORES-web02.Experimental is OK: DISK OK [08:40:34] 10ORES, 10Scoring-platform-team, 10Edit-Review-Improvements, 10Growth-Team: Save ORES-guided edit-review decision data to improve accuracy of ORES models - https://phabricator.wikimedia.org/T145161 (10Liuxinyu970226) [08:42:02] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10Growth-Team, 10Wikidata: Support ORES for propagated Wikidata edits - https://phabricator.wikimedia.org/T158025 (10Liuxinyu970226) [08:43:43] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Growth-Team, 10MediaWiki-Recent-changes: Add HTML element or class for patrolled edits to Special:Recentchanges - https://phabricator.wikimedia.org/T134689 (10Liuxinyu970226) [13:06:20] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Tiger3018) So what's going on about this issue? Need a PR with python source on github, a more detailed / traditional Chinese support list or somethi... [14:34:47] o/ [14:35:12] halfak: o/ [14:53:41] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) a:05ToAruShiroiNeko→03None Hi @Tiger3018! I've added this to our backlog. Now that we have the basic list you have provided, we need to... [15:01:18] halfak: how are you today [15:02:08] Hey Zppix. Doing OK. The weather in MN is hitting me though. It's been a slog this last month with snow, cold, and house issues due to the snow and cold. [15:02:11] How about you? [15:02:53] halfak: Good looking for something to do thats simple and "administrative"/clerical in a way [15:04:23] Hmm. I wonder if I have something like that. [15:04:33] I could find a productive task if you wanted :) [15:04:41] halfak: Always willing to lend a hand [15:07:39] OK give me a sec. I'll find something. [15:08:52] OK here's what I propose. Look at all of the README's for each repo in https://github.com/orgs/wikimedia/teams/scoring-platform/repositories and not what is missing and ought to be added. [15:08:56] I have a task for this. [15:08:58] Let me dig it up. [15:09:25] https://phabricator.wikimedia.org/T208338 [15:09:50] halfak: alright will start looking at it :) [16:03:14] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.284 second response time [16:03:22] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.276 second response time [16:03:42] I'm starting to get over-notified for these non-issues on wmflabs. [16:03:48] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 6520 bytes in 5.050 second response time [16:03:56] Hmm. OK now I'm paying attention. [16:04:23] Looks like we're up from my point of view. [16:04:28] Might be a mwapi burp. [16:05:12] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 1.242 second response time [16:05:46] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 971 bytes in 0.386 second response time [16:06:24] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 0.807 second response time [16:08:25] akosiaris, I think that all this is due to a hiccup in the mwapi. Does that jibe? [16:08:38] I can't think of another reason that wmflabs and prod would be returning 500s at the same time. [16:09:13] * akosiaris looking [16:12:17] there is an increase in 503s indeed [16:12:34] but not something large [16:12:48] and it's not timing well with the ores alerts [16:14:50] I think the worker check hits testwiki. [16:14:58] I wonder if that's the reason we see something interesting here. [16:15:12] But then again, the testwiki model doesn't actually talk to testwiki-- I don't think. [16:15:58] there was a minor bump in CPU usage in the API cluster a bit prior to the alerts https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=api_appserver&var-instance=All&from=1551196389862&to=1551197194741 [16:16:04] that actually times better [16:16:16] Ha! It does after all! [16:16:28] I think we hit testwiki in order to help Audiences teams test their code. [16:16:41] So if testwiki goes down, we'll return a 500 from icinga. [16:20:32] indeed meanlatency increased from 100ms to 125ms for a bit [16:27:26] It's hard to believe that to be the issue. [16:28:55] https://grafana.wikimedia.org/d/000000255/ores?refresh=1m&panelId=2&fullscreen&orgId=1&from=1551196726515&to=1551197619944 [16:29:00] Seems it only affected eqiad. [16:29:18] the API server cluster also had a minor burp https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=api_appserver&var-instance=All [16:30:16] PROBLEM - ores-extension grafana alert on icinga2001 is CRITICAL: CRITICAL: ORES extension ( https://grafana.wikimedia.org/d/000000263/ores-extension ) is alerting: Service hits for obtaining thresholds alert. [16:34:00] RECOVERY - ores-extension grafana alert on icinga2001 is OK: OK: ORES extension ( https://grafana.wikimedia.org/d/000000263/ores-extension ) is not alerting. [16:47:55] ^ Mediawiki deploy? [17:10:42] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Tiger3018) Got it, thank you. >>! In T109366#4984678, @Halfak wrote: > Hi @Tiger3018! I've added this to our backlog. Now that we have the basic l... [18:58:03] o/ srrodlund [18:58:24] ? [18:58:27] Nothing for ORES docs from me today except for a quick status update that I can give over IRC. [18:58:42] that is fine I have a meeting right now for an ERG anyway [18:58:49] harej, do you have anything for the agenda? [18:58:52] Oh! I see. [18:59:05] Status update is that I've picked up your task re. READMEs [18:59:19] I have Zppix doing an audit today and I hope to address what he finds later. [18:59:54] O/ [19:00:07] i'm actually working on that rn [19:01:05] and I have nothing, hence me declining [19:01:45] halfak: am i going through all the github repos? or just the ones that are actively used [19:02:06] The ones that are listed in the task have the highest priority [19:02:11] ok [19:05:18] I can tell you that ORES's readme probably just needs a complete overhaul I don't think that readme has been touched a whole lot since for 6 months and its only mostly minor changes [19:05:56] Right. I'm hoping you can apply the list that Sarah put together in the task. [19:06:31] So you want me to write up a draft on the task and/or pr? [19:27:21] Zppix, I think a draft on the task. But if you want to contribute a PR, I'd review it :) [19:27:37] (Sorry I'm jumping in and out of IRC. Feel free to ping if I miss something) [19:27:48] halfak: no worries im multitasking too :P [20:40:47] OK! Finally got out from underneath something. So now I get to go back to working on (1) Designer for Jade and (2) Topic embeddings proposal. [20:41:00] Speaking of (2), harej have you taken a look at what I have? [20:41:05] Also applicable to (1) [20:58:11] halfak: I've been taken up by other priorities, but I will be getting to those things today [21:08:11] halfak: I've also been talking with Comms -- they're interested in working with Technology to improve our technical story telling [21:08:40] (This is something I came across the other day; haven't committed to anything.) [21:08:40] Right. I have a long history of working with comms to that end. [21:09:43] I'd really like to develop our AI story with them, but as it stands, they just don't seem to be that interested. Legal, on the other hand, seems to be very interested. [21:10:56] I'm working with Jan and Sherwin to set up a speaking tour -- probably this spring or summer. I think doing some comms around that would be complimentary. [21:11:07] Question you can answer: would you like me to add you to this thread? [21:12:07] Oh yes. Sure. [21:17:42] also for next week's sync meeting I may have a proposed new meeting pattern on the table. My general idea is to look at the practices of other teams and create our own version based on that [21:18:02] Slightly related, what is your opinion of daily standup? My thinking is that it would be a decent idea if we were a larger team. We're not though so it seems kind of excessive. [21:25:29] harej, one of my main concerns recently is how we've fallen off our pace. I think a daily stand-up sounds like a good idea in the short-term. [21:25:58] harej, I think you should bring the meeting restructuring proposal to the staff meeting tomorrow and we can bash/adopt it. [21:26:09] Ooh that's right, we have the staff meeting tomorrow. [21:28:24] :D [21:30:37] Yes, we have fallen off pace. I think there are a few factors behind that. But I think things are getting back on track. [21:58:24] halfak: is there any backlog to groom today? There wasn’t when I checked yesterday. [21:59:12] Yes. We still haven't visited most of the columns. Also you were going to try to groom some technical tasks in prep for today. [22:01:00] Cool, today we’ll work on the other columns. [22:03:40] 10ORES, 10Scoring-platform-team, 10Edit-Review-Improvements, 10Growth-Team: Save ORES-guided edit-review decision data to improve accuracy of ORES models - https://phabricator.wikimedia.org/T145161 (10Halfak) Essentially, this maps to #jade integration for ERI. See {T209653} [22:05:26] 10Jade, 10Scoring-platform-team, 10Edit-Review-Improvements, 10Growth-Team: "Report error" button for ORES recent changes filter - https://phabricator.wikimedia.org/T209653 (10Halfak) [22:05:33] 10ORES, 10Scoring-platform-team, 10Edit-Review-Improvements, 10Growth-Team: Save ORES-guided edit-review decision data to improve accuracy of ORES models - https://phabricator.wikimedia.org/T145161 (10Halfak) [22:06:12] 10Scoring-platform-team: AWight to summarize CSCW notes to a blog - https://phabricator.wikimedia.org/T210662 (10Halfak) 05Open→03Declined [22:06:55] 10ORES, 10Scoring-platform-team: Streamline workflow for new ORES model lifecycle - https://phabricator.wikimedia.org/T192504 (10Halfak) 05Open→03Declined [22:13:12] 10ORES, 10Scoring-platform-team (Current): [Discuss] Storage of model training/testing datasets - https://phabricator.wikimedia.org/T193789 (10Halfak) 05Open→03Resolved a:03Halfak We decided to store the file on a public repository (figshare) and reference that directly from the makefile. See https://nd... [22:16:53] 10Scoring-platform-team, 10Analytics: [Discuss] ORES model development and deployment processes - https://phabricator.wikimedia.org/T216246 (10Halfak) Meeting scheduled for Thursday, Feb 28th @ 1630UTC. I've preemptively made an a notes document here: https://etherpad.wikimedia.org/p/ores_usecases_for_ml_infr... [22:19:00] 10Scoring-platform-team (Current), 10Analytics: [Discuss] ORES model development and deployment processes - https://phabricator.wikimedia.org/T216246 (10Halfak) p:05Triage→03Normal a:03Halfak [22:20:10] 10Jade, 10Scoring-platform-team: Regression: Judgment validation allows for multiple judgments with the same value e.g. 2x {damaging, badfaith} - https://phabricator.wikimedia.org/T210804 (10Halfak) p:05Triage→03High [22:24:27] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 4 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [22:24:34] 10Jade, 10Scoring-platform-team (Current), 10MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), 10Patch-For-Review: Rename "JADE" extension to "Jade" - https://phabricator.wikimedia.org/T211046 (10Halfak) 05Open→03Resolved a:03Halfak Looks like this is done. [22:26:15] 10Jade, 10Scoring-platform-team (Current), 10MediaWiki-Configuration, 10Patch-For-Review: Rename JADE->Jade in beta cluster configuration - https://phabricator.wikimedia.org/T212182 (10Halfak) [22:28:36] 10Jade, 10Scoring-platform-team (Current), 10MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), 10Patch-For-Review: Rename "JADE" extension to "Jade" - https://phabricator.wikimedia.org/T211046 (10Halfak) 05Resolved→03Open I'm unresolving this because some of the child tasks are not quite done yet. [22:28:40] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 4 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10Halfak) [22:29:55] 10Jade, 10Scoring-platform-team, 10Gerrit: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10Halfak) p:05Triage→03Normal [22:30:20] 10Jade, 10Scoring-platform-team, 10Continuous-Integration-Config, 10Patch-For-Review: Rename JADE->Jade in continuous integration - https://phabricator.wikimedia.org/T212181 (10Halfak) p:05Triage→03Normal [22:30:36] 10Jade, 10Scoring-platform-team (Current), 10Continuous-Integration-Config, 10Patch-For-Review: Rename JADE->Jade in continuous integration - https://phabricator.wikimedia.org/T212181 (10Halfak) [22:32:26] 10Jade, 10Scoring-platform-team (Current), 10Core Platform Team Backlog (Watching / External), 10Multi-Content-Revisions (Reactive): JADE\Tests\Hooks\MoveHooksTest::testOnMovePageIsValidMove is broken - https://phabricator.wikimedia.org/T211473 (10Halfak) [22:35:35] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: ORES should not check highly trusted user's edits - https://phabricator.wikimedia.org/T200908 (10Halfak) 05Open→03Resolved a:03Halfak I think that this is resolved now. We use use status as a predictor and... [22:40:27] 10Scoring-platform-team, 10Wikilabels, 10UX-Debt: Wikilabels should better handle "revision not found" - https://phabricator.wikimedia.org/T209926 (10Halfak) [22:40:44] 10Scoring-platform-team, 10Wikilabels, 10UX-Debt: Wikilabels should better handle "revision not found" - https://phabricator.wikimedia.org/T209926 (10Halfak) p:05Triage→03Low [22:43:59] 10ORES, 10Scoring-platform-team (Current), 10Wikibase-Quality-Constraints, 10Wikidata: quick overview of the quality of an item - https://phabricator.wikimedia.org/T195703 (10Halfak) 05Open→03Resolved a:03Halfak Given the inactivity, we're going to mark this task as resolved. We'd be very happy to w... [22:44:42] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Watchlist integration - https://phabricator.wikimedia.org/T210560 (10Halfak) p:05Triage→03High [22:48:35] 10Scoring-platform-team (Current), 10drafttopic-modeling: Surface draft topic predictions in English Wikipedia - https://phabricator.wikimedia.org/T194738 (10Halfak) 05Open→03Resolved a:03Halfak @SQL seems to have already implemented this. Thank you! [22:50:09] 10Jade, 10Scoring-platform-team (Current), 10Design, 10Epic: Re-surface judgments for continuous evaluation - https://phabricator.wikimedia.org/T204064 (10Halfak) p:05Triage→03Low [22:55:00] harej, I'm just about to run out the door. Forgot to say as we were getting off the call. Anything you need re. topic embeddings or reviewing design stuff? [22:55:41] Yeah, I need to sit down and read/review those things :D [22:58:08] Got the links and stuff handy? [23:03:22] Remind me which wireframes I haven't reviewed yet [23:03:34] halfak ^ [23:13:18] I am not sure. [23:13:31] Honestly :( Maybe you could glance at each of them via the tasks [23:13:46] Sure, I’ll do that