[00:16:27] (03CR) 10Awight: Hooks to maintain judgment link tables (036 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [00:29:19] (03CR) 10Awight: Maintenance scripts for judgment indexes (034 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [00:29:25] (03PS16) 10Awight: Render Judgment pages as wikitext [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464737 (https://phabricator.wikimedia.org/T206346) [00:29:27] (03PS15) 10Awight: Tests to demonstrate SpamBlacklist integration [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464727 (https://phabricator.wikimedia.org/T206255) [00:29:29] (03PS50) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [00:29:31] (03PS9) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [00:29:34] (03PS8) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [00:52:44] o/ [10:06:41] o/ [10:29:45] Yesterday I had some fun incident with my work laptop trying to upgrade to bionic causing me to wipe out everything, I'll be slow :( [13:37:16] o/ [13:37:37] oh no Amir1! What's bionic? [13:37:50] ubuntu 18.04 :D [13:38:38] Ohhhhh [13:39:01] * halfak is scared of upgrades. They are productivity sucks. [13:40:22] I think for debian things will be more smooth [13:40:45] but I'm not going to try another migration any time soon [13:49:39] 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: Create a newcomerquality meta-model for revscoring - https://phabricator.wikimedia.org/T205926 (10Halfak) @notconfusing and I worked together on looking at what it would take to implement a `user_oriented` d... [13:56:55] halfak: I'm testing with CPU usage on ores-staging [13:57:21] the doubling happened before I started the second precache [13:57:32] I think it's because it had to get to speed [13:57:43] hmm. that wasn't my experience [13:58:00] I watched it stabilize for a while and the jump happened as soon as i started the second. [13:58:19] Oh! I need to work on mwpfh 0.5.2 [13:58:36] halfak: yeah for that one I didn't enable the redis task tracker [13:58:51] oh! so were we using the celery task tracker? [13:59:10] nothing [13:59:28] because I applied the patch but forgot to change the config :D [13:59:43] right now, it stays the same [13:59:46] around 60% [14:00:31] it goes up and down though, the speed of edits vary I guess [14:00:55] Where are you looking? [14:01:11] https://grafana-labs.wikimedia.org/dashboard/db/ores-labs?panelId=5&fullscreen&orgId=1&edit&tab=general&from=now-1h&to=now [14:02:48] I tested it another way though to make sure it responds properly, put time.sleep(10) and made two parallel requests at different times, they responded at the same time [14:02:49] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:03:43] now, I stop the second one, let's see if it reduces the CPU load [14:05:57] Oh this is better than what I was doing in graphite directly. [14:06:49] graphite is gross :D This looks modern [14:08:28] Also if feature extraction is happening on uwsgi, some cpu increase is expected. We can test it better by deploying on labs though [14:08:33] halfak: ^ [14:09:01] No work should be duplicated. Feature extraction does not happen on uwsgi. [14:09:25] The IO for basic datasources is done in uwsgi. This does not incur additional CPU. [14:09:44] If there's an in-progress job, that IO doesn't happen. [14:17:38] datasource extraction I meant [14:18:16] can we test it on ores.wmflabs.org? That has a different setup, making it pretty useful to do this test [14:18:48] Amir1, still datasource extraction will not incur substantial CPU costs. [14:18:55] What are you seeing in your ores-staging test? [14:21:06] after stopping the second precache the cpu usage dropped from 70% to 50% [14:21:11] it's not half [14:21:19] that's good [14:22:17] halfak: ^ [14:22:49] That's pretty bad honestly. That means we're catching about half of the duplicate jobs. [14:22:58] But I agree about using ores.wmflabs.org [14:23:17] It'll help separate the CPU usage between uwsgi (non-negligible for just handling requests) and celery. [14:23:50] If I'm right about uwsgi, then we'll see that celery still shows higher CPU usage than expected. [14:26:23] I'm running it with celery deduplicator to make sure how much overhead we have [14:26:29] (in ores.wmflabs.org) [14:28:08] Amir1, our celery deduplicator has some known issues. I think your redis deduplicator will beat it in performance. [14:28:25] What sort of delay are you using? [14:28:37] delay 0.2 [14:28:46] Gotcha. [14:29:09] I forget what my original tests with celery dedupe looked like. [14:29:12] Been a while [14:29:29] Also, ARG! I thought that mwparserfromhell 0.5.2 was released. Turns out it is not. [14:30:01] I just pinged earwig to ask for a release. So it looks like I'll need to wait for that. [14:37:15] o/ notconfusing [14:37:25] I updated the task with my understanding. [14:37:31] Re. user-oriented [14:38:30] I'm going to keep thinking through what it means to have a list of dependents. I think it means that everything inside of it is a tree of lists as well. [14:39:06] E.g. you have user.first_session.text as an array of texts [14:39:22] halfak: ok, great. I couldn't push the branch to the main repo yesterday (permissions) it's on my form [14:39:25] Still working it out. [14:39:45] Oh yeah. That is probably related to us moving to the "wikimedia" org from our own "wiki-ai" [14:39:47] Amir1, ^ [14:39:57] What's the best way to get notconfusing rights on our repos. [14:40:02] *? [14:40:23] Can we give him permission to all of them in one swoop? [14:40:24] can be added easily as collaborator, it should be easy [14:40:27] fork* . yeah it brought up some fundamental stuff about how revscoring works, it's about out of my familiarity with the repo unfortunately. [14:40:29] maybe team member [14:41:42] * halfak swoops. [14:41:46] notconfusing, you should have an invite [14:41:48] thanks Amir1 [14:43:01] notconfusing, indeed this work has sent us spinning. I think that we'll get a big benefit from doing this the hard way. [14:43:21] Since we'll be able to dig into some of the nuanced temporal features. [14:47:14] Yes, althought I thought we could get the "purely" temporal features from doing it the easy way. Ie. comparing timestamps. The really nuanced stuff like "trendline of bytes over time" or whatever, is what this could unlock, I believe. [14:51:32] Oh yes. We'll build an entirely separate pile of code to process this [14:51:51] That pile of code will live out of revscoring and would be difficult to integrate [14:52:15] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:59:08] revscoring has some nice facilities for mixing and matching features if we can get this foundation right. [14:59:54] I think I might need to re-implement revision_oriented.Revision [15:00:02] to have session_oriented.RevisionList [15:00:21] so RevisionList.text is an array of text [15:00:44] And revisionList.diff contains an array of diff-related datasources [15:02:08] I'll have a proposal explaining this ready for next week's sync meeting. [15:02:17] We can check if I'm crazy :) [15:07:58] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.025 second response time [15:08:18] Amir1, your testing ^ ? [15:08:39] yup [15:08:54] :) Rock on. [15:09:47] Ugh. I think I might need to take a sick day folks. I'm feeling pretty bad. I figured I'd give it a couple of hours to see if I felt better as the day went on but this isn't working. I'm struggling to read through doc.s [15:10:36] I'll drop an email to the list. That means I won't be at staff, but there's no org updates (do to presentation slide hell this week taking over management meetings) [15:10:45] My OOO is up to date. [15:12:19] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 442 bytes in 0.543 second response time [15:12:38] halfak: feel better soon [15:13:12] Thanks man. [15:15:49] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.023 second response time [15:16:23] That's me :D [15:18:09] thanks for the invite, i was able to create a new branch on revscoring [15:18:09] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 5.731 second response time [15:18:43] halfak: the worker cpu stays the same with redis task tracker [15:18:45] ^^ [15:19:26] \o/ Amir1 do you see a spike in uwsgi CPU? [15:20:17] (03PS9) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [15:20:36] yup [15:20:39] halfak: ^ [15:21:01] Amir1: btw I've integrated hoo's changes and ext-JADE stuff is ready for re-review any time [15:21:19] cool [15:21:41] I will take a look [15:21:58] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:58] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 0.581 second response time [15:23:00] still me [15:23:16] I think I'll ask for DBA review in SoS [15:24:07] halfak: I'm supposed to report "mwparserfromhell" for SoS but it's not clear what that means. [15:24:11] halfak: do you think we can merge and deploy now? [15:24:12] Did we upgrade the library? [15:24:19] Are we switching to a different parser? [15:24:37] we haven't upgraded yet, it's not released yet :D [15:25:02] soo, nothing to report? [15:25:23] we can say, we are facing this issue, we will upgrade soon [15:25:36] nothing in the active workboard, strange [15:25:59] OK, I'll mention the parsing issue if that's all [15:28:59] awight, the task has to do with linearly increasing CPU [15:29:07] halfak: kk, I have: [15:29:12] ** ORES cluster CPU usage has been slowly climbing, from 5% to almost 100%. This seems to be caused by a bug in mwparserfromhell, we're going to upgrade soon. No downtime so far. [15:29:37] So nice that I haven't had to be involved in that bug yet :-) [15:29:50] Might note that we have alerts in place and an issue report will be written. [15:30:09] Another question, Amir1 is there a Phab bug for converting our task tracking from Celery to Redis? [15:30:32] kk [15:30:41] awight: I'm using https://phabricator.wikimedia.org/T152012 :D [15:31:05] excellent, thansk [15:31:45] one interesting note, it reduces the load on redis nodes as well [15:32:03] I'm still not clear on the change we're making, will paste a summary here for review. [15:32:59] ** We're switching some of our task tracking from Celery to Redis, which will help us with an upgrade to Celery 4, and reduces load on Redis. https://phabricator.wikimedia.org/T152012 [15:34:05] cool [15:34:48] thanks [15:36:30] wikimedia/revscoring#1544 (user_oriented - fda780e : Max Klein): The build failed. https://travis-ci.org/wikimedia/revscoring/builds/442740402 [16:09:41] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Silence or address E_WOULDBLOCK warning - https://phabricator.wikimedia.org/T152012 (10Ladsgroup) https://github.com/wikimedia/ores/pull/270 [16:28:18] harej: Looks like your alias is shortened at https://wikiconference.org/wiki/2018/Schedule#Saturday.2C_October_20 ? [16:28:32] It's just your last name, donno if that was intentional. [16:59:37] going to eat something [16:59:56] :) [17:10:05] awight: for some reason the schedule lists the presenters as “ from ” [17:10:25] Just a default setting, I guess [18:10:36] 10ORES, 10Scoring-platform-team (Current), 10Growth-Team, 10MediaWiki-extensions-PageCuration, and 2 others: Merge articlequality and itemquality - https://phabricator.wikimedia.org/T206037 (10kostajh) > I'm going to push this forward, but would first like to reopen discussion of the name. @awight I see t... [19:02:45] 10ORES, 10Scoring-platform-team (Current), 10Growth-Team, 10MediaWiki-extensions-PageCuration, and 2 others: Merge articlequality and itemquality - https://phabricator.wikimedia.org/T206037 (10awight) @kostajh Sorry for the delay, we haven't merged any patches that will affect ORES yet. I'll post here onc... [19:47:26] (03PS14) 10Awight: Split user schema into local ID, central ID, and IP [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) [19:47:28] (03PS17) 10Awight: Render Judgment pages as wikitext [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464737 (https://phabricator.wikimedia.org/T206346) [19:47:30] (03PS16) 10Awight: Tests to demonstrate SpamBlacklist integration [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464727 (https://phabricator.wikimedia.org/T206255) [19:47:32] (03PS51) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [19:47:34] (03PS10) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [19:47:36] (03PS10) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [19:47:38] (03PS1) 10Awight: Test helper for judgment text; add defaults [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468076 [19:47:40] (03PS1) 10Awight: Comment fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468077 [19:47:42] (03PS1) 10Awight: Link table model [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468078 (https://phabricator.wikimedia.org/T202596) [19:57:48] (03CR) 10Hoo man: [C: 032] Spaces to tabs in JSON files [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466829 (owner: 10Awight) [20:16:22] (03Merged) 10jenkins-bot: Spaces to tabs in JSON files [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466829 (owner: 10Awight) [20:25:05] (03CR) 10jenkins-bot: Spaces to tabs in JSON files [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466829 (owner: 10Awight) [23:07:29] (03CR) 10Hoo man: [C: 032] Test helper for judgment text; add defaults [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468076 (owner: 10Awight) [23:07:50] niice [23:08:07] (03PS17) 10Awight: Tests to demonstrate SpamBlacklist integration [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464727 (https://phabricator.wikimedia.org/T206255) [23:08:09] (03PS52) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [23:08:11] (03PS2) 10Awight: Link table model [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468078 (https://phabricator.wikimedia.org/T202596) [23:08:13] (03PS11) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [23:08:15] (03PS11) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [23:08:17] (03PS1) 10Awight: Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 [23:08:32] (03CR) 10Hoo man: [C: 032] Comment fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468077 (owner: 10Awight) [23:13:14] (03Merged) 10jenkins-bot: Test helper for judgment text; add defaults [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468076 (owner: 10Awight) [23:14:37] (03CR) 10jerkins-bot: [V: 04-1] Link table model [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468078 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:16:43] (03CR) 10jerkins-bot: [V: 04-1] Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:16:49] (03CR) 10jerkins-bot: [V: 04-1] Link table model [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468078 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:16:58] (03Merged) 10jenkins-bot: Comment fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468077 (owner: 10Awight) [23:17:05] (03CR) 10jerkins-bot: [V: 04-1] Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:17:24] (03CR) 10jerkins-bot: [V: 04-1] Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [23:20:26] (03CR) 10jerkins-bot: [V: 04-1] Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:20:35] (03CR) 10jerkins-bot: [V: 04-1] Tests to demonstrate SpamBlacklist integration [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464727 (https://phabricator.wikimedia.org/T206255) (owner: 10Awight) [23:21:31] suave [23:22:30] (03CR) 10jerkins-bot: [V: 04-1] Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:25:22] (03CR) 10jerkins-bot: [V: 04-1] Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:27:59] (03CR) 10jerkins-bot: [V: 04-1] Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [23:34:14] Ooh I hate when the tests pass locally [23:38:59] (03CR) 10Hoo man: [C: 04-1] "We're almost there :)" (034 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight) [23:40:42] (03PS2) 10Awight: Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 [23:46:02] (03CR) 10jerkins-bot: [V: 04-1] Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [23:46:08] (03CR) 10Awight: Split user schema into local ID, central ID, and IP (034 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight) [23:46:16] (03PS15) 10Awight: Split user schema into local ID, central ID, and IP [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) [23:48:07] (03PS18) 10Awight: Render Judgment pages as wikitext [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464737 (https://phabricator.wikimedia.org/T206346) [23:48:09] (03PS3) 10Awight: Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 [23:54:25] (03CR) 10jerkins-bot: [V: 04-1] Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [23:54:30] (03CR) 10jerkins-bot: [V: 04-1] Unit tests for JudgmentContent [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [23:55:01] (03CR) 10Hoo man: [C: 04-1] "I missed some stuff earlier … ParserCache and language mismatches are pure fun… :/" (038 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464737 (https://phabricator.wikimedia.org/T206346) (owner: 10Awight) [23:57:45] (03CR) 10Hoo man: [C: 032] Split user schema into local ID, central ID, and IP [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight)