[02:44:45] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.039 second response time [02:45:29] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.059 second response time [02:47:19] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 0.140 second response time [02:48:05] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 0.432 second response time [02:50:35] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 6519 bytes in 5.052 second response time [02:51:49] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 972 bytes in 0.162 second response time [06:15:47] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for hewiki - https://phabricator.wikimedia.org/T215363 (10eranroz) Sure I would like to help. Is there a specific message to village pump you would like me to translate or write requesting people to participate in campign? (i don't see h... [08:10:55] 10ORES, 10Scoring-platform-team, 10Scap: Investigate: why are we getting overload errors during ORES deployments? - https://phabricator.wikimedia.org/T213116 (10akosiaris) >>! In T213116#4929800, @Halfak wrote: > @akosiaris I'd like your input here. We're running into a lot of "After moving to kubernetes, n... [09:47:10] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10WMSE-Development-Support-2019 (Automatic article quality assessment), 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Sebastian_Berlin-WMSE) @Halfak, could you, or some oth... [10:53:27] o/ [11:20:04] wikimedia/revscoring#1610 (master - 07e7bca : Amir Sarabadani): The build was fixed. https://travis-ci.org/wikimedia/revscoring/builds/489483702 [12:21:33] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 (10Ladsgroup) Okay, after rebuilding bswiki models with revscoring 2.3.3, It works. Now I rebuild all models with that. [12:22:26] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) p:05Triage→03High [14:29:24] o/ [14:59:21] 10Scoring-platform-team (Current), 10draftquality-modeling, 10artificial-intelligence: Implement and test words_to_watch features for draftquality and articlequality - https://phabricator.wikimedia.org/T214091 (10Halfak) [15:00:39] 10ORES, 10Scoring-platform-team, 10Services (designing): Investigate a Hyperswitch frontend for ORES - https://phabricator.wikimedia.org/T214874 (10awight) >>! In T214874#4925938, @mobrovac wrote: > If I understand this task correctly, it seems to be a duplicate of {T107196}. Thanks for mentioning this, it... [15:00:57] Hi Amir1! Are you running the technical advice meeting today? [15:01:22] Hey, no, I'm in another meeting [15:01:26] Oh! I see. [15:02:02] Does 1830 UTC work for you? That's after our BS meeting. [15:02:29] Or we could just meet in 30 [15:02:31] Looks like that works. [15:02:42] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @CindyCicaleseWMF - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:07:37] 10Scoring-platform-team (Current): Prototype a bot framework that utilizes newcomerquality - https://phabricator.wikimedia.org/T211434 (10awight) I've only had very general conversations about this project so far, and didn't realize there was working code :-) @notconfusing Would you mind dropping a link to the... [15:14:04] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for hewiki - https://phabricator.wikimedia.org/T215363 (10Halfak) Great! Thank you! We still need to generate a sample. I'm hoping to get you set up today. You will eventually see hewiki at that URL. I'll also try to dig up an ann... [15:19:02] halfak: sure. [15:20:00] Hey harej, I think that https://phabricator.wikimedia.org/T199520 looks good. It's sitting in the review column. Should I move it to "done" or is there somewhere else you'd like to document this criteria -- e.g. somewhere near mw:JADE [15:20:04] Amir1, cool. [15:21:40] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Tiger3018) Great. So start to write the **problemtic/words_to_watch expression** regex now? (Or other expression/ALL) As for Manual... Just click "La... [15:23:42] halfak: depends on what you consider “done” — is it done when I actually get those wikis to sign on, or is it done when I said “we are going with wikis X Y and Z” [15:24:31] That's up to you as the writer of the task :) [15:24:40] I just want to remove it from "review" under some condition. [15:24:58] Might go back to "active". Might go to "done" [15:27:12] Let’s put it in done [15:27:28] Arg. I'm running into limitations in our makefile config. I have a wiki where one sample is a balanced 5k and now I'm adding a new sample that is not a balanced 5k. [15:27:57] I'm kicking myself for not moving forward with modularization. I guess I'll be looking at that PR again today. [15:28:51] Note: nobody is signed up for Scrum of Scrums this morning [15:29:36] Also, I'm definitely out of commission again, probably for the remainder of the week [15:31:43] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Tiger3018) p:05Lowest→03Normal [15:52:42] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @CindyCicaleseWMF - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:56:17] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for fiwiki - https://phabricator.wikimedia.org/T215359 (10Zache) Damaging model is better than in summer 2017 (before flagged revs data) and it is useful. This is mostly because it gives more stable results (ie bad edits get a higher num... [16:22:39] 10ORES, 10Scoring-platform-team: Weigh Kubernetes vs. streaming architecture for ORES "2.0" - https://phabricator.wikimedia.org/T215431 (10awight) [16:22:47] 10ORES, 10Scoring-platform-team: Weigh Kubernetes vs. streaming architecture for ORES "2.0" - https://phabricator.wikimedia.org/T215431 (10awight) [16:35:44] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10WMSE-Development-Support-2019 (Automatic article quality assessment), 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Halfak) Yes! Thank you for the ping. We're pretty ba... [16:36:09] brb gonna go shovel. [16:38:40] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10WMSE-Development-Support-2019 (Automatic article quality assessment), 10artificial-intelligence: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Halfak) a:03hoo [16:45:38] "shovel"? [16:45:48] is this a reference to that mythical substance known as 'snow'? [16:48:08] Didn't get out there after all. [16:48:17] Indeed. We got a lot of it yesterday and today. [16:49:56] Classic [16:50:09] Amir1: are you going to be at the meeting in ten minutes? [16:52:15] harej: yes but only first bits [16:52:27] I need to catch the train [18:02:09] harej, are you coming for the BS session today? [18:02:16] If not, I'll actually get out there to shovel. [18:02:33] Feel free to shovel :) [18:03:01] OK will do. [18:03:15] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for fiwiki - https://phabricator.wikimedia.org/T215359 (10Zache) @Halfak btw. there is already slowly ongoing second campaign which we could try to finalize it meets the requirements: https://labels.wmflabs.org/stats/fiwiki/ [18:06:40] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for fiwiki - https://phabricator.wikimedia.org/T215359 (10Halfak) Oh yes! I remember setting that up. Finish that up would be great. It looks like we're pretty close. [19:11:14] Back! Forgot to say. [19:11:17] Been deep in email. [19:53:53] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) This looks very useful. It's hard for me to interpret without some command of the language. Machine translation kind of jumbles up what I ca... [20:12:22] (03CR) 10jerkins-bot: [V: 04-1] build: Updating mediawiki/mediawiki-codesniffer to 24.0.0 [extensions/JADE] - 10https://gerrit.wikimedia.org/r/488545 (owner: 10Libraryupgrader) [20:13:01] (03CR) 10Umherirrender: "recheck" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/488545 (owner: 10Libraryupgrader) [20:16:31] yikes. I need to eat lunch. [20:18:44] (03CR) 10jerkins-bot: [V: 04-1] build: Updating mediawiki/mediawiki-codesniffer to 24.0.0 [extensions/JADE] - 10https://gerrit.wikimedia.org/r/488545 (owner: 10Libraryupgrader) [20:29:06] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:36:56] 10Jade: Unit tests of Jade extensions failing - https://phabricator.wikimedia.org/T215459 (10Umherirrender) [20:37:02] (03CR) 10Umherirrender: "For the test failure: T215459" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/488545 (owner: 10Libraryupgrader) [20:39:18] wikimedia/ores#1267 (bump_yamlconf_version - 883b17a : halfak): The build passed. https://travis-ci.org/wikimedia/ores/builds/489731653 [20:42:57] wikimedia/revscoring#1612 (bump_yamlconf_version - 8d62087 : halfak): The build passed. https://travis-ci.org/wikimedia/revscoring/builds/489732458 [20:53:53] 10ORES, 10Scoring-platform-team: Create new editquality labeling campaign for hewiki - https://phabricator.wikimedia.org/T215363 (10Halfak) I created a new sample here https://quarry.wmflabs.org/query/33235 But I have to work on other things, so hopefully I can come back to this soon. [20:57:06] RECOVERY - puppet on ORES-worker01.experimental is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [21:55:03] I'm just about to head to my last meeting. Then I'm out of here. [21:57:03] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Implement words_to_watch in editquality models - https://phabricator.wikimedia.org/T215473 (10Halfak) [21:58:16] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Implement words_to_watch in editquality models - https://phabricator.wikimedia.org/T215473 (10Halfak) 05Open→03Resolved See https://github.com/wikimedia/editquality/pull/185 I advised @hoo that his work was good, but t... [22:27:06] halfak, awight: So, bad news re itwiki/dewiki. We had planned to work on deploying them today, but it turns out all of my "tools" for determining thresholds etc have broken [22:27:43] I say "tools" in scare quotes because it's a spreadsheet and a jsfiddle (Adam put the links in a task somewhere), and the jsfiddle doesn't play well with the new, quite strict, rate limiting, so it gets 429ed and breaks [22:29:03] The "good" news is that now Kosta and Stephane have seen all this stuff, and they agree that we should improve the tools and document a bunch of the things I do when adding ORES models, and also that we should go back through the already-enabled ones and re-evaluate their thresholds [22:29:29] Stephane in particular seemed eager to expand on Adam's new React UI to add some tables that would be useful for this purpose [22:30:05] But we probably need to have a chat about whether this kind of tooling and support stays in Collaboration Growth, or if it should be Scoring Platform's responsibility [22:30:20] And regardless, I need to write a bunch of stuff down [22:34:41] What would be required to hand off the tooling to new ownership? [23:20:47] Me documenting the hell out of things :P) [23:21:24] Cool, please do that :) [23:21:33] If I remember correctly, the reason we currently own it is a combination of, 1) the Collaboration team (when it still existed under that name) was happy to keep it and 2) I was the only one who knew how things worked anyway [23:22:04] harej: Oh that's rght, you are on Scoring Platform now, right? [23:22:13] Partially [23:22:53] lol, well that's a good enough answer for me to justify doing this work with "if I do this, we'll likely be able to hand over one of our 137 responsibilities to another team" [23:25:55] At this point I'm reluctant to sign my team up for more things, but if it's documentation... we probably need it anyway?