[00:20:26] off! [01:25:38] (03Merged) 10jenkins-bot: Adjust for core change I4764c1c78 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461441 (https://phabricator.wikimedia.org/T204669) (owner: 10Anomie) [01:26:10] (03CR) 10jenkins-bot: Adjust for core change I4764c1c78 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461441 (https://phabricator.wikimedia.org/T204669) (owner: 10Anomie) [01:26:40] (03CR) 10Krinkle: Introduce ext.ores.api (033 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/459549 (https://phabricator.wikimedia.org/T201691) (owner: 10Ladsgroup) [01:29:40] (03CR) 10jenkins-bot: Adjust for core change I4764c1c78 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461441 (https://phabricator.wikimedia.org/T204669) (owner: 10Anomie) [04:49:26] 10ORES, 10Scoring-platform-team (Current), 10Documentation: Draft of ORES threshold optimization documentation - https://phabricator.wikimedia.org/T198232 (10srodlund) p:05Triage>03Lowest The worked example is great. This document feels complete. Can we call it resolved? [08:58:18] o/ [08:58:20] 10ORES, 10Scoring-platform-team (Current), 10WMF-JobQueue, 10MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), and 4 others: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T204753 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup [10:30:49] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10akosiaris) >>! In T206654#4656931, @awight wrote: > I found this config on production, so in theory our workers should be restarting themselves: >... [10:56:45] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Ladsgroup) Celery has timeouts and it should catch them. I guess revoking tasks might be at fault here. [11:45:33] mwparserfromhell honors it's name [11:45:43] (gdb) py-bt [11:45:43] Traceback (most recent call first): [11:45:43] [11:45:43] Python Exception Type does not have a target.: [11:45:46] sigh [11:46:12] good luck figuring out now the python stacktrace [11:59:58] LOL [12:15:29] afk for lunch [13:33:06] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Ladsgroup) One other observation, CPU usage doesn't go up linearly, it actually goes up with jumps. Since we restarted. Some nodes stayed stable bu... [13:46:19] \o/ [13:46:23] That mwbase thing is merged! [13:46:24] WOOOO [13:47:33] akosiaris, thanks for digging into this. I think that we ought to (1) monitor CPU usage. Should never go above 50% for any sustained period of time and (2) update mwparserfromhell in hopes that the issue has already been addressed. [13:47:38] What do you think? [13:47:55] * halfak looks into setting up a grafana alert. [13:49:13] Looks like we have new celery processes pegged at 100% on 2003 [13:49:14] I can work on the mwparserfromhell upgrade [13:49:19] So the problem persists. [13:50:07] https://grafana.wikimedia.org/dashboard/db/ores?panelId=5&fullscreen&orgId=1&from=now-3h&to=now-1m&refresh=1m [13:50:11] It looks fine [13:50:12] Thinking this through, we have a strategy for timing out that should be robust to even mwparserfromhell's strange C. [13:50:28] Amir1, we have some processes that have been at 100% for 835 minutes. [13:50:31] Amir1, ^ [13:50:35] Woops. Double ping [13:51:42] https://phabricator.wikimedia.org/T206654#4658398 [13:52:07] so given what you said, once in a while a process goes 100% and stays there [13:52:13] then next one joins [13:52:32] that explains the jumps [13:53:12] halfak: We already do monitor CPU usage, just don't alert for it. And that's because experience has shown that those aren't really worth pursuing much. All that an alert like that ever tells you it's "something just maybe wrong" nothing more. Usually digging deeper to figure out what's going on is much more useful [13:53:33] +1 on the update of mwparserfromhell [13:54:13] adding some way for the celery master to kill runaway child process is bound to help here [13:54:46] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Upgrade mwparserfromhell in production. - https://phabricator.wikimedia.org/T206763 (10Ladsgroup) p:05Triage>03High [13:55:42] The only problem is that it requires deploying new version of revscoring and I don't want to do that because it would need rebuilding wikidata stuff and changing itemquality/articlequality [13:55:53] probably will backport it [13:57:10] Fixed a long-standing performance issue with deeply nested, invalid syntax [13:57:10] (issue #42). The parser should be much faster on certain complex pages. The [13:57:10] "max cycle" restriction has also been removed, so some situations where [13:57:10] templates at the end of a page were being skipped are now resolved. [13:57:18] https://github.com/earwig/mwparserfromhell/releases [13:58:21] Amir1, I think that rebuilding models should be do-able. [13:58:34] We should rebuild just in case mwparserfromhell changed some things. [13:58:47] I don't think we use mwparserfromhell for editquality [13:58:50] * halfak checks. [13:59:40] Hmm.. Maybe we do. [14:03:48] we already have the newest version of parserfromhell in revscoring [14:04:29] the wheels is also updated [14:05:23] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Ladsgroup) [14:05:29] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Upgrade mwparserfromhell in production. - https://phabricator.wikimedia.org/T206763 (10Ladsgroup) 05Open>03Invalid We already use 0.5.1, the latest version, in production. [14:08:02] Amir1, oh. Damn. [14:08:28] Next plan: Fix logging so we can figure out what revision is getting stuck. [14:10:20] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Halfak) From @akosiaris in chat ``` [06:45:33] mwparserfromhell honors it's name [06:45:43] (gdb) py-bt [06:45:43] let me dig in hadoop responses, if it's external we can get a hint by cross-examining with increases time-wise [14:11:59] halfak: can you give me UTC time of some of cpu jumps? [14:12:04] Amir1, I'm guessing it's changeprop. It keeps retrying the same revision [14:13:08] 2018-10-10 23:56:00 is when I first notice something on ores2003 [14:13:13] Amir1, ^ [14:13:56] noted [14:13:58] thanks [14:18:26] FYI, services switchover in 12 mins [14:18:40] Thanks akosiaris [14:18:59] I think I out to be working out why these jobs aren't timing out. [14:19:17] Also, changeprop should not infinitely retry the same revision. If it times out 5 times, stop trying. [14:22:22] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Halfak) @Ladsgroup just noted in IRC that we are running the most recent version of mwparserfromhell already (0.5.1). So I'm looking into why th... [14:26:24] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Halfak) We are *supposedly* using Unix signals to time out a processing job that takes too long. See https://github.com/wikimedia/ores/blob/master... [14:26:58] halfak: its maximum retries is 30 [14:27:05] gives up after 30 retries [14:27:13] We should cut that down, I think. [14:27:26] it's possible [14:29:04] in the ten minutes around the jump you mentioned, neither hadoop (external requests only) or logstash (internal and external) show sending any 5xx error [14:29:48] which probably means we don't even respond [14:30:50] Amir1, I think that uwsgi is certainly responding. [14:30:54] But celery gets locked. [14:31:18] uwsgi would just send a scoring timeout error in a 200 response. [14:31:28] As in an individual scoring job timed out. [14:31:31] it sends 503 [14:31:35] I changed it [14:31:41] no 504 [14:31:42] No way [14:31:45] sorry [14:31:55] Because you can have a multi-score job where some time out and others do not. [14:32:05] So sending anything other than 200 doesn't make sense. [14:33:01] hmm, let me check for proper responses [14:33:43] maybe 206? [14:34:22] That'd be OK I guess. [14:34:30] Either way, it won't solve the problem at hand. [14:34:49] I guess that change prop can read the error code and to know to stop requesting. [14:35:41] OK so I need a degenerate wikitext that will make mwparserfromhell spiral out of control so I can prove whether or not this Signal-based timeout thing is working or not. [14:35:52] Let's see if I can find an old bug. [14:37:26] https://github.com/earwig/mwparserfromhell/issues/190 [14:41:22] Hmm. Cannot replicate the issue with mwparserfromhell==0.5.0 [14:42:02] Let's try 0.4.4 [14:42:22] aha! [14:42:23] That works. [14:42:30] * halfak watches a single core go crazy [14:43:41] ...and we're able to timeout using Signals... so that's a dead end. [15:00:08] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Halfak) I've demonstrated that ORES timeout mechanism works for an old mwparserfromhell recursion bug in version 0.4.4 based on this report: https:... [15:01:06] Amir1, ^ either we *are* timing out on ORES end and ChangeProp is just re-submitting the job over and over again. Or the issue with parsing we are running into now is somehow special and the signal timeout doesn't work. [15:06:39] hmm let check configs [15:06:49] halfak: (if you’re not too busy putting out fires) not sure if you caught this yesterday, but yesterday during the tech engagement offsite I came up with this idea for using JADE as the backend for “did you find this page useful” feedback for docs on wikitech. We’re probably going to end up writing a gadget; question is whether the data posts to a judgment page or some other arbitrary wiki page. Scoring Platform Me is not too hot on giving it [15:06:49] priority, but could we consider it as a use case if it would not require much work? [15:07:43] (wikitech and possible mediawiki.org as well) [15:11:25] afk for lunch [15:34:08] 10ORES, 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Ottomata) @Pchelolo merging https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/439917/9 will requi... [15:37:29] 10ORES, 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Pchelolo) @Ottomata I think there should be 2 steps here - first we just stop emitting events completely -... [15:45:42] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Ladsgroup) I also checked if ores extension has changed the way it does requests and jobs. [[https://www.mediawiki.org/wiki/MediaWiki_1.32/wmf.22#O... [15:46:04] back now [15:55:42] going to step back for a bit, I don't feel alright [16:00:30] 10ORES, 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Ottomata) +1 plan sounds good to me. [16:29:36] 10ORES, 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Pchelolo) https://github.com/wikimedia/change-propagation/pull/295 [17:02:25] Amir1: maybe consider having a period of not-work between each work day :p [17:02:38] Drink some tea, I hope you feel better! [17:02:47] <3 thanks [17:02:52] Good idea [17:08:39] 10JADE, 10Scoring-platform-team, 10Discovery-Search, 10Elasticsearch: Extract judgment data for search indexing - https://phabricator.wikimedia.org/T206352 (10EBjune) p:05Triage>03Normal [17:23:30] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Merge logging config to the main config in ORES - https://phabricator.wikimedia.org/T206221 (10Ladsgroup) https://github.com/wikimedia/ores/pull/272 [17:28:52] wikimedia/ores#1057 (logging_config - 2a376b8 : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/ores/builds/440259081 [17:42:33] https://github.com/wikimedia/ores-wmflabs-deploy/pull/99 [17:42:58] The build failed because the branch doesn't exist anymore [17:44:26] hahaha [17:44:34] oops. [17:45:21] wait, I'm confused. That wasn't the PR I just merged, so why is the branch missing? [17:46:09] ah nvm, the last URL was something different. [17:46:10] we usually delete the branch after we merge :D [17:46:28] I did! [17:46:42] maybe nearly instantaneously after [17:46:51] yup [17:47:43] I see, the problem is just that I merged before the original Travis build finished. [17:58:12] (03PS1) 10Ladsgroup: Use the new ores logging config handling [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/466708 (https://phabricator.wikimedia.org/T206221) [18:01:45] https://github.com/wikimedia/ores/pull/270 [18:01:49] ahm ahm :D [18:01:59] (03CR) 10Ladsgroup: [V: 032 C: 032] Use the new ores logging config handling [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/466708 (https://phabricator.wikimedia.org/T206221) (owner: 10Ladsgroup) [18:02:24] (03CR) 10Awight: [C: 031] "Looks equivalent" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/466708 (https://phabricator.wikimedia.org/T206221) (owner: 10Ladsgroup) [18:13:17] o/ Amir1. I saw it. Will get to it. [18:13:33] Morning meetings done. Taking lunch. Will review when I get back ^_^ [18:13:35] Thanks [18:17:21] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10akosiaris) >>! In T206654#4658539, @Halfak wrote: > I've demonstrated that ORES timeout mechanism works for an old mwparserfromhell recursion bug i... [18:26:11] are you still using mwparserfromhell ? [18:26:48] halfak|Lunch: ^ [18:27:45] cscott: yup we do [19:11:48] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10Halfak) Yes. All calls to mwparserfromhell happen within that timeout() call. [19:15:40] cscott, we still haven't made the leap to parsoid output. We'd have to make a separate API call and that'd cost us some extra IO time. [19:20:26] halfak: alas. https://github.com/wikimedia/parsoid-jsapi can be run locally and is pretty much a straight port of the mwparserfromhell API. [19:21:04] halfak: i need to touch it up a bit, we forked it out of our codebase and I don't have docs being generated automatically any more (but the docs were pretty good) [19:24:30] akosiaris: fyi https://gerrit.wikimedia.org/r/c/operations/puppet/+/466716, This should not affect production as the code and final config changes has not been deployed to prod [19:26:13] halfak: awight please clean your stale branches: https://github.com/wikimedia/ores-wmflabs-deploy/branches/stale [19:28:39] Amir1: thanks for the nudge [19:29:11] Amir1: btw, this is the most fun and approachable patch in the CR queue, https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/464737/ [19:29:31] That strategy seems to give us all the integrations we've been worried about. [19:30:49] cool [19:30:54] Have you ever had that moment while working in MediaWiki code, where you think "hey, that's not completely arbitrary and it even works!"? [19:30:56] I have some nitpicks [19:31:01] :D please do [19:31:23] "hey, that's not completely arbitrary and it even works!" <- oh yeah :D [19:31:40] btw https://en.wikipedia.org/wiki/Nitpicking [19:32:28] not always so metaphorical :p [19:32:39] lol [19:35:05] Amir1, one of those is an open pull request. https://github.com/wikimedia/ores-wmflabs-deploy/pull/97 [19:35:13] (03CR) 10Awight: [C: 032] "Thanks, bulk update buddy!" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466141 (owner: 10Libraryupgrader) [19:35:41] halfak: it conflicts so it's not mergable at this shape [19:35:50] It's so old! [19:36:13] yeah :/ [19:36:31] same for staging, I think we deploy staging from master [19:36:37] and prod from deploy [19:38:37] cscott, re. parsoid-jsapi, is that actually running locally or just calling out to an API/ [19:38:43] Also, got apython version? [19:39:31] * halfak waits for ores-wmflabs-deploy to finish updating [19:39:58] I'm done for the day [19:40:13] have fun people [19:48:28] halfak: running locally, although it will call out to an API if you want to expand templates (no way around that) [19:48:41] halfak: no python version, but a php version is probably Coming Soon (tm) [19:48:56] o/ Amir1 [19:49:11] cscott, alas. That's a dead end for us then :| [19:51:43] OK this is ready for a merge now. https://github.com/wikimedia/ores-wmflabs-deploy/pull/97 [19:51:55] Looks like our wmflabs deploy is a bit behind on the configuration. [19:52:13] awight, ^ if you want to quickly merge that, I'll do a followup to match us to prod. [19:56:03] merging [19:56:37] thanks [19:56:41] Followup coming [19:57:41] halfak: If you find time for CR, https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/461502/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/464737/ are both regarding topics we've discussed lately, feel free to leave notes. [19:58:22] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for Galician Wikipedia - https://phabricator.wikimedia.org/T201146 (10Halfak) @Elisardojm OK! I finally have the translations deployed. I think we're ready for an announcement. [20:00:18] awight, I see you decided to not do oneOf. I'm OK with this, but I'm curious about your reasoning for the min: 1, max: 1 strategy [20:01:09] halfak: As I remember from yesterday, it communicates the same thing and I'm unclear on either one being a readability advantage. [20:01:14] Do you think it makes a big improvement? [20:01:36] Let's see what it looks like... [20:01:47] * awight jumps into letter-mopping suit [20:02:26] :D good news, this patch is a the top of the heap so no rebases will be harmed. [20:02:48] Not big. But yeah, I think oneOf was meant to make these situations explicit. [20:02:49] https://json-schema.org/understanding-json-schema/reference/combining.html [20:04:10] I think that in the end, it means the same thing. [20:04:36] (03CR) 10Halfak: [C: 031] "I've verified the changes to the v1 schema. But I'm not qualified to review the PHP changes." (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight) [20:05:05] (03CR) 10jenkins-bot: build: Updating npm dependencies for security issues [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466141 (owner: 10Libraryupgrader) [20:06:01] Good point. I don't like the new, boilerplate "type": "object", but the visual indentation due to oneOf makes sense. [20:06:43] Ah "properties" goes away, how strange. [20:07:06] That is strange. I didn't notice that. [20:08:33] say, I'm looking at how you handle value names of contentquality.. I'm wondering if quality scales with differing numbers of values will work. [20:08:37] E.g. enwiki has 6 [20:08:41] wikidatawiki has 5 [20:12:31] It's nice to look at these things wholistically. E.g. it looks like https://github.com/wikimedia/mediawiki-extensions-JADE/blob/master/jsonschema/judgment/v1.json#L55 will allow for a judgement that includes all three (damaging, goodfaith, contentquality). That doesn't quite seem right, does it? [20:12:56] It seems to me that we should be referencing a schema where damaging/goodfaith come together. [20:12:57] (03CR) 10Awight: Split user schema into ID or IP; validate (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight) [20:13:03] And where contentquality is entirely separate. [20:14:26] halfak: Some validation isn't appropriate at the schema level. For example, (schema x entity_type) compatibility is set by configuration. [20:15:00] Oh right. I'm just saying that we talked about combining damaging/goodfaith but not other types of potential diff schemas. [20:15:09] E.g. edittype. [20:17:35] That's my understanding. [20:17:41] halfak: See https://phabricator.wikimedia.org/diffusion/EJAD/browse/master/includes/JudgmentValidator.php$103-116 [20:17:47] Maybe an example of what I expected would be helpful. [20:18:00] that's the code that disallows contentquality for a Diff judgment page. [20:18:38] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for Galician Wikipedia - https://phabricator.wikimedia.org/T201146 (10Elisardojm) According to https://labels.wmflabs.org/stats/glwiki/, we need 37 campaigns more, isn't it? I don't kn... [20:18:50] I'll take your suggestion about (damaging, goodfaith) separately, and yes fully agree that we can combine these into one schema. It'll make the point that we don't just deal in scalars. [20:20:01] halfak: lmk your opinion of the snippet in my comment https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/461502/4/jsonschema/judgment/v1.json@90 [20:20:54] (03CR) 10Halfak: [C: 031] Split user schema into ID or IP; validate (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) (owner: 10Awight) [20:21:01] Looks good to me. [20:25:32] halfak: you're using one of the python ml libraries? or is it stuff you've rolled yourself? [20:25:55] We've rolled a lot of stuff. But yeah. Also python is great for ML libraries. [20:26:10] halfak: i think the "proper" solution is for mediawiki-core to feed you edits in a parsed form [20:26:12] I don't think Node or PHP are good at all for ML [20:26:23] cscott, yes. That would be amazing. [20:26:53] halfak: latency is a concern, right? but parsoid is triggered immediately on edit anyway, and stashes its parsed output away as soon as it has computed it [20:27:12] halfak: are you actually looking at syntax features? or are you just extracting the text? [20:27:30] Right now, we work with text and parse it. [20:27:39] Does parsoid keep historical parses? [20:27:42] We score old edits [20:29:26] parsoid does not store anything [20:29:45] restbase would be the place were the html of old revisions could be stored [20:30:00] but that's opportunistic (it's more like a cache in this respect) [20:30:10] with retention policies and evictions and so on [20:31:30] but calling out to restbase could mean an implicit calling out to parsoid if restbase does not have the html [20:33:26] That could work. You might find that ORES users DOS you by scoring large swaths of Wikipedia history [20:33:39] that's a concern I do indeed have [20:33:39] ^_^ [20:34:04] Do you think that parsoid is more efficient than mwparserfromhell? [20:34:25] hahaha ^ that one is a candidate for the Bash page [20:34:29] Because we can keep up with *our* parsing. But then again, we don't expand templates. [20:34:41] lolol [20:34:51] I have no data about that tbh [20:35:11] not even hearsay [20:36:07] AFAIMC, if anything the gain would be in using the already generated html stored in restbase [20:39:46] for what is worth I am failing up to now to get a proper python bt in gdb.. I have bt of ~300 frames and I have no idea what's going on in python yet [20:41:16] Amir1, was checking on hadoop. I don't think he had any success. [20:41:49] I think that might be our best bet of getting the rev_id that is causing this (assuming my hypothesis is right) [20:41:59] And once we have that, we can try to reproduce the problem. [20:42:19] I still don't have a way to explain how our timeouts might not be working. [20:42:51] that one has me puzzled too [20:44:08] I really think we should be catching this with signal. [20:44:13] text='== Participaci\xf3n ==\n\n{| class="wikitable" class="sortable wikitable"\n|-----\n!width=4%|A\xf1o!!width=12%|Pa\xeds que representa !!Cantante(s) !!width=15%|Canci\xf3n
(M\xfasica / Letra)!!width=6%|Final!!width=4%|Pts.!!width=2%|Semifinal!!width=5%|Pts.!!width=6%|J!!width=6%|Pts!!width=6%|T!!width=6%|Pts!!width=6%|Odds!!width=4%|{{bandera|Spain}}!!width=10%|{{bandera|Spain}}12?\n|-\n| [[Festival de la C [20:44:13] anci\xf3n de Eurovisi\xf3n 2877|2877]]\n| {{bandera|Israel}} \'\'\'[[Israel]]\'\'\'\n| style="background:" |\'\'\'[[Daniel Alfredo]]\'\'\'\n|[[Fly until ma own|\'\'\'\'\'Menos (Low)\'\'\'\']]\n| [20:44:14] 10[1] 04https://meta.wikimedia.org/wiki/Template:bandera [20:44:15] 10[2] 04https://meta.wikimedia.org/wiki/Template:bandera13 => [20:44:18] does this help at all ? [20:44:18] 10[3] 04https://meta.wikimedia.org/wiki/Israel13 => [20:44:21] halfak: Remind me, you wanted damaging and goodfaith to both be required for an editquality judgment? No rush. [20:44:21] 10[4] 04https://meta.wikimedia.org/wiki/Daniel_Alfredo13 => [20:44:24] 10[5] 04https://meta.wikimedia.org/wiki/Fly_until_ma_own [20:44:30] ah I triggered by mistake asimovbot ? [20:44:31] hehe [20:44:48] I think I just got something for the very first time in gdb [20:45:03] awight, yeah. I have a thingie. Will share. [20:45:10] so it seems, akosiaris [20:45:11] akosiaris, that could be useful! Let me check in a minute. [20:46:55] akosiaris: that wikitext comes from https://es.wikipedia.org/wiki/Usuario:Danielalfredo/Taller [20:47:12] awight, https://gist.github.com/halfak/2308f2b32f818582a97343715e8bad7b [20:47:20] Platonides: you are fast! [20:47:25] hehe [20:47:27] Thank you Platonides [20:48:16] that 2877 Eurovision Song Contest link was flashy [20:48:31] 10ORES, 10Scoring-platform-team (Current): ORES workers using dramatically higher CPU, increasing linearly with time - https://phabricator.wikimedia.org/T206654 (10akosiaris) I 'm adding some extra gdb info in P7669 [20:48:53] halfak: hehe I was literally typing that exact same code [20:49:02] :) [20:50:04] Well... We can run mwparserfromhell on that page no problem [20:50:05] :( [20:50:15] it could be some old revision [20:50:22] The explicit schema is great for giving us granular control over rules. [20:50:46] * halfak tries the the version from Sept. 22nd -- when the problem started. [20:50:51] * akosiaris same thing [20:51:15] Also parse-able [20:51:17] hmm [20:51:20] dammit [20:51:23] Right [20:54:01] akosiaris, maybe not this text? [20:55:05] Hmm. Looks like we *do* timeout when parsing this with mwparserfromhell==0.5.0 [20:55:18] We *should* be running mwparsefromhell==0.5.1 [20:55:19] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466747 (owner: 10L10n-bot) [20:55:21] I'ma check [20:56:15] frozen-requirements.txt:mwparserfromhell==0.5.1 in the current deploy repo [20:56:18] so it should indeed [20:56:41] yeah. Confirmed on ores2003 [20:56:44] 0.5.1 [20:57:38] Well damn [21:01:42] the wheels repo shows that we last deployed that wheel in April, [21:01:42] mwparserfromhell-0.4.4-cp35-cp35m-linux_x86_64.whl | Bin 164558 -> 0 bytes [21:01:45] mwparserfromhell-0.5.1-cp35-cp35m-linux_x86_64.whl | Bin 0 -> 179637 bytes [21:08:39] (03PS1) 10Awight: Always require damaging and goodfaith together [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466781 [21:17:14] (03PS2) 10Awight: Always require damaging and goodfaith together [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466781 [21:17:16] (03PS1) 10Awight: Followup: convert user subschema to use oneOf [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466786 [22:03:32] 10Scoring-platform-team, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for Galician Wikipedia - https://phabricator.wikimedia.org/T201146 (10Halfak) When this campaign is complete, we'll be able to build an article quality prediction model in ORES for glwik... [22:03:37] I'm out of here. Have a good one folks! [22:14:16] (03PS3) 10Awight: Always require damaging and goodfaith together [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466781 [22:14:19] (03PS2) 10Awight: Followup: convert user subschema to use oneOf [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466786 [22:14:20] (03PS5) 10Awight: Split user schema into ID or IP; validate [extensions/JADE] - 10https://gerrit.wikimedia.org/r/461502 (https://phabricator.wikimedia.org/T206573) [22:14:22] (03PS7) 10Awight: Render Judgment pages as wikitext [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464737 (https://phabricator.wikimedia.org/T206346) [22:14:26] (03PS7) 10Awight: Tests to demonstrate SpamBlacklist integration [extensions/JADE] - 10https://gerrit.wikimedia.org/r/464727 (https://phabricator.wikimedia.org/T206255) [22:14:43] ^ Reordered with schema stuff first. [22:39:42] * awight stands still and listens to the howling wind [22:43:40] nice to see that the 1 kloc secondary schemas patch doesn't conflict with any of these! [23:00:42] (03PS44) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [23:00:44] (03PS1) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466804 (https://phabricator.wikimedia.org/T202596) [23:00:46] (03PS1) 10Awight: PSR-4 autoload tests by namespace rather than listing each class. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466805 [23:00:48] (03PS1) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [23:01:06] (03PS45) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [23:01:08] (03PS2) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [23:02:08] (03Abandoned) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466804 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:03:30] (03PS46) 10Awight: Secondary schema for JADE indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) [23:04:58] (03PS2) 10Awight: PSR-4 autoload tests by namespace rather than listing each class. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466805 [23:05:00] (03PS3) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [23:05:02] (03PS1) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [23:09:32] (03CR) 10Awight: "PS 46: split out the hooks and maintenance scripts into separate patches, in an act of mercy towards would-be code reviewers, and at the h" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/456078 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:16:11] (03CR) 10jerkins-bot: [V: 04-1] PSR-4 autoload tests by namespace rather than listing each class. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466805 (owner: 10Awight) [23:16:45] (03CR) 10jerkins-bot: [V: 04-1] PSR-4 autoload tests by namespace rather than listing each class. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466805 (owner: 10Awight) [23:17:50] 10JADE, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 3 others: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10awight) @Marostegui These are the proposed indexes, if you want to discuss something concrete: https://gerrit.wikimedi... [23:30:40] (03PS3) 10Awight: PSR-4 autoload tests by namespace rather than listing each class. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466805 [23:30:42] (03PS2) 10Awight: Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) [23:30:44] (03PS4) 10Awight: Hooks to maintain judgment link tables [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466806 (https://phabricator.wikimedia.org/T202596) [23:31:16] o/ [23:36:26] (03CR) 10jerkins-bot: [V: 04-1] Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [23:42:22] (03CR) 10jerkins-bot: [V: 04-1] Maintenance scripts for judgment indexes [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight)