I'm hoping maybe we can reduce it to a simple model and move the html wrapping back to PHP. I hope the iteration over the lines and wrapping is simple enough to do fast enough in plain PHP [05:36:17] yeah, it is faster if you format on the C++ side [05:37:04] g2g [16:07:48] Need to do the same thing to 6 documents.... breaking up lines [16:07:52] Fuck doing it manually [16:39:46] Reedy: possibly relevant -- https://xkcd.com/1319/ [16:40:06] hahah [16:40:15] I'm done with it now.. I think [16:40:19] Unfortunately GIGO [16:40:26] So we have missing data to begin with [16:40:40] Because, you know, who needs part numbers on production drawings for parts used for gas meters [16:41:28] And then, we won't be consistent anyway with descriptions [16:45:55] I've got an UBN! logging issue that could use code review -- https://gerrit.wikimedia.org/r/#/c/251679/ [16:46:01] ori, Krinkle: ^ [16:47:50] Checking [16:48:05] thanks [16:48:10] Aye, it changed [16:48:46] yeah I got fancier than PS1 rpoposed [16:49:36] I started working on a Filter to make things UTF-8 safe and then realized that it was an edge case better handled in the Formatter on failure [16:50:52] I don't feel good merging, but that's because I don't know the area very well. [16:50:55] Looks good though. [16:53:07] Krinkle: thanks. I can probably get ori or legoktm to merge if they agree [16:53:47] also, you ask a lot of logging questions so I should try to make you feel more comfortable with the stack we use [16:54:35] I suppose a good place to start would be to finally get around to documenting how WMF uses Monolog in prod [16:56:55] Reedy: What was the outcome of enabling error channel (w stacks) [16:57:06] I've not had chance [16:57:08] Been busy most of the day [16:57:31] Oh, I recall you having a patch [16:58:37] Yeah [16:58:45] Not had chance to enable it, monitor for 5 or so and disable again [17:00:01] Reedy: Could also start by enabling it for mw1017 dynamically and leave it there (flourine only) [17:00:39] https://wikitech.wikimedia.org/wiki/Debugging_in_production#Accessing_mw1017_with_a_magic_header [17:03:32] Or even let it to go logstash, why not. Something like channel error = array( enable => wfHostname === .. ) [17:57:24] bd808: reviewing [17:57:40] oh, I saw this yesterday [17:57:48] I still don't understand what about this problem is unique to mediawiki [17:58:03] in other words, why is not an upstream issue that should be fixed upstream? [17:58:11] upstream made it a runtime error on purpose [17:58:21] I'm not entirely sure why [17:58:42] i suspect that they just don't have our volume of traffic so they don't run into crazy encoding issues as reliably as we do [17:59:31] i'm not opposed to having a temporary workaround in our codebase if we can agree that it's temporary and at least start a convo with upstream [17:59:52] csteipp: Ping for basic status on the SessionManager security review. [17:59:55] or maybe this problem is specific to us or specific to mediawiki in a way i'm not seeing, in which case please explain [18:00:17] ori: well, see https://github.com/Seldaek/monolog/issues/545 [18:00:32] they did it on purpose [18:00:58] yeah, but that doesn't mean they thought it through [18:01:03] "Would it perhaps not be better to throw an error in toJson()?" [18:01:21] "no, it wouldn't, because such issues exist in the wild and there isn't much you can do about them; they are not an indication of an application error." [18:01:27] * YuviPanda nervously waits for talk [18:01:35] "deciding on a way to normalize them so they can be serialized would be a better approach" [18:02:51] errors should be actionable, right? I don't see value in emitting errors because a remote user has a shitty browser [18:03:44] again, my style is to come out swinging, but i'm not gung ho about this, if i am not seeing something here please let me know [18:03:51] anomie: Should be done this week [18:03:54] ok [18:04:18] ori: I'm going to poke them about the decision upstream [18:04:29] but I think we should fix now if we can [18:04:29] thanks [18:04:35] OK [18:04:56] let me look it over one more time to see if there are any silly errors and i'll merge it if it looks good [18:07:31] it's a shame json doesn't have a bytes type [18:07:52] json is a shame [18:08:11] you've been hanging out with tim too much :P [18:08:59] lack of comments was a dumb design decision [18:52:39] imma kill sync-dblist. [18:52:46] It's useless now that we have them all in a single dir. [18:53:19] jfdi [18:53:37] I wrote a patch, just needs someone to accept [18:54:07] https://phabricator.wikimedia.org/D43 [18:56:03] ewww, Differential sucks [18:56:41] repository name is somewhere below and you need to remeber all those crazy acronyms [18:57:05] Those acronyms are going away soon [18:57:19] TAaGAS [18:58:06] (I like Differential, FWIW) [19:40:06] ori, bd808: I wonder if we could move CDB development to Phab. It's not updated often and it'd be good to get a least one MW library in that pipeline soon. [19:41:54] fine by me. AhoCorasick would be fine too -- I kept it around because it could be useful for people, but we didn't end up using it ourselves [19:42:00] RunningStat too [19:42:12] RunningStat we are using, but I'd be fine with it in Phab [20:18:53] ostriches: sure. I'm not sure that CDB is going to see a lot of development though really [20:20:42] ori: Upstream at monolog they think we should just wrap everything up in https://github.com/Seldaek/monolog/blob/master/src/Monolog/Handler/WhatFailureGroupHandler.php [20:21:00] it's an error suppressing handler [20:21:54] More thought about getting something moved so we can get composer and the vendor submodule, etc all working right with Phab as needed. [20:23:07] bd808: seems Seldaek still hasn't considered your solution (to normalize/escape the encoding) [20:23:14] prpose that maybe? [20:23:48] I guess that means I need to get it to work on php5.3 first :/ [20:24:09] * bd808 blames iconv integration bugs [20:24:20] could just propose it and see if he thinks it's a good idea first [20:49:01] ori: the other scrubbing thing I was thinking about was more along the lines of what the less command does. When it sees a byte that is invalid UTF-8 it outputs '` where XX is the hex value of the offending byte. [20:49:29] that would be more reliable than the iconv stuff seems to be [20:50:33] I think I could whip up a small library that just did that with reasonable performance [22:06:37] AaronSchulz, hoo: do you know why there would be queued LocalRenameUserJob for over 24h that still haven't run yet? https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Zaher.Kadour is stuck, I manually ran the arwikisource one just now and it was fine... [22:06:58] legoktm@mw1152:~$ mwscript showJobs.php --wiki=aswikisource --group [22:06:58] LocalRenameUserJob: 1 queued; 0 claimed (0 active, 0 abandoned); 0 delayed [22:07:50] hm [22:07:59] will have a quick look [22:08:37] I didn't see anything in exception logs that indicated that they had run once and failed [22:08:57] (and by manually run, I mean runJobs.php --type=LocalRenameUserJob) [22:11:31] the job has attempts=1 [22:11:37] So it failed before [22:14:07] o.O [22:14:25] hoo: where do you see that? [22:14:42] legoktm: showJobs.php --list [22:14:55] aha. [22:15:03] okay, but shouldn't it have been re-queued then? [22:15:27] It has [22:15:35] Well, it's still in the queue [22:17:21] so...the job queue is just behind? :/ [22:17:45] It will pick it up again at some point [22:17:49] might take a bit though [22:17:55] I'm not that into the details [22:18:01] mwscript extensions/WikimediaMaintenance/getJobQueueLengths.php | grep aswiki [22:18:03] gives nothing [22:18:10] so it's not in the "queues with jobs" hash [22:18:47] Oh, I see [22:19:07] csteipp: https://gerrit.wikimedia.org/r/252339 [22:19:19] in theory, that eventually gets rebuilt by jobchron, maybe it could just be made less racey to begin with though instead of relying on that, which can take a while (though it shouldn't take days either) [22:22:11] is it alright if I run a foreachwiki loop to clear out the individual LocalRenameUserJob queues? or should I leave it alone for debugging? [22:23:18] AaronSchulz: ^ [22:36:52] legoktm: https://gerrit.wikimedia.org/r/#/c/252342/ [22:39:00] * legoktm learns about redis's WATCH [22:41:16] AaronSchulz: I assume that won't retroactively fix this situation? You said jobchron was supposed to rebuild it though... [22:41:33] I'm trying to rebuild the hash but now it's stuck empty [22:41:37] * AaronSchulz will have to hack around more [22:41:52] ok [22:41:59] * legoktm posts a quick update on-wiki [22:43:22] Thanks :) [22:44:36] * AaronSchulz manually deletes the :lock key and rebuilts [22:47:15] legoktm: ok, done, aswikisource is there now [22:47:25] yay [22:47:44] https://grafana-admin.wikimedia.org/dashboard/db/job-queue-health [22:47:50] haha, the 0 was me, it will go up again ;) [22:47:51] will the other wikis begin to just work, or will it require manual fixing? [22:48:51] they should have their jobs popped now [22:49:33] > There are no renames in progress for Zaher.Kadour. They may have already finished. [22:49:35] yay, thanks :D [22:50:08] cursive: ^ [22:51:33] Great, thanks AaronSchulz et. al [22:56:03] legoktm: I bet lots of jobs on rdb3 (1007) didn't end up in the hash since jobchron had to be restarted manually today [22:57:31] so the background script never added them (not sure why the notifyNonEmpty calls didn't work though, maybe the tried to pop() before mw was aware of 1007 since workers run for a min or two before exiting) [22:57:38] *they tried [22:57:56] ahhh