[00:30:15] 3MediaWiki-Core-Team, Librarization, MediaWiki-General-or-Unknown: [Regression] MediaWiki should detect absent or outdated vendor - https://phabricator.wikimedia.org/T74777#989369 (10Cookjamesmw) I'm getting exactly this error. In the apache logs: ``` Class 'MWLogger' not found in ... mediawiki/includes/Globa... [00:51:03] 3MediaWiki-Core-Team, Librarization, MediaWiki-General-or-Unknown: [Regression] MediaWiki should detect absent or outdated vendor - https://phabricator.wikimedia.org/T74777#989384 (10Cookjamesmw) Of course I forgot to symlink to the new "vendor" directory from the wiki instance. Duh! Writing the comment above he... [05:28:54] legoktm: about? exception-json log (and logstash) are full of "Unrecognized job type 'LocalRenameUserJob'." [05:29:11] Any idea what we did wrong? [05:31:52] umm [05:31:58] looking [05:32:05] me too [05:32:18] $wgJobClasses['LocalRenameUserJob'] = 'LocalRenameUserJob'; [05:32:24] look right to me [05:32:46] but I really don't know much about this [05:33:02] s/this/the job queue and runner/ [05:34:16] bd808: oh, this is vewikimedia [05:34:18] this may be what is killing logstash [05:35:19] oh.. yeah all of them are the same wiki [05:35:34] so, the CentralAuth db had references to a "vewikimedia", which at one point was connected to SUL. It was recently re-opened as a fishbowl, meaning it's not SUL and CA isn't installed. But a user who the db thought existed on vewikimedia was global renamed, and it queued a job on vewikimedia except the job class doesn't exist there [05:35:59] 10000 log messages in the last 50000 exception.log lines [05:36:10] all for that [05:36:20] I fixed the db but never deleted the job... [05:36:27] I had assumed it would give up after 3 tries [05:36:37] apparently not so much [05:37:25] legoktm@terbium:~$ mwscript showJobs.php --wiki=vewikimedia --group <-- outputs nothing [05:38:17] hmmm... one job popping and pushing so fast you'd have to catch it between iterations? [05:39:13] just tried 15 times, nothing showed up :P [05:40:44] tail exception.log [05:41:19] holy crap [05:41:31] we may need to do surgery on redis to kill this off [05:41:41] can we manually just drop the entire job queue for vewikimedia? [05:41:51] probably [05:42:04] I did something like that for beta before [05:42:21] Aaron's help would be nice but... [05:43:10] ori: about? Have you poked manually at jobrunner queues in redis? We'd got a stuck job that needs to be eliminated [05:44:14] * bd808 goes to look at redis in beta and see if he can remember how this works [06:10:14] the runner trying to pop the job out of the queue is what throws the exception and so it stays stuck because it can't be marked as bad [06:10:39] so we have to get the list key and drop it via redis-cli or similar [06:11:11] print_r( JobQueueGroup::singleton()->get('LocalRenameUserJob')->getSize() ); show it in the list [06:11:21] but pop() throws the exception [06:18:33] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989647 (10bd808) 3NEW [06:20:43] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989655 (10bd808) [21:35] < legoktm> so, the CentralAuth db had references to a "vewikimedia", which at one point was connected to SUL. It was recently re-opened as a fish... [06:21:40] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989656 (10bd808) The crazy volume of this exception event may be what is killing logstash as well. [06:21:41] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989659 (10Legoktm) (See also {T87264}) [06:23:43] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989662 (10bd808) I tried this: ``` $ mwscript eval.php --wiki=vewikimedia > print_r( JobQueueGroup::singleton()->get('LocalRenameUserJob')->getSize() ); 1 > print_r( JobQu... [06:25:46] legoktm: I've got no other tricks up my sleeve. I think we need Aaron or another redis job queue guru [06:26:56] me neither [06:26:59] my eval.php attempt didn't seem to do anything. print_r( JobQueueGroup::singleton()->get('LocalRenameUserJob')->getSize() ); is printing 0 now for me but the errors are still spewing [06:28:38] I don't think it's hurting anything other than logstash at the moment [06:28:51] ok [06:28:53] I'm going to sleep then [06:28:58] do that [06:29:03] see you tomorrow [06:29:07] o/ [07:30:25] bd808|BUFFER: hey, i missed your pings [07:30:35] i'm actually in the lobby of CQ but going to head home now, I think [07:30:44] there are docs on wikitech about purging jobs from redis [07:31:05] but if it's not hurting anything other than logstash than yeah, let's not do anything about it until tomorrow [14:13:45] ori, about? [14:13:50] got a quick hhvm/job question [14:18:44] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989960 (10Krenair) @tgr found a way to kill jobs from redis for T87040#984282 (path refers to tin), perhaps the same thing could be used here. [14:23:58] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#989966 (10Krenair) Although getSize now returns 0 for me, the exception log is still getting flooded. [15:43:35] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#990005 (10Glaisher) Just deleting vewikimedia would solve this problem? It's currently not in use and they requested that it should be a redirect to their current wiki inst... [16:05:10] 3MediaWiki-Core-Team, SUL-Finalization: Bad LocalRenameUserJob stuck in jobrunner for vewikimedia - https://phabricator.wikimedia.org/T87360#990013 (10bd808) >>! In T87360#989960, @Krenair wrote: > @tgr found a way to kill jobs from redis for T87040#984282 (path refers to tin), perhaps the same thing could be us... [18:28:33] 3MediaWiki-Core-Team, Wikimedia-General-or-Unknown: Users are unable to change account password (HTTP 503) - https://phabricator.wikimedia.org/T87305#990090 (10hoo) Is this a problem for anyone still? If not, I'd vote to close this bug. [18:48:19] 3MediaWiki-Core-Team, MediaWiki-extensions-TitleBlacklist: Title blacklist intermittently failing, allowing users to edit things they shouldn't be able to - https://phabricator.wikimedia.org/T85428#990099 (10tstarling) 5Open>3Resolved Confirmed, no instances of an empty blacklist being saved to cache were lo... [22:32:14] 3MediaWiki-Core-Team, MediaWiki-extensions-OAuth: Support a nice sso experience with MediaWiki's OAuth - https://phabricator.wikimedia.org/T86869#990365 (10csteipp) [22:49:23] 3MediaWiki-Core-Team, MediaWiki-extensions-OAuth: Support a nice sso experience with MediaWiki's OAuth - https://phabricator.wikimedia.org/T86869#990400 (10csteipp) [22:53:34] 3MediaWiki-Core-Team, MediaWiki-extensions-OAuth: Support a nice sso experience with MediaWiki's OAuth - https://phabricator.wikimedia.org/T86869#990402 (10csteipp) [23:35:56] 3MediaWiki-Core-Team: Create separate job loop for LocalRenameUserJob - https://phabricator.wikimedia.org/T87397#990437 (10Legoktm) 3NEW [23:37:29] 3MediaWiki-JobRunner, MediaWiki-Core-Team: Job runner inifinite loop when job class does not exist - https://phabricator.wikimedia.org/T87398#990444 (10Legoktm) 3NEW