[14:51:10] I encountered a weird issue upgrading from 1.41.4 to 1.43.0 just now; I feel like T299947 migrations are perhaps applied out of order? one of them adds pl_namespace column/index (and some other stuff), the other removes the same pl_namespace column/index. I am getting DB errors from jobs stating the column does not exist... [14:51:11] T299947: Normalize pagelinks table - https://phabricator.wikimedia.org/T299947 [14:51:49] the files are patch-pagelinks-drop-pl_title.json and patch-pagelinks-target_id.json [14:53:38] [2024-12-30T14:52:08.272257+00:00] exception.ERROR: [720bc3307854735a80ef10ce] [no req] Wikimedia\Rdbms\DBQueryError: Error 42703: ERROR: column "pl_namespace" does not exist LINE 1: ...ate\GenericPageLinksTable::fetchExistingRows [15:04:27] disregard the out of order part, it looks like it is supposed to be removed by 1.43.0... time to figure out why it is still referred to [15:08:07] outdated extension? [15:08:43] path was truncated so I couldn't see which file generated that error [15:09:33] includes/deferred/LinksUpdate/GenericPageLinksTable.php [15:10:01] [15:08:07] outdated extension? <- I used REL1_43 versions for everything :\ [15:10:45] hmm I think I found the reason, thank you [15:11:44] the 'path' part was what tipped me off — it seems like the job runner was somehow stuck in the past [15:18:59] I guess with this resolved I should find out why TimedMediaHandler ignores $wgFFmpegLocation in some cases but not others [15:20:23] I have it set correctly to /usr/local/bin/ffmpeg, and after setting it I am getting proper thumbnail images, but if I enable transcoding I get exec.WARNING: Possibly missing executable file: '/usr/bin/ffmpeg' in the error log [15:24:06] (adding a symlink solves the issue but I do not like this) [15:24:14] is there a stacktrace along with it? [15:25:19] sadly no, this is just a warning from the job :( [15:25:57] oho, but maybe the cause is the same [15:26:15] which proves that I am an idiot, thank you mooeypoo [15:26:19] moonmoon* [15:27:10] * Remilia needs to pay more attention to the complete log messages :( [15:27:29] going to double check this now [15:27:50] incomplete upgrade? a second install somewhere for job runners that wasn't kept in sync with your main install? [15:27:52] * moonmoon is curious :P [15:28:08] the job runner script does not cd in the loop :D [15:28:31] so when I `mv w w.old` it kept running inside that [15:29:02] aha [15:29:30] well glad you got it sorted ;) [15:29:34] but on the other hand the timedmediahandler issue is from before I updated [15:30:12] as far as I can tell, the only thing that hardcodes /usr/bin/ffmpeg is inside the test suite [15:30:18] everything else uses $wgFFmpegLocation [15:30:47] yeah I came to the same conclusion :\ [15:31:07] but I think the default is /usr/bin, which could also imply that the job is for some reason not using your LocalSettings? which would be a bit odd [15:31:23] or if you have a wikifarm set up it is potentially using the wrong LocalSettings [15:32:14] this one is not part of a farm, so I'm currently waiting for transcode timeout to check again [15:32:46] hmm [15:34:50] I generally encounter more issues than most because I run it on FreeBSD + PostgreSQL haha [15:37:55] has the (semi-)recent change to abstract schema helped on the postgres front? [15:42:06] honestly I have not had significant issues with postgresql for a while now, the only one that spams the error log is SpecialMobileHistory with incorrect timestamp format [16:16:18] moonmoon: I noticed that REL1_41 TimedMediaHandler outright uses globals; maybe something was weird with that somewhere… the one from REL1_43 tag uses $config and seems to have no issues with the FFmpegLocation setting — I just checked and everything works without a symlink [16:19:05] shouldn't have made a difference since all config is exported to globals anyway [16:19:36] but who knows, there might've been a phab issue about it but I'm honestly not curious enough to go digging :< [16:19:56] would've been more curious if it was still broken in 1.43 [16:20:28] 1.41 is EOL so who cares haha [17:57:01] I'm trying to use MUST and SHOULD as detailed at https://www.mediawiki.org/wiki/Help:CirrusSearch/Logical_operators#Lucene,_MUST,_and_SHOULD , but it seems like the words are being interpreted literally; am I missing something? [18:04:44] ZLima12: did you note the bold section of your link? "Note that CirrusSearch does not currently support MUST or SHOULD operators in user queries." [18:05:24] moonmoon: ah. thanks [18:07:12] it seems AND and OR work, but get converted to MUST/SHOULD internally via non-intuitive rules documented by that URL [18:09:10] in general it appears that complex filters are just not really doable [18:09:36] moonmoon: hmm, I'm not seeming to have luck with OR at least [18:10:01] what is the search you are attempting? [18:12:19] actually, it seems like it worked when I used Special:Search, but not Special:MediaSearch (this is on commons) [18:41:53] moonmoon: I remembered the most current issue with postgresql while upgrading the other 4 wikis; https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/tags/1.43.0/maintenance/postgres/archives/patch-pagelinks-drop-pl_title.sql is missing IF EXISTS for at least some of the statements (lines 5-8 inclusive for upgrading from 1.42, as an example) [18:42:31] `run.php update` bails on these [18:45:49] the updater does checks a bit differently; rather than encoding it in the file it's supposed to check that before executing the file in the first place [18:47:00] (but it also might just be my cursed installs really, one of them had been getting upgrades since 2009, others since 2011, 2013, 2017 and 2020) [18:47:19] moonmoon: the other statements in these files still apply [18:48:04] unless the updater knows to skip parts of it hmm [18:48:36] in this particular case, for my 2020-installed wiki that was on 1.41, only the first statement caused an error [18:48:57] for the 2009 wiki running 1.42, it was the first 4 lines (3 statements) [18:49:30] it'll run the file if it detects that pagelinks.pl_title exists [18:49:53] unfortunately if the db isn't in the expected "before" state then things get messed up and may need manual intervention :( [18:49:55] yeah that one existed in all cases, but pl_namespace did not [18:50:05] yep [18:50:39] I'm guessing it's all the old Postgres issues accumulating that results in schema inconsistencies [18:50:48] probably [18:50:54] but I've seen it happen in mysql too [18:51:53] I will forever curse the day I let myself get convinced into installing Cargo on one of the wikis though, that was a huge mistake which cannot be undone [18:52:28] you could try to dump table schema to a file and then diff it against maintence/postgres/tables.sql maybe to see what else might be off [18:53:11] oh, good idea, thanks for the suggestion [20:38:37] one question that has been plaguing me for a few years now is how to debug 'Duplicate get()' warnings for caches, happens with both CACHE_DB and CACHE_MEMCACHED; there are no mentions of this in search results… but maybe these can just be ignored :\