[00:00:05] I used to hear that it was complicated in the traditional mysql databases [00:01:05] no, just mysql -B SHOW TABLES | sed 's/.*/RENAME TABLE \0 TO newwiki.\0;' | mysql [00:01:58] centralauth etc. also complicates it [00:02:14] also external store? [00:02:16] and not just node.js services, every service adds a step, whether it is redis or memcached [00:02:43] external store is just that same one-liner [00:02:54] but you have to run it on each ES cluster [00:03:59] https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/renameWiki.php [00:04:05] It's almost like someone thought about this [00:04:46] yeah, if only it was that simple, right? [00:05:24] :) [00:05:36] maybe it should call a hook so that extensions can update their own data [00:05:48] Plus all the services... Anything that stores dbnames (CentralAuth.. Wikibase) [00:06:32] you know the no/nb background right reedy? [00:07:44] I've got a few norwegian friends... And when I've mentioned Bokmal vs Nynorsk [00:07:50] It does illicit some interesting responses [00:08:36] we were asked to make a norwegian wikipedia at no.wikipedia.org, and we said fine [00:08:45] I mean, we have macrolanguage wikis like zh that worked out just fine [00:09:16] but on nowiki it didn't take long for them to write policies saying you must only write in proper bokmal [00:09:38] and nyorsk speakers should go get their own wiki [00:10:36] This sounds familiar, yeah [00:10:55] so a couple of months ago, someone changed the no autonym to "norsk" instead of "norsk bokmal", because that is the correct and proper autonym according to ISO or something [00:11:05] thereby breaking all sidebars [00:11:31] and we cannot change it back because that would be a crime against ISO which you know is almost like a crime against humanity [02:07:13] legoktm: does it make sense to introduce some default firejail restriction level? [02:07:58] MaxSem: you mean actual on by default and not ->restrict( Shell::RESTRICT_DEFAULT ) ? [02:08:19] yep, if no restrict() was called [02:11:43] maybe, I think RESTRICT_DEFAULT is reasonably safe for most things already. But I'd like to get it in usage by everywhere we shell out in Wikimedia first to see what breaks - which I think we can have done before it comes close to 1.31 release time [02:15:11] we can make it configurable and try tweaking it to see if shit breaks [02:15:48] I would suspect that RESTRICT_NETWORK is one thing we'll especially want [02:15:48] also I think ->whitelistPaths() is going to be one of the more important things to get right, since that will take away the possibility of LocalSettings/PrivateSettings disclosure [05:15:56] legoktm: Ping re https://github.com/JetBrains/phpstorm-stubs/pull/289/files ; did you find out if there's something we can do about this on the phan side? Or do we just have to convince upstream and/or fork it? [15:58:58] + $this->localBootstrapVendorPath = __DIR__ . '/../../../../../vendor/twbs/bootstrap'; [15:59:00] That looks robust [16:47:44] What's the command from deployment-tin to make something log to Beta Cluster's SAL? [17:27:23] anomie: there isn't one. instead you manually !log in either #wikimedia-cloud or #wikimedia-releng with "!log deployment-prep ..." [17:27:44] we have never setup an ircecho service in the beta cluster [17:28:14] bd808: Thanks. That's what I ended up doing. [17:32:28] Actually, I wonder if `scap log` works on beta [17:32:48] Oh, no ircecho [17:32:50] So no [17:41:40] I tried it. And `scap sal` after I noticed that in the help. Neither worked. [17:46:41] greg-g: Regarding long-running scripts, the Deployments page says "it is required to add an entry in the calendar for the task with a window that accounts for the anticipated start time and estimated length for the task". Are there any details or examples, particularly when the estimated length is "I have no idea"? This is regarding T181731 right now, but I'll have more as things get merged and schema changes happen. [17:46:42] T181731: Run maintenance/cleanupUsersWithNoId.php on all wikis - https://phabricator.wikimedia.org/T181731 [17:48:26] anomie: it's mostly for the benefit of eg DBAs. is there any interaction with other processes (iow: should it running block anything else happening?) [17:49:06] greg-g: No interactions that I know of. [17:49:27] then pick a time to start it off, give yourself an hour just for good measure [17:50:15] It took an hour to run for Beta Cluster, so I'm sure it'll take much more than that for production. [17:51:17] there's also the "week of" area at the top of the week which can be used for this kind of long running thing [17:51:46] If the "week of" area is ok to use, I'll just do that. [17:58:16] bd808: Is it known that doing !log in #wikimedia-cloud makes stashbot comment on the task twice? [17:58:59] hmmm... no, I didn't know that [17:59:12] https://phabricator.wikimedia.org/T181731#3800347 [18:01:07] ugh. I think I see why it happens [18:03:11] there is magic in stashbot that is pretty much just there because some people didn't want to switch to !log in -releng for beta cluster stuff. When you '!log deployment-prep ...' in -cloud it handles it and then also processes the event like it was done as '!log ...' in -releng [18:03:17] its a bug :/ [18:05:13] We could remove that special casing for #-releng [18:07:59] stashbot.sal._store_in_es() handles the phab comment addition and right now it has no guard that can tell if this is a duped message [18:07:59] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [18:08:07] * bd808 pets stashbot [18:09:29] I think I can fix this... [18:18:20] no_justification, anomie: I think this will fix it -- https://gerrit.wikimedia.org/r/#/c/394356/ [18:22:06] bd808: Seems sane to me. But I haven't tested and don't really know python. [18:24:28] bd808: You shouldn't use bare except [18:24:31] :p [18:24:50] :P [18:25:22] except in a long running bot that wants to recover from *any* error in a controlled way [18:25:34] * bd808 punched flake8 in the nose [18:29:48] "except Exception" should be enough to shut up flake8? [18:29:54] Functionally idential [18:30:00] *identical [18:31:50] not exactly though. There is even an example on https://docs.python.org/3/tutorial/errors.html showing an "except: raise" useage [18:32:07] but flake8 isn't smart enough to see that I guess [18:32:54] plus code can randomly raise anything. python is duck typed [18:34:35] I always read that as fuck typed [18:34:40] Which isn't too far from the truth tbh [18:36:05] I'm a fan of duck typing and passing dicts but opinions vary [18:59:40] anomie: hey, sorry, I went into a long workshop. Yeah, just use the week of section [19:00:40] anomie: Looks like it is fixed now -- https://phabricator.wikimedia.org/T181731#3800669 [19:46:29] anomie: What is a cross-wiki block exactly? I assume not a global block, right? [19:46:54] I know user rights can use @wikiid, but that's a different thing afaik. [19:48:19] Oh it does use that. Interesting. So at run-time CentralAuth creates its own Block object that emulates the built-in one, and uses the same interwiki syntax to represent the blocker. [19:49:44] Krinkle: I may have screwed up some terminology. CentralAuth inserts blocks into local block tables with ipb_by = 0. [19:50:22] ... for each sul-connected wiki? [19:50:44] I'd have sworn there is a global_blocks table of sorts that CA just queries from a hook or something. [19:51:54] Hm.. I see now. [20:13:23] Derp? [{exception_id}] {exception_url} MWException from line 644 of /srv/mediawiki/php-1.31.0-wmf.10/includes/logging/LogFormatter.php: Expected title, got null [20:35:13] AaronSchulz: if you call LoadBalancer::getConnection() for the local wiki's database, do you still need to call reuseConnection()? [20:46:54] nope, it will just no-op the later call [21:14:16] so, in lua templates (like https://sh.wikipedia.org/wiki/Module:Navbox?action=edit) on line 237 where there is the :wikitext call .. where can I find the code for that helper method? I assume part of the Scribunto library? [21:15:44] * subbu found it in the Scribunto/engines/LuaCommon/lualib/mw.html.lua [22:06:16] I tried to fix session management in Collection [22:06:17] It hurt [22:06:18] I had to give up [22:08:02] I don't think I've ever seen so much $_SESSION usage [22:08:03] * no_justification cries [22:15:27] I know the feeling, same example actually. couple weeks ago looking at /deprecated in logstash :( [22:15:50] I wanted to try and fix it [22:15:52] But it hurt [22:21:17] I remember doing the original code review of Collection before it was deployed [22:21:48] the session usage was unusual by MW standards, but it seemed like a reasonable way to do the thing they were doing [22:23:59] nowadays, we would probably just stick it into LocalStorage... [22:24:07] yay, moar JS:( [22:36:19] or sessionStorage [22:44:50] legoktm: there's 102 files in autoload.php that have multiple classes in them [22:47:10] I'm working on an automated file splitter at the moment, figuring out what its logic should be [22:49:08] in the AST we have a comment array attached to each node, the idea is to take the first comment in the file, and see if it looks like a file comment [22:49:21] specifically if it has @file or license text [22:49:36] and if so, that comment will be duplicated into all the resulting output files [22:51:23] there's two ways to generate the output: do string operations based on the file offsets given in the AST, or use the format-preserving printer to print the AST [22:51:40] I'm leaning towards the former [22:56:45] Krinkle: Swapping for Session::get() and friends seems like the easiest 1:1 swap [22:56:47] But still messy [22:56:52] The code isn't the cleanest [22:59:05] TimStarling: I like the former too [23:01:09] I wonder if automating it will result in some bad comments [23:01:43] Ie: some class got tacked onto a file and its overarching @file comment is mostly bogus [23:02:09] yes, there will be some bad comments [23:03:13] Although I guess the transition is automated not the review. [23:03:32] Long as the resulting git commits are broken up into manageable chunks [23:03:51] Uh, the goal is to do it all in one commit [23:04:33] I hate omnibus commits 😭 [23:04:39] a lot of files have a little bit of descriptive text at the start of the @file comment, for example on PathRouter.php we have "Parser to extract query parameters out of REQUEST_URI paths." [23:05:18] so the proposed scheme will duplicate that to a new file PathRouterPatternReplacer.php [23:06:23] or in Feed.php we have "Contain a feed class as well as classes to build rss / atom ... feeds" which doesn't make sense as a duplicated comment [23:07:55] the idea is to have the automatically generated changes become a single commit, that should make rebasing a bit simpler [23:09:12] you can comment on the changes to the generation script, those are more fine-grained than the final mediawiki/core commit will be [23:10:20] fair 'nuff [23:10:27] anyway, we could do manual fixes to file comments prior to the scripted migration commit [23:11:29] no_justification, I've tried to mess with the Collection code before [23:11:36] I think my attempts are probably still in gerrit [23:11:54] at some point I too gave up [23:17:50] I got 99 problems.... [23:24:43] TimStarling: radical idea... what about not using the @file block (besides license) and putting these in the/a @class block instead [23:24:50] I've always found them confusing/duplicative. [23:24:59] I like that idea ^ [23:25:00] esp. given the one class per file rule [23:25:23] it's only an invitation for the author to put useful text in @file which is then not picked up by doxygen where you'd expect it. [23:25:29] that means I need a license parser [23:26:02] well, basically any test before "This program is free" [23:26:13] except if it starts with "@" or "Copyright". [23:26:22] should catch most of it, but I wouldn't be surprised if I was wrong. [23:27:02] copyright/author notices are mostly wrong, but I guess it is easier to preserve them [23:27:05] Feed.php has: [23:27:10] * Copyright © 2004 Brion Vibber [23:27:20] pretty sure it's been edited since then [23:28:03] Well, there was talk about that stuff being kinda lame anyway on wikitech-l [23:28:09] Good time to remove said lines? [23:28:21] need to ask permission, otherwise it is a GPL violation [23:29:16] well, not that we follow the GPL anyway [23:29:42] a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. [23:30:51] * no_justification writes tool to dump `git log` output into all file headers [23:31:03] A few of us did self-removals, but yeah bulk removal would technically require agreement from every past copyright claimant [23:31:39] TimStarling: nearly everything should have GPL file headers, and if core stuff doesn't it should be safe to add them [23:33:37] bd808: I wonder what the distribution of those headers are. Like if we convince brion is that most of them? [23:33:39] Hehe [23:34:43] I could remove mine there aren't many [23:37:52] what about @ingroup/@defgroup [23:44:57] no_justification: https://gerrit.wikimedia.org/r/#/c/394502/1 [23:45:42] TimStarling: those should also be on the @class block afaik. [23:45:48] At least for files that contain a class only, that is. [23:45:54] TimStarling: I think those grouping markers will be pretty useless once you namespace all the things [23:46:00] Not sure if other member objects are group-able. [23:46:03] currently both files and classes are in groups in doxygen [23:46:11] https://doc.wikimedia.org/mediawiki-core/master/php/group__Ajax.html [23:46:22] Yeah [23:46:49] But only if the source file speified it twice, right? [23:47:02] having files in a group might be useful if the file is not a class file. [23:47:10] like Resources.php, in theory. [23:47:20] otherwise, seems like specifying twice isn't so useful. [23:50:35] doxygen does not appear to do any kind of grouping by namespace [23:50:41] I think grouping @file is kind of redundant when you only have one class per file [23:51:09] oh, no it does group by namespace in the class list, never mind [23:53:33] so the idea is to remove class files from groups, and to remove any descriptive comments from @file blocks [23:55:01] merge into the @class block ideally. [23:55:34] I don't mind either way, but at least currently, groups seem separate from namespaces in that they can be cross-component. [23:56:18] I'm not sure what the point of having license headers in doc comments is [23:56:38] it just puts the license into the doxygen file pages, like https://doc.wikimedia.org/mediawiki-core/master/php/ApiCheckToken_8php.html [23:56:44] does it need to be there? [23:57:25] if license headers were in normal comments /* .. */ instead of doc comments /** ... */ then they would presumably be omitted from doxygen [23:59:56] Well, personally I doubt whether these belong in PHP files at all. I've long believed that per-file license headers are not needed, even for GPL. But oh well.