[00:44:26] @rhinosf1 is MDR still a thing or are we only doing RDW [03:06:01] probably won't be awake at 9am yet, my sleeping schedule is too broken :/ [03:09:02] type shi 🤝 [06:37:31] MDR is gone. [07:01:00] so from dec 11 on its full status vacay? [07:01:41] There are still some inconsistencies to work out with that date. So it is not 100% solid yet. [07:02:47] I think that's supposed to be the 18th [07:03:04] "This phase starts on the Friday closest to Christmas Day, but at least 5 working days before Christmas, and ends after the New Year." which would be the 18th [07:04:09] But also I kinda just want to simplify things and get consistent so it's always the same and just do 5 days before Christmas to 5 days after New Years. [07:05:47] i got an event from rhi thats labeled as dec/11 [07:10:58] Aware. I think the schedule got confused somehow. Maybe looking at calander and got wrong week on accident or something. But the 11th doesn't match up to what is already on the policy page either. [08:09:36] We are going to change it to the 18th [08:10:07] The 11th will be calculated right but I can get the point that it's a bit long [08:10:53] I don't like 5 days before and after as a rule as you need to pay attention to when the pre travel will be which does change every year [08:11:03] Per what the policy says wouldn't that be the 18th though anyway? If not maybe this should be clarified a bit on the policy also. [08:12:24] Tbh I'm happy changing the dates each year because the mad weekend of pre Christmas travel will move depending on what day Christmas is [08:12:57] That's fair enough. [08:13:53] Maybe we should remove that entirely then and leave it more open to case by case decisions each year. [08:14:13] Can do [08:14:17] Will do actually [08:15:19] Also @rhinosf1 I will likely be around during maint. If you need me, ping me if no one else is around. [08:17:14] Updated, can you check the policy? [08:17:22] Will do one moment [08:18:22] I found the major window was quite long and also a little confusing on what was major based on last year. It's effectively still there but a lot more use common sense in the lead up to not rush big things in rather than a fixed period. [08:18:52] Like I still don't expect you to go do a huge change the day before that runs over into the window and falls apart ruining it [08:19:32] Looks fine to me [08:22:04] I will send out Mattermost announcements then [09:00:44] @cosmicalpha check is now running [09:00:49] Thanks [09:01:14] and already 3 tables showing as possible corrupt [09:01:30] Ping if you need me and keep updated if you find anything else [09:06:07] searchindex is widely showing as possibly not closed properly [09:06:27] nothing so far as definately corrupt though [09:10:49] @cosmicalpha we're up to b and 16 wikis so far, i think it's easier to put it into RO and let it auto repair and optimize everything [09:11:11] searching isn't broken on all of the wikis but the table is bad [09:11:40] I approve of readonly. [09:11:49] doing then [09:17:45] @cosmicalpha repair is now running in screen for a first pass, I will run optimize after too [09:18:05] and then a final pass of check to make sure we look fine [09:24:11] @paladox @abaddriverlol @cosmicalpha: If it stays at this pace, I expect 90 minutes - 2 hours for the first pass. I may then break for lunch and do the optimise this afternoon as that will take longer unless anyone wants to take over. [09:24:58] I will likely be going to sleep in ~3 hours so I don't want to get into it myself as I may not be able to stay awake [09:26:22] for visibility, it was sent to the wrong channel [09:29:21] Our next batch of upgrades will still be a couple days out. [09:33:36] ideally next week is fine [09:44:14] @cosmicalpha currently on att [09:46:39] we've passed att [10:06:08] @cosmicalpha we're on avid now [10:06:30] so going quite slow but 1 big wiki down and 1 in progress [10:07:02] if it isn't done by 11:30/12:00, i plan to break and let it run watching for disaster only from my phone for a bit [10:16:14] I am not actively looking at my screen but very close by [10:37:53] @rhinosf1 Is there any reason we need read only for the next check run, we should only need it for optimize I think, no? [10:38:28] i am thinking about just not doing a final check at the end and hoping repair and optimise are enough to fix anything marked bad [10:38:30] which they should be [10:38:38] because this is taking way longer than estimated [10:38:58] check was throwing various warnings running live though [10:39:12] i think repair + optimise will fix everything though [10:39:47] at the rate it's going, repair and optimise will probably take about a day [10:39:55] Oh we don't need readonly for optimize either just repair [10:40:08] optimise will lock things up [10:40:15] it locks the tables while it does it [10:40:20] mediawiki will not like that [10:40:50] this would be a good argument for database replicas as we could do this with no impact if we had replicas [10:41:11] True. replicas are a goal for next year btw. [10:41:26] I will be putting the tech budget together over the next few days also. [10:41:35] I don't think mediawiki will handle lots of tables being locked well so I think RO is safer [10:41:40] we don't want more corruption [10:41:43] That's fine [10:41:53] You're probably right [10:42:01] cool [10:42:13] it's on bluepages now [10:42:17] which is quite big [10:42:28] we've already passed att, avid and nearly bluepages [10:42:36] Side note: how it is already almost 2027?? [10:42:43] don't even [10:42:46] this year has been insane [10:42:53] it was like july 5 minutes ago [10:43:05] i've owned a home for 2.5 months now [10:43:47] i am going to take a break soon as there's no chance this will finish within the next hour [10:43:53] and leave it running [10:44:06] i won't be watching progress but i will watch any alerts from icinga etc [10:44:09] Sounds good. Ping if you need me in emergency. I will likely be around for the next 1.5 hours. [10:44:16] or maybe 2 hours [10:46:26] I have mostly reversed my sleep schedule entirely now so I am up all night and get IRL + MH work done most of the night which has been better as I sleep full time (from about 6-7AM until 2ish PM) rather than 1-2 hours I was getting before. [10:47:03] Doesn't always work out as there are some things I need done at certain times though [10:47:23] But better than getting 1-2 hours of sleep a night of weeks without end. [10:49:48] at least it's a schedule [10:50:50] taking break now [11:18:47] [1/4] https://cdn.discordapp.com/attachments/1006789349498699827/1571484603707957398/20261106_031719.png?ex=6afae696&is=6af99516&hm=a75b66bab6cc3b2d55f222af9bfb7638e8892a17fdd23dad8f6d2ca88379e582& [11:18:47] [2/4] https://cdn.discordapp.com/attachments/1006789349498699827/1571484604328841257/20261115_000507.png?ex=6afae696&is=6af99516&hm=38591864bbc8206aa76f72352625238aaa5d070d37de47f69e5f9291ca8a82af& [11:18:48] [3/4] https://cdn.discordapp.com/attachments/1006789349498699827/1571484605133885440/20261114_215909.png?ex=6afae696&is=6af99516&hm=88675d7e3b76fcaec45f3179bc56f1e3da918d036e6fdef012a551ce85790751& [11:18:48] [4/4] https://cdn.discordapp.com/attachments/1006789349498699827/1571484605859635220/PXL_20261106_114613_502.png?ex=6afae696&is=6af99516&hm=db648b5406fa16b9064df8ca8b7c047c4476a5b7d7002de4537bcdd083c9d43c& [12:00:13] @rhinosf1 how's it going [12:00:33] up to er on the repair [12:00:39] we will do an optimise after [12:00:57] I expect repair will finish around 1800-2000 depending on how fast it goes [12:01:28] optimise will be longer, probably 15-20 hours after [12:01:46] i will be out most of tomorrow but i will set a logsalmsg to ping me when done and keep checking as I can [12:01:55] it's in a screen [12:02:19] so we will probably finish between 10am and 5pm tomorrow @paladox [15:44:48] @paladox should finish the repair between 17:30 & 18:00, then will move onto the optimise [15:45:05] I will be out a lot tomorrow but I will make it ping me via logsalmsg at the end [15:45:26] [1/3] my plan is to run [15:45:26] [2/3] > mysqlcheck -o --all-databases 2>&1 | tee /root/mysqlcheck_optimize.log; EXIT_CODE=${PIPESTATUS[0]}; logsalmsg "RhinosF1: done (exit code $EXIT_CODE)" [15:45:26] [3/3] so I will get pinged [15:45:52] if anyone sees it log that, you can check it definately ran on all wikis and then restart writes [15:46:13] I would expect some point tomorrow afternoon, I will try and give a better estimate once we are a few hours in [17:08:52] still on target to finish by 6 so we can move onto -o [17:13:56] is it fine if I deploy to prod in a couple minutes, or is there any reason not to do that rn [17:14:09] deploys are fine [17:26:07] should be about 20 minutes left before I move onto -o [17:51:27] @paladox moved onto -o [18:15:25] @paladox disk space is down 10% already on db161 [18:15:29] that's bloody good [18:15:44] we really need to get replicas so we can do proper maint without long downtime [18:16:12] this is just too slow to do everywhere unless we can make a way to do it putting 1 wiki RO at a time [18:17:57] I will check on progress in around half an hour to an hour [18:48:36] @paladox @abaddriverlol @cosmicalpha any of you planning to be awake until about 2am UTC? [18:48:50] it is running a lot faster than --repair was [18:49:22] basically wait for the message in SAL when done or watch /root/mysqlcheck_optimize.log [18:49:38] once I go to bed at 2100 UTC as I have to be up early [18:49:48] and once it is done, bring things back up [18:49:55] I will probably go to bed before that [18:50:10] the current estimate is between midnight and 2am [18:50:53] which is what about 6-8 hours, it's 1 hour in [18:51:57] I will have a look at progress about 2030 UTC and come up with a better guess, it ran at a pretty steady pace mostly doing --repair [19:01:00] If there’s something you need me to run at that time I can (I won’t be able to stay up / monitor it tho, and if it’ll be longer then 2am probably not) [19:01:50] basically just check it actually has finished (so the last database in the log is at the end of the alphabet) and if it is, we should be good to come out of read only [19:02:11] it should finish by 1am at this pace [19:02:26] i have set it to do a !log when it finishes [19:02:35] so either watch for that or tail the log [19:03:10] i will stay watching it until about 9pm, by that point we should have a pretty accurate estimate of finishing time. [19:03:35] So that I understand right, once it is finished, you want me to take it out of read only. But before check the log to see if the last wiki has been logged (which is z?) [19:04:06] yes [20:09:38] got an error when trying to link git account to phorge [20:09:43] https://cdn.discordapp.com/attachments/1006789349498699827/1571618219679162489/message.txt?ex=6afb6306&is=6afa1186&hm=577ca2fecae6959eaf7629370d4ba94fa27116602c2541ff000db65f982e0c50& [20:09:58] git says it's linked but phorge doesn't [20:10:52] I wouldn't advise sending this in a public server [20:16:26] @paladox expect it to be between 00:30 and 1:15am [20:17:39] its a known issue that affects all phorge installs, not just us [20:17:54] with phorge or github [20:18:21] seemingly GH, iirc it just randomly started happening one day [20:18:38] [1/3] • Entering f Now (56.25 GB): Expect mysqlcheck to clear f around 20:50 GMT, then sprint through g, h, and i by 21:10 GMT. [20:18:38] [2/3] • Halfway Mark (k/l): It will cross the 50% disk-space mark right around 21:40 GMT. [20:18:38] [3/3] • Final Heavy Blocks (p and s): Expect p (52.44 GB) around 22:45 GMT and s (78.73 GB) around 23:30 GMT, coasting across the finish line between 00:30 and 01:00 GMT. [20:19:00] @paladox if you are watching the log with tail, use ^ as a guide as to if it's still running to schedule. [20:20:03] Remember to update #announcements when done or if it is running very late [20:20:34] 1am utc right? [20:21:32] [20:21:34] ? [20:21:54] wait no [20:22:19] yes [20:22:19] wait yes? [20:22:20] idk [20:22:26] im tired and want to sleep [20:22:37] and? [20:22:49] i dunno [20:23:08] lie in bed then [20:23:12] maybe have some soup [20:23:26] we dont have good soup here sadly [20:23:36] send your roommate to find one [20:23:43] they can find soup instead of sec bugs [20:23:46] roommate is on a call right now [20:23:57] i won't be able to figure out how long it has left but i trust your maths on it. [20:23:58] which also makes slep harder [20:24:00] it's on G rn [20:24:04] thats wasnt him lol [20:24:16] if it stays to them timings, it's running fine [20:24:35] i should do some amount of work but sleep sounds so much nicer (has gotten no class work done today) [20:24:36] i just gave gemini the count of wikis by letter on the database and the disk space by letter [20:24:52] and then give it the time and how far we are through [20:25:00] it's been accurate to a few minutes each time [20:25:27] no you sleep, no work when unwell [20:25:33] more work doesn't get you better [20:26:32] me when i have a meeting im still going anyways in 4 hours [20:27:13] look after yourself kid [20:27:39] yeah ive never really been great at that bit [20:28:06] i may not always be doing what i should be doing but im always doing something [20:28:14] [1/4] • Halfway Mark (k/l): Will cross 50% disk space in about 30 minutes (around 20:55–21:00 GMT). [20:28:14] [2/4] • Heavy Cluster (p — 52.44 GB): Expected around 21:55 GMT, clearing into q/r by 22:25 GMT. [20:28:14] [3/4] • Final Big Block (s — 78.73 GB): Expected around 22:40 GMT, clearing into t by 23:25 GMT. [20:28:15] [4/4] • Finish Line (z): Expected right around 23:50 to 00:15 GMT, triggering logsalmsg "RhinosF1: done (exit code 0)" right around midnight! [20:28:28] @paladox current estimate as my terminal wasn't refreshing ^ [20:29:14] i will moan at you [20:29:16] https://we.phorge.it/T16936 [20:29:31] opened an upstream ticket ^ as this issue is still ongoing [20:29:53] ty mac [20:30:37] i want to add an option for cross-wiki abuse to the centralauth menu when filling out a reason for locking [20:30:42] that's in MirahezeMagic right? [20:31:12] is it not there already? [20:31:25] no! [20:31:28] i thought i was going insane [20:31:33] i just always assumed it was [20:31:45] huh [20:31:56] let me look for ya [20:32:02] waitwaitwait [20:32:05] how do you [20:32:19] look [20:32:37] this is a teaching moment [20:32:41] or a learning moment for me [20:32:54] I believe we can just edit https://meta.miraheze.org/wiki/MediaWiki:Centralauth-admin-status-reasons [20:33:17] ohhhh nice [20:33:25] thanks mac [20:33:26] i wasn't sure if it was a page or something on git [20:33:35] it sometimes depends i think [20:33:44] and- no permissions [20:33:47] 🙁 [20:33:53] good mindset [20:33:58] !! [20:34:31] my lazy approach was to open up the MirahezeMagic github repo and use the search bar for words I knew should be in the same config, like abuse or multiple [20:34:40] when that drew nothing I tried on mw-config [20:34:43] https://github.com/search?q=repo%3Amiraheze%2Fmw-config+multiple&type=code [20:35:01] but yes in this case it seems to be stored on wiki [20:35:07] i have MM forked [20:35:12] is that the word? [20:36:04] yes! A fork is a copy of a repository that you own and control, you can add your own commits to it and then ask to merge your changes 'upstream' to the parent repository [20:36:19] looks like reading your essay actual paid of [20:36:22] like 6 months ahead [20:36:37] codespaces is cool [20:36:38] https://cdn.discordapp.com/attachments/1006789349498699827/1571624994021056532/iu_.png?ex=6afb6955&is=6afa17d5&hm=e87babc633fa06276159cfe3b39ee41588d00e64f9b0c26d1822932de95cfe3b& [20:36:42] it is [20:36:53] if you have the free student plan its even better i think [20:37:03] makes me very happy to hear [20:37:45] why is it called i18n [20:38:05] short for internationalisation [20:38:10] and l10n is localisation [20:38:12] i think [20:38:13] you see we hate letters here [20:38:20] developers are lazy [20:38:22] so we take big word [20:38:28] chop out the middle 18 letters [20:38:31] and replace it with [20:38:32] 18 [20:38:53] @paladox I'm pretty much gone from now [20:39:06] we do the same with a11y (accessibility) which is objectively a dick move since that is horribly not accessible [20:39:14] heh [20:39:29] I have to go to the Isle of Wight tomorrow so I'll be up about 4:30am [20:39:42] id rather die [20:39:47] thats far too early [20:39:48] That's perfect [20:39:53] also when creating a git account i also finally figured out where reception got his pfp [20:40:03] I'm not too happy about 6am train either [20:40:04] oh yeah [20:40:06] gravatar [20:40:16] [1/2] it looks so happy [20:40:16] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1571625907914018867/image.png?ex=6afb6a2f&is=6afa18af&hm=150e4d9d44d996a3106f4224efbfc0379158e307b609960aca53982b49af53d8& [20:42:10] sorry no [20:42:13] not gravatar [20:42:17] thats the other thing [20:42:21] those are identicons [20:42:25] https://en.wikipedia.org/wiki/Identicon [20:42:37] @crystalite13 I've gone ahead and added the reason [20:42:43] https://github.blog/news-insights/company-news/identicons/ [20:42:46] thank you [20:43:13] does cross wiki abuse need an appeal message though? [20:43:18] [1/2] appartently this is mine [20:43:18] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1571626671231078451/pixDeVl.png?ex=6afb6ae5&is=6afa1965&hm=bfd8f4fbd1fe66cf422ce50d3db55898177af6f94d2165799446085abf01eb60& [20:43:21] Pix would know [20:43:31] would he [20:43:41] i copied and modified the one from global blocking [20:43:57] i mean [20:43:59] maybe [20:44:06] you're the authority CVT figure pal [20:44:21] https://klipy.com/gifs/chihuahua-uncomfortable [20:44:52] this git thing goes deep [20:45:02] why does a codespace have a silly name [20:45:05] it's git all the way down [20:45:09] why not [20:45:20] so many languages [20:45:20] I'll add it [20:46:19] oh already done [20:46:46] mmmm idk if we want that on steward or cvt email [20:46:50] ill ask in mm later [20:47:03] i need a nap [20:47:13] cya [20:47:21] sadly not [20:47:25] i have things to do [20:57:03] Things are still looking good [20:57:28] @cosmicalpha replicas could be huge cause this would be so much easier and we could optimise the db when we set them up [20:57:54] I don't want to do an optimise on every db and cause 18 hours of issues but it looks like it will have a decent impact on disk [20:58:04] I suspect because of all the links migrations and such [20:58:20] I genuinely am gone now though [20:58:30] Yeah next year that is definitely on the plan [21:10:26] We've dropped like 15% in disk space [21:10:31] Some of it will be binlogs [21:10:35] But not that much [21:13:40] Nice! [22:17:57] has db161 still not shit it's binary pants? [22:26:50] [1/2] hi! can someone merge my PR please if all looks good [22:26:50] [2/2] https://github.com/miraheze/mw-config/pull/6641 [22:31:34] Done. Will deploy in a moment. [22:32:33] Oh I was on IRC lol I meant to reply from this side. [22:32:55] Deployed @whostacking [22:37:15] [f04cbd338577bcd41703bc8c] 2026-11-15 22:36:36: Erreur fatale de type « Wikimedia\Rdbms\DBQueryError » [22:38:00] hmm [22:38:08] link? [22:39:30] oh right one moment [22:40:17] [22:40:29] the files sharing thing works though [22:40:38] its just that when going to file details that it starts acting up [22:42:32] Fixed @whostacking [22:43:18] nice. thank you so much! [22:43:26] No problem! [23:00:44] @cosmicalpha are you able to do the thing rhinos asked above (when it finishes). I'll be around for a bit but incase i'm not and it takes longer then expected [23:00:53] it's currently on the S [23:01:03] Maybe but it might be a little later then that.