[00:40:37] well its a linear gradient [00:40:55] [1/9] ```css [00:40:56] [2/9] .key-melee { [00:40:56] [3/9] width:30px; [00:40:56] [4/9] height:30px; [00:40:57] [5/9] border:3px solid #1b2935; [00:40:57] [6/9] background:linear-gradient(transparent, var(--background-gradient-bottom)) fixed, repeating-conic-gradient(#b23738 0% 25%, #d44241 25% 50%) 0 / 30px 30px; [00:40:57] [7/9] animation:keytilt 1s infinite ease-in-out; [00:40:57] [8/9] } [00:40:58] [9/9] ``` like this is it [00:44:31] 💤 [03:30:07] Common.css is purely a choice if i want to use right [03:56:17] you could use the skin css pages [04:27:38] element needs to be inline, a span [05:45:11] Miraheze is down [05:45:22] same error as last time [05:45:47] Caught that too. [05:45:52] can confirm [05:45:55] Was planning to edit OSCA. [05:46:14] why is miraheze so buggy [05:46:27] there have been 2 website crashes in the last week [05:46:35] same error and everything [05:46:47] [1/2] Error 503 Backend fetch failed via cp37 at Wed, 05 Mar 2025 05:46:15 GMT [05:46:48] [2/2] Varnish XID 659687813 [05:47:10] Yup [05:47:13] trust me, it's been so much worse 3 years ago on old server [05:47:25] really? [05:47:27] [1/2] cannot reach to my wiki, getting this error [05:47:28] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346720749293015040/image.png?ex=67c936ef&is=67c7e56f&hm=820e69a76834cc706732ed13350ee46581b26cbc7bfca8306e411384abb418a1& [05:47:44] i'd hide my ip if I were you btw [05:48:14] I'm the 6666th member 😎 [05:50:04] [1/2] Getting this error when connecting to my wiki [05:50:04] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346721404414066810/image.png?ex=67c9378b&is=67c7e60b&hm=d7849caaca322fa19646acce26e6d8c9c37bf6a3122c3fa866d5e12ecd79cc57& [05:50:12] man we got it [05:50:16] Feels like maintenance started 13 hours too early [05:50:47] i think these revent crashes show why the maintenance will happen tommorow [05:51:04] it is working now [05:51:05] eclair u here? or who shall I ping [05:51:52] now there is too many connections [05:52:11] please know that everything you post on discord stay on irc, no matter if you deleted or edited your message here [05:52:20] it's down [05:52:27] https://cdn.discordapp.com/attachments/407537962553966603/1346722004375703574/Screenshot_2025-03-04_at_9.52.07_PM.png?ex=67c9381a&is=67c7e69a&hm=a75d5f26437cfc16aa8fa1472918b2c0a1bb6768460f61175e650648d86b7ba4& [05:52:42] so posting deleting and posting looks like span on irc side [05:52:44] pages that had the previous error infinetly load and new pages being this error [05:53:00] give it a rest a bit [05:54:39] its interesting that atleast to db181, the same thing is happening as it did exactly a month ago. The servers are overloaded [05:55:02] [1/2] this is how it looks like [05:55:02] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346722653938913301/Screenshot_2025-03-04_at_9.53.56_PM.png?ex=67c938b5&is=67c7e735&hm=057ccfafbc62426528000c8613a7644b6370ee00cb783f924bc0e6d074d17613& [06:00:15] I think they're updating early [06:06:33] Where are the admins [06:06:38] i dont know [06:06:51] there is less attention to this crash then the last 2 ones [06:07:00] and the numbers just got worse [06:08:39] https://tenor.com/view/panic-anime-shookt-shocked-gif-10583115 [06:08:50] volunteering and time zones [06:09:02] Well it is my bedtime anyways [06:10:46] I think the Miraheze team needs to improve the db181 servers [06:11:13] its not a coincidence that only that server crashes exactly a month apart [06:12:38] oh db181 is down? [06:12:43] yeah [06:13:21] [1/2] this is the updated screenshot [06:13:21] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346727263655166023/Screenshot_2025-03-04_at_10.12.50_PM.png?ex=67c93d00&is=67c7eb80&hm=b33957c51b7ab78e56d06f818391c400e48cb46f3f4e3893a6887d662d3646f6& [06:15:43] [1/2] why? [06:15:44] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346727862173962240/Screenshot_20250305-131439.png?ex=67c93d8f&is=67c7ec0f&hm=0054597f36fc869cfeaecb97feb72622434e487158341f1983e92bfbe199ec2f& [06:17:16] check my screenshot right above yours [06:17:28] the servers are overloaded with information, so it cant laod [06:18:42] macfan [06:18:56] blankeclair [06:19:02] nya? [06:19:18] aware of 503? [06:19:38] seems like you're the only tech online [06:19:53] ah crud, infra is not my thing ^^; [06:20:00] rip [06:20:04] paladox: you there? [06:20:34] oi Rhinos [06:20:39] RhinosF1 [06:20:41] :p [06:21:01] they are both "inactive" [06:21:04] h [06:22:29] Hello [06:22:34] oh hello [06:22:38] db181 is having a stroke atm [06:24:05] c4 [06:25:38] BlankEclair: you can do this live on mwtask181, towards the top of LS.php you'll see wgDatabaseClustersMaintenance = []; change it to wgDatabaseClustersMaintenance = ['c4']; and deploy it [06:25:46] ah oki [06:25:56] BlankEclair: how well is the server [06:25:57] huh, direct on mwtask181 and not a pr to the repo? [06:26:06] it's um [06:26:09] > CPU: 9334% [06:26:09] BlankEclair: it's temp so you can do it live [06:26:12] has seen better days [06:26:16] Can you ssh in BlankEclair ? [06:26:30] mwtask181? should be able to [06:26:45] BlankEclair: db181 [06:26:48] of course, bloody linux usb bug with yubikeys makes me wait 10s after plugging it in [06:26:50] doubt but i can try [06:27:07] Try after you've deployed that [06:27:41] blankeclair@db181.wikitide.net: Permission denied (publickey). [06:28:40] You're only mediawiki admin [06:28:43] So ye [06:28:49] That is about as much as you can do [06:29:00] maint clusters edited and mwdeploy running [06:29:01] But it'll kill mediawiki traffic to c4 [06:29:17] So it should keep everyone else up [06:29:40] And if it's mediawiki related, it'll stop it [06:29:46] oh god yeah, instant drop lol [06:30:02] i wonder, can we automatically put clusters in maintenance if they start to use too much cpu? [06:30:12] No idea [06:30:48] might be scuffed idea, but perhaps a systemd timer that grabs all the databases' cpu usage from grafana and automatically writes affected clusters to some .php file that we can then import [06:31:38] when is it safe to unmaint c4? [06:32:44] BlankEclair: give it 5 minutes at least to make sure it's fully stopped doing whatever upset it [06:32:59] And for an infra to turn up to see what's up with the db [06:33:00] oki ^_^ [06:33:05] https://cdn.discordapp.com/attachments/407537962553966603/1346732231367987271/image.png?ex=67c941a1&is=67c7f021&hm=e5f6994b17cd5f718c4bf181aa6b8383a69fa3a69ee249ccc7e2850c4b085f6d& [06:34:33] Yup, that's exactly what I told Claire to do [06:34:49] Ah [06:34:56] It means only 1/4 of the farm is down instead of all of it [06:35:47] Ah i see [06:42:34] i wanted to see no life rod stats bcus i heard it got buffed but i cant 😭 [06:42:40] this pops up [06:43:00] https://cdn.discordapp.com/attachments/407537962553966603/1346734728530628618/1469_20250305154226.png?ex=67c943f4&is=67c7f274&hm=f478afb56e34fccdfc35194d8842de74a5bfda64c8cbcefc19ca671f0a82645a& [06:43:41] none of this adds a value to solution or this chat in general [06:46:29] Be nice please [06:46:52] There's not a lot users can do to add value to this outage tbh [06:47:00] db sad due to backups [06:47:09] We're waiting for them to finish [06:48:08] (we technically can kill them, but it's a good idea to have backups for the maintenance tomorrow) [06:48:21] sorry [06:48:51] You did nothing wrong tbh @jrball0572_31948 [06:49:04] what I meant is that it was clear from your and eclair's convo you are working on it [06:49:30] I'd rather badly drawn sad faces than being told how rubbish it is we're down tbh [06:49:39] I'm happy for a bit of randomness as long as it's polite [06:49:49] true yeah lol [06:50:04] for some reason, the purple C makes me think of the intersex flag [06:58:01] [1/2] he's hard at work figuring this out [06:58:01] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346738504213860382/IMG_20250305_175629_850.jpg?ex=67c94778&is=67c7f5f8&hm=930bb4524059216403c0abd202e1e33ef7baa84cd8a8384710b54f9293baa5a7& [06:58:25] CA has arrived [06:59:14] We apologize for the issues for those encountering them at the moment. Normally we would kill backups for this circumstance but I decided not to because it is good to have backups right before maintenance as we have learned the hard way before for those that remember those incidents. [06:59:31] (and yeah, backups are to blame again >.<) [07:00:40] absolutely backups are a must [07:01:22] On the bright side after maintenance tomorrow we should be able to avoid this in the future by expanding db memory resources. [07:03:29] > should [07:03:34] can't be too pessimistic xD [07:06:01] [1/6] For those curious what exactly it is we are doing (or plan to anyway) — [07:06:01] [2/6] * Increase RAM (memory) to 128GB for all database servers [07:06:02] [3/6] * Double total swift (file storage) disk capacity (which will be about 18TB in total going to file storage capacity) [07:06:02] [4/6] * Add more memcache (mediawiki caching) resources [07:06:02] [5/6] * Add more opensearch resources for MediaWiki (CirrusSearch) which means we will be able to finally make CirrusSearch available unrestricted as well. [07:06:03] [6/6] Now this is if everything goes according to plan though. [07:14:42] How much time is it gonna take [07:14:49] :ThinkerMH: [07:15:40] Oh nvm mb [07:17:10] seems like CA is planning to kill the backups if they're still running when he's going to sleep [07:17:31] Indeed. [07:17:38] 1 AM, right? [07:17:45] maybe lol [07:17:53] just like me sleeping at 10 pm then [07:18:00] I was in till 6AM last night [07:18:09] > The time for CosmicAlpha is 12:17 AM [07:18:19] um, when do you need to get up? ^^; [07:18:25] 8AM [07:18:30] 2 hour nap :3 [07:18:37] Lol yep [07:18:44] i slept at uh [07:18:46] when?? [07:19:17] 12:41 am [07:19:20] 6AM last night and 5AM the night before so I don't think I can do it that late tonight lol [07:19:53] to many things to do to little time. [07:20:01] heh [07:22:30] [1/6] timestamps of when i went to bed for the past five dated sleeps: [07:22:31] [2/6] 2025-02-27T03:35:07 AEDT [07:22:31] [3/6] 2025-02-28T03:59:03 AEDT [07:22:31] [4/6] 2025-03-03T04:01:14 AEDT [07:22:32] [5/6] 2025-03-03T23:27:03 AEDT [07:22:32] [6/6] 2025-03-05T00:41:15 AEDT [07:22:33] 10/10 healthy [07:23:56] There's 2/5 good nights [07:24:20] [1/17] 2025-02-12T04:02:07 AEDT [07:24:20] [2/17] 2025-02-13T00:20:55 AEDT [07:24:20] [3/17] 2025-02-14T01:42:55 AEDT [07:24:21] [4/17] 2025-02-17T02:17:18 AEDT [07:24:21] [5/17] 2025-02-18T00:24:52 AEDT [07:24:21] [6/17] 2025-02-19T00:15:07 AEDT [07:24:21] [7/17] 2025-02-20T00:01:45 AEDT [07:24:22] [8/17] 2025-02-21T03:15:39 AEDT [07:24:22] [9/17] 2025-02-24T02:16:03 AEDT [07:24:22] [10/17] 2025-02-25T03:14:16 AEDT [07:24:23] [11/17] 2025-02-26T02:24:11 AEDT [07:24:23] [12/17] 2025-02-26T02:24:20 AEDT [07:24:23] [13/17] 2025-02-27T03:35:07 AEDT [07:24:24] [14/17] 2025-02-28T03:59:03 AEDT [07:24:24] [15/17] 2025-03-03T04:01:14 AEDT [07:24:25] [16/17] 2025-03-03T23:27:03 AEDT [07:24:25] [17/17] 2025-03-05T00:41:15 AEDT [07:24:37] It could be a lot worse [07:24:40] true [07:24:49] i've been using my laptop as an alarm clock btw [07:25:16] and here's the line i used to generate the list: `grep XcQ.m4a$ ~/.zsh_history | cut -d':' -f2 | tr -d ' ' | while read -r i; do date -d@"$i"; done` [07:27:51] [1/2] mine looks so much worse lol wake up at some times I go to sleep the next day then randomly fall asleep at 10PM one day wake up at 4AM then go to sleep at 5AM then 6AM, then there is one day I don't sleep like at all because it's 7AM and I'm just sitting looking at my clock thinking why bother with an hour of sleep then I loose that hour of sleep because I'm sitting thinking if I [07:27:52] [2/2] should use it but by noon that hour of sleep sure sounds nice... am I making any sense? lol [07:28:22] why are you like this [07:29:09] When I was sick I wasn't. That part was actually kinda nice to have been caught up on my sleep for awhile lol [07:29:17] oh true lol [07:30:00] we need someone to take care of us 🥺 [07:34:13] We need a safeguarding officer just to make sure volunteers sleep and eat [07:34:52] TRUE [07:36:52] like long haul truck drivers have lol [07:37:13] every 8h they have to stop to eat and nap etc [07:40:22] do we need a safeguarding officer for the safeguarding officer? [07:40:43] probably [07:40:47] Hopefully not [07:42:44] we used to use a timer when coding to log time for billing and it would cap us at set times to stop for a period. Timers work but only.if people use them properly which only half did [07:51:26] there are tachographs now tho? [08:41:27] :ThinkerMH: [08:41:41] :hello: [11:07:26] [1/2] cool i guess. [11:07:27] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346801274011779135/image.png?ex=67c981ee&is=67c8306e&hm=ce4b3b4b30ff2186b316ce4a9230cebab4f6d43f4876b310e65bbf7f9be37f3f& [11:08:01] yeah it worked [11:08:02] thanks mate [11:17:38] [1/2] sensational [11:17:38] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1346803839545643048/image.png?ex=67c98451&is=67c832d1&hm=ff2252b049b1dc08afe070c1deb25c72b1f4a3feae7c6ee322353f190616b6a1& [12:32:47] YOU FINALLY MADE A DISCORD [12:40:00]