[00:37:16] Am i the only one who just got a Bad Gateway? [00:37:58] refresh [00:38:05] it will fix itself usually [00:38:17] Miraheze just went down [00:38:40] ok I stand corrected [00:38:44] db161 went kaboom [00:39:54] I just got it.. [00:40:09] appears to be back [00:40:15] Hooray for failover [00:40:49] [1/2] Umm... [00:40:50] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1494135443611062382/image.png?ex=69e18191&is=69e03011&hm=03287e500f961f992a74f4d88485062784f0292f2be751051c5e922f0cd97936& [00:41:09] hooray for random depooling db161 [00:41:13] give it a second, poor guy had a stroke [00:42:02] you blew it up [00:42:30] I just blame you for everything tbh [00:42:32] nothing personal [00:42:56] why is miraheze down [00:42:58] can we get Fs in the chat for db161 [00:43:02] classic zippee move, blaming everything on women [00:43:28] classic wwr move, blaming me for blaming everything on women [00:43:28] smhead my head [00:43:45] gotta earn my placement somehow [00:43:49] [1/2] i was trying to add a font bro what did i do🥹 [00:43:50] [2/2] https://cdn.discordapp.com/attachments/407537962553966603/1494136198946492486/IMG_7727.png?ex=69e18245&is=69e030c5&hm=7a36ce7ba8282a209503a2e7bfabd3ac49ef47431a38cfc101ad94e2db7d9229& [00:43:55] Skye (Miraheze) will gban you into the abyss [00:44:02] I ain't #1 for nothing you know [00:44:09] this happened to me too [00:44:13] she's already threatened that probably [00:44:13] you're 1 of 2 [00:44:16] i was worried for a second i thought it was just me [00:44:28] NotAracham (Miraheze) will make you become MirahezeGDPR [00:44:29] okay so its a global issue thats good to know [00:44:31] she definitely once threatened to rename me to "Skye I dare you" [00:44:36] it's not global lmao it's one cluster [00:44:38] it'll be back in a minute give it a second thanks [00:44:39] i genuinely thought i just Destroyed my wiki [00:44:41] it's like 25% of wikis lol [00:44:42] when will miraheze be back ip [00:45:05] like right now [00:45:15] What's the next battle cats banner [00:45:16] really? [00:45:19] still #1 [00:45:23] true true [00:45:58] I like how people always say the whole of miraheze is down when it's only like 25% [00:46:02] 25% is rookie numbers [00:46:07] over at wikioasis its all or nothing [00:46:13] usually all [00:46:24] on which note the board are now considering my proposals [00:46:27] 25% is like 10 wikioasises [00:46:37] why the board need to be involved in the proposals for global rights having community elections is no idea [00:46:38] everything goes down if one db server is really slow and not depooled [00:46:40] seems very counterintuitive [00:46:42] because of worker exhaustion [00:46:51] If mh ever gets a server tage it should be `db161` [00:47:01] would it not be wiser to route traffic to certain mw servers based on which db they are on [00:47:03] otherwise it's kinda [00:47:10] reliant on someone being there to depool [00:47:19] probably [00:47:20] [[mh:battlecats:Worker's Revolution (Monthly Event)]] [00:47:21]

[00:47:27] could always just like get orchestrator to failover [00:47:34] but then again you need a replica then [00:47:44] I assume someone's looked at whether or not that's feasible [00:47:58] rn it's going fine because basically every time there's an outage, I'm still online and fix it because of my terrible sleeping schedule [00:48:05] (it's 2:48 am) [00:48:14] one of these days you'll be asleep [00:48:35] haha miraheze tech never sleeps [00:48:36] its ok i can depool too but im slow and i was getting an energy drink [00:48:37] then again you'd have to then scale based on how much traffic each cluster gets so [00:48:41] who knows [00:48:50] before becoming miraheze tech I was in bed by 1am [00:48:52] yeah but you can't fix db161 if it's in a state like it just was lol [00:49:05] you need to press the scary "Reset" button in proxmox [00:49:06] didnt need it anyway /j [00:49:10] before wikioasis I would sleep at like midnight [00:49:13] now it's regularly 2am [00:49:16] https://cdn.discordapp.com/attachments/407537962553966603/1494137569623932949/image.png?ex=69e1838c&is=69e0320c&hm=9ccfc07d3d0c107a6d94266307e92de644b2f9e6fda7eed8fcd735f77fa106bb& [00:49:26] I've always just used stop and then start [00:49:30] gives the same effect [00:49:33] honestly we oughta have some sort of mechanism for MWS to do that as well [00:49:48] restart doesn't work in the state it was in so I assume stop wouldn't either [00:49:56] you can force stop [00:50:01] shutdown and stop are different iirc [00:50:10] oh right [00:50:12] but reset is easier [00:50:23] oh ffs [00:50:28] I need to now deal with more boardroom politics [00:50:33] I miss the days when it was just fun tech shit [00:50:41] now there's too much politics involved [00:51:33] what is the main wiki on miraheze? [00:51:52] what do you mean "main"? [00:51:56] like [[m:]]? [00:51:57]

[00:52:03] meta [00:52:18] ok [01:17:35] sleep is weird [01:18:08] i got into the habit of sleeping past 1am over the summer and kept it through the fall, after winter break, entirely lost the ability to do so [01:18:43] I got 5:30 of sleep and had to take 3 power naps just to get things moving [02:17:36] Bro go sleep lol [02:18:00] Nobody like politics, especially me tbh [02:18:08] in any case I'm taking 2 weeks off [02:18:36] I acted very hastily and I shouldn't hav [02:19:49] Yeah when you overheat it is natural to cool down [02:20:11] some of it is burnout lol [02:20:50] some of it will just be what it is and until I have the ability to change it I'm not going to be able to and I'll either suck it up or I'll have to leave [02:44:22] I’m kind of confused about the way terms of use enforcement works. It always feels extremely abrupt without much warning and sometimes after people actively said things were okay. [02:50:28] if you want my 2c then I wouldn't go from having my wiki dropped for replicating wikidata to going back to replicating wikidata [02:51:18] I’m very confused, like really really confused [02:51:55] Because I don’t think I did that? [02:52:28] I’m not sure if I might have accidentally ran an old script. But I’m very baffled at this situation [02:52:45] Is this about old content [02:55:49] Where’s the global wiki feed? [02:57:41] #global-wiki-feed , I think you need to #verify to see it [03:00:41] [1/2] Okay, I see it now, but I'm still a bit confused. Did you just remove it based off of seeing that it had the absurdly large amount of wiki base items, because all of those were added before the terms of use enforcement on the Shinto wiki? I also had confirmation that they were okay given on the server, so kind of confusing. I'm not sure if you would have wanted me to have manually r [03:00:42] [2/2] emoved them or something. [03:01:07] Admittedly, I did a search for the most recent ones, but I'm not sure if the search properly worked, so I'll still have to manually scroll through to ensure that everything's okay. [03:15:51] Yeah, I am pretty sure if it is specifically related to wikidata stuff, then you are talking about information that has been on that wiki for at least four months. [03:21:37] I believe it was very clear last time that mass-copying of articles from the WMF is not allowed? [03:27:54] Yes, but I also don't believe that I have done anything like that since that last time. I can't really view the recent changes, so I don't know if something's screwed up, but I don't think I did any kind of mass copying since then. [03:36:38] [1/2] Those wikidata items flew under the radar then. You got away with it before since no one has noticed or did not act upon it. They are still a policy violation regardless of whether they were found out. [03:36:38] [2/2] In this particular instance Tech had to lock the wiki down due to excessive server activity bringing down the whole farm. And this is when we noticed a massive number of articles coming from WMF-hosted wikis (again). [03:37:25] Ultimately it's a Trust and Safety decision what to do with your wiki, so as Tech I can't give definitive answers, just what I know of the situation. [03:45:23] [1/3] Recently pages such as https://wiki.order.life/wiki/Category:Wikipedia_administrators and https://wiki.order.life/wiki/Portal:The_Ash_Cloud_(Edge_Event) were mass-created with no content and just a category. Many more appear in RecentChanges. [03:45:23] [2/3] While empty pages by themselves are not a problem, when you are actively creating them with a bot on a wiki with 17 million revisions, it leads to disproportionate server resource usage. [03:45:23] [3/3] https://cdn.discordapp.com/attachments/407537962553966603/1494181888540282932/image.png?ex=69e1acd2&is=69e05b52&hm=3ce115e19d1b1e5dafff56e9925c510b2ced3b7da54d484a694a0f3ad9a1e1d7& [03:51:43] Oh. Didn’t get that this would be a particular issue. And it was enough of an issue to cause server problems for the wiki? Does making a page scale in expense with the size of the wiki? [03:53:19] Either way can you give me 24 hours to get what I want off the wiki before closing it? [03:59:33] The immediate goal of stopping the bleeding has been accomplished [04:00:02] Please do not do any more automated mass imports in the meantime [04:00:15] I will not [04:00:42] I don’t care about the wiki staying up as long as it isn’t immediately deleted [04:01:08] We will be in touch shortly, this is a situation on top of other situations for our volunteers, so we need some rest before we can fully assess and discuss next steps [04:01:34] Thanks for bearing w delay, it's been an insanely busy period [04:02:01] You’re welcome! I hope you get everything sorted out. [04:10:38] [1/4] This particular incident was due to a data dump generation which exhausted server resources. Mass article creation added fuel to the fire by inflating the size of the wiki, though merely creating them did not cause immediate problems for the server. [04:10:38] [2/4] In general, a lot of the things you did (copying from wikidata or creating empty pages) would've been considered minor issues had they not been done on a massive scale. I don't think anyone would be alarmed if you imported a few hundred items from wikidata or if your bot accidentally created a few hundred empty pages. [04:10:39] [3/4] This is a combination of multiple factors: each issue on its own would've been considered minor. Lifeism Wiki also has the most edits on the whole platform (17 million edits, whereas the largest human-maintained wikis only have less than 3 million edits in their many years of history, see [list](https://communities.miraheze.org/wiki/List_of_all_wikis)). P [04:10:39] [4/4] lus, most items/articles on the wiki seemingly serve no purpose and are unrelated to the topic/scope of the wiki. [04:16:58] That’s understandable. Tbh I don’t really care about it that much. The wiki was a mess of technical debt on my part. Many pages that I had no clue why they were there. I was trying to clean it up but I understand the decision. [04:17:52] I would just like the wiki to stay up for another 24 hours so I can get the stuff I want off of it. [04:22:16] [1/2] Thanks for being upstanding of the situation. I don't see any reasons for immediate wiki deletion since the original concern about bringing down the whole farm was mitigated. [04:22:16] [2/2] The worst case scenario is that the wiki is put on read-only mode for a month or so: this is standard procedure for wikis closed due to policy violations. This should be more than enough time for you to get data out of the wiki. [04:24:28] Okay, that's good to know. Thank you for being understanding about this. And I'm sorry about the server issues. [04:25:36] Yep, thank you as well. We understand it wasn't a situation of intentional/malicious bent, but it is unfortunately reaching a point with two service disruptions. [04:25:40] I expect that I will probably have been able to retrieve the data that I want in a few hours. [04:27:41] Well, I'll be very happy to try to work things out with you. I am very sad that I have caused the farm issues and also a bit surprised at the same time. [04:34:30] is this something you will manage on your own? we're not too keen on generating new backups for the tremendous amount of (non-)content [04:36:03] Yes I have a script running right now that is grabbing the data I care about. ETA 2 hours to completion. Although I would like to have more than 2 hours in case something goes wrong. [04:36:05] [1/5] (No pressure, just want to respond before I go to bed and this gets buried more.) [04:36:05] [2/5] What happens is, I log in on Meta just fine, then go to log in on PPC Wiki using the exact same credentials as stored in my browser, and I get this error: [04:36:05] [3/5] > The supplied credentials could not be authenticated. [04:36:06] [4/5] Thinking back, I know how I probably broke it, too. When I added the email address to the account in Meta, in the confirmation email, I clicked the "did not register" link by accident. Slightly panicked, I went and changed the password (still in Meta) in order to generate a new confirmation, which I then accepted. PPC Wiki doesn't seem to have gotten the me [04:36:06] [5/5] ssage? [04:37:59] I was in the process of trying to clean up the wiki and retrieve what I wanted from it at the time. It inadvertently led to the issues in question. [05:04:38] You will have at least many hours to export data unless something explodes, and I don't think the wiki will blow up the farm now that it's on read-only. [07:53:22] And done. Finished my retrieval [08:03:30] I'm seeing slowness on some wikis. [08:09:14] I do really wonder how rare is the event that brings to farm down [08:10:41] took an embarassing amount of time for me to dig out the fact that the HAWelcome effectively disables the account that is used as the welcome username [08:12:56] forsaken wiki launch [08:13:05] 💀 [08:13:16] Ah yes roblox wikis [08:14:14] TBH the farm did fine during wiki launch. If the pool counter didn't block repeated main page parsing it would've brought the farm down. [08:14:42] at least the farm had a heads up before they sent the announcement message [08:14:55] who knows what wouldve happened if it came out of the blue [08:14:58] So Miraheze is really vulnerable into going offline apparently [08:15:13] Every site is [08:17:36] could be worse [08:17:50] How much worse... [08:18:38] data corruption [08:19:16] i was thinking one notaracham login taking down the farm worse but yeah that too [08:19:42] is his login that powerful? [08:20:29] [[Special:CA/NotAracham]] [08:20:30]

[08:20:34] jokes aside, we do have performance issues when an admin logs into our company application [08:20:44] mr. worldwide [08:20:45] Lol [08:20:46] because it has access to everything and stresses the server [08:21:05] just because it can load everything doesnt mean it should 😭 [08:21:23] What happens when NotAracham changes to YesAracham, MH servers are going to have a field trip lol [08:21:46] nah the rename would just take a while [08:22:04] What about microediting in the GUP? [08:22:12] Last time it gave you guys [08:22:16] that was mostly fixed [08:22:17] A lot of j*bs [08:22:18] and it happens multiple times because for some reason they decided site info and user settings are realtime instead of a database fetch done once [08:22:46] excuse me [08:22:50] i left my laptop idle and it had 1GB xhr requests after 1-2 hours .. no images or videos btw [08:23:30] trash the whole thing oml [08:23:38] it used to be worse when i joined [08:24:05] we were doing the usual "read a csv and insert it into a database" because of course CSVs are the most used CMS in the world [08:24:16] so far so good except [08:24:30] https://cdn.discordapp.com/attachments/1467017687124414554/1468913000844103792/IMG_1015.gif [08:24:38] Does tech team even sleep [08:24:39] they were doing one insert per row to update the same row 5400 times [08:24:49] no 💔 [08:25:24] so people were told to leave the laptop idle for 30 minutes to make all the requests [08:26:17] i was shocked so i just read every row and submit a single request, they were impressed and i got hired [08:28:06]