[01:38:48] apparently gdnsd has us as the second entry on their users page https://gdnsd.org/users.html [01:40:27] soon won't be >:D [01:43:49] huh [01:44:32] or idk [01:44:44] isn't mh planning to use another dns server? [01:45:00] We will be using cloudflare rather than gdnsd [01:45:05] I think we may go with the 'fuck it cloud- [01:45:06] yea [01:45:17] who wants to tell the maintainer [01:45:23] So both Varnish and GDNSD will be gone (two less things to maintain us nice) [01:45:23] i could [01:45:39] Can wait till we finish migrating [01:45:51] I wonder how many people have found out about Miraheze from that page [01:45:51] Cloudflare is replacing both load balancing AND caching? Nice [01:45:53] can or can't [01:46:03] tax: yep [01:46:05] can probably [01:46:21] meanwhile i'm here like "nooo my -X PURGErinos" [01:46:38] oh btw tax i saw you triggered one of the abuse filters earlier today lmao. random but funny [01:46:44] oh? [01:47:35] I cheated and made the edit with my staff account https://login.miraheze.org/wiki/User:Harej?diff=15676&oldid=14449 [01:47:39] There's an external link [01:47:54] yeah that's annoying >_> [01:48:02] wait did it block you? [01:48:03] when does the filter trigger? when not autoconfirmed on the wiki? [01:48:09] I thought it was log only [01:48:21] It did block me [01:48:31] not a warn either [01:48:38] autoconfirmed [01:48:40] well [01:48:45] virtually all of the rainverse wiki editors have experience lol [01:48:45] skipcaptcha right [01:50:12] what the hell is firejail [01:50:52] Sandboxing for security [01:50:53] thing to prevent certain syscalls iirc [01:50:53] this is the wackiest filter syntax ive seen [01:54:08] why are cp* redlinks https://meta.miraheze.org/wiki/Tech:Server_usage [01:54:21] No pages [01:54:24] no one made articles for them [01:54:58] I will be heading off for a few hours this morning. I'll be back this afternoon [01:55:01] well ye [01:55:08] wow he sleeps? [01:55:21] https://mis.thecomicseries.com/comics/190/#comment-3770787; heh [01:55:26] No, I have things to do lol. it is 10AM [01:55:34] of course [01:55:38] did you sleep? [01:55:42] probably [01:55:58] Not much, I went to sleep at 5 and woke up at 6:45 or so [01:56:15] god i gotta step up my insomnia game [01:56:24] also why is icinga restricted to tech, does it show internal stuff? [01:56:28] i felt drowsy at only 12! 12!! [01:56:30] BlankEclair no no [01:57:18] Yes icinga shows restricted private stuff sometimes. Which is redacted when sent events but not on icinga itself. [01:57:36] bring back guest account 2024 [01:57:49] and guest ldap account email spam as well [01:57:50] how many things could the guest account do? [01:58:58] oh [01:59:02] It used to be an ldap account... which just so happened to have email access [01:59:20] yeah i knew that (i looked through closed security tasks out of boredom) [01:59:39] like it had a guest@miraheze.org email or other emails, well emails [01:59:44] bring back email spam :< [01:59:46] BlankEclair that checks [01:59:47] but like, what could you do with ldap [02:00:16] you can't really do much with ldap access other than email, at least by default [02:00:21] maybe spam ldapwikiwiki [02:00:35] rickroll someone using email access :3 [02:00:37] Matomo needs additional flags to grant you access, as does Graylog [02:00:57] maybe spamming Grafana? [02:00:58] what's the guest account in ldap for then? or was it accidentally enabled [02:01:03] Agent: reminding me of an old DEF CON talk I watched where some guys talk about profiting off stock pum n dumb scams [02:01:20] BlankEclair: it was used to grant public access to icinga [02:01:28] oh [02:01:33] PixDeVl: heh [02:02:20] im still not entirely sure what icinga is [02:02:29] same [02:02:41] in my head its just the thing that screams in #miraheze-tech-ops ngl [02:03:11] then again i still dont understand ldap either, though NOT for lack of trying [02:03:15] basically what -tech-ops is or whatever it is that replaced our beloved -sre [02:03:43] -sre became -tech, -sre-ops became -tech-ops iirc [02:03:44] i.e. load alerts, pretty much and a few SSL cert issues every so often [02:03:50] (-sre-ops was a thing, right?) [02:04:00] oh, was -sre finally split into a further channel? [02:04:03] finally [02:04:09] I had always wanted an -sre-bots [02:04:47] oh neat [02:05:25] iirc we had a general tech channel and a tech team ops channel since i joined discord [02:05:31] https://youtu.be/fQPspL_VuD8 found it [02:06:12] defcon 17 dang [02:06:17] this year was 32 [02:07:02] Agent so it's job is to see and yell [02:07:36] 13sounds bout right13 [02:07:38] Icinga is useless [02:07:46] especially when we have a far better error reporting system [02:07:59] it screams at us when mh is down i guess [02:08:00] users screaming in #general about 502s or whatever the latest and greatest error is [02:08:10] 13I mean13 [02:08:13] yeah but i wanna be early in the riide [02:08:14] 13He's not wrong?13 [02:08:35] it can also be funny at times [02:08:47] I miss this error https://usercontent.irccloud-cdn.com/file/jO3ZhQzv/1725156516777.png [02:09:03] now it's all just about 502s :( [02:09:09] 13looking at tech ops can be a good indicator how fucked we are13 [02:09:09] > RECOVERY - wiki.moores.tech - LetsEncrypt on sslhost is OK: OK - Certificate '*.sakura.ne.jp' will expire on Sun 18 May 2025 11:59:59 PM GMT +0000. [02:09:10] > Eh [02:09:12] > That is not OK icinga [02:09:13] > Why on earth do you think that's ok [02:09:28] 13lmao13 [02:09:41] ngl i kinda dig the peachy pink [02:09:57] 13Agent we never get vanlla 500s though13 [02:09:59] 13ty13 [02:10:14] 13i was debating betwee this and 1306this one06 [02:10:21] 13but i like this more13 [02:10:35] 13im a purple and black color scheme enjoyer13 [02:10:35] I forgot about dark mode in error pages [02:10:37] purple is my favorite color (fun fact ig), but ngl i kinda like the pink a bit more [02:10:39] very high tech [02:10:44] 13same13 [02:10:57] 13the new error pages are always dark13 [02:11:00] 13as god intended13 [02:11:05] PixDeVl: bet you'd like my laptop color scheme then [02:11:44] https://files.catbox.moe/i9fhf0.png [02:11:51] 13although boring at times. someone suggested putting xena's drawing of a robotic bee with a glitch face on the error page13 [02:12:04] 13konversation13 [02:12:16] 13oh13 [02:12:24] 13the color is different for you then me13 [02:12:30] yeah i made it lighter [02:13:08] 13oh13 [02:13:32] 13ill send you a screenshot of my wallpaper tmr13 [02:13:45] 13not like i ever see it13 [02:13:56] i'm sorry, is this a personal attack? [02:15:15] 13no13 [02:15:24] 13i just am never on my wall paper13 [02:15:30] 13i always have a window open13 [02:15:32] me neither [02:15:48] (i probably should've added a /j) [02:15:48] 13also youtube ads just reminded me zenless zone zero existed13 [02:16:03] 13should get back to playing that at some point13 [02:16:05] 13was fun13 [02:16:11] btw i sent you the wallpaper i use privately if you haven't notied [02:16:18] 13a13 [02:16:23] w [02:33:55] https://meta.miraheze.org/wiki/Tech:Salt so you connect to puppet to use salt to use puppet [02:34:54] yum [02:34:56] salt [02:35:41] spice of life [02:36:52] someone needs to update ldapwiki's main page [02:39:00] MacFan4000 do the stewards cvt etc ldap groups still exist since moving email to google? [02:39:02] https://meta.miraheze.org/wiki/Tech:Ldap#Existing_LDAP_groups [02:42:35] They exist since I didn't remove them yet but don't do anything. I should probably remove them. [02:43:03] maybe [02:43:18] also is the bot server not documented on wiki [02:43:53] i think bots171 is a late addition to the original server blueprint [02:44:05] hence why there's no info on it [02:44:28] makes sense [02:44:39] did we always have a dedicated bot server [02:45:01] no [02:45:11] it used to be combined with mon server [02:45:32] is there any private info access on bots? [02:46:52] I suppose just the PWB password maybe [02:46:58] and the IRC passwords [02:47:09] which is PWB again [02:47:19] too many acryonums [02:47:23] PyWikiBot [02:47:25] spelling 100 [02:47:28] oh ya [02:47:29] aka the bot that does the archiving [02:47:52] I think we named it BeeBot now [02:47:56] aka whatever AgentBot's successor is since Miraheze maintenance script was axed [02:48:02] [[User:BeeBot]] [02:48:03] ah yes, BeeBot [02:48:10] even though this isn't WikiApiary [02:48:14] wm-bot2 is eepy [02:48:19] Wait what happened to Miraheze maintenance script? [02:48:20] I have a discord bot named BEe [02:48:24] totally differeny [02:48:52] CosmicAlpha: the account used for PWB was changed from the Miraheze maintenance script user to BeeBot [02:49:18] Oh I thought you meant Miraheze maintenance script account itself vanished lol [02:49:24] I didn't know that though [02:50:16] I thought the DT (or whatever acronym is used for DSRE's new name) was the one who spearheaded this [02:50:43] CosmicAlpha is it worth adding pages for undocumented servers like bots and such on wiki [02:51:19] I had nothing to do with any change to pywikibot and was not even aware of it [02:51:54] :thonk: [02:56:08] Also yes replying to an earlier conversation, #miraheze-sre basically became #miraheze-tech-ops, but both this channel and that one repurposed so this channel is used for what #miraheze-sre used to be while #miraheze-tech-ops became more bots. [02:56:43] that don't relay to discord [02:56:58] having them relay to discord would be nice [02:57:33] I thought I enabled bots to the Discord relay after seperation but realized yesterday I didn't. I will at least relay icinga now lol [02:59:46] hopefully icinga has calmed down now [02:59:57] I removed all but wm-bot fron relay ignore list [03:00:37] It has yes [03:00:48] why wm-bot [03:01:02] it's a pain to not be able to see irc links sometimes [03:01:09] not that he works [03:01:21] wonder whats with that [03:01:37] oh [03:01:41] relay died [03:01:51] we're in our on- [03:01:57] nevermind relays back [03:01:59] Because I forgot if the bot will handle no Ignore configure so I needed to leave one for now lol [03:02:04] It restarts when config is changes [03:02:08] *changed [03:03:05] ping [03:03:11] pong [03:03:45] ping [03:04:39] pong [03:04:59] I should add a ping pong auto bot response to the relay lol [03:05:18] or a !test or something [03:05:38] but it is to much work for a minor thing [03:14:09] heh [03:35:28] CF free includes DDoS protection but not DDoS alerts [03:35:30] silly [03:37:01] does MH have free Business or did they leave us with free Enterprise? [03:37:20] that is, referring to CloudFlare plans [03:37:29] We have enterprise on the miraheze.org domain right now [03:37:34] (It’s per domain) [03:37:49] We have an open ticket to get it for wikitide.org [03:37:58] All other domains are on free [03:38:33] interesting [03:38:39] it's nice that they never corrected it [03:38:49] corrected? [03:40:11] I was checking my old WikiTide email when I found an email from CF explaining that they couldn't give us the Business plan due to a technical error so we could get Enterprise until they fixed the issue [03:40:26] If they haven't downgraded us then they probably haven't fixed the issue :p [03:41:14] let's hope they don't rhinos has been having a field day bitch slapping naughty bots [03:41:45] Account wide Enterprise access would be soooo nice [03:42:23] im rechecking the differences between business and enterprise [03:42:54] Cache Purge - By Tag [03:42:55] Cache Purge - By Host [03:42:55] Custom Cache Keys [03:42:56] Prioritized IP Ranges [03:42:56] Prefetching URLs [03:43:43] smt called subdomain zones [03:43:50] hopefully not important [03:44:18] Enterprise DDoS Mitigation is its own thing [03:44:41] https://www.cloudflare.com/plans/ [03:46:51] I'm surprised things like Argo and load balancing aren't included [03:47:17] CosmicAlpha: did we ever look into having CF cover R2 for us? [03:47:29] It's something they do for other projects [03:48:02] R2? [03:48:14] CloudFlare's answer to S3 [03:48:18] aka file storage [03:52:11] hey PTW is mentioned on test wikipedia lol [03:52:33] https://test.wikipedia.org/wiki/Wikipedia:Requests/Tools [03:52:46] not sure why it says restricted [03:53:03] maybe that means full no but didnt wanna use red cause its not a bad thing [03:55:13] Not yet depeneds on if the performance issues of it can be resolved if we will use it or not. [03:57:33] PixDeVl: hmm, well we still don’t have enough notable sources to have a Miraheze page on WP [03:57:37] Agent: it is still enterprise yeah. We also have account wide load balancer, and a ton of other services we have requested also. [03:58:24] Also it says CheckUser is restricted, because it is [03:58:52] Restricted and yellow implies limited access but access [03:59:04] A couple of the others have a modified version that allows users to run CU only on themselves or their own IP [03:59:20] What is the point? [03:59:24] to me [03:59:30] testing [03:59:49] Privileges users can still have full access [04:00:07] checkuser vs checkuser-limited permission [04:00:48] Why are we stopping using gdnsd? Miraheze speed issue should be looked into per us noticing that it was faster connecting directly to cp3* indicating there was some problem with connections. [04:03:06] (And one of the listed wikis is run and maintained by me) [04:03:28] all is macfan [04:47:40] @paladox interesting though, when was it you tested out directly to cp [04:48:32] (also how do you directly connect to a proxy and tell it which wiki huh) [05:19:08] @paladox because it doesn't work for cloudflare, everything will go through cloudflare. [05:20:43] cloudflare protects us against all the DDOSes is the primary reason for switching, but it will also be used for load balancing and CDN since we get it all completely free. [05:21:08] i wonder, what happens if the bastion hosts are offline [05:21:43] It has happened and causes quite a mess due to firewall. We will be launching a backup bastion external to DC to prevent it in the future. [08:01:01] (sending questions here that i initially asked elsewhere just in case i could get a response) [08:01:03] > [Saturday, 31 August 2024] [11.58.05 GMT+10] where do i ask for more info about why a thing in a commit was done that way? the linked task doesn't give any clues either [08:01:04] > [Saturday, 31 August 2024] [11.58.08 GMT+10] here's the commit: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/5265579c0f9a11254811b50b3e72cbd8fbd83ede%5E%21/ [08:01:06] > [Saturday, 31 August 2024] [11.58.16 GMT+10] no idea why it's done in javascript and not in php... [08:01:07] > [Saturday, 31 August 2024] [12.03.25 GMT+10] doesn't look like anything else uses the new "magnify-resource" parameter that the commit introduced either: https://codesearch.wmcloud.org/search/?q=magnify-resource [08:46:45] It's such a shame Miraheze is taking such a big step back from free and open source software. Guess that's one of the key principles WTF have gotten rid of [08:54:29] [1/2] Why do we need open source DNS anyway? Most everything is still using open source, and cloudflare is free for us? Why not take advantage of what we can as a nonprofit? Yeah its definitely a shame we are prioritizing user ended performance instead rather than OSS for just DNS/CDN. [08:54:29] [2/2] If this is the view point I get as a volunteer who puts effort into things. I honestly don't know why I even do this. I don't intend to quit, but I just can't seem to understand some view points here but you're entitled to your opinion and I respect that. [08:54:52] [1/2] do we have our own resources for protection against aggressive scraping and ddos? we aren't even wmf [08:54:52] [2/2] I don't trust cf that much myself, but it looked like the best solution against having the entire farm down or being slow like we are back to WD servers, or blocking a whole region [08:56:27] Because Miraheze was founded on a commitment which has been the same until WikiTide took over - there's a legacy that has been binned that's managed to survive a lot. [08:57:06] The FAQ needs updating as it mentions a commitment which has been the same for 7 years, which is a) no longer true and b) not even on the wiki anymore [08:57:27] So we did with MHL [08:58:07] like we were down for a whole day or more? [08:58:19] That was resourcing constraints - nothing to do with DDoS [08:58:41] Things change as the need arises. We are still open source for 99% of services. You can't think technological things are the same now as they were 7 years ago. [08:59:23] I will refrain from arguing about it right now. I respect your point of view, while I disagree, you are entitled to it. [09:00:15] I'm not saying technology is the same, it's just clear the commitment from WikiTide Foundation isn't, and two different orgs would have sought to protect two different things. [09:01:11] I'd be curious to see what process was undertaken when selecting CloudFlare and what alternative OSS options were considered [09:01:51] If the answer is none, then you can't argue the commitment was maintained, it was a jump from private, closed and proprietary paid software [09:03:53] Also the argument of "we get it for free so might as well use it" does not sit right with me as justification for moving away from open source software. Open source software is also free, so why not just keeping using it? [09:07:01] removing varnish feels a bit :/ to me, but it is what it is [09:09:37] [1/2] We will likely be keeping open sourced DNS using bind rather than gdnsd but using Cloudflare for CDN to avoid so many outages with DDOSing, cache filling up etc... [09:09:37] [2/2] At least its the current plan. [09:10:52] Why the change away from GDNSD? It's worked for 9 years and powers one of the largest websites in the world [09:11:28] question: how would you use cloudflare without pointing your nameservers over to them? [09:11:39] Because it doesn't support apex cname or alias records we need [09:11:59] We would use our dns using bind and then apex alias records. [09:12:24] cf lets you do that? [09:12:40] Yes [09:12:43] huh, til [09:12:46] You need because of closed source software? Mkay [09:12:55] Won't argue that point any furtjer [09:14:43] To be honest, I don't blame the move away from all the existing software. I guess it makes sense when the tech team doesn't understand the existing infrastructure [09:14:55] Probably easier to build from fresh [09:38:53] There is a valid argument tbh for cloudflare [09:38:58] We need ddos protection [09:39:18] We have seen ddos attacks that we simply can't fight off on our own [09:39:35] The WMF use cloudflare too (just in a different manner) [09:41:02] And regarding GDNSD, we are considering the move to bind cause it seems to be the best option for the features we need [09:41:17] GDNSD is too slow to support what we need [09:41:23] Bind has a copy left license [09:41:45] It is open source [09:41:47] @owenrb [09:42:28] The DDoS protections we had were about as good as a chocolate teacup [09:42:43] i want chocolate now... [09:43:00] BlankEclair: not a chocolate teacup though [09:43:38] I didn't say bind wasn't, the comment was originally in relation to the comment of using CloudFlare for DNS management [09:44:29] That was originally the plan that has since changed as we realized it wouldn't work. I like the new plan of using bind, which is still OSS better [09:44:50] We only have cloudflare for the 10 domains we own [09:45:10] how does wmf use cf? [09:45:11] All other ones are staying in the dns repo [09:45:22] BlankEclair: it's called spectrum I think [09:45:43] It sits in the path the ip takes rather than proxying through them [09:46:20] BlankEclair: https://www.cloudflare.com/en-gb/case-studies/wikimedia-foundation/ [09:46:22] Magic transit [09:46:54] I remember that ddos [09:47:07] That is what I originally wanted us to use but we couldn't it was way to expensive and we couldn't get it free. [09:47:38] That WMF attack was a very large one [09:47:43] It actually took us down too [09:47:46] Cause instant commons [09:47:51] But that was poor config [09:48:52] That was in 2019 or 2020 wasn't it? [09:49:13] 2019 according to the article [09:49:19] It was before I joined Miraheze but I think I remember it also. [09:49:20] huh interesting [09:50:04] Oh yeah, September 7, 2019 [09:50:06] BlankEclair: the guy was tweeting about it that supposedly did it [09:50:22] It went on quite a while in waves [09:50:29] The fact it took us down was funny [09:50:37] Instant commons didn't have a timeout in our config [09:50:47] And connections to WMF were taking forever [09:50:51] hehe the funny no timeout cause miraheze to die 🥰 [09:50:56] (i have no idea how my mind works) [09:51:06] So it used up all php workers [09:51:11] I set a timeout eventually [09:53:53] BlankEclair: we used to have a lot of preventable outages [09:54:04] *gets popcorn* [09:57:26] BlankEclair: we do make a lot up as we go along [09:57:38] And have mostly people who lack actual experience [09:57:42] make a lot of what up? [09:57:50] BlankEclair: everything [10:00:48] wow [10:01:49] Why is relay so laggy [10:02:21] To which bit? [10:02:44] how wmf ddos took miraheze down [10:02:52] Ah [10:02:58] the more you know [10:04:05]