[09:34:09] fellas [09:35:40] things got so slow I can't even reply w/ DT [09:36:05] it fails to load input block and throws server error message [09:38:10] any updates on the matter? [09:42:51] check https://grafana.wikitide.net at all time, it seems that loads are at it's highest ever recorded [10:22:05] its been like that all week tbf [10:22:15] grafana won't even open now [10:23:23] [02/08/2024 20:22] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:26:55] looks like "system call" refers to system(3), not just any regular syscall (that'd be absolutely horrifying tbh) [10:28:49] looks like it timed out executing nslookup [10:35:01] i don't think monitoring is doing too hot: https://files.catbox.moe/jihvbh.png [10:35:33] Rip [10:36:27] oh wow i think like 75% of everything failed [10:36:36] thought it was just the mediawiki dashboard [10:36:53] oh it's back up i think [10:48:13] i'd be the type of person who'd run wikiteam3dumpgenerator on a wiki even though miraheze is 502ing 50% of the time [10:48:21] (uh, should i stop?) [11:04:12] BlankEclair: yes [11:04:22] done [11:04:45] BlankEclair: if lots of prom stats are failing then it's more likely that prom is broken [11:05:17] accessing wikis has been lagging and 502ing constantly though, especially today [11:05:23] though i also felt lag yesterday [12:06:49] [1/2] I don't think mw171 is doing so good [12:06:49] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1268902826428338267/1722600407872.png?ex=66ae1d58&is=66accbd8&hm=0ce24d094dc36eb80481b64aafc21d885160f4adb1cd5a9ca29ae5d181edfaec& [12:07:47] man [12:08:08] yum watermelon [12:08:27] is anyone able to investigate? don't feel like watching drives failing again [12:09:58] brb gonna make a quick xml dump [12:31:29] Hard drives won't fail [12:32:25] top ten moments said before disaster [12:36:55] Theres no gurantee of that [12:37:09] BlankEclair: I didn't say anything else wouldn't fail [12:37:10] DC harddrives are less likely to fail but not impossible [12:37:30] I meant as a direct result of the high load [12:37:38] yeah we're joking :p [12:37:43] It's not going to directly cause a failure [12:37:51] wait do you see the strikethroughs? [12:37:54] They could still fail for other reasons [12:38:06] And other components can fail [12:38:11] Yes strike throughs are bridged also [12:38:24] what about RhinosF1's irc client? [12:38:37] oh nvm responded on discord [12:38:38] I can yes [13:10:39] Can't a reboot be usefull? Wouldn't take long to do that? [13:11:50] I'm tired of waiting and waiting for each page to load and the 502's every now and then popping up. Feels like we're back in February where we constantly had 502's [13:12:03] It's been bad for days now [13:13:38] i'd prefer a proper diagnosis, though i suppose nothing ever went wrong from the legendary "have you tried turning it off and on again?" [13:20:44] BlankEclair: it could definitely worsen things [14:34:33] @bluemoon0332 someone is looking at our cf ticket [15:04:00] Is this the thing that needs to get done to unblock server-side perf improvements? I seem to recall that OS had a few different ideas on how to improve the turbo-latency lately [15:07:24] Cloudflare is part of the issue [17:56:22] @bluemoon0332 you about? [17:56:32] or anyone with cf access (cc @Infrastructure Specialists ) [18:18:50] setting up load balancing now [18:19:02] @bluemoon0332 thanks [18:19:10] i don't have permissions to fiddle with it [18:19:23] i messaged UO to see if he can add them [18:19:46] not sure who else can give me that page [18:20:13] there's another super admin but only UO is allowed to action access requests [18:20:41] it's not technically an access request [18:20:55] the service just never existed to add me to before [18:40:09] Finally tried connecting to my VPS via my phone [18:40:12] Pretty painless [18:40:44] Maybe I’ll try send set up phorge [18:41:29] Though on a phone that sounds painful [18:45:36] I'm not sure if that's brave or insane [18:45:37] Or both [18:45:51] One of those has been well established [18:46:15] God we should really set up a quips instance [18:46:26] Also [18:46:38] [1/2] WHY ARE THERE SO MANY DNS RECORDS [18:46:39] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1269003446225469470/image0.jpg?ex=66ae7b0e&is=66ad298e&hm=6e52f39d0fe6aad70f2ceebb8802e4cf4d60e33d99f6cb471560b9d15e5f1d7c& [18:46:50] Many reasons [18:46:55] And also just do it [18:47:51] Cry [18:48:12] I’m reading a quick primer on DNS so I can set up domains for my dev env [18:48:24] https://howdns.works is adorable [18:49:34] @bluemoon0332 you need to delete the existing * records too [18:49:48] or i can [18:51:43] they're gone [18:51:49] they are now [18:51:54] page didn't update for me [18:52:09] from what I can see traffic is actually load balanced according to the x-served-by header [18:52:46] there's monitoring as well, I'll document everything later [18:52:51] I'm off for dinner [18:52:55] ye i can't see anything yet [18:53:15] what do you mean? [18:53:23] cause I can't see load balancing [18:53:38] or I'd have a look at how it was working [18:53:42] ahh [18:53:47] well I assure you it is there [19:05:32] @bluemoon0332 rate limting been active for minutes and 1 badly behaving crawler is already banned [19:05:45] well being challenged for the next day [19:07:55] something weird started yesterday [19:09:58] i'm challenging the ASN now [19:10:02] it's Byteplus [20:04:01] Does the name part of a DNS record affect anything or is it a note [20:04:33] On the thing I’m using the default parking record is named @, is just just mean all subdomains of a domain [20:04:45] #notadnsnerd [20:08:49] I should read more on DNS [20:08:59] Makes my head hurt ~_~