[05:51:19] 10DBA, 10Operations: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10Marostegui) [06:00:24] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: db1140 (backup source) crashed - https://phabricator.wikimedia.org/T250602 (10Marostegui) p:05Triage→03High a:03jcrespo This server is fully broken apparently, it is not powering ON :-( - I have tried multiple combinations: ` hpiLO-> power on status=0 s... [08:52:20] I'll ask this question here in the case that some magical DBA knowledge might fly my way :) [08:53:00] I'm running a bunch of wikis on a single db currently (150), and have not yet setup external storage for text, it is one of my next things to do.. [08:53:32] Am I actually best to go down the external storage sql route as documented on mediawiki .org, or is there potentially something new / more moden that I should / could be using? [09:40:35] addshore: what do you mean? you mean using mysql as the database for external store data? [09:40:45] or potentially using another dB engine? [12:55:31] potentially another [12:56:13] I actually just saw that there are a few different ExternalStoreMedium implementations in core! [12:56:30] Including a ExternalStoreMwstore which uses a configured filebackend, which is an interesting idea [15:40:36] addshore: the problem is the abstraction layer and whether MW actually supports any other engine no? like rocksdb which is key:value store for instance [15:41:40] It looks like I might actually be able to hook it all up to write historic revision content to a generic S3 API interfaces bucket, which could be interesting [15:41:49] And then keep current text revisions closer [15:42:39] Unrelated question, We have some form of sql proxy in prod right? (I remember seeing these dbproxy things) what powers them? [15:43:07] no [15:43:16] they are on misc clusters [15:43:36] and for labsdb [15:43:48] we just use haproxy to re-route things [15:44:15] misc clusters are m1,m2... which host stuff like Gerrit, phab, otrs etc [18:08:46] otcha [18:08:48] *gotcha [18:08:58] Yeah, I figured something like ha proxy might be all I need