[01:53:55] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2400928 (AlexMonk-WMF) a:AlexMonk-WMF I'm having a go at this. > It blows up and rebuilds all wikis on every run. It truncates the meta_p.wiki... [07:09:16] DBA, Operations, Patch-For-Review: reimage or decom db servers on precise - https://phabricator.wikimedia.org/T125028#2401018 (jcrespo) It is a bit more complex than that- we need to failover the slave actions to the master (and use only the master). Then (for example, the following week) we need to... [07:14:06] DBA, Operations, Phabricator: Upgrade m3 (phabricator) db servers - https://phabricator.wikimedia.org/T138460#2401019 (jcrespo) [07:15:54] DBA, Operations, Patch-For-Review: reimage or decom db servers on precise - https://phabricator.wikimedia.org/T125028#1972522 (jcrespo) I have created T138460 specifically for Phabricator. Related to T137928#2389155, too. [07:18:55] DBA, Labs, Labs-Infrastructure, Operations, and 2 others: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2401040 (jcrespo) labsdb1002 will never get fixed. [08:09:51] DBA, Labs, Labs-Infrastructure, Operations, and 2 others: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2401109 (Jdforrester-WMF) stalled>Resolved a:Jdforrester-WMF In that case, I'm declaring this fixed. [08:09:54] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2401114 (Jdforrester-WMF) [08:22:11] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2400728 (jcrespo) I have to add a view to a newly created labs-only table, so it is created for new wikis, too: ``` MariaDB L... [08:23:25] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2401137 (jcrespo) [08:23:28] DBA, Labs, Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2401136 (jcrespo) [12:39:38] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2401809 (jcrespo) >> It blows up and rebuilds all wikis on every run. >It truncates the meta_p.wiki table but it doesn't drop... [13:58:06] DBA, Labs, Labs-Infrastructure, Operations, and 2 others: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2402068 (Krenair) a:Jdforrester-WMF>ori [14:54:21] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2402245 (AlexMonk-WMF) >>! In T138450#2401809, @jcrespo wrote: >>> It blows up and rebuilds all wikis on every run. >>It trunc... [14:55:53] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2402249 (jcrespo) > Who in the ops group could be its 'sole owner'? No one else has any access to these systems. Maybe labs a... [16:07:54] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2402468 (chasemp) I'm 100% on board for being on the hook for this process, or at least being a partner. We can coparent :)... [16:09:49] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2402474 (chasemp) p:Triage>High [16:51:13] DBA, Operations, Wikimedia-Etherpad: etherpad database issues - https://phabricator.wikimedia.org/T138516#2402605 (jcrespo) [16:53:07] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2402631 (jcrespo) [17:03:00] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2402704 (jcrespo) p:Triage>High [17:44:17] DBA, Labs, Labs-Infrastructure, Operations, and 2 others: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2286195 (ksmith) Thanks to everyone who helped get this unstuck and fixed! [17:55:12] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2402828 (jcrespo) [19:01:54] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2400728 (scfc) I once thought of a tool that does something like `diff -u <(mysqldump --no-data) <(what-views-and-triggers-and... [19:05:20] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2402968 (jcrespo) @scfc redactatron is a horrible piece of software and we do not want to expand it, but kill it. It has its f... [20:05:06] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2403090 (AlexMonk-WMF) >>! In T138450#2401133, @jcrespo wrote: > I have to add a view to a newly created labs-only table, so i... [20:20:13] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2403157 (AlexMonk-WMF) @jcrespo: I was wrong in my last comment and have uploaded https://gerrit.wikimedia.org/r/295751 which,... [20:32:00] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403185 (jcrespo) So good news: we have been able to recover until just a few minutes before crashing (which means virtually no data loss). The problem is we have yet to reimpor... [20:35:18] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2403187 (AlexMonk-WMF) >>! In T138450#2402468, @chasemp wrote: > * For [[ https://phabricator.wikimedia.org/T135029#2400629 |... [20:36:29] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403189 (Effeietsanders) Can you please make sure to not overwrite the things added later? I re-did a bunch of the work I did this afternoon in preperation of the discussions tom... [20:40:31] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403193 (jcrespo) @Effeietsanders as I sent on my email- no data will be added, deleted or overwritten on the current etherpad. **I promised that and I will maintain that.** We t... [20:42:21] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403195 (jcrespo) Clarification: no data will be added, deleted or overwritten on the current etherpad **by us (operators)**, you are expected to do that as usual (use the curren... [20:47:57] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2403204 (jcrespo) @scfc BTW, I actually documented [[ https://wikitech.wikimedia.org/wiki/MariaDB/Sanitarium_and_Labsdbs | red... [20:53:00] DBA, Operations, Wikimedia-Etherpad, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403209 (jcrespo) @akosiaris I managed to reimport the tables, with two different timestamps. They are on the same host (m1-master), and I have granted permission to the same use... [21:23:09] DBA, Labs, Labs-Infrastructure, Blocked-on-Operations, Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2403326 (ori) >>! In T138450#2403187, @AlexMonk-WMF wrote: >>>! In T138450#2402468, @chasemp wrote: >> * For [[ https://phabri... [21:39:48] DBA, Operations, Wikimedia-Etherpad, Patch-For-Review, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403399 (akosiaris) Thanks to @jcrespo 's efforts and using the `etherpadlite_restore2` database, we now have http://etherpad-restore.wikimedia.org. This is... [22:44:49] DBA, Operations, Wikimedia-Etherpad, Patch-For-Review, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403642 (jcrespo) It is finally working: https://etherpad-restore.wikimedia.org (if it does not, wait for your DNS cache to update). Please recover anythin... [22:46:16] DBA, Operations, Wikimedia-Etherpad, Patch-For-Review, User-notice: etherpad database issues - https://phabricator.wikimedia.org/T138516#2403643 (jcrespo)