[09:48:33] 10DBA, 10Operations, 10Puppet, 10User-jbond: Request new database for pki.discovery.wmnet - https://phabricator.wikimedia.org/T268329 (10jcrespo) 2 tables and its schema were backed up yesterday, with around 4K in size after gzip compression. If that seems right I would call out the backups "working". Ple... [09:49:35] 10DBA, 10Operations, 10Puppet, 10User-jbond: Request new database for pki.discovery.wmnet - https://phabricator.wikimedia.org/T268329 (10jbond) >>! In T268329#6678525, @jcrespo wrote: > 2 tables and its schema were backed up yesterday, with around 4K in size after gzip compression. If that seems right I w... [09:53:42] 10DBA, 10Operations, 10Puppet, 10User-jbond: Request new database for pki.discovery.wmnet - https://phabricator.wikimedia.org/T268329 (10jcrespo) >>! In T268329#6678526, @jbond wrote: >>>! In T268329#6678525, @jcrespo wrote: >> 2 tables and its schema were backed up yesterday, with around 4K in size after... [09:58:59] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10jcrespo) @kostajh backups ran yesterday on both datacenters, but the database backups seem to be empty. Please reopen/ping me if that is unexpected. [10:01:44] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10jcrespo) @hnowlan backups ran yesterday on both datacenters, and they returned an empty database. Please reopen/ping me if that is unexpected. [10:04:32] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10kostajh) >>! In T267214#6678535, @jcrespo wrote: > @kostajh backups ran yesterday on both datacenters, but the database backups seem to be empty. Please reopen/ping me if t... [10:05:28] 10DBA, 10Patch-For-Review: Add a link engineering: Database for link recommendation service - https://phabricator.wikimedia.org/T267214 (10jcrespo) Thanks for the confirmation. Then backups were setup correctly. Feel free to recheck with me after first import. [14:33:33] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10gmodena) hey @jcrespo & @hnowlan just chiming in to let you know that we have not ingested data yet. Empty backup was expected. [14:36:45] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10jcrespo) Thanks for checking. Let me know if you want me to recheck when data introduced. [17:36:12] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on es1023 - https://phabricator.wikimedia.org/T268796 (10Cmjohnson) 05Open→03Resolved The disk has been replaced and is rebuilding, please re-open if the problem persists [17:45:04] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on es1023 - https://phabricator.wikimedia.org/T268796 (10jcrespo) As usual, Chris, thank you for the quick response! [17:50:16] we should keep an eye on dbstore1004, potential, albeit very small but builds-over-time memory leak, I had to restarting twice in the last 6 months, may send a warning soon [17:50:32] (icinga, not me) [18:05:13] too late, it started already: WARN Memory 90% used. Largest process: mysqld (29717) = 42.4% [21:26:03] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Jclark-ctr) Replaced more Dimms per HP no errors at this [21:37:23] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot (issue continues after board change) 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Jclark-ctr) 05Open→03Resolved