[11:12:30] 10DBA, 10Operations: dbstore2001 low on disk space - https://phabricator.wikimedia.org/T214831 (10jcrespo) [11:13:47] 10DBA, 10Operations: dbstore2001 low on disk space - https://phabricator.wikimedia.org/T214831 (10jcrespo) [11:15:22] 10DBA, 10Operations: dbstore2001 low on disk space - https://phabricator.wikimedia.org/T214831 (10jcrespo) I think we can delete `/srv/backups/`? As those are duplicated from those on es2001 / bacula. Agree? [12:35:25] for some reason prometheus exporter is failing on db2062 [13:37:45] 10DBA, 10Operations, 10Packaging: db2085 doesn't boot with 4.9.0-8-amd64 - https://phabricator.wikimedia.org/T214840 (10jcrespo) [14:20:46] 10DBA, 10Operations: dbstore2001 low on disk space - https://phabricator.wikimedia.org/T214831 (10Marostegui) +1 to get rid of it: ` root@dbstore2001:/srv/backups# ls -lhrt | tail -n5 drwx------ 2 dump dump 24K Feb 28 2018 s1.20180228121150 -rw-r--r-- 1 dump dump 86 Feb 28 2018 dump.s3.log drwx------ 2 du... [15:20:40] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 crashed - https://phabricator.wikimedia.org/T214720 (10jcrespo) revision was not affected: ` root@cumin1001:~$ ./wmfmariadbpy/wmfmariadbpy/compare.py enwiki revision rev_id db1067 db1114 --step=100000 [...] 2019-01-28T15:13:27.821882: row id 8799... [16:38:22] 10DBA, 10Operations: dbstore2001 low on disk space - https://phabricator.wikimedia.org/T214831 (10jcrespo) 05Open→03Resolved a:03jcrespo https://grafana.wikimedia.org/d/000000274/prometheus-machine-stats?panelId=12&fullscreen&orgId=1&var-server=dbstore2001&var-datasource=codfw%20prometheus%2Fops&from=154... [17:02:25] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Zoranzoki21) >>! In T174802#4857155, @Urbanecm wrote: >>>! In T174802#4764490, @NickK wrote: >> Is it there a guide on how to ma... [17:18:21] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 crashed - https://phabricator.wikimedia.org/T214720 (10jcrespo) I checked the core tables at https://gerrit.wikimedia.org/r/486872 I propose to repool the host, as persistent replication stats worked well as expected: https://gerrit.wikimedia.or... [17:39:59] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Urbanecm) >>! In T174802#4914863, @Zoranzoki21 wrote: >>>! In T174802#4857155, @Urbanecm wrote: >>>>! In T174802#4764490, @NickK... [17:53:40] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 crashed - https://phabricator.wikimedia.org/T214720 (10Marostegui) Go for it! Thanks for checking it!