[10:47:29] DBA, Operations: Email spam from some MariaDB's logrotate - https://phabricator.wikimedia.org/T127638#2363711 (jcrespo) Open>Resolved a:jcrespo I have deleted all cron jobs on db* hosts trying to rotate mysql logs (we do not allow debian to access mysql in production). We can create later a... [10:49:12] DBA, Operations: Email spam from some MariaDB's logrotate - https://phabricator.wikimedia.org/T127638#2363720 (jcrespo) The above tickets exists and it is: T127636 [10:51:03] DBA, Patch-For-Review: MariaDB missing logrotate for error and slow logs - https://phabricator.wikimedia.org/T127636#2363733 (jcrespo) Some of the above have been reimaged already/there are blocking tasks to do to reduce its verbosity in the first place. As I mention on T127638 I would like to work towa... [15:42:58] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2364726 (Papaul) @jcrespo I am sending the log file to the Dell support engineer, I will update you on the status. [18:44:06] DBA, Beta-Cluster-Infrastructure, Flow, Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2365401 (Mattflaschen-WMF) Really sorry, I accidentally started running it without the dry run option. So it did it for real... [19:04:31] DBA, Beta-Cluster-Infrastructure, Flow, Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2365447 (Mattflaschen-WMF) Dry run completed: P3226 Skimming it, it looks good. @jcrespo Let me know when I should proceed t... [19:17:05] chasemp: I'm unable to tunnel into my DB no longer [19:17:12] The problem started just now. [19:17:23] My SSH connection to labs is still open. [19:17:58] mysql workbench just gives me network is unreachable. [19:57:23] CP678|Laptop: Chase is out on vacation for a while. [19:57:34] :-/ [19:58:14] There is ongoing maintenance in Labs that may be causing brief connectivity issues. Not aware of anything systemic being broken [19:58:41] ok [20:57:01] DBA, Operations, ops-codfw: es2017 and es2019 crashed with no logs - https://phabricator.wikimedia.org/T130702#2365897 (Papaul) @jcrespo here is the reply back from the Dell support team so I will need another down time on those systems tomorrow or Friday. Hello Papaul, We have just found a fix fo... [21:46:29] bd808: It seems to be working again