[06:44:24] I have switched x1 to STATEMENT to run a few schema changes [07:02:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1186:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [07:02:31] ^me [07:14:11] Can I get a review for https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/826202 [07:17:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1186:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [07:20:34] marostegui: done [07:20:42] thanks :* [07:36:33] thank you both [07:41:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1188:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [07:42:00] ^me [07:56:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1188:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [08:48:45] if someone has time to review ips and hostname here https://gerrit.wikimedia.org/r/c/operations/puppet/+/826220/ - I would be grateful! [08:50:44] on it [08:51:14] <3 [09:07:34] backups don't depend hardly on writing stats (they continue even if writing to the db fails) but they are nice to have [09:08:38] sorry to insist on dbbackups vs backups on the patch, but for me that names vs bacula and mediabackups help my mental map [09:11:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1190:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [09:20:13] yep, makes sense :) [09:56:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1190:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [10:59:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1191:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [11:49:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1191:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [12:18:09] Amir1: for tomorrow's s4 failover, db1143 will fail to be moved, that's ok. I am leaving it up as I am troubleshooting with mariadb [12:18:30] marostegui: sure, sgtm [17:28:37] https://grafana-rw.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&viewPanel=14 [18:15:16] oh nice [18:30:26] duh, of course it's here :) [18:31:18] Hello, I am wondering if someone could check an existing database grant for me. [18:31:38] basically I want to know if a new server can already access the same databases as an old server it replaces or not [18:31:49] like if there is an .% or just limited to really that one IP [18:31:55] mutante: I'm not next to a computer but I could check tomorrow morning if no one else does it before [18:32:29] marostegui: thank you, yes, sounds good [18:34:24] mutante: if you can provide the details [18:36:30] marostegui: I have existing server phab1001.eqiad.wmnet (10.64.16.8, 2620:0:861:102:10:64:16:8) which has _various_ grants to various databases (phab has many) and then I have new server phab1004.eqiad.wmnet (10.64.16.101, 2620:0:861:102:10:64:16:101) and I want to do if it we need to add it to all of them or if it will "just work" because there are wildcards in the existing GRANTS [18:36:41] s/do/know [18:37:04] also I want to avoid that my new server talks to any DB before it's supposed to [18:38:54] let me turn it into a phab ticket? [18:39:20] would you like that if I just make a ticket and assign it (to team or individual) [18:44:37] yeah, that would be easier to handle [18:44:48] no need to assign it, we'll take it [18:46:14] will do, thanks [18:46:54] ty [19:28:26] I'm seeing a lot of swift storage errors 1 ____▁__ 1903 1903 UploadChunkFile..... .25 i/u/UploadFromChunks:359 Error storing file in '/tmp/phpAGr8gY': backend-fail-internal; local-swift-eqiad