[00:02:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:00:06] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:02:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:00:07] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:09:40] bonjour 👋 [07:38:04] I'll depool db2189 [08:02:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:00:07] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:41:20] welcome back arnaudb [11:12:24] thanks Amir1! [12:02:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:11:36] hello folks! [12:11:51] I am roll restarting cassandra on restbase-codfw to pick up the new openjdk [13:00:07] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:08:19] Hello. I wonder if anyone could help me with a MariaDB rights question, please. Details are here: https://phabricator.wikimedia.org/T371991#10057528 [13:09:34] I would expect the `s53272` user to have the standard `labsdbuser` role and to be able to see the `btmwiki_p` database. But for some reason it can't see that database. [13:11:54] I'll check btullis [13:17:13] nothing obvious :o [13:23:55] hello btullis and welcome back arnaudb :) [13:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:24:08] btullis: let me check if I can spot what's wrong with that user [13:25:48] was that database created recently? [13:26:49] yep I now remember it's a new-ish wiki [13:38:01] thanks dhinus ! [13:43:38] I can see the grant is set correctly in clouddb1020, but not in an-redacteddb1001 [13:44:27] in mysql.s5, SHOW GRANTS FOR 'labsdbuser' includes btmwiki_p [13:55:50] OK, thanks. So all I have to do is execute this on an-redacteddb1001 (s5): GRANT SELECT, SHOW VIEW ON `btmwiki_p`.* TO `labsdbuser` [13:55:55] Then flush privileges? [13:57:39] probably, but I'm trying to understand why they're missing :) [13:57:48] and if other dbs are also missing [13:58:00] Thanks. I don't quite understand where these grants are created normally, either. It must have got skipped during the migration of clouddb1021 to an-redacteddb1001, but I'm not exactly sure where. [13:58:49] yep my hunch is that something was skipped when you created the new host [13:59:07] so far I found modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py [13:59:44] I don't think that any other dbs are missing the grants. We had an issue with the monthly sqoop from mariadb to HDFS, but every other project database was fine. https://groups.google.com/a/wikimedia.org/g/data-engineering-alerts/c/WDztmSceWD8/m/e6dk-OBIAAAJ [14:00:04] I think it's fine to run the manual GRANT [14:00:18] I'll let you know if I find an explanation of why it was missing [14:00:38] I tried to do a maintain-views on that database here: https://phabricator.wikimedia.org/T368066#9939333 but it didn't seem to make any difference. [14:00:57] Great! Thanks so much. I'll do the manual run and keep looking myself, as well. [14:01:07] I mean, manual grant. [14:04:02] I think if the db already exists it will just skip it (line 469 of maintain-views.py) [14:04:22] that's probably something we could improve in maintain-views.py [14:08:14] Oh I see. Yes, got it. So my second run of maintain-views wouldn't have corrected the missing grant. That is counter-intuitive, isn't it? [14:10:14] kind of, yes, but in theory the grant and the db are created at the same time. so why is the db present but the grant isn't? [14:12:38] elukey: heya, re: T371874 do you want me to take over the restbase restarts? [14:13:03] not sure where you are on that, but thought I'd offer... I expect you have more going on than I [14:14:04] urandom: o/ I am doing codfw atm, should be mostly finished, if you want to do eqiad later on it would be great, otherwise I'll do it tomorrow! :) [14:14:27] elukey: ok, sgtm [15:37:25] FIRING: [2x] SystemdUnitFailed: ferm.service on ms-be1078:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:47:25] FIRING: [2x] SystemdUnitFailed: ferm.service on ms-be1078:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:00:07] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:40:42] update globalusers set gu_password = replace(gu_password, ':A:', ':B:') where gu_password like ':A:%' limit 100; [19:40:55] Amir1: Hey I would like to run something like that ^ [19:41:12] Is that possible to just do or would that be to expensive? [19:42:06] zabe: there shouldn't be any index on gu_password? [19:43:15] yes there is none on gu_password [19:43:24] if there is, that should be fine (but very very very very very careful, at least run it on beta first), if not, add a condition on something that has index, e..g gu_id between 0 and 1000, then another one, and another one [19:43:38] that's what we do on categorylinks [19:44:25] alright [19:44:33] there is no index on gu_password [19:44:47] so I will do it together with gu_id between ... [19:45:00] update globalusers set gu_password = replace(gu_password, ':A:', ':B:') where gu_password like ':A:%' and gu_id between 0 and 1000; [19:45:05] have fun [19:45:25] but again, be very (5x) careful [19:45:38] this is canonical data and direct query on master [19:45:52] yes [19:46:20] thats why I wanted to talk to you about it before just doing it [19:46:48] I will run some select queries on analytics to find out in which id ranges the passwords with the ':A:' prefix live [19:47:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:00:07] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:24:04] FIRING: PuppetFailure: Puppet has failed on thanos-be2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:47:40] FIRING: SystemdUnitFailed: systemd-timedated.service on thanos-be2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed