[00:17:19] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:16:56] FIRING: SystemdUnitFailed: systemd-timedated.service on ms-be1075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:17:19] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:31:56] FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be1075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:17:20] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:21:56] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1246:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:26:11] I'm having trouble with dbctl https://phabricator.wikimedia.org/P69475 [09:26:25] I'm trying to depool a host [09:29:03] ok, I fixed it [09:29:04] did you check it depooled the host? at fist sight it seems a failure to connect to the SAL logger [09:29:05] I think [09:29:36] idk what went wrong, I'll check the log after our meeting but I'm betting you're right [09:30:02] actually no it seems related to the new audit function of conftool [09:30:12] released reacently that is generating ECS logs [09:30:22] and sending them to I guess logstash [09:30:33] ah, maybe because of the progressive repooling there was some schema validation issues? [09:30:56] _joe_: does this rings a bell? https://phabricator.wikimedia.org/P69475 seems related to the new audit functionality [09:31:43] <_joe_> volans: yeah sorry [09:31:58] <_joe_> I just stopped conftool2git to understand why some hooks weren't working [09:32:01] <_joe_> I'll restart it now [09:32:13] lol, ok so just operational unluckiness :) [09:32:15] <_joe_> your change should still have worked [09:32:19] <_joe_> yeah [09:32:22] reassuring :D [09:32:23] thanks [09:33:43] <_joe_> but maybe it's good to create a task stating we should silence errors in logging from that handler [09:33:56] <_joe_> I have no idea how to do it though [09:34:46] <_joe_> arnaudb: things should be back to normal [09:35:14] <_joe_> and to be clear [09:35:19] <_joe_> your changes were still applied [09:36:54] _joe_: for the suppressing errors I guess implementing handleError() should do it [09:37:06] <_joe_> ack [10:10:01] thanks, noted! [12:17:20] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:13:55] hello folks! [13:14:13] I have a code review that is swift-related (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1078380) but I noticed that Matthew is out [13:14:27] is there anybody that can review it in their absence? [13:21:56] FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be1075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:17:20] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:21:56] FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be1075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:17:20] FIRING: [2x] PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:21:56] FIRING: [2x] SystemdUnitFailed: systemd-timedated.service on ms-be1075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed