[03:22:38] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [06:47:02] 10SRE-tools, 06DBA, 06Infrastructure-Foundations: Raid handler for broadcom disk did't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10894402 (10Volans) 05Resolved→03Open [06:47:25] 10SRE-tools, 06DBA, 06Infrastructure-Foundations: Raid handler for broadcom disk did't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10894408 (10Volans) From IRC logs the handler was triggered fine: ` Jun 07 04:22:14 alert1002 icinga[1502926]: SERVICE EVENT HANDLER: db2226;Del... [06:58:30] 10SRE-tools, 06DBA, 06Infrastructure-Foundations: Raid handler for broadcom disk did't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10894411 (10Volans) As a side note `sudo /usr/lib/nagios/plugins/check_nrpe -4 -H db2226 -c get_raid_status_broadcom` returns an exit status of... [07:03:09] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations: Raid handler for broadcom disk did't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10894413 (10Marostegui) [07:22:42] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [09:36:56] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations, 10Observability-Alerting: Raid handler for broadcom disk didn't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10894787 (10elukey) I've tried to disable puppet on alert1002, and manually change the service_descrip... [11:09:52] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations, 10Observability-Alerting: Raid handler for broadcom disk didn't automatically open task on db2226 - https://phabricator.wikimedia.org/T396319#10895014 (10elukey) 05Open→03Resolved To keep archives happy, cross posting from the parent ta... [11:23:25] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [14:34:36] 10Mail, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#10895613 (10jhathaway) p:05Triage→03Medium [14:34:40] 10Mail, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE: Add DMarcian trial-account address to the dmarc-ruf@wikimedia.org mailing list - https://phabricator.wikimedia.org/T396062#10895615 (10jhathaway) p:05Triage→03Medium [15:23:34] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [19:23:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:23:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts