[11:48:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=drmrs&var-instance=dns6001:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [11:53:20] FIRING: [5x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [11:58:20] RESOLVED: [6x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:03:50] FIRING: [7x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:04:05] RESOLVED: [7x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:08:50] FIRING: [12x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:13:50] FIRING: [11x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:14:05] FIRING: [12x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:18:50] RESOLVED: [10x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:47:50] uh? [12:48:27] I will check later what happened [13:25:56] 06Traffic, 10conftool, 06Data-Persistence, 06serviceops, 13Patch-For-Review: Switch conftool to use the version 3 etcd datastore - https://phabricator.wikimedia.org/T350565#11154454 (10PEPE1234.13) 05Open→03Invalid p:05Triage→03Unbreak! a:05Scott_French→03None [14:04:04] 06Traffic, 10conftool, 06Data-Persistence, 06serviceops, 13Patch-For-Review: Switch conftool to use the version 3 etcd datastore - https://phabricator.wikimedia.org/T350565#11154527 (10Aklapper) 05Invalid→03Open p:05Unbreak!→03Triage a:03Scott_French [21:13:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [21:18:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS