[00:33:56] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [02:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:33:56] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:16:06] Netbox 4.4.0 released [08:08:10] 10netops, 10Ganeti, 06Infrastructure-Foundations: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11142484 (10ayounsi) 05Open→03Resolved [08:08:21] 10netops, 10Ganeti, 06Infrastructure-Foundations: magru: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T402372#11142489 (10ayounsi) a:03ayounsi [08:11:01] 10netops, 10Ganeti, 06Infrastructure-Foundations: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580 (10ayounsi) 03NEW p:05Triage→03Low [08:25:53] easy +1 for anyone who feels like it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1184471 [08:32:40] 10netops, 10Ganeti, 06Infrastructure-Foundations, 13Patch-For-Review: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580#11142603 (10ayounsi) [08:33:56] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:05:13] fixed --^, same problem that 1002 had yesterday [09:05:20] git diff for homer public shoed [09:05:22] *showed [09:05:30] modified: tests/generate_schema_docs.py [09:05:30] modified: utils/check-style.sh [09:05:30] modified: utils/format-code.sh [09:05:52] all permission-bits related. I did a git restore for all files, puppet doesn't try to fix them anymore [09:06:36] I wonder if at the next git fetch/pull/push it will re-happen [09:23:56] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:26:54] elukey: ^^^ ;) [09:32:54] it is strange, I don't see it in the puppet logs [09:36:20] the alert shows only cumin2002 (just fixed it) [09:41:41] 10netops, 10Ganeti, 06Infrastructure-Foundations: esams: move sandbox vlan to routed Ganeti - https://phabricator.wikimedia.org/T403580#11142999 (10ayounsi) @MoritzMuehlenhoff I tried to create the VM using `sudo cookbook sre.ganeti.makevm --vcpus 2 --memory 2 --disk 50 --network sandbox --os none --cluster... [09:58:56] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cumin2002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:50:31] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11144499 (10Papaul) manually disable OSPF (using commit confirmed) make the mgmt goes down when done on mr1 or cr3/cr4 . But mr1-ulsfo.oob.wikimedia.org and mr1 loopback stil... [16:18:55] FIRING: MaxConntrack: Max conntrack at 82.73% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [16:38:55] RESOLVED: MaxConntrack: Max conntrack at 83.93% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:11:56] FIRING: MaxConntrack: Max conntrack at 80.59% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [21:16:55] RESOLVED: MaxConntrack: Max conntrack at 80.59% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [21:24:56] FIRING: MaxConntrack: Max conntrack at 80.7% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [21:29:56] RESOLVED: MaxConntrack: Max conntrack at 80.7% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [22:48:56] FIRING: [4x] SystemdUnitFailed: squid-logrotate.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed