[01:30:10] FIRING: SystemdUnitFailed: wmf_auto_restart_atftpd.service on install7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:38:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [01:53:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [03:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [03:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.3%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [05:30:10] FIRING: SystemdUnitFailed: wmf_auto_restart_atftpd.service on install7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:27:19] ^ made https://gerrit.wikimedia.org/r/1156648 for this [07:14:34] 10netops, 06Infrastructure-Foundations: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#10911942 (10ayounsi) From JTAC: > Hello Team, > > Engineering found out the issue and I am awaiting the details of the RC from them and will share it here as soon as they have finalized the... [07:25:41] 10netbox, 06Infrastructure-Foundations, 06SRE: Traceback in sre.dns.netbox - https://phabricator.wikimedia.org/T396834 (10MoritzMuehlenhoff) 03NEW [07:26:17] 10netbox, 06Infrastructure-Foundations, 06SRE: Traceback in sre.dns.netbox accessing a virtual interface - https://phabricator.wikimedia.org/T396834#10911964 (10MoritzMuehlenhoff) p:05Triage→03Medium [07:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:33:00] 10netbox, 06Infrastructure-Foundations, 06SRE: Traceback in sre.dns.netbox accessing a virtual interface - https://phabricator.wikimedia.org/T396834#10911971 (10ayounsi) This match this old IP deletion change: https://netbox.wikimedia.org/extras/changelog/228764/ `assigned_object_id: 3934` My guess is that... [07:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.36%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [07:47:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [07:57:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [09:05:10] FIRING: [5x] SystemdUnitFailed: squid-logrotate.service on install7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:52:25] FIRING: [2x] SystemdUnitFailed: dhcp-helper.service on ganeti7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:05:21] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: [infra] Reports of slow connectivity from APAC - https://phabricator.wikimedia.org/T395135#10912818 (10Nokib_Sarkar) ` $ curl https://upload.wikimedia.org/wikipedia/commons/e/eb/SMS_Arcona_NH_65764_-_Restoration.jpg --output test.... [11:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [11:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.61%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [13:50:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [13:55:10] FIRING: [2x] SystemdUnitFailed: dhcp-helper.service on ganeti7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:55:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [13:55:55] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [14:10:10] FIRING: [3x] SystemdUnitFailed: dhcp-helper.service on ganeti7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.6%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [15:46:51] 10Mail, 06Infrastructure-Foundations: Prevent OoO auto-responses for external messages to noc@ and other similar aliases - https://phabricator.wikimedia.org/T396884 (10taavi) 03NEW [17:55:55] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [18:10:10] FIRING: [3x] SystemdUnitFailed: dhcp-helper.service on ganeti7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:17:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895 (10RobH) 03NEW [19:19:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914404 (10RobH) [19:27:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914443 (10RobH) Support Email Draft: Support, When the power maintainance took place via CHG0247347, we lost power to the secondary feeds in our rack B... [19:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [19:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.59%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [19:38:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914499 (10RobH) CS1117758 filed [19:50:55] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [20:05:10] FIRING: [3x] SystemdUnitFailed: dhcp-helper.service on ganeti7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:05:55] RESOLVED: [2x] MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [20:10:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914579 (10RobH) Ticket accepted, changed from open to in progress. No further updates at this time. [20:11:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914580 (10RobH) [20:11:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914581 (10RobH) [20:58:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru loss of power redundancy - https://phabricator.wikimedia.org/T396895#10914653 (10RobH) 05Open→03Resolved a:03RobH Validated, the reported equipment is installed in U44, and the PSU on the equipment was Down. Reconn... [23:29:50] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:34:02] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (94.58%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure