[02:52:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11217960 (10Papaul) [02:53:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11217962 (10Papaul) [02:59:25] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (98.52%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [03:07:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [03:11:04] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [03:11:05] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [04:11:04] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:05] RESOLVED: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [04:12:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [06:07:55] FIRING: MaxConntrack: Max conntrack at 85.07% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:12:55] RESOLVED: MaxConntrack: Max conntrack at 82.54% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:59:25] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (98.47%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [08:19:57] 10SRE-tools, 06Infrastructure-Foundations, 10GitLab (Infrastructure): CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706 (10ABran-WMF) 03NEW [08:21:36] 10SRE-tools, 06Infrastructure-Foundations: CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706#11218316 (10taavi) [08:24:30] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: DeprecationWarning: datetime.datetime.utcnow() is deprecated - https://phabricator.wikimedia.org/T401581#11218318 (10A_smart_kitten) [10:59:25] FIRING: GanetiMemoryPressure: Ganeti: High memory usage (98.88%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [11:34:47] 10CFSSL-PKI, 06Infrastructure-Foundations, 10Cloud-VPS (Debian Bullseye Deprecation): Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11218940 (10taavi) [11:35:28] 10CFSSL-PKI, 06Infrastructure-Foundations, 10Cloud-VPS (Debian Bullseye Deprecation): Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11218945 (10taavi) Repurposing this task for that, please close or adjust if you want to track that somewhere else! [12:45:14] 10SRE-tools, 06Infrastructure-Foundations: CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706#11219110 (10LSobanski) Adding @ltoscano as this is likely to be related to a Dell firmware change. [13:40:57] the alert for ganeti1036 was caused by the reboots for vmscape, I'm shuffling a big VM around, it should resolve soon [13:44:10] RESOLVED: GanetiMemoryPressure: Ganeti: High memory usage (98.66%) on ganeti1036:9100 - https://wikitech.wikimedia.org/wiki/Ganeti#Memory_pressure - https://grafana.wikimedia.org/d/gd6vep5Iz/ganeti-memory-pressure?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DGanetiMemoryPressure [14:09:57] 10Mail, 06Infrastructure-Foundations, 06Product Safety and Integrity, 06SRE, and 3 others: Forward emails addressed to privacy@wikidata to privacy@wikimedia - https://phabricator.wikimedia.org/T255733#11219899 (10OKryva-WMF) [14:10:05] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 06Product Safety and Integrity, 07Privacy: Provide a sender email address alias for use in Special:Emailuser (aka 2-way email relay) - https://phabricator.wikimedia.org/T150421#11219903 (10OKryva-WMF) [15:07:20] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 06Product Safety and Integrity, and 3 others: Provide a sender email address alias for use in Special:Emailuser (aka 2-way email relay) - https://phabricator.wikimedia.org/T150421#11220226 (10OKryva-WMF) [15:08:46] 10Mail, 06Infrastructure-Foundations, 06SRE, 06Trust-and-Safety, and 3 others: Forward emails addressed to privacy@wikidata to privacy@wikimedia - https://phabricator.wikimedia.org/T255733#11220234 (10OKryva-WMF) [15:56:38] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops, 10Spicerack, 13Patch-For-Review: Spicerack's `Discovery.resolve_with_client_ip` should set a timeout on `udp_with_fallback` - https://phabricator.wikimedia.org/T405397#11220443 (10Scott_French) I've gone ahead and merged https://gerrit.wikimedia.or... [17:15:04] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [18:15:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [20:29:55] FIRING: MaxConntrack: Max conntrack at 80.61% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:34:55] RESOLVED: MaxConntrack: Max conntrack at 80.93% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [22:53:55] FIRING: MaxConntrack: Max conntrack at 82.15% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [22:58:55] RESOLVED: MaxConntrack: Max conntrack at 82.15% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack