[00:58:44] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10201389 (10Dzahn) @nisrael I think at this point we really need to see one of the mails that arrived at lisa@wikimedia.org to say anything more. Do you... [09:15:25] FIRING: SystemdUnitFailed: prometheus-puppet-agent-stats.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:45:25] RESOLVED: SystemdUnitFailed: prometheus-puppet-agent-stats.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:52:30] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462 (10aborrero) 03NEW [11:04:05] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202145 (10cmooney) This patch covers the delegation for the openstack-managed ranges, I think it's correct? https://g... [11:14:24] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202157 (10aborrero) [11:16:06] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202161 (10cmooney) To be more clear, you need to make sure these two zones are working on the openstack authdns: ` 0.0.0.0.0.... [11:54:48] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202237 (10aborrero) before and after merging the tofu-infra patch above: `lang=shell-session arturo@nostromo:~ $ dig SO... [12:00:14] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202238 (10aborrero) 05Open→03In progress p:05Triage→03Medium [13:09:20] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10202360 (10MBywater-WMF) lisa@wikimedia.org is an alias of lgruwell@wikimedia.org [13:50:42] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10202535 (10Papaul) [14:20:13] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10202600 (10nisrael) Ok good to know. I will set up a call with Lisa next Monday and we can get an example of one of the mails. She has access to that inbox. [14:43:43] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10202667 (10MBywater-WMF) Thanks, @nisrael! [22:16:54] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10204075 (10Papaul)