[06:17:33] 10Packaging, 06Infrastructure-Foundations: Package ipxe-qemu - https://phabricator.wikimedia.org/T369136#11026610 (10ayounsi) 05Open→03Resolved a:03ayounsi It's now packaged for trixie (1.21.1+git20250501.dad20602) and sid (1.21.1+git20250602.5b3ebf8b) [09:46:45] 10netbox, 06Infrastructure-Foundations: Improve Netbox "locations" use - https://phabricator.wikimedia.org/T333948#11027075 (10ayounsi) We agreed on using the `pod` term instead of `row`, so for example it would be `codfw-pod-ab` We sync up the `locations` from Netbox to Puppet with the `sre.puppet.sync-netbo... [10:21:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [10:24:04] 10netops, 06Infrastructure-Foundations, 06SRE: Inaccurate stats reported by cr2-codfw - https://phabricator.wikimedia.org/T400205#11027210 (10cmooney) Ok so I drained cr2-codfw of traffic and tried issuing the commands. Commands as supplied by Juniper aren't 100% correct either which is reassuring when medd... [10:26:39] 07Puppet, 10Beta-Cluster-Infrastructure, 06SRE: Puppet configures kernel.core_pattern |/usr/lib/systemd/systemd-coredump, but systemd-coredump is not installed - https://phabricator.wikimedia.org/T400247 (10Lucas_Werkmeister_WMDE) 03NEW [11:41:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [12:12:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Link errors: ssw1-d1-codfw <-> ssw1-f1-codfw - https://phabricator.wikimedia.org/T400253 (10cmooney) 03NEW p:05Triage→03Medium [13:35:50] 10netops, 06Infrastructure-Foundations, 06SRE: Homer: PyEz "ignore_warnings" does not work for port-block speed change warning - https://phabricator.wikimedia.org/T400261 (10cmooney) 03NEW p:05Triage→03Medium [13:50:55] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07Python3-Porting: Puppet tox: properly lint both Py2 and Py3 files - https://phabricator.wikimedia.org/T184435#11027873 (10jijiki) Hey folks, I ran into this issue myself, having CI failing my patches over and over again. ` py2-pep8: skipped because could... [18:06:25] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:11:25] RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:43:48] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Homer: PyEz "ignore_warnings" does not work for port-block speed change warning - https://phabricator.wikimedia.org/T400261#11029477 (10cmooney) [21:00:55] FIRING: MaxConntrack: Max conntrack at 85.7% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [21:05:55] RESOLVED: MaxConntrack: Max conntrack at 85.87% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [21:53:26] 10Mail, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#11029549 (10nisrael) Great thank you Jesse! Just want to confirm, am I safe toinstruct our rep at DMarcian to restart our free trial? [21:55:04] 10Mail, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Infrastructure Security, and 2 others: Re-opening our DMarcian Trial - https://phabricator.wikimedia.org/T394788#11029551 (10jhathaway) >>! In T394788#11029549, @nisrael wrote: > Great thank you Jesse! Just want to confirm, am I safe toinstruct our r... [22:14:05] 10Mail, 06Infrastructure-Foundations, 06SRE, 10SRE-Access-Requests: Access Request to DMarcDigests - https://phabricator.wikimedia.org/T399976#11029580 (10jhathaway) @nisrael I sent you an invite, let me know if you can get in. [23:54:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:59:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed