[04:03:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 80.74% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[04:08:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.74% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[08:28:32] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455 (10SLyngshede-WMF) 03NEW
[08:28:41] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11245018 (10SLyngshede-WMF) p:05Triage→03Medium
[10:06:56] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.65% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[10:11:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 84.65% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[13:04:56] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.69% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[13:09:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 84.23% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[14:37:36] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11246363 (10cmooney) p:05Triage→03Low
[14:42:29] <wikibugs>	 10Mail, 06Infrastructure-Foundations: Investigate options for outbound email redundancy for mediawiki on kubernetes - https://phabricator.wikimedia.org/T370006#11246380 (10jhathaway) p:05High→03Medium
[14:56:41] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11246477 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test1005.wikimedia.org with OS trixie
[15:15:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 80.51% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[15:20:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 83.69% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[15:46:50] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11246700 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test1005.wikimedia.org with OS trixie completed: - idp-test1005 (**PASS**)   -...
[17:15:04] <jinxer-wm>	 FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[18:15:04] <jinxer-wm>	 RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[22:02:56] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[22:08:01] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[23:24:26] * swfrench-wmf appears wearing oncall hat
[23:24:37] <swfrench-wmf>	 hello I/F friends - it looks like we've had CertAlmostExpired alerts firing for `lsw1-[ef][567]-eqiad.mgmt.eqiad` for the last couple of days. is that expected?