[07:46:22] 07Puppet, 10MobileFrontend (Tracking): Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425#11223314 (10Krinkle) [07:46:57] 07Puppet, 10MobileFrontend (Tracking): Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425#11223316 (10Krinkle) [08:20:30] 10SRE-tools, 06Infrastructure-Foundations: CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706#11223453 (10elukey) https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1192056 [08:36:16] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops, 10Spicerack: Spicerack's `Discovery.resolve_with_client_ip` should set a timeout on `udp_with_fallback` - https://phabricator.wikimedia.org/T405397#11223585 (10elukey) Spicerack 11.9.0 deployed on all cumin nodes :) [08:44:53] 10SRE-tools, 06Infrastructure-Foundations: CI error on operations/cookbooks - https://phabricator.wikimedia.org/T405706#11223632 (10ABran-WMF) 05Open→03Resolved a:03elukey [[ https://integration.wikimedia.org/ci/job/tox/7677/console | CI went through ]], thanks for the fix! [12:19:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:24:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:52:04] 10CFSSL-PKI, 06Infrastructure-Foundations, 10Cloud-VPS (Debian Bullseye Deprecation): Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11224973 (10elukey) p:05Triage→03Medium [14:52:11] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: DeprecationWarning: datetime.datetime.utcnow() is deprecated - https://phabricator.wikimedia.org/T401581#11224976 (10elukey) p:05Triage→03Low [14:52:24] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: DeprecationWarning: datetime.datetime.utcnow() is deprecated - https://phabricator.wikimedia.org/T401581#11224978 (10elukey) [15:17:02] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops, 10Spicerack: Spicerack's `Discovery.resolve_with_client_ip` should set a timeout on `udp_with_fallback` - https://phabricator.wikimedia.org/T405397#11225075 (10Scott_French) 05Open→03Resolved a:03Scott_French Amazing - thank you very much,... [15:39:56] FIRING: MaxConntrack: Max conntrack at 82.26% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [15:44:55] RESOLVED: MaxConntrack: Max conntrack at 81.13% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [15:52:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11225279 (10Papaul) [16:07:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11225393 (10Papaul) I had a meeting today with @Jgreen about the new switch configuration. what we will be doing is to move the... [18:46:49] 10CAS-SSO, 06cloud-services-team, 10Striker, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#11226188 (10Arendpieter) @SLyngshede-WMF, do you know how to create OIDC_KEY and OIDC_SECRET in idp.wikimedia.org in order to test this solution: https://gerrit... [19:00:01] 10Mail, 06Infrastructure-Foundations, 06SRE Observability, 07Epic: Parse DMARC reports and create a dashboard from data - https://phabricator.wikimedia.org/T404888#11226276 (10andrea.denisse) Hi folks, LGTM, I'd just suggest defining a lifecycle for how long do we plan on storing the reports, that we stor... [20:41:04] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11226861 (10Papaul) 05Open→03Resolved We decided that the next time we have a n open window again for this @cmooney will himself drive the test. For this test, the m... [20:43:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11226867 (10Papaul) [20:44:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11226869 (10Papaul) p:05Triage→03Medium [21:16:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11226958 (10Papaul) [21:31:28] FIRING: NodeTextfileStale: Stale textfile for config-master1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:35:28] FIRING: [2x] NodeTextfileStale: Stale textfile for puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:36:28] FIRING: [2x] NodeTextfileStale: Stale textfile for config-master1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:40:28] FIRING: [4x] NodeTextfileStale: Stale textfile for puppetmaster1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:45:28] FIRING: [7x] NodeTextfileStale: Stale textfile for puppetmaster1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:50:28] FIRING: [8x] NodeTextfileStale: Stale textfile for puppetmaster1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:55:28] FIRING: [9x] NodeTextfileStale: Stale textfile for puppetmaster1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [22:00:28] FIRING: [10x] NodeTextfileStale: Stale textfile for puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale