[05:22:13] 06Traffic: Error message says "%error_body_content%" - https://phabricator.wikimedia.org/T371424#10030519 (10Pppery) [05:22:50] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:23:54] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:24:11] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:27:50] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:28:54] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:34:11] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:39:11] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:44:11] RESOLVED: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:33:50] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:codfw:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371434#10030695 (10ayounsi) a:03Papaul [10:02:40] 06Traffic, 06Data-Engineering: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031#10031080 (10fgiunchedi) untagging o11y since benthos is going to be removed from cp hosts [10:03:44] 06Traffic, 06Data-Engineering, 13Patch-For-Review: Install benthos on single esams host to check performances under higher load - https://phabricator.wikimedia.org/T365968#10031082 (10fgiunchedi) untagging o11y since benthos is going to be removed from cp hosts [10:09:50] 06Traffic, 06Data Products, 06Data-Engineering: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10031099 (10fgiunchedi) untagging o11y since this is unrelated [12:40:41] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10031746 (10Fabfur) [13:59:50] 10netops, 06Infrastructure-Foundations, 06SRE: Configure DSCP marking for cloudceph* hosts - https://phabricator.wikimedia.org/T371501 (10cmooney) 03NEW p:05Triage→03Low [15:06:13] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10032281 (10Fabfur) [15:06:59] 06Traffic: Adding IP Addresses to SPF (Dayforce) - https://phabricator.wikimedia.org/T371304#10032283 (10ssingh) @jhathaway: Thoughts on this? We have the DKIM selector `corporate._domainkey` and Dayforce wants to add SPF records for `wikimedia.org` for the IPs above. [15:07:39] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10032300 (10Fabfur) Puppet run on all cp-ulsfo hosts has been completed, HAProxy configuration now doesn't send logs anymore to Benthos.... [15:23:27] 06Traffic: Adding IP Addresses to SPF (Dayforce) - https://phabricator.wikimedia.org/T371304#10032342 (10jhathaway) >>! In T371304#10032282, @ssingh wrote: > @jhathaway: Thoughts on this? We have the DKIM selector `corporate._domainkey` and Dayforce wants to add SPF records for `wikimedia.org` for the IPs above.... [16:11:20] 06Traffic: Error message says "%error_body_content%" - https://phabricator.wikimedia.org/T371424#10032644 (10ssingh) [16:22:01] 06Traffic: Adding IP Addresses to SPF (Dayforce) - https://phabricator.wikimedia.org/T371304#10032736 (10ssingh) Should we ask them for that in this round? That's the question I am debating. [16:33:12] 06Traffic: Adding IP Addresses to SPF (Dayforce) - https://phabricator.wikimedia.org/T371304#10032802 (10ssingh) On one hand, we have fr-tech already sending emails from `ip4:74.121.51.111` but that has existed since 2012. The concern here is allowing Dayforce to send emails from @ for `wikimedia.org` and we are... [16:39:48] 06Traffic: Adding IP Addresses to SPF (Dayforce) - https://phabricator.wikimedia.org/T371304#10032817 (10jhathaway) if the request is not urgent, I would love to try using a subdomain, my preference would be to have their name included in the subdomain, so `dayforce.wikimedia.org` rather than `corporate.wikimedi... [18:08:38] FIRING: LVSHighRX: Excessive RX traffic on lvs6001:9100 (enp175s0f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [18:13:38] RESOLVED: LVSHighRX: Excessive RX traffic on lvs6001:9100 (enp175s0f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:48:57] 06Traffic, 10ops-magru: Degraded RAID on cp7015 - https://phabricator.wikimedia.org/T371554#10033696 (10BCornwall) 05Open→03In progress p:05Triage→03High [20:50:27] 06Traffic, 10ops-magru: Degraded RAID on cp7015 - https://phabricator.wikimedia.org/T371554#10033702 (10BCornwall) ` Jul 31 20:19:29 cp7015 kernel: mpt3sas_cm0: log_info(0x31110d00): originator(PL), code(0x11), sub_code(0x0d00) Jul 31 20:19:33 cp7015 kernel: mpt3sas_cm0: log_info(0x31110d00): originator(PL), c... [21:08:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp7015:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [21:09:13] ^oops, will downtime