[04:06:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp5023:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5023 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:11:00] FIRING: [9x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:16:00] FIRING: [10x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:21:00] FIRING: [13x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:26:00] FIRING: [11x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:31:00] FIRING: [14x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:36:00] FIRING: [15x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:41:00] FIRING: [13x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:46:00] RESOLVED: [13x] PurgedHighEventLag: High event process lag with purged on cp5018:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [13:15:32] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11327066 (10Papaul) @Dwisehaupt yes Wednesday 11/5 is ok with me. Let us do 10:00am CT. Thank you. [13:34:12] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558#11327142 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=aacadee6-1bf1-45b7-bbed-963884cb38ed) set by cmooney@cumin1003 for 5 d... [15:28:08] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11327839 (10Jhancock.wm) 05In progress→03Resolved a:03Jhancock.wm [16:00:14] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11328122 (10JVanderhoop-WMF) [16:00:49] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11328131 (10Dwisehaupt) Cool. Let's do it then. [16:32:43] FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [16:37:43] FIRING: [25x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [16:47:43] RESOLVED: [25x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [18:57:50] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857 (10BCornwall) 03NEW [18:59:02] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11329256 (10BCornwall) [19:01:05] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11329269 (10BCornwall) [19:01:22] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11329272 (10BCornwall) p:05Triage→03Low [19:02:02] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11329275 (10BCornwall) [20:45:46] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11329606 (10Dzahn) config example kindly provided by Chris Danis: {P84490} [22:41:50] 06Traffic: ncmonitor: Migrate from deprecated API to new API - https://phabricator.wikimedia.org/T408857#11329958 (10BCornwall)