[06:52:51] 06Traffic, 06MediaWiki-Platform-Team (Radar): [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11235772 (10Krinkle) [07:35:01] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11235854 (10Gehel) A few notes: * `wdqs-scholarly.discovery.wmnet` and `wdqs-main.discovery.wmnet` are the 2 WDQS public endpoints. Both of those have a 95% u... [07:38:14] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11235857 (10Gehel) I'm not entirely sure which alerts paged during the last WDQS outage. I think it was about the number of servers depooled from LVS and not f... [08:28:18] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#11235985 (10cmooney) [11:38:29] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11236548 (10cmooney) >>! In T404959#11229706, @VRiley-WMF wrote: > Hey @cmooney is there a good time to schedual this move? Hey @VRi... [13:20:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:21:08] yeah that is expected [13:25:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:28:05] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11236873 (10ssingh) @BCornwall from Traffic will be working on this, thanks! [13:30:11] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11236890 (10ssingh) >>! In T406141#11235857, @Gehel wrote: > I'm not entirely sure which alerts paged during the last WDQS outage. I think it was about the num... [13:31:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:36:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:41:50] FIRING: [3x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:46:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:51:50] FIRING: [3x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:54:35] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11236988 (10cmooney) >>! In T404959#11236872, @ssingh wrote: > @BCornwall from Traffic will be working on this, thanks! Thanks @ssing... [13:56:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-c state mismatch on dns1006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:02:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:03:20] sigh [14:03:53] so we really don't depool the NTP service given it's a quick restart and most (all?) clients should be in sync anyway and then there is sufficient redundancy [14:04:06] but it seems like since this check is too picky, we will probably need to add that [14:04:57] silenced [14:13:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11237050 (10Jclark-ctr) a:03Jclark-ctr [14:36:56] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: row C/D switch refresh configuration task - https://phabricator.wikimedia.org/T402588#11237167 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=626cec35-f6f7-443b-90fb-3024162d9dc9) set by cmooney@cumin1003 for 0:10:00 on 3 host(s) and... [18:52:19] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11238347 (10BCornwall) @cmooney Sounds good! Should I be scheduling with you or @VRiley-WMF? [19:18:46] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11238425 (10cmooney) >>! In T404959#11238347, @BCornwall wrote: > @cmooney Sounds good! Should I be scheduling with you or @VRiley-WMF... [20:48:15] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11238731 (10BCornwall) Sure, no problem. Have at it. LMK if you need any help. [21:44:37] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406166#11238814 (10MoritzMuehlenhoff) Looks good, please use any of row/group B, C or D. [21:45:07] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406167#11238819 (10MoritzMuehlenhoff) Looks good, please use any of row/group B, C or D.