[05:33:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11979903 (10ayounsi) [05:34:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11979906 (10ayounsi) [05:34:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11979908 (10ayounsi) [06:32:54] 07Puppet, 06collaboration-services, 10Gerrit, 06Infrastructure-Foundations, 13Patch-For-Review: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#11979958 (10ABran-WMF) good idea @Dzahn I pinged @MoritzMuehlenhoff on... [07:38:27] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A5 maintenance - https://phabricator.wikimedia.org/T428020#11980094 (10MLechvien-WMF) [08:11:05] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980301 (10ayounsi) [08:15:50] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980302 (10ayounsi) [08:19:50] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980303 (10ayounsi) @cmooney see task description, let me know what you think of: * eqiad A/B : 208.80.155.32/28 - 2620:0:861:6::/64 * eqiad C/D : 208.80.155.48/28 - 2620:0:861:7::/64 *... [08:22:29] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980305 (10cmooney) >>! In T422043#11980303, @ayounsi wrote: > @cmooney see task description, let me know what you think of: > * eqiad A/B : 208.80.155.32/28 - 2620:0:861:6::/64 > * eqiad... [09:19:28] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11980483 (10elukey) I think we could study something that logs to a task if needed, as optional feature to toggle via parameter. We could d... [09:41:27] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11980580 (10Volans) Would adding the duration to the existing log messages in [1], [2] and [3] be enough? [1] https://gerrit.wikimedia.org... [09:51:24] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11980630 (10CWilliams-WMF) @elukey yes, I did have an idea... but @Volans suggesting that making it part of the log messages from the calls... [09:56:21] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11980650 (10Marostegui) I think that'd be good for me yeah! Thank you all [10:07:59] o/ just noticed that https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/network/data/data.yaml#53 uses a /128 and that feels wrong [10:09:20] it probably should be this /64 https://netbox.wikimedia.org/ipam/prefixes/529/ip-addresses/ (cc topranks) [10:10:29] taavi, XioNoX: indeed correct that's my mistake, I'll file a task thanks <3 [10:26:16] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE, 13Patch-For-Review: Improve port-utilisation alerting to take QoS into account - https://phabricator.wikimedia.org/T384052#11980758 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this one. Alert is in place and fir... [10:30:45] 10netops, 06Infrastructure-Foundations, 06SRE: Change codfw dns hosts BGP peering to top-of-rack switch - https://phabricator.wikimedia.org/T376894#11980777 (10cmooney) 05Open→03Declined I'm going to close this one for now. Given we are moving the dns hosts to new vlans under T422043, during which t... [10:31:06] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980783 (10cmooney) [10:31:08] 10netops, 06Infrastructure-Foundations, 06SRE: Change codfw dns hosts BGP peering to top-of-rack switch - https://phabricator.wikimedia.org/T376894#11980784 (10cmooney) [10:35:01] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11980807 (10ayounsi) [13:59:27] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11981430 (10ayounsi) [13:59:56] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11981442 (10ayounsi) [14:00:50] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11981458 (10ayounsi) [19:20:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11982896 (10BCornwall) When pooling cp5032 I noticed that connection to kafka-jumbo1016.eqiad.wmnet:9093 (`10.64.154.15 via 10.132.1.1 de... [19:37:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11982956 (10BCornwall) @ayounsi helpfully pointed out that I needed to update hieradata/common.yaml with the new IP addresses. Thanks! [21:26:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11983173 (10BCornwall) [21:26:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11983180 (10BCornwall) I was advised by @taavi to also update mediawiki-config's `wmf-config/reverse-proxy.php` ranges. I've updated... [21:27:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11983181 (10BCornwall)