[03:14:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [07:14:41] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [09:15:36] Hello, if no objection I'm going to test the --move-vlan reimage cookbook on sretest1002, 1005 and 1006 (all in eqiad rows C or D) [09:57:34] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562 (10ayounsi) 03NEW [09:57:57] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11586452 (10ayounsi) [09:57:58] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11586451 (10ayounsi) [10:01:22] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563 (10ayounsi) 03NEW [10:01:36] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#11586477 (10ayounsi) [10:01:38] 10netops, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11586478 (10ayounsi) [11:13:51] feature request for os-reports.w.o, if the list under "The following hosts are missing in the migration data" could either be sorted or using nodeset notation :-P [11:14:41] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [11:57:38] 10netops, 06Infrastructure-Foundations, 06Traffic: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11587007 (10Vgutierrez) [13:29:37] moritzm: just fyi I pushed some homer changes to our mgmt routers relating to bast1004 IPs being added to ACLs [13:29:43] so that is already done if it was on your list [13:39:30] many thanks [15:14:41] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [15:57:43] is https://librenms.wikimedia.org/device/359/overview missing a 'ports' tab that's there for Juniper switches? is that same info now available elsewhere? [16:01:24] taavi: interesting, they show up on some devices, not sure what's the difference [16:01:25] https://librenms.wikimedia.org/device/369 [16:01:35] but also it's better to use Grafana now [16:02:54] taavi: for example here https://grafana-rw.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?orgId=1&from=now-3h&to=now&timezone=utc&var-site=000000006&var-device=lsw1-c4-eqiad:9804&var-interface=$__all&refresh=30s [16:03:18] but there is a bit of harmonization to do with all the dashboards [16:08:16] moritzm: is there a way to get a list of hosts that are up but in `insetup` mode in eqiad ? [16:10:22] XioNoX: cumin "A:insetup and A:eqiad" [16:10:29] follow-up question: how sampled is netflow? [16:11:08] taavi: 1/1000 but nfacct extrapolates the bits/packets, so it depends on where you look :) [16:11:58] so the 'bytes' count in Turnilo is roughly accurate? [16:12:40] taavi: yep [16:13:00] thanks! [16:50:43] 10netops, 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, and 3 others: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11588444 (10taavi) This is still primarily blocked on {T415193} (for DE). In the meantime, some numbers/data: * [... [17:41:24] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Cookbook sre.hosts.reimage: DHCP snippet created with old IP when --move-vlan is used - https://phabricator.wikimedia.org/T416401#11588732 (10ayounsi) 05Open→03Resolved a:03ayounsi fixed. [19:14:41] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [23:14:41] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag