[08:32:48] 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10A189605) Can you possibly explain why our ISP's interface connected to our network (using IP 84.19.61.194) can successfully ping 91.198.174.192 and 91.198.174.208 (used earlier in t... [11:03:41] 10Traffic, 10netops, 10SRE: BGP Policy on aggregate routes prevents them being created in some circumstances. - https://phabricator.wikimedia.org/T283163 (10cmooney) Ok configuration has been added to cr1-eqiad, cr2-eqiad and cr2-codfw (routers with transport links to eqord). Looks to have been successful.... [12:25:43] 10Traffic, 10SRE, 10observability: Implement SLI measurement for Varnish Frontend - https://phabricator.wikimedia.org/T284576 (10ema) I wrote a varnishlog consumer to see the actual maximum values for each timestamp we're currently getting: ` #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ VarnishMaxTi... [12:45:26] 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) Thus far we have: 1. Validated your IP range or any subset thereof is not on any ban or block lists. 2. Validated we can route from our front-end load-balancer IPs to your... [13:36:42] 10Traffic, 10netops, 10SRE, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) > INFO:homer.transports.junos:Committing the configuration on cr4-ulsfo.wikimedia.org > INFO:homer:Homer run completed successfully on 2 devices:... [13:43:28] 10Traffic, 10netops, 10SRE, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) I can confirm the 185.71.138.0/24 prefix is now being announced to peers from ulsfo, for example: ` cmooney@cr4-ulsfo> show route advertising-pro... [14:26:32] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in eqiad - https://phabricator.wikimedia.org/T284348 (10ssingh) 05Open→03Resolved a:03ssingh doh1001 and doh1002 have been created; closing this task. Thanks for the help! [14:32:48] 10Traffic, 10netops, 10SRE, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) > INFO:homer.transports.junos:Committing the configuration on cr2-eqiad.wikimedia.org > INFO:homer:Homer run completed successfully on 2 devices:... [14:37:20] 10Traffic, 10netops, 10SRE, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) Additional confirmation, since I am enjoying the reduced latency of the new Toronto -> eqiad route instead of the old Toronto -> codfw :) ` kdig... [14:38:01] 10Traffic, 10netops, 10SRE, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Yep! Seeing very nice latency from NY to wikidough now :) ` root@nyc2:~# mtr -b -w -z -c 5 185.71.138.138 Start: 2021-06-10T16:35:02+0200 HOST:... [15:21:24] 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) [15:25:04] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10Volans) I'm not sure yet how the automation side of things will look like, but there is a good chance that it could use redfish. In... [15:56:25] 10Traffic, 10SRE: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [15:56:33] 10Traffic, 10netops, 10SRE: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) 05Open→03Resolved a:03ssingh Marking this as resolved as we have completed all the intended tasks for now and the routers have been configured. On our (Traffic's... [16:00:00] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) From IRC conversation: We're going to do a 1 off to ease DCops pain of upgrading a large amount of firmwares. Once those 40... [16:56:52] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10odimitrijevic) a:03odimitrijevic [17:07:33] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) a:03jbond @jbond & @MoritzMuehlenhoff: Would it be ok for me to temp push the Dell firmware files to our install server via... [17:20:57] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7149531, @RobH wrote: > @jbond & @MoritzMuehlenhoff: > > Would it be ok for me to temp push the D... [17:53:47] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7149561, @MoritzMuehlenhoff wrote: >>>! In T283771#7149531, @RobH wrote: >> @jbond & @MoritzMuehlenhoff: >> >>... [17:59:04] 10netops, 10DC-Ops, 10SRE, 10SRE-tools: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7149663, @RobH wrote: >> Which size are these files? That's fine, if it's not more than say 5 G, t...