[01:08:55] 10netops, 10Operations, 10fundraising-tech-ops: possible routing issue between eqiad and Maxmind network - https://phabricator.wikimedia.org/T233672 (10Jgreen) [01:12:04] 10netops, 10Operations, 10fundraising-tech-ops: possible routing issue between eqiad and Maxmind network - https://phabricator.wikimedia.org/T233672 (10Jgreen) p:05Triage→03Unbreak! Flipping this to "Unbreak Now!" since it's a timely issue, and service outage interfering with the donation pipeline. We do... [02:03:11] 10netops, 10Operations, 10fundraising-tech-ops: possible routing issue between eqiad and Maxmind network - https://phabricator.wikimedia.org/T233672 (10ayounsi) [02:03:46] 10netops, 10Operations, 10fundraising-tech-ops: possible routing issue between eqiad and Maxmind network - https://phabricator.wikimedia.org/T233672 (10ayounsi) a:03ayounsi All those IPs are behind Cloudflare. Opened a ticket with them. [02:47:57] 10netops, 10Operations, 10fundraising-tech-ops: possible routing issue between eqiad and Maxmind network - https://phabricator.wikimedia.org/T233672 (10ayounsi) 05Open→03Resolved Resolved by Cloudflare. [06:41:32] 10Traffic, 10Core Platform Team, 10Operations, 10Performance-Team, and 6 others: RFC: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Nikerabbit) [16:57:02] https://blog.apnic.net/2019/09/23/dns-resolver-centrality/ [17:44:49] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Free up 185.15.59.0/24 - https://phabricator.wikimedia.org/T211254 (10ayounsi) 05Open→03Resolved https://gerrit.wikimedia.org/r/c/operations/puppet/+/509140 was the last thing to do and it has been merged some time ago. [17:44:51] 10netops, 10Operations: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - https://phabricator.wikimedia.org/T207753 (10ayounsi) [17:54:48] 10netops, 10Operations: BGP session down for AS4739 on cr4-ulsfo - https://phabricator.wikimedia.org/T230005 (10ayounsi) 05Open→03Resolved Sessions are now established. Thanks! [18:15:02] 10netops, 10Operations: Connection timeout from 195.77.175.64/29 to text-lb.esams.wikimedia.org - https://phabricator.wikimedia.org/T189689 (10ayounsi) 05Open→03Resolved No new updates since March 2018, feel free to reopen if the issue is still there. [18:45:54] 10netops, 10Cloud-Services, 10Operations: Intermittent bandwidth issue to labs proxy (eqiad) from Comcast in Portland OR - https://phabricator.wikimedia.org/T136671 (10brion) 05Open→03Resolved Haven't encountered this in a while; Comcast etc may have improved the intermediate routes. Closing out as resol... [19:29:52] 10netops, 10Operations: configure BGP route damping on IX sessions - https://phabricator.wikimedia.org/T222424 (10ayounsi) [19:37:05] 10netops, 10Operations: configure BGP route damping on IX sessions - https://phabricator.wikimedia.org/T222424 (10ayounsi) a:03ayounsi Keeping the default damping settings (per the doc) here is what I think we should push to our routers: `lang=diff [edit protocols bgp group IX4] + damping; [edit protocols... [21:11:07] 10Traffic, 10Operations: rack/setup/install ganeti400[123] - https://phabricator.wikimedia.org/T226444 (10Dzahn) These hosts showed up as special cases as part of T147074 because commands over remote IPMI did not work. ssh login works. Feels like IPMI over LAN might be disabled in BIOS. [21:40:24] 10netops, 10Operations: Outbound BGP graceful shutdown - https://phabricator.wikimedia.org/T211728 (10ayounsi) FYI I tested the policy from the description successfully during T226422 and T226424. [21:54:07] 10netops, 10Operations, 10Wikimedia-Incident: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645 (10ayounsi) The logs rolled over the weekend... And neither the ones shipped to central logging nor the RSI had any useful information according to JTAC. From there we can: 1/ replace the switch wi... [22:12:28] 10netops, 10Operations, 10observability: Netflow Collector Project - https://phabricator.wikimedia.org/T83119 (10ayounsi) 05Open→03Resolved a:03ayounsi https://wikitech.wikimedia.org/wiki/Netflow