[06:14:57] 10netops, 10SRE: Lumen link between cr2-eqiad and cr2-esams down - https://phabricator.wikimedia.org/T279820 (10ayounsi) For the record: Ticket opened at **2021-04-10 11:29:36 GMT** > **2021-04-10 12:50:02 GMT **- Hello from Lumen, > We are observing a network Local Fault from our Subsea portion. We will send... [07:38:09] 10Traffic, 10SRE, 10User-notice: Rate limit requests in violation of User-Agent policy more aggressively - https://phabricator.wikimedia.org/T224891 (10Legoktm) >>! In T224891#6983370, @ayounsi wrote: > Even with the current rate limiting, some crawling are regularly causing issues, wasting precious SRE time... [08:35:26] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10ayounsi) PCC result: https://puppet-compiler.wmflabs.org/compiler1003/28963/ [09:23:18] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10ema) >>! In T265864#6993281, @ayounsi wrote: > Now to figure out the actual service impact and if it's safe to merge. > To re-iterate, 185.15.56.0... [09:35:52] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10Majavah) >>! In T265864#6993418, @ema wrote: > On cache hosts, the result of that change would be removing the IP from **wikimedia_nets** and **wi... [09:53:40] 10Traffic, 10SRE, 10Patch-For-Review: cache_upload cache policy + large_objects_cutoff concerns - https://phabricator.wikimedia.org/T275809 (10ema) cp5001 has been running with the exp policy for 5 days now: compared to other upload nodes in eqsin, its [[https://grafana.wikimedia.org/d/A__2L7eWz/cache-hosts-... [10:04:57] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10ayounsi) They're currently coming from the 172.16.0.0/12 space but T209011 is going to change it so they will come from the 185.15.56.0/24 space.... [10:58:37] 10Traffic, 10SRE: Network unreachable after network-online.target is brought up - https://phabricator.wikimedia.org/T237243 (10ayounsi) [13:01:55] 10Traffic, 10Performance-Team, 10SRE, 10serviceops: Decide on details of progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10fgiunchedi) p:05Triage→03Medium [13:03:10] 10Traffic, 10netops, 10SRE: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10fgiunchedi) p:05Triage→03Medium [13:04:09] 10Traffic, 10Analytics, 10SRE: Add Traffic's notion of "from public cloud" to Analytics webrequest data - https://phabricator.wikimedia.org/T279380 (10fgiunchedi) p:05Triage→03Medium [16:54:33] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10Dzahn) Speaking for phab1001: I am not aware of anything that needs to connect from cloud to phab in production but adding @20after4 as well Spe... [16:56:40] 10netops: BGP: prioritize directly connected peers - https://phabricator.wikimedia.org/T280054 (10ayounsi) p:05Triage→03Low [16:59:03] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10mmodell) @dzahn: would this block toolhub tools from connecting to phab? Things like #stashbot and #wikibugs come to mind. [17:37:21] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10Dzahn) @mmodell Yes, given that `stashbot.toolforge.org has address 185.15.56.11` it seem it would break it. [17:55:01] 10netops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Remove 185.15.56.0/24 from network::external - https://phabricator.wikimedia.org/T265864 (10Legoktm) >>! In T265864#6995415, @mmodell wrote: > @dzahn: would this block toolhub tools from connecting to phab? Things like #stashbot and #wi... [18:49:01] 10Traffic, 10Performance-Team, 10SRE, 10serviceops: Decide on details of progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10Krinkle) a:03Krinkle