[09:55:39] 10netops, 10Operations: Configure conditional advertising in eqdfw and knams - https://phabricator.wikimedia.org/T236785 (10ayounsi) `lang=diff,name=cr2-eqdfw [edit routing-options rib inet6.0 aggregate route 2620:0:860::/46] - policy BGP_aggregate_contributors_eqiad; + policy BGP_from_LVS; [edit routi... [10:15:31] 10Traffic, 10netops, 10Operations: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) digging more into it, codfw already advertise its own /23 (and v6 /48). [10:36:10] 10netops, 10Operations: Configure conditional advertising in eqdfw and knams - https://phabricator.wikimedia.org/T236785 (10ayounsi) `lang=diff,name=cr3-knams [edit routing-options rib inet6.0 aggregate route 2620:0:862::/48] - policy BGP_aggregate_contributors; + policy BGP_from_LVS; [edit routing-opt... [11:18:45] <_joe_> ema, vgutierrez today I will convert the mwdebug* servers to use envoy [11:21:53] nice [11:25:51] <_joe_> we might start thinking about sending to envoy some of its control-flow headers from ats-be [12:05:23] I see some puppet failures on cp3* hosts, you may want to give a look [12:17:34] 10netops, 10Operations: Configure conditional advertising in eqdfw and knams - https://phabricator.wikimedia.org/T236785 (10ayounsi) Previous change rolled back as: `* 2620:0:862::/48 Self I` was not being advertised to the world anymore While there are contributing... [13:32:50] jynus: puppet is broken fleet wide :/ [13:34:16] see -ops [13:34:17] fixed now [13:34:38] cool [14:06:48] sorry, I only see it on cp3* hosts first on icinga [14:19:33] hmm [14:20:09] it looks like your icinga it's deprecated [14:20:10] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations, 10Developer Productivity: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10Krinkle) [14:20:37] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations, 10Developer Productivity: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10Krinkle) [14:20:49] 10Traffic, 10MediaWiki-Debug-Logger, 10Operations, 10Developer Productivity: noc.wikimedia.org doesn't route to the docroot when WikimediaDebug browser extension is live - https://phabricator.wikimedia.org/T245552 (10Krinkle) [15:31:45] 10netops, 10Operations: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) [15:38:22] 10netops, 10Operations: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) Deployed to codfw but caused an outage. Incident report in progress [15:51:55] 10Traffic, 10Operations, 10Product-Infrastructure-Team-Backlog: Elevated 503 responses between 2020-03-15 and 2020-03-19 - https://phabricator.wikimedia.org/T248132 (10Mholloway) [16:04:20] 10Traffic, 10Analytics, 10Analytics-Kanban, 10EventStreams, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10ema) >>! In T242767#5998492, @gerritbot wrote: > Change 583295 **merged** by Ema: > [operations/puppet@produ... [16:13:50] 10Traffic, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2] - https://phabricator.wikimedia.org/T247340 (10Papaul) [19:48:36] 10netops, 10Operations, 10Wikimedia-Incident: Add linecard diversity to the router-to-router interconnect in codfw - https://phabricator.wikimedia.org/T248506 (10CDanis) [22:03:58] Interesting read: - https://blog.cloudflare.com/why-we-started-putting-unpopular-assets-in-memory/ [22:04:47] I suspect that given our frontend+backend model of CDN caching, there is possibly much fewer exposure to the SSD latency part they talk about improving by having fewer writes. [22:05:21] so that might be less important for us, but reducing writes for longevity of the drives certainly sounds interesting. Are the ats backend disks SSDs though? [22:35:29] 10Traffic, 10Operations, 10Performance-Team: Review socket balancing in ATS/Varnish traffic layers - https://phabricator.wikimedia.org/T248522 (10Krinkle) [22:39:21] 10HTTPS, 10Traffic, 10Operations, 10Research, and 4 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276 (10Krinkle)