[06:16:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12493620 (10ayounsi) [08:36:34] XioNoX, topranks I've added lvs3010 to the pool of active LBs in esams but it isn't getting any traffic... could you check on your side of things? [08:39:00] https://www.irccloud.com/pastebin/RKbdYYMy/ [08:40:22] mainly cause I intend to depool lvs3009 next and I want to be sure that lvs3010 will get the load :) [08:43:10] vgutierrez: looking, as they're in the same rack a guess is that they're not doing ECMP on the switch (so active/passive and not active/active) [08:44:50] https://www.irccloud.com/pastebin/epmx9iiz/ [08:45:12] vgutierrez: yeah they're active/passive, but .2 should take over as soon as .3 is gone [08:49:21] XioNoX: thx :D [08:54:24] vgutierrez: if we want proper active/active we need to add `asw1-bw27-esams# set protocols bgp group PyBal multipath` [08:54:51] XioNoX: we might need it... all the load shifted to lvs3008 [08:56:55] vgutierrez: is it some kind of emergency or something we can carefully look at rolling out everywhere? I want to avoid having special config in one site [08:56:56] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Download button on Commons pages doesn't download - https://phabricator.wikimedia.org/T442313#12493907 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [08:57:17] XioNoX: I'd like to go back to active/passive on esams before the weekend [08:57:44] XioNoX: so my plan was to have the secondary one (lvs3010) handling traffic in active/active while I reimage the other two [08:58:11] and bring lvs3008 and lvs3009 back online only with the original set of services [08:58:44] mainly cause I don't have a way in liberica of depooling just a set of services and not the whole instance [08:59:16] vgutierrez: ok, so I can set it manually on the switch and remove it before the end of the day? [08:59:31] XioNoX: that's the goal yes [08:59:38] is there a tracking task? [08:59:52] T401832 [08:59:53] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [09:00:30] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12493942 (10Vgutierrez) [09:00:33] done https://www.irccloud.com/pastebin/udGTSYJF/ [09:00:57] asw1-bw27-esams# run show route 185.15.59.224/32 [09:00:57] inet.0: 1471 destinations, 2937 routes (1471 active, 0 holddown, 0 hidden) [09:00:57] + = Active Route, - = Last Active, * = Both [09:00:58] 185.15.59.224/32 *[BGP/170] 00:00:31, localpref 100 [09:00:58] AS path: 64600 I, validation-state: unverified [09:00:59] to 10.80.0.2 via irb.321 [09:00:59] > to 10.80.0.3 via irb.321 [09:01:00] [BGP/170] 00:27:57, localpref 100 [09:01:00] AS path: 64600 I, validation-state: unverified [09:01:01] > to 10.80.0.2 via irb.321 [09:01:04] oops [09:01:08] https://www.irccloud.com/pastebin/mXz7MQwZ/ [09:01:52] yep.. lvs3010 is getting traffic now: https://grafana.wikimedia.org/goto/shjq7l?orgId=default [09:02:03] let me reimage lvs3009 then [09:02:42] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12493980 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs3009.esams.wmnet with OS trixie [09:09:57] XioNoX: looks like traffic betwen lvs3008-3010 is more stable than it was for lvs3008-3009 [09:10:07] vgutierrez: what do you mean? [09:10:20] I see less non-syn LRU misses on the loadbalancers now [09:10:53] nice [09:10:56] https://grafana-rw.wikimedia.org/goto/szdqcq?orgId=default [09:14:18] speaking of which, I think we only need to apply `asw1-bw27-esams# show forwarding-options enhanced-hash-key ecmp-resilient-hash` on the switch to not have that issue anymore. Not sure for the routers though. [09:16:47] maybe this for the MX routers: `set policy-options policy-statement load-balance term resilient-consistent then load-balance consistent-hash` to be investigated of course [09:30:30] vgutierrez: thinking of that more, could katran have a resilient hash too, and redirecting a flow to the same backend server, whatever LVS nodes is used? [09:30:48] that's maglev [09:31:04] we would need to change our operations a little bit [09:31:17] like before depooling a realserver we should drain it [09:31:46] so the maglev rings stay the same across load balancers while the server is being removed from traffic [09:32:00] and then you can depool it safely [09:32:06] nice! [09:32:19] is there a task for that ? :) (I'm opening a task for the junos side) [09:32:25] s/junos/network/ [09:32:44] and same for the internal operation itself... if a healthcheck fails.. drain the host and then depool it [09:33:06] not yet IIRC [09:37:45] 10netops, 06Infrastructure-Foundations: Investigate consistent/resilient ECMP - https://phabricator.wikimedia.org/T442996 (10ayounsi) 03NEW p:05Triage→03Low [09:37:47] vgutierrez: https://phabricator.wikimedia.org/T442996 [09:45:55] ` As the new server doesn't know about those flows it terminates them, which has a small user impact.` [09:45:59] I don't think this is accurate [09:46:25] katran performs consistent hashing when it gets an LRU miss [09:46:57] so assuming that both lvs share the same view of the realservers, traffic will be balanced to the same cp server [09:47:23] ah nice that's what I was trying to ask earlier [09:47:50] so https://grafana-rw.wikimedia.org/goto/szdqcq?orgId=default doesn't have any user impact ? [09:48:04] nope [09:48:12] just increased CPU usage for the LB [09:48:32] it's cheaper to use the LRU than repeat the constant hashing op per packet [09:48:41] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12494125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs3009.esams.wmnet with OS trixie completed: - lvs3009 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled... [09:49:34] ahhh, awesome then [09:50:35] can you put that one the task? so there is less need for consistent hashing on the routers, mostly for bird anycast [09:50:39] enabling it on the network layer makes sense though [09:50:51] yeah it would be a nice to have [09:51:00] but less "reduce user issues" [09:52:22] 10netops, 06Infrastructure-Foundations: Investigate consistent/resilient ECMP - https://phabricator.wikimedia.org/T442996#12494128 (10Vgutierrez) users haven't been impacted by this as katran performs consistent hashing for LRU misses, so assuming they share the same view of the realservers, traffic will end u... [09:54:15] hmm if flows don't jump across LBs it would improve user issues during cp server maintenance/server issues [09:55:22] 10netops, 06Infrastructure-Foundations: Investigate consistent/resilient ECMP - https://phabricator.wikimedia.org/T442996#12494138 (10ayounsi) [09:55:27] cool! [09:58:18] ok, lvs3009 is ready to be repooled, it will be repooled as high-traffic2 (upload) only [10:00:10] https://www.irccloud.com/pastebin/VmpztKTO/ [10:00:16] so it will handle those two VIPs [10:07:07] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12494190 (10Vgutierrez) [10:10:47] vgutierrez https://www.irccloud.com/pastebin/71FtvaHI/ [10:11:13] lvs-high-traffic2-esams should be fixed already [10:12:40] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1360851 --> that's the commit that fixed high-traffic2 [10:20:30] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12494248 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs3008.esams.wmnet with OS trixie [10:32:59] 06Traffic, 06ServiceOps, 10Wikidata, 06Wikidata Platform Team, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12494460 (10Gehel) [10:45:24] XioNoX: and I just merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1360858, so cumin should be happy about high-traffic1 as well (mind the required puppet runs ofc) [10:48:28] cool [11:04:09] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12494595 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs3008.esams.wmnet with OS trixie completed: - lvs3008 (**PASS**) - Downtimed on Icinga/Ale... [11:15:52] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12494619 (10Vgutierrez) [11:20:00] XioNoX: lvs3010 is back to its passive role https://www.irccloud.com/pastebin/h4mSjtqc/ [11:20:16] give it 5 minutes so traffic can shift back to lvs3008 and lvs3009 and you can restore the switch config [11:48:53] vgutierrez: good now? [12:09:40] yes [12:19:03] vgutierrez: done! [12:19:50] thx [12:24:45] vgutierrez: https://phabricator.wikimedia.org/T443003 [12:24:50] 10netops, 06Infrastructure-Foundations: Investigate enabling multipath for Pybal BGP group - https://phabricator.wikimedia.org/T443003 (10ayounsi) 03NEW p:05Triage→03Low [12:25:39] only pybal uses med btw [12:25:52] liberica leverages bgp communities [12:26:59] vgutierrez: yeah, but the mechanism is the same, bgp communities then set local-pref on the router [12:27:40] so two devices advertising a similar prefix, one with the bgp community, one without, will be active/passive because they will have different local-pref [13:15:36] o/ I'd like to add a DNS record for a microsite that's being worked upon for the mobile app - does this look okay? https://gerrit.wikimedia.org/r/c/operations/dns/+/1356256 [13:35:25] microsite behind the cdn, right? [13:41:58] also, I don't know that we've ever considered a "high traffic" and/or "high reliability" usecase in the engineering of the generic microsite stuff. Is it potentially the wrong tool for the job? [13:42:37] like, is it going to have a high rate of uncached requests, and are people going to be angry if it fails sometimes? [13:44:55] maybe they're scalable now on k8s? [14:00:34] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12495291 (10SBurnett-WMF) @ssingh - would Q3 work for this? [14:19:53] vgutierrez: yep - it'll be on the miscweb cluster [14:20:25] bblack: it'll be a static site with very little to serve - it will hit the APIs though but that'll go through the usual paths [14:21:10] "it" being the microsite, will forward to the APIs? if so, I wouldn't call that static [14:22:23] (and even if it's just static redirects or whatever, there's the question of whether it matches expectations of traffic volume for how microsites are designed) [14:22:33] (and/or reliability) [14:24:13] bblack: clients will make requests of the relevant API via JS, not via the app [14:24:15] I could dig on the details myself too. What jarred me as seeming ~abnormal was the combination of "microsite" and "for the mobile app" :) [14:25:03] https://phabricator.wikimedia.org/T438505 here's the top-level task for the site [14:25:36] I'm simply matching the pattern for other miscweb sites on wikimedia.org, if there's a better route I'm happy to follow [14:29:22] yeah I'm not saying it's structurally wrong, I just don't recall us ever using "microsite" to host something that e.g. mobileapps or browser-clients might hit at high volume before. [14:29:50] and I wonder if it meets expectations in that role: re scalability/reliability vs the usual mw-web/wm-api stuff [14:30:05] oh yeah this won't be hit by actual mobile apps instances, it's a simulator of the app experience [14:30:33] for clients viewing the simulator on wikimediafoundation.org ? [14:30:55] yeah I guess that's probably a relatively-small population [14:31:57] I guess I initially misunderstood the "for the mobile app" part [as being, a microsite that would be accessed by our mobile app] [14:36:26] oh yeah that would definitely require more than a miscweb [14:41:58] Did we end up moving one more host (per dc) to text? Or it got de-prioritized due to *gestures at the world* [14:44:29] nope AFAIK [14:44:40] we are at 10 - 6 at the moment [14:45:22] yes, 10/6 except codfw 11/6 AFAICT [14:46:00] unoptimized one-liner: [14:46:00] for i in eqiad codfw esams ulsfo eqsin drmrs magru; do sudo cumin "A:cp-text_$i"; sudo cumin "A:cp-upload_$i"; done [14:46:46] yeah.. there is a supermicro test host running in codfw [14:46:58] for a second I thought the one liner is to move the hosts. I was like "wow, everything is automated now" [14:47:43] :D [14:59:53] 06Traffic, 06SRE Observability, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301#12495543 (10herron) >>! In T426301#12439621, @CDanis wrote: > But it might also be interesting to try defining a faux alert EdgeServiceDepoole... [17:39:28] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12496383 (10wiki_willy) 05Open→03Resolved Thanks, we have the credit received back now, so closing this task