[06:12:16] is there a task for the project of migrating from ats-tls to something else? [06:12:28] I looked around phabricator but couldn't find it [06:15:51] are we migrating away from ats-tls?? [06:54:59] well, that's what bblack told me was likely going to happen, or that at least it was being considered [07:03:38] 10netops, 10Operations: Consider balancing VRRP primaries to cr1/cr2 - https://phabricator.wikimedia.org/T263212 (10ayounsi) a:03ayounsi [07:09:57] gilles: ok I was not aware of it, will ask around :) [07:10:49] what I know is that we'd want to get rid of the varnish frontends leaving only ats-tls doing all the work, but there were some difficulties [07:19:34] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Remove aliases `minnan` and `zh-cfr` for the Min Nan Wikipedia - https://phabricator.wikimedia.org/T230382 (10Ladsgroup) I assume someone from langcom should take a look and approve this at least. [07:43:43] 10netops, 10Operations: Upgrade Fastnetmon to 1.1.7 - https://phabricator.wikimedia.org/T257035 (10ayounsi) Thanks @MoritzMuehlenhoff I installed in on netflow4001 and it is working fine. Surprisingly though one new CLI tool `fastnetmon_api_client` is missing from the DEB. Was there any issues during the buil... [08:47:40] ema vgutierrez we will add a new lvs service [08:47:48] just heads up [08:48:44] effie: ack, thanks! [10:52:17] ema: we are going to restart pybal in ~5m [10:53:59] (sorry it took this long, we run on another issue) [11:56:43] ema, around? [12:11:27] gilles: AIUI there's nothing concrete yet, merely an idea that we might [12:11:59] and elukey AIUI getting rid of varnish-frontend also seems like potentially more worth than it is worth :/ [12:17:02] effie: yes, what's up? [12:17:24] ema: we broke it and fixed it [12:17:32] effie: did you get a tshirt? [12:17:37] no [12:17:38] :( [12:17:44] not cool [12:18:10] very uncool [12:18:21] :( [12:18:34] btw ema will you have a chance to look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/627629 sometime soon? [12:18:55] ah, it's not WIP anymore \o/ Sure cdanis [12:19:04] thanks \o/ [12:19:14] one open question from me on the VTC [12:19:46] cdanis: maybe you can use !~ [12:19:59] reasonable enough [12:23:11] cdanis: you can use 'loop' in the server block instead of multiple rxreq/txresp if you think it's cleaner [12:23:30] ah! [12:23:37] loop 5 { [12:23:39] that sounds great [12:23:48] I don't actually know VTC, I just copypasta things [12:25:54] let's not get into epistemology, copypasting things till tests are green is sufficient on a Monday [12:26:14] 🍻 [12:27:31] cdanis: I think in the regexp you want office\.wikimedia (slash) [12:27:57] yay nitpicks [12:27:58] ack, you are right, I missed one [12:29:18] +1 other than that! [12:32:03] thanks! [12:32:21] catching up on some other things and having a bit more coffee before I deplo [12:43:14] 10netops, 10Operations: Set the same OSPF weight on eqiad/codfw wavelenghts - https://phabricator.wikimedia.org/T263230 (10ayounsi) 05Open→03Resolved [12:43:16] 10Traffic, 10netops, 10Operations, 10Epic: Capacity planning for (& optimization of) transport backhaul vs edge egress - https://phabricator.wikimedia.org/T263275 (10ayounsi) [13:07:53] ema: https://phabricator.wikimedia.org/P12686 [13:07:57] I am baffled [13:10:25] ahh! [13:10:41] apparently you can expect blah == [13:10:54] not that this magic token appears anywhere in the VTC "reference" [13:14:34] okay! I am going to merge now, I'll let it roll out with normal Puppet deploys [13:15:09] 10netops, 10Operations: Consider balancing VRRP primaries to cr1/cr2 - https://phabricator.wikimedia.org/T263212 (10faidon) BTW, one dangerous impact of this (as with all ECMP!) is that it would harder to notice a situation where we don't have enough capacity to carry regular amounts of traffic when one of the... [13:23:19] cdanis: !~ also does not appear in the man page, and yet... :) [13:23:48] more adventures in DSLs [13:29:42] 10Traffic, 10netops, 10Operations: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10elukey) > are the netflow boxes and also the Analytics pipelines involved going to be okay if we are sending a great number of more flows? Do we have a high level estimate of what will... [13:30:23] https://logstash.wikimedia.org/goto/40f37d21f2470c6fa86347c8ac9671ce it is alive [13:36:07] a bit more interesting if you also add ` -report_body.type:abandoned` [13:37:41] and also `-report_body.status_code:404` [13:41:11] and then that leaves you with three reports from my IP after I broke my local machine on purpose, and 4 real oddities [14:25:42] cdanis: I see that data is coming in nicely, excellent! [14:36:10] yeah! now there's still the question of what we *do* with this data, but :D [14:36:39] very nice cdanis ! [14:37:10] I'm really not sure what to make of the events with status_code 200 and failure type 'unknown' [14:49:18] I created today's pad btw [14:54:57] <3 [15:32:27] https://phabricator.wikimedia.org/project/view/13/ just added you bblack [15:32:44] although it does seem like maybe moving between columns on the same board doesn't work for batch edits 🙃 [15:33:29] well, it might work for some initial thing where I throw all the tickets into some new triage-like column for the new board, who knows [15:34:02] phab wishlist: put your session in "bulk maintenance" mode for all the manual moves you're doing. [15:36:35] bblack: re ulsfo Netbox-DNS patches. They are now splitted in private/public and the public part includes the public bits in wikimedia.org. [15:37:08] Do you see any blocker to move forward? To you want more time to review them? [15:43:05] volans: LGTM :) [15:43:38] great! do you think we should pre-emptively depool the DC for this first one? [15:44:05] with the automated diff and manual check we're pretty confident the records are the same, but it's anyway a risky migration [15:44:37] yeah we may as well, Just In Case [15:44:57] just make sure we're not depooling ulsfo for this, right at the same time as some other risk (e.g. planned network maint or other sites being depooled, etc) [15:44:58] do we have eqiad still depooled? that might complicate a bit the things I guess [15:45:08] last I heard it was re-pooled [15:45:19] * volans double checking [15:46:01] yeah seems all polled [15:46:03] *pooled [15:47:54] yes, it was repooled because the codfw/eqdfw link was saturation [15:48:11] and then we discovered that, if you don't have swift repooled in eqiad, and you have both eqiad and esams pooled in geodns [15:48:14] you saturate the codfw/eqiad link [16:28:38] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) 05Open→03Resolved [16:28:41] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [16:28:44] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [16:30:54] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10Nuria) a:05fdans→03mforns [18:42:25] 10Traffic, 10Operations, 10Performance-Team (Radar): experiment with a "unified" ATS-BE pool - https://phabricator.wikimedia.org/T263291 (10Krinkle) [20:40:52] bblack: I made some trivial edits to https://gerrit.wikimedia.org/r/c/operations/dns/+/626656/ and also wrote https://gerrit.wikimedia.org/r/c/operations/dns/+/628935/ [20:41:05] if you're still around for the day I would appreciate a quick peek