[06:59:40] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10047372 (10Fabfur) >>! In T370741#10041302, @Vgutierrez wrote: > we had some alerts ongoing during the weekend due to this task: > > `... [13:57:04] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10048344 (10Fabfur) [13:57:13] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10048350 (10Fabfur) 05Open→03Resolved This has been completed and verified on hosts in ulsfo [17:26:13] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049089 (10greg) This is waiting on review+merge of the patch by the SRE Traffic team: https://gerrit.wikimedia.org/r/c/operations/dns/+/1059412 Adding their proje... [17:28:28] just a drive by friendly request for review of this patch: https://gerrit.wikimedia.org/r/c/operations/dns/+/1059412 (phab: https://phabricator.wikimedia.org/T370963 ) adds a TXT record so we (just fundraising) can get spam/complaint feedback from Yahoo. [17:28:55] greg-g: just added a comment, reviewing and merging now [17:29:07] (we missed it because Traffic was not added but thank you for doing that) [17:29:15] sukhe: awesome! thank you! [17:29:28] that reminded me we should probably auto-add traffic to ops/dns patches so I will do that as well once I am done with this [17:29:35] * greg-g nods [17:29:42] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049096 (10ssingh) Thanks for adding Traffic @greg. We missed this patch as the team was not added. I will deploy it right now. [17:33:15] I am trying to figure out if this is a TXT record on @ or under the spop1024 [17:33:29] should be on spop1024 [17:33:33] because the conversation on the task indicates the latter but the patch is for @ [17:33:37] ok [17:33:38] ahhh [17:37:11] kinda confusing hmm [17:37:21] you might want to edit the regexes on https://www.mediawiki.org/wiki/Git/Reviewers to get auto reviewers [17:37:29] there is already some DNS stuff there [17:37:35] mutante: yes thanks, I will do that [17:37:38] cool:) [17:37:54] I would have had to search for this link so thanks :> [17:38:23] oh, actually. that's all DNS stuff in the puppet repo, but operations/dns just isn't in it [17:38:30] different repo [17:38:38] yeah will add it I guess [17:39:33] I have been wondering similar things for my team. What we are missing is like a feature to add a team alias as reviewer in gerrit. [17:39:43] which then notifies the members [17:39:56] that would be so nice [17:39:59] greg-g: merging [17:40:01] but also, how many reviewers is too many.. where is the line [17:44:05] mutante: as the page points out, I was able to add it to the gerrit setting itself [17:44:11] and on the other page, I added myself and brett for now [17:46:03] sukhe: oh, I don't think I was even aware of that similar feature [17:46:08] nice [17:48:44] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049141 (10ssingh) Hi all: Apologies for missing this patch. Like I said, Traffic wasn't added so we were not notified. We have fixed this for future to be automati... [17:50:16] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049143 (10bsisolak) Yahoo's interface confirmed and new feedback loop is setup. [17:52:02] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049147 (10greg) Thanks @ssingh ! [18:03:30] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049162 (10nisrael) @greg @bsisolak can I confirm that now that the new feedback loop is setup that we are safe to continue sending emails? [18:08:08] 06Traffic, 10fundraising-tech-ops, 13Patch-For-Review: Add a TXT record to the Yahoo sending domain - https://phabricator.wikimedia.org/T370963#10049177 (10bsisolak) Yes [18:59:10] sukhe: https://phabricator.wikimedia.org/T353817#10049308 [18:59:56] we need a routing rule somewhere to go /beacon/event -> some_endpoint. right now that endpoint is mediawiki.org/w/rest.php/EventLogging/v0/beacon [19:00:18] we were considering doing that routing in Apache, but we need mod_proxy and other things because reasons, and it is not gonna work. [19:00:30] ok [19:00:48] would it be bad to do that (proxy?) routing on frontends instead? [19:01:05] i guess the q is: which is worse, apache rewrite for this, or some frontend routing logic? [19:02:57] ottomata: reading [19:05:16] yeah, seems like a better fit for VCL based on what I am reading [19:05:32] the "better" question is a bit hard to answer in that sense. like I said, we have all sorts of rewrite rules that are in VCL but should not be [19:05:40] but we are trying to move them to Apache wherever possible [19:07:59] ottomata: [19:08:01] we need mod ssl and SSLProxyEngine on to allow the request to be proxied to https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon [19:08:15] ^ or something, yeah. [19:08:15] but [19:08:16] and this works for you without any issues? [19:08:45] Hm, no I don't know if that is the solution for sure. All I know is that mod_proxy is needed, and whatever is installed is not sufficient [19:09:18] with whatever is currently enabled for apaches, I get: [19:09:19] No protocol handler was valid for the URL /beacon/event (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule. [19:09:33] _joe_: has tried things and said it probably isn't going to work. [19:09:55] <_joe_> well there was a solution that worked, right? [19:10:11] yes yes [19:10:28] <_joe_> and you could still have a script that just translates REQUEST_URI [19:10:29] I have another way: php file in docroot should work with mod_rewrite [19:10:51] if traffic prefers the rule in apache, we should do it in apache. [19:11:04] this rule will probably be needed for another 5 years unfortunetly (according to cindy) [19:11:51] sukhe: thank you, i think traffic's preference for apache is enough to do the docroot solution. i'll proceed with that [19:13:30] ottomata: if for whatever reason it doesn't work out, let's talk about the VCL [19:16:45] okay, thank you! [20:52:49] 10netops, 06Infrastructure-Foundations, 06SRE: cloudsw1-d5-eqiad instability Aug 6 2024 - https://phabricator.wikimedia.org/T371879#10049699 (10Dzahn) We got paged at 20:19 UTC for "primary outbound port utilisation over 80%" on both cloudsw1-d5 and cloudsw1-f4 today. Shortly after it resolved. But somethi... [20:54:57] 10netops, 06Infrastructure-Foundations, 06SRE: cloudsw1-d5-eqiad instability Aug 6 2024 - https://phabricator.wikimedia.org/T371879#10049702 (10Dzahn) {F57154133}