[03:26:48] 10Traffic, 10DNS, 10Mail, 10SRE: ITS request to update SPF & DNS Records for Trust & Safety - https://phabricator.wikimedia.org/T272750 (10drochford) Apologies for the tardiness @pkang - Following up with Nasma (Ops Manager). Will revert then. [09:50:04] 10netops, 10Anti-Harassment, 10SRE, 10SRE-tools: Surprising new svc.eqiad.wmnet dns entry deployed: similar-users on host decommission - https://phabricator.wikimedia.org/T273275 (10jcrespo) 05Open→03Resolved a:03hnowlan I don't think there is further actionables here except for @Volans to read this... [09:51:50] 10netops, 10Anti-Harassment, 10SRE, 10SRE-tools: Surprising new svc.eqiad.wmnet dns entry deployed: similar-users on host decommission - https://phabricator.wikimedia.org/T273275 (10Volans) @jcrespo I'm aware of this conversation, I just didn't had anything to add as @akosiaris had already gave all the rel... [10:05:57] vgutierrez: hey, is this part needed? https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/ipsec.pp#L22 I couldn't find ipsec being applied to cp nodes anymore (it's only used in redis multidc it seems) [10:19:09] I don't think so Amir1, we got rid of IPSec after we moved from varnish-be to ats-be [10:19:41] that's what I suspected [10:19:44] okay, I make a patch to remove the whole section [10:22:58] thanks [12:03:34] vgutierrez: for when you have time https://gerrit.wikimedia.org/r/c/operations/puppet/+/661073 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/661070 [12:03:41] ack [12:04:39] Thanks! [13:33:46] hi folks, there's someone on #mediawiki who is getting a 502 from a cp when saving a large page [13:56:20] 10Traffic, 10SRE, 10Wikisource: Error when trying to create new page on Romanian Wikisource - https://phabricator.wikimedia.org/T273623 (10Majavah) [15:11:03] bblack: thank you for looking at those scary patches! I am going to roll out the first two (which are no-ops), let things sit for a while (up to a day depending on how clinic duty and some other pieces of work go), and then do the scary inline-C one [15:36:57] cdanis: ack, thanks! [16:05:49] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10elukey) @hashar I applied the nginx change to bypass Jetty, can you test again? [16:14:10] 10HTTPS, 10Cloud-VPS, 10User-bd808, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) a:03bd808 [16:22:03] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) Fetching https://archiva.wikimedia.org/repository/mirrored/junit/junit/4.13.1/junit-4.13.1.jar it still takes a while until the transfer starts: | time... [16:26:59] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10elukey) I think that we should make tests inside the wikimedia network, testing from home is not reliable (as you said there are too many variables, one above a... [17:54:54] 10HTTPS, 10Cloud-VPS, 10Patch-For-Review, 10User-bd808, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) Closure of POST loophole announced: https://lists.wikimedia.o... [17:55:13] 10HTTPS, 10Cloud-VPS, 10Patch-For-Review, 10User-bd808, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) [18:02:50] 10HTTPS, 10Cloud-VPS, 10Patch-For-Review, 10User-bd808, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) 05Open→03Resolved [18:02:59] 10HTTPS, 10Traffic, 10Cloud-Services, 10Cloud-VPS, and 3 others: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627 (10bd808) [20:12:44] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) From my connection something else is broken download a 2.13M [[ https://archiva.wikimedia.org/repository/releases/com/googlesource/gerrit/plugins/javame... [21:34:44] 10Traffic: Wikidough: Upgrade to dnsdist 1.6.0 - https://phabricator.wikimedia.org/T273679 (10ssingh) [21:35:12] 10Traffic: Wikidough: Upgrade to dnsdist 1.6.0 - https://phabricator.wikimedia.org/T273679 (10ssingh) [21:35:17] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [21:36:44] 10Traffic: Wikidough: Upgrade to dnsdist 1.6.0 - https://phabricator.wikimedia.org/T273679 (10ssingh) p:05Triage→03Medium [21:55:18] would be nice if someone wants to confirm tlsproxy::prometheus can be removed https://gerrit.wikimedia.org/r/c/operations/puppet/+/659377 [22:06:02] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10K6ka) The bot has now been unblocked and is editing again. I will report here if the bot is seen blanking WP:CHUS again. [22:48:38] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Cyberpower678) Yes it turns out the bot had robust error handling for prod errors. It didn’t have robust handling for reused OAuth nonces. For some reason it got... [22:53:30] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Reedy) >>! In T273003#6798157, @Cyberpower678 wrote: > Yes it turns out the bot had robust error handling for prod errors. It didn’t have robust handling for reuse...