[00:36:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11924035 (10Papaul) 05Open→03Resolved The last BGP session between cr3 and asw1-23 is now up, We ca now close this task. Thanks to all that did he... [00:38:17] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: FY2526 Q3 ulsfo: switch refresh - https://phabricator.wikimedia.org/T408510#11924038 (10Papaul) 05Open→03Resolved The ULSFO switch refresh is complete. Good to close this task. [07:05:51] 06Traffic, 10ContentTranslation, 06LPL Hypothesis, 06Product Safety and Integrity, 06Security-Team: CX dashboard can't load page collections on some wikis (blocked by CORS) - https://phabricator.wikimedia.org/T426323#11924265 (10kostajh) [07:19:12] 06Traffic, 06ServiceOps new, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11924275 (10achou) [07:25:40] 06Traffic, 06ServiceOps new, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11924293 (10achou) a:05klausman→03DPogorzelski-WMF [08:04:56] 06Traffic, 06ServiceOps new, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11924324 (10achou) [08:31:15] 10netops, 06Infrastructure-Foundations: POPs - free up 2xQSFP ports - https://phabricator.wikimedia.org/T424611#11924376 (10cmooney) a:03cmooney [08:35:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new line card in cr2-eqiad slot 0, move card from slot 1 to cr1-eqiad slot 0 and configure - https://phabricator.wikimedia.org/T426343#11924396 (10cmooney) Actually now that I think of it we should probably combine this work wi... [08:36:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new line card in cr2-eqiad slot 0, move card from slot 1 to cr1-eqiad slot 0 and configure - https://phabricator.wikimedia.org/T426343#11924398 (10cmooney) [10:14:24] 06Traffic, 10ContentTranslation, 06LPL Hypothesis, 06Security-Team, and 2 others: CX dashboard can't load page collections on some wikis (blocked by CORS) - https://phabricator.wikimedia.org/T426323#11924804 (10Dreamy_Jazz) [10:33:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new line card in cr2-eqiad slot 0, move card from slot 1 to cr1-eqiad slot 0 and configure - https://phabricator.wikimedia.org/T426343#11924902 (10cmooney) [10:35:58] 06Traffic, 10ConfirmEdit (CAPTCHA extension), 10hCaptcha, 05Bot detection and mitigation (WE4.10 hCaptcha), and 2 others: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin - https://phabricator.wikimedia.org/T426178#11924906 (10jijiki) If there is something #s... [14:08:47] 06Traffic, 07SEO: Bing can't search images from Commons, is Wikimedia denying their requests? - https://phabricator.wikimedia.org/T425850#11925597 (10Aklapper) What to look for on Wikimedia's side without knowing the IPs which they use? [14:13:41] 06Traffic, 06SRE: Nakavo - Rate Limiting Query - https://phabricator.wikimedia.org/T422872#11925607 (10Aklapper) > I will change the logic so that we always default to a thumbnail in this case, thank you for sharing. I will monitor the changes and hopefully no more 429 will show up @NakavoDev Hi, is there sti... [14:28:13] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11925694 (10xcollazo) @AKhatun_WMF doe the WARNINGs continue? [14:58:14] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11925783 (10AKhatun_WMF) @xcollazo Yes, we still have very frequent warnings. https://airflow.wikimedia.org/dags/refine_webrequest_hourly_text... [15:59:06] 06Traffic, 10ConfirmEdit (CAPTCHA extension), 10hCaptcha, 05Bot detection and mitigation (WE4.10 hCaptcha), and 2 others: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin - https://phabricator.wikimedia.org/T426178#11926100 (10kostajh) 05Open→03Resolve... [19:36:44] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11927041 (10xcollazo) Maybe we should double all thresholds.