[07:24:19] 06Traffic: purged issues while kafka brokers are restarted - https://phabricator.wikimedia.org/T334078#10163315 (10Vgutierrez) purged 0.24 shipping the patch mentioned above has been deployed everywhere. Now we should expect the following behavior: purged will start logging and gathering metrics of kafka errors.... [08:19:04] 06Traffic, 06Trust-and-Safety, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253 (10jeremyb) 03NEW [08:21:00] 06Traffic, 06Trust-and-Safety, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10163405 (10jeremyb) p:05Triage→03High is this UBN? high for now. [08:21:06] 06Traffic, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10163389 (10akosiaris) a:03akosiaris @matmarex , I am pretty confident I 've found the reason and have a patch at https://gerrit.wikimedia.org/r/c/operations/... [08:23:41] 06Traffic, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10163411 (10Bugreporter) [08:27:36] 06Traffic, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10163433 (10jeremyb) [08:36:48] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10163497 (10Aklapper) [08:49:33] 10netops, 06Infrastructure-Foundations, 10cloud-services-team (FY2024/2025-Q1-Q2): cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10163529 (10aborrero) [08:50:25] 10netops, 06Infrastructure-Foundations, 10cloud-services-team (FY2024/2025-Q1-Q2): cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10163533 (10aborrero) p:05Triage→03Medium I'm tagging #netops in the hope they can help us figure out what happens her... [08:55:36] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10163560 (10Kitabc12345) Our link all is refer to https://zh.wikipedia.org/zh-hk/Te... [09:11:10] 10netops, 06Infrastructure-Foundations, 06SRE: netbox: create IPv6 entries for Cloud VPS - https://phabricator.wikimedia.org/T374712#10163608 (10cmooney) I've added the network containers as discussed on the previous task in Netbox now: https://netbox.wikimedia.org/ipam/prefixes/?within_include=2a02%3Aec80%... [09:23:28] 10netops, 10CFSSL-PKI, 06Infrastructure-Foundations: sre.network.tls cookbook - CFSSL error: bad request - https://phabricator.wikimedia.org/T375179#10163644 (10ayounsi) a:03ayounsi Thanks for having a look ! You're right, looks like the `preserve` directory isn't preserved across upgrades. So the workarou... [09:47:54] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Add cloudsw to gnmic interface stats collection - https://phabricator.wikimedia.org/T365012#10163710 (10ayounsi) 05Open→03Resolved Certs regenerated, so we're good for the next 12 months. Hopefully we will setup automation by then :) [10:10:04] 10netops, 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: sre.network.tls cookbook - CFSSL error: bad request - https://phabricator.wikimedia.org/T375179#10163746 (10ayounsi) Patch above tested and works fine. ` INFO:__main__:================================================== INFO:__main__:E... [10:21:56] any objections to me merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047509 on Monday (this is part of the wider cleanups to eventually drop our Puppet 5 setup) [11:27:51] moritzm: nope, that's fine IMHO [11:30:33] ack, thx [12:35:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10163923 (10MoritzMuehlenhoff) Ganeti row C and D have been rebalanced. [14:01:23] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164110 (10Kitabc12345) The Chinese Wiki News Stories live channel has also been o... [14:28:17] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164157 (10Kitabc12345) Increase subscribers according to the had revised https://... [14:33:11] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164163 (10Kitabc12345) according to https://zh.wikinews.org/zh-hk/User:Kanashimi [15:08:02] 06Traffic: haproxykafka features - https://phabricator.wikimedia.org/T374128#10164241 (10Fabfur) [15:15:42] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164264 (10jeremyb) >>! In T375253#10163560, @Kitabc12345 wrote: > privacy concern... [15:29:25] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikipedia, zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164276 (10jeremyb) [17:02:56] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikipedia, zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10164557 (10jeremyb) @Kitabc12345 also I'm wondering if there's been a... [17:40:54] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10164709 (10CDanis) As best I know here's the current state of this task: A couple weeks ago we discussed on -k8s-sig IRC that the "some coredns pod IPs get assigned staticall...