[08:20:41] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887 (10LSobanski) 03NEW [08:20:54] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp2001:9100) - https://phabricator.wikimedia.org/T402888 (10LSobanski) 03NEW [08:32:59] hello, would someone be able to help me get https://gerrit.wikimedia.org/r/c/operations/puppet/+/1181128 deployed? [08:46:21] kostajh: I can help with that although I don't have deep knowledge about the hcaptcha proxy. Will you be able to validate it's functionality after deploy? [08:47:49] jayme: yes, I can validate it [08:49:10] kostajh: cool. I'll merge it and let you know when the puppet run on urldownloader1004 is done [08:50:03] thanks! [08:50:09] ^ cc mszabo [08:50:29] kostajh: thanks [08:53:23] kostajh: done [08:59:18] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887#11118034 (10JMeybohm) a:03jijiki @jijiki could you take a look? I think this is alerting since you set the instances up in {T276994} [09:33:46] jayme: just to confirm, the puppet run is done? [09:34:06] kostajh: yeah, that's what I meant by "done" - sorry :) [09:34:21] e.g. the change should be live [09:35:37] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887#11118205 (10Clement_Goubert) I pushed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1181673?usp=dashboard yesterday but apparently that wasn... [09:35:43] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887#11118206 (10Clement_Goubert) p:05Triage→03Medium [09:37:23] jayme: got it, thanks. We have one follow-up patch coming. [09:39:28] kostajh: ack [09:39:54] kostajh: feel free to ping me when you have it ready - assuming we don't need to roll-back? [09:40:18] no need to roll back [09:40:26] 👍 [09:40:29] just a follow-up to also target the `Set-Cookie` header [10:03:57] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp2001:9100) - https://phabricator.wikimedia.org/T402888#11118327 (10Clement_Goubert) Same issue as T402887 [11:41:30] 06serviceops, 06MediaWiki-Platform-Team: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914 (10Clement_Goubert) 03NEW [11:48:58] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887#11118756 (10Clement_Goubert) 05Open→03Resolved Permission fix looks like it resolved the alert. [11:49:29] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp2001:9100) - https://phabricator.wikimedia.org/T402888#11118761 (10Clement_Goubert) →14Duplicate dup:03T402887 [11:49:30] 06serviceops, 07sre-alert-triage: Alert in need of triage: PuppetConstantChange (instance wikikube-worker-exp1001:9100) - https://phabricator.wikimedia.org/T402887#11118763 (10Clement_Goubert) [12:33:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11118868 (10Jhancock.wm) it's on site.pp and the pressed [12:55:16] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-17 (2025-08-26 to 2025-09-09)), 07OKR-Work: [SPIKE] Test plan for rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400152#11118967 (10HCoplin-WMF) [13:34:38] 06serviceops, 13Patch-For-Review: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11119138 (10Ladsgroup) >>! In T402873#11117352, @Scott_French wrote: > @Ladsgroup - Two questions for you: > > 1. What's the desired execution concurrency limit? F... [15:21:58] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-17 (2025-08-26 to 2025-09-09)), 07OKR-Work: [SPIKE] Test plan for rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400152#11119620 (10aaron) a:03aaron [15:27:15] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#11119697 (10OWresch-WMF) [15:35:55] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#11119777 (10OWresch-WMF) a:03aaron [16:08:07] 06serviceops, 10envoy, 06SRE, 06Traffic, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11120051 (10MoritzMuehlenhoff) >>! In T402584#11115783, @RLazarus wrote: >>>! In T402584#11113754, @MoritzMuehlenhoff wrote: >> We also have 237 baremetal host... [16:22:37] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11120163 (10Scott_French) [16:28:26] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11120191 (10Scott_French) Alright, today went a lot more smoothly than the last attempt. We've completed migrating codfw and plan to apply the same in eqiad tomorrow. As noted on the... [17:27:23] 06serviceops, 10Cassandra, 06Growth-Team, 10MW-1.45-notes (1.45.0-wmf.15; 2025-08-19), 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11120485 (10Scott_French) [17:56:36] 06serviceops, 10envoy, 06SRE, 06Traffic, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11120660 (10Dzahn) [22:57:39] 06serviceops, 10envoy, 06SRE, 06Traffic: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11121811 (10RLazarus) Validated on mathoid and mw-debug (mathoid still on envoy-future, mw-debug back on 1.23 for now). One config warning in the logs from mw-debug: ` [2025-08-26... [23:00:58] !log reprepro -C main includedeb bullseye-wikimedia /srv/wikimedia/pool/component/envoy-future/e/envoyproxy/envoyproxy_1.26.8-1_amd64.deb # T402584 [23:01:09] wrong channel :) [23:34:41] 06serviceops, 10envoy, 06SRE, 06Traffic: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11121920 (10RLazarus) More deprecation warnings from the API Gateway (started locally after modifying charts/api-gateway/values-devel.yaml to use envoy-future: ` [source/common/pro...