[07:13:36] 06serviceops, 10envoy, 06SRE, 06Traffic, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11113754 (10MoritzMuehlenhoff) We also have 237 baremetal hosts with Envoy, how shall we handle these? We could e.g. add a profile parameter $use_future to pro... [07:33:41] 06serviceops, 10envoy, 06SRE, 06Traffic, 13Patch-For-Review: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11113776 (10hashar) I have updated the [[ https://integration.wikimedia.org/ci/job/helm-lint/ | helm-lint ]] job to the new image :) [10:42:54] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11114340 (10Vgutierrez) >>! In T352245#11112038, @Scott_French wrote: > I'd like to get this moving again early next week, ideally Tuesday or Wednesday during Europe / Americas overlap... [13:58:23] 06serviceops, 06MW-Interfaces-Team, 07Epic: Exploratory testing on PHP 8.3 for MediaWiki Interfaces Team components - https://phabricator.wikimedia.org/T402809 (10Krinkle) 03NEW [14:15:13] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11115074 (10Krinkle) a:03Krinkle [16:00:49] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11115652 (10Scott_French) Thanks, @Vgutierrez. I am cautiously optimistic that this should be low-touch for you. The current plan is move forward with this tomorrow (Tuesday) the 26th... [16:19:49] 06serviceops, 10envoy, 06SRE, 06Traffic: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11115783 (10RLazarus) >>! In T402584#11113754, @MoritzMuehlenhoff wrote: > We also have 237 baremetal hosts with Envoy, how shall we handle these? We could e.g. add a profile parame... [16:56:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116003 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm [18:16:39] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116267 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors: - deploy2003 (**... [19:15:17] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116454 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm [19:16:01] hi folks. gerrit is unhappy ATM. probably some scarping -- I can take a look too but just sharing here [20:01:14] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116698 (10Jhancock.wm) @Papaul this one is going to fail again. looks like there might be a missmatch between hardware and the site.pp or preseed. I'm not sure which, but they both exi... [20:01:27] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116699 (10Jhancock.wm) [20:34:46] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116894 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors: - deploy2003 (**... [20:51:59] heya, alertmanager is complaining about k8s-ingress-dse_30443 being marked down yet still pooled - What's the status of that? [20:55:53] brett: 302 to data platform sre, they own k8s-dse :) [21:02:53] thanks [21:05:22] brett I can take a look at that [21:08:09] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11116995 (10Papaul) @Jhancock.wm no entry on the wrong puppet server for this server. Please check site.pp. Thanks [21:09:04] I ACKed all the dse-k8s related alerts. We are standing up a dse-k8s-codfw cluster in T397293, looks like the alerts are related. I'll get a Slack thread started in #data-platform-sre to raise awareness [21:20:35] thank you! [22:45:34] 06serviceops: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873 (10Scott_French) 03NEW [22:51:29] 06serviceops: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11117352 (10Scott_French) 05Open→03In progress p:05Triage→03Medium @Ladsgroup - Two questions for you: 1. What's the desired execution concurrency limit? For reference `category... [22:51:42] 06serviceops: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11117355 (10Scott_French) a:03Scott_French [23:09:35] 06serviceops, 13Patch-For-Review: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11117399 (10Scott_French) [23:45:13] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737#11117438 (10RLazarus) 05Open→03Resolved