[06:58:01] 10serviceops, 10Parsoid (Tracking), 10Patch-For-Review: Upgrade nodejs on testreduce1001 - https://phabricator.wikimedia.org/T345220 (10Marostegui) [07:33:16] 10serviceops, 10MW-on-K8s, 10MediaWiki-Engineering: EtcdConfig using stale data: lost lock in /srv/mediawiki/php-1.42.0-wmf.1/includes/config/EtcdConfig.php on line 218 - https://phabricator.wikimedia.org/T349376 (10Joe) EtcdConfig uses eventually APCUBagOfStuff, see https://gerrit.wikimedia.org/r/plugins/gi... [07:38:41] 10serviceops, 10MW-on-K8s, 10MediaWiki-Engineering: EtcdConfig using stale data: lost lock in /srv/mediawiki/php-1.42.0-wmf.1/includes/config/EtcdConfig.php on line 218 - https://phabricator.wikimedia.org/T349376 (10Joe) p:05Triage→03Medium Setting the priority to medium as we do clearly read correctly f... [08:16:08] 10serviceops, 10GrowthExperiments-Homepage, 10GrowthExperiments-ImpactModule, 10SRE, and 2 others: RefreshUserImpactJob consumes too many file descriptors - https://phabricator.wikimedia.org/T344428 (10Joe) Sorry for the silence, I was first at a conference then in bed sick (and I'm still not in a great he... [10:41:03] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Gracefully handle pod termination in mw-on-k8s - https://phabricator.wikimedia.org/T331609 (10JMeybohm) 05Resolved→03Open I noticed there are still differences in mw-debug config between codfw and eqiad originating from this task (https://gerrit.wikimedia.or... [11:50:20] 10serviceops, 10CX-deployments, 10MinT, 10Prod-Kubernetes, and 3 others: Remove the use of :latest image tags in production - https://phabricator.wikimedia.org/T348856 (10Nikerabbit) [11:50:50] 10serviceops, 10MinT, 10Prod-Kubernetes, 10Kubernetes, and 2 others: Remove the use of :latest image tags in production - https://phabricator.wikimedia.org/T348856 (10Nikerabbit) [12:54:12] 10serviceops: Rebalance kafka partitions in main-{eqiad,codfw} clusters - 2023 edition - https://phabricator.wikimedia.org/T341558 (10brouberol) If someone lands on this ticket in the future, please note that usage of the tooling has been streamlined and documented: https://wikitech.wikimedia.org/wiki/Kafka/Admi... [13:21:14] 10serviceops, 10Data Engineering and Event Platform Team, 10Data-Engineering, 10Discovery-Search (Current work), and 2 others: Improve the flink-app chart to provide more useful defaults - https://phabricator.wikimedia.org/T346315 (10Ottomata) [13:50:11] 10serviceops: Rebalance kafka partitions in main-{eqiad,codfw} clusters - 2023 edition - https://phabricator.wikimedia.org/T341558 (10Ottomata) <3 [14:44:14] hello folks! [14:44:30] https://gerrit.wikimedia.org/r/c/mediawiki/services/change-propagation/+/966029 is ready to go (changeprop on node18) [14:44:42] my plan is to build the img, deploy in staging and test there for a while [14:45:00] ok for you? [14:45:57] go for it! [14:48:35] sounds good! [14:50:46] 10serviceops, 10Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting: Introduce PCS cache management layer - https://phabricator.wikimedia.org/T348995 (10Jgiannelos) a:03Jgiannelos [15:16:57] Failed to fetch http://mirrors.wikimedia.org/debian/dists/bookworm/InRelease Could not connect to mirrors.wikimedia.org:80 (208.80.154.139), connection timed out [15:17:02] sigh [15:18:13] forcing a retry in jenkins, changeprop doesn't like me [15:18:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [15:20:16] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10elukey) npm warnings in CI: ` #15 10.83 npm WARN EBADENGINE Unsupported engine { #15 10.83 npm WARN EBADENGINE package: 'eslint-plugin-jsdoc@39.2.... [15:36:53] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10elukey) [15:37:14] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/967957 - changeprop to the future [16:08:37] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10Jdforrester-WMF) [16:08:57] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 8 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Jdforrester-WMF) [16:46:12] 10serviceops, 10Data-Engineering, 10Event-Platform: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10elukey) Deployment to staging was fine, no errors in the logs etc.. The only thing that I noticed is: https://grafana.wikimedia.org/d/000300/change-propagation?orgId=1&va... [16:46:32] hnowlan: same cpu increase as the last time :( --^ [16:46:37] I'll try to debug it tomorrow [16:48:01] elukey: agh, ack [16:59:21] 10serviceops, 10iPoid-Service: Implement proxy configuration for kubernetes deployment - https://phabricator.wikimedia.org/T349171 (10jijiki) @tchanders please use the environmental variable `HTTPS_PROXY` (vs `HTTP_PROXY`), as is the convention we use in the repo, along with `https_proxy`. This is a bit on me...