[09:39:34] hello on-callers [09:39:49] as FYI I am going to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1243727 since there are no deployments ongoing [09:40:16] the /v2/restricted prefix in the docker registry (MW images essentially) will be routed to a new backend [09:40:25] I'll test it etc.. [09:40:56] cc: Emperor [09:45:33] elukey: thanks for the heads up [09:47:21] elukey: ack [09:57:51] currently deploying [10:11:32] less than 20 mins to push all the MW Docker images, with the forced 300s wait time [10:11:46] now the k8s workers are pulling the images [10:14:42] sync looks good [10:26:28] Emperor: the interesting bit is https://grafana.wikimedia.org/d/1a13678a8d55/ceph-pools-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-datasource=000000005&var-cluster=$__all&var-job=$__all&var-topk=15&refresh=5m&viewPanel=panel-12, but nothing dramatic [10:41:40] deployment finished [10:47:32] I left some notes in cortobot about how to rollback (essentially https://phabricator.wikimedia.org/T412951#11653669) [11:54:25] GitLab needs a short maintenance break in one hour [13:00:59] gitlab maintenance done [14:02:02] Amir1: Why is wgThumbnailStepsRatio=0 on private wikis? [14:02:13] officewiki is accessing commons with non-standard widths [14:02:32] e.g. https://office.wikimedia.org/wiki/Contact_list -> .jpg/284px [14:02:48] (most are fine, it's based on aspect ratio, and height) [14:03:09] * Krinkle sees PTO on cal [15:30:48] i love how gitlab has just decided to render a part of a command on https://gitlab.wikimedia.org/repos/sre/slothslos as math [15:33:35] oh wow, that's funny [15:33:39] lol, it's missing a newline on line 17 [15:44:58] <_joe_> taavi: you've been volinted! [15:49:02] lol, it's indeed funny but that's just inline math from https://docs.gitlab.com/user/markdown/#math-equations [18:07:03] oncallers, be aware that traffic is pooling two fresh hosts, cp2043/cp2044: These are running on Trixie with HAProxy 3.0 rather than the usual Bullseye/HAProxy 2.8. We're keeping an eye on them but they seem to be performing well. [18:07:27] nice job! \m/ [18:26:25] brett: glad to hear, nice job indeed! [18:36:02] inflatador: do you think you could look at https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1240701 when you've a minute, please? :) [19:16:09] Emperor 👀 [19:16:51] +1'd, sorry for the trouble. I thought I fixed that one up already ;( [19:17:57] NP :) [19:48:58] side note: h-ashar had mentioned adding dpkg commands to check changelog formatting to that repo's CI, might be worth pursuing. Let me see if I can find the ticket... [19:51:38] T283855 is the task , looks like most of the work is done already [19:51:39] T283855: operations/docker-images/production-images has no CI - https://phabricator.wikimedia.org/T283855 [21:57:48] > This afternoon we deployed a patch for Thanos to introduce a dedicated Store Gateway for the Ruler component... [21:57:53] The rollout is finished. At the moment, the patches that prevent cross-DC traffic between Queriers and Rulers are still in place. I’ll revert them tomorrow or on Monday after further checks.