[08:24:12] <kostajh>	 jayme: I have another proxy update patch, if you (or someone else) has time for review and deployment: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182106
[08:26:12] <jayme>	 kostajh: I'd like to leave the review to someone who speaks nginx :D
[08:29:51] <kostajh>	 jayme: fair enough!
[08:54:36] <stevemunene>	 hello, getting an error on a homer diff check `Policy error: Policy kubedse_import referenced but not defined In [edit] (policy-options)` on the devices `cr2-codfw.wikimedia.org` and `cr1-codfw.wikimedia.org` 
[09:00:19] <kostajh>	 jayme: suggestions for who I could ask for reviews? 
[09:01:07] <jayme>	 stevemunene: I'd suggest to contact #-traffic about that
[09:01:37] <jayme>	 kostajh: v.gutierrez is probably a good option, given he also did the review for the last patch
[09:02:01] <kostajh>	 ok
[10:26:06] <kostajh>	 vgutierrez: would you have time to review / deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182106 ? 
[10:28:34] * vgutierrez looking
[10:41:37] <mszabo>	 looks like pcc is failing on an unrelated resource there
[10:41:55] <mszabo>	 because of logic added in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182148
[10:44:14] <hnowlan>	 tappof ^ 
[10:45:14] <vgutierrez>	 kostajh: happy to review it.. but deploying hcaptcha stuff is definitely out of my scope :)
[10:45:34] <mszabo>	 we can do all required testing post deploy
[10:47:21] <tappof>	 mszabo: Could you share the link to the PCC compiler output?
[10:47:32] <mszabo>	 tappof: https://puppet-compiler.wmflabs.org/output/1182106/7327/urldownloader1004.wikimedia.org/change.urldownloader1004.wikimedia.org.err
[10:47:53] <tappof>	 thx mszabo 
[10:51:58] <kostajh>	 vgutierrez: what mszabo said above, we can do the verification after deployment. the visible change is that a `__cf_bm` cookie should no longer be seen in the client, because the proxy will have removed it
[10:57:03] <tappof>	 mszabo: I’ve got a fix, sorry... I’ll submit a patch in a few minutes.
[11:06:19] <vgutierrez>	 kostajh, mszabo looks good.. it's a pity that nginx doesn't let you filter response headers natively 
[11:07:18] <mszabo>	 vgutierrez: yeah its an issue with multiple headers specifically
[11:07:28] <mszabo>	 $upstream_http_foo works if you expect only one
[11:07:41] <mszabo>	 but set-cookie may legitimately appear more than once
[11:07:46] <vgutierrez>	 yup
[11:18:25] <tappof>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182148 reverted mszabo 
[11:18:32] <mszabo>	 ty
[11:35:03] <kostajh>	 vgutierrez: please let us know if you would like any other changes on the patch
[11:41:20] <vgutierrez>	 looks good to me kostajh 
[11:44:34] <kostajh>	 cool. let's ship it?
[12:41:10] <vgutierrez>	 kostajh: sure, let me know when
[12:41:45] <kostajh>	 we're ready when you are 
[12:43:09] <vgutierrez>	 ack, let's go then
[12:43:36] <vgutierrez>	 which servers are taking care of serving hcaptcha.wm.o?
[12:43:46] <vgutierrez>	 urldownloader1004 alone?
[12:44:09] <vgutierrez>	 hmm it looks like that
[12:45:33] <hnowlan>	 yep
[12:46:24] <vgutierrez>	 running puppet on urldownloader1004 as we speak
[12:47:37] <vgutierrez>	 https://www.irccloud.com/pastebin/hJbQcaUH/
[12:47:57] <vgutierrez>	 ok... /etc/nginx/lua isn't there? :)
[12:49:32] <kostajh>	 hmm
[12:50:17] <vgutierrez>	 yeah.. you need to create /etc/nginx/lua first
[12:50:25] <vgutierrez>	 could you submit a CR?
[12:50:28] <kostajh>	 seems to be referenced in a few other places but I guess not here https://codesearch.wmcloud.org/search/?q=etc%2Fnginx%2Flua&files=&excludeFiles=&repos=
[12:51:54] <kostajh>	 I'm guessing we want something like https://gerrit.wikimedia.org/g/operations/puppet/+/6f691134a3c3c758816f57a661d1f3fe26717fd7/modules/dynamicproxy/manifests/init.pp#179, but not sure where we'd add that for the proxy
[12:52:14] <vgutierrez>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182539/
[12:52:38] <kostajh>	 thanks!
[12:53:05] <kostajh>	 vgutierrez: does that need `Hosts: urldownloader1004.wikimedia.org` ?
[12:53:25] <kostajh>	 ah, you've added it 
[12:58:58] <vgutierrez>	 kostajh: looks good? 
[13:00:57] <kostajh>	 vgutierrez: seems like it's worth trying 
[13:03:00] <vgutierrez>	 merged and running puppet again in urldownloader1004
[13:03:29] <vgutierrez>	 Notice: /Stage[main]/Profile::Hcaptcha::Proxy/File[/etc/nginx/lua]/ensure: created
[13:03:29] <vgutierrez>	 Notice: /Stage[main]/Profile::Hcaptcha::Proxy/File[/etc/nginx/lua/filter_set_cookie.lua]/ensure: defined content as '{sha256}ffaf1ffec362c7e5ec3102faba9ba0eb0716270fb958d97b4ef31aba0c496083'
[13:03:36] <vgutierrez>	 ok, that's better
[13:04:28] <vgutierrez>	 kostajh: your change is now live 
[13:05:18] <kostajh>	 vgutierrez: thanks! looking 
[13:07:44] <kostajh>	 vgutierrez: looks good, thank you!
[15:46:29] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11124927 (10Scott_French)
[15:47:28] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11124940 (10Scott_French) p:05Triage→03Medium
[16:15:02] <wikibugs>	 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Route test2wiki rest.php APIs through rest-gateway - https://phabricator.wikimedia.org/T402412#11125147 (10Clement_Goubert) p:05Triage→03High
[17:28:11] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125489 (10Jhancock.wm) a:03Jhancock.wm
[17:32:48] <wikibugs>	 06serviceops: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11125533 (10Scott_French) Alright, the next and final phase of this is to devise and execute a plan for migrating etcd itself to cfssl-PKI. As noted in the task description, the functionality is already pre...
[17:45:55] <wikibugs>	 06serviceops, 10envoy, 06SRE, 06Traffic: Envoy config updates from v1.26 - https://phabricator.wikimedia.org/T403101 (10RLazarus) 03NEW
[18:12:25] <wikibugs>	 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11125783 (10Scott_French)
[18:25:01] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125886 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm
[18:25:14] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125888 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2012.codfw.wmnet with OS bookworm
[18:25:22] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125889 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2013.codfw.wmnet with OS bookworm
[18:25:33] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125890 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2014.codfw.wmnet with OS bookworm
[18:40:34] <wikibugs>	 06serviceops, 10envoy, 06SRE: Envoy config updates from v1.26 - https://phabricator.wikimedia.org/T403101#11125946 (10ssingh)
[18:41:03] <wikibugs>	 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.26.8 and drop buster - https://phabricator.wikimedia.org/T402584#11125948 (10ssingh)
[18:41:15] <wikibugs>	 06serviceops, 10envoy, 06SRE: Upgrade Envoy to >= 1.24 - https://phabricator.wikimedia.org/T380211#11125960 (10ssingh)
[18:41:57] <wikibugs>	 06serviceops, 10envoy, 06SRE: Upgrade Envoy to >= 1.24 - https://phabricator.wikimedia.org/T380211#11125964 (10ssingh) For awareness: I checked with @RLazarus and removing the Traffic tag. We can add back later as required.
[18:42:38] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11125967 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm executed with errors: - maps2011 (**FA...
[19:06:31] <wikibugs>	 06serviceops: Identify areas covered by the Production Readiness checklist - https://phabricator.wikimedia.org/T400476#11126116 (10bking)
[20:31:47] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11126375 (10VRiley-WMF) a:03VRiley-WMF
[20:37:03] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11126381 (10Jhancock.wm)
[20:37:51] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11126384 (10Jhancock.wm) ran into an issue where the installer thinks that there is a raid when the config does not call for one. need to go back and compare and check for issues.
[20:56:35] <wikibugs>	 06serviceops, 13Patch-For-Review: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11126430 (10Scott_French) @Ladsgroup - Thanks for the follow-up!  > I don't think so. These stuff are not considered canonical data. The small counters have self-he...
[21:49:37] <wikibugs>	 06serviceops: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11126653 (10Scott_French)
[23:21:32] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11126899 (10Scott_French) Looking at what the PCC diffs on [[ https://gerrit.wikimedia.org/r/1182658 | r/1182658 ]] look like, this looks promising in terms of requiring zero tweaks to...