[08:15:23] claime: lol, ofc. :D my bad that I did not check in advance [08:31:39] 06serviceops, 10ChangeProp, 06Content-Transform-Team, 10MediaWiki-Core-HTTP-Cache, and 3 others: Reduce the number of resource_change and resource_purge events emitted due to template changes - https://phabricator.wikimedia.org/T369898#10399790 (10daniel) [09:12:34] 06serviceops, 10ChangeProp, 06Content-Transform-Team, 10MediaWiki-Core-HTTP-Cache, and 3 others: Reduce the number of resource_change and resource_purge events emitted due to template changes - https://phabricator.wikimedia.org/T369898#10399898 (10Joe) >>! In T369898#9990753, @cscott wrote: > For completen... [10:49:51] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10400061 (10Krd) Request from :1227:f5ff:fec7:7ec1 via cp3070 cp3070, Varnish XID 926533531 Upstream caches: cp3070 int Error: 429, at Thu, 12 Dec 2024 10:46:40 GMT The same problem appears again and ag... [10:56:28] folks I've removed a bunch of Kubernetes hosts BGP sessions from CR's in codfw [10:56:29] https://phabricator.wikimedia.org/P71702 [10:56:42] all the sessions were down, hosts aren't in netbox under those names so assume that was ok [10:59:00] topranks: thanks! these where vlan moved yesterday. sorry for the noise [10:59:06] (cc jelto) [10:59:09] np [10:59:33] thanks :) [10:59:34] some good news I've merged a patch to homer which should speed up it running against the CRs [10:59:48] not exactly super-fast, but it looks good compared to the OLD speed :P [11:01:00] <3 [11:03:19] <3 [11:30:02] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: Decommission kubernetes20[07-14].codfw.wmnet - https://phabricator.wikimedia.org/T379788#10400168 (10Clement_Goubert) [12:18:35] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10400297 (10Clement_Goubert) Hi, Sorry for the delay in responding. Could you describe how you are creating the accounts? Are you using your browser and the `Special:CreateAccount` page, a script, a gadget? This... [12:41:07] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10400370 (10Krd) Special:CreateAccount in a normal browser. [13:03:35] 06serviceops, 10[DEPRECATED] wdwb-tech, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes - https://phabricator.wikimedia.org/T293063#10400421 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Thanks for the... [13:16:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400450 (10ops-monitoring-bot) depool host wikikube-worker2127.codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [13:16:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400459 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker2127.codfw.wmnet complete... [13:18:38] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400468 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2127.codfw.wmnet with OS bookworm [13:20:23] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10400469 (10Clement_Goubert) I've tweaked a rate limiting rule, could you please try again? If possible, could you enable the developer tools in your browser, and get the request headers sent in the `POST` reque... [13:32:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10400507 (10Jelto) [13:33:05] 06serviceops, 06SRE: VRT wiki fails to create account - https://phabricator.wikimedia.org/T359901#10400510 (10Krd) The account was created in the meantime. I suggest I test this at the next opportunity and report here. Thank you! [13:33:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400511 (10Jelto) [14:01:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400583 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2127.codfw.wmnet with OS bookworm compl... [14:03:25] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400588 (10ops-monitoring-bot) pool host wikikube-worker2127.codfw.wmnet by jelto@cumin1002 with reason: None [14:03:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10400590 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2127.codfw.wmnet completed:... [14:04:48] 06serviceops, 06SRE: HTTP 429 error on VRT wiki trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10400598 (10Aklapper) [15:59:34] 06serviceops, 06SRE: blank 429 error attempting to create accounts on checkuserwiki - https://phabricator.wikimedia.org/T382048#10400945 (10JJMC89) Similar to {T359901}, which appears to be caused by the mitigation for {T341908}. [16:03:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10400962 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1270-1275].eqiad.wmnet [16:06:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10400967 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1270-1275].eqiad.wmnet [16:08:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10400973 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1270.eqiad.wmnet with OS boo... [16:49:15] 06serviceops, 06SRE: HTTP 429 error on private wikis trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10401070 (10Clement_Goubert) p:05Triage→03High [16:49:44] 06serviceops, 06SRE: HTTP 429 error on private wikis trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10401073 (10Clement_Goubert) [16:49:45] 06serviceops, 06SRE: blank 429 error attempting to create accounts on checkuserwiki - https://phabricator.wikimedia.org/T382048#10401076 (10Clement_Goubert) →14Duplicate dup:03T359901 [16:50:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401077 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1270.eqiad.wmnet with OS bookwor... [16:52:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401084 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1271.eqiad.wmnet with OS boo... [16:56:20] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: Decommission kubernetes20[07-14].codfw.wmnet - https://phabricator.wikimedia.org/T379788#10401093 (10Jhancock.wm) [16:57:10] 06serviceops, 06SRE: HTTP 429 error on private wikis trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10401106 (10Clement_Goubert) >>! In T359901#10400510, @Krd wrote: > The account was created in the meantime. I suggest I test this at the next opportunity and repo... [16:57:21] 06serviceops, 06SRE: HTTP 429 error on private wikis trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10401108 (10Clement_Goubert) 05Open→03In progress [17:13:45] 06serviceops, 06SRE: HTTP 429 error on private wikis trying to create account via Special:CreateAccount - https://phabricator.wikimedia.org/T359901#10401161 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert The problem should be resolved for all private wikis now. [17:20:30] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: Decommission kubernetes20[07-14].codfw.wmnet - https://phabricator.wikimedia.org/T379788#10401197 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [17:33:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401280 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1271.eqiad.wmnet with OS bookwor... [17:35:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401286 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1272.eqiad.wmnet with OS boo... [18:17:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401439 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1272.eqiad.wmnet with OS bookwor... [18:19:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401441 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1273.eqiad.wmnet with OS boo... [19:02:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401540 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1273.eqiad.wmnet with OS bookwor... [19:04:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401543 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1274.eqiad.wmnet with OS boo... [19:46:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401647 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1274.eqiad.wmnet with OS bookwor... [19:48:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401648 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS boo... [20:33:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401683 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS bookwor... [20:35:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401687 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker1276.eqiad.wmnet [20:37:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401688 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS boo... [20:38:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401689 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1276-1277].eqiad.wmnet [20:40:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401703 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1276.eqiad.wmnet with OS boo... [21:20:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401766 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS bookwor... [21:25:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401790 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1276.eqiad.wmnet with OS bookwor... [21:25:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401794 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1277.eqiad.wmnet with OS boo... [22:09:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10401886 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1277.eqiad.wmnet with OS bookwor...