[06:59:08] <wikibugs>	 10serviceops, 10Patch-For-Review: Test running php7.2 and php7.4 in parallel on the beta cluster - https://phabricator.wikimedia.org/T295578 (10tstarling) I inspected the logs from the last 5 days of PHP 7.4 job runner execution on beta. The only relevant log entries I could find were three exceptions with the...
[07:57:41] <wikibugs>	 10serviceops, 10SRE, 10Patch-For-Review: Update conf1* servers - https://phabricator.wikimedia.org/T310062 (10SLyngshede-WMF) p:05Triage→03Medium
[08:35:13] <wikibugs>	 10serviceops, 10SRE, 10Patch-For-Review: Update conf1* servers - https://phabricator.wikimedia.org/T310062 (10SLyngshede-WMF) p:05Medium→03High
[08:38:16] <wikibugs>	 10serviceops, 10SRE, 10Traffic: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10SLyngshede-WMF) p:05Triage→03Medium
[08:41:47] <wikibugs>	 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Wikimedia-production-error: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10SLyngshede-WMF)
[08:42:09] <wikibugs>	 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Wikimedia-production-error: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10SLyngshede-WMF) Merged with an unrelated bug, and the relevant tags was dr...
[08:50:51] <wikibugs>	 10serviceops, 10SRE: Requesting SSH keypair for deployment server keyholder to push to Gerrit - https://phabricator.wikimedia.org/T310620 (10SLyngshede-WMF) p:05Triage→03Medium
[08:51:41] <wikibugs>	 10serviceops, 10SRE: Requesting SSH keypair for deployment server keyholder to push to Gerrit - https://phabricator.wikimedia.org/T310620 (10SLyngshede-WMF) This doesn't appear to be an SRE-Access-Request. Adding the ServiceOps tags, as they are involved in the Kubernetes migration and it makes sense to loop t...
[10:49:30] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 4 others: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10joanna_borun)
[10:49:50] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 3 others: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10joanna_borun)
[13:45:53] <ottomata>	 o/ i'm getting  a strange failure when trying to deploy eventstreams to staging k8s
[13:46:04] <ottomata>	 command "/usr/bin/helm3" exited with non-zero status
[13:46:11] <ottomata>	 STDERR:
[13:46:12] <ottomata>	   WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/eventstreams-deploy-staging.config
[13:46:12] <ottomata>	   Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
[13:52:17] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm)
[13:53:15] <jayme>	 ottomata: uh, "nice" (I wanted to capture a picture of that state :D)
[13:54:08] <jayme>	 I'll quickly collect some stuff before fixing the situation if that's oka
[13:54:12] <jayme>	 *okay
[14:38:48] <ottomata>	 ya no rush!
[15:00:50] <jayme>	 ottomata: I'm done, you should be all good to deploy again. Happy Helming!
[15:06:51] <ottomata>	 ty!
[15:06:52] <ottomata>	 trying
[15:12:20] <ottomata>	 jayme: ... i
[15:12:29] <ottomata>	 i'm not sure.  it seems to be hanging?
[15:12:34] <ottomata>	 been at
[15:12:34] <ottomata>	 Upgrading release=production, chart=wmf-stable/eventstreams
[15:12:40] <ottomata>	 for maybe 5+ mintues?
[15:12:47] <jayme>	 hanging == doing something I guess :)
[15:12:53] <ottomata>	 i guess....?
[15:13:05] <ottomata>	 hm
[15:13:11] <jayme>	 let me check
[15:13:41] <jayme>	 the new pod is not starting/crashing
[15:13:49] <jayme>	 there will be a rollback shortly
[15:14:15] <ottomata>	 okay, probably a config problem or something then.
[15:14:17] <ottomata>	 lets see
[15:15:55] <ottomata>	 hmm curl 10.64.75.198:8092/_info works fine
[15:16:09] <ottomata>	 logs look good too
[15:16:10] <ottomata>	 hm
[15:16:26] <ottomata>	 10.64.75.198:8092/_info is the readinessProbe
[15:17:05] <ottomata>	 oh, eventstreams container is good?
[15:17:05] <ottomata>	 hm
[15:17:31] <ottomata>	 it is eventstreams-tls-proxy that is failing
[15:17:45] <jayme>	 oh, looks ..yeah, that
[15:18:04] <ottomata>	 Configuration does not parse cleanly as v3. v2 configuration is deprecated
[15:18:04] <ottomata>	 ?
[15:19:01] <jayme>	 I guess there is custom templating in the chart regarding envoy 
[15:19:04] <ottomata>	 hmmm
[15:19:11] <jayme>	 and that does not work with the latest update
[15:19:14] <ottomata>	 aye
[15:19:19] <jayme>	 (of envoy ot 1.18 IIRC)
[15:19:19] <ottomata>	 when did that latest update happen?
[15:20:12] <jayme>	 I have to check. But we tend to not roll out the update if there is a diff in the deployment (other than the new envoy version)
[15:20:51] <jayme>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/771053
[15:20:57] <jayme>	 May 17
[15:21:06] <ottomata>	 okay
[15:22:27] <jayme>	 quick fix is pin the version in your deployment back to 1.15.5-1
[15:24:19] <ottomata>	 hm, the image version has not changed
[15:24:20] <ottomata>	 image_version: 1.13.1-2
[15:25:08] <ottomata>	 wait
[15:25:12] <ottomata>	 something is overriding it 
[15:25:15] <ottomata>	 -           image: docker-registry.wikimedia.org/envoy:1.15.5-1
[15:25:15] <ottomata>	 +           image: docker-registry.wikimedia.org/envoy:1.18.3-1
[15:25:18] <ottomata>	 the chart has 1.13.1-2
[15:26:38] <jayme>	 yes, yes. We override that with default. You may override our default by specifying it in helmfile.d/services/eventstreams/values.yaml
[15:26:56] <jayme>	 that will take precedence
[15:27:05] <ottomata>	 shouldn't the one in the chart values.yaml take precidence anyway?
[15:27:27] <ottomata>	 how is it overridden?
[15:28:18] <jayme>	 the values.yaml in the chart itself is the one with the lowest priority, one second
[15:29:42] <jayme>	 here is a proper description https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/services/README.MD#conventions
[15:31:28] <ottomata>	 nice okay, good to know
[15:31:32] <ottomata>	 pinning to 1.15.5-1
[15:35:35] <ottomata>	 The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition.
[15:35:36] <ottomata>	 oops
[15:35:45] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/805843
[15:37:44] <hnowlan>	 just FYI - RunJobs.php is being removed from mediawiki-config. It was disabled in the apache config on jobrunners over a year ago though so nbd https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/805775/ 
[15:39:43] <ottomata>	 hehe jayme 
[15:39:45] <ottomata>	 WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/eventstreams-deploy-staging.config
[15:39:45] <ottomata>	   Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
[15:39:51] <ottomata>	 is puppet touching something it shouldn't!?
[15:40:23] <jayme>	 did you ^C at some point maybe?
[15:40:38] <ottomata>	 i may have!
[15:40:50] <ottomata>	 maybe when it was hanging 
[15:41:17] <ottomata>	 how can I fix so I don't have to bother you if I do it again?
[15:41:17] <ottomata>	 :)
[15:42:57] <jayme>	 I'd say https://phabricator.wikimedia.org/T310714 but ....
[15:43:36] <jayme>	 https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency
[15:44:51] <jayme>	 bascically: rollback to the last state: deployed version
[15:50:26] <ottomata>	 gr8 ty
[15:50:29] <jayme>	 about the "hanging" part: helmfile will wait 600 seconds for the new deployment to become ready and will roll back in case that does not happen. That process is client side, so if you ^C it does not happen
[15:50:39] <jayme>	 leaving your latest release in the pending stage
[15:50:41] <jayme>	 *state
[15:51:33] <ottomata>	 aye so avoid ctrl c
[15:51:35] <ottomata>	 :)
[15:51:37] <jayme>	 you may lower the timeout (top of your helmfile.yaml) if you wish. The 600 sec are just something we defined as default
[15:52:02] <ottomata>	 grr codfw also in weird state...maybe some change was attempted be rolled out by someone else in the past that also failed?
[15:52:33] <ottomata>	 Mon Mar 21 16:02:23 2022	pending-upgrade	eventstreams-0.4.1	           	Preparing upgrade
[15:52:56] <ottomata>	 same in eqiad
[15:52:57] <ottomata>	 fixing
[15:53:20] <jayme>	 yeah, possible :-(
[15:53:49] <jayme>	 https://phabricator.wikimedia.org/T310714 is about detecting and alerting for releases in that state
[15:55:42] <ottomata>	 aye
[15:55:51] <ottomata>	 oo, also need to remember to rollback canary release
[15:56:18] <jayme>	 ah, indeed
[15:57:38] <ottomata>	 alrigh! deploy successful.  thank you!
[15:57:41] <jayme>	 and, obviously: Please update the chart to the latest common_templates so it supports more recent envoy versions :)
[15:57:49] <ottomata>	 yeahHhhhHhhhhhh....
[15:57:50] <jayme>	 ya
[15:57:57] <jayme>	 :D
[15:58:25] <ottomata>	 i'll make a task for eventstreams too.
[15:58:29] <ottomata>	 there is one for eventgate already
[15:58:42] <jayme>	 ack
[19:01:12] <wikibugs>	 10serviceops, 10DNS, 10SRE, 10WMF-Legal, 10wikimediafoundation.org: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Varnent) As the #WMF-Legal project tag was added to this task, some general information to avoid wrong expecta...
[19:20:49] <wikibugs>	 10serviceops, 10DNS, 10SRE, 10WMF-Legal, 10wikimediafoundation.org: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Dzahn) just a note for serviceops: policy.wikimedia.org is not currently under the control of SRE/prod servers...
[19:28:54] <wikibugs>	 10serviceops, 10DNS, 10SRE, 10WMF-Legal, 10wikimediafoundation.org: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Dzahn) There are incoming redirects into policy.wikimedia.org:  https://wikimedia.org/stopsurveillance -> http...