[00:24:58] 10serviceops, 10Kubernetes: Document how k8s logging works - https://phabricator.wikimedia.org/T289639 (10Legoktm) [07:30:35] 10serviceops, 10Prod-Kubernetes, 10Shellbox, 10Kubernetes: Docker container logs (stdout, stderr) can grow quite large - https://phabricator.wikimedia.org/T289578 (10JMeybohm) >>! In T289578#7306073, @Legoktm wrote: > Just to clarify, are these the logs that are stored at `/var/log/pods// 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) >>! In T251305#7304773, @JMeybohm wrote: > That is pretty cool, thanks! Did you actually deploy something using helm3 with `tillerNamespace:` still set? Is it just ignored in that... [08:37:03] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Reduce number of shards in redis_sessions cluster - https://phabricator.wikimedia.org/T280582 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mc2033.codfw.wmnet'... [09:09:49] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Reduce number of shards in redis_sessions cluster - https://phabricator.wikimedia.org/T280582 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2033.codfw.wmnet'] ` and were **ALL** successful. [09:32:35] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Enable the Priority admission plugin - https://phabricator.wikimedia.org/T289131 (10JMeybohm) Test in staging-codfw looks as expected: ` root@deploy1002:~# kubectl -n miscweb apply -f /home/jayme/debug_pod.yaml... [10:09:44] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Reduce number of shards in redis_sessions cluster - https://phabricator.wikimedia.org/T280582 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` ['mc2035.codfw.wmnet'... [10:41:57] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Reduce number of shards in redis_sessions cluster - https://phabricator.wikimedia.org/T280582 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2035.codfw.wmnet'] ` and were **ALL** successful. [10:46:43] 10serviceops, 10Platform Engineering, 10SRE, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster and away from the memcached cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) [10:46:49] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Reduce number of shards in redis_sessions cluster - https://phabricator.wikimedia.org/T280582 (10jijiki) 05Open→03Resolved [11:00:42] 10serviceops, 10CX-cxserver, 10Wikidata, 10wdwb-tech, 10Language-Team (Language-2021-July-September): cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503 - https://phabricator.wikimedia.org/T285219 (10Nikerabbit) [11:43:20] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10JMeybohm) [11:43:56] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review: Evaluate Dragonfly for distribution of docker images - https://phabricator.wikimedia.org/T286054 (10JMeybohm) 05Open→03Resolved This is done. Docs at https://wikitech.wikimedia.org/wiki/Dragonfly [13:57:16] rzl, legoktm: FYI I've released a new spicerack with the latest fixes for the switchdc on cumin2002. Please test them at your earlier convenience. [14:00:59] (and on cumin1001 too now) [14:35:22] hello folks [14:35:34] on deneb I see the following error while trying to build a new image [14:35:37] ERROR - Build failed: devmapper: Thin Pool has 144957 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior (image.py:205) [14:36:04] 10serviceops, 10MediaWiki-Uploading, 10SRE, 10Traffic, 10Wikimedia-production-error: Unexpected upload speed to commons - https://phabricator.wikimedia.org/T288481 (10Majavah) [14:37:39] that I guess is related to the docker [14:37:54] *docker dedicated dev, if any, but I have zero experience with it [14:41:19] ahh okok Data loop file: /var/lib/docker/devicemapper/devicemapper/data [14:41:48] and docker info says [14:41:49] Data Space Used: 97.87GB [14:41:49] Data Space Total: 107.4GB [14:41:49] Data Space Available: 9.5GB [14:42:53] what is the usual cleanup procedure? [14:44:35] https://phabricator.wikimedia.org/T287222 [14:47:18] akosiaris: just to double check, can I proceed in dropping with docker rmi some old images on deneb? I'll start with the ml ones [14:47:23] Cc: jayme --^ [15:02:47] ok I am dropping some old ml images [15:03:22] sure, go ahead [15:13:56] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-7): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10jcrespo) [15:29:56] I cleaned up a lot of old images, now things are a little better [15:29:57] Data Space Used: 66.39GB [15:29:57] Data Space Total: 107.4GB [15:29:57] Data Space Available: 40.98GB [15:30:09] still we'd need some cleanup for other images [15:31:30] 10serviceops, 10SRE: Clean up old Docker images on deneb - https://phabricator.wikimedia.org/T287222 (10elukey) Cleaned up some old istio/knative/kubeflow images, got down to this: ` Data Space Used: 66.39GB Data Space Total: 107.4GB Data Space Available: 40.98GB ` We should try to reduce the images even... [15:33:44] elukey: heads up, I am gonna reopen the task about adding calico to the kubernetes ml master nodes. I am fully catching up to it, but unless I am really mistaken, calico had nothing to do with it. It's the addition of kube-proxy (which came in via the kubernetes::node profile) that made everything work [15:34:22] but I don't like much the idea of making the control plane also be part of the execution plane. Makes things more difficult to reason about. [15:35:04] akosiaris: definitely yes, sorry if we added too much in the first place, with some more context it makes sense that kube-proxy may be the only thing needed [15:35:26] that being said, IIF I have understood well what the need was, we might very easily fix it with actually having the ML service ips advertised to the entire DC, not just internally to the cluster [15:35:42] which is something we want to do for the main clusters too [15:36:31] yeah, assuming I have understood correctly, if all that was needed was for the api server to talk to service ips that are in the cluster, kube-proxy is the component that translates from service ip -> pod ip and back [15:36:36] the main problem that we (as ML) have is that the k8s api needs to be able to contact pods on the worker nodes when a webhook is invoked, and of course istio/knative/kubeflow are full of them [15:37:10] pod IPs are able to be contacted already though. From every node in all DCs (barring firewalling rules ofc) [15:37:11] but there is always a service ip involved [15:37:37] but the service ips, those are currently utterly unknown aside the cluster. With the exception of kubestage-codfw [15:37:48] where we are testing announcing those to BGP [15:38:03] and it works, minus some HA issues [15:38:09] exactly yes, the k8s api tries to contact a service ip, that was unknown when we tested istio for the first time (it was the first showing the issue) [15:38:45] we are still at a very early stage of the cluster so any change is fine (even an invasive one) [15:39:14] we got one ORES model to work recently, but there are still too many things to fix (PSPs, Network policies, etc..) [15:39:21] re: HA https://github.com/projectcalico/calico/issues/4607 [15:39:59] volans: yay! I was just going to ask you about that :D [15:40:06] cool thanks. I 'll catch up to it and then come up with a suggestion. Just so we are clear, is a master reimage ok(ish?). [15:40:18] akosiaris: https://phabricator.wikimedia.org/T287238 was another "fun" problem, that I haven't fully fixed yet, not sure if you already went through it. [15:40:40] depending on when people have time we should do a dry run & live test later this week or early next week [15:40:41] (fixed in the sense that if we upgrade nodes to buster the problem is automagically handled) [15:40:57] legoktm: ack [15:41:08] lmk if you need anything from my side [15:46:48] elukey: you got by the iptables thing ? [15:46:52] dammit [15:47:19] https://phabricator.wikimedia.org/T245272 [15:47:37] "iptables in buster is 1.8.2, however we want to at least target 1.8.3 which is in buster-backports. The rationale of for that decision is based on https://github.com/kubernetes/kubernetes/issues/71305. However, https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/#ensure-iptables-tooling-does-not-use-the-nftables-backend way more [15:47:37] clearly says to switch to iptables-legacy. Both should be evaluated." [15:47:55] so yeah, 1.8.2 is hugely problematic [15:48:03] s/is/was/ [15:48:19] akosiaris: yeah I added a component with a new version of iptables, but that is not enforced for the moment (namely I had to apt-get install) [15:48:40] huge thanks again to topranks for the help :) [15:50:24] yeah, it's one of the myriad of reasons why we stalled on using buster. [15:50:59] up until https://phabricator.wikimedia.org/T287238#7238601 I was wondering what was going on [15:51:14] and then just saw iptables and was like "oooh, 1.8.2..." [15:51:57] that being said, a valid solution is also to update-alternatives and use iptables-legacy [15:52:08] it's in fact what's proposed in kubernetes docs [15:52:36] IIRC it didn't work or it was already set when I tried, but I could be mistaken [15:52:55] I can say that the stack works fine now :D [16:13:42] does envoy (maybe in our serviceproxy configuration) not support proxying requests over HTTPS? the same request works fine over HTTP: https://phabricator.wikimedia.org/P17083 [16:15:26] legoktm: hm, something is misconfigured, it's connecting on the same port but then including a port number in the Host header [16:16:20] yeah... [16:16:28] I updated the paste to show how HTTP works just fine [16:17:00] let me verify if MW's HTTP wrapper has the same issue [16:20:34] uh, it seems to work fine in MW? [16:21:04] so, what's happening is, MW is already listening for both HTTP and HTTPS traffic [16:21:45] (it's not working fine in MW, I didn't actually set the proxy) [16:21:46] that Envoy is *sending* HTTPS for everything, but it's only listening for plain HTTP, because we're only using it on localhost (I think) [16:22:34] I might have this all wrong -- but think of 6501 as the equivalent of port 80, not port 80-and-also-443 [16:22:42] hmm [16:23:35] so ideally I'd like to have MW configuration say "https://...." and it just work [16:24:06] the goal of setting this proxy in one place was so we didn't have to hunt down every call site and change it to use localhost:... with the Host header, if we have to do https -> http for it all, that defeats the point a bit [16:25:11] hmm [16:25:38] ok, confirmed that when I set the proxy in MW, it does not work properly [16:25:54] "Received HTTP code 404 from proxy after CONNECT" same thing (unsurprising, it's also using libcurl under the hood) [16:26:07] from googling around a little it *is* possible to serve http and https on the same envoy port, it's just weird -- so we might choose to go that route [16:26:09] (https://github.com/envoyproxy/envoy/issues/5514 has some notes) [16:26:28] but I think that doesn't solve your `Host: meta.wikimedia.org:443` problem [16:27:04] can we just serve HTTPS? [16:27:19] oh I'm totally wrong! port numbers are legal in a host header, somehow I had no idea [16:27:47] maybe -- it depends on whether something else is already expecting that to be an HTTP port, I'm not sure [16:28:06] I just introduced it 15 minutes ago :p so nothing yet [16:28:10] (also ftr there's a performance hit in doing TLS on both hops) [16:28:13] ha, fair enough [16:29:58] by both hops you mean TLS for envoy to connect, and then TLS for the HTTPS request being sent? [16:30:30] for MW (curl, here) to connect to envoy and then for envoy to connect upstream [16:31:05] one of the performance benefits of using Envoy is that *second* one is a persistent connection -- Envoy does the TLS handshake once and then stays connected, whereas php-fpm would have thrown it away on each request [16:31:24] if you do TLS-to-localhost, I think you go back to having to do a handshake to talk to the local envoy every time [16:32:09] maybe the right answer is get it working as a proof of concept and only then worry about performance, I can totally get behind that -- but that's why it wasn't the original plan [16:33:15] maybe I'm misunderstanding, but I still want to use a HTTP proxy for envoy, I just want it to proxy HTTPS requests [16:35:38] we might be talking past each other :) can you say more? I think that's what you already have [16:35:51] is the end goal "MW speaks HTTP to Envoy, Envoy speaks HTTPS to upstream"? [16:36:17] the goal is requests like: [16:36:18] >>> $req=MWHttpRequest::factory('https://meta.wikimedia.org/w/api.php?action=query&meta=siteinfo&format=json', ['proxy' => 'localhost:6501']); [16:36:21] work [16:37:21] AIUI that should be equivalent to the curl command I pasted [16:37:27] but right now we're getting the 404 error [16:42:19] ohhh I wonder if we just need to turn on strip_matching_host_port, in https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto [16:43:23] as in, I wonder if it's that Host header after all -- when curl handles the proxied request, it writes `Host: meta.wikimedia.org:443` to indicate that it should be proxied to SSL please, but then envoy's host matching says "I've never heard of any virtual hosts with a colon in them" and sends the 404 [16:43:34] oooh [16:44:22] so if I'm reading this right, setting `strip_matching_host_port: true`in the HTTPConnectionManager config would internally rewrite that to `Host: meta.wikimedia.org`before trying to route it [16:44:46] and we'd still get HTTPS coming out the other side because that's always how Envoy talks to meta.wm.o [16:45:54] or no, that can't be it, because we have `domains: ['*']` [16:49:06] ahhh I was close though! we do match on `{prefix: /}` which matches every URL path... but CONNECT requests don't have a path [16:49:17] so *that's* why the 404 [16:49:31] ( https://www.envoyproxy.io/docs/envoy/latest/faq/debugging/why_is_envoy_404ing_connect_requests ) [16:50:27] that's what threw me off before -- I thought the CONNECT request was part of a TLS handshake, which is why I was thinking the proxied request was trying to speak HTTPS on both hops [16:50:43] ahhhh [16:50:57] but it's not, it's just trying to establish a persistent TLS-free connection, because that's how HTTP-to-HTTPS proxying works [16:51:14] (I'm having a very educational morning) [16:51:37] I'm learning all of this too :p here I was thinking I could just plug it all in and it would work out of the box [16:53:52] can we set a wildcard connect_matcher? (I'm struggling to find the structure of what it's supposed to look like) [16:55:40] yeah the connect_matcher has no fields, so in yaml it just looks like `connect_matcher: {}` [16:56:09] the question is what the associated route should look like [16:58:29] https://github.com/envoyproxy/envoy/blob/main/configs/proxy_connect.yaml is a config that proxies the connect request upstream, https://github.com/envoyproxy/envoy/blob/main/configs/terminate_http1_connect.yaml is one that terminates the connect and sends the payload -- https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades#connect-support says more [17:06:20] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-7): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10ldelench_wmf) Great, thank you for your help connecting us with serviceops @jcrespo ! @Daimona is the engineer leading this project; he will be better equipped than... [17:10:40] rzl: are envoy configs normally that complex? I'm wondering if it would be simpler to have MW just downgrade the https to http (for internal/local domains) before making the request [17:10:52] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-7): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10jcrespo) Don't worry, pings have been sent to the right people- and the manager confirmed someone will get back to you soon. Please ping me again if it doesn't happ... [17:30:37] legoktm: there's a lot of bulk but we generate a lot of it in puppet, I don't know if that makes it better or worse :P [17:35:24] heh [17:36:19] rzl: are you still looking into it? or should I start attempting to work on coming up with a patch? [17:37:40] I think I've gone as far as I can without knowing more about how these proxied requests from MW work -- in particular, if a connect_matcher is what we want here, I'm not sure which of those two configs is closer to the behavior we want [17:39:45] might be the thing to do is just claim a machine for experimenting on, disable puppet, and experiment directly on envoy.yaml until the curl works as expected, then puppetize it [17:40:17] ack [17:40:21] I think https://github.com/envoyproxy/envoy/blob/main/configs/terminate_http1_connect.yaml looks like what we want [17:40:31] or if you think that MW-sided approach is viable, that might be worth a look too [17:41:19] nod [17:42:12] mostly based on the curl example being roughly what I was trying [18:31:44] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Cmjohnson) [21:01:47] Is there a "best practice" for making a new operations/deployment-charts/helmfile.d/services subdirectory for the first time? [21:08:09] * bd808 will just copy operations/deployment-charts/helmfile.d/services/shellbox and edit from there [21:10:07] there is a create_new_service.sh script [21:10:22] in deployment_charts [21:11:34] wkandek: *nod* that script creates a new directory under charts/ [21:11:40] I don't think that one creates a new helmfile.d structure unless I missed a new addition [21:11:57] however even for shellbox I copied from the most recently added one too, which was similar-users then [21:12:24] I will cargo cult my way to victory! :) [21:13:03] note that I did fix the shellbox values.yaml yesterday, so copy from master please :) [21:15:14] https://wikitech.wikimedia.org/wiki/Kubernetes#Deploy_a_service_to_staging really just skips over this step entirely [21:15:39] https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial misses it too [21:17:04] How about this one: https://wikitech.wikimedia.org/wiki/Kubernetes/Kubernetes_Workshop/Step_10 [21:17:16] I think it mentiones it but lacks detail. [21:20:01] "A helmfile is configured through YAML file needs. Take a look at the file for calculator-service in helmfile.d/services/calculator-service/helmfile.yaml where we are specifying 1 replica for staging and 2 replicas for production. " [21:20:16] at least it explains to copy it :) [21:20:24] I agree that there are mentions, and most of the things you need to know after making the initial files seem to exist. I think all that is undefined right now is how to make the helmfile.yaml file. Maybe just a "copy from ..." is what is needed somewhere in the docs? [21:21:12] it's definitely copy/paste [21:21:31] these docs are really not bad though. I hope everyone who has worked on them knows that they are appreciated! [21:22:24] That "beginner's mind" bit needed when writing the docs is hard to hold when the services have grown up over time. [21:24:46] here's what I added for now: https://wikitech.wikimedia.org/w/index.php?title=Kubernetes&type=revision&diff=1923242&oldid=1921545 [21:35:39] legoktm: another one for you... is service.port.nodePort === tls.public_port always? I see that this is true in the shellbox/values.yaml, but the duplicate places to record that number made me want to stop and ask. [21:37:14] * legoktm looks [21:41:58] this...may be a bad case of cargo culting [21:42:37] as far as I can tell (in shellbox), services.port.nodePort is only used in a `{{ if not .Values.tls.enabled }}` block [21:43:34] That sort of makes sense. Either you directly expose a nodePort or you use envoy as an ingress bascially? [21:45:09] yeah, if you look in _tls_helpers.tpl, it has `nodePort: {{ .Values.tls.public_port }}` [21:45:27] so that seems to be a leftover from pre-TLS days [21:47:13] https://phabricator.wikimedia.org/P17084 [21:47:36] apparently I tried to set nodePort first, and then later set tls.public_port: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/695432 [21:48:08] cool. I will try without the nodePort set at all and eventually we will see if that works :) [21:48:34] :D [21:49:24] * bd808 hopes to try a deploy into the staging cluster sometime tomorrow [21:50:08] woot! [21:52:47] https://integration.wikimedia.org/ci/job/helm-lint/4995/console <-- CI confirms removing nodePort is a no-op [22:30:06] In some not logged IRC discussion I had in the past with _j.oe_ he mentioned that I might need an mcrouter sidecar to connect to memcached for Toolhub. Is that something I should talk more about with akosiaris or effie or ??? [22:33:07] I can copy in the chart bits that the mediawiki chart has for setting up the sidecar, but I'm not sure if that will aim me at the pool that should ideally be used for this non-MediaWiki thing.