[00:22:47] 10Traffic, 06Operations: Sort out vcl_deliver vs vcl_synth mess with v4 VCL - https://phabricator.wikimedia.org/T135696#2307560 (10BBlack) [02:38:57] 10Traffic, 06Operations: Sort out vcl_deliver vs vcl_synth mess with v4 VCL - https://phabricator.wikimedia.org/T135696#2307697 (10BBlack) Actually it doesn't turn out to be as messy as it seems at first. It's mostly the analytics, debugging, etc headers in common code that need invocation in vcl_synth, but n... [09:36:44] 10Traffic, 10DNS, 06Operations, 13Patch-For-Review: arbcom-nl.wikipedia.org doesn't have a functioning mobile website - https://phabricator.wikimedia.org/T135480#2308155 (10Joe) 05Open>03Resolved [09:36:56] 10Traffic, 10DNS, 06Operations, 13Patch-For-Review: arbcom-nl.wikipedia.org doesn't have a functioning mobile website - https://phabricator.wikimedia.org/T135480#2300317 (10Joe) The mobile site now works. [09:52:29] 10Traffic, 10DNS, 06Operations, 13Patch-For-Review: arbcom-nl.wikipedia.org doesn't have a functioning mobile website - https://phabricator.wikimedia.org/T135480#2308184 (10Sjoerddebruin) Thanks! [12:37:56] 10netops, 06Operations: codfw-eqiad Zayo link is down (cr2-codfw:xe-5/0/1) - https://phabricator.wikimedia.org/T134930#2308571 (10faidon) 05stalled>03Resolved And here's the formal RFO: {F4030747} [12:40:15] 10netops, 06Operations: cr2-codfw LUCHIP/trinity_pio error messages - https://phabricator.wikimedia.org/T134932#2308582 (10faidon) The case is making progress. The latest are: - We should reboot FPC 0 and see if that will stop the error messages. - If not, we can RMA. [13:31:15] ema: any thoughts on https://gerrit.wikimedia.org/r/#/c/289588/ ? [13:31:46] bblack: looking [13:32:06] http://book.varnish-software.com/4.0/chapters/VCL_Basics.html vs http://book.varnish-software.com/3.0/VCL_Basics.html flowcharts is what the logic is based on [13:32:41] (3.0 *everything* delivered goes through vcl_deliver. 4.0 some requests bypass vcl_deliver via vcl_synth, but we still want client-facing headers set and analytics and X-Cache, etc) [13:33:39] I noticed this because I know cache_misc can synthesize from a backend, which would show "int-remote/int-local" stats in the new caching grafana thingy, but I'm seeing zero counts. [13:33:55] because our varnish4 clusters aren't really setting X-Cache type stuff for synthetic responses at all :) [13:40:06] bblack: looks good! I think it should be @req_method though [13:40:33] and perhaps we could add a vtc test [13:40:40] oh I thought it was throughout? previous it was v3-only for the purge-check, but since vcl_synth has purge traffic too, I just put the check in universally [13:41:31] oh, the @ you mean [13:41:33] yeah ok [13:41:41] right :) [13:42:27] yeah could do a vtc for http->https, that's an easy universal synthetic [13:42:36] (and check for x-cache, x-analytics, etc) [13:42:42] yep [13:43:44] I still really don't have a good workflow around VTC in puppet repo [13:44:12] heh it's not optimal at the moment [13:44:14] since I can't (and shouldn't) forward auth over ssh to labs instances, I work from local on the repo, and local is a stupid mac [13:44:46] I should raise priority on the virtual phab ticket in my head to reinstall my mac with debian :P [13:44:53] +1! [13:45:07] 10Traffic, 10Beta-Cluster-Infrastructure, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2308967 (10hashar) Well / is reported as full by the filesystem: ``` # df -h / Filesystem Size Used Avail Use% Mounted on /dev/vda3 19G 18G... [13:45:15] I just know it means probably days of productivity loss, been putting it off for longer than I can remember now [13:45:35] ^^that bug above is for beta cluster / labs not a big deal imho [13:45:52] I mean more in general we should be able to write/run vtc tests on our development machines [13:46:06] (somehow) [13:46:09] bblack: I bought myself an Intel NUC with some SSD + Jessie [13:46:14] and use that over ssh from the mac ;) [13:46:23] hashar: might peek at /var/cache/varnishkafka? when prod caches get full disks, it's often kafka spam in there that can be wiped. I don't know if kafka runs at all on the beta caches. [13:46:28] 10Traffic, 06DC-Ops, 06Operations, 10hardware-requests, 10ops-esams: Decomission amssq31-62 (32 hosts) - https://phabricator.wikimedia.org/T95742#2308975 (10mark) Yes, I'll remove this in the near future - likely before we refresh the other (newer) low range cp* hosts. [13:46:33] so I still have OS X nicer GUI [13:47:11] bblack: I am not sure what happened, apparently varnishlog has kept a fd open attempting to write to it while the file inode got moved / removed or something [13:47:14] I really don't even think the GUI is all that awesome. But I do like not having to worry about driver compatibility and all that jazz, and their built-in full disk crypto is easy to use. [13:47:23] so maybe it is writing to an inexistent file [13:47:30] restarting the process fixed it ;) [13:48:14] top reason I use the mac is the GUI really, specially the font and their rendering. I have never ever managed to get something to render nicely on Jessie or Ubuntu :( [13:48:14] and I still tend to mostly think of my local machine as a browser+ssh terminal and then I do code-work on e.g. gdnsd on a virtual machine at linode. But I'm not forwarding my WMF-related auth keys over ssh to linode just to work heh. [13:49:22] maybe a virtual box on the mac will do :) [13:49:32] for most WMF things it works ok, but varnish and nginx code work can be painful, moving around between git on my local machine and copper for building and a labs VM for testing [13:49:47] (pulling in the changes via gerrit un-merged cherry-picks and such anonymously on copper/labs) [13:49:59] (or just scp-ing patchfiles over) [13:51:01] I'm using a self-hosted puppetmaster in labs for that, it works! [13:51:15] that == vtc/testing varnish stuff [13:51:20] maybe you can build on the labs instance instead of on copper [13:51:29] or we can get CI to build the stuff for you whenever you push [13:51:39] ema: yeah but what's your patch workflow then? write the new vtc on labs, scp back to your host to commit? [13:52:06] the copper solution works well for building [13:52:32] (assuming the patches you're building are already in gerrit, at least unmerged) [13:52:50] bblack: sometimes, other times when the patch is "easy enough" I just create a CR on gerrit first and download it from the labs instance [13:53:17] and hope it works :) [13:53:33] if it doesn't, scp back and forth yes [13:54:43] which is not great, also because by being able to run the vtc tests without puppet we could carry on with the varnishtest CI work [13:55:32] I don't see how we're going to reach vtc without puppet really [13:55:44] since puppet templates the config under test [13:56:12] with something like this, but better? https://gerrit.wikimedia.org/r/#/c/276733/ [13:56:23] :) [13:57:10] I donno, it doesn't seem like a great direction to go [13:57:38] lots of duplication with the puppet structure and variables, will probably never cover them all for all clusters and variable settings, etc... [13:57:48] it would be a nightmare keeping the two in sync [13:58:28] that's true [13:58:33] <_joe_> my advice would be to have jenkins run the tests on a machine where you compile the puppet code for the various clusters and run vtc on it [13:58:54] well, so stepping back a little into a different approach: [13:59:48] 1) We could structure the VTC tests such that they're split up by what cluster/layer they're applicable to (or for some tests, all clusters and/or all layers) [14:00:52] 2) Structure the Rakefile so that puppet-compiler can run them in the context of a host (whatever clusters/layers that host has) [14:01:22] 3) At that point, can't we hook into it manually with puppet-compiler GUI runs on integration.wm.o with a list of test hosts in various clusters/dcs? [14:01:43] and then we're just back to the universal problem of automating puppet-compiler checks for applicable hosts on commits [14:02:01] <_joe_> :) [14:02:58] I guess what I mean is, ideally the VTC is structured such that you can run "rake test" or whatever to invoke all the applicable VTC directly on cp4011.ulsfo.wmnet, which would run all the files relevant to cache_maps + ulsfo (frontend + indirect backend) [14:03:24] and it's just a matter then of having CI automagically simulate various host role examples [14:03:28] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2309076 (10Joe) [14:03:35] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) p:05Triage>03Normal [14:04:04] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) p:05Normal>03Low [14:04:41] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) a:03Joe [14:04:46] we need puppet compilation just to get the templated full VCL config correct all the time. We're not getting away from that easily without basically duplicating the complexity of modules/varnish + modules/role/manifests/cache and everything related in ruby. [14:05:04] (multiplied by #clusters + #dcs) [14:05:34] (if we want the tests to eventually extend to nearly-complete coverage of our contracts with the application layer, which I think is the goal) [14:07:25] not that that list of contracts is explicitly defined at the appropriate level of abstraction, anywhere [14:07:33] (yet!) [14:07:40] thinking about the contracts problem... [14:07:50] the first-pass mental approach goes like: [14:08:37] 1. Write a wikitech page documenting traffic-layer contracts. It treats the Traffic-layer as a black box agnostic of datacenter tiering, front/backe-ends, Varnish itself, etc... [14:09:21] (you can imagine it documenting things like X-Cache, X-Analytics, how we treat Cache-Control in general and the exceptions on text cluster, how we handle Vary in general and the Vary:Cookie specifics on text, etc, etc....) [14:10:06] (there's a level of detail it doesn't go beyond, though, by explicitly referencing that anything not covered here, assume standards-complaint HTTP behaviors from various spec links) [14:10:06] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2309108 (10hashar) I have kept this one open in case #traffic want to investigate whether it can be a problem in production. For beta, the resta... [14:10:46] 2. Flesh out the VTC testsuite to cover everything in the Wikitech document (which I guess has hierarchical section indices to reference) [14:12:07] but long-term that gets messy. what happens when someone updates wikitech and not the VTC. who audits that the prose on the wikitech lists and the functionality of the VTC test lines up and how? You at least need direct links between the two somewhere, like a comment in a VTC file that says "This checks cache_text contract 4.3.2.a" [14:12:54] It seems better to approach this in reverse and abstract the contract in some explicit format tied to the VTC. [14:13:13] vcl_load( contract-4.3.2.a.yaml ) [14:13:35] by contract do you mean a clear definition of what the VCL are supposed to do ? [14:13:38] (In other words, the VTC files or something related in the repo actually contains the relevant contract's human text alongside the test for it, and the human-readable full contract is generated from the VTC testsuite automatically) [14:13:48] maybe that could be written down directly in puppet.git ? [14:13:58] so if you update the contract, you get to update the VCL as well [14:14:03] and tests [14:14:09] hashar: by contract I mean a clear definition of what the Traffic layer does to traffic between user<->application as an abstract black box [14:14:11] or the tests could be seen as the contract [14:15:02] it goes beyond varnish in a number of ways [14:15:14] it would cover what nginx is doing, too, actually. [14:15:31] and much like a high-coverage testsuite makes it easy to refactor the codebase of one little project. [14:15:59] a high-coverage contract like this could let us refactor the whole traffic layer with less risk, because we know exactly what functionality we've gauranteed to the consumers. [14:16:44] (of course we'd need new implementations of the tests if we moved away from varnish, and we need different tests than VTC even today for the nginx part of the problem) [14:17:10] (but even having an explicit and self-documenting testsuite at all is a step down the right path) [14:17:39] in theory when a patch is proposed we could setup a whole mini cluster dynamically with the patch applied + puppet run to provision them [14:17:50] then run some kind of end-to-end test against that [14:18:14] well we don't necessarily need a mini-cluster for every commit [14:18:40] hopefully just a puppet-compiler run against an example of every role+dc combo (16x virtual test hostnames, currently) [14:19:43] none of this is novel in the abstract. it's like a well-maintained piece of C or Python software, right? It needs a testsuite with full coverage, and you commit new tests for bugs, new tests with all new functionality, etc... [14:20:10] it's just aiming towards applying those ideals to an entire layer of infrastructure instead of one code project. [14:25:26] hashar: hmmm yeah there is a disconnect from isolated puppet-compiler runs I guess. We can do unit tests that support the contracts that way. but we can't test end-to-end I guess. [14:26:08] it's trivial to think about how we'd structure VTC for that, right? We can have VTC simulate multiple stacked varnish instances [14:26:29] but we don't have a good way to generate the VCL for all those layers. puppet never does that for any single host, and puppet is what generates it. [14:27:31] so, we'd need a way to generate the puppet-compiled VCL output for all the relevant test hosts (4 for a cluster?), and have the end-to-end VTC load up various machines' VCL into layered daemons. [14:28:00] the layering part can be done in VTC itself right? [14:28:05] yeah [14:28:14] but the input VCL files that VTC loads... [14:28:26] would need to come from different puppet hostnames/roles/etc [14:28:46] yes [14:28:51] which is what the gerrit commit earlier tries to do, but the complexity and duplication of that approach will get very unwieldy if we try to test everything [14:29:11] (and it's still hardcoded for the needs of the one simplest cluster) [14:29:58] if one can get a single test to run for a single hostname/role that would be a good step forward [14:30:08] yeah [14:30:09] then from that base figure out how to add more test/ more combinations [14:30:22] I guess puppet doesn't have something like chef-solo right? :) [14:30:33] :) [14:30:44] and I have no idea how the puppet compiler works :( [14:30:53] the whole catalog compilation scares me off [14:30:55] Step 1 - migrate WMF from puppet to chef (estimated time: 3 days) :P [14:31:04] lol [14:32:04] Ryan Lane wrote a nice post about puppet to (SaltStack || Ansible) http://ryandlane.com/blog/2014/08/04/moving-away-from-puppet-saltstack-or-ansible/ [14:32:32] so, one thing we could do, that might help us on many levels, is try to move away from puppet doing so much of the VCL-templating work [14:32:42] he was probably biased toward salt but still evaluated ansible [14:33:00] because of ERB? [14:33:17] as in, the input variables (@vcl_config and such) just go into a host-local file like /etc/traffic/varnish.conf, and some local process (a tiny python script) templates things out [14:33:23] scap3 is using python / jinja2 with some success apparently [14:33:43] that abstraction would make it easier in some test scenarios. It would make puppet-compiler less-useful vs today, though. [14:34:07] well given a list of @variables, we can already manually run erb to expand the templates [14:34:17] in any case, it seems like a noble goal to get complexity out of puppet and put it into things that can be driven by puppet|anything-else [14:34:47] hashar: yeah, the problem is the complexity of that in puppet in practice today. variables come from all over the place and enter VCL templating in non-obvious ways. [14:34:54] puppet.git:/utils/expanderb.rb is supposed to do handle the expansion [14:35:22] something like expanderb.rb -f modules/varnish/templates/foobar.vcl.erb vcl_config=bar vcl_dies=false [14:35:26] with an extra layer of abstraction in there, we can say concretely that puppet generates this configfile with these 17 variables in it, and a local VCL-templating script only consumes those [14:35:32] it most probably does not support hash / list variables though [14:35:48] yeah that sounds like a plan [14:36:46] <_joe_> please let Ryan's musings on config managers out of our discussions [14:37:43] <_joe_> :) [14:37:52] (and then we just need the VCL templatefiles + templatescript + config file with 17 vars. and to simulate real hosts from puppet, we just need examples of what that 17-var file looks like on the relevant cluster+layer) [14:38:17] that sounds more like something CI can consume independently of puppet easier with less confusion and duplication [14:38:21] <_joe_> it's easier to puppet-compile for a host, and extract the vcl files from there [14:38:26] bblack: we also have stuff like @varnish_version4, @beresp_obj and such, but that's a limited number of variables that simply depend on the varnish version [14:38:30] are all the variables in hiera already ? [14:38:45] hashar: no, there's a lot of places [14:39:17] they're mostly in the role manifests in @vcl_config, but then also random hiera bits, and god knows what else [14:47:40] ema: hopefully we'll be past that soon enough on the @varnish_version4 -related things :) [14:47:51] yes! [14:48:04] I can't wait for the commit that un-templates all that :) [14:48:27] and removes all the v4 versions of python stuff [14:48:33] v3 I mean of course [14:49:48] back on the topic of templating outside of puppet: we'll need to handle the confd directors.vcl case too [14:49:57] it would probably improve that situation dramatically [14:50:27] well currently we mock the directors entirely in vtc [14:50:38] instead of double-templating erb+go->VCL, we could have confd's go-template just put down a json file that the VCL-templater local script consumes, and trigger re-templating like puppet does when it writes its variables down. [14:51:02] "trigger" could just be the templating script having a daemon mode to watch input files, too [14:52:30] (and not vcl-reload on no-op, and maybe rate-limit the changes a bit if confd+puppet spam it multiple times in a few seconds) [14:55:07] (on change requiring vcl-reload: if time-since-last-change > 5: pause 5 seconds and then coalesce all updates into one vcl-reload at the end) [14:55:17] s/>/ because I think varnish still isn't very efficient at reloading vcl like crazy. it tends to leak memory in the form of old VCL files and their dynamic allocations. [14:56:09] (or VCL shared objects, I mean) [14:58:05] almost 24h of data now in https://grafana-admin.wikimedia.org/dashboard/db/varnish-caching [14:58:51] awesome [15:00:08] those big int-front rates on text, at first I thought maybe they were TLS-redirects, but it didn't add up [15:00:18] turns out the biggest chunk is /beacon/ 204s [15:00:34] we seem to be overusing beacons a bit, that they're such a large fraction of overall reqs :P [15:01:35] wikibugs found something better to do [15:02:51] I want to play with lots of things that will spike those graphs a bit, but I'm trying to make myself wait until we have some decent "normal" history to compare [15:02:57] uh 09-chunked-response-add-cl.vtc on misc seems to be failing [15:03:10] from the new package? [15:03:24] oh, that's normal [15:03:26] ---- c1 0.6 EXPECT resp.http.Content-Length () == "5" failed [15:03:36] we haven't un-reverted the CL-sensitive VCL [15:03:41] for cache_misc I mean [15:03:41] right! [15:03:47] should we? [15:03:51] probably, yeah [15:04:30] I re-applied and reverted parted of it, confusing the commit history [15:04:43] I'll write the vtc for deliver+synth unless you want to do that [15:05:00] this is the one to revert I think: https://gerrit.wikimedia.org/r/#/c/288231/ [15:05:12] ema: if you can that'd be awesome. I'll go fix the @ thing. [15:05:17] alright [15:07:14] heh, this is one of those tests that apply to all clusters though, and the vtc stuff is cluster specific [15:07:35] we have several of those I think [15:07:59] oh wait, we don't [15:08:13] but still [15:08:16] 09-chunked-response-add-cl.vtc is misc-specific [15:08:57] https://gerrit.wikimedia.org/r/#/c/288231/ revert is what puts that magic in place, and it's misc VCL not shared VCL [15:09:34] (because we don't want to kill streaming on text just to get absolute limits on content-length, since we don't expect problematically-large objects there) [15:10:05] in general we probably have a few tests that cover shared VCL, but we somehow always need the cluster type to load the vcl in the vtc [15:10:18] eg: include "/usr/share/varnish/tests/wikimedia_maps-backend.vcl"; [15:10:23] yeah [15:10:32] test-templating? :) [15:10:52] yes but then I guess we need to specify all test names in puppet [15:11:01] with static files we just use the dir name [15:11:20] that's probably my puppet-ignorance speaking though [15:12:11] as in, now we do this: [15:12:12] # VTC tests [15:12:12] file { '/usr/share/varnish/tests/': [15:12:12] source => 'puppet:///modules/varnish/tests', [15:12:12] owner => 'root', [15:12:15] group => 'root', [15:12:17] recurse => true, [15:12:20] } [15:12:22] can we do the same thing with templates? [15:12:36] kind of, but not really :) [15:12:58] we can make it less painful with create_resources() or whatever it's called [15:13:06] and just have a list of the testnames [15:15:14] on the other hand, we probably need to specify the Host header anyways to avoid getting a 404 not served here [15:15:26] and that is cluster-specific [15:16:12] we could use one hostname that works everywhere [15:16:23] well, there isn't one though :) [15:16:49] text can use all most of the misc hostnames in theory (the ones in wikimedia.org) [15:16:53] upload can only use upload though [15:17:02] I don't think maps checks [15:17:08] maps doesn't [15:17:16] (but it probably should!) [15:17:36] all of that falls under the bin of unifying cache_misc's applayer-backend magic to the other clusters [15:17:55] and having some mode that works for text where all other non-matching hosts/URLs default to a certain backend [15:19:53] such as vtc_backend perhaps [15:20:04] which is there already for testing purposes [15:20:17] basically cache_misc now has a half-assed variant of what we want in https://phabricator.wikimedia.org/T110717 . maps and upload could use it roughly like it is now, but text needs something more-advanced. [15:23:55] probably something to be careful of when un-varnish3-ing cache_misc VCL. the supporting VCL templates for that stuff, we may want to copy to text/upload before they convert. [15:24:02] (well, or to common code anyways) [15:27:09] uh I haven't merged https://gerrit.wikimedia.org/r/#/c/289433/ yet [15:27:25] bblack: is there anything else to do except for merging? [15:27:37] looks pretty safe to me [15:27:54] alright! [15:28:06] for ops/dns, the replacement for puppet-merge on palladium is "authdns-update" on any of the authdns nodes [15:28:13] baham.codfw, radon.eqiad, eeden.esams [15:28:27] ah yes [15:28:28] I think it needs a full root shell too [15:28:31] IIRC [15:28:54] running on any of the 3 will update the others [15:31:32] done [15:33:20] nice [15:33:27] now we can shut off eqiad+codfw and save lots of power :) [15:34:03] and improve performance on cache misses! [15:34:24] the cache stats are fascinating. even when looking indirectly through ganglia, I've not often tried to infer remote-cache-hits. they're tiny for most clusters, except text. [15:34:47] but of course that doesn't mean they're pointless. they play a big role if we wipe/lose/stale-ify remote backends. [15:36:41] 09-chunked-response-add-cl.vtc is green again [15:37:01] yay [15:38:39] it might be kind of interesting (in the long term, after we get over some other related hurdles so we're not blendering up too much complex change at once...), to hack in some VCL support in cache_miss to look for hits in DCs outside the linear path [15:39:08] e.g. even though eqiad can talk to the applayer directly, on cache_miss it could ask codfw with some kind of Only-If-Cache request before hitting the application. [15:39:50] I think especially in varnish4 that kind of thing is easier, with how cache_miss and vcl_backend_error and retry-logic isolated to the backendside without touching the frontend-side works, etc [15:40:48] I think with v4 and roughly the structure we have today, we could at least make eqiad<->codfw pull hits from each other like that to refill on loss easier. [15:40:49] but only to "close" DCs? Or would esams also be a candidate? [15:41:23] I think it would be risky to automate e.g. a codfw miss asking esams with how things work today. it would have to retry-step serially through eqiad first, it adds a lot of latency. [15:41:54] it might be that for certain requests it's still a win, I don't know [15:42:01] in all such cases we have to avoid loops, too [15:42:34] as a starting point, given today's cache-routing, you might say "in vcl_miss in eqiad, if the source DC was not codfw, try Only-If-Cached to codfw before trying applayer" [15:42:52] (request source DC immediately up the chain) [15:43:43] I guess in general terms, the conditional would be "if cache_miss, and $another_dc was not in the request chain, and we would normally contact the applayer directly at this point, try Only-If-Cached to $another_dc first" [15:44:12] and yeah, could loop that over all other possible $another_dc starting with the closest ones latency-wise [15:44:57] whether jumping back out to a very remote cache DC is "worth it" is debateable though. For an expensive fresh page render in mediawiki, maybe yes. for a maps tile, probably not [15:45:34] for static stuff, surely not [15:46:14] in any case, for a first pass, could just include eqiad+codfw in the set of $another_dc to check, and it adds some resiliency in some scenarios with less intervention on our part. [15:47:33] (and since esams+ulsfo don't ever do direct applayer contact for misses, regardless of how we factor it they'd never in practice try to actually do it) [15:48:03] so if eqiad caches were wiped, requests into codfw (or ulsfo which flows through codfw) would go straight to eqiad->applayer if they miss that far. [15:48:23] but user requests coming into eqiad (or esams which flows through eqiad) would try Only-If-Cache on codfw before trying the app. [15:49:45] somewhere between that stuff and active:active cache routing plans with general support for applayer at any DC, etc.... [15:49:58] eventually you reach some ideal state where the caches act more like peers and forwarders for each other. [15:50:04] instead of a strict hierarchy [15:54:16] it doesn't fix the whole problem for persistence still (if we lose all caches everywhere due to an induced crash) [15:54:54] but it makes some of the less scenarios more-palatable (where today we'd have to manually switch up geodns and/or cache::route_table to cope with total cache loss at 1x DC, maybe with this it's automagic) [15:56:08] s/less/lesser/ [15:57:06] bblack: 'obj.hits': Not available from method 'wm_common_xcache_deliver' [15:57:21] oh, interesting [15:57:28] makes sense :) [15:57:37] synthetic objects aren't hittable [15:57:54] so the obj.hits bit needs to move back to deliver-only [15:58:11] will amend, unless you've already amended locally? [15:58:17] nope! [15:58:20] ok [15:59:17] so the whole else if can go away I guess [16:02:19] ok updated patch [16:02:36] cool [16:04:04] there are probably better ways to factor that, but I'll save it for the next round of general no-op refactoring [16:04:38] bblack: it works and https://phabricator.wikimedia.org/P3145 passes [16:05:21] nifty! [16:05:34] now, should we just drop it under misc or should we start actually tackle cluster-agnostic tests properly? [16:05:53] *tackling I guess :) [16:06:03] eh I'd say drop it under misc for now. cluster-agnostic tests applies to more than just that anyways [16:06:49] ok [16:07:49] amended w/ vtc [16:07:56] awesome [16:08:38] +1ed [16:09:28] ok let's see if we can get some int-* stats from maps/misc :) [16:15:22] on a unrelated note, we still have the vsm files permission issue on v4 [16:15:45] my attempt of fixing it failed weeks ago with https://gerrit.wikimedia.org/r/#/c/281918/ [16:16:08] $ curl -v https://foo.wikimedia.org/ --resolve foo.wikimedia.org:443:91.198.174.217 [16:16:11] would it be so bad to just have them 644, which was v3 behavior anyways? [16:16:13] < HTTP/1.1 404 Domain not served here [16:16:18] < X-Cache: cp1061 int, cp3007 miss, cp3010 miss [16:16:22] ^ should generate int-remote :) [16:16:56] ema: the real problem is, even if we decide 644 is ok, how do we enforce it when varnish re-creates it? [16:17:11] I guess we're lucky so far that it merely overwrites it even on fresh start [16:17:28] does it have a config param for it? [16:17:30] with some ugly post-restart systemd hack? [16:17:49] config param, great question [16:18:36] we probably could/should do this "right" and add various users to the varnish group [16:18:47] but I'm not any clearer than you are on how we puppetize that properly :) [16:19:11] (given the ganglia problem) [16:19:40] it would also solve the mess that is all of those varnishlog python scripts running as root, I think [16:19:51] we could give them per-daemon users [16:21:35] seeing int-front stats for maps+misc now. and int-remote for misc, but it's rare (bad domainname) [16:21:47] still, now it shows 0.0% instead of 0%, which means it had at least one little datapoint [16:23:09] cool! [16:23:48] really only misc-VCL's 404/503 code from the automatic backend selection should be doing synth on backends at all, and that could totally be moved to frontend [16:24:01] which makes me question the utility of even logging or categorizing backend synthetics [16:24:32] but we may see them in the future, maybe for some types of varnish-internal 503s if can't contact applayer. and for our loop-prevention 503ing on inter-cache loop. [16:27:46] the test I really want to try for the new graphing (next week, after we have more normal data to see) is wiping an esams upload backend's storage and seeing where the graphs go [16:28:03] how much of it does eqiad catch in practice, and how much does it temporarily increase full-misspass, etc [16:28:50] also, I want some good history on text before merging https://gerrit.wikimedia.org/r/#/c/288936/ which doubles text-frontend memory size [16:29:32] we should see the restarts zap hit-front and replace that mostly with hit-local temporarily, and ramp back to recovery. [16:30:06] question is how the impact looks given a certain timing on the frontend-restarts (time to recovery per node / overall), and whether after recovery looks done the extra memory actually helped or not. [16:31:07] these are all the sorts of things that I think I have a blurry intuitive grasp of, because I've inferred it from other data in the past. but having explicit data is way nicer. [17:06:48] I'm feeling ambitious today https://gerrit.wikimedia.org/r/#/c/289696/ [17:07:10] (it's never going to work) [17:12:48] ema: does the ganglia debian package create the user? [17:13:05] bblack: I believe so [17:13:34] maybe the first thing we need is way over in whatever installs the ganglia package... [17:14:12] user { 'ganglia': ... } which just creates the puppet-level user, and depends on the package install, and isn't expected to actually create the user, just validate that it's present? [17:14:40] so yes, the debian package creates ganglia as a system user [17:14:40] then we'd have a puppet ganglia user object to work with at all [17:14:59] and I've double checked that puppet doesn't scream if the user is there already [17:15:00] at which point maybe this becomes bikeshedding about how different other roles/modules can add groups to it [17:15:19] yeah but it might pre-create it wrongly if that executes before ganglia package install [17:15:41] right so that user definition should depend on the package I guess [17:15:46] I'm not even sure if there's a sane way to do user { 'ganglia': that requires package install and won't still try to create it on first run because it looked at the user list before the package inst? [17:15:58] I have no idea, but I tend to mistrust puppet :) [17:17:21] I think maybe: user { 'ganglia': ensure => present, require => Package[...] } [17:17:25] will work in the ganglia class [17:18:41] but then we face the question: how do you puppetize it such that N other modules/roles can add personalized groups to that user? [17:19:23] User['ganglia'] { group => ... } would do it once, but would conflict if several tried. I don't think there's an array-merge for that kind of thing [17:19:42] in my quest for knowledge I bumped into a thing called virtual resources [17:20:19] the fact that we need "virtual" stuff to add a user to a group makes me slightly sad but still [17:20:43] does that even work for this pattern? [17:22:05] oh, duh, we're looking at this backwards... [17:22:36] 1. We could define user { 'ganglia': in a ganglia module somewhere for completeness with just ensure=>present and dependency on the package, but I'm not sure that's even necessary... [17:23:31] 2. In some varnishy place, we define: group { 'varnish': ensure => present, members => ['ganglia', ...], require => [Package['varnish'], some ganglia class] } [17:23:59] I think that kind of pattern can work. you can define members for groups, instead of groups for users [17:24:19] right! Probably 1. is not necessary, depending on the ganglia package/class means the user must be there [17:24:42] oh wait, silly [17:25:15] even in the latest puppet docs, the provider for "group" that Linux uses is "groupadd", which lacks the feature "manages_members" that the members attribute requires... [17:26:23] we could make an exec shellscript that idempotently adds users to group varnish faster than digging through puppet stupidity :P [17:26:37] that's for sure [17:26:57] look at the admin module stuff maybe [17:27:04] that's where we started :) [17:27:05] I had this similar journey for groupadd sillyness [17:27:07] ha [17:27:20] chasemp: we started here https://gerrit.wikimedia.org/r/281918 [17:27:32] both the user and group in question are created by packages, they're not explicit in puppet or belong in admin-module really... [17:27:51] user varnish is created by varnish package, installed by varnish classes. user ganglia is created by ganglia package, installed by ganglia classes [17:28:00] now in role foo, we want to ensure user ganglia is a member of group varnish... [17:28:25] (and ideally, puppetized in a way that future role foo's are not prevented from adding other users to group varnish, or adding other groups to user ganglia) [17:30:42] so we don't need to create teh varnish group really, and we don't need to create teh ganglia user [17:30:48] but we are creating both bblack and ema users [17:30:52] so maybe we could [17:30:58] use an instance of admin::groupmembers [17:31:03] pass in the array you want [17:31:07] well bblack and ema are nice side-features [17:31:10] and use metaparameters to handle the dependency [17:31:12] sorry guys I've gotta go. Thanks chasemp for jumping in! [17:31:16] see you tomorrow :) [17:31:24] the main thing is getting ganglia-user+varnish-group hooked up [17:31:28] cya ema [17:32:13] and probably other software-users too [17:32:18] what you were saying w/ making an exec that is more sane etc [17:32:26] pretty much that is teh approach in groupmembers.pp [17:32:41] catch is it's exclusive so it will nuke any users not defined by it [17:33:01] but since the users in question are just an array you could pass in users you know will be created by packages [17:33:20] yeah and I guess some requires stuff to not do it until after packages are installed [17:33:22] and handle the dependency w/ Package[ganglia] dependency maybe [17:34:00] it's not ideal but it gets you where you want to be maybe [17:34:38] it's arguably better than defining the users as a shadow and depending on them directly [17:34:46] even tho the resource is basically a noop in real terms [17:36:58] right [17:37:00] thanks :) [17:37:42] this almost makes me want to go back down the rabbithole of making a saner shared user/group/auth provider with nss + pam support.... [17:38:03] and then making a puppet provider for user+group that matches it and does all the right things [17:38:20] please do :) [17:38:39] I've always thought there's a big gaping hole between "standard local files" and "LDAP (or worse, ancient NIS/NIS+ stuff)" [17:39:22] something that simply rsyncs what look a lot like the normal file-based user/group/shadow files around securely, and has some trivial stuff layered on top to handle pwchange sync, etc [17:39:26] I started to once years ago [17:39:49] not directly related, but kinda [17:39:49] cool [17:40:15] I once bypassed the issue using pam_auth_radius and freeradius for all the auth things [17:40:23] it was shockingly simple and effective [17:40:28] yeah [17:40:41] basically any solution for the auth part that works for pam and keeps things in network-sync [17:40:57] toss in secure distribution of the basic passwd/group/shadow files via rsync+ssh or whatever [17:41:10] and an nss module that can read them from the alternate directory you sync to [17:41:23] I just haven't seen anyone publish a good standard solution [17:41:54] I think sssd was one that was close? [17:42:31] or it was when the project started, but it kept gaining features? [17:42:55] I never saw it [17:44:47] looks interesting tho [17:45:12] this is fairly accurate description for nslcd and nss ldap: It’s broken, convoluted, and not well documented. Worst, there’s a lot of bad advice floating around the Internet in places like StackOverflow, ServerFault, ExpertsExchange, etc. [17:45:15] ha [17:47:47] hey at least it's not partman [20:18:13] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310391 (10BBlack) From irc conversation in wikimedia-operations w/ @Nuria, @Krinkle, and myself. This is the varnish-level pseudo-code proposed (ignore arbitrary names and con... [20:26:39] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310419 (10Nuria) Looks great, one minor nit: rather than having distinct cookies per feature we can have one weblab cookie that contains all features and bucketing for those.... [20:26:47] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310422 (10BBlack) Other nits and notes: 1. Don't send the cookie to the applayer, just the feature header 2. Validate the cookie's value, clear+reset if invalid 3. Block the f... [21:05:24] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310635 (10BBlack) Better pseudo-code, after more conversation: Data structure (which we update as we add/remove experiments): ``` experiments => { # 100 total bins to use: 0-9... [21:10:20] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310677 (10BBlack) If the `setRequestHeader` bits didn't use else-if, you could have overlapping buckets with multiple features in play too, but this seems simpler for the momen... [21:11:41] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310694 (10BBlack) Another complication that didn't come up in conversation earlier: what about domainnames for these cookies? They'll be getting binned independently for every... [21:14:46] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310716 (10BBlack) Another thought: with the above code, I've intentionally set it so that both sides of the experiment will initially share an empty cache split (they'll have t...