[07:19:55] XioNoX: topranks: I'd like to flip dumps https traffic to the new load balancer service, do you have any concerns or objections about doing it now? [07:25:50] greetings [08:08:45] taavi: no fire away let’s hope it goes smooth [08:09:39] great, doing [14:29:48] taavi: wanted to confirm that https://phabricator.wikimedia.org/T422046 is a builds-api use case you are interested in. [14:31:15] Damian made a patch but I can’t merge until at least someone else is strongly interested in it. [14:55:23] andrewbogott: did you manage to test if by any chance that change would fix the logs? [14:55:29] * volans curious [14:55:46] I haven't made it back to that task yet, will let you know when I do [14:56:19] ack [15:23:38] doesn't seem to help. I'm not 100% sure that logging.conf is getting used with the rest of the config [16:22:31] I was exploring our S3-compatible object storage stuff and found that `s3cmd ws-create s3://html` raises an `ERROR: S3 error: 405 (MethodNotAllowed)` failure. I couldn't figure out if we have deliberately disabled the static website functionality, accidentally not enabled it, or if Ceph radosgw doesn't support this no matter what. [16:23:42] as I understand it, the ws-create change would make a URL like https://object.eqiad1.wikimediacloud.org/84baa6f9fe8d41afb4b7ca99891161f3:etherpads/ serve up an index.html object if present rather than giving the bucket index xml response. [16:25:45] bd808: have you checked our setup does not serve index.html from a 'normal' bucket? [16:26:50] it did not when I uploaded an index.html to that bucket, but I can try that again to give a reproduction or prove I did something wrong. :) hang on... [16:29:07] taavi: https://object.eqiad1.wikimediacloud.org/84baa6f9fe8d41afb4b7ca99891161f3:etherpads/index.html works, and https://object.eqiad1.wikimediacloud.org/84baa6f9fe8d41afb4b7ca99891161f3:etherpads/ is still the xml listing. [16:46:14] bottom of https://ceph-users.ceph.narkive.com/gyevg9AB/rgw-s3website-methodnotallowed-on-jewel-10-2-3 implies that there's a flag we need to flip. But so far I do not see such a flag in the radosgw config docs... [16:46:37] oh wait, maybe I see it... [16:47:53] bd808: how hard would it be for you to reproduce this in codfw1dev? [16:57:41] I'm confused by the 'rgw_dns_s3website_name' which makes it seem like the static website config supports hosting /one/ website rather than hosting n websites... [17:00:46] andrewbogott: It should be easy if the gateway is all setup in codfw and I can get my hands on ec2 credentials for a bucket. I basically was just following things from https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#S3_API, especially the "s3cmd example" collapsed section, and then playing with other s3cmd functions. [17:01:11] T422958 [17:01:14] assuming you can log into horizon there, everything else should be the same. [17:01:15] T422958: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958 [17:02:18] Getting that command to work might be as simple as https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270497 but I am confused about the idea of a global rather than per-container site name [17:03:50] here is an example of why I think that command may be pointless for a public cloud: https://www.ibm.com/docs/en/storage-ceph/8.0.0?topic=hosting-static-web-gateway-setup [17:05:21] I was expecting the URLs to stay something like https://object.eqiad1.wikimediacloud.org/$PROJECT_ID:$BUCKET/ in our case. [17:07:22] yeah, it might work like that, but if so those settings are strangely named [17:08:12] openstack config strangely named... madness! ;) [17:08:24] the docs might just be saying that we need a second domain like 'website.eqiad1.wikimedia.cloud.org' to get the index resolution... [17:08:36] Oh, this is ceph actually, a totally different team naming things strangely :) [17:09:36] Is this a feature that you imagine would be useful for you/others? If so we can poke at it more or I can make myself a ticket. [17:12:44] hmmm... needing a domain to serve the indirect default index content over does make some sense. [17:13:57] Yeah, it avoids regular s3 lookups doing surprising things [17:16:29] I can try to write up the use case(s) for sure. The main place I remember it coming up before was discussions about how the next gen zuul setup we are slowly working towards will work. It will be using object storage for job artifacts (logs, reports) and upstream expects that folks can get the artifacts back easily. [17:17:15] We have direct http access now though, just not default index support so it's not like there is a huge missing function. [17:17:37] ok [17:18:01] it's easy for me to mock up a test, I'm just reluctant to twiddle the ceph settings in eqiad1 until we know that we're getting somewhere [17:19:06] yeah, breaking things unnecessarily is never fun. Thanks for jumping to look into it. I was mostly curious if we knew that it didn't work first. [17:21:21] definitely seems to be disabled by default, based on various forum questions [21:49:06] andrewbogott: I made T423194. It isn't super compelling yet, so don't feel bad about ignoring it. We have the idea though and maybe can figure out if it is useful. [21:49:07] T423194: Consider enabling static website support in the WMCS radosgw implementation of S3 buckets - https://phabricator.wikimedia.org/T423194