[06:28:55] I am going to switch phabricator master, expect around 1 minute of RO time https://phabricator.wikimedia.org/T398818 [06:32:00] Done, RO was 30 seconds [06:55:06] thank you! [07:22:35] hnowlan nemo-yiannis can we revert or dial down the mobileapps 10k logs/s "outbound request" firehose? I believe that's https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1167160 [07:22:59] we are rate limiting ingestion on the logstash side with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167287 [07:23:05] I think hnowlan turned it off yesterday [07:23:08] let me revert [07:23:19] thank you nemo-yiannis [07:24:46] unfortunately codfw kafka consumers can't keep up with the load of consuming from eqiad [07:24:59] otherwise the rate limit on the logstash side would be enough [07:26:25] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1167453 [07:26:44] LGTM thank you nemo-yiannis [07:28:30] thanks godog, waiting for CI and will deploy [07:28:59] cheers [08:00:13] FYI; I need to reboot puppetserver1001, Puppet will be disabled on eqiad and the edges for about 15 minutes starting in a few minutes (no puppet merges will be possible during that period either) [08:02:47] actually, no. I'll wait until the current mediawiki deployment window has completed [08:14:25] I will be rolling out a minor Netbox update in a few minutes. See: https://phabricator.wikimedia.org/T397300 [09:04:32] FYI; I need to reboot puppetserver1001, Puppet will be disabled on eqiad and the edges for about 15 minutes starting in a few minutes (no puppet merges will be possible during that period either) [09:20:40] reboot is complete and Puppet has been re-enabled [11:13:02] GitLab needs a short maintenance restart at 12:00 UTC [12:09:03] GitLab maintenance done [17:54:47] does config-master have an API route? Like if I wanted to get the output of https://config-master.wikimedia.org/pybal/eqiad/wdqs without the comments, is there a way to do that? I tried adding `'accept: application/json'` to my curl cmd, but that didn't do it [17:58:13] from inside production you can query etcd directly, but maybe that doesn't help? [17:58:52] it depends what's needed for indeed [17:59:13] also I think the etcd service owner would prefer to keep the number of clients connecting to ectd small and known ;) [18:05:52] inflatador: the machinery behind config-master is quite simplistic, those generated files are the same confd templates used to configure pybal [18:07:15] modules/profile/manifests/configmaster.pp -> modules/pybal/manifests/web.pp -> modules/pybal/manifests/conf_file.pp [18:10:14] oh and I forgot, if you can live with the data being a few minutes stale, you can use this: curl -s https://config-master.wikimedia.org/pools.json | jq .eqiad.wdqs [18:43:50] thanks volans and cdanis , stale is fine. I'll check out pools.json [18:45:37] FWiW it's a hacky way to figure out which cirrus hosts are in which cluster without having to consult puppet or the opensearch API. We have 3 clusters, 1 is on all hosts, the other 2 are on a subset of hosts [18:46:02] it's gross and we hope to do something about it soon ;) [18:51:30] in requestctl, can a site pattern have more than one "site"? More like multiple virtual hosts per site. Example: the "site" phabricator consists of phabricator.wikimedia.org AND phab.wmfusercontent.org vhosts. For some rules I am tempted to have a single site pattern that contains them both. But I see nobody else has done this before and in the web UI form there is just a "Header" field and [18:51:36] doesn't really say if I can have a logical OR or | in there. If that is considered an anti-pattern then it's also ok but I have to edit each existing action to add 2 site patterns with an OR there. [18:59:19] inflatador: an option is to get it from confctl from a cumin host, for example: confctl select 'dc=eqiad,cluster=wdqs,service=wdqs' get [19:02:21] mutante: requestctl works with regexes so yes you can match multiple Host headers with one regex OR have rules that match multiple site patterns [19:02:31] an example is any-wikipedia [19:04:57] yeah, and patterns aren't compose-able, unfortunately; only actions can do that [19:05:04] (although I've been mulling over changing that possibly) [19:05:57] ACK, we can't use cumin since not everyone has access, but pools.json + a little jq massage will work ;) [19:10:02] thank you, volans and cdanis, I see. [19:19:22] I am adding the OR to my actions and stick to one site per site pattern. Slightly prefer over using regex. Harder to make mistakes.