[05:19:02] <marostegui>	 thanks paladox and thcipriani
[05:19:48] <vgutierrez>	 now marostegui is happy again with his github stats going higher and higher
[05:20:37] <marostegui>	 no, I just use github to look at the code :)
[05:21:32] <vgutierrez>	 yeah, I do the same
[05:21:57] <vgutierrez>	 the code browser is slightly better than gitiles
[06:27:58] <_joe_>	 vgutierrez: his stats are bound to go down now
[06:28:19] <vgutierrez>	 in August because he's spanish?
[06:28:25] <vgutierrez>	 }:)
[06:28:41] <vgutierrez>	 (jokes aside.. dbctl)
[07:29:20] <marostegui>	 thcipriani paladox I have re-opened https://phabricator.wikimedia.org/T229945 as I believe replication isn't working properly yet
[07:33:04] <marostegui>	 cdanis volans would be hard to allow "-w" to be an alias for set-weight? same as we have -p %
[07:38:23] <volans>	 I don't see why not :)
[07:38:47] <marostegui>	 You don't see why not it won't be hard? :p
[07:38:59] <marostegui>	 volans: Do you want me to create a task with the dbctl tag for discussion?
[07:40:07] <marostegui>	 mmmm install2002 is short on disk space
[07:40:11] <marostegui>	 I am going to take a look
[07:44:33] <marostegui>	 heh: https://grafana.wikimedia.org/d/000000377/host-overview?refresh=5m&panelId=12&fullscreen&orgId=1&from=now-24h&to=now&var-server=install2002&var-datasource=codfw%20prometheus%2Fops&var-cluster=misc
[07:44:38] <marostegui>	 Going to create a task for this
[07:46:21] <volans>	 marostegui: nah, just send a patch ;) https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/conftool/+/master/conftool/extensions/dbconfig/__init__.py#66
[07:46:39] <volans>	 ah wait a sec
[07:46:47] <volans>	 set-weight is a command not an option
[07:47:07] <marostegui>	 Yep
[07:47:18] <volans>	 sorry still waking up :)
[07:47:30] <marostegui>	 I am fine creating a task just for this to be on the backlog
[07:47:35] <marostegui>	 Doesn't need to be done this week
[07:47:45] <volans>	 but what would be the UI then?
[07:48:32] <volans>	 the general one is: instance LABEL COMMAND [OPTIONS]
[07:48:41] <volans>	 so we need a COMMAND for set-weight
[07:49:07] <marostegui>	 volans: yeah, I don't know really, I just thought about it when I was running
[07:49:17] <marostegui>	 That we do -p % so it would be nice to do -w 100
[07:49:45] <volans>	 but -p is an option to pool
[07:50:15] <marostegui>	 yeah, true
[07:50:40] <marostegui>	 I think I need to stop thinking about work when I run!
[07:50:46] <volans>	 rotfl
[07:50:57] <marostegui>	 volans: don't know, it just popped up
[07:51:11] <marostegui>	 Hence my question about how hard it would be :)
[07:51:29] <marostegui>	 but yeah, it would change many things indeed
[07:51:55] <marostegui>	 just give it a thought
[07:52:12] <marostegui>	 again, if it is easier, I am happy to create a task and then we can decline or whatever
[07:52:45] <volans>	 sure feel free but I'm wondering how to keep the functionality of --section and --group, that seems to require set-weight to be a command with its own options
[07:53:04] <volans>	 because then -w what would be? which section? which group?
[07:54:03] <marostegui>	 good questions, I don't know :)
[07:54:27] <volans>	 let me ask you another one... what do you see "wrong" or hard to use in the current UI?
[07:54:49] <marostegui>	 So basically dbctl instance db1100 -p 50 feels natural
[07:54:55] <marostegui>	 And dbctl instance db1100 -w 300
[07:54:59] <marostegui>	 Would also feel natural
[07:55:16] <volans>	 db1100 pool -p 50 ;)
[07:55:46] <marostegui>	 yeah
[07:55:57] <volans>	 so db1100 WHAT -w 300?
[07:56:00] <marostegui>	 I guess it is a matter of getting used to it :)
[07:56:24] <marostegui>	 nah, in that case set-weight -w 500 wouldn't make sense XD
[07:56:33] <volans>	 eh
[07:56:41] <volans>	 that's why I asked WHAT :D
[07:56:50] <marostegui>	 ah
[07:57:03] <marostegui>	 then dbctl instance db0000 pool -w 500
[07:57:45] <marostegui>	 volans: let's forget it for now, I haven't played that much with weights yet
[07:57:52] <marostegui>	 mostly pool,repool and edit
[07:57:55] <volans>	 but that's mixing pool with weight, feels a bit weird to me
[07:58:30] <volans>	 sure, play more and let us know :D
[07:58:34] <marostegui>	 will do
[07:58:36] <marostegui>	 thanks! :)
[07:58:45] <marostegui>	 now, time to do an edit XD
[08:05:54] <_joe_>	 not mixing operations on weights and pooled state was a deliberate decision
[08:06:13] <_joe_>	 it seemed easier to me if all the actions you can do from cli are individual
[08:06:34] <_joe_>	 for more complex changes you should use edit, in my original idea :)
[08:12:38] <_joe_>	 observability people: I have a software exposing prometheus-compatible metrics at /stats, not /metrics
[08:13:01] <_joe_>	 how can I collect those in our puppet abstraction? something specific I need to change?
[08:15:28] <marostegui>	 _joe_: Yeah, for complex changes I am using edit, it is easier
[08:15:42] <marostegui>	 _joe_: I am not fluent enough with set-weight as I am not using it that much
[08:15:56] <marostegui>	 We generally don't change weights that often, we mostly repool/pool
[08:16:05] <_joe_>	 so you can do the warmup
[08:16:19] <marostegui>	 Yeah, the warm up I am doing it with -p
[08:16:22] <_joe_>	 where you don't change weights but you change the dampening factor on the original weight
[08:16:32] <marostegui>	 yeah, that we do use a lot
[08:16:43] <marostegui>	 And I have been using it with dbctl already a bunch of times
[08:16:56] <marostegui>	 And it is _awesome_ not to have to do a commit per action
[08:17:03] <godog>	 _joe_: you can change the endpoint to scrape in the prometheus job configuration
[08:17:21] <marostegui>	 commit as in MW commit
[08:41:46] <marostegui>	 Someone bored enough to check this IP for me? :) https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/528713/
[08:44:46] <vgutierrez>	 right now I'm https://xkcd.com/303/
[08:45:07] <marostegui>	 hahaha
[08:45:08] <volans>	 {done}
[08:45:13] <marostegui>	 thanks! :)
[08:45:29] <vgutierrez>	 damn latency
[08:45:31] <marostegui>	 Oh, two +1!!!
[08:45:38] <marostegui>	 that makes it a...+2!
[08:45:51] <marostegui>	 thanks guys
[08:45:54] <vgutierrez>	 <3
[08:46:03] <volans>	 marostegui: nope, remember, I can only give 0.8 :-P
[08:46:09] <marostegui>	 hahaha
[08:46:56] <marostegui>	 I thought the conversation in dublin with BOTH directors asking you to be able to do full +1 would change your mind
[08:47:04] <marostegui>	 But you are not sinking with the pressure
[08:47:06] <marostegui>	 damn!
[08:47:57] <volans>	 because they didn't implement it in gerrit :-P
[08:48:23] <marostegui>	 paravoid!!!! ^
[08:48:44] * marostegui runs
[09:55:13] <apergos>	 ah rats, I missed the call. so late in the day though :-P :-D
[11:11:57] <paladox>	 Hmm
[11:11:58] <paladox>	 Gerrit down
[11:12:22] <marostegui>	 check -operations
[11:12:54] <apergos>	 it's back already after restart
[11:13:59] <paladox>	 Thanks
[11:36:57] <paladox>	 marostegui the replication plugin is missing (https://gerrit.wikimedia.org/r/#/admin/plugins/ isn't showing it as installed and running the ssh command is failing)
[11:38:48] <marostegui>	 paladox: can you report that on the task I re-opened?
[11:38:55] <paladox>	 yup
[11:38:58] <marostegui>	 Probably that's the cause
[11:38:59] <marostegui>	 Thanks :)
[13:50:34] <jbond42>	 hi all, FYI i plan to disable puppet fleet wide so i can restart puppetdb.  should only be disabled for a a short window and plan to start at 14:00 UTC
[14:45:42] <marostegui>	 shdubsh: I added you as a subscriber for https://phabricator.wikimedia.org/T229997 as I saw you were on clinic duty
[14:46:18] <cdanis>	 systemd::timer::job has something silly happening wrt: logging dirs
[14:47:24] <cdanis>	 both it and systemd::syslog want to append their $title to the base dir (usually /var/log)
[14:47:37] <shdubsh>	 marostegui: thanks for the heads up!
[14:47:39] <cdanis>	 so you get, for instance, /var/log/fetch_dbconfig/fetch_dbconfig/fetch_dbconfig.log
[14:48:26] <cdanis>	 also, nothing ever defines a directory resource for /var/log/fetch_dbconfig, so Puppet fails
[14:50:07] <cdanis>	 it looks like most existing users work around this by either setting logging_enabled=>false, or by overriding logging_basedir
[14:57:45] <_joe_>	 cdanis: yeah
[14:57:58] <_joe_>	 cdanis: that was a consequence of how the job was setup initially
[14:58:03] <_joe_>	 I never came around to fix that
[14:58:08] <_joe_>	 just not having time
[14:58:30] <_joe_>	 let's make a trade
[14:58:47] <_joe_>	 I look into fixing it, and you explain to me what's wrong with
[14:58:53] <_joe_>	 topk(5,histogram_quantile(0.95, sum(rate(mediawiki_http_requests_duration_bucket{cluster="$cluster",handler="proxy:unix:/run/php/fpm-www.sock|fcgi://localhost",method="$method", code="$code"}[2m])) by (le,instance,code)))
[14:59:03] <_joe_>	 why is this not showing me 5 entries only??
[14:59:16] <cdanis>	 it's showing you the top 5 at every instant it is evaluated
[14:59:32] <cdanis>	 https://stackoverflow.com/questions/38783424/prometheus-topk-returns-more-results-than-expected
[14:59:35] <cdanis>	 https://www.robustperception.io/graph-top-n-time-series-in-grafana
[14:59:53] <cdanis>	 this trade seems pretty good for me 🙃
[15:01:49] <_joe_>	 sadly we don't have grafana 5.3
[15:01:58] <_joe_>	 are you sure the trade is that good?
[15:02:12] <cdanis>	 we have grafana 5.4.2
[15:02:12] <_joe_>	 oh we do
[15:02:20] <_joe_>	 so why does $__range_s not work
[15:02:22] <cdanis>	 I've been meaning to upgrade to 6.x
[15:02:39] <_joe_>	 oh damn
[15:02:48] <_joe_>	 I hate editing queries in that box
[16:25:08] <cdanis>	 jaufrecht: if you don't mind, could you do: /msg chanserv access #wikimedia-sre add cdanis
[16:25:12] <cdanis>	 that would let me set the topic in here
[16:25:47] <jaufrecht>	  /msg chanserv access #wikimedia-sre add cdanis
[16:26:19] <cdanis>	 but without the space at the front ;)
[16:27:12] <jaufrecht>	 Try now
[16:28:02] <cdanis>	 thank you !
[16:31:18] <chaomodus>	 yay
[23:43:10] <mutante>	 "Description: CPU 1 machine check error detected.
[23:43:14] <mutante>	 always nice