[09:04:57] rzl: totally agree on fixing it, but I had tried to repro locally running tox with all supported python versions and I was *never* able to repro. So something makes me thing that there is something in the way the CI env is setup that might be related, although I can't be 100% sure of it. [09:05:32] The other aspect is that the failing tool is pylint that is a pure static analyzer, no code gets executed. [10:18:51] godog: I have 2 feedbacks to share, one good one bad :-P [10:20:44] arturo: haha! sure, shoot [10:21:03] which one you prefer first? [10:21:15] hehe [10:22:35] heheh the bad one first [10:23:12] ok, so it seems I still cannot edit this grafana dashboard https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad1 [10:23:48] clicking the sign in button redirects to the same URL. I can further debug the requests in my browser if you want [10:24:28] if I type https://grafana-rw.wikimedia.org/d/000000579/wmcs-openstack-eqiad1 in the URL bar, it redirects to the same but without `-rw` [10:24:52] mhhh that's a sign you are not logged in with sso [10:24:59] ok, let me try [10:25:08] arturo: do you see the 'edit' button? does it fail when saving only? [10:25:19] I don't have the edit button [10:26:26] godog: so now I identified into idp.wikimedia.org. I see same behavior when clicking the sign in button [10:27:21] arturo: ack, does https://grafana.wikimedia.org/login work? i.e. redirect to -rw ? [10:27:44] godog: it does! [10:28:35] and this time, if I type https://grafana-rw.wikimedia.org/d/000000579/wmcs-openstack-eqiad1 it allows me to edit! (so it doesn't redirect back to the RO version) [10:29:49] yeah the grafana-rw -> grafana redirect is the tell tale you are not logged in, the idea is to DTRT when e.g. we're sharing -rw links with folks that can't log in [10:30:18] interesting... If I open a 'private' tab, and try to login, it just redirects me again to non-rw (it goes to /login -> rw -> non-rw, but no user/pass prompt and I end up not being logged in), maybe expected in a private tab though [10:31:18] dcaro: yeah, although grafana.w.o/login should ask you to login with sso [10:31:56] I'm logged in in the non-private tab though, might get confused and think I'm logged in there too [10:32:37] godog: anyway the good feedback now. I debugged the very first issue using logstash. Amazing! [10:32:51] https://phabricator.wikimedia.org/F33931500 [10:33:04] mmhh no that shouldn't happen dcaro, afaik when in private cookies aren't shared [10:33:24] arturo: nice! good times indeed [10:33:31] maybe the sso uses the source ip somehow? [10:33:40] (in any case not an issue) [10:33:50] arturo: make sure to try https://logstash-next.wikimedia.org too if you aren't already [10:34:22] * arturo checking [10:34:25] arturo: thanks for the feedback, always welcome [10:35:14] godog: dashboard aren't shared with the old logstash? [10:35:55] I don't think so [10:36:01] arturo: they get synced from logstash.w.o periodically, so new dashboards might not be there yeah [10:36:13] excellent [10:38:40] thanks! [10:54:33] np! [17:11:58] godog, moritzm: is it known that the "sign in" link from a grafana dashboard doesn't work? [17:12:06] from https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1 when I click "sign in", it goes to https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&forceLogin=true, which 302s to https://grafana.wikimedia.org/login, which 302s to https://grafana-rw.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1, which 302s back to https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1 where I started [17:12:37] rzl: it's called Grafana tour :-P [17:12:59] I can work around by hitting the sign in link from https://grafana.wikimedia.org/?orgId=1 instead [17:13:16] was about to say "grafana-admin" but that is history [17:15:22] rzl: bizarre, signing in and the redirect works for me, now that you have a fresh sso session does it work if you try again ? [17:16:17] it == the original "sign in" link for the puppet dashboard [17:16:49] yep, works fine if I'm already signed into sso [17:17:05] the only thing that doesn't work is getting me to the login prompt [17:19:53] yeah I see what you mean, "sign in" from a dashboard should DTRT the same way that grafana.w.o/login does if you are not logged in [17:20:04] want me to file a task? [17:20:20] yes please, appreciate it! [17:20:29] 👍 [20:37:39] here's a nice one: on our mirrors server where we pull from Ubuntu, Tails and Openstack to provide mirrors.. the first line of the Ubuntu sync script has been "#/bin/dash" forever. Note how it's missing a !. Now this did not matter all this time when cron started it and there is also no issue if you manually run it. Just once you try to convert this to let systemd start it everything fails and the [20:37:45] journal just tells you " Exec format error". Eventually found it and adding the ! does the job. [20:38:23] these are the kinds of things that could also be happening for the backup jobs and why we have to babysit each one [20:58:31] ouch :-)