[05:20:52] razzi: reprepro is managed via puppet, see modules/aptrepo/files/distributions-wikimedia, puppet will create the necessary repo component [05:21:04] for importing external repos see https://wikitech.wikimedia.org/wiki/Reprepro#Updating_external_repositories [05:24:12] but if it's just a one off import (and seems to be the case here), using includedeb is also fine, but make sure to download using Secure Apt (i.e. adding the nodesource repo key and run apt-get download nodejs) [08:32:32] moritzm: when you have a minute, I have two quick patches that need your approval: https://gerrit.wikimedia.org/r/c/operations/puppet/+/675812 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/675722 [08:34:46] good morning, having a look in a bit [08:45:37] thanks! [08:55:46] dcaro: both done [08:57:12] thanks! [09:05:22] heh, got a breakdown of CPU/Memory utilization per component on our mw servers. https://grafana-rw.wikimedia.org/d/0VjCCwwGk/mediawiki-server-clusters-utilization?orgId [09:05:42] sigh grafana-rw... just remove -rw if you don't have an account. [09:06:27] it redirected me to it without -rw [09:06:33] well, that's something [09:06:47] mcrouter seems to be wrong btw, /me checking [09:07:11] then if I try to sign-in I get redirected to the same page without -rw over an over and never getting logged in... [09:07:28] sigh grafana-rw indeed [09:07:40] :-( [09:09:09] FWIW the "sign in" link should work and redirect you to the -rw dashboard version _if_ you already have a valid sso session [09:09:38] I have a valid session [09:09:38] and indeed it doesn't quite work yet on SSO session renewal or no session [09:09:54] is "yet" the key word here? [09:09:58] as in there are plans to fix it? [09:10:58] akosiaris: yes, though between me and John we haven't had quite the time to work on it, context is https://phabricator.wikimedia.org/T269272 [09:11:07] ah thanks! [09:11:09] volans: do you mind and/or have time for debugging ? [09:11:44] godog: what can I do for you? getting the list of redirects? [09:13:10] volans: yeah the first few redirects would be super helpful, specifically what I did from https://grafana.wikimedia.org/d/0VjCCwwGk/mediawiki-server-clusters-utilization?orgId is hit "sign in" on bottom left and the -rw redirect worked [09:13:33] hmpf... google package signing key is expired xd (2021-03-31), reprepro is complaining [09:15:20] ah no mcrouter is correct. It's just consuming half a core per server [09:15:22] interesting [09:21:25] godog: define "session renewal" because things are getting weird [09:21:27] moritzm: should I force the reprepro checks? the fact that the google key is expired only affects the kubernetes repos [09:22:02] I was gathering the info, went to idp.w.o and showed I was logged in, then went to the homepage of grafana, hit login again and now is showing me the login form, same if I go now to idp.w.o [09:24:03] dcaro: yeah, feel free to force, but can you please open a Phab task so that the key used for Kubernetes get replaced (just tag it with serviceops)? [09:24:15] ack [09:25:42] volans: mmhh what I had in mind is the case where you are logged in but mod_cas from time to time redirects you to idp.w.o to renew the session/ticket, which turns POST into GET and gets the XHRs from grafana quite confused [09:26:22] I've updated the tasks with my data [09:30:21] volans: thank you! that's helpful [09:31:03] ironically I think is the -rw -> non-rw redirect (for folks that don't have an account) that's sometimes faulty [09:39:48] moritzm: adding the bypass to the google key https://gerrit.wikimedia.org/r/c/operations/puppet/+/676292 [09:40:06] np [13:12:37] moritzm: I messed up with the reprepro forcing, the bang goes at the end, but I found a better solution, packages are double-signed, the other key is not: https://gerrit.wikimedia.org/r/c/operations/puppet/+/676342 [13:13:03] ack, having a look in a bit [13:37:11] why do we have two separate versions of keyholder, one in operations/puppet:modules/keyholder and one in operations/software/keyholder? [13:41:58] dcaro: not sure about that specific key, but I've added the relevent people as reviewers [13:42:06] Majavah: good question :-) [13:42:13] moritzm: ack, thanks [13:46:05] moritzm: found https://phabricator.wikimedia.org/T203003, so looks like operations/software/keyholder is meant to replace the one on puppet [13:48:51] I need keyholder for another project of mine, and while packaging the one on operations/software/keyholder I found out that it depends on a crypto library that has not been updated on upstream since 2014 and no longer available on bullseye for some reason :/ [16:05:25] Just wanted to wish a bunch of folks Happy Work Anniversaries! moritzm, paravoid, bblack, wkandek, _joe_. Congrats! [16:05:53] <3 [16:16:35] Thanks! [16:41:52] I think janis as well :) [16:41:56] started same day as wolfgang [16:41:58] congrats! [17:07:35] Bikeshed time! Wikitech should get a new logo, and y'all SREs are the main voice that I think should be heard. T279087 [17:07:36] T279087: Updated logo for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T279087 [17:27:25] is the half-full flower supposed to be representative of the state of our documentation [17:35:29] I always thought about more as the state of projects, that we are always building on more. But I can see the cynical view too. :) [17:42:51] :D [17:44:32] I thought of it as building (the infrastructure, the code), not really about the docs [17:44:42] for the docs we alreay have bug 1 [17:45:49] which was closed :(( [17:46:37] it will always be open i my heart :-D [17:48:07] * bd808 waits for quiddity to pop up and say "Zanzu!" [17:48:33] https://www.mediawiki.org/wiki/User:Quiddity/Zanzu [17:48:39] <3 [17:49:42] these are the children of bug 1. never say it is forgotten! https://phabricator.wikimedia.org/tag/documentation/ [17:49:45] Grr, that page still isn't in the top10 google results! (The mw page for "Naming Things" is the top result now!) [19:22:18] is there a mythical bug 1? For me, that was https://bugs.launchpad.net/ubuntu/+bug/1 for a long time [19:24:05] It was the founding ethos of that organization and was closed quasi-ceremoniously at one point. For WMF, bug 1 might be more akin to our spread / reach / impact to everyone in the world. But what is bug 1? [19:24:39] balloons: https://phabricator.wikimedia.org/T2001 was #1 in the original bugzilla installation [19:25:10] <3, thank you! [19:25:24] 👍 [19:31:33] Bug 1 is "docs bad" and can never be closed [19:34:10] paravoid: if we want to reflect state of our documentation, we should go with the almost empty one: https://commons.wikimedia.org/wiki/File:Wikitech_variant_1.svg [19:34:50] I like that one [19:35:06] and bd808 thought I was the cynic [19:35:37] it's a bug that T2001 isn't T1 [19:35:37] T2001: [DO NOT USE] Documentation is out of date, incomplete (tracking) [superseded by #Documentation] - https://phabricator.wikimedia.org/T2001 [19:35:38] T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1 [19:37:44] we have the OG cynic: https://bash.toolforge.org/quip/AU7VVnI_6snAnmqnK_yu [19:42:36] It is hard to out cynic Chad :) [19:45:07] <_joe_> I disagree, if it needs to represent the state of our documentation, it should be a half-flower, with half the petals from the new logo, and half from the old [19:45:12] <_joe_> plus some comic sans [19:46:20] I think that the docs on wikitech have been steadily improving over the years. Yes there are stale pages and yes some things are entirely missing, but a lot of people have spent time making good things there. [19:47:07] <_joe_> I agree, that's why I didn't add "and the flower is half on fire, half rotting" [19:47:57] <_joe_> Also, I just realized I spent more than half of my career in tech at the WMF [19:49:02] _joe_: not as poignant as the day Yuvi realized he has spend half his life here, but impressive [19:52:09] I think this year I reach the "30% of my career" mark. It's an interesting metric! [19:59:54] 100% here \o/ (60% of life in Wikimedia) [20:02:13] https://static-bugzilla.wikimedia.org/bug1.html balloons, this is the original bug 1, in all its glory [20:03:57] 👍 [20:04:28] Percentage of life is an interesting way to measure tenure. Heh, I certainly have never thought of it like that! [20:07:58] https://en.wikipedia.org/w/index.php?title=Wikipedia:Bug_reports_and_feature_requests&oldid=20748713 [20:08:05] ^ 2005 bug reports [20:08:32] there must have been a bug that we need a bug tracker [20:09:35] "The wikimedia web server didn't return any response to your request." - THE server - :) [20:11:31] those are just about a year younger than I am, wow [20:21:36] * bd808 does the math: ~31% of pro career and ~17% of life [20:46:23] yeah we're within 1% of each on both metrics I think. This probably implies other similarities :) [20:50:30] Majavah: that makes you a similar age to me [20:52:46] I would have guessed you a good few years older [21:01:24] RhinosF1: guessing ages of people you know in real life is difficult enough, I wouldn't even try online [21:01:42] Majavah: heh [21:02:31] but you're definitely not the first one to think that I'm older than I am in reality :P [21:06:12] Tbh I get told I seem older way too often on irc [21:06:24] I do sound way older on irc [21:07:38] Went to the dentist and got asked when I was born, then the reaction was "oh, so you didn't experience WWII then". lol, I know I look old, but that is 2 generations off :p [21:09:04] That's harsh [21:09:17] or something about education [21:09:40] I had orthodontist today [21:09:40] or maybe they just say that to everyone [21:10:07] "and when were you born?" "2013" "ahh, so you're too young for WWII" [21:10:18] That's a random comment [21:11:03] maybe they'll get WWIII instead :// [21:11:08] I gave up on age guessing games when I found out that an active tech volunteer that I would have sworn was late-20's/early-30's based on my assessment of their work was actually 16. :) [21:11:49] sounds familiar [21:11:59] I always confuse people when I say when I started learning to program we had no trained IT teachers. We taught the teachers. [21:12:04] bd808: those reviewing my Puppet code are often shocked to find I'm old enough that I actually ought to be able to read and write [21:12:25] I would say we didn't have git but college PCs still don't [21:12:33] rzl: well, puppet so you have an excuse. ;) [21:16:56] https://bash.toolforge.org/quip/M0pLj3gB1jz_IcWux5-o [21:19:12] bash is my favourite tool [21:23:11] it was fun to build :) [21:24:20] I've spent hours laughing at its contents [21:24:37] It's how I survive long coach journeys [21:24:46] And not being able to sleep [21:26:34] there are a few that I really wish it had the full context instead of a line or two, https://bash.toolforge.org/quip/M0pLj3gB1jz_IcWux5-o for example [21:26:59] https://bash.toolforge.org/quip/AWaHt9IcfM03vZ1od1Yf [21:27:11] copying things is hard, that's the correct one [21:34:42] Majavah: that was a good one -- https://bots.wmflabs.org/browser/index.php?start=10%2F18%2F2018&end=10%2F18%2F2018&display=%23wikimedia-cloud -- search for "jc86035" and read from there [21:36:34] I take it they left before the actual snark comment quoted? [21:38:11] I assume so yeah. pretty common in that channel to have pop-in !help shouts with a quick escape [21:39:06] I already closed the shell to my raw irc logs or I guess I could confirm that