[09:20:09] <_joe_>	 mutante: giving complete access to logs needs admin / sudo privileges, and it's a good thing.
[11:35:46] <dcaro>	 moritzm: ryankemper: the hiera settings for that project on openstack enc: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/master/wikidata-query/
[13:23:54] <jbond42>	 hi all i notice a few people set or have functions to set http_proxy in there profile.  To avoid everyone having to add theses exports/functions to there individual profiles i wonder if its worth thinking about this as global option/config as such i have created https://phabricator.wikimedia.org/T278315 to gather ideas/issues
[13:40:30] <volans>	 jbond42: thx, replied, playing the devil's advocate
[13:43:54] <jbond42>	 volans: thanks, also worth mentioning i dont have this in my profile currently so im very keen to here advocats both devil and otherwise before making any changes :)
[13:44:37] <volans>	 {same}
[13:47:02] <kormat>	 i have Opinions
[14:15:14] <Amir1>	 herron: hey, mailman2 is on python2. Right?
[14:15:27] <Amir1>	 just double checking
[14:16:05] <moritzm>	 it is
[14:17:18] <Amir1>	 this software is beyond rotten. Isn't it
[14:17:30] <Reedy>	 Amir1: scap runs on python 2 :P
[14:17:53] <Amir1>	 Reedy: but it doesn't store passwords in plain text
[14:17:59] <Amir1>	 while that's a really low bar
[14:18:03] <Reedy>	 lol
[14:18:06] <volans>	 lol
[14:18:24] <Amir1>	 trying to list mailman2 issues so far came up with this "Pretty old user interface (specially for admins and moderators), storing passwords in plain text, lack of any database (all are files), pretty old code, lack of ability to search in archives or send email from web interface, running on EOL python (python2), encoding issues with non-Latin languages are some of many many issues the current software has."
[14:19:15] <jynus>	 +"archive is not easily editable to fix privacy issues"
[14:19:18] <rzl>	 "unable to expunge a message from the archive without breaking all other archive links" is hard to state compactly but is a pretty key privacy issue
[14:19:25] <rzl>	 ahaha beaten, thanks jynus
[14:19:34] <jynus>	 we thought the same, rzl :-)
[14:19:35] <Amir1>	 Thanks!
[14:19:40] <jynus>	 I guess you read my last email
[14:19:50] <rzl>	 (n.b. I'm not positive that's fixed in mailman 3, I imagine so but haven't checked)
[14:20:02] <moritzm>	 + dropped from Debian bullseye, no upgrade path other than mailman3
[14:20:19] <jynus>	 rzl, I havent checked, but given it is db-based and not render-based, I am confident it will be
[14:20:29] <rzl>	 yeah, that's what I was thinking too
[14:20:54] <Amir1>	 added
[14:21:01] <Amir1>	 Thanks
[14:22:16] <Majavah>	 are you drafting an announcement or something?
[14:24:45] <jynus>	 Amir1, on the negative, I am worried about scaling issues- we had some for db-backed mails in OTRS already: https://phabricator.wikimedia.org/T138915 Hopefuly testing it demonstrates it will not create additional problems
[14:27:20] <Amir1>	 Majavah: yup, lists-next.wikimedia.org will be live soon ^^
[14:27:52] <Amir1>	 jynus: yeah, I know. I wish I had some way to tackle it, maybe some consistent sharding?
[14:28:15] <jynus>	 Amir1, no need to tackle it, just to observe for now/have it into account
[14:28:23] <Amir1>	 yeah
[14:28:38] <jynus>	 maybe it is an irratial fear
[14:29:38] <Amir1>	 I'm sure we can drop archive of some mailing lists. Like these automated ones that send email for phabricator changes
[15:50:51] <dancy>	 Hi SRE folks! It's that time again..  another logspam mod in need of a +2:  https://gerrit.wikimedia.org/r/c/operations/puppet/+/674387
[15:54:32] * legoktm looks
[15:55:50] <legoktm>	 dancy: merged, do you want me to run puppet manually on a specific host?
[15:56:17] <dancy>	 Sure. mwlog1001
[15:56:26] <dancy>	 Thanks alot.
[15:58:27] <legoktm>	 done
[17:27:16] <mutante>	 _joe_: ok, ACK, will suggest do add sudo privs to run journalctl on an admin group
[18:19:55] <legoktm>	 chaomodus: hi, can you help me get a service IP for lists-next.wikimedia.org? (https://gerrit.wikimedia.org/r/c/operations/dns/+/673638)
[18:20:30] <chaomodus>	 legoktm: certainly
[18:22:15] <chaomodus>	 does it need to be separate, i see you have the lists ip in there already
[18:22:20] <chaomodus>	 in that patch
[18:22:29] <chaomodus>	 which, i think that's currently the preferred thing to do
[18:23:00] <legoktm>	 well I'm trying to copy the setup of lists.wikimedia.org, which has a service IP
[18:23:14] <chaomodus>	 ahh
[18:23:20] <chaomodus>	 what machien will this be on then?
[18:23:57] <legoktm>	 lists1002.wikimedia.org
[18:25:50] <chaomodus>	 hm does it need to be different from the 'real' ip for the machine?
[18:26:12] <chaomodus>	 i think the current best practice is to make service aliases be cnames that point at the real name
[18:26:49] <mutante>	 hey chaomodus, so one part of this is _exactly_ like what you did for me yesterday
[18:26:59] <mutante>	 a VM with 2 public IPs, same thing
[18:27:14] <chaomodus>	 yah I see that but in this case it seems unecessary to have an extra ip?
[18:27:15] <mutante>	 it's just that additionally this also has MX and TXT records to deal with
[18:27:21] <chaomodus>	 mmh
[18:27:27] <mutante>	 so this problem can be separated
[18:27:35] <mutante>	 into part a "same as the other day"
[18:28:01] <mutante>	 and part b "where do MX and TXT records go currently, should it still be a DNS repo change"
[18:28:17] <legoktm>	 tbh I'm not sure if a cname wouldn't work, but I would like to keep it as similar to lists.wm.o as possible just to reduce variance in unrelated things
[18:28:54] <chaomodus>	 interesting, could it just be an extra name that points to the same ip? I'm wondering why lists.wm.o has an extra IP I guess
[18:30:08] <legoktm>	 the only thing I can think of is that mail servers are typically given antispam reputation based on IP, and using a service IP allows us to move it to a new server without losing that reputation?
[18:30:09] <mutante>	 _not_ adding the second IP as before is potentially making it harder than doing it
[18:31:03] <legoktm>	 I can commit to opening a ticket asking why lists.wm.o has an extra IP to see if we can get rid of it :)
[18:31:09] <mutante>	 it's a mail service.. yes.. that kind of thing and maybe ACLs
[18:31:16] <chaomodus>	 interesting
[18:32:04] <mutante>	 also if you ask traffic then CNAMES are almost always technicaly bad for $reaons
[18:32:20] <chaomodus>	 yah
[18:32:20] <mutante>	 but.. dunno, ticket seems righte way
[18:32:29] <chaomodus>	 i realize we do service ips as just separate ips a lot otoo
[18:32:31] <chaomodus>	 too
[18:32:40] <chaomodus>	 or as extra names for the same ip
[18:32:53] <chaomodus>	 i guess the later is more common and accepted
[18:33:01] <mutante>	 also netops might prefer it if they can separate public traffic from where we ssh to it
[18:37:32] <mutante>	 legoktm: confirmed this, when we did the last migration it included the step https://gerrit.wikimedia.org/r/c/operations/dns/+/233642/3/templates/wikimedia.org
[18:37:53] <mutante>	 so "switch service IP from one host to another" is how we want to move it again
[18:38:06] <mutante>	 in the future.. I would assume at least
[18:38:31] <chaomodus>	 yep okay
[18:38:49] <chaomodus>	 2620:0:861:1:208:80:154:16 and 208.80.154.16
[18:39:06] <chaomodus>	 when dns generation is run those a recorlds should show up as lists-next.wikimedia.org
[18:39:34] * legoktm runs the cookbook
[18:40:32] <mutante>	 here's the example how you can add it to the NIC with puppet and IP in Hiera:  https://gerrit.wikimedia.org/r/c/operations/puppet/+/674446
[18:42:39] <legoktm>	 chaomodus: there's a typo, "lists-next.wikiwedia.org."
[18:42:56] <chaomodus>	 ah :)
[18:43:04] <chaomodus>	 apologies
[18:43:05] <mutante>	 we probably own that domain :)
[18:43:50] <chaomodus>	 probably :) okay fixed
[18:45:35] <legoktm>	 ty :)
[18:51:36] <legoktm>	 cookbook finished
[18:51:53] <legoktm>	 chaomodus: just to make sure, I don't need to add an entry like https://gerrit.wikimedia.org/g/operations/dns/+/5a9207990f43596e1fc5c6c162e959b5942c8812/templates/154.80.208.in-addr.arpa#15 correct?
[18:52:17] <chaomodus>	 that's correct
[18:52:33] <legoktm>	 ok, thanks for all your help!
[18:57:33] <chaomodus>	 no worries glad to help :)
[20:36:20] <legoktm>	 chaomodus, mutante: if you have a moment, now that I switched in the service IP in https://gerrit.wikimedia.org/r/c/operations/dns/+/673638, CI is failing with "error: Name 'lists-next.wikimedia.org.': All TTLs for A records at the same name should agree (using 300)"
[20:36:37] <legoktm>	 as far as I can tell, I'm only adding one A record...
[20:37:59] <mutante>	 I would expect that change to not have a A/AAAA now.
[20:38:10] <mutante>	 Since that has been added by the DNS cookbook meanwhile
[20:38:38] <legoktm>	 oh, duh
[20:38:41] <chaomodus>	 :)
[20:38:45] <mutante>	 lists-next.wikimedia.org has address 208.80.154.16
[20:38:48] <mutante>	 it's already there
[20:39:38] <legoktm>	 thanks (and thanks netbox!)
[20:41:04] <legoktm>	 and now jenkins likes it ^.^
[21:33:28] <andrewbogott>	 legoktm: I merged your patch
[21:33:37] <legoktm>	 thanks :)
[22:03:50] <legoktm>	 aw crap, sorry about the cron spam
[22:04:22] <legoktm>	 if I could get a sanity check on https://gerrit.wikimedia.org/r/674724 - adjusting the m5-master firewall, I'd appreciate that