[09:22:50] godog: you happy for me to m,erge yours [09:23:10] jbond42: sure [09:23:11] thanks [09:23:23] np, merging now [09:23:24] heehe was about to ping you jbond42 for the lock, good timing [09:23:29] :) [09:23:38] done [09:44:37] hi all just a heads up that i plan to start the CA change in 15 minutes which will mean disabling puppet for some time [09:48:26] * volans hands over some 🧯 🧯 🧯 to jbond42 and run for cover :-P [09:48:40] :) [11:39:41] This new CA has now been distributed [11:45:29] \o/ [11:48:13] great! [12:05:50] \o/ [14:12:31] <_joe_> and I still see the site! [16:16:03] robh: do you have a link of hw providers (I guess dell or hp) to check available ssd models? (I don't want to buy anything, just see what capacities are available and its aproximate prices) [16:16:26] I don't have anything with prices nope [16:16:33] Since they vendor won't furnish such a list [16:16:33] ok, and without them? [16:16:39] i do have a list of SSD optoins with dell though [16:16:44] a pdf of dell ssd options =] [16:16:49] ok, that would work [16:16:54] HP not so much they just toss in what i ask [16:16:54] could you share them [16:16:57] ill email it righ tnow to ya [16:17:00] thanks! [16:17:22] I think ssd capacities change faster than I realize [16:23:10] jynus: don't forget to look at nvmes too [16:23:18] or HHHL PCIe cards [16:23:26] if I wanted performance, sure [16:23:30] with high-end SSD-like things these days, often the interface is the bottleneck [16:23:33] but they are not hotplug [16:23:49] yeah, they're not :) [16:24:10] for dbs we want something in between hds and pci persistent memory [16:24:18] they could be in theory (Linux supports PCIe hotplug), but I doubt the physical/hardware part in practice on our dells :) [16:24:22] in performance and capacity [16:24:27] but hotplug is a must [16:25:06] yeah, some pci version (or maybe since the beginning) theortically suports hotplug, but board/software/etc. rarely supports it [16:25:49] so you're stuck with slightly-slower interfaces [16:26:14] they do make all the latest fancy things in SATA form too [16:26:15] well most of them [16:26:54] can we hotplu U.2 form factor, or only traditional SATA, on the dells? [16:26:57] *hotplug [16:27:22] https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/optane-dc-ssd-series/optane-dc-d4800x-series/d4800x-1-5tb-2-5-inch.html [16:27:33] ^ the optanes only come in U.2, not legacy SATA stuff [16:28:11] (optane has the best latency AFAIK, but not necessarily better IOPS throughput than the best non-optane competitors) [16:41:26] question: suppose I have multiple commits under review. on commit 2, jenkins failed but in commit 3, jenkins passed, fixing the issue in commit 2. [16:41:36] in such a case, should I manually set +2 and then submitt commit 2? [16:42:14] in this case, I am both the committer and the owner of the repository but I did have one external review [16:44:23] is there no way to fix commit 2 to not trip the failure? [16:44:30] (sometimes that's the case) [16:46:27] yeah "E303 too many blank lines (3)10:21:36 1 E303 too many blank lines (3)" it's an easy fix but with commit 3, I already did that so I am not sure if I should go back and fix 2 and then have a conflict (?) for 3? [16:46:51] if you fix the lines in the same way a rebase will fix everything [16:47:04] just use rebase --interactive SHA1 of commit 1 [16:47:07] the reason I am having this problem is because the CI was added after I had pushed 3 commits :) [16:47:09] mark commit 2 as edit [16:47:34] fix the line in the same way they are in commit 3 and you should not have any conflict [16:47:46] ok let me do that. I am glad I didn't manually set +2 on commit 2 [16:47:53] (before asking here) [16:50:34] i recently added notes on how to manuly rebase on a parent her https://www.mediawiki.org/wiki/Gerrit/Advanced_usage#Manually_rebase_(on_parent) (althugh vola.ns method sounded a bit simpler) [16:51:02] "git rebase -i" is something I type like 7000 times a day heh [16:51:50] It's wonderful. [16:53:27] I have a bunch of aliases for this [16:53:41] r = rebase [16:53:41] ri = rebase -i [16:53:41] rc = rebase --continue [16:53:59] amend = commit --amend --no-edit [16:54:08] my braindamage of never customizing my shell extends to git aliases too [16:54:24] I don't want to become dependent on them and then my fingers fail me when I'm in some other environment and don't have them :) [16:54:45] I've them in prod too :D but yeah it's painful when you're used to and not have them [16:55:02] actually I should sync my prod ones with my local ones, has been a long time [16:55:16] heh, I meant to automate that kind of sync for my setup, and then never did [16:58:00] lots of alerts, restbase, mwapi,... [16:58:22] recommendation_api [17:56:45] hiya jbond42, do I need to change the puppet CA as is in helm charts? [17:57:32] e.g. [17:57:32] https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/services/eqiad/eventgate-logging-external/values.yaml#L24-L25 [17:57:50] as it is* [17:59:12] loking [18:03:08] ottomata: yes that one will need to be updated as well. alternativly (and im not familure with helm so this may be a none starter) you could just use the file located at /etc/ssl/certs/Puppet_Internal_CA.pem [18:12:41] switched Phabricator to new backend on buster..finally done.. then it turns out new server is set to use legacy IDE mode for the drive instead of AHCI in BIOS, making it slower. when you go to BIOS and change that "DATA LOSS will occur"..so we have to switch back..reimage and switch a third time. as godog would say: "sad_trombone.wav" ;P [18:15:32] ottomata: i have created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/554584 but im not sure of the impact of merging. i.e. services restarting etc. added you and alex as reviewer pleas add anyone elses how may know [18:17:46] jbond42: thanks, in this case the service isn't used yet, so we can just do it [18:18:08] i don't think that the puppet deployed file exists in the k8s pod [18:18:21] ty, will merge that today [18:18:22] oh probably not [18:18:44] great thanks and please let me know if there are any issues or if you think of anything else like this [18:18:47] jbond42: so this ca is used to validate the certs used by the kafka brokers [18:18:57] those certs are signed by the puppet ca [18:19:01] will that work now if i merge it? [18:19:23] jbond42: the other ones I know of are the ones used for envoyproxy tls in k8s too [18:19:28] yes it should work with out issue. its the same cert just extended the ca [18:19:29] those are in the private repo tho [18:19:41] hmm maybe the ca is not deployed for that hough [18:19:48] ok cool [18:19:52] if its the private repo on the puppetmasters i know of them [18:20:16] i think most things in ops referce to the file listed above like this https://github.com/wikimedia/puppet/blob/production/modules/profile/templates/rsyslog/output_kafka.conf.erb#L63 [18:20:37] but the ca file is still copied to the private repo just not used [18:20:50] ya cool [18:20:50] ok [18:21:46] either way ill update the files in private and ill make a node to check in service ops tomorrow to make sure there are no other k8 things to consider [18:22:42] I have another question, if I may. what is the etiquette for code review? one part of that is jenkins/CI and that's fine. but for the manual human review, how does it work? [18:22:53] should I just find someone (anyone) and ask them to quickly review it? [18:23:12] does it hold true for small changes as well that don't affect the code (typo, formatting)? [18:23:37] sukhe: first try adding some people just in gerrit UI so they get notified but not necessarily realtime. if that doesn't work, escalate to IRC pings [18:23:56] as an example, for the first pass, moritzm looked at things. but I was wondering about what to do going forward [18:24:08] sukhe: on the first part its a bit of gusse work. i tend to use blame to get a feel of who is familure with th code and add them [18:24:15] mutante: yeah but *which* people is the question? :) [18:24:36] jbond42: in this case, the code is mine and so far, no external contributors [18:24:49] the second bit is personal jugment [18:24:55] sukhe: i'm afraid there isn't much more than "use good judgement" when it's about deciding if it's trivial enough for self-merge or not. depends how comfortable you are with it and what's the worst that can break. [18:25:15] for example much more likely that self-merge is ok if it's a new thing anyways vs an existing service [18:25:40] I see [18:25:54] sukhe: there is the special wiki page where people can "subscribe" to certain file names/pathes/modules and "reviewer-bot" adds them [18:26:05] so if they do they get added automagically [18:26:20] feel free to add me if its a puppet change [18:26:33] and maybe just "git log" on the files to see who changed them before [18:26:35] mutante: ...THERE IS? [18:26:39] I think this decision (at least for me) is easier if I modifying some other project but I am not sure what to do for my own project as I also don't want to bother people by saying hey can you review this for me [18:26:51] s/if I/if I am [18:26:59] ottomata: yes https://www.mediawiki.org/wiki/Git/Reviewers [18:27:09] sukhe: what project? [18:27:12] mutante: TIL cool thanks [18:27:15] ottomata: you can add yourself and a regex about stuff you want to be added to [18:27:36] or by repo [18:27:46] ottomata: your on that page :D [18:27:47] weow [18:27:50] ottomata: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/censorship-monitoring/ [18:28:08] DID I add myself there? [18:28:08] HM [18:28:09] cool [18:28:31] sukhe: i'd recommend volans he loves reviewing ops python projects [18:28:35] :p [18:28:49] i think he gets added automatically to anything ending in .py :) [18:28:52] haha [18:28:58] only in the puppet repo [18:28:59] :D [18:29:10] volans: ! [18:29:21] I felt summoned [18:29:35] that's what I am hesitant in doing and hence the initial question :P [18:29:36] * volans sensed a disturbance in the force [18:30:39] feel free to add me to python stuff, happy to review in general [18:30:46] yo can add me to changes to that repo, just as im curious but volans is your best bet :) [18:30:55] thank you all [18:31:13] I am sorry for these inane questions [18:31:14] for existing projects a gerrit bot might add people automatically [18:31:30] based on either settings in gerrit or opt-in signup in https://www.mediawiki.org/wiki/Git/Reviewers [18:31:32] oops I'm late, also happy to do python reviews 👋 [18:32:22] for new stuff is indeed harder to guess [18:32:36] in particular if it's a single person effort and not team effort [18:32:39] volans: also this is the first time I am doing this here so trying to get a sense of the culture [18:33:10] as an example, I still haven't figured out if it's acceptable to DM people without asking first. in my previous job, that was not a good thing to do [18:33:13] it's a learning experience [18:33:31] that's mostly a personal preference I guess [18:34:03] DM or ping, same same to me, prefer ping so I can pass off responsibilty to others in chan if possible :p [18:34:24] sukhe: i think it never hurts to ask for help, be bold, etc. [18:34:31] i don't mind direct DMs if they have content. then you can answer later and it's great across timezones. it's just the contentless-pings (around?). you see them later but still don't know what it was about. [18:34:35] you can ping folks and ask them who to ask too [18:35:15] hehe, that's why i like to sign off of IRC when not working. if i'm not on IRC...that's what email is for! [18:35:39] i feel like staying on IRC when not there is like putting a microphone in an office at work so you can listen in on conversations happening [18:35:44] i dont' mind missing those if i'm not there :) [18:35:46] well, ok. but at least google chat stays off :)