[05:21:42] bstorm: sorry I couldn't reconnect yesterday night, glad you find the issue. Feel free to ping me if you have any additional issue. [08:14:37] FYI I'll be switching grafana.w.o to codfw/grafana 7 shortly [09:12:35] mutante: fyi yes you should be able to use `Hosts: C:`. however a node which uses that class name needs to exist in the pcc puppetdb. without going to much into detail this means that you cant test new classes. if you ping when online can help look at the specifics [13:32:37] Hi everyone, where is the best channel to discuss the creation of a new deployment window for services at Monday european time? [13:37:24] -releng I believe [14:10:55] heya _joe_ qq, is it possible to create symlinks in a sane way in binary deb packaging? [14:11:18] I'm used to dh_links, but now I'm just running dpkg-deb directly and skipping debhelper stuff [14:11:54] i've tried manually creating them, but of course the target paths are messed up, since the local file paths in debian/ are different than the final install system paths [14:15:05] <_joe_> ottomata: you're pinging the wrong person :) [14:16:02] oh, sorry the binary package idea i got from you so pinged you :) [14:16:56] i'd ping moritz but i don't see him in this room which maybe means he is off? [14:18:11] ottomata: ema is a fine person to bother about debian packaging :) [14:18:33] (he's also off though) [14:18:43] (ah. what a 'concidence') [14:19:57] wait was that my cue? [14:21:10] kormat: coincidence but personally neither exist. [14:21:35] ottomata: I'm not familiar with what you are trying to do, is there a bit more context ? [14:44:20] godog: i'm making a anaconda deb package [14:44:29] which is just a distribution of a conda environment with lots of packages [14:44:38] it also has some custom tweaks for us [14:44:46] but i'm not building anything from source [14:44:53] and joe recommended i do just a 'binary' package [14:44:58] i'm mostly following this [14:44:58] https://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/ [14:45:01] its all working great [14:45:08] except, i'd like to create some symlinks [14:45:13] into e.g. /usr/bin [14:45:48] so basically i'm just prepping the debian/ dir with files in their insttall locations, like everything in debian/usr/lib/anaconda-wmf [14:45:52] on install [14:46:23] i'd like to have a symlink for a few things there, like /usr/bin/conda-create-stacked -> /usr/lib/anaconda-wmf/bin/conda-create-stacked [14:46:24] etc. [14:46:33] if I were using debhelper [14:46:40] i'd know what to do with the .links file [14:46:49] but, i'm just arranging files in a dir hierarchy [14:46:54] and then running dpkg-deb [14:48:38] ottomata: ah, in that case I think you can ln -s /usr/lib/anaconda-wmf-bin/conda-create-stacked debian//usr/bin/conda-create-stacked, where "something" is either "tmp" or the package name, "it depends" [14:49:22] tmp? [14:49:37] ottomata: assuming I understood correctly, if you have a repo or a code review I can take a look [14:49:42] repo yes [14:49:55] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/anaconda-wmf/+/refs/heads/debian [14:49:56] but [14:50:02] wait.. [14:50:29] there we go [14:50:29] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/anaconda-wmf/+/refs/heads/debian [14:50:34] "... there's more!" [14:50:35] had to merge a patch [14:50:41] so there's not much there [14:50:45] debian/ is empty [14:50:53] the README.debian.md file has ll the instructions [14:52:04] i want to add steps there after Copy our custom conda user env bin scripts into usr/lib/anaconda-wmf/bin [14:52:06] to make the sylminks [14:54:16] ah I see, then yeah with the instructions you have already you should be able to ship a symlink by copying it within debian/, it seems to me [14:54:30] so ln -s /usr/lib/anaconda-wmf-bin/conda-create-stacked debian/usr/bin/conda-create-stacked [14:56:21] ottomata: FYI I recently did something similar to that "a debian package that downloads things and builds a binary package" but in a different way, https://gerrit.wikimedia.org/r/c/operations/debs/grafana-plugins/+/618953 [14:56:22] hmm i think i tired that [14:57:06] ya godog i thought about making a script to do all this [14:57:11] but some of the steps take a long time [14:57:31] and esp while developing restarting from the beginning every time would take for ever [14:57:40] godog: will try the s ymlinks again... [14:58:36] ottomata: sounds good! re: the script, making the time-consuming steps idempotent is a possibility too [14:59:23] having said that, I'm not familiar with conda or its build process or whether it can be made reasonably unattended [14:59:51] godog: won't a symlink like that be broken after it is installed? [14:59:58] HMM mayb enot. [14:59:59] hmmm [15:00:16] if the destination exists after it is installed it won't, no [16:09:32] random q for for folks [16:09:48] what does 'opaque ID' mean to you? [16:10:01] (trying to make sure my own definition isn't wrong...writing a blog post here)( [16:16:17] it doesn't have any significance except that it's different for different entities [16:16:31] no ordering, no internal structure or fields, the only sensible operation is "equal or not equal" [16:22:44] hmm ok lemme give a sentence for context [16:23:10] ... [16:23:10] To do either of these, we need to be able to find the schema of an event. [16:23:10] Confluent does this with opaque schema IDs which are used to look up schemas in their remote Schema Registry service. [16:23:30] rzl: in ^, what does 'opaque' mean to you? [16:23:43] not quizzing you, just trying to see if my meaning of the word will be understood by readers [16:42:48] ottomata: same answer I think [16:43:14] hmm ok [16:43:21] I get from that sentence that the ID is used for lookup in that way, but that I shouldn't expect to examine the ID itself and learn anything about the schema [16:43:33] ok cool yea that's right [16:44:01] the point i'm trying to make is that the ID carries no meaning outside of the schema registry installation [16:44:16] i'm adding more now to explain that point [16:44:28] i was asking dan in analytics eng too and he thought I shoudl do that [17:54:57] hi all, I could use some input on determining where we want to draw the line between ganeti VM and hardware. So .. the existing ganeti VMs usually have RAM between 1G and 8G, only 2 special cases with 16GB, puppetdb and deneb. none have more than 16. Now we have a request to upgrade the webperf VMs from 8G to 16G or 32G if possible. I looked at "gnt-node list" to try to determine how much is left [17:55:03] that can be allocated and the MFree column tells me it it is anything from 12 up to 62 depending on the node. So I am not sure how to calculate it but either way it seems to mean we can't do 2 x 32GB. So is this the line where we say webperf* should move to metal ? [17:55:59] ..or does it just mean the ganeti cluster needs more resources [17:57:25] dpifke: just a cc: for you ^ [18:00:54] If it wasn't clear from the request, only webperfX002 (one VM in each cluster) needs more RAM. webperfX001 can stay with 8GB. [18:02:12] Given the X001 hosts have different function than X002 hosts, perhaps we use this as an excuse to rename them. [18:02:26] ah yes, one per cluster [18:02:38] dpifke: yea, actually.. let's do the renaming [18:02:52] I *think* arclamp is the only thing running on X002, let me double check. [18:04:13] i guess i can do "8 GB more" and first try it in codfw. so the docs say it can be done without downtime [18:04:30] Yup, just arclamp, so we could rename them arclampX001. I think originally xhgui was intended to share the X002 VMs, but that's now separate. [18:05:07] no, wait. it can't "The same command can also be used to increase the memory available on an instance, provided that enough free memory is available on its node, and the specified size is not larger than the maximum memory size the instance had when it was first booted (an instance will be unable to see new memory above the maximum that was specified to the hypervisor at its boot time, if it needs [18:05:13] to grow further a reboot becomes necessary)." [18:05:32] It's batch processing, so OK if downtime is needed. It'll catch up when it comes back. [18:06:09] dpifke: ok, i can help with adding a new "cluster" arclamp [18:06:37] i will try to give more RAM to webperf2002 [18:06:41] while it's running [18:06:48] and then we can reboot it [18:07:03] SGTM. [18:07:59] well, it did not like that. Instance webperf2002.codfw.wmnet must have memory between 8192 and 8192 MB of memory unless --force is given [18:08:53] searching a bit more [18:11:29] It's OK if you need to shut it down, then add the memory while it's offline, then bring it back up. Esp since we were going to need to reboot anyways. [18:12:18] ok, looks like i got it. the right way to raise it above the previous max is "modify -B memory=16G" [18:12:50] alright, yes needs reboot. and just doing it right now then.. but with icinga downtime [18:16:59] i had to reboot on ganeti level, not OS level. but that is very quick [18:17:01] [webperf2002:~] $ cat /proc/meminfo [18:17:01] MemTotal: 16436708 kB [18:17:20] dpifke: there you go, so the downtime was very short [18:17:29] Looks good on this end, thanks! [18:19:04] ok, so the same thing for eqiad right now or in a few minutes [18:20:20] Right now works. Actually, give me 1 min to stop the process that's writing the logs there. (It should shut down ~gracefully, but just in case...) [18:20:41] sure, i should add this to wikitech docs, brb [18:21:47] oh, it already was:) [18:22:46] You're good to go. [18:22:54] optionally we can change hieradata/eqiad.yaml:arclamp_host: webperf1002.eqiad.wmnet [18:24:55] Nah, it's fine. If the flamegraphs aren't accessible for a minute or so I don't think anyone will notice. [18:25:24] And the generation will catch up when it comes back. [18:27:02] dpifke: ok, done! it is back. took about 40 seconds [18:27:28] Also looks good from here. Appreciate the help! [18:27:29] MemTotal: 16436708 kB [18:27:39] ok. you're welcome [18:28:21] i will claim it's resolved and if it turns out you need more than 16 then let's reopen it and that discussion from the start [18:29:16] because then they would become special cases among the other VMs [18:30:41] OK. The long-term plan is to have this run as a stateless service of some sort rather than in a VM. So hopefully 16GB will hold us over until then. [18:31:03] alright. sounds good [19:31:39] looking for a dashboard or public info about how many reqs/second mw app servers handle [19:32:23] i found https://grafana.wikimedia.org/d/000000550/mediawiki-application-servers?orgId=1 but it only shows 1 server at a time [19:32:40] hm i guess i can just sum in prometheus, but was hhoping for a public linkable reference [19:33:20] ottomata: https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1 that's aggregate but only shows one of app or api at a time [19:33:35] ah hmm [19:33:41] was about to link the same yeah [19:34:19] wow this grafana explore feature is so cool [19:34:28] i can just expore based on one of those panels [19:34:53] sum(rate(mediawiki_http_requests_duration_count[2m])) [19:34:55] perfect [19:34:57] so cool [19:35:30] thanks! [19:36:22] yeah, just keep in mind that /explore is logged-in-only functionality [19:37:12] k that's fine i guess i don't really need a link ref [19:39:31] you can always just copy those panels into your own public dashboard, with modification :) [19:39:56] aye