[09:05:42] I could use an Apache config tweak for the CI Jenkins. It is to let us change json.gz artifact files to be considered as application/json with gzip encoding [09:06:09] namely for some super large trace/debug files that I need to compress on the server side, but still need to be able to drag'n drop the link in the Chrome performance tool [09:06:17] I have live hacked it on the server and that seems to work: https://gerrit.wikimedia.org/r/c/operations/puppet/+/675515/2/modules/contint/templates/apache/proxy_jenkins.erb ;] [09:06:44] doesn't affect the CI service itself since we don't rely on the web frontend at all beside for showing the jenkins ui. So it is rather harmless [09:46:29] moritzm: when you have a minute, quick review https://gerrit.wikimedia.org/r/c/operations/puppet/+/675722 [09:47:48] oh, and https://gerrit.wikimedia.org/r/c/operations/puppet/+/675812 xd [10:51:43] in deployment-prep, what's the distinction between THING.eqiad1.wikimedia.cloud and THING.eqiad.wmflabs? which is preferred? [10:59:08] answer https://wikitech.wikimedia.org/wiki/News/Phasing_out_the_.wmflabs_domain [11:26:39] yeah new instances get the new form of the name (I say, having one such, and I had to poke some things in the manifests to get it to work) [13:00:59] hi all i have drafted an incident document for the jobrunner incident yesterday. could those involved please give it and the actionable tasks a review to make sure i have captured things correctly [13:01:22] legoktm: Urbanecm: akosiaris: mutante: rzl: jynus: ^^^ [13:02:15] jbond42: i can have a look, is the google doc doc still the canonical version, or is it on wiki somewhere? [13:04:39] Urbanecm: i think the gdoc one and the wiki one have seperate audiences. The gdoc one has all the detail including a complete guidline. the wikitech one is provide so intrasted parties can get a good overview of what happened and what needs to be done next, without getting to bogged down in all the detail [13:05:17] knowthing in this incident was private so if there is something in the gdoc that you think is usefull and should be on the wiki doc please make the update [13:05:26] (rzl, cdanis ping to make sure i got this right ^^) [13:05:58] s/complete guidline/complete timeline/ [13:07:06] makes sense jbond42. I always thought the wiki one is the gdoc one minus PII, but apparently they might be partially different too [13:09:08] ill wait for rz.l or cda.nis to give an authorative answer but, i think its more that for small incidents most people dont need or whant the full verbose version as such its more about providing soething that is usefull and will get read and not about trying to redact (however we would of course allways redact PII) [13:13:11] jbond42: sure, send me the link. [13:13:34] akosiaris: https://docs.google.com/document/d/1kvNvBQb94KHddd8an9Lf2HyekQeq4HzGU-xK8x4Cqes/edit# and https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-03-30_Jobqueue_overload [13:13:40] Thanks! [13:13:54] yes that would be usefull :) thanks Urbanecm [13:14:02] np [13:14:46] for the lightweight vs full thing the guidelines are at https://wikitech.wikimedia.org/wiki/Incident_response/Lightweight_report#Decision_criteria [13:15:37] And I think Urbanecm ticks the last box ;-) [13:16:26] well...i can read it anyway. I was just curious when i asked, thanks for the link akosiaris [13:16:36] the rest are not ticked AFAICT (the 2nd one may be argued, but unique/surprising is a very subjective thing) [13:17:35] yes i think it was on the edge, happy to update it to a full report if its wanted [13:17:52] it was very surprising to me at least. I performed a lot of server side uploads before, some even of a similar size, and i don't recall any irregulaties happening as a result [13:18:37] It gives me a good reason to say even the WMF jobrunners get overloaded now when someone moans at me to clear ours [13:29:54] heads up, merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/675237. Per https://phabricator.wikimedia.org/T278220#6956779 it's going to take a while until puppet is enabled again in all mediawiki hosts [14:25:38] jbond42: good use of the lightweight format IMO, thanks for doing the work! [14:27:48] FWIW even the longer format isn't identical to the gdoc -- the timeline is mostly the same, but most of the extra effort is involved in taking the rough notes and polishing them into a narrative with analysis and conclusions [14:28:24] thanks rzl [14:28:33] in this case we could certainly find stuff to fill in those sections if we wanted to, but I don't think the effort is warranted, we already have a good set of actionables [14:28:35] the gdocs serve a different purpose and are intended to be ephemeral [14:28:51] (they won't ever actually be that, but) [14:29:33] ha, that would certainly be a good forcing function to get us to finish IRs faster [14:29:34] rzl, I would like to suggest a different format for reports, I don't think they are blameless enough [14:29:40] but not at this time [14:29:49] oh interesting! yeah let's chat sometime [14:30:35] (based on past experiences) I will try to catch you at some point in the next weeks [14:31:07] 👍 [17:12:42] Any SRE mind a quick pm? [17:15:55] cdanis: ^ [17:19:44] Handled [19:55:04] Hi SRE, I'm looking to write a script to install nodejs version 14 on a build server to compile the frontend of an upstream application. elukey pointed out the NodeSource recommended `curl ... | bash` is a bad way to do this, but is it good enough to download a .deb and then install that? [19:55:45] (I ask here because Luca is offline for the day and I'm hoping to release the newer version today) [20:02:01] razzi: we should import the debs to apt.wikimedia.org ideally [20:02:50] I think we already use the nodesource debs, so it would be a matter of creating a new component for the v14 packages [20:03:03] legoktm: ok cool [20:05:07] legoktm: would I use reprepro for this? [20:05:15] Yeah [20:05:49] https://wikitech.wikimedia.org/wiki/Reprepro#Updating_external_repositories [20:06:31] no, we never used the nodesource debs. these were either built internally (node6) and with buster we simply use the buster default [20:11:13] if it's just for building something, importing these seems kind of okay, but the quality of the nodesource debs is really poor (e.g. they bundle Openssl and various other libs), so we need to make sure it's not used for running things in prod [20:11:53] ah, my bad then [20:12:24] what's wrong with bundling multiple copies of openssl? zoom does it/s [20:12:27] moritzm: indeed, this will only be used in a temporary docker container on deneb.codfw.wmnet [20:33:21] moritzm: would you recommend against putting a nodesource .deb on apt.wikimedia.org, then, so that it won't accidentally be used in production? [21:39:37] nah go ahead with an import, but e.g. add a comment to the reprepro config, so that people don't use it to run things in prod [21:45:02] Sounds good moritzm. Do you have a few minutes to get me started? Do I need to create a new git repository for this? [22:02:44] Perhaps it's as easy as downloading the deb and running `reprepro -C thirdparty/node14 includedeb buster-wikimedia nodejs_14.15.5-1nodesource1_amd64.deb`?