[11:51:06] if anyone has any insights here, I would appreciate it for a reimage [11:51:31] I am getting a Error: Unable to establish IPMI v2 / RMCP+ session if I run ipmitool -I [11:51:56] however, if I ssh to the manegment inteface, the password works, so it is not it [11:52:30] effie: I'd suggest hard idrac restart, that usually fixes it [11:54:03] I an trying to locate this in teh docs [11:54:56] oh dear, dont tell me I need the UI for this [11:55:02] no, you do not [11:55:21] you do it from the ssh mgmt interface [11:55:31] or from the host itself if it is up [11:55:49] I fount it online, I will update the docs [11:56:42] racadm racreset soft [11:57:57] effie: https://wikitech.wikimedia.org/wiki/Management_Interfaces [12:00:27] I was reading SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation. ... [12:00:38] than you manuel [12:12:29] I had to racadm set iDRAC.Users.2.Password [12:12:30] sigh [14:15:35] Emperor: can I stop puppet on ms-fe to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1300077 ? I'd like to put it on one to see that it works as intended then roll it out [14:20:18] claime: ack; please don't use ms-fe{1,2}009 as your canary. [14:20:43] Emperor: I won't :) [14:21:23] cool. It's just easier for me to always say that than to assume folk remember which nodes are weird :) [15:14:13] possibly a bad idea and hence the discussion: I was discussing this with Moritz as well, but wondering if we can SRV urldownloader [15:14:29] right now it is like 'urldownloader' => 'http://url-downloader.eqiad.wikimedia.org:8080', [15:14:38] so it's essentially a single host [15:14:50] the full context is that another critical service depends on it from MW, and that's hcaptcha [15:15:19] sukhe: should we instead liberica it? [15:16:55] cdanis: I mean we could yeah, but I haven't really thought that through so I was wondering if SRVing it instead for now should achieve what we want [15:17:00] https://logstash.wikimedia.org/goto/31009dcbb90cd1f554d66a993bb860af [15:17:09] trying to avoid things like this basically since it's a single host [15:17:12] yeah [15:17:16] we should absolutely do something [15:18:10] possibly also route it through the service mesh somehow if we can? [15:18:35] you mean hcaptcha specifically? [15:18:52] or any usage of urldownloader from mediawiki [15:19:00] but yeah [15:19:01] yeah... [15:19:28] anyway I guess for now we should do whatever is the least work while also being minimally gross [15:19:34] I tried to go down that path as well [https://wikitech.wikimedia.org/wiki/Envoy#Use_a_listener] [15:19:40] I guess the first thing is a new VM with a public IP? [15:19:44] but since I don't know everything about those bits and pieces, I gave up :P [15:22:25] sukhe, cdanis: I'd suggest using the HTTP proxies instead, and indeed setting them up as active/active in each site - https://phabricator.wikimedia.org/T242715 [15:22:26] Another service is going to arrive for those too, so we better do something solid - https://phabricator.wikimedia.org/T427457#11993782 [15:22:47] XioNoX: they allegedly serve different purposes [15:23:00] (urldownloader is non-caching, for one) [15:24:10] if the client uses https it won't cache anything neither (and we can also set the proxies to be non-caching) [15:24:23] or maybe I'm confused by the name "urldownloader" :) [15:24:24] https://phabricator.wikimedia.org/T254011#6181867 [15:24:48] yeah there is a distinction -- whether they can be merged, I am personally not sure [15:25:33] I think the better distinction is perhaps batch processing and user-critical-path-blocking fetches [15:25:39] cdanis: great, thx! [15:25:53] yeah there is a tradeoff between having 2 systems to maintain vs. one [15:25:55] yeah [15:26:01] "maintain" [15:27:39] cdanis: :) if we need to start un-dusting some of them, we could make a conscious choice whether merging them or not, or having 2 deamons on the same machines for example, with the proper doc, etc [15:27:56] yeah [15:28:15] not sure if this is I/F or serviceops or both :) [15:28:46] I guess it's "managers" for now [15:29:06] but yeah maybe it's a good time to have a good look at both those services [15:29:10] +1 [15:29:30] as a "manager", I will take this on [15:30:20] there is one other critical service that depends on the urldownloaders for example [15:30:27] the image processing one for example. so yeah [15:32:29] sukhe: happy to help with documentation and with design [15:33:10] cdanis: yeah I will need your input for sure, I definitely do not fully understand this [15:33:59] I am trying to currently parse through wmf-config [15:34:19] to see how $wmgLocalServices is consumed, to see if SRVing urldownloader will work in the interim [15:44:32] yeah same, happy to help [16:01:54] for folks who use `git review -d ` is there a way to diff between patchsets in git? [16:03:19] Emperor: actually didn't have time for it because meetings, so I'll do it tomorrow maybe [16:04:38] jhathaway: you can add a comma and patchset number after the change number, to download that patchset. once you have them both locally, you can diff as you like [16:04:51] git review -d 1234567,1 [16:05:12] thanks MatmaRex, the `-m` option also seems relevant, should have read the man page [16:07:33] ah, -m automates those steps MatmaRex, e.g. git review -m 1293593,17-18 [16:09:32] oh, neat [16:20:05] oh neat [16:40:55] claime: I'm only working a couple of hours tomorrow, and am then on post-oncall TOIL until Monday. [16:41:11] [I'm happy for you to go ahead anyway, but thought you'd like to know :) ]