[08:15:45] <godog>	 I'd like to set "prometheus::default_web_instance: 'tools'" in hiera for toolforge, what's the recommended place?
[10:25:34] <dhinus>	 godog: probably https://horizon.wikimedia.org/project/puppet/
[10:43:16] <taavi>	 godog: iirc we already did set that, but the switch is/was broken?
[10:43:39] <taavi>	 since I remember breaking prod prometheus's lvs health checks when trying to fix it
[11:00:06] <godog>	 dhinus: ack thank you
[11:00:38] <godog>	 taavi: ah mhhh could be, I don't see the redirect at https://prometheus.svc.toolforge.org hence my question, good point re: health checks tho
[11:03:09] * godog lunch
[11:37:38] <taavi>	 godog: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1146973 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1151612, it seems
[12:01:12] <godog>	 taavi: ok thank you, do you remember offhand why/how it works in production?
[12:02:39] <taavi>	 godog: basically, the LVS checks except a 200 from / but that patch "fixed" it to be a redirect instead
[12:03:55] <godog>	 ack
[13:53:32] <andrewbogott>	 godog: shall I work on a zookeeper setup or are you already halfway through that?
[14:28:18] <godog>	 andrewbogott: heh I haven't thought about the implementation in terms of priority tbh
[14:29:03] <andrewbogott>	 I'll probably have a look. If it's the last thing that makes our cloudcontrols not proper failovers it seems important-ish
[14:29:28] <andrewbogott>	 Looks like dzahn has already set up some nicely-documented zookeeper puppet classes.
[14:29:50] <godog>	 ok SGTM
[14:30:44] <Raymond_Ndibe>	 Need +1 for testplatform vps project creation https://phabricator.wikimedia.org/T423226
[14:31:35] <andrewbogott>	 Raymond_Ndibe: done
[14:33:36] <godog>	 in terms of failover as in "host drops off the network entirely" FWIW we also have T422820 for rabbit/oslo, then that's it as far as I'm aware
[14:33:37] <stashbot>	 T422820: oslo.messaging does not failover to the next rabbit host on traffic blackhole situations - https://phabricator.wikimedia.org/T422820
[14:34:21] <andrewbogott>	 yeah, that one seems messier :(
[14:36:25] <godog>	 andrewbogott: I was thinking we could at least get upstream's opinion/ideas, just in case we're holding it wrong, surely we can't be the only ones having this problem
[14:36:57] <andrewbogott>	 yeah
[14:37:42] <andrewbogott>	 I guess maybe the normal openstack mailing list, or possibly in #openstack-oslo on OFTC although that's more for devs than operators
[14:39:37] <godog>	 sounds good, I'll try irc tomorrow and the ML if that fails
[14:39:59] <taavi>	 andrewbogott: Raymond_Ndibe: the project name of T423226 seems awfully close to the "do not do this" part of https://phabricator.wikimedia.org/project/view/2875/
[14:39:59] <stashbot>	 T423226: Request creation of testplatform VPS project - https://phabricator.wikimedia.org/T423226
[14:42:04] <andrewbogott>	 taavi: you mean because it sounds team-based rather than project-based?
[14:42:39] <taavi>	 yeah
[14:43:49] <andrewbogott>	 testplatformtests also seems bad :)
[14:44:30] <taavi>	 maybe that's a sign that the project scope should be a bit more generic than 'any tests in this general area' :)
[14:49:29] * andrewbogott suggests 'ciperformance' on task
[14:49:33] <andrewbogott>	 also not perfect
[14:50:40] <aputhin>	 ciplatformtests? 
[14:54:15] <andrewbogott>	 that's not bad
[15:18:45] <andrewbogott>	 Are these recurring maintaindbusers emails something that's already understood/handled?
[16:24:44] <dhinus>	 andrewbogott: I'll have a look
[16:24:53] <andrewbogott>	 thx
[16:25:20] <andrewbogott>	 isn't your day pretty much over though? I can also look, just didn't want to duplicate effort
[16:25:49] <dhinus>	 my plan was exactly to look at this before ending my day :D
[16:26:03] <dhinus>	 (I noticed the alert before but I had a meeting)
[16:26:27] <andrewbogott>	 ok! lmk if there's anything to hand off
[16:26:39] <dhinus>	 I will
[16:35:37] <dhinus>	 the error is: "Can't connect to MySQL server on 'an-redacteddb1001.eqiad.wmnet'"
[16:35:45] <dhinus>	 that host is currently being reimaged
[16:38:54] <dhinus>	 btullis: do you expect an-redacteddb1001 will be back online soon? ^
[16:39:25] <dhinus>	 otherwise we can remove it from maintain-dbusers
[16:39:31] <btullis>	 Oh what! It should be fully up and running, all services started.
[16:40:04] <btullis>	 All services are green in icinga. https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=an-redacteddb1001
[16:40:33] <btullis>	 Ah, it is probably because the IP got changed. DO we have to update the zarcillo database manually, or something?
[16:41:08] <dhinus>	 maybe, let me check
[16:42:37] <btullis>	 It's fine in orchestrator.
[16:47:39] <dhinus>	 looks like it's a firewall problem
[16:47:46] <dhinus>	 I cannot connect on the mysql port from cloudcontrol1007
[16:47:49] <dhinus>	 but I can connect to clouddbs
[16:48:01] <dhinus>	 I can ping it though
[16:50:09] <dhinus>	 maybe a repeat of T368316
[16:50:09] <stashbot>	 T368316: maintain-dbusers.service failing on cloudcontrol1005 - https://phabricator.wikimedia.org/T368316
[16:55:07] <dhinus>	 hmm in that case a rule was missing in homer, but the rule was added and should still be valid
[16:56:30] <btullis>	 Yes, I found this: https://gerrit.wikimedia.org/g/operations/homer/public/+/691bd9251cabb7554600ef7f055c7553f3d28494/policies/cr-cloud-hosts.yaml#121
[16:57:09] <btullis>	 Maybe we need to run homer against the cloud switches again, if it needs to updated with fresh netbox data, or something.
[16:57:10] <dhinus>	 yep exactly, but it doesn't seem to be working
[16:57:38] <dhinus>	 topranks: maybe you know? ^
[17:00:48] * taavi looking
[17:03:16] <taavi>	 yeah, the latest run of the GetHosts netbox script is a week old
[17:04:34] <taavi>	 (T361549 the task for is automating noticing this)
[17:04:34] <stashbot>	 T361549: Automatically run Capirca Netbox script regularly - https://phabricator.wikimedia.org/T361549
[17:07:28] <topranks>	 seems like taavi has the answer yep, run the script then re-run homer for the cloudsw and crs
[17:07:39] <topranks>	 let me know if you need a hand
[17:10:45] <dhinus>	 thanks both! taavi can you please run the script, or link me to some docs? I don't know how to run the script
[17:11:11] <topranks>	 dhinus: it was run a few mins ago by taavi 
[17:11:34] <topranks>	 This is the url, running is just a matter of clicking the button:  https://netbox.wikimedia.org/extras/scripts/1/
[17:11:36] <taavi>	 dealing with it, and as usual merging a bunch of other new/decom'd hosts at the same time
[17:11:59] <taavi>	 topranks: you need to check that checkbox first!
[17:12:04] <taavi>	 very difficult, I know
[17:12:14] <taavi>	 (I failed it the first time)
[17:12:26] <topranks>	 ha yeah
[17:13:33] <topranks>	 I'll discuss in netops, to me it seems a little mad we don't have it running automatically every night or something 
[17:16:39] <taavi>	 dhinus: btullis: try again now?
[17:17:33] <dhinus>	 seems to be working
[17:18:25] <dhinus>	 yep, I see some "Created account in an-redacteddb1001.eqiad.wmnet:" in the maintain-dbusers logs
[17:24:06] <dhinus>	 the alert should clear soon
[17:24:09] <dhinus>	 thanks everyone!
[17:24:46] <dhinus>	 andrewbogott: should be all sorted
[17:24:54] <andrewbogott>	 great!
[17:29:27] * dhinus off
[17:46:36] <btullis>	 Nice. Thanks, all.
[18:32:10] <taavi>	 bd808: I assume you also got the github email about https://github.com/toolforge/toolforge-php-mysql webhook token needing to be rotated?
[19:34:01] <taavi>	 andrewbogott: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271029 will not work, the .sources format is different from the .list format
[19:40:04] <andrewbogott>	 dang, ok.  I guess it was only warnings when both files were present
[20:21:37] <bd808>	 taavi: yes. I haven't made time to do anything about the email yet, but I have it.
[20:37:07] <andrewbogott>	 stashbot, what's your deal
[20:37:07] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[20:37:19] <andrewbogott>	 wmopbot, do you also have a help page?
[20:37:27] <andrewbogott>	 seems not
[20:47:33] <bd808>	 andrewbogott: https://wmopbot.toolforge.org/, but you may not have the IRC creds to get in
[20:50:28] <bd808>	 actually you have the founder bit here so I think wmopbot should answer if you knock
[21:04:20] <andrewbogott>	 thanks bd808 -- I was trying to answer bliviero's general question "what is with all these bots?" and I asserted that they were self-documenting which turns out to be only partly true :)
[23:30:56] <bd808>	 I filed T423354 to get the object count quota bumped up for etherpads3
[23:30:57] <stashbot>	 T423354: Increase object storage object count for etherpads3 project - https://phabricator.wikimedia.org/T423354