[08:58:32] <marostegui>	 We are going to deploy the new es5 section, please hold any scap deployment until we are done
[09:13:29] <volans>	 marostegui: you can put a lock on scap on deploy1001
[09:13:37] <marostegui>	 Yeah, it is done
[09:13:42] <volans>	 :)
[09:20:18] <liw>	 volans, I did not know that
[09:20:26] <liw>	 about locking scap
[09:20:30] <liw>	 how is it implemented?
[09:21:19] <marostegui>	 liw: touch /var/lock/scap.operations_mediawiki-config.lock
[09:21:20] <marostegui>	 XD
[09:21:30] <liw>	 ack, thanks
[09:22:03] <liw>	 reason I asked: releng has been discussing whether to run scap from our laptops instead of deploy1001, and this is another nail in that coffin
[09:23:08] <jynus>	 from our laptops? how does that work?
[09:23:51] <liw>	 I'm not sure it does
[09:32:37] <volans>	 liw: what would be the reasoning for this requirement?
[09:34:16] <liw>	 I'm not sure it's a requirement. More of an accident of wiki page editing.
[09:34:33] <marostegui>	 es5 deployment window finished - scap deploys can resume as normal
[09:34:57] <volans>	 than just rever it ;)
[09:35:33] <liw>	 volans, the discussion is about what we actually need, for train; better have that first
[13:42:03] <cdanis>	 godog: https://phabricator.wikimedia.org/P10675
[13:51:33] <godog>	 cdanis: oohh interesting! logstash hosts are kinda expected I think because they just got reimaged, not sure about wdqs tho
[13:52:32] <cdanis>	 9105 is ... rsyslog exporter?
[13:53:41] <godog>	 yeah that's right
[13:54:36] <godog>	 (meeting in 6)
[13:55:43] <cdanis>	 I'm not sure what changed in prom, or if this has been happening all along and we've just missed it so far, but it is concerning
[13:56:30] <cdanis>	 Last puppet commit: (2f6fc0aad7) Cmjohnson - Adding wdqs101[1-3] to site.pp role:spare
[13:56:42] <cdanis>	 ah so those wdqs hosts might be new too?
[13:57:11] <godog>	 ah yeah that might be! puppet races all the way down
[13:59:07] <cdanis>	 otoh i just ran puppet-agent on prom1003/1004 and, no change
[15:37:08] <chaomodus>	 o/
[15:41:14] <RhinosF1>	 hi
[17:19:15] <andrewbogott>	 marostegui: With the latest mariadb, what do I need to do to complete package installation?  I'm running into lots of issues with /opt/wmf-mariadb101/bin/mysql_secure_installation
[17:19:52] <marostegui>	 andrewbogott: you mean 10.1 or 10.4
[17:19:55] <marostegui>	 I am in a meeting though
[17:20:06] <andrewbogott>	 I'm getting 10.1.44-0+deb9u1 from apt
[17:21:04] <marostegui>	 can you paste the issues somewhere so I can check later?
[17:21:12] <andrewbogott>	 sure
[17:31:27] <andrewbogott>	 marostegui: T247328
[17:31:27] <stashbot>	 T247328: issues completing installation of mariadb 10.1.44-1  - https://phabricator.wikimedia.org/T247328
[17:31:29] <andrewbogott>	 thanks
[17:39:14] <jynus>	 so 2 things, mysql_secure_installation needs to be run from basedir
[17:39:32] <jynus>	 and 2, the host seem misconfigured, pointing to /tmp/mysql.sock
[17:40:09] <jynus>	 you can just force the socket to the right location with -S, but later you should fix you mysql user config
[17:57:27] <andrewbogott>	 jynus: ok, will try
[17:57:29] <andrewbogott>	 ty
[18:00:01] <andrewbogott>	 jynus: I'm not sure I know what you mean by 'basedir'
[18:01:39] <andrewbogott>	 and, for the socket to exist wouldn't the server have to be running?
[18:16:45] * andrewbogott figured it out
[21:43:03] <mutante>	 CRITICAL: the following (9) node(s) change every puppet run: elastic2060.codfw.wmnet, logstash1029.eqiad.wmnet, elastic2056.codfw.wmnet, elastic2057.codfw.wmnet, elastic2059.codfw.wmnet, elastic2058.codfw.wmnet, elastic2055.codfw.wmnet, cloudvirt2003-dev.codfw.wmnet, an-tool1006.eqiad.wmnet
[21:44:34] <ebernhardson>	 mutante: fwiw, those elastic nodes are not fully working yet, they are being added to the cluster this week afaik
[21:45:26] <ebernhardson>	 hmm, they have joined the cluster at least though..so not sure
[21:45:27] <mutante>	 ebernhardson: interesting. let's see if that happens when servers are installed without being assigned a role
[21:46:35] <mutante>	 confirmed, they are already assigned a role
[21:47:00] <mutante>	 runs pupet on elastic2057 .. and it starts the service called:
[21:47:07] <mutante>	 mjolnir-kafka-msearch-daemon
[21:47:37] <ebernhardson>	 oh, i wonder if that doesn't come up right on new servers. checking
[21:47:37] <mutante>	 something else stops it and then puppet starts it again
[21:47:42] <mutante>	 that's why the alert is real.. yea
[21:48:00] <mutante>	 thanks
[21:59:12] <ebernhardson>	 looks like however deployments are initialized by scap is incomplete, or maybe wrong ordering, not sure. Need to dig into the puppet logs, but forcing a redeploy ran the appropriate scap scripts and it looks to start the daemons now
[22:01:43] <mutante>	 ebernhardson: cool, confirmed on elastic2057. puppet run does not (need to) start it anymore :)
[22:02:08] <mutante>	 that should solve the elastic* part of that alert
[22:02:16] <ebernhardson>	 cool :