[11:32:30] I would like to update facts on puppet compiler hosts (modules/puppet_compiler/files/compiler-update-facts). Can someone grant me access to the puppet-diffs horizon project? [11:41:16] jayme: added [11:43:06] jbond42: thanks! [11:43:12] np [12:22:21] hurm. anyone know why/how https://grafana.wikimedia.org/d/000000303/mysql-replication-lag?orgId=1 is marked as non-edittable? i'm only familiar with that from when dashboards are deployed via the file system, but i can't see any evidence of that in this case [12:35:32] kormat: there's a "Make Editable" button in dashboard settings, but, idk why it was marked as non-editable in the first place [12:35:54] looks like you can't see revision history when it's non-editable either [12:37:17] cdanis: that's how i knew it was not edittable :P [12:37:31] i'm trying to figure out the source of the dashboard [12:39:16] I suggest beginning with making it editable and looking at revision history :) [12:39:59] ok, now i'm left with more questions [12:40:23] there's 3 difference versions that talk about making it edittable. what's making it unedittable? [13:13:06] cdanis: in any case, thanks for that suggestion :) [13:16:46] We're going to replace eqiad's mgmt switch in ~5/10min, which mean no more eqiad mgmt access for ~30min please let me or cmjohnson1 know if we should delay it [14:14:34] fyo, the mgmt switch replacement is completed, let me know if you see any issue [14:18:16] kormat: the source is usually grafana's db, except when the dashboard is tagged with source:git or source:puppet (IIRC), my understanding is that for non-git dashboard it is humans that mark dashboard non editable [14:18:39] damn those humans [14:19:24] ls [14:19:31] whoops wrong window :D [14:21:20] kormat: ikr? terrible [17:00:37] hey herron yt? [17:00:58] hey ottomata [17:01:16] q about logstash, am trying to verify that the validation error log messages from eventgate do indeed make it into logstash [17:01:23] but i can't seem to find them via kibana [17:01:29] i have some examples from the eventgate instance [17:01:40] log output, but not in logastsh [17:02:00] also, every time i use kibana i barely understand how to make it work [17:02:03] so i might be doing something wrong [17:02:20] is there an easy way to verify that a log message has arrived? [17:02:22] ok, can you shoot me the examples? I can have a look [17:02:59] https://gist.github.com/ottomata/a5c2259e5de82664cea1014b6320311d [17:03:02] herron: ^ [17:03:45] thanks [17:12:51] ottomata: looks like yes they are arriving, https://logstash.wikimedia.org/goto/e9a5edf9c60a73f78652b212256f813e [17:13:16] you are a wizard [17:13:39] that seems so simple but so many of the things i tried just brought up random results [17:13:42] thank you [17:13:50] hehe sure np! [17:15:10] hmm herron ok cool, now that I'm finally looking at these, i think it might actually be useful to have the error event in logstash after all [17:15:15] the error event has the raw event (as a string) too [17:15:25] so, a dev could actually see the event value that caused the validation error [17:15:39] this one https://gerrit.wikimedia.org/r/c/operations/puppet/+/589597 [17:16:07] ah ok [17:16:15] they can indeed look elsewhere, but it would be nice to know exactly why there is a problem and be able to fix it rather than having to find the offending event in hive [17:18:37] yeah makes sense. it looks like there might be an open todo on that about the truststore password, but if/when ready happy to give it a try [17:18:46] I’ll re-add myself [17:19:20] ya [17:19:23] looking at that now [17:19:23] ok thanks [17:43:59] ok herron [17:44:00] https://gerrit.wikimedia.org/r/c/operations/puppet/+/589597 [17:44:15] updated the class to take a hash of truststore pws [17:44:35] if you think that's an ok wayt to go, i will merge https://gerrit.wikimedia.org/r/c/labs/private/+/609828 and run PCC [17:45:07] and if that works, i'll add the ::passwords hash to private, then we can merge, then we can remove the ::password from private hiera [17:46:01] ok I’ll have a closer look here shortly [18:16:11] ottomata: ok I'm game to try this now if you have time [18:16:24] ya perfect [18:16:31] we should disable puppet on the logstash hosts while hiera is being shuffled around [18:16:35] ok I'll disable [18:16:45] hang on lets get PCC etc. confirmed first [18:16:48] but ya [18:17:11] so if you like approach i will merge [18:17:12] https://gerrit.wikimedia.org/r/c/labs/private/+/609828 [18:17:14] and we can try PCC [18:17:52] herron: ^ [18:18:21] yup sounds good [18:19:41] logstash1010 a node to apply to ya? [18:19:47] hm no [18:20:10] logstash1007, logstash1023 [18:20:20] should cover it [18:23:10] hmm says noop herron https://puppet-compiler.wmflabs.org/compiler1003/23714/ [18:23:23] yeah was just looking at that as well [18:23:25] hmm [18:28:34] oh, of course, these hosts compile correctly only with the change [18:29:28] ? [18:29:35] ya but we should see a diff, no? [18:29:49] there's no prod catalog to diff against, since that hiera lookup breaks it [18:29:57] ? [18:30:22] https://puppet-compiler.wmflabs.org/compiler1003/23714/logstash1007.eqiad.wmnet/prod.logstash1007.eqiad.wmnet.err [18:30:25] for example [18:30:34] OH the diff huh [18:30:36] hmmm [18:30:48] ok i guess i can re-add the ::password value [18:30:53] to labs-private for PCC purposes [18:31:02] kk [18:34:19] ok https://puppet-compiler.wmflabs.org/compiler1001/23717/ [18:34:51] heyy much better [18:35:20] ya lgtm,ok [18:35:35] going to add ::passwords to private (and keep ::password there for now too) [18:35:57] ack [18:38:22] ok, done [18:38:31] herron: if you want to disable puppet now we can merge and apply on one node [18:38:46] ok ready when you are, puppet is disabled already [18:40:30] ok lets go [18:41:52] ok herron which node shall we try on, and do you want to run puppet or shall I? [18:42:03] (puppet merged) [18:42:07] I can do it, will use logstash1009 first [18:42:10] ok [18:45:09] puppet is applying, slowly... [18:45:28] Notice: Applied catalog in 152.58 seconds [18:46:25] cooool [18:46:40] so far so good, looks look ok [18:46:40] should be seeing these in kibana now [18:47:36] https://logstash.wikimedia.org/goto/3a6d95ba3ff329a94e3b27cb52086367 [18:49:24] s/looks/logs [18:51:07] ya ok and [18:51:09] https://gist.github.com/ottomata/a5c2259e5de82664cea1014b6320311d#file-validation-error-event-json [18:51:14] is an example of the event as it comes from kafka [18:51:40] oh ho and they are there! [18:51:41] cool! [18:52:19] sweet! ok I'll re-enable puppet and let it do it's thing [18:53:40] awesome thank you [18:53:54] herron: is it posssible to query logstash from grafana? :) i would guess not :) [18:54:31] yeah not with how things are currently, but we have been talking about some ideas of how to get logs in as metrics [20:14:21] herron: should be ok to remove the private ::password now ya? [20:14:39] think so yea ottomata [20:14:44] or, hm i guess let's wait til tomorrow in case we need emergency revert [20:14:46] i'll do that then :) [20:17:10] cool sounds good to me