[09:23:54] moritzm, jbond42_: by any change any one of you could look into the puppetdb queue at some point? I'm leaving for the rest of the week in the afternoon and cannot debug it, sorry [09:24:09] https://grafana.wikimedia.org/d/000000477/puppetdb?panelId=19&fullscreen&orgId=1&from=1564236470272&to=1565167416489 for reference [09:25:32] ack, I'll have a look later on; I reckon we don't have a task yet? If so, I'll create one [09:28:10] not AFAIK [09:28:21] I just noticed it today after you opened the other task :) [09:31:29] I created https://phabricator.wikimedia.org/T230002 [09:31:38] Might be related to the new Puppet 5 canaries, as more canary hosts were pointed to puppetmaster1003 around the time. [09:32:35] that's one of the possibilities I was thinkng yes [09:46:19] taking a look now [10:37:08] volans: have updated the task, tl;dr it looks like it is an issue with the upgrade, there is a schema incompatibility between the puppetmaster and puppetdb. Advice from puppetlabs is just upgrade everythign all at once :( [10:37:48] meh :-/ [10:47:42] <_joe_> rotfl [10:47:56] <_joe_> they're really a gift that gives on giving [10:49:13] and i naivly thought puppetlabs had matured and now understood the benefits of a smooth upgrade path [10:49:20] * jbond42 stupid john [10:55:18] the puppetdb host is selected via the puppetdb_host variable in Hiera, so we can probably setup the new puppetdb hosts instances in parallel and point the canaries which were designated for puppetmaster1003 to the new puppetdb hosts [10:56:12] although I'm wondering if that means to specifically only migrate to puppetdb 5 instead of 6 given their lockstep recommendation [10:58:01] moritzm: i was wondering the same thing in relation to puppet 6. goign through puppet docs and bugs now to see. Although i would have have expected bugs to have been raised in debian if they where not compatible? [10:59:15] that's a good point, given that Buster has puppetmaster 5 and Puppetdb 6 [10:59:26] eitherway the upgrade to puppet6 will add more pain. its not something i have looked at in depth but a lot of the things that are deprecated in 5 are removed in 6 and we still use some of them. Further the have removed some core types and migrated them to external modules which will cause pain the cron type is an example of this [10:59:52] also, probably using a higher version of puppetdb is probably less probeblematic that a higher version of the puppet masters [11:03:37] moritzm: in relation to the suggesttion above re spinning up a new puppetdb. this will work for simple tests but lots of things would not work specificly exported_resources (sshkey's, icinga) and anything that uses query_{resources,hosts,facts} [11:05:38] jbond42: ofc, because synchronous upgrades is even bettet than synchronous deploys :-P [11:08:44] in response to this 'Is there any way for 5.x servers to talk to 4.x puppetdb? That has been the way with the 3.x to 4.x upgrade, why the change in behaviour?' [11:08:59] we get 'I believe the upgrade from 3.x to 4.x coincidentally worked, but wasn't explicitly planned that way.' :facepalm: [11:17:06] in https://tickets.puppetlabs.com/browse/PUP-8901 is "Jean Bond" your Alter Ego because you don't want to file Puppet tasks under your real name? :-) [11:19:12] lol [11:19:52] the ironic thing the value thats causing us a problem i.e. job_id is set to `null` in every report [11:44:45] I had a look at the "source package" from the current puppetdb 4 package, but it's a 25MB Jar embedding lots of other classes and based on Clojure, seems unrealistic to patch out the job_id handling [11:49:29] if we went that route it would probably be easier to patch iot out of puppetdb-terminus or from puppet-master but we could just end up going down a rabbit hole constantly finding new issues [11:51:28] yeah [12:26:33] can we patch it at apache/nginx level? :-P [12:26:48] * volans half joking [13:08:07] volans: if we where to modify it on the fly then i think we would have to modify the stream between the puppet master and the puppetdb so it would have to be on the puppetdb nginx which means we could in theory do some lua hacking e.g. https://stackoverflow.com/a/22788730/3075306 [13:08:43] eheheh [13:08:53] thanks puppet [14:00:38] chaomodus, moritzm, jbond42: FYI I've made a new spicerack release yesteday and uploaded it to apt.w.o, but failed to find the time to test it. Is it ok if it stays there untile Monday? [14:01:04] sure thing [14:01:20] Basically don't upgrade it, but if it can be a problem I can downgrade it on apt.w.o [14:01:34] not expecting any reimages of Cumin hosts for the rest of the week :-) [14:01:47] ok then [14:14:45] volans moritzm fyi i managed to get puppetdb to process one of the new reports by removing the job_id paramter so hacking at nginx may actully work :S [14:16:39] cool :-) [14:20:52] rotfl [16:04:28] moritzm XioNoX joining the meeting ? [16:06:07] I'm off most of today [16:06:10] enjoy! [16:06:44] oh! ok enjoy XioNoX [16:18:17] http://docs.ganeti.org/ganeti/current/man/gnt-backup.html, but not comparable to what e.g. Vmware offers [16:20:32] thanks moritzm [18:50:40] FYI all this is what the nginx filter hack to fix the puppetdb schema thing could look like: https://gerrit.wikimedia.org/r/c/operations/puppet/+/528906 [18:51:50] i personaly think its safe to test but as to if its what we should use im undecided