[00:31:17] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[00:39:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4080501 (10EddieGP) The title for swift::init_device comes from a hiera lookup (hiera key `swift_storage_drives`) . Openstack browser shows this key is set to the value 'lv-a' on deployment-...
[00:46:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4094961 (10EddieGP) Related: T184236 and the attached patches.
[07:25:31] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[10:35:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4095116 (10MarcoAurelio) I've been looking into https://horizon.wikimedia.org/project/puppet/ and apparently I cannot do anything from there but to simply see. I am also unfamiliar with Pupp...
[10:38:32] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for shell access on deployment-prep - https://phabricator.wikimedia.org/T190925#4095118 (10MarcoAurelio) +1 from me. Eddie has proven to be helpful and knowledgeable. No concerns from me.
[10:54:08] <shinken-wm>	 PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130)
[10:54:52] <shinken-wm>	 PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211)
[11:43:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4095140 (10EddieGP) Actually this is a duplicate. After https://gerrit.wikimedia.org/r/#/c/361648/ the "/dev/swift/" part will be implicit as well as the trailing "1", and after https://gerr...
[11:43:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-ms-be03 - https://phabricator.wikimedia.org/T190683#4095142 (10EddieGP)
[11:43:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#4095145 (10EddieGP)
[12:37:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for access to the beta cluster - https://phabricator.wikimedia.org/T190755#4095176 (10EddieGP)
[12:40:12] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4095178 (10zeljkofilipin) >>! In T179188#4094387, @zeljkofilipin wrote: > Created testing job: [[ https://integration.wikimedia.org/ci/view/...
[12:43:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 3 others: Ferm/DNS library weirdness causing puppet errors on some deployment-prep instances - https://phabricator.wikimedia.org/T153468#4095180 (10EddieGP)
[12:48:06] <wikibugs>	 (03PS1) 10Hashar: Explain why we require composer dev dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423239
[12:48:08] <wikibugs>	 (03PS1) 10Hashar: Factor out code to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/423240
[12:54:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4095186 (10EddieGP) 05Open>03Resolved a:03EddieGP >>! In T132259#3879429, demon wrote: > Is this really best as a tracking task or should we add it to the...
[14:02:21] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994#4095222 (10zeljkofilipin)
[14:02:25] <wikibugs>	 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Investigate if WebdriverIO `sync: false` would be useful to us and document how to use it - https://phabricator.wikimedia.org/T182412#4095221 (10zeljkofilipin)
[14:02:27] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), and 2 others: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#4095223 (10zeljkofilipin)
[14:02:40] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994#4090304 (10zeljkofilipin)
[14:02:44] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), and 2 others: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822905 (10zeljkofilipin)
[14:53:16] <shinken-wm>	 PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76)
[15:11:57] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4095239 (10zeljkofilipin)
[15:48:42] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1003 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[17:04:46] <wikibugs>	 (03CR) 10Hashar: [C: 032] Explain why we require composer dev dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423239 (owner: 10Hashar)
[17:05:04] <wikibugs>	 (03CR) 10Hashar: [C: 032] Factor out code to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/423240 (owner: 10Hashar)
[17:05:13] <wikibugs>	 (03Merged) 10jenkins-bot: Explain why we require composer dev dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423239 (owner: 10Hashar)
[17:05:36] <wikibugs>	 (03PS2) 10Hashar: Factor out code to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/423240
[17:05:41] <wikibugs>	 (03CR) 10Hashar: [C: 032] Factor out code to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/423240 (owner: 10Hashar)
[17:06:07] <wikibugs>	 (03Merged) 10jenkins-bot: Factor out code to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/423240 (owner: 10Hashar)
[17:08:25] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4095296 (10zeljkofilipin)
[17:48:50] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#4095325 (10zeljkofilipin)
[18:34:37] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.004 second response time
[18:35:59] <shinken-wm>	 PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[18:54:29] <shinken-wm>	 PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:38:53] <shinken-wm>	 PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:39:36] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:41:28] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:41:31] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[19:47:24] <paladox>	 no_justification i found this https://gwtmaterialdesign.github.io/gwt-material-demo/#about lol
[19:47:37] * paladox wishes there was a polymer javascript version of that site :)
[19:47:46] <paladox>	 all the ui in there looks pretty :)
[20:02:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4095458 (10MarcoAurelio) Still erroring:  ``` maurelio@deployment-mira:~$ sudo puppet agent -tv Info: Using configured environment 'future' I...
[20:03:40] <Hauskatze>	 eddiegp: ^^ 
[20:03:51] <eddiegp>	 Hauskatze: give me a minute
[20:03:59] <Hauskatze>	 sure
[20:04:07] <paladox>	 puppetmaster may need to be updated
[20:04:12] <paladox>	 if there is merge conflicts
[20:05:47] <Hauskatze>	 ran on puppetmaster02 and no errors there
[20:07:07] <eddiegp>	 Umm, what? "Could not find data item"
[20:07:39] <eddiegp>	 That's the same thing as before, but that file it's now in is definitely part of the hiera for deployment-mira.
[20:07:47] <mutante>	 hallo Katze
[20:07:58] <mutante>	 du kannst es nochmal probieren aber der Puppenmeister braucht vielleicht sync :pp
[20:08:01] * mutante hides
[20:08:24] <paladox>	 i meant updating the git repo
[20:08:25] <mutante>	 and thanks Eddied too
[20:08:28] <paladox>	 on the puppetmaster
[20:08:35] <mutante>	 what paladox says :)
[20:08:37] <paladox>	  /var/lib/git/operations/puppet
[20:08:39] <eddiegp>	 Hmm, is the deployment-prep puppetmaster auto-fetching the operations/puppet repo at all? I guess not.
[20:08:45] * eddiegp is too slow
[20:08:52] <paladox>	 eddiegp it should
[20:08:59] <paladox>	 but won't if there are merge conflicts
[20:09:08] <Hauskatze>	 mutante: I'm tired of socks and puppets
[20:09:10] <Hauskatze>	 :P
[20:09:36] <mutante>	 Hauskatze: haha, ok :) but just follow paladox on this one :)
[20:09:42] <paladox>	 :)
[20:09:47] <Hauskatze>	 so _what_ needs to be done here?
[20:09:56] <paladox>	 git fetch origin
[20:09:58] <mutante>	 git pull  .. i think
[20:10:02] <paladox>	 and then git rebase origin
[20:10:03] <mutante>	 what he says 
[20:10:05] <Hauskatze>	 I've found http://shinken.wmflabs.org/host/labs-puppetmaster
[20:10:12] <paladox>	 git pull will block any future auto updates
[20:10:17] <Hauskatze>	 paladox: right, where? which server?
[20:10:18] <paladox>	 as i found in the past :)
[20:10:27] <mutante>	 good point. mayeb that is _exactly_ what happened
[20:10:29] <paladox>	 Hauskatze on the deployment-prep puppetmaster
[20:10:31] <mutante>	 somebody like me did git pull
[20:10:32] <mutante>	 hehe
[20:10:40] <paladox>	 and in /var/lib/git/operations/puppet
[20:10:52] <Hauskatze>	 paladox: okay, I'll check there
[20:10:53] <eddiegp>	 Or it's just some crazy cherry-pick, which I heard to be normal on deployment-prep ;)
[20:11:01] <mutante>	 yea, that too
[20:11:02] <paladox>	 yeh :)
[20:11:10] <mutante>	 probably it's both on top of each other
[20:11:33] <mutante>	 https://en.wikipedia.org/wiki/Turtles_all_the_way_down
[20:11:41] <paladox>	 heh
[20:11:45] <Hauskatze>	 paladox: var/lib/git has two folders
[20:11:52] <paladox>	 Hauskatze /var/lib/git/operations/puppet
[20:11:55] <Hauskatze>	 labs and operations
[20:11:55] <paladox>	 cd /var/lib/git/operations/puppet
[20:12:22] <Hauskatze>	 Your branch is ahead of 'origin/production' by 18 commits.
[20:12:22] <Hauskatze>	   (use "git push" to publish your local commits)
[20:12:22] <Hauskatze>	 nothing to commit, working tree clean
[20:12:24] <Hauskatze>	 heh
[20:12:40] <Hauskatze>	 git reset --hard origin/master && git pull origin master
[20:12:46] <Hauskatze>	 ?
[20:12:48] <eddiegp>	 Umm, no.
[20:12:51] <paladox>	 Hauskatze nope doint run git reset
[20:12:56] <eddiegp>	 That'd drop all the cherry-picks
[20:12:56] <paladox>	 run
[20:12:59] <paladox>	 git fetch origin
[20:13:00] <paladox>	 and then
[20:13:03] <paladox>	 git rebase origin
[20:13:23] <Hauskatze>	 maurelio@deployment-puppetmaster02:/var/lib/git/operations/puppet$ git fetch origin
[20:13:23] <Hauskatze>	 error: cannot open .git/FETCH_HEAD: Permission denied
[20:13:24] <Hauskatze>	 lol
[20:13:29] <paladox>	 sudo 
[20:13:46] <eddiegp>	 https://xkcd.com/149/
[20:13:51] <eddiegp>	 ;)
[20:13:57] <paladox>	 lol
[20:14:18] <Hauskatze>	 maurelio@deployment-puppetmaster02:/var/lib/git/operations/puppet$ sudo git rebase origin
[20:14:18] <Hauskatze>	 Current branch production is up to date.
[20:14:28] <mutante>	 sudo make me a deployment burger
[20:14:37] <Hauskatze>	 after sudo git fetch origin
[20:15:04] <Hauskatze>	 still:
[20:15:06] <Hauskatze>	 maurelio@deployment-puppetmaster02:/var/lib/git/operations/puppet$ sudo git status
[20:15:06] <Hauskatze>	 On branch production
[20:15:06] <Hauskatze>	 Your branch is ahead of 'origin/production' by 18 commits.
[20:15:06] <Hauskatze>	   (use "git push" to publish your local commits)
[20:15:06] <Hauskatze>	 nothing to commit, working tree clean
[20:15:13] <paladox>	 yep that's ok :)
[20:15:19] <paladox>	 so hmm, it's current
[20:15:21] <mutante>	 what does git log show on top?
[20:15:30] <mutante>	 does it show the Hiera change?
[20:15:47] <paladox>	 aha
[20:15:54] <paladox>	 it's asking for the group now
[20:16:01] <mutante>	 ;) 
[20:16:02] <paladox>	 profile::kubernetes::deployment_server::git_owner: trebuchet
[20:16:04] <paladox>	 is one fix
[20:16:06] <paladox>	 now it wants
[20:16:13] <paladox>	 profile::kubernetes::deployment_server::git_group
[20:16:14] <Hauskatze>	 maurelio@deployment-puppetmaster02:/var/lib/git/operations/puppet$ sudo git log
[20:16:14] <Hauskatze>	 commit 82af816b705f6a103b165408adb5f02aa8f09652
[20:16:14] <Hauskatze>	 Author: Stephane Bisson <sbisson@wikimedia.org>
[20:16:14] <Hauskatze>	 Date:   Tue Mar 27 15:23:23 2018 -0400
[20:16:14] <Hauskatze>	     Make 'style' and 'storage id' available to maps services
[20:16:14] <Hauskatze>	     Make 'style' and 'storage id' variables available as config
[20:16:16] <mutante>	 you fixed one bug and then see the next. situation normal
[20:16:16] <Hauskatze>	     vars for kartotherian, tilerator, and tileratorui.
[20:16:18] <Hauskatze>	     Bug: T112948
[20:16:19] <stashbot>	 T112948: All map location names should be in the user's language - https://phabricator.wikimedia.org/T112948
[20:16:20] <Hauskatze>	     Change-Id: Iec7a99a7360d71c8fd57545d41800474f78e5c52
[20:16:20] <mutante>	 but progress
[20:16:27] <paladox>	 Hauskatze we know what's wrong now
[20:16:35] <Hauskatze>	 aleluya
[20:16:38] <paladox>	 lol
[20:16:46] <paladox>	 profile::kubernetes::deployment_server::git_group
[20:16:50] <paladox>	 eddiegp ^^
[20:16:50] <eddiegp>	 Well, top of git log will show the cherry-picks anyway
[20:16:52] <mutante>	 that's exactly why i wanted to show how to fix it for next time too 
[20:16:56] <mutante>	 heh, but nice
[20:17:45] <Hauskatze>	 eddiegp change is not on git log mutante 
[20:17:54] <paladox>	 wikidev
[20:17:59] <paladox>	 i guess it should be ^^
[20:18:06] <eddiegp>	 Hauskatze: Sure? Should be the 19th from top, so scroll a bit ;)
[20:18:08] <paladox>	 https://github.com/wikimedia/puppet/blob/3a2551ce098b17259b12baea6e683486df8dd28c/hieradata/role/common/deployment_server.yaml#L190
[20:18:26] <mutante>	 i expected it on top but there are 18 others above it because they are cherry pciked...
[20:18:29] <mutante>	 meh
[20:18:51] <Hauskatze>	 nope, it is not there
[20:19:12] <paladox>	 cherry picks are at the top
[20:19:12] <mutante>	 weird.. since the puppet run says otherwise ?
[20:19:56] <Hauskatze>	 running git log -n50 --oneline
[20:20:03] <eddiegp>	 paladox: Where you've got the git_group error message from?
[20:20:11] <Hauskatze>	 167f282b8d hiera: fix deployment-mira, lacking ::git_owner
[20:20:17] <paladox>	 eddiegp from https://phabricator.wikimedia.org/T191110#4095458
[20:20:29] <mutante>	 if you test something on this you are really testing all these 18 cherry-picks and not what production is ...
[20:20:48] <paladox>	 mutante https://gerrit.wikimedia.org/r/#/c/423256/
[20:21:18] <eddiegp>	 Well yeah, the puppet master was updated, I just misread the error message to be still the old one. My bad :)
[20:22:04] <mutante>	 ok, merged on prod master
[20:22:06] <paladox>	 eddiegp Hauskatze fixed in https://gerrit.wikimedia.org/r/c/423256/
[20:22:07] <mutante>	 you should repeat this
[20:22:15] <mutante>	 fetch the next fix
[20:22:18] <mutante>	 and run puppet
[20:22:22] <Hauskatze>	 git fetch origin && git rebase origin ??
[20:22:26] <paladox>	 yep
[20:22:31] <Hauskatze>	 okay...
[20:23:03] <eddiegp>	 'git pull --rebase' is shorter and does the same in one step btw
[20:23:18] <mutante>	 how many hosts are there starting with deployment- in host name?
[20:23:54] <eddiegp>	 mutante: 74
[20:23:58] <mutante>	 lol?
[20:24:07] <Hauskatze>	 madness
[20:24:18] <mutante>	 is this "mira" one special?
[20:24:21] <eddiegp>	 According to the "instances" list on https://tools.wmflabs.org/openstack-browser/project/deployment-prep
[20:24:29] <mutante>	 because production doesnt have "mira" anymore
[20:24:47] <mutante>	 is it used?
[20:24:55] <eddiegp>	 Yes, it is.
[20:25:00] <mutante>	 ok
[20:25:03] <Hauskatze>	 okay so I ran the git fetch/git rebase thing
[20:25:15] <Hauskatze>	 doing sudo puppet agent -tv now
[20:25:36] <eddiegp>	 deployment-prep also uses deployment-tin and will continue to do so even if there's no "tin" in prod any more
[20:26:12] <Hauskatze>	 sudo puppet agent -tv on puppetmaster -> okay
[20:26:16] <Hauskatze>	 now on mira
[20:27:25] <Hauskatze>	 no error now
[20:27:34] <Hauskatze>	 but a bunch of things I'll paste
[20:28:07] <paladox>	 also
[20:28:12] <paladox>	 deployment-tin puppet is failing
[20:28:14] <paladox>	 [20:38:53]  <shinken-wm>	PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[20:28:57] <Hauskatze>	 https://phabricator.wikimedia.org/P6917
[20:30:54] <Hauskatze>	 no 'recovery' message from shinken?
[20:31:05] <Hauskatze>	 I did this for the bot message :|
[20:31:07] <Hauskatze>	 :P
[20:31:16] <eddiegp>	 That paste looks like a normal puppet run to me :)
[20:31:29] <eddiegp>	 Shinken runs every 5-10min ish, so just wait for it ;)
[20:31:34] <paladox>	 yep
[20:31:49] <paladox>	 Hauskatze oh i wonder why deployment-tin showed as failing?
[20:31:50] <Hauskatze>	 on -tin
[20:31:53] <Hauskatze>	 The last Puppet run was at Thu Mar 29 14:14:27 UTC 2018 (3257 minutes ago).
[20:32:02] <Hauskatze>	 now looking there
[20:32:19] <paladox>	 oh
[20:32:30] <paladox>	 is it the same error as mira
[20:32:59] <Hauskatze>	 Error: Error while evaluating a Function Call, Could not find data item profile::kubernetes::deployment_server::git_owner in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/kubernetes/deployment_server.pp:5:16 on node deployment-tin.deployment-prep.eqiad.wmflabs
[20:33:16] <paladox>	 yep same error
[20:33:19] <Hauskatze>	 diving into var lib git operations puppet
[20:33:26] <eddiegp>	 Yeah, we can copy what we added to deployment-mira to deployment-tin. Or just move it to some common file.
[20:33:38] <eddiegp>	 Probably the latter is a better idea.
[20:34:34] <Hauskatze>	 paladox: there's a problem as there's no operations/puppet on deployment-tin on var/lib/git
[20:34:46] <mutante>	 yes, avoid using hieradata/hosts/
[20:34:51] <paladox>	 Hauskatze mutante https://gerrit.wikimedia.org/r/#/c/423257/
[20:34:51] <mutante>	 avoid using host names from prod
[20:34:58] <mutante>	 try using hieradata/role/
[20:35:02] <mutante>	  ./common/
[20:35:07] <paladox>	 Hauskatze yep
[20:35:12] <paladox>	 that is only on the puppet master
[20:35:15] <paladox>	 not on the clients
[20:35:27] <eddiegp>	 mutante: labs has no role-based hiera lookup
[20:35:39] <eddiegp>	 Just common, project specific, host specific
[20:35:49] <mutante>	 and prefix
[20:35:52] <mutante>	 by hostname
[20:35:59] <mutante>	 you can use deployment-prefix ?
[20:36:00] <paladox>	 fixed tin in https://gerrit.wikimedia.org/r/#/c/423257/
[20:36:26] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
[20:36:31] <Hauskatze>	 hah
[20:36:33] <eddiegp>	 l
[20:37:12] <Hauskatze>	 still I'm seeing a critical error: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since 4M 2w 5d 1h 34m 24s
[20:37:21] <eddiegp>	 paladox: Really let's not have the same line in both the deployment-mira.yaml and deployment-tin.yaml. move it to deployment-prep/common.yaml instead and get rid of it in -mira
[20:37:24] <mutante>	 ok, i will merge that one more change for tin
[20:37:27] <mutante>	 but then i'm also out 
[20:37:33] <paladox>	 ok
[20:37:35] <mutante>	 and i dont see a point in  doing this for 74 hosts
[20:38:26] <eddiegp>	 deployment-prep has 74 hosts in total. It doesn't have 74 deployment servers, so there's no need to do this for all hosts ;)
[20:38:43] <paladox>	 done
[20:39:09] <mutante>	 just dont expect me to have time to merge this kind of thing for each VPS that gets created becaues we hardcode the host names
[20:40:45] <paladox>	 mutante this https://gerrit.wikimedia.org/r/#/c/423257/ should fix it for other deployment hosts too
[20:41:01] <paladox>	 deployment-prep/common.yaml
[20:41:40] <Hauskatze>	 ready to run stuff when told
[20:41:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4095478 (10Paladox)
[20:41:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4095477 (10Paladox) 05Open>03Resolved
[20:42:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4093928 (10Paladox)
[20:42:06] <mutante>	 first i'm merging my own actual change that i came here for
[20:42:14] <mutante>	 now i'm rebasing that
[20:42:38] <mutante>	 Hauskatze: done
[20:42:59] <Hauskatze>	 shall I git fetch etc on puppetmaster again?
[20:43:23] <eddiegp>	 Well hiera in labs is just insane, being splitted across wikitech, horizon, operations/puppet and labs/private. I'd be all in for killing three of those, preferably storing everything in horizon and not needing a operations/puppet merge to change labs hiera.
[20:43:55] <Hauskatze>	 paladox: fetch and rebase on puppetmaster again?
[20:44:41] <mutante>	 i abandoned your change because i already did that before seeing it
[20:44:44] <mutante>	 just to clean up
[20:45:01] <paladox>	 Hauskatze yep
[20:45:20] <Hauskatze>	 mutante: no problems, thanks
[20:46:01] <Hauskatze>	 done
[20:46:07] <mutante>	 ok guys, i have to get off the bus
[20:46:12] <mutante>	 which means i lose my wifi
[20:46:17] <mutante>	 which is part of the bus :P
[20:46:26] <Hauskatze>	 thanks mutante -- see you later :)
[20:46:29] <mutante>	 cya
[20:46:31] <paladox>	 heh :)
[20:46:35] * paladox has to go too
[20:46:39] <eddiegp>	 thanks mutante :)
[20:46:41] <mutante>	 paladox knows i like to us busses
[20:46:43] <mutante>	 and work from them
[20:46:48] <paladox>	 yeh heh
[20:46:51] * mutante waves
[20:47:20] <Hauskatze>	 puppet ran on deployment-tin
[20:48:20] <Hauskatze>	 https://phabricator.wikimedia.org/P6918
[20:49:30] <eddiegp>	 Looks all good, just waiting for shinken again
[20:49:48] <Hauskatze>	 deployment-eventlog05/Puppet staleness <-- I think we can fix this with sudo puppet agent -tv
[20:50:23] <Hauskatze>	 now a silly question: are we suposed to do everyday the fetch and rebase on the puppetmaster server?
[20:50:41] <paladox>	 nope
[20:50:57] <eddiegp>	 paladox said it should auto-update, I don't know how it does though.
[20:51:01] <paladox>	 it should refresh by it's self as long as there is no merge conflicts
[20:51:10] <paladox>	 every 30 mins
[20:51:20] <paladox>	 i think
[20:53:29] <Hauskatze>	 evenlog05 error is different
[20:55:26] <Hauskatze>	 tin is now down from 100% to CRITICAL: 30.00% of data above the critical threshold [0.0]
[20:58:56] <shinken-wm>	 RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0]
[20:59:02] <Hauskatze>	 :)
[21:03:48] <mutante>	 i'm back .. hanging out behind the railway station. but as logn as that train doesn't leave i can leech oif Amtrak wifi 
[21:04:19] <mutante>	 nice to see that recovery :)
[21:04:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx - https://phabricator.wikimedia.org/T191151#4095484 (10MarcoAurelio)
[21:05:28] <Hauskatze>	 my house cat demands that I feed him now
[21:05:35] <Hauskatze>	 so brb
[21:05:53] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0]
[21:05:54] <eddiegp>	 Hauskatzes Hauskatze :D
[21:05:56] <mutante>	 geb der Katze Futter
[21:06:09] <Hauskatze>	 ah oh, another recovery
[21:06:23] <Hauskatze>	 but.. I did sudo puppet agent disable there
[21:06:25] <mutante>	 surprise recoveries are the best recoveries
[21:06:27] <Hauskatze>	 leaving it as it was
[21:06:31] <mutante>	 yea, you should call the cat "Meta"
[21:06:42] <Hauskatze>	 it's called 'Miko'
[21:06:44] <mutante>	 Hauskatze of Hauskatze is a good one
[21:07:02] <mutante>	 Neko Mimi
[21:08:22] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:08:23] <mutante>	 regarding that recoery on deploynment-eventlog .. thanks to using common.yaml ;)
[21:08:34] <mutante>	 more coming in, nice
[21:08:47] <Hauskatze>	 mx02
[21:09:07] <Hauskatze>	 I guess I can reenable the puppet there on eventlog
[21:12:36] <Hauskatze>	 mx02 looks like some letsencrypt stuff
[21:13:33] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx - https://phabricator.wikimedia.org/T191151#4095500 (10EddieGP)
[21:13:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#4095503 (10EddieGP)
[21:14:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx02 due to some Letsencrypt stuff - https://phabricator.wikimedia.org/T191152#4095504 (10MarcoAurelio)
[21:16:56] <eddiegp>	 Hauskatze: See Krenairs sentence "Created a new system, ran into the problem that https://gerrit.wikimedia.org/r/#/c/403326/ fixes" on T184244. I guess T191152 is just another duplicate to that one.
[21:16:57] <stashbot>	 T191152: Puppet broken on deployment-mx02 due to some Letsencrypt stuff - https://phabricator.wikimedia.org/T191152
[21:16:57] <stashbot>	 T184244: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244
[21:17:19] <Hauskatze>	 oh, sorry for duplicating then
[21:17:38] <eddiegp>	 No problem, I'll just merge this one as well (already did for the first)1
[21:18:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx02 due to some Letsencrypt stuff - https://phabricator.wikimedia.org/T191152#4095516 (10EddieGP)
[21:18:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#4095518 (10EddieGP)
[21:19:14] <Hauskatze>	 deployment-etcd-01 should be fixed too
[21:20:32] <Hauskatze>	 or not
[21:20:33] <Hauskatze>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::etcd::tlsproxy::listen_port in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/etcd/tlsproxy.pp:6:35 on node deployment-etcd-01.deployment-prep.eqiad.wmfla
[21:20:34] <Hauskatze>	 bs
[21:20:51] * Hauskatze headesks
[21:23:48] <Hauskatze>	 eddiegp: that's not reported yet I think
[21:23:56] <Hauskatze>	 the tlsproxy.pp stuff
[21:23:58] * eddiegp looking
[21:24:09] <Hauskatze>	 profile::etcd::tlsproxy::listen_port
[21:24:16] <Hauskatze>	 no idea what's that
[21:24:59] * eddiegp laughs
[21:25:02] <eddiegp>	 T191107
[21:25:03] <stashbot>	 T191107: deployment-etcd-01 puppet errors - https://phabricator.wikimedia.org/T191107
[21:25:08] <Hauskatze>	 https://github.com/search?q=org%3Awikimedia+profile%3A%3Aetcd%3A%3Atlsproxy%3A%3Alisten_port&type=Code
[21:25:08] <eddiegp>	 And you were the one to report that :D
[21:25:42] * Hauskatze ashamed
[21:26:12] <eddiegp>	 have to run
[21:26:35] <Hauskatze>	 ok, me too
[21:28:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#4095534 (10MarcoAurelio)
[21:28:26] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: deployment-etcd-01 puppet errors - https://phabricator.wikimedia.org/T191107#4095535 (10MarcoAurelio) https://github.com/search?q=org%3Awikimedia+profile%3A%3Aetcd%3A%3Atlsproxy%3A%3Alisten_port&type=Code
[21:37:14] <eddiegp>	 Actually back, that alert turned out to be nothing.
[21:37:56] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: deployment-etcd-01 puppet errors - https://phabricator.wikimedia.org/T191107#4095540 (10MarcoAurelio) @Paladox Should we add `profile::etcd::tlsproxy::listen_port: <number>` to https://github.com/wikimedia/puppet/blob/production/hieradata/labs/deployment-prep/host/deplo...
[21:40:08] <mutante>	 Hauskatze: sup
[21:40:44] <mutante>	 ok, laters
[21:41:34] <Hauskatze>	 I'm here
[21:41:39] <Hauskatze>	 maps03 fixed
[21:41:49] <Hauskatze>	 puppet was disabled and now catalog updated
[21:42:18] <Hauskatze>	 !log Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
[21:42:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:43:20] <Hauskatze>	 and critical threshold is lowering now
[21:50:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka04 - https://phabricator.wikimedia.org/T191154#4095542 (10MarcoAurelio)
[21:52:18] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [3600.0]
[21:52:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: Puppet broken on deployment-mira - https://phabricator.wikimedia.org/T191110#4093928 (10Dzahn) this also fixed puppet runs on a bunch of other deployment-* hosts thanks to using common.yaml instead of ./hosts/  bonus token for that ! thanks
[21:54:24] <Hauskatze>	 mutante: maps03 fixed as well :D
[21:54:43] <mutante>	 edited my comemnt
[21:58:09] <Hauskatze>	 heh 17:52
[21:58:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka04 - https://phabricator.wikimedia.org/T191154#4095555 (10EddieGP) Role deleted from puppet git by @Ottomata in 661eea7bda, but still applied to deployment-kafka04 according to https://tools.wmfl...
[21:58:15] <Hauskatze>	 23:58 here
[22:04:02] <Hauskatze>	 CRITICAL: deployment-puppetmaster02/Long lived cherry-picks on puppetmaster
[22:04:10] <Hauskatze>	 shot shot shot
[22:08:35] <eddiegp>	 That one probably wins the "alert no-one cares about" award of the year.
[22:09:26] <Hauskatze>	 hopefully your request will be approved and so someone with knowledge about friking puppet can help there
[22:10:29] <eddiegp>	 I'm optimistic about that :)
[22:14:00] <Hauskatze>	 :)
[22:14:58] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4095558 (10EddieGP)
[22:15:08] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4095542 (10EddieGP) Affects deployment-kafka05 as well.
[22:26:06] <mutante>	 just make tickets where you ask if that isntance is even used and if not then andrew will be happy to delete it and the thing can be removed
[22:26:30] <mutante>	 based on "role deleted in production" that seems pretty likely
[22:27:00] <mutante>	 and you save the time of dealing with shinken details
[22:27:12] <wikibugs>	 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4019041 (10KATMAKROFAN) Why can't we just merge it into Meta-Wiki?
[22:40:47] <eddiegp>	 true, will do that in the future. I've already pinged ottama in that ticket, so he'll probably tell if that instance is useless.
[23:00:21] <wikibugs>	 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4095598 (10KATMAKROFAN) After renaming foundationwiki, we should enable use of LDAP accounts on there.