[00:56:28] <shinken-wm>	 PROBLEM - Puppet failure on wmfbranch is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[01:01:59] <grrrit-wm>	 (03PS3) 10Legoktm: [FEAT] Report multiple errors on each line [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247047 (owner: 10XZise)
[01:02:05] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] [FEAT] Report multiple errors on each line [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247047 (owner: 10XZise)
[01:02:52] <wikibugs>	 10Deployment-Systems, 6operations: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585#1883535 (10Dzahn) after taking a look at tin and mira (and finding more inconsistencies and "system" users with UIDs over 10000) i suggest we do  UID 120 for l10nupdate
[01:02:53] <grrrit-wm>	 (03Merged) 10jenkins-bot: [FEAT] Report multiple errors on each line [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247047 (owner: 10XZise)
[01:04:31] <grrrit-wm>	 (03PS6) 10Legoktm: [FEAT] Require phabricator bug ids [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247048 (owner: 10XZise)
[01:06:03] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] [FEAT] Require phabricator bug ids [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247048 (owner: 10XZise)
[01:06:50] <grrrit-wm>	 (03Merged) 10jenkins-bot: [FEAT] Require phabricator bug ids [integration/jenkins] - 10https://gerrit.wikimedia.org/r/247048 (owner: 10XZise)
[01:15:48] <wikibugs>	 10Continuous-Integration-Infrastructure: Move commit-message-validator.py tool out of integration/jenkins repository - https://phabricator.wikimedia.org/T121609#1883558 (10Legoktm) 3NEW
[01:17:48] <wikibugs>	 10Deployment-Systems, 6operations, 5Patch-For-Review: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585#1856865 (10Dzahn)
[01:31:26] <shinken-wm>	 RECOVERY - Puppet failure on wmfbranch is OK: OK: Less than 1.00% above the threshold [0.0]
[01:57:28] <shinken-wm>	 PROBLEM - Puppet failure on wmfbranch is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[02:37:31] <shinken-wm>	 RECOVERY - Puppet failure on wmfbranch is OK: OK: Less than 1.00% above the threshold [0.0]
[04:48:45] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:48:45] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:43] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:43] <testing-shinken->	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:44] <testing-shinken->	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:45] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:47] <testing-shinken->	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:47] <testing-shinken->	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:48] <testing-shinken->	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:49] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:33:23] <shinken-wm>	 PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[08:33:23] <testing-shinken->	 PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[08:33:41] <legoktm>	 ^ labs nfs issues, everything is down
[08:33:44] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #286: 04FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/286/
[08:33:53] <testing-shinken->	 RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0]
[08:33:53] <testing-shinken->	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:33:53] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:33:54] <shinken-wm>	 RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0]
[08:34:17] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #838: 04FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/838/
[08:34:39] <wmf-insecte>	 Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3154: 04FAILURE in 1 hr 11 min: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3154/
[08:34:40] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #815: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/815/
[08:34:43] <wmf-insecte>	 Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #798: 04FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/798/
[08:34:48] <testing-shinken->	 PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[08:34:48] <testing-shinken->	 PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[08:34:48] <testing-shinken->	 PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[08:34:48] <testing-shinken->	 PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]
[08:34:49] <testing-shinken->	 PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[08:34:49] <testing-shinken->	 PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[08:34:50] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39736 bytes in 0.863 second response time
[08:34:50] <testing-shinken->	 PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[08:34:50] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.938 second response time
[08:34:50] <testing-shinken->	 PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[08:34:50] <testing-shinken->	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.887 second response time
[08:34:50] <testing-shinken->	 PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[08:34:50] <testing-shinken->	 PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[08:34:50] <testing-shinken->	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.876 second response time
[08:34:50] <testing-shinken->	 PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[08:34:56] <testing-shinken->	 PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[08:35:02] <testing-shinken->	 PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[08:35:34] <testing-shinken->	 PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[08:35:46] <testing-shinken->	 PROBLEM - Puppet failure on deployment-tmh01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[08:35:48] <testing-shinken->	 PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[08:35:57] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.876 second response time
[08:35:57] <testing-shinken->	 PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0]
[08:36:35] <testing-shinken->	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39737 bytes in 1.718 second response time
[08:36:35] <testing-shinken->	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39411 bytes in 1.389 second response time
[08:37:02] <testing-shinken->	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31061 bytes in 0.975 second response time
[08:37:10] <testing-shinken->	 PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]
[08:37:34] <testing-shinken->	 PROBLEM - Puppet failure on mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[08:38:13] <shinken-wm>	 PROBLEM - Puppet failure on deployment-kafka04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[08:38:51] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39736 bytes in 0.863 second response time
[08:38:53] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39413 bytes in 1.162 second response time
[08:40:29] <testing-shinken->	 PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[08:40:39] <testing-shinken->	 PROBLEM - Puppet failure on deployment-conf03 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]
[08:40:53] <testing-shinken->	 PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]
[08:43:11] <testing-shinken->	 PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[08:45:47] <testing-shinken->	 PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [0.0]
[08:48:03] <testing-shinken->	 RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.056 second response time
[08:49:30] <shinken-wm>	 RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.044 second response time
[08:53:58] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:53:58] <testing-shinken->	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:53:58] <testing-shinken->	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:53:58] <testing-shinken->	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:57:01] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.460 second response time
[08:57:32] <testing-shinken->	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39744 bytes in 0.584 second response time
[08:57:32] <testing-shinken->	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39410 bytes in 0.545 second response time
[08:58:01] <testing-shinken->	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31061 bytes in 0.533 second response time
[09:01:29] <testing-shinken->	 RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:02:54] <testing-shinken->	 RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:10:48] <testing-shinken->	 RECOVERY - Puppet failure on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:13:56] <testing-shinken->	 RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:15:11] <testing-shinken->	 RECOVERY - Puppet failure on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:15:51] <testing-shinken->	 RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:17:14] <testing-shinken->	 RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:20:31] <testing-shinken->	 RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:20:55] <testing-shinken->	 RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0]
[09:22:48] <testing-shinken->	 RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
[09:23:13] <shinken-wm>	 RECOVERY - Puppet failure on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:24:28] <testing-shinken->	 RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:25:36] <testing-shinken->	 RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:25:48] <testing-shinken->	 RECOVERY - Puppet failure on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:25:48] <testing-shinken->	 RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:27:02] <testing-shinken->	 RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:27:25] <testing-shinken->	 RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:27:41] <testing-shinken->	 RECOVERY - Puppet failure on mira is OK: OK: Less than 1.00% above the threshold [0.0]
[09:28:06] <testing-shinken->	 RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0]
[09:29:54] <testing-shinken->	 RECOVERY - Puppet failure on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0]
[09:30:55] <testing-shinken->	 RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:34:34] <testing-shinken->	 RECOVERY - Puppet failure on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:38:23] <wmf-insecte>	 Yippee, build fixed!
[09:38:24] <wmf-insecte>	 Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #705: 09FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/705/
[10:16:04] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Where should we branch for Wikimedia wikis? - https://phabricator.wikimedia.org/T121570#1884027 (10hashar) We would need to speed up the make-wmf-branch. From a quick glance at the code it:  - `git clone` everything afresh from Gerrit or `git pull` if --path has...
[10:20:55] <grrrit-wm>	 (03PS1) 10Hashar: Revert "Deleted failing Flow browsertests Jenkins job" [integration/config] - 10https://gerrit.wikimedia.org/r/259470 
[10:20:59] <grrrit-wm>	 (03PS2) 10Hashar: Revert "Deleted failing Flow browsertests Jenkins job" [integration/config] - 10https://gerrit.wikimedia.org/r/259470 
[10:23:34] <grrrit-wm>	 (03PS3) 10Hashar: Restore Flow browser tests on Win8.0/IE 10 [integration/config] - 10https://gerrit.wikimedia.org/r/259470 
[10:24:14] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "Need the gem to be bumped in Flow https://gerrit.wikimedia.org/r/#/c/257338/" [integration/config] - 10https://gerrit.wikimedia.org/r/259470 (owner: 10Hashar)
[10:28:10] <grrrit-wm>	 (03CR) 10Hashar: "With mediawiki_selenium >=1.6.3 , we can restore the IE/Android jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/242880 (https://phabricator.wikimedia.org/T94151) (owner: 10Zfilipin)
[11:03:26] <wikibugs>	 7Browser-Tests, 10VisualEditor, 5Patch-For-Review: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1884172 (10zeljkofilipin) a:3zeljkofilipin
[11:17:13] <testing-shinken->	 PROBLEM - Host deployment-cache-parsoid04 is DOWN: CRITICAL - Host Unreachable (10.68.19.197)
[11:36:28] <hashar>	 zeljkof: good morning :-}
[11:36:49] <hashar>	 zeljkof: so Flow can probably get IE added :-}
[11:36:52] <zeljkof>	 hashar: morning!
[11:36:58] <zeljkof>	 will test that today
[11:37:23] <zeljkof>	 need to pay a bill or two and back to T114241
[11:37:46] <hashar>	 if you get time this afternoon, I would love to discuss about CentralAuth failing ( https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ ).  Somehow it does not send the mediawiki_user  but some previous value of `user` variable :-(
[11:38:00] <hashar>	 and I can't reproduce locally (i.e.: it just works for me)
[11:38:32] <zeljkof>	 hashar: I should be working on that in 30-60 minutes
[11:38:44] <hashar>	 about to head to lunch 
[11:38:57] <hashar>	 I am really curious about CentralAuth, seems to be a low hanging fruit to get it green again
[11:39:14] <hashar>	 and there is most probably a bug in the way we run the tests on CI
[11:41:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1884245 (10hashar) That...
[11:43:08] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1884251 (10hashar)
[11:44:39] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#747903 (10hashar)
[11:45:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1884276 (10hashar) a:5...
[11:57:43] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta not picking up merged change - https://phabricator.wikimedia.org/T75659#1884288 (10hashar)
[11:57:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1884290 (10hashar)
[12:19:10] <grrrit-wm>	 (03PS1) 10Paladox: [BlueSpiceFoundation] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/259481 
[12:19:54] <paladox>	 hashar: Could you merge https://gerrit.wikimedia.org/r/#/c/259481/ please.
[12:22:26] <hashar>	 paladox: will look in a few; digging logs right now :(
[12:26:36] <paladox>	 hashar: Ok thanks.
[12:34:36] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] [BlueSpiceFoundation] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/259481 (owner: 10Paladox)
[12:34:44] <grrrit-wm>	 (03PS1) 10Phedenskog: If one batch fails for WPT, fail the whole job. [integration/config] - 10https://gerrit.wikimedia.org/r/259484 (https://phabricator.wikimedia.org/T120365) 
[12:34:44] <hashar>	 paladox: great thank you :-)
[12:36:18] <grrrit-wm>	 (03Merged) 10jenkins-bot: [BlueSpiceFoundation] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/259481 (owner: 10Paladox)
[12:38:03] <grrrit-wm>	 (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/259481 (owner: 10Paladox)
[12:38:17] <grrrit-wm>	 (03CR) 10Hashar: "Deployed. Did a recheck on the last merged change https://gerrit.wikimedia.org/r/#/c/255446/ and it is all green:" [integration/config] - 10https://gerrit.wikimedia.org/r/259481 (owner: 10Paladox)
[12:38:21] <hashar>	 paladox: all good thanks!
[12:38:28] <paladox>	 paladox: Ok thanks.
[12:46:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 7WorkType-Maintenance: ContentTranslation phpunit run very slow due to inclusion of Scribunto and Wikibase - https://phabricator.wikimedia.org/T121595#1884317 (10Amire80) p:5Triage>3Normal
[12:46:35] <wmf-insecte>	 Yippee, build fixed!
[12:46:36] <wmf-insecte>	 Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3155: 09FIXED in 34 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3155/
[12:49:10] <grrrit-wm>	 (03PS2) 10Hashar: If one batch fails for WPT, fail the whole job. [integration/config] - 10https://gerrit.wikimedia.org/r/259484 (https://phabricator.wikimedia.org/T120365) (owner: 10Phedenskog)
[12:49:46] <hashar>	 phedenskog: do you know how to deploy Jenkins jobs ? :-}  would be happy to introduce you to the magic
[12:52:12] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Tested on a Trusty instance with bash 4.3.11. I have refreshed both jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/259484 (https://phabricator.wikimedia.org/T120365) (owner: 10Phedenskog)
[12:53:50] <grrrit-wm>	 (03Merged) 10jenkins-bot: If one batch fails for WPT, fail the whole job. [integration/config] - 10https://gerrit.wikimedia.org/r/259484 (https://phabricator.wikimedia.org/T120365) (owner: 10Phedenskog)
[12:54:50] <wmf-insecte>	 Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #691: 04FAILURE in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/691/
[12:58:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 7WorkType-Maintenance: ContentTranslation phpunit run very slow due to inclusion of Scribunto and Wikibase - https://phabricator.wikimedia.org/T121595#1884328 (10Amire80) > I'm not sure why this extension is loading these two...
[13:15:55] <hashar>	 !log Gerrit , on mediawiki/services/mathoid force pushed gh-pages branch from Github to Gerrit repo . Attempting to fix Gerrit replication issue ( https://phabricator.wikimedia.org/T121635 )
[13:15:59] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:18:00] <hashar>	 !log Gerrit created https://github.com/wikimedia/operations-debs-bloomd | https://phabricator.wikimedia.org/T121635
[13:18:04] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:18:57] <hashar>	 !log Gerrit created https://github.com/wikimedia/thumbor-svg-engine | https://phabricator.wikimedia.org/T121635
[13:19:00] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:37:43] <Nikerabbit>	 hashar: there is nothing like list of unit tests in order of time spent, to kill some low hanging fruit?
[13:48:38] <testing-shinken->	 PROBLEM - Puppet failure on deployment-conf03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:54:44] <hashar>	 Nikerabbit: hey
[13:55:02] <hashar>	 Nikerabbit: the Jenkins job runs phpunit with JUnit output which capture the time to run each test
[13:55:16] <hashar>	 Nikerabbit: that is parsed by Jenkins and show up on each build under 'Test results'
[13:55:48] <hashar>	 so from https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/9524/console
[13:55:56] <hashar>	 on the left side bar click 'Test Results'
[13:56:20] <hashar>	 you get a summary of all tests with a '(root)'  link that gives further details
[13:56:44] <hashar>	 ie https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/9524/testReport/(root)/
[13:56:49] <hashar>	 you can then sort by "Duration"
[13:57:17] <hashar>	 For Zend, I have extracted the parsertests to their own Jenkins job https://integration.wikimedia.org/ci/job/mediawiki-phpunit-parsertests-zend/250/testReport/(root)/
[13:57:52] <Nikerabbit>	 hashar: wow
[13:57:58] <hashar>	 there one can see that a test takes at a minimum 140ms and we have more than 1300
[13:58:16] <hashar>	 a bunch of them takes 200-300 ms
[13:58:33] <hashar>	 one of the trouble is the overhead caused by our phpunit wrapper 
[13:58:45] <hashar>	 we do a lot of things before/after each parser tests that slow down stuff :-(
[13:58:54] <Nikerabbit>	 hashar: https://integration.wikimedia.org/ci/job/mwext-testextension-zend/17047/testReport/(root)/ here the thing to blame is very obvious
[13:59:14] <hashar>	 Nikerabbit: yup
[13:59:35] <hashar>	 Nikerabbit: though the tests probably consume a dataprovider
[13:59:44] <hashar>	 so it is really a lot more tests being run behind the hood
[14:00:04] <Nikerabbit>	 hashar: naturally, but what is the empty named test?
[14:00:17] <hashar>	 the 50 seconds one Scribunto_LuaUriLibraryTests::testLua  apparently has ~300 tests https://integration.wikimedia.org/ci/job/mwext-testextension-zend/17047/testReport/(root)/Scribunto_LuaUriLibraryTests__testLua/
[14:00:35] <hashar>	 the empty test, I have no idea :(
[14:00:44] <hashar>	 something is bugged somewhere :-((((
[14:01:20] <Nikerabbit>	 1000 tests 30 ms each
[14:01:50] <hashar>	 that is how you end up with 30 sec delay :-}
[14:02:30] <hashar>	 iirc Scribunto shells out to  a binary
[14:03:30] <phedenskog>	 Hashar: no I don't  yes please. let setup a time so we can talk.
[14:08:01] <hashar>	 phedenskog: I guess anytime tomorrow between 11am and 5pm  CET (UTC+1)  would do
[14:08:44] <hashar>	 phedenskog: either over hangouts or IRC  I dont mind. We could even talk about having Jenkins to run the webperf tests in parallel  to provide earlier feedback or run them more often
[14:09:09] <phedenskog>	 hashar: great, I'll send an invite, how much time do you think we should have? 
[14:09:27] <hashar>	 Nikerabbit: for core I found out we update the Sqlite/Mysql SearchEngine on each tests  even when there is no reason to do so.   https://gerrit.wikimedia.org/r/#/c/247560/ 
[14:09:49] <hashar>	 Nikerabbit: and I have a very lame patch to enable the profiler when running phpunit.php  https://gerrit.wikimedia.org/r/#/c/247554/ :}
[14:10:11] <hashar>	 phedenskog: if you are familiar with python,  30 - 40 minutes would do
[14:10:17] <hashar>	 phedenskog: else 1 hour or so.
[14:10:28] <phedenskog>	 hashar: ok!
[14:10:29] <hashar>	 phedenskog: we can have several sessions as needed
[14:11:45] <hashar>	 phedenskog: my calendar is  on amusso @ wikimedia.org  and should be up-to-date
[14:11:53] <phedenskog>	 hashar: perfect, thanks:)
[14:13:47] <hashar>	 yeah 12 hours should covers everything  11:00 – 23:45 :D
[14:14:45] <hashar>	 if you feel adventurous this afternoon, there is a basic tutorial at https://www.mediawiki.org/wiki/CI/JJB
[14:14:48] <Nikerabbit>	 hashar: thanks for all the work again ;)
[14:14:54] <hashar>	 if you get some spare time, might be worth looking at
[14:15:34] <hashar>	 Nikerabbit: maybe I should fill a tracking tasks to improve the phpunit.php wrapper speed and a bunch of child tasks with ideas/low hanging fruits
[14:18:23] <Nikerabbit>	 I have no idea how much overhead the wrapper adds
[14:28:34] <hashar>	 Nikerabbit: well each time we do an article edit, the search engine is updated. But in almost case we never look at the search db so those inserts/deletes are unneeded
[14:28:47] <hashar>	 Nikerabbit: the parser test also keep trying and deleting a bunch of files on each test :/
[14:34:19] <grrrit-wm>	 (03PS3) 10Hashar: Mathoid: Use mathoid-deploy-npm for the deploy repo patches [integration/config] - 10https://gerrit.wikimedia.org/r/259282 (owner: 10Mobrovac)
[14:35:39] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "\O/ thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/259282 (owner: 10Mobrovac)
[14:36:55] <grrrit-wm>	 (03Merged) 10jenkins-bot: Mathoid: Use mathoid-deploy-npm for the deploy repo patches [integration/config] - 10https://gerrit.wikimedia.org/r/259282 (owner: 10Mobrovac)
[14:40:23] <hashar>	 mobrovac: I have screwed up some beta cluster puppet patches apparently :(
[14:40:36] <mobrovac>	 ah?
[14:40:44] <hashar>	 mobrovac:  /var/lib/git/operations/puppet on deployment-puppetmaster had a dirty space
[14:40:44] <hashar>	 so I did a rebase --abort
[14:41:10] <mobrovac>	 hashar: not a pb, my rb patches have been merged today :)
[14:41:14] <hashar>	 http://paste.openstack.org/show/482081/
[14:41:14] <hashar>	 oh
[14:41:17] <hashar>	 yeah that is the reason
[14:41:18] <hashar>	 will rebase
[14:41:46] <hashar>	 !log beta puppetmaster: drop cherry pick 02c2006 RESTBase: Switch to service::node --  got merged
[14:41:50] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[14:43:26] <hashar>	 !log beta puppetmaster is rebased and up-to-date (upstream at d3e1f70 )
[14:43:29] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[14:43:30] <hashar>	 mobrovac: solved!
[14:43:37] <mobrovac>	 :)
[14:53:42] <testing-shinken->	 RECOVERY - Puppet failure on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:53:54] <grrrit-wm>	 (03PS4) 10Hashar: Restore Flow browser tests on Win8.0/IE 10 [integration/config] - 10https://gerrit.wikimedia.org/r/259470 
[14:55:06] <grrrit-wm>	 (03CR) 10Sbisson: [C: 031] "Sounds good, but stabilizing the tests in IE is not gonna be a walk in the park..." [integration/config] - 10https://gerrit.wikimedia.org/r/259470 (owner: 10Hashar)
[14:56:46] <grrrit-wm>	 (03CR) 10Hashar: "Well we can hold on this until Flow developers stabilize the tests on their local machine? There is little point in having a job spammin" [integration/config] - 10https://gerrit.wikimedia.org/r/259470 (owner: 10Hashar)
[15:11:42] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1884499 (10zeljkofilipin)
[15:19:01] <Nikerabbit>	 hashar: is there some kind of hidden queue not shown on https://integration.wikimedia.org/zuul/ ?
[15:19:14] <Nikerabbit>	 I don't see https://gerrit.wikimedia.org/r/#/c/259509/ there
[15:20:11] <Nikerabbit>	 ok, now it appeared there in back of the test  queue
[15:21:33] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] Enable submodules for operations/mediawiki-config phpunit tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[15:22:05] <hashar>	 Nikerabbit: there is a 5 - 10 secs delay
[15:23:02] <Nikerabbit>	 hashar: more like 25 seconds ;)
[15:29:12] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1884527 (10Qgil) 5Open>3Resolved It was published on Monday: http://blog.wikimedia.org/2015/12/14/new-mediawik...
[15:34:20] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1884537 (10zeljkofilipin)
[15:34:41] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1689201 (10zeljkofilipin)
[15:42:27] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1884554 (10zeljkofilipin)
[15:50:12] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1884581 (10zeljkofilipin)
[15:51:00] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 2 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1689201 (10zeljkofilipin)
[15:56:19] <andrewbogott>	 thcipriani: we’re still seeing evidence of a bunch of puppet breakage in staging.  Did you get a change to look through those?
[15:56:27] <andrewbogott>	 latest list is at the bottom here:  https://etherpad.wikimedia.org/p/remaining-ldap
[16:05:21] <ottomata>	 hey, am reading new (to me) scap3 docs, thanks so much for updating!  docs looking very nice :)
[16:06:39] <ottomata>	 i have a few qs, lemme know if someone is around who might answer (mostly about scap/ directory)
[16:07:53] <thcipriani>	 andrewbogott: I'll check it today. I thought I got them all, but I must've missed some. There's not too much use of the staging project at the moment, so it's not terribly critical. Thanks for the heads up.
[16:08:20] <thcipriani>	 ottomata: I could probably help out, what's up?
[16:08:48] <ottomata>	 just not sure what the best practice is around where to put that...maybe there isn't one yet
[16:08:53] <ottomata>	 i don't really want to add it to my repo
[16:09:01] <ottomata>	 i suppose a separate git repo would be fine, that I just clone on tin.
[16:09:04] <ottomata>	 but, it would be nice if that was puppetized
[16:09:05] <ottomata>	 so
[16:09:16] <ottomata>	 mabye I should puppetize a git::clone on tin at $deploypath/scap
[16:09:16] <ottomata>	 ?
[16:10:54] <thcipriani>	 yeah, one thing that may not be mentioned in the docs is that scap is agnostic about how you include the directory in the deploy path
[16:11:02] <ottomata>	 yeah, it kinda says that
[16:11:16] <ottomata>	 maybe i'll just do a git repo and manually clone for now? hmm. 
[16:11:32] <thcipriani>	 that would work, sure.
[16:11:40] <ottomata>	 here is a quick easy q:
[16:11:44] <ottomata>	 doc says "When you specify a service_port, the port specified will be checked to see if it is accepting connections. "
[16:11:47] <ottomata>	 waht does accespting connections mean?
[16:11:48] <ottomata>	 just tcp?
[16:12:11] <thcipriani>	 yeah, basically does: nc -vz localhost port
[16:12:16] <ottomata>	 k
[16:12:19] <thcipriani>	 but fancier and python-y
[16:12:36] <thcipriani>	 you can also specify the timeout for that (not sure if that's in docs or not)
[16:12:58] <ottomata>	 i think i tis
[16:13:01] <ottomata>	 also custom checks
[16:13:16] <ottomata>	 thcipriani: is scap3 used in prod with any other service yet?
[16:13:18] <ottomata>	 (or in beta?)
[16:13:30] <thcipriani>	 yeah, marxarelli did a lot of cool work with those checks.
[16:14:13] <thcipriani>	 ottomata: no, we're working on getting service things moved over. RESTBase used it briefly, then reverted back to trebuchet as it diverged too much with production.
[16:14:37] <thcipriani>	 (using scap for the config files diverged too much with production)
[16:15:56] <thcipriani>	 that being said, the way scap works from an end-user perspective is unlikely to change, mostly internal refactoring, docs, puppet work being done at this point.
[16:16:04] <ottomata>	 k cool
[16:16:12] <ottomata>	 gonna try in in beta now
[16:16:54] <thcipriani>	 ottomata: let me know how it goes, if anything fails for you I'm happy to troubleshoot. The more usage feedback we get the better the tool becomes.
[16:17:45] <ottomata>	 thcipriani: if I made an environments/beta/mockbase file
[16:17:49] <ottomata>	 that had targets for deployment in beta
[16:17:59] <ottomata>	 would that override a plain mockbase file in scap/
[16:17:59] <ottomata>	 ?
[16:18:27] <thcipriani>	 only if you do: deploy -e beta
[16:18:31] <ottomata>	 aye
[16:18:36] <ottomata>	 so it will override dsh targets too?
[16:18:46] <thcipriani>	 you have to pass the environment specifically at that point.
[16:18:52] <ottomata>	 then my scap.cfg wouldn't have to specify different server_groups?
[16:19:08] <thcipriani>	 let me doublecheck real quick. I did make a stink about that on the patch that added this functionality :P
[16:20:28] <ottomata>	 hm, how does ssh_user work?
[16:20:33] <ottomata>	 does it need a special key?
[16:21:42] <thcipriani>	 so yes to your question: an environments/beta/mockbase file should override a base mockbase file if you pass -e on the command line.
[16:22:03] <thcipriani>	 so ssh_user should be a user that can ssh into each target from tin.
[16:23:02] <wikibugs>	 10Deployment-Systems, 6operations, 5Patch-For-Review: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585#1884698 (10bd808)
[16:23:02] <thcipriani>	 since port forwarding is disallowed in production, keyholder is likely the only way to make it happen at this point.
[16:23:08] <ottomata>	 ok, so management of that user isn't magic
[16:23:24] <ottomata>	 keyholder?  as in i need to puppetize keys for that user appropriately?
[16:23:56] <ottomata>	 HMmMMM https://wikitech.wikimedia.org/wiki/Keyholder
[16:23:59] <ottomata>	 not heard of this...
[16:24:04] <thcipriani>	 indeed. I outlined a lot of what I've done thus-far for services here: https://doc.wikimedia.org/mw-tools-scap/scap3/ssh-access.html
[16:24:34] <ottomata>	 awesome, reading...
[16:29:04] <ottomata>	 cool thcipriani, and if i wanted to puppetize the git::clone of my scap config repo, i could add it to role::deployment::mything, since that is only applied on deploy servers, ja?
[16:29:30] <wikibugs>	 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should be aware of security patches - https://phabricator.wikimedia.org/T118477#1884740 (10hashar) git apply trials -------  D80 is straightforward.  It attempts to dry-run revert the security patches on the branch and will error out whenever it can...
[16:30:40] <thcipriani>	 ottomata: yeah, I believe that you also then have to add role::deployment::mything to role::deployment::server otherwise it wouldn't get applied anywhere, IIRC.
[16:30:51] <ottomata>	 right
[16:31:06] <ottomata>	 aye
[16:31:36] <wikibugs>	 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should be aware of security patches - https://phabricator.wikimedia.org/T118477#1884753 (10hashar) If you could find a way to record a patch has been published in Gerrit, it would be helpful to propose to get rid of the associated .patch file.  That...
[16:32:22] <grrrit-wm>	 (03PS7) 10Hashar: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:39:18] <grrrit-wm>	 (03PS8) 10Hashar: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:41:00] <ottomata>	 thcipriani: 
[16:41:04] <ottomata>	 "	1.	Adds the keyfile at puppet:///private/ssh/tin/deploy-mockbase_rsa to the keyholder" ?
[16:41:09] <ottomata>	 on deployment-bastion as well?
[16:41:19] <ottomata>	 i see there /etc/puppet/private/files/ssh/tin, which is a little funny?
[16:41:21] <hashar>	 !log updating operations-mw-config-phpunit to have it process submodules
[16:41:23] <ottomata>	 sorry
[16:41:24] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[16:41:26] <ottomata>	 deployment-puppetmaster?
[16:41:29] <grrrit-wm>	 (03CR) 10Hashar: "I have added a comment as to why the git definition is overridden. Made submodule to be explicit." [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:41:40] <andrewbogott>	 thcipriani: puppet master was crashed on staging-palladium and also there’s a merge error in /var/lib/git/operations/puppet which prevents puppet runs.
[16:41:47] <andrewbogott>	 So, fix that and probably everything else will fall in line :)
[16:41:57] * andrewbogott stops meddling
[16:43:55] <thcipriani>	 ottomata: you'd add keys to: deployment-puppetmaster:/var/lib/git/labs/private/files/ssh
[16:45:04] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "I did a recheck on https://gerrit.wikimedia.org/r/#/c/255135/" [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:45:29] <hashar>	 ebernhardson: tyler pointed me to your Jenkins patch to get submodule processed on mw-config ( https://gerrit.wikimedia.org/r/#/c/256979/ )
[16:45:30] <ebernhardson>	 hashar: thanks! i was kinda taking a stab in the dark with that patch
[16:45:35] <hashar>	 ebernhardson: kudos, deployed and working fine!
[16:45:54] <hashar>	 ebernhardson: it was fine.  I just added a comment and explicitly defined  submodule: disabled: false
[16:46:23] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:46:30] <hashar>	 ebernhardson: it is deployed.  So I guess now you can write some more tests that rely on whatever is in the submodules :-))))  Thank you a ton!
[16:46:48] <grrrit-wm>	 (03CR) 10Hashar: "For your tirelessly work on Jenkins, CI, tests, Librarifizification, here is the official Continuous integration barnstar!" [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[16:46:56] <hashar>	 ebernhardson: here is your CI barnstorm ^^^^
[16:47:00] <ebernhardson>	 :)
[16:47:00] <hashar>	 barn star
[16:47:10] <ottomata>	 thcipriani:  aye, but why is there a tin directly in there?
[16:47:14] <hashar>	 :D
[16:47:34] <ottomata>	  /var/lib/git/labs/private/files/ssh/tin
[16:48:07] <gwicke>	 thcipriani, ottomata: we haven't used trebuchet in about a year
[16:49:01] <gwicke>	 we used custom shell loops for a while, but then switched to ansible
[16:49:14] <hashar>	 ottomata: that is where puppetmaster on labs look for $passwords::
[16:49:30] <hashar>	 and I guess tin is hardcoded in the puppet manifest
[16:49:33] <ottomata>	 :(
[16:50:03] <hashar>	 but maybe some hiera data let us vary the file name :-}
[16:51:54] <ottomata>	 thcipriani:  why would you set  recurse => true on     file { '/srv/deployment/mockbase':
[16:51:54] <ottomata>	  in the deployment target?
[16:52:03] <ottomata>	 wouldn't that cause puppet to delete the deployed files?
[16:52:09] <ottomata>	 since they are not managed by puppet?
[16:53:10] <thcipriani>	 ottomata: the intent was to ensure that permissions for all files in that directory are the same.
[16:54:40] <ottomata>	 thcipriani:  have you tried it?  i think it deletes any non puppet managed file in that dir
[16:55:27] <thcipriani>	 ottomata: this was in a cherry-picked manifest on deployment-puppetmaster and I hadn't observed that
[16:55:42] <ottomata>	 hm
[16:56:20] <ottomata>	 This also enables the purge attribute, which can delete unmanaged files from a directory. See the description of purge for more details.
[16:56:29] <ottomata>	 purge
[16:56:29] <ottomata>	 Whether unmanaged files should be purged. This option only makes sense when ensure => directory and recurse => true.
[16:56:47] <ottomata>	 maybe the defeault for purge is false
[16:57:03] <ottomata>	 and 'enables the purge attribute' just means you can now use the purge attribute
[16:57:20] <ottomata>	 guess i will try it :)
[16:58:08] <thcipriani>	 ottomata: if it is the case that the unmanaged files are purged, let me know and I'll update docs.
[16:58:51] <ottomata>	 k
[16:58:53] <hashar>	 Niharika will send a patch to add a new extension (PageAssessment ) in integration/config
[16:59:02] <hashar>	 straightforward, should just change Zuul layout config. 
[16:59:33] <hashar>	 patch and her should show up here in a few :}
[16:59:37] <hashar>	 I am off! *wave*
[17:05:17] <grrrit-wm>	 (03PS1) 10Niharika29: Setup CI for PageAssessments extension [integration/config] - 10https://gerrit.wikimedia.org/r/259529 
[17:19:33] <ottomata>	 thcipriani:  does the data.yaml group deploy-mockbase need to be added to the deploy server ?
[17:19:39] <ottomata>	 "Finally, you’ll want to modify: modules/admin/data/data.yaml in the operations/puppet repo to create the deploy-mockbase group and add users to that group.
[17:19:39] <ottomata>	 "
[17:19:46] <ottomata>	 just adding the group isn't enough, right?
[17:19:57] <ottomata>	 or, is the group membership looked up elsewhere than posix group membership on the deploy server?
[17:23:23] <thcipriani>	 so data.yaml is only used for production, so you'll have to add the group and the group members to ldap
[17:23:44] <thcipriani>	 but data.yaml does handle group and group membership.
[17:23:58] <ottomata>	 to ldap?
[17:24:01] <ottomata>	 hm
[17:24:19] <ottomata>	 thcipriani:  data.yaml handles it?
[17:24:21] <ottomata>	 ok well
[17:24:22] <ottomata>	 in prod
[17:24:24] <ottomata>	 i'm asking
[17:24:30] <ottomata>	 does the posix group need to exist on tin
[17:24:36] <ottomata>	 with its members
[17:24:37] <ottomata>	 or
[17:24:47] <ottomata>	 is just the presense of the group config in data.yaml enough?
[17:24:48] <ottomata>	 e.g.
[17:25:05] <thcipriani>	 the group needs to exist on tin with its members.
[17:25:21] <ottomata>	 ok, so somewhere in puppet tin will need to declare all of the groups it needs
[17:25:22] <ottomata>	 hm.
[17:25:50] <ottomata>	 via admin::groups:
[17:25:53] <ottomata>	 i guess
[17:26:22] <ottomata>	 ergh, which probably means that every deployment target will need a new group created, eh?
[17:27:01] <ottomata>	 i guess i'll defer that until production...although i'm not sure what group to use for keyholder trusted_group
[17:27:22] <thcipriani>	 every deployed thing will likely need a group of deployers.
[17:27:49] <ottomata>	 aye, yeah, we have a few eventlogging groups right now, i can use one of those i think
[17:27:59] <ottomata>	 we'll have to figure out some smarter sudo rules
[17:29:44] <ottomata>	 thcipriani:  how do I add a group in ldap/labs?
[17:30:27] <ottomata>	 also, what is the proper way to add a private key to labs private?
[17:30:34] <ottomata>	 just manually put it in there, and git commit ?
[17:30:39] <ottomata>	 on deployment-puppetmaster?
[17:30:57] <ottomata>	 and the proper path is /var/lib/git/labs/private/ssh/tin/deploy-mockbase_rsa 
[17:30:58] <ottomata>	 ?
[17:31:20] <thcipriani>	 ottomata: I was just digging through wikitech to try to figure that out if there's a way to add a group via that interface. This may be a question for -labs :\
[17:31:34] <thcipriani>	 yes, that is the correct way to add a key for deployment-prep.
[17:31:42] <ottomata>	 ook
[17:38:19] <ottomata>	 thcipriani:  oh, i see you added a passphrase file too?
[17:38:24] <ottomata>	 for servicedeploy
[17:38:32] <ottomata>	 or well, somewone did
[17:38:35] <ottomata>	 should I do that?  
[17:38:42] <ottomata>	 or shoudl the key be passphrase less?
[17:49:07] <thcipriani>	 ottomata: yeah, I think the best practice is to have a passphrase although I'm not sure that's a hard-requirement. I just did it like that because it was done that way for the others.
[17:53:13] <wikibugs>	 7Browser-Tests, 5Patch-For-Review, 3Reading Web Sprint 62 - DJ-Jazzy-Jeff-and-the-Fresh-Sprints: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1884983 (10KLans_WMF)
[18:04:16] <wikibugs>	 10Deployment-Systems, 10Architecture, 10Wikimedia-Developer-Summit-2016-Organization, 7Availability: WikiDev 16 working area: Software engineering - https://phabricator.wikimedia.org/T119032#1885039 (10daniel) Here is my current take on what sessions we should have to cover the Software Engineering (aka Co...
[18:54:25] <shinken-wm>	 PROBLEM - Puppet failure on wmfbranch is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]
[19:15:54] <ottomata>	 thcipriani: k another q
[19:16:04] <ottomata>	 if i want to do different deployments to different targets
[19:16:10] <ottomata>	 in production, from the same codebase
[19:16:22] <ottomata>	 would it be better to make a new deployment source and target
[19:16:31] <ottomata>	 or, to use scap config files (maybe environments?) to manage this?
[19:17:23] <thcipriani>	 so you mean that you would be using a different dsh_targets? 
[19:18:21] <ottomata>	 yeah, either with that
[19:18:28] <ottomata>	 or just different dsh files in different environments
[19:18:32] <ottomata>	 although, maybe that is an abuse
[19:18:42] <ottomata>	 i think just a list of different dsh_targets is wrong, since that does staged deploys, right?
[19:19:05] <ottomata>	 that's not what I want, i want to manage deployments of the same codebase for different purposes (e.g. analytics, production production (eventbus), etc.)
[19:19:15] <ottomata>	 i'm thinking a different deploy target is more appropriate, eh?
[19:19:25] <thcipriani>	 so, if you have different dsh_targets in the same scap.cfg file, then, yeah, that's staged deploys
[19:19:53] <thcipriani>	 if you want a completely different set of dsh_target for deployment-prep vs production, then I'd say that's the use-case we built environments for.
[19:20:03] <ottomata>	 hmm, it smore than that
[19:20:04] <ottomata>	 in production
[19:20:09] <ottomata>	 i wnat different dsh_targets
[19:20:47] <ottomata>	 as there are different many services that are run out of the same repo
[19:21:10] <ottomata>	 i really only need 2 right now
[19:21:17] <ottomata>	 analytics, and uhhh let's say 'eventbus'
[19:21:27] <ottomata>	 but, i want to be able to deploy new code to each of those separately
[19:21:43] <ottomata>	 so, environments woudl work
[19:21:48] <ottomata>	 but, maybe that is an abuse of environments
[19:22:05] <thcipriani>	 yeah, that seems like it may not be the cleanest way to do things in that instance.
[19:22:14] <ottomata>	 as the clone of the repo on the deploy server would have to be checked out at certain commits or tags to deploy the desired commit
[19:22:20] <ottomata>	 rather than the usual git pull and deploy the latest
[19:22:43] <ottomata>	 yeah, will make another entry in deployments.yaml for the new deployment case
[19:22:50] <ottomata>	 and configure scap3 for that one for now
[19:23:29] <ottomata>	 for any deploy target, all that is necessary is that the configured user can ssh there and copy files to a deploy path, right?
[19:23:47] <ottomata>	 there's no special config needed from scap.cfg or from puppet other than that, ja?
[19:24:16] <thcipriani>	 scap will need to be installed on the target as well.
[19:24:24] <ottomata>	 oh, what does that?
[19:24:52] <thcipriani>	 one second, there's a puppet class for this, lemme find it.
[19:24:54] <ottomata>	 ah
[19:24:55] <ottomata>	 i see it
[19:24:57] <ottomata>	 scap
[19:24:58] <ottomata>	 module
[19:24:59] <ottomata>	 ja?
[19:25:00] <ottomata>	 just include scap?
[19:25:07] <ottomata>	 oh maybe not
[19:25:28] <ottomata>	 that looks like old scap
[19:26:52] <thcipriani>	 so there are two classes: scap::init and role::scap::target. First installs scap + dependencies, second makes sure there's a ferm rule for scap.
[19:28:03] <ottomata>	     # Using trebuchet provider while scap service deployment is under
[19:28:03] <ottomata>	     # development—chicken and egg things
[19:28:03] <ottomata>	     #
[19:28:03] <ottomata>	     # This should be removed once scap3 is in a final state (i.e. packaged
[19:28:03] <ottomata>	     # or deployed via another method)
[19:28:03] <ottomata>	     package { 'scap':
[19:28:06] <ottomata>	 that is the proper scap?
[19:28:25] <thcipriani>	 indeed.
[19:28:53] <ottomata>	 oh i see, it ijust using trebuchet's git clone to scap3 code to /srv/deployment/scap/scap?
[19:29:10] <ottomata>	 ok
[19:29:16] <thcipriani>	 we have done the work for packaging; however, we'd like to get more things deployed in beta before commiting to using the package exclusively.
[19:29:20] <testing-shinken->	 RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:29:35] <ottomata>	 why doesn't role::scap::target just include scap
[19:29:36] <ottomata>	 ?
[19:30:05] <ottomata>	 or, better yet, why not put scap::target  in the scap module?  
[19:30:20] <ottomata>	 role doesn't quite make sense for that i thinkk... (I also dont' like including roles from other modules)
[19:30:37] <niedzielski>	 anyone having trouble cloning from diffusion? i thought git clone https://phabricator.wikimedia.org/diffusion/APAW should work but i'm getting a 503
[19:31:09] <niedzielski>	 nevermind. just magically started working now :|
[19:31:46] <thcipriani>	 ottomata: yeah, the puppet stuff for deployment is in need of some work. Most of this split was done to get all the scap stuff out of the mediawiki module.
[19:31:58] <ottomata>	 aye k
[19:33:31] <ottomata>	 thcipriani:  i'm going to make a patch, and add you as reviewer, please add whoever else you think shoudl look at it
[19:33:38] <ottomata>	 (for that role thing :) )
[19:33:41] <ottomata>	 shoudl be really easy
[19:33:45] <thcipriani>	 ottomata: awesome! Thanks.
[19:34:23] <shinken-wm>	 RECOVERY - Puppet failure on wmfbranch is OK: OK: Less than 1.00% above the threshold [0.0]
[19:36:23] <ottomata>	 thcipriani:  do you know if there is a specific reason (role::)scap::target doesn't just include scap itself?
[19:36:34] <ottomata>	 i'd thikn if you wanted the ferm rule, you'd also want the package, otherwise, no point?
[19:37:18] <ottomata>	 hm, i guess because some of the old mediawiki stuff doesn't use the new scap3?  hmmm
[19:37:56] <thcipriani>	 iirc it's mostly to do with role::deployment::mediawiki
[19:38:21] <ottomata>	 yeah ok, i won't touch that
[19:38:23] <ottomata>	 will just add a comment
[19:40:20] <thcipriani>	 kk, yeah, this is definitely something that should be addressed; however, pulling on that thread can definitely eat some time.
[19:41:12] <ottomata>	 yeah
[19:41:12] <ottomata>	 hehe
[19:41:36] <ottomata>	 thcipriani:  what is your gerrit nick?
[19:41:43] <ottomata>	 oh same
[19:41:43] <ottomata>	 nm
[19:41:45] <ottomata>	 just typed it wrong
[19:41:47] <ottomata>	 found it :)
[19:41:49] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/259542/
[19:41:57] <thcipriani>	 yeah, I'm thcipriani or tcipriani everywhere :)
[19:57:04] <testing-shinken->	 PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]
[20:00:41] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1885513 (10hashar) Thank you @Qgil (and other writers)
[20:00:42] <ottomata>	 don't think thats me...but i'm the only one making changes sooOoo..
[20:00:42] <ottomata>	 hm
[20:01:05] <ottomata>	 hmm maybe its my new deployment target
[20:01:58] <ottomata>	 thcipriani:  typo in your ssh-access docs
[20:01:59] <ottomata>	         source => puppet://modules/mockbase/deploy-test_rsa.pub,
[20:02:02] <ottomata>	 should have 3 slashes
[20:02:06] <ottomata>	         source => puppet:///modules/mockbase/deploy-test_rsa.pub,
[20:04:33] <thcipriani>	 ottomata: right. good catch—will update.
[20:18:17] <ottomata>	 hmmMm ok not sure what i'm doing wrong atm
[20:18:19] <ottomata>	 so
[20:18:23] <ottomata>	 i defined a new deployment target here
[20:18:23] <ottomata>	 https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep
[20:18:26] <ottomata>	     eventlogging/eventbus:
[20:18:27] <ottomata>	         upstream: https://gerrit.wikimedia.org/r/eventlogging
[20:18:27] <ottomata>	         checkout_submodules: true
[20:18:44] <ottomata>	 shouldn't a puppet run make that show up at /srv/deployment/eventlogging/eventbus on deployment-bastion
[20:18:45] <ottomata>	 ?
[20:19:48] <ottomata>	 thcipriani: ^?
[20:22:10] <testing-shinken->	 RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0]
[20:23:46] <ottomata>	 oh myabe i need a puppet run elsewhere...
[20:24:02] <ottomata>	 ahh yes, on deployment-salt
[20:28:12] <ottomata>	 Cool ja there it is
[20:32:11] <ottomata>	 ok thcipriani trying my first deploy with beta environment dsh_targets
[20:32:17] <ottomata>	 it iddn't pick up the proper dsh file
[20:32:19] <ottomata>	 still around?
[20:32:23] <wmf-insecte>	 Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #724: 04FAILURE in 1 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/724/
[20:33:01] <testing-shinken->	 PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0]
[20:33:03] <testing-shinken->	 PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0]
[20:35:33] <thcipriani>	 ottomata: yup, I'm around
[20:35:48] <ottomata>	 k, btw, not a question, but yes, keyholder keys need a passphrase
[20:35:52] <ottomata>	  /etc/keyholder.d/eventlogging_rsa is not an acceptable key. Does it have a passphrase?
[20:35:54] <ottomata>	 thcipriani:  ja so
[20:36:01] <ottomata>	 on deployment-bastion
[20:36:15] <ottomata>	 look at my scap config in  /srv/deployment/eventlogging/eventbus/scap
[20:36:58] <ottomata>	 (oh it looks like keyholdearm it added my new key anyway)
[20:37:00] <ottomata>	 guess it was just a warning?
[20:37:23] <ottomata>	 Agent admitted failure to sign using the key.
[20:37:24] <ottomata>	 maybe not
[20:37:24] <ottomata>	 :)
[20:37:34] <ottomata>	 thcipriani:  anyway, with my scap/ config
[20:37:35] <ottomata>	 i have
[20:37:39] <Krinkle>	 thcipriani: ostriches: Reedy: I'm thinking of bringing back the weekly mails about log warnings, broken down by project. We used to do this a while back. We need to re-introduce this sense of responsibility.
[20:37:45] <ottomata>	 scap/environments/beta/eventbus
[20:37:51] <ottomata>	 which lists the beta host i want to deploy to
[20:37:53] <ottomata>	 when I do
[20:37:56] <Reedy>	 Krinkle: :)
[20:37:56] <ottomata>	 deploy -e beta
[20:38:02] <ottomata>	 it pulls the hosts out of scap/eventbus
[20:38:04] <testing-shinken->	 RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0]
[20:38:05] <Krinkle>	 I'm also thinking of proposing a zero-tolerance where projects are not allowed to roll out new functionality before fixing warnings from previous train rides. 
[20:38:24] <ostriches>	 Krinkle: +1000
[20:38:59] <Krinkle>	 The error count is so massive right now that it makes it impossible to find regressions that stand out from the large amount of noise, and also hard to find out what error might have caused a hard-to-pinpoint regression. FOr example, this week our Save Timing metric regressed by 200ms. Being able to quickly see (or rule out) using error logs would've been
[20:39:00] <Krinkle>	 useful.
[20:39:11] <thcipriani>	 ottomata: hmm, lemme double check something, might be a scap out of date thing.
[20:39:17] <ottomata>	 k
[20:39:22] <Krinkle>	 Thanks, I'll write one up then (warnings) to start it off again
[20:39:40] <testing-shinken->	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[20:40:20] <shinken-wm>	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[20:41:22] <ostriches>	 Krinkle: You're familiar with https://grafana.wikimedia.org/dashboard/db/production-logging, right? We should use those graphs to back up our e-mails.
[20:41:37] <ostriches>	 ie: "Log volume has increased by 10% this week. See? That's bad"
[20:41:38] <ostriches>	 Etc.
[20:42:06] <wmf-insecte>	 Yippee, build fixed!
[20:42:07] <wmf-insecte>	 Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #725: 09FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/725/
[20:43:56] <grrrit-wm>	 (03Merged) 10jenkins-bot: Setup CI for PageAssessments extension [integration/config] - 10https://gerrit.wikimedia.org/r/259529 (owner: 10Niharika29)
[20:44:37] <grrrit-wm>	 (03CR) 10Hashar: "Well done. I have deployed the configuration change :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/259529 (owner: 10Niharika29)
[20:47:50] <thcipriani>	 !log update scap on deployment-prep
[20:47:56] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[20:49:25] <thcipriani>	 ottomata: give that a shot, if that doesn't work, I'll have to take a look at the scap.cfg (can't see it right now since it's in your ~ :))
[20:49:49] <ottomata>	 :p
[20:49:59] <ottomata>	 i can fix that, i was doing that before i had things working better
[20:53:23] <testing-shinken->	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms
[20:53:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 7Easy, 7Mobile, 5Patch-For-Review: MobileFrontEnd on Beta Cluster should display a different logo so that it is clearly not a production site - https://phabricator.wikimedia.org/T115078#1885775 (10Jdlrobson)
[20:54:29] <ottomata>	 naw thcipriani no good
[20:54:34] <testing-shinken->	 PROBLEM - Puppet failure on mira is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0]
[20:54:36] <ottomata>	 it still grabbed the main dsh file
[20:54:38] <ottomata>	 not the -e beta one
[20:54:49] <thcipriani>	 :(
[20:54:52] <ottomata>	 $ deploy -e beta
[20:54:52] <ottomata>	 20:54:17 Started deploy_eventlogging/eventbus
[20:54:52] <ottomata>	 Updated tag 'scap/sync/2015-12-16/0003' (was 0000000)
[20:54:52] <ottomata>	 Entering 'config/schemas'
[20:54:52] <ottomata>	 20:54:17
[20:54:52] <ottomata>	 == DEFAULT ==
[20:54:52] <ottomata>	 :* kafka1001.eqiad.wmnet
[20:54:53] <ottomata>	 :* kafka1002.eqiad.wmnet
[20:55:17] <thcipriani>	 yeah, I was running `deploy-log` so I could see that :)
[20:55:36] <ottomata>	 my keyholder thing isn't working yet iether, so i'll have to figure that out too
[20:55:41] <thcipriani>	 lemme check the scap.cfg real quick...
[20:55:43] <ottomata>	 ok
[20:55:50] <shinken-wm>	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms
[20:56:48] <ottomata>	 hm, i can switch that weird git_deploy_dir back
[20:56:54] <ottomata>	 i'm not worried about that now that I have a new target
[20:59:25] <thcipriani>	 ottomata: give it another try, I'll watch deploy-log, this is a bug, I just tried to fix ad-hoc.
[21:00:07] <ottomata>	 k just a few, got a couple of things i need to fix too
[21:00:10] <ottomata>	 in puppet stuff
[21:07:00] <ottomata>	 ok thcipriani trying now!
[21:07:12] <ottomata>	 hmm locked?
[21:07:44] <ottomata>	 someone deploying mw?
[21:07:51] <ottomata>	 can only one deploy happen at once?
[21:07:58] <ottomata>	 oh jenkins!
[21:07:58] <ottomata>	 hm!
[21:08:15] <thcipriani>	 I thought we merged something for this, hang on.
[21:09:34] <thcipriani>	 ottomata: oh, right, just add: lock_file: /tmp/eventbus to your config
[21:09:46] <ottomata>	 hm ok
[21:10:04] <ottomata>	 maybe the default should be ./scap/lock ?
[21:10:04] <thcipriani>	 it's configurable and defaults to the scap lockfile. There is a bug for this.
[21:10:23] <ottomata>	 can I set a relative path? :p
[21:10:29] * bd808 needs to get rid of his "scap" stalk word
[21:10:40] <thcipriani>	 bd808: :D
[21:10:48] <thcipriani>	 ottomata: no, not at this time.
[21:10:57] <ottomata>	 ok
[21:11:06] <ottomata>	 can I refer to other configs within the config file?
[21:11:18] <ottomata>	 e.g. $git_repo/$git_deploy_dir ?
[21:11:18] <ottomata>	 :p
[21:11:31] <ostriches>	 Krinkle: Warning: data error in /srv/mediawiki/php-1.27.0-wmf.9/includes/Revision.php on line 1325	 
[21:11:31] <ostriches>	 Warning: Revision::decompressRevisionText: gzinflate() failed [Called from Revision::decompressRevisionText in /srv/mediawiki/php-1.27.0-wmf.9/includes/Revision.php at line 1328] in /srv/mediawiki/php-1.27.0-wmf.9/includes/debug/MWDebug.php on line 300
[21:11:37] <ostriches>	 bad gzip?
[21:11:41] <ostriches>	 on a rev
[21:11:53] <ottomata>	 (i assume not :) )
[21:12:02] <thcipriani>	 ottomata: you assume correctly :)
[21:12:42] * Krinkle edits https://grafana.wikimedia.org/dashboard/db/production-logging
[21:12:50] <Krinkle>	 added history view and added a per-channel breakdown on the bottom
[21:13:58] <ottomata>	 rats
[21:14:00] <ottomata>	 thcipriani: 
[21:14:00] <ottomata>	 $ deploy -e beta
[21:14:00] <ottomata>	 21:13:46 Started deploy_eventlogging/eventbus
[21:14:00] <ottomata>	 Updated tag 'scap/sync/2015-12-16/0004' (was 0000000)
[21:14:00] <ottomata>	 Entering 'config/schemas'
[21:14:00] <ottomata>	 21:13:46
[21:14:00] <ottomata>	 == DEFAULT ==
[21:14:01] <ottomata>	 :* kafka1001.eqiad.wmnet
[21:14:01] <ottomata>	 :* kafka1002.eqiad.wmnet
[21:14:55] <thcipriani>	 sigh. ok, lemme check my fix again here.
[21:15:09] <testing-shinken->	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[21:15:14] <thcipriani>	 ottomata: can I try deploys to test some stuff?
[21:15:48] <ostriches>	 Krinkle: looks good, thx for the improvements
[21:15:53] <shinken-wm>	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[21:18:14] <ottomata>	 yes thcipriani go ahead
[21:18:21] <ottomata>	 i haven't got keyholder to work yet
[21:18:27] <thcipriani>	 kk
[21:18:31] <ottomata>	 but you can at least attempt the deploy
[21:18:34] <ottomata>	 to see what hosts get chosen
[21:24:59] <wikibugs>	 10Deployment-Systems, 3Scap3: scap environment-specific host file not working - https://phabricator.wikimedia.org/T121705#1885896 (10thcipriani) 3NEW
[21:25:56] <thcipriani>	 ^ ottomata I filed a bug, for the time being for beta, it'll be easier to use an absolute path. I'll try to get that bug fixed fairly quickly, should be quick.
[21:26:31] <ottomata>	 absoulte path to dsh file?
[21:27:29] <thcipriani>	 yep. That will work. 
[21:27:50] <ottomata>	 oh or to env dir?
[21:28:20] <testing-shinken->	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.68 ms
[21:28:31] <thcipriani>	 wrt to the keyholder, I just checked that auth-d yaml file for keyholder. It seems you set the authorized group to project_deployment-prep. I'm in project-deployment-prep
[21:28:53] <thcipriani>	 if you use an absolute path, you can set it to anywhere.
[21:30:50] <shinken-wm>	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms
[21:32:22] <hashar>	 hmm
[21:33:06] <ottomata1>	 thcipriani:  i saw that too
[21:33:08] <ottomata1>	 have fixed
[21:33:24] <ottomata>	 still same result though...oh maybe keey needs reamred?
[21:33:48] <ottomata>	 hmm, no that shouldn't matter, right?
[21:33:59] <thcipriani>	 yeah, changing the auth needs a restart of keyholder then a key rearm IIRC.
[21:34:10] <thcipriani>	 loads up perms at service start
[21:34:17] <ottomata>	 oh reastart
[21:34:18] <ottomata>	 ok
[21:34:30] <chasemp>	 twentyafterfour: ostriches https://gerrit.wikimedia.org/r/#/c/259560/
[21:34:34] <chasemp>	 let's get real ip's
[21:34:52] <andrewbogott>	 thcipriani: the st-*.services-testbed instances also look unmaintained.  May I delete them?
[21:35:28] <ottomata>	 thcipriani:  rats
[21:35:29] <ottomata>	 restarted
[21:35:31] <ottomata>	 re-armed
[21:35:33] <ottomata>	 with passphrases
[21:35:37] <ottomata>	 lookgs good in keyholder status
[21:35:39] <ottomata>	 but still
[21:35:41] <twentyafterfour>	 chasemp: +1'd
[21:35:42] <ottomata>	 Agent admitted failure to sign using the key.
[21:36:04] <thcipriani>	 andrewbogott: looking
[21:36:38] <thcipriani>	 ottomata: which instance are you trying to get to?
[21:36:52] <ottomata>	 deployment-eventlogging04
[21:36:59] <ottomata>	 in auth.log there
[21:37:09] <ottomata>	 i don't see it tryint ot use the key fingerprint  from keyholder
[21:37:17] <ottomata>	 i think keyholder isn't allowing me access to the agent
[21:38:37] <ottomata>	 ha, not much helpful in upstart logs
[21:38:37] <ottomata>	 debug1: type 11
[21:38:38] <ottomata>	 debug1: XXX shrink: 3 < 4
[21:38:46] <thcipriani>	 andrewbogott: I'd say you can delete everything except st-trusty and proba, you'd have to check with services about those, everything else I set up for scap3 things. May want to doublecheck in -services about all those, though.
[21:39:04] <andrewbogott>	 thcipriani: sounds good, thank you
[21:39:37] <thcipriani>	 ottomata: which user?
[21:39:46] <ottomata>	 eventlogging
[21:41:47] <thcipriani>	 hmm, yes, that is strange...
[21:43:42] <wikibugs>	 10Deployment-Systems, 7I18n: Translation from translatewiki.net (MediaWiki:View-foreign/nl) don't show - https://phabricator.wikimedia.org/T66230#1885965 (10Nemo_bis)
[21:43:53] <wikibugs>	 10Deployment-Systems, 7I18n: Translation from translatewiki.net (MediaWiki:View-foreign/nl) don't show - https://phabricator.wikimedia.org/T66230#1885966 (10Nemo_bis) 5Open>3Invalid
[21:46:16] <ottomata>	 thcipriani:  fyi i can confirm that deploy -e /srv/deployment/eventlogging/eventbus/scap/environments/beta does the right thing
[21:47:36] <thcipriani>	 kk, looking at this keyholder thing, strange that it's not working
[21:49:50] <ottomata>	 ja afaict everything is in line...
[21:50:52] <thcipriani>	 indeed.
[21:51:43] <ottomata>	 thcipriani:  maybe the keyholder-proxy needs restarted too?
[21:52:45] <thcipriani>	 ottomata: yep
[21:53:00] <ottomata>	 did you restart it?
[21:53:02] <ottomata>	 if not i will try now
[21:53:10] <thcipriani>	 I did not, please do
[21:53:14] <ottomata>	 i just noticed that keyholder restart only restarts the main agent
[21:53:54] <ottomata>	 better!
[21:53:59] <ottomata>	 now it is using the proper key
[21:53:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Staging, 6Collaboration-Team-Backlog, 10DBA: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1885983 (10Mattflaschen)
[21:54:08] <ottomata>	 buuut not yet allowed to get into the target
[21:54:17] <ottomata>	 Failed publickey for eventlogging from 10.68.16.58 port 57033 ssh2: RSA 02:9b:99:e2:f0:16:70:a3:d2:5a:e6:02:a3:73:0e:b0 :)
[21:54:27] <ottomata>	 thcipriani:  should keyholder restart restart the proxy then too?
[21:54:38] <ottomata>	  "/sbin/${command}" keyholder-proxy
[21:54:39] <ottomata>	 ?
[21:54:53] <ottomata>	 if so I can make patch...
[21:55:30] <thcipriani>	 I can't offhand think of a circumstance in which you would restart one without restarting the other.
[21:55:52] <ottomata>	 ja if you are trying to reload configs, the proxy will need restarted in order to pick up the auth dir changes
[21:55:58] <ottomata>	 k submitting patch, i'll add you and ori as reviewers
[21:58:30] <ottomata>	 thcipriani:  https://gerrit.wikimedia.org/r/#/c/259596/
[22:00:03] <ottomata>	 SO CLOSE!
[22:00:19] <ottomata>	 now what is wrong with this keyYyYYYy
[22:00:20] <ottomata>	 hm
[22:01:24] <shinken-wm>	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[22:01:40] <ottomata>	 thcipriani:  is it possible I have to do something with this eventlogging user in ldap in labs?
[22:01:54] <ottomata>	 i see it doing  error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1
[22:01:57] <ottomata>	  pam_access(sshd:account): access denied for user `eventlogging' from `deployment-bastion.deployment-prep.eqiad.wmflabs'
[22:02:08] <ottomata>	 but, it is trying to use the proper key
[22:02:23] <ottomata>	 and, the public key is at /etc/ssh/userkeys/eventlogging
[22:02:45] <Krenair>	 um
[22:03:00] <Krenair>	 Does andrewbogott know you're talking about this?
[22:03:10] <ottomata>	 me? :)
[22:03:17] <ottomata>	 i suppose now he will :)
[22:03:20] <testing-shinken->	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[22:03:36] <ottomata>	 i aint changin no pam things, don't you worry
[22:03:43] <Krenair>	 ok :)
[22:04:09] <ottomata>	 i *think* for this case it shouldn't matter, ssh is just trying to lookup a username in ldap and it is failing (because it doesn't exist there)
[22:04:13] <ottomata>	 so, it doesn't find a public key for it 
[22:04:15] <ottomata>	 which is fine
[22:04:17] <ottomata>	 i think it shoudln't matter
[22:04:22] <Krenair>	 that does sound like a pam thing
[22:04:29] <ottomata>	 yeah, its trying to use pam to do that
[22:04:32] <ottomata>	 as it will for all logins i guess
[22:04:39] <ottomata>	 but, i'm trying to log in with a systemuser account that is not in ldap
[22:04:44] <ottomata>	 but configuring ssh keys via keyholder in puppet
[22:05:01] <ottomata>	 so, there is a private key in an agent on deployment-bastion, and a public key deployed via puppet on my target host
[22:05:03] <Krenair>	 I think this might be a known problem
[22:05:22] <ottomata>	 hmmm yeah, maybe hte pam thing is causing a problem
[22:05:39] <ottomata>	 is labs configured in such a way that ONLY ldap public keys are allowed?
[22:06:05] <andrewbogott>	 ottomata: as opposed to…?
[22:06:18] <ottomata>	 as opposed to this https://wikitech.wikimedia.org/wiki/Keyholder
[22:06:50] <ottomata>	 andrewbogott:  i'm trying to use scap3 to deploy 
[22:06:51] <ottomata>	 in beta
[22:07:03] <ottomata>	 i've set up private key in keyholder on deployment-bastion
[22:07:12] <ottomata>	 and a public key in /etc/ssh/userkeys
[22:07:13] <andrewbogott>	 is this something that you think ever worked in labs?
[22:07:14] <ottomata>	 i think its all set
[22:07:19] <Krenair>	 Yes
[22:07:22] <ottomata>	 andrewbogott:  no idea! i'm doing this for the first time
[22:07:33] <andrewbogott>	 ok...
[22:07:37] <ottomata>	 it seems like it should, according to these instructions, but maybe I need to add a public key for a user in ldap?
[22:07:44] <andrewbogott>	 I don’t know.  It’s quite possible that pam changes broke it
[22:07:45] <ottomata>	 that would be unfortunate if so :)
[22:07:52] <ottomata>	 this user doesn't exist in ldap
[22:07:56] <ottomata>	 it is a system user
[22:07:59] <ottomata>	 managed by puppet
[22:08:13] <andrewbogott>	 I’d suggest you open a ticket, cc: faidon, and make sure you include something along the lines of when/how it used to work
[22:08:23] <ottomata>	 thcipriani:  do you know if it used to work?
[22:08:32] <ottomata>	 or are all your scap3 system users in ldap in labs?
[22:08:35] <ottomata>	 with public keys?
[22:09:18] <thcipriani>	 they are, I haven't tried just using a key in userkeys, just assumed it should work.
[22:09:46] <Krenair>	 mediawiki deployments still appear to work from deployment-bastion
[22:10:05] <ottomata>	 hm, what is a user that scap3 deploys with from -bastion?
[22:10:08] <ottomata>	 mwdeploy
[22:10:09] <ottomata>	 ?
[22:10:11] <thcipriani>	 yeah, mwdeploy
[22:10:35] <ottomata>	 hmm, /usr/sbin/ssh-key-ldap-lookup mwdeploy doesn't seem to find a public key..>.
[22:10:36] <Krenair>	 I don't know about scap3, but normal scap uses keyholder
[22:10:50] <ottomata>	 Krenair:  yeah, to get the private key, i thikn that is working
[22:11:01] <thcipriani>	 same mechanism for scap v. scap3
[22:11:05] <ottomata>	 its that pam may not be configured auth with a public key not in ldap
[22:11:10] <ottomata>	 on the target host
[22:11:52] <ottomata>	 do either of you know how those keys are added to ldap?
[22:12:35] <ottomata>	 via wikitech?
[22:12:57] <ottomata>	 hmmm nawww, can't be :)
[22:13:14] <Krenair>	 wikitech controls the ssh keys in ldap, yes
[22:13:20] <ottomata>	 but for system users?
[22:13:22] <ottomata>	 i guess:
[22:13:26] <Krenair>	 system users shouldn't be in ldap
[22:13:29] <ottomata>	 does mwdeploy have a public key in ldap, and if so, how did it get there?
[22:14:01] <Krenair>	 no it doesn't
[22:14:04] <ottomata>	 ok
[22:14:09] <ottomata>	 so that must not be my issue then
[22:14:32] <ottomata>	 hmm, thcipriani is there somethign else deployed with scap3 in beta right now? maybe i can check auth.log on a target for that and compare
[22:15:13] <testing-shinken->	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.24 ms
[22:15:15] <thcipriani>	 restbase01 and restbase02 were deployed (as recently as a few weeks ago)
[22:15:25] <thcipriani>	 using deploy-service
[22:15:52] <shinken-wm>	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms
[22:16:24] <ottomata>	 ok via the deploy-service user?
[22:16:29] <thcipriani>	 yup
[22:17:33] <ottomata>	 ok cool, i see the same ldap error for that
[22:17:35] <ottomata>	 so that is not my problem
[22:18:20] <ottomata>	 Hmmmm
[22:18:30] <ottomata>	 no public key in /etc/ssh/userkeys for deploy-service
[22:18:32] <ottomata>	 HMmMM
[22:19:00] <ottomata>	 Hmmm where could it be?!
[22:20:37] <hashar>	 for what it is worth deployment-aqs01 logs "Error: Cannot find module '/srv/deployment/restbase/deploy/restbase/server.js'"
[22:21:17] <hashar>	 and 
[22:21:18] <hashar>	 fatal: Access denied for user eventlogging by PAM account configuration [preauth]
[22:21:30] <hashar>	 (instances syslog are sent to logstash)
[22:22:01] <ottomata>	 aye yeah
[22:22:01] <ottomata>	 k
[22:22:31] <ottomata>	 dunno anything about that aqs one, but the pam eventlogging one is what i'm trying to solve
[22:23:15] <hashar>	 is that `eventlogging` user in the deployment-prep project?
[22:23:36] <ottomata>	 hm, no
[22:23:37] <ottomata>	 but I am
[22:24:05] <ottomata>	 does it have to be?
[22:24:26] <hashar>	 might be wong
[22:24:33] <hashar>	 but IIRC pam is configured to lookup in ldap 
[22:24:35] <ottomata>	 i guess i can just manually add and try
[22:24:39] <ottomata>	 yah, i can see it doing that
[22:24:42] <hashar>	 and if the user is not in the project, it is rejected magically
[22:24:55] <hashar>	 but then, some changes have been made to pam recently so...
[22:25:07] <ottomata>	 i thiiiink that is not my problem, but i will try it...
[22:25:36] <hashar>	 ottomata: I am wrong
[22:25:44] <ottomata>	 oh?
[22:25:44] <hashar>	 the  mwdeploy  user is not in the group :D
[22:26:37] <ottomata>	 aye
[22:26:39] <ottomata>	 hm yeah
[22:26:47] <ottomata>	 ok, right now, i'm looking at the deploy-service user, because that one certainly uses scap3
[22:27:04] <hashar>	 deployment-aqs01:~$ getent passwd deploy-service
[22:27:04] <hashar>	 deploy-service:x:998:998::/var/lib/scap:/bin/bash
[22:27:08] <ottomata>	 and, it was used to deploy restbase to deployment-restbase01 recently
[22:27:09] <hashar>	 but
[22:27:19] <hashar>	 deployment-aqs01:~$ getent passwd eventlogging
[22:27:20] <hashar>	 $
[22:27:21] <ottomata>	 and i see the same ldap key lookup error
[22:27:26] <ottomata>	 right hashar
[22:27:28] <ottomata>	 i am not deploying to aqs
[22:27:31] <ottomata>	 deployment-eventlogging04
[22:27:45] <ottomata>	 aqs is totally unrelated
[22:28:10] <hashar>	 oh sorry ignore me
[22:28:15] <hashar>	 I am not any useful :-/
[22:28:17] <ottomata>	 hehe
[22:28:17] <ottomata>	 :)
[22:28:30] <ottomata>	 well, almost, righ tnow, i need to find where the deploy-service public key is on restbase01
[22:28:40] <ottomata>	 that may be a clue
[22:28:53] <ottomata>	 or, maybe its possible whatever crazy pam changes happened over the weekend just broke scap3
[22:29:07] <ottomata>	 the last restbase01 deploy i'm looking happened on dec 8
[22:29:11] <thcipriani>	 so, it seems like, the deploy-service user for restbase01 no longer works
[22:29:30] <ottomata>	 ah!
[22:29:30] <thcipriani>	 this could be because of the un-cherry-picked puppetmaster patch
[22:29:31] <ottomata>	 ok
[22:29:34] * hashar blames puppet
[22:29:39] <ottomata>	 oh?
[22:29:44] <ottomata>	 by me?  no
[22:29:47] <ottomata>	 right?
[22:30:22] * thcipriani walks home from café
[22:30:30] <thcipriani>	 I'll be back in a minute
[22:30:32] <ottomata>	 :)
[22:30:33] <ottomata>	 k
[22:40:14] <ottomata>	 thcipriani:  ok, so, i guess, if deploy-service user isn't working, then neither will this, eh?
[22:40:21] <ottomata>	 i still am not sure how the deploy-service thing works at all anyway
[22:40:31] <ottomata>	 since I can't find its public key on restbase01
[22:41:06] <hashar>	 ah 
[22:41:07] <hashar>	 odules/beta/manifests/deployaccess.pp
[22:41:16] <hashar>	 that is the crazy rule I was looking for
[22:41:48] <hashar>	 that tells pam to allow deploy-service and mwdeploy users from the deployment-bastion
[22:41:57] <ottomata>	 bwaaa
[22:42:04] <ottomata>	 bwaaaaaa
[22:42:15] <hashar>	 I am not sure how / where it is applied though
[22:43:00] <ottomata>	 looks lke in wikitech ui
[22:43:02] <hashar>	 2015-11-03 22:05 bd808: applied ::beta::deployaccess on deployment-bastion via Special:NovaInstance
[22:43:02] <ottomata>	 https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=088a0575-c09b-4c42-88a5-4ef57d8705c0&project=deployment-prep&region=eqiad
[22:43:05] <ottomata>	 oooook
[22:43:22] <ottomata>	 hmmm i guess i should just add my user there, althouhg, seems a little crazy!
[22:44:21] <hashar>	 jenkins-deploy is not listed there though
[22:44:32] <ottomata>	 yeah iuNnoo man lots of stuff going on here..>
[22:44:34] <hashar>	 that is the user used by Jenkins to ssh in a few instances
[22:44:43] <ottomata>	 hm i can manually add access.conf.d entry and just try it i guess
[22:44:45] <ottomata>	 on my target
[22:45:33] <hashar>	 but then jenkins-deploy user is in ldap
[22:45:39] <hashar>	 (created via wikitech)
[22:45:49] <hashar>	 and it is a member of the  group   project-deployment-prep
[22:45:56] <hashar>	 so pam let it in
[22:46:03] <ottomata>	 hm ja ok
[22:46:04] <ottomata>	 trying this then
[22:46:38] <hashar>	 for mwdeploy ,  we had it added to admins data.yam with a 6xx uid
[22:46:48] <hashar>	 and a corresponding entry has been added to ldap iirc
[22:46:57] <hashar>	 so we have the same UID on all instances
[22:47:16] <hashar>	 but the mwdeploy is unknown to wikitech and thus is not in the project-deployment-prep  group
[22:47:23] <hashar>	 probably why there is the beta::access class
[22:48:10] <ottomata>	 Accepted publickey for eventlogging from 10.68.16.58 port 58491 ssh2: RSA 02:9b:99:e2:f0:16:70:a3:d2:5a:e6:02:a3:73:0e:b0 :)
[22:48:11] <ottomata>	 :D 
[22:48:20] <ottomata>	 ok so this is def the problem
[22:48:21] <ottomata>	 cool
[22:48:32] <hashar>	 oh
[22:48:40] <hashar>	 !!!!!!!!!!!!!!!!
[22:48:42] <hashar>	 now
[22:48:56] <hashar>	 you get a fully qualified ticket
[22:49:03] <ottomata>	 going to add this?
[22:49:04] <ottomata>	     security::access::config { 'beta-allow-eventlogging':
[22:49:04] <ottomata>	         content  => "+ : eventlogging : ${bastion_ip}\n",
[22:49:05] <hashar>	 time to estate to level 3 support :D
[22:49:44] <ottomata>	 hmmmm, this is pretty cumbersom though
[22:49:47] <hashar>	 estate -> escaladate
[22:49:55] <ottomata>	 so many steps to take in order to deploy a new repo in beta
[22:50:21] <hashar>	 what is the eventlogging user for anyway?
[22:50:25] <ottomata>	 for every new deploy user, one must add a special pam exception
[22:50:28] <ottomata>	 for eventlogging!
[22:50:34] <ottomata>	 we were deploying with trebuchet
[22:50:37] <hashar>	 oh
[22:50:38] <ottomata>	 which is root for everything
[22:50:42] <ottomata>	 i'm trying to do scap3
[22:50:48] <hashar>	 and will now deploy using scap3 as  eventlogging ?
[22:50:56] <hashar>	 so that would be the equivalent of service-deploy right ?
[22:50:58] <ottomata>	 well, the eventlogging-service for eventbus
[22:51:06] <ottomata>	 next quarter we'll migrate the other stuff over
[22:51:07] <ottomata>	 but ja
[22:51:09] <ottomata>	 yes
[22:51:10] <ottomata>	 exactly
[22:51:26] <hashar>	 so dont quote me
[22:51:44] <hashar>	 but i would assign eventlogging-service a UID in 6xx via data.yaml
[22:51:50] <hashar>	 add it in LDAP with matching uid
[22:52:10] <hashar>	 then do the security::access::config { 'beta-allow-eventlogging':  
[22:52:21] <ottomata>	 hmm, i dunno, the eventlogging user already exists in prod and on a few beta hosts
[22:52:23] <ottomata>	 it is a system user
[22:52:28] <ottomata>	 managed by puppet
[22:52:42] <ottomata>	 why would i need a ldap user?
[22:52:47] <hashar>	 fixed uid ?
[22:53:06] <ottomata>	 why though?
[22:53:10] <ottomata>	 nfs?
[22:53:10] <ottomata>	 :)
[22:53:12] <ottomata>	 no nfs here :)
[22:53:29] <ottomata>	 hashar:  i would do it if that was the standard for system users, but i don't think it is :/
[22:54:37] <ottomata>	 id' prefer to not have to deal with ldap here, especially since we won't be doing that in prod anyway
[22:54:38] <testing-shinken->	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[22:54:42] <hashar>	 so maybe  security::access::config { 'beta-allow-eventlogging': is enough
[22:54:46] <ottomata>	 heah
[22:54:49] <ottomata>	 it will let me in
[22:54:52] <ottomata>	 pretty sure
[22:54:57] <hashar>	 you can craft a puppet patch + cherry pick on beta cluster puppetmaster
[22:55:07] <hashar>	 then add the puppet class on the node and see what happens
[22:55:10] <hashar>	 should do 
[22:55:33] <hashar>	 would be worth asking on ops whether the system user should be given a unique id across the fleet
[22:55:44] <ottomata>	 i think i need to give it a real shell though, instead of /bin/false
[22:56:01] <shinken-wm>	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[22:56:13] <hashar>	 that cause other troubles with puppet though.  If puppet can't reach ldap while doing User[johndoe]  puppet creates a local johndoe user with a local uid
[22:56:45] <ottomata>	 naw, its a systemuser
[22:56:55] <ottomata>	 so it should create a local user
[22:56:57] <ottomata>	 i want it to
[22:57:02] <hashar>	 ok ok :-}
[22:57:31] <hashar>	 so seems security::access::config  is the unblocker
[22:57:34] <ottomata>	 yeah
[22:57:35] <ottomata>	 thanks!
[22:57:37] <ottomata>	 patching now
[22:57:45] <ottomata>	 it seems crazy though that I have to do all that! :) but ok!
[22:57:59] <ottomata>	 thcipriani:  is gonna have to update his https://doc.wikimedia.org/mw-tools-scap/scap3/ssh-access.html docs :)
[22:58:05] <hashar>	 yeah that is a constraint due to labs
[22:58:22] <ottomata>	 yeah, i guess it is just a labs thing, but mehHhH :0
[22:58:23] <ottomata>	 :)
[22:58:26] <hashar>	 all labs projects share the same underlying infra (same ldap / no network isolation etc)
[22:58:50] <hashar>	 so the isolation has been done at the instance level using trick such as the pam rules
[22:59:14] <hashar>	 I think coren and andrewbogott are working on better isolating labs projects and definitely reworked the labs pam system
[22:59:24] <chasemp>	 ottomata: there have been mucho changes in this arena as of late
[22:59:35] <hashar>	 so it is getting better.  2015 largely improved labs for sure 
[22:59:39] <ottomata>	 chasemp:  aye i have heard only alittle :)
[22:59:43] <chasemp>	 and some really weird consequences fyi, if you wouldn't mind making a ticket for what you are seeing
[22:59:49] <ottomata>	 but, i think that we ahve found the problem
[22:59:51] <chasemp>	 only because, I'm sure whatever it is, isn't just you
[22:59:59] <ottomata>	 chasemp:  i think its just a new scap3 related thing
[23:00:17] <ottomata>	 i'm the first non releng person trying to use it
[23:00:19] <ottomata>	 (i think)
[23:00:39] <ottomata>	 scap3 tries to ssh as a configured system user
[23:00:41] * hashar looks for a scap3 barn start to offer
[23:00:56] <chasemp>	 still worth a ticket :) ...if you can
[23:00:57] <ottomata>	 and labs doesn't allow that
[23:00:59] <ottomata>	 oooOk
[23:01:04] <chasemp>	 otehrwise good odds we stomp on the fix man
[23:01:08] <chasemp>	 as we are known to do
[23:01:16] <ottomata>	 should I assign to you? what project?
[23:01:25] <chasemp>	 labs project and you can leave traige
[23:01:29] <chasemp>	 and I'll talk to ppls
[23:02:12] <hashar>	 cc Bryan Davis to it since apparently he has hit that wall previously and introduced the magic pam rule
[23:02:45] <bd808>	 which one? The deploy keys in beta cluster?
[23:03:19] <hashar>	 bd808: sshing to an instance with a user not being part of deployment-prep project
[23:03:37] <hashar>	 worked around via modules/beta/manifests/deployaccess.pp
[23:03:51] <bd808>	 yeah. there is deep puppet magic I copied from somewhere else
[23:03:56] <hashar>	 which tells pam to allow deploy-service / mwdeploy
[23:04:05] <hashar>	 andrew is trying to add another user for eventlogging deployment via scap3
[23:04:41] <bd808>	 ah, yes. that would be the same issue
[23:05:09] <testing-shinken->	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms
[23:05:29] <hashar>	 jenkins-deploy  (used by Jenkins to ssh to instances) is a member of the labs project
[23:05:36] <bd808>	 It seems like there used to be a pam hack too
[23:05:37] <hashar>	 so not impacted :-}
[23:05:46] <bd808>	 but maybe that's not needed anymore
[23:05:51] <shinken-wm>	 RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms
[23:05:56] <ottomata>	 chasemp: https://phabricator.wikimedia.org/T121721
[23:05:58] <ottomata>	 danke
[23:06:20] <ottomata>	 for now i'm going to add an exception so i can test my deploys, but hopefully we wont' have to eventually :)
[23:07:14] * thcipriani just got home
[23:07:22] <hashar>	 oh no
[23:07:25] <hashar>	 tyler missed all the fun
[23:07:34] <thcipriani>	 evidently.
[23:07:46] <thcipriani>	 just catching up on the scrollback.
[23:07:47] <hashar>	 TL;DR:   modules/beta/manifests/deployaccess.pp
[23:08:05] <hashar>	 assuming pam auto reject users not being part of deployment-prep
[23:08:26] <hashar>	 we need a custom pam rule to whitelist users not being members of the project
[23:08:37] <hashar>	 jenkins-deploy (used by Jenkins) works just fine because it is a member
[23:08:49] <hashar>	 mwdeploy does thanks to the pam rule
[23:09:10] <hashar>	 eventlogging lacks a proper passport/visa and is rejected at the border
[23:10:11] <thcipriani>	 ah, that makes sense.
[23:10:19] <ottomata>	 thcipriani: so so so close!
[23:10:26] <ottomata>	 i need sudo rules?
[23:10:30] <ottomata>	 eventlogging is not allowed to run sudo on deployment-eventlogging04.  This incident will be reported.
[23:11:05] <thcipriani>	 ottomata: I think I know what this is, let me verify
[23:11:13] <hashar>	 ah sudo
[23:11:16] <ottomata>	 Command 'mkdir -p '/srv/deployment/eventlogging/eventbus-cache/cache'' returned non-zero exit status 1
[23:11:44] <ottomata>	 hm oh or do I need to make /srv/deployment/eventlogging ahve proper perms?
[23:12:18] <ottomata>	 i have to go very SOoOoON :)
[23:12:35] <hashar>	 deployment-prep has a bunch of sudo rules on https://wikitech.wikimedia.org/wiki/Special:NovaSudoer which I never really understood :/
[23:12:36] <thcipriani>	 https://phabricator.wikimedia.org/P2431
[23:12:49] <hashar>	 some might be provisioned via puppet as well
[23:12:55] <thcipriani>	 so the eventlogging user needs a sudo rule to sudo as itself
[23:13:02] <ottomata>	 AH HA
[23:13:03] <ottomata>	 ook
[23:13:04] <ottomata>	 :p
[23:13:19] <hashar>	 wrong test!
[23:13:37] <ottomata>	 thcipriani:  i think it will fail anyway
[23:13:38] <ottomata>	 since
[23:13:46] <ottomata>	  /srv/deployment/eventlogging
[23:13:48] <ottomata>	 is owned by root
[23:14:04] <ottomata>	 because it was originally created by trebuchet
[23:14:13] <ottomata>	 but, i think it shoudlnt' ahve to be owned by the deploy user
[23:14:37] <ottomata>	 what if I deployed different services as different users into that dir (unlikely, but maybe!)
[23:14:45] <ottomata>	 (in labs maybe it is likely )
[23:15:06] <ottomata>	 is the location of the cache dir configurable?
[23:15:37] <ottomata>	 i have 5 mins!  Ahhhh i so wanted to make this work before I left today, oh well!  lots of progress though!
[23:15:45] <ottomata>	 man, so many thanks to all of you
[23:15:48] <ottomata>	 so much help all day here today
[23:15:49] <hashar>	 heck you can ssh to it!
[23:16:00] <hashar>	 progress [#####___________]
[23:16:19] <ottomata>	 closer than that!
[23:16:19] <hashar>	 damn it is past midnight already
[23:16:29] <hashar>	 where is the midnightkickbot when i need it
[23:16:43] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: Continuous integration: wikimedia/fundraising/tools/DjangoBannerStats needs V+2 jobs - https://phabricator.wikimedia.org/T121723#1886315 (10awight)
[23:16:47] <thcipriani>	 ottomata: the cache dir is always: git_deploy_dir / git_repo '_cache'
[23:17:08] <thcipriani>	 ottomata: thanks for trying it out! :)
[23:17:59] <ottomata>	 thcipriani:  i think that will have problems, since git_deploy_dir is maanged by scap, and git_deploy_dir/eventlogging/eventbus is managed by scap
[23:18:04] <ottomata>	 git_deploy_dir/eventlogging is not
[23:18:30] <ottomata>	 so attempting to create git_deploy_dir/eventlogging/eventbus_cache could easily fail
[23:18:53] <ottomata>	 i think either put cache in git_deploy_dir/git_repo/.cache (or something)
[23:18:55] <ottomata>	 or make it configurable
[23:18:58] <ottomata>	 or put it in /tmp
[23:18:59] <ottomata>	 :p
[23:19:24] <ottomata>	 or
[23:19:31] <thcipriani>	 configurable is not a bad idea. Generally the thinking was that the transition would need some initial puppet work.
[23:19:34] <ottomata>	 git_deploy_dir/.cache/git_repo
[23:19:49] <ottomata>	 well, one that that is really coll about this
[23:20:14] <ottomata>	 is that techincally, is that in puppet the target doesn't even have to know final deploy_dir (if scap can create it)
[23:20:20] <ottomata>	 so, if I were to set
[23:20:32] <ottomata>	 git_deploy_dir: /home/otto/path/to/anything/i/want
[23:20:44] <grrrit-wm>	 (03PS1) 10Krinkle: Set up test pipeline for rcstream repository [integration/config] - 10https://gerrit.wikimedia.org/r/259616 
[23:20:57] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Set up test pipeline for rcstream repository [integration/config] - 10https://gerrit.wikimedia.org/r/259616 (owner: 10Krinkle)
[23:20:57] <ottomata>	 i think scap shoudl work, as long as deploy user can write in /home/otto/path/to/anything/i/want
[23:21:09] <ottomata>	 it shoudln't require puppet work to do it :)
[23:21:37] <ottomata>	 SOOoOOO, all places that deploy user needs to write to should be known by scap
[23:21:40] <ottomata>	 git_deploy_dir
[23:21:41] <ottomata>	 is fine
[23:21:44] <ottomata>	 git_deploy_dir/git_repo 
[23:21:45] <ottomata>	 is fine
[23:21:46] <ottomata>	 but
[23:21:56] <ottomata>	 git_deploy_dir/something/something_cache isn't
[23:22:10] <ottomata>	 becuase git_deploy_dir/something might be owned by who knows what :)
[23:22:28] <ottomata>	 OK
[23:22:31] <ottomata>	 yes, time to go
[23:22:38] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set up test pipeline for rcstream repository [integration/config] - 10https://gerrit.wikimedia.org/r/259616 (owner: 10Krinkle)
[23:22:40] <ottomata>	 thank you So mUch!  I'll probably bug you some more tomorrow
[23:22:42] <ottomata>	 we are really close
[23:22:43] <ottomata>	 :D
[23:22:51] <thcipriani>	 so the thinking was that in most cases the git_deploy_dir is /srv/deployment and since we were trying to scope deployments of a particular service to a particular user it would be easier to ensure the remote user could write to /srv/deployment/[repo]-cache than /srv/deployment/.cache/[repo]
[23:23:06] <Krinkle>	 !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/259616
[23:23:11] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[23:23:18] <ottomata>	 thcipriani:  i think that would be fine, if repo was not two directories
[23:23:23] <ottomata>	 e.g. eventlogging/eventbus
[23:24:14] <thcipriani>	 repo being enforced as two directories is sort of an arbitrary decision by trebuchet that's reasoning is a bit opaque. Wanted to move away from enforcing that, generally.
[23:26:41] <hashar>	 sleepy time. *wave*
[23:28:03] <hashar>	 kudos on scap3 / eventbus progressing somehow!
[23:28:26] <thcipriani>	 toodles hashar 
[23:48:01] <twentyafterfour>	 I'd like to go on the record that I hate PEP8, 80 character lines are for suckers
[23:52:58] <thcipriani>	 twentyafterfour: does that not make your code better?
[23:54:05] <twentyafterfour>	 short lines make code better? Not necessarily. Often it makes code worse, because it forces me to do stupid things
[23:54:43] <twentyafterfour>	 most of pep8 is ok but wtf is wrong with #comment   or ## comment
[23:55:11] <testing-shinken->	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[23:56:11] <twentyafterfour>	 and I haven't used an 80 column editor since wide-screen monitors became standard and unavoidable.  Even if I run my screen in portrait orientation it's a lot more than 80 cols.
[23:56:36] <shinken-wm>	 PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67)
[23:56:43] <thcipriani>	 pep8 related https://youtu.be/wf-BqAjZb8M
[23:57:07] <thcipriani>	 (my comment was sarcasm, by the by :))