[00:02:34] <Krenair>	 Okay, so someone had deleted https://gerrit.wikimedia.org/r/#/c/247587/2 from the puppetmaster
[00:06:51] <ostriches>	 We should use letsencrypt instead.
[00:09:26] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2513777 (10greg)
[00:11:10] <paladox>	 ostriches it seems that https://gerrit.wikimedia.org/r/#/c/299164/ is failing to be built
[00:11:17] <paladox>	 https://gerrit.wikimedia.org/r/#/c/302371/
[00:11:20] <paladox>	 which is strange
[00:11:33] <paladox>	 since it works here https://gerrit.wikimedia.org/r/#/c/301841
[00:11:58] <ostriches>	 00:03:09 gbp:error: upstream/2.12.2-wmf is not a valid treeish
[00:12:32] <paladox>	 yep
[00:12:39] <paladox>	 but it works in https://gerrit.wikimedia.org/r/#/c/301841
[00:12:49] <paladox>	 but breaks in  https://gerrit.wikimedia.org/r/#/c/299164/
[00:12:52] <paladox>	 ostriches ^^
[00:13:20] <ostriches>	 Because your patch introduces gbp.conf
[00:13:22] <ostriches>	 Mine doesn't have it
[00:13:43] <paladox>	 Yep
[00:13:53] <paladox>	 Which should fix  00:03:09 gbp:error: upstream/2.12.2-wmf is not a valid treeish
[00:14:06] <paladox>	 since yours fails too
[00:14:58] <ostriches>	 I know it fails
[00:15:04] <ostriches>	 It never worked, so I don't pay attention ;-)
[00:15:14] <paladox>	 oh
[00:15:16] <paladox>	 but it did
[00:15:38] <paladox>	 ostriches it built https://integration.wikimedia.org/ci/job/debian-glue-non-voting/44/artifact/gerrit_2.12.2+0~20160801235111.44+jessie+wikimedia~1.gbp0d41ac_all.deb
[00:16:28] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[00:20:00] <paladox>	 ostriches i got it working
[00:20:01] <paladox>	 https://integration.wikimedia.org/ci/job/debian-glue-non-voting/49/console
[00:20:02] <paladox>	 yay
[00:21:46] <paladox>	 ostriches i got the deb now
[00:21:47] <paladox>	 https://integration.wikimedia.org/ci/job/debian-glue-non-voting/49/artifact/gerrit_2.12.2-wmf.1+0~20160802001844.49+jessie+wikimedia~1.gbp8a56c5_all.deb
[00:21:55] <paladox>	 That is based on your change :)
[00:22:13] <paladox>	 all we need now is gerrit 2.12.3 :)
[00:27:07] <paladox>	 ostriches i think the deb will work now
[00:46:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[00:52:44] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[00:58:16] <Krenair>	 tada: https://upload.beta.wmflabs.org/wikisource/en/thumb/6/62/Wind_in_the_Willows_%281913%29.djvu/page7-1024px-Wind_in_the_Willows_%281913%29.djvu.jpg
[00:58:18] <Krenair>	 note the cert
[00:58:19] <Krenair>	 greg-g, ^
[00:59:12] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-Q1-July-Sep-2016, 13Patch-For-Review, 07Performance: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#2513846 (10Mattflaschen-WMF)
[01:07:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:21:28] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:29:15] <shinken-wm>	 RECOVERY - SSH on deployment-elastic06 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0)
[01:35:07] <shinken-wm>	 PROBLEM - SSH on deployment-elastic06 is CRITICAL: Server answer
[01:47:58] <wikibugs>	 10Deployment-Systems, 06Collaboration-Team-Triage, 10Flow, 07I18n: Message Mediawiki:Flow-terms-of-use-edit on nowp is English - https://phabricator.wikimedia.org/T133571#2514026 (10Mattflaschen-WMF)
[01:55:16] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 569 bytes in 0.002 second response time
[01:57:28] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[01:58:44] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[01:59:08] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 567 bytes in 0.003 second response time
[02:00:08] <shinken-wm>	 RECOVERY - SSH on deployment-elastic06 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0)
[02:06:09] <shinken-wm>	 PROBLEM - SSH on deployment-elastic06 is CRITICAL: Server answer
[02:12:52] <shinken-wm>	 RECOVERY - SSH on deployment-elastic06 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0)
[02:13:03] <wmf-insecte>	 Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #103: 04FAILURE in 2.7 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/103/
[02:13:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:30:20] <Krenair>	 I'll deal with the other domains of the text part of that tomorrow
[02:30:29] <Krenair>	 certainly the ones affecting monitoring
[02:32:31] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:41:08] <wmf-insecte>	 Project selenium-CirrusSearch » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #103: 04FAILURE in 7.6 sec: https://integration.wikimedia.org/ci/job/selenium-CirrusSearch/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/103/
[02:44:59] <Krenair>	 hm... I can fiddle with the https-forcing to leave those domains for now I guess
[02:49:12] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 44821 bytes in 1.919 second response time
[02:50:17] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 32491 bytes in 0.983 second response time
[03:05:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 06Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#2514079 (10Krenair) a:03Krenair
[03:06:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 06Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#527800 (10Krenair) This is now working for meta.wikimedia.beta.wmflabs.org and deployment.wikimedia.beta.wmflabs.org (and their...
[03:43:28] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[06:46:53] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[06:58:16] <shinken-wm>	 PROBLEM - Puppet run on phab-beta is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[07:26:51] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:05:10] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 567 bytes in 0.003 second response time
[08:06:18] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 569 bytes in 0.004 second response time
[08:09:47] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[08:21:29] <wikibugs>	 10Deployment-Systems, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics-Cluster, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2514384 (10MoritzMuehlenhoff) @elukey: I've dropped Yuvi's expired key from pwstore, so new entries can be added now.
[08:23:32] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[08:24:26] <shinken-wm>	 PROBLEM - Puppet run on deployment-parsoid07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[08:58:31] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:59:23] <shinken-wm>	 RECOVERY - Puppet run on deployment-parsoid07 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:03:29] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:14:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:44:30] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[09:53:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:49] <shinken-wm>	 RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:27] <shinken-wm>	 RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:34] <shinken-wm>	 RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:22:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Parsoid, 06Services, and 2 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.3 - https://phabricator.wikimedia.org/T125003#2514761 (10mobrovac)
[10:41:51] <wikibugs>	 10Deployment-Systems, 03Scap3: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#2514822 (10fgiunchedi) 05Open>03Resolved this is completed!
[11:20:23] <wikibugs>	 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Regression, 07Unplanned-Sprint-Work: [Regression] Fix browser tests for language switching on the beta cluster - https://phabricator.wikimedia.org/T141647#2514959 (10phuedx) @dr0ptp4kt: [selenium-MobileFrontend](https://...
[11:35:07] <shinken-wm>	 PROBLEM - SSH on deployment-elastic06 is CRITICAL: Server answer
[11:50:29] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[11:58:07] <grrrit-wm>	 (03PS2) 10Hashar: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) 
[11:58:16] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[11:58:27] <grrrit-wm>	 (03PS3) 10Hashar: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) 
[11:58:36] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[11:59:24] <grrrit-wm>	 (03PS4) 10Paladox: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[11:59:33] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[11:59:59] <grrrit-wm>	 (03PS5) 10Hashar: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) 
[12:00:08] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[12:01:16] <wmf-insecte>	 Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #99: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/99/
[12:01:20] <wmf-insecte>	 Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #99: 04FAILURE in 19 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/99/
[12:03:23] <grrrit-wm>	 (03PS6) 10Paladox: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[12:03:57] <grrrit-wm>	 (03CR) 10Paladox: "Rebased." [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar)
[12:25:29] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2515119 (10Aklapper) I appreciate this as I considered this my work so far. :P (I tried more like every other week, plus only pinging / naggin...
[13:10:06] <shinken-wm>	 RECOVERY - SSH on deployment-elastic06 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0)
[13:16:07] <shinken-wm>	 PROBLEM - SSH on deployment-elastic06 is CRITICAL: Server answer
[13:44:25] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #99: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/99/
[13:46:08] <shinken-wm>	 RECOVERY - SSH on deployment-elastic06 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0)
[13:47:22] <gehel>	 ^ I'm having a look at deployment-elastic06 ... strange...
[14:02:54] <gehel>	 !log deployment-prep rebooting deployment-elastic06 (unresponsive to SSH and Salt)
[14:09:20] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-elastic06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0]
[14:19:19] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [3600.0]
[14:34:58] <paladox>	 ostriches hi, would it be correct do this https://gerrit.wikimedia.org/r/#/c/302416/2 in the grrrit-wm bot
[14:35:06] <ottomata>	 morning
[14:35:14] <ottomata>	 possible bug in scap3  on deployment-tin
[14:35:31] <ottomata>	   File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 608, in checks_setup
[14:35:31] <ottomata>	     if not os.path.exists(checks_path):
[14:35:31] <ottomata>	   File "/usr/lib/python2.7/genericpath.py", line 18, in exists
[14:35:31] <ottomata>	     os.stat(path)
[14:35:32] <ottomata>	 TypeError: coercing to Unicode: need string or buffer, NoneType found
[14:35:32] <ottomata>	 14:34:55 deploy failed: <TypeError> coercing to Unicode: need string or buffer, NoneType found
[14:35:32] <wikibugs>	 10Deployment-Systems, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics-Cluster, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2515412 (10elukey) Created the keys in the private repo and encrypted them with the pass stored in pwstore under analytics-deplo...
[14:35:39] <paladox>	 Since it seems message.uploader.name carnt be used in the gerrit edit in browser
[14:35:44] <paladox>	 ?
[14:46:56] <thcipriani|afk>	 ottomata: which repo are you seeing that on?
[14:47:27] <ottomata>	 thcipriani:  i saw it on both eventlogging/eventbus and eventlogging/analytics
[14:47:32] <ottomata>	 brb...
[14:47:35] <thcipriani>	 ack, checking
[14:49:18] * paladox brb restarting my pc due to high disk usage slowing my pc down, plus a windows 10 update to install. Bash gets released to all windows 10 users today :) 400+ million users will have access to bash in a few hours once microsoft starts rolling out the windows 10 update :)
[14:54:34] <paladox>	 Im back
[14:54:44] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:55:12] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:58:43] <paladox>	 ostriches i changed message.uploader.name to message.author.name here https://gerrit.wikimedia.org/r/#/c/302416/ would that work?
[14:58:55] <ostriches>	 I don't know :)
[14:59:54] <paladox>	 Oh ok
[15:00:13] <paladox>	 ostriches but i did a test today and found that uploading through ssh or http worked
[15:00:22] <paladox>	 but doing it through the web ui and editing files didnt
[15:00:30] <ostriches>	 Probably not, actually.
[15:00:36] <paladox>	 Oh
[15:00:42] <ostriches>	 Eh, author.name might work.
[15:00:58] <paladox>	 Oh yeh
[15:01:03] <ostriches>	 patchSet.*.uploader.name might also work
[15:01:05] <paladox>	 since that is what patch comment uses
[15:01:08] <ostriches>	 Where * is the latest patch set.
[15:01:08] <paladox>	 Oh
[15:01:10] <ostriches>	 :)
[15:01:20] <paladox>	 message.patchSet.*.uploader.name
[15:01:27] <paladox>	 something like ^^ ostriches
[15:01:47] <ostriches>	 Well, you'd need to iterate over the patchSet array items and figure out which the latest is.
[15:01:54] <ostriches>	 * could be 1, or it could be 45 :)
[15:02:08] <paladox>	 Oh but woulden message.author.name that do it
[15:02:08] <ottomata>	 (back)
[15:02:19] <ostriches>	 Oh wait....
[15:02:25] <ostriches>	 It's always just the latest in stream-events
[15:02:29] <paladox>	 Yeh
[15:03:00] <paladox>	 It's just probaly the web ui dosent set off message.uploader.name so that just uses the authers name
[15:03:13] <ostriches>	 patchSet.uploader.name.
[15:03:18] <paladox>	 Yeh
[15:03:24] <paladox>	 That is what openstack uses too
[15:03:25] <ostriches>	 Or patchSet.author.nae
[15:03:27] <ostriches>	 *name
[15:03:30] <paladox>	 Oh
[15:03:33] <ostriches>	 Either way, we want the patchSet one
[15:03:37] <paladox>	 Ok
[15:03:41] <paladox>	 I will do that now
[15:03:41] <ostriches>	 Not the top-level author :)
[15:03:47] <paladox>	 thanks
[15:04:12] <ostriches>	 I've never once looked at this code, so I'm just guessing really ;-)
[15:04:34] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44501 bytes in 1.038 second response time
[15:04:38] <thcipriani>	 ottomata: I see the problem I have a patch https://phabricator.wikimedia.org/D302, this patch is also live on deployment-tin if you'd like to try it.
[15:04:38] <ottomata>	 thcipriani: one possibly unique fact, I am trying to deploy an in review change from gerrit
[15:04:43] <ottomata>	 i fetched and checked out the change
[15:04:49] <ottomata>	 oh ok
[15:04:50] <ottomata>	 trying
[15:04:51] <paladox>	 ostriches ok, yeh but either one of those could fix it, were not really sure
[15:04:57] <paladox>	 since it works
[15:04:59] <paladox>	 for ssh and http
[15:05:02] <paladox>	 but not web ui
[15:05:03] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 44499 bytes in 0.973 second response time
[15:05:11] <ostriches>	 Author is probably better than uploader.
[15:05:24] <paladox>	 Oh ok
[15:05:27] <paladox>	 i will change it now
[15:05:30] <paladox>	 thanks
[15:05:51] <ostriches>	 Although, I wonder if *both* would be best :)
[15:06:05] <ottomata>	 thcipriani:  looks good!
[15:06:18] <thcipriani>	 ottomata: thank you so much for that catch!
[15:06:31] <ostriches>	 Prolly harder though :)
[15:06:59] <ostriches>	 So you'd get something like....
[15:07:07] <ostriches>	 (CR) Ottomata: [C: 2] Hieraize eventlogging_kafka_handler to allow selection of different kafka clients [puppet] - https://gerrit.wikimedia.org/r/301126 (owner: Ottomata, author: Chad)
[15:07:11] <ottomata>	 thanks for the fast fix! :)
[15:07:14] <ostriches>	 (Sorry for the ping ottomata, just needed an example)
[15:07:59] <paladox>	 Oh
[15:08:00] <thcipriani>	 ottomata: deployment-tin has the latest scap package that is pretty close to going out live! Close call.
[15:08:38] <thcipriani>	 feature list: https://www.mediawiki.org/wiki/Deployment_tooling/Cabal/2016-08-01#Scap_v.3.2.1_tagged_.28live_on_beta.29
[15:08:41] <paladox>	 ostriches so doing message.patchSet.uploader.name would do what you done in the example
[15:08:50] <paladox>	 and put both owner and author
[15:09:01] <ostriches>	 Yeah :)
[15:09:05] <paladox>	 Ok
[15:09:09] <paladox>	 Thanks for explaning
[15:09:19] <paladox>	 Ive changed it to message.patchSet.author.name now
[15:09:23] <ostriches>	 Then it's just super clear to everyone who all is involved :)
[15:09:32] <paladox>	 Yep
[15:09:57] <paladox>	 Now we need to find one of the authors to deploy it 
[15:10:00] <paladox>	 so we can test
[15:10:08] <paladox>	 weather we fix it or broke more things lol
[15:11:40] <paladox>	 ostriches do you have to build the war for gerrit 2.12.3 or can you use the one they bult?
[15:11:44] <paladox>	 bult = built
[15:12:13] <ostriches>	 I build them myself. I also have to build the its-* plugins so it's no big deal.
[15:12:33] <paladox>	 Oh
[15:12:38] <paladox>	 I tryed building it
[15:12:40] <paladox>	 but failed
[15:12:49] <paladox>	 LOL
[15:13:24] <paladox>	 ostriches do you know when they will be built ?
[15:14:38] <ostriches>	 Soon :)
[15:15:55] <paladox>	 Ok, soon as in today, a week or next week.
[15:15:56] <paladox>	 ?
[15:16:06] <paladox>	 ostriches ^^
[15:17:54] <paladox>	 :)
[15:19:14] <paladox>	 I now get error
[15:19:15] <paladox>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item network::external in any Hiera data file and no default supplied at /etc/puppet/modules/network/manifests/constants.pp:5 on node gerrit-test3.git.eqiad.wmflabs
[15:19:15] <paladox>	 Warning: Not using cache on failed catalog
[15:19:15] <paladox>	 Error: Could not retrieve catalog; skipping run
[15:19:28] <paladox>	 when running sudo puppet agent -tv
[15:19:31] <paladox>	 ostriches ^^
[15:33:08] <wmf-insecte>	 Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #100: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/100/
[15:35:07] <shinken-wm>	 PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[15:38:27] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #100: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/100/
[15:43:17] <Amir1>	 ostriches: hey, we already deploy from gerrit. We have a ores submodule which is mirrored from github but the real repo is  https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/deploy
[15:44:13] <ostriches>	 Apologies if I misunderstood then. Github deploys scare meh :p
[15:44:18] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 07Wikimedia-Incident: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2515594 (10greg) >>! In T141066#2513427, @greg wrote: >>>! In T141066#2496346, @faidon wrote: >> "First responders" or "f...
[15:44:26] <Amir1>	 updating submodule requires a commit which means it needs to be done in gerrit 
[15:44:39] <Amir1>	 https://gerrit.wikimedia.org/r/#/q/project:mediawiki/services/ores/deploy
[15:44:40] <paladox>	 You can do it from gerrit gui
[15:44:42] <paladox>	 :)
[15:44:42] <shinken-wm>	 PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[15:44:58] <greg-g>	 Krenair: !!!!!!!!!!!!!!!!!!
[15:45:06] <greg-g>	 Krenair: re the LE cert on beta
[15:45:14] <Krenair>	 Hi greg-g
[15:45:14] <ostriches>	 Amir1: Ah, so it's a submodule pointing at Github but the submodule is Gerrit-maintained? Eh, better. But still not best :)
[15:45:42] <shinken-wm>	 PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[15:45:43] <Amir1>	 yeah
[15:47:02] <Amir1>	 (I'm not huge of this current complex gerrit-diffusion-github-based system too)
[15:48:50] <Amir1>	 greg-g: One of the things Aaron and I came to agreement is that we will do deployments in the "Services" window only from now on. Is it okay. Or we should get another window?
[15:49:18] <greg-g>	 Amir1: that is correct, which is what you did yesterday, yes?
[15:49:56] <Amir1>	 that wasn't intentional 
[15:50:05] <Amir1>	 but it was at the same time
[15:50:23] <greg-g>	 Amir1: what window were you using?
[15:50:28] <greg-g>	 I don't see it on the calendar
[15:50:38] <greg-g>	 did you just pick a random time? that's a very bad idea
[15:51:07] <Amir1>	 greg-g: not randomly, when there was no other windows 
[15:51:13] <greg-g>	 Amir1: that's not how it works
[15:51:19] <Amir1>	 yeah and I know it
[15:51:30] <Amir1>	 that's why I suggested to get a window 
[15:51:59] <greg-g>	 well, now I'm even less happy about what happened yesterday
[15:52:02] <greg-g>	 to be fully honest
[15:52:45] <shinken-wm>	 PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[15:52:53] <greg-g>	 I'm not going to be responsive on IRC for a bit (a meeting) but, no more picking a window yourself, schedule *with me* or use the services window (in coordination with them)
[15:52:57] <shinken-wm>	 PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[15:53:05] <shinken-wm>	 PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[15:53:49] <shinken-wm>	 PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[15:55:02] <Amir1>	 greg-g: let's talk once you're done. Please ping me you have some free time
[15:56:09] <greg-g>	 or continue the thread on the ops list and/or the talk page
[15:57:02] <greg-g>	 but I'll try to ping when I can
[15:57:07] <Amir1>	 thanks
[15:59:15] <ottomata>	 hm, puppet repo on tin seems a little borked
[15:59:19] <ottomata>	 trying to run 
[15:59:25] <ottomata>	  /usr/local/bin/git-sync-upstream
[15:59:28] <paladox>	 Yeh
[15:59:30] <ottomata>	 error: could not apply 2218144... puppetization for thumbor
[15:59:38] <paladox>	 See -operations
[16:00:35] <ottomata>	 hm paladox not sure i follwed in my quick skim there, but that doesn't seem related
[16:00:43] <ottomata>	 sounds like they are talking about a labs wide hiera lookup problem?
[16:00:49] <paladox>	 Oh yeh sorry
[16:00:50] <ottomata>	 i'm just having problems doing the rebase
[16:00:54] <ottomata>	 on deployment-puppetmaster
[16:00:58] <paladox>	 Oh
[16:01:01] <ottomata>	 (oh sorry, not tin, deployment-puppetmaster
[16:01:01] <thcipriani>	 ottomata: hmm, looks like that patch merged, but may also be cherry-picked?
[16:01:01] <ottomata>	 )
[16:01:09] <thcipriani>	 https://gerrit.wikimedia.org/r/#/c/300827/
[16:01:10] <paladox>	 i think someone merged patches related to your error
[16:01:19] <ottomata>	 thcipriani:  should I remove the thing from thumbor?
[16:01:24] <ottomata>	 the commit from the rebase list?
[16:02:32] <thcipriani>	 yeah, I'd remove that cherry-pick from the rebase list
[16:02:37] <thcipriani>	 should work then, I reckon.
[16:02:45] <thcipriani>	 (unless there are other conflicts)
[16:03:44] <ottomata>	 k
[16:04:23] <Krenair>	 it's probably an old version of a now-merged commit, highly likely to conflict
[16:05:08] <shinken-wm>	 RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0]
[16:09:44] <andrewbogott>	 ok, y'all, I'm going to break CI in about 30 seconds.
[16:12:06] <thcipriani>	 andrewbogott: ack.
[16:15:01] <godog>	 oh yeah sorry about thumbor ottomata thcipriani, I merged the change in production so that likely broke it
[16:15:26] <ottomata>	 aye, i am in standup but haven't been able to fix it yet
[16:15:34] <ottomata>	 godog:  got a sec to see if you can make the rebase happy?
[16:15:44] <ottomata>	 even if i remove that commit from the rebase list, i get Could not apply 2218144cb26430d971f4b546889477588a583fe2... puppetization for thumbor
[16:15:50] <ottomata>	 error: could not apply 2218144... puppetization for thumbor
[16:16:01] <Krenair>	 okay, everyone stop for a moment.
[16:16:55] <godog>	 ok!
[16:17:08] <Krenair>	 I'm looking
[16:19:25] <Krenair>	 godog, ottomata: Ok
[16:19:31] <Krenair>	 It should be fixed now
[16:19:48] <Krenair>	 I removed these two:
[16:19:52] <Krenair>	 9d13910 puppetization for thumbor
[16:19:54] <Krenair>	 2218144 puppetization for thumbor
[16:20:09] <Krenair>	 using `git rebase -i HEAD~20`
[16:20:16] <Krenair>	 then ran `git pull --rebase origin production`
[16:20:35] <Krenair>	 root@deployment-puppetmaster:/var/lib/git/operations/puppet# git log --oneline | grep "puppetization for thumbor"
[16:20:35] <Krenair>	 e959e6a puppetization for thumbor
[16:20:35] <Krenair>	 root@deployment-puppetmaster:/var/lib/git/operations/puppet# 
[16:21:11] <wikibugs>	 10Beta-Cluster-Infrastructure: Admin permissions for cawiki at beta-cluster - https://phabricator.wikimedia.org/T141890#2515732 (10Toniher)
[16:21:40] <godog>	 thanks Krenair !
[16:21:55] <Krenair>	 godog, is tht the only "puppetization for thumbor" commit you expect?
[16:21:58] <wikibugs>	 10Beta-Cluster-Infrastructure: Admin permissions for cawiki at beta-cluster - https://phabricator.wikimedia.org/T141890#2515744 (10Toniher)
[16:22:12] <godog>	 Krenair: the only I cherry-picked yeah iirc
[16:22:28] <Krenair>	 okay then I think we're good
[16:22:50] <sachins301>	 hi
[16:23:16] <Krenair>	 hi sachins301 
[16:23:30] <godog>	 I'm sure this has been discussed forever, I wonder if there's a way to make git-sync-upstream DTRT in cases like this, where the change has been merged but another PS was previously cherry-picked
[16:24:01] <Krenair>	 maybe using Change-Ids
[16:24:12] <Krenair>	 it has "# TODO: rewrite in python?" near the top
[16:24:13] <thcipriani>	 git-cherry is for this iirc
[16:24:17] <Krenair>	 may want to do that first
[16:24:37] <thcipriani>	 https://git-scm.com/docs/git-cherry
[16:26:19] <ostriches>	 git pull --rebase :)
[16:26:30] <paladox>	 ostriches the deb is being built
[16:26:33] <paladox>	 and uploaded
[16:26:36] <ostriches>	 I know.
[16:26:39] <godog>	 thcipriani: nice! though I'm not sure it'd work if the diffs are different like in different PSes
[16:26:39] <paladox>	 Ok
[16:27:09] <paladox>	 ostriches do you know if gerrit 2.12.3 will be built today?
[16:27:16] <ostriches>	 No, I don't.
[16:27:19] <paladox>	 ok
[16:27:21] <ostriches>	 I said soon.
[16:27:25] <paladox>	 ok
[16:27:45] <shinken-wm>	 RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:28:05] <shinken-wm>	 RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:28:47] <shinken-wm>	 RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0]
[16:29:45] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:30:11] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[16:30:17] <shinken-wm>	 PROBLEM - Puppet run on integration-puppetmaster is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[16:31:40] <thcipriani>	 hrm, yeah, different diffs would be problematic... Krenair is probably right about Change-Ids. Something like doing a --theirs in cases where there are conflicts with the same change-id. Although a switch to differential would break that.
[16:32:21] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[16:32:33] <shinken-wm>	 PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[16:32:57] <shinken-wm>	 RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:34:32] <greg-g>	 should probably kill shinken during the labs upgrade window?
[16:36:48] <shinken-wm>	 PROBLEM - Puppet run on deployment-poolcounter01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[16:37:24] <shinken-wm>	 PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[16:38:42] <Krenair>	 these are all "Failed to determined $::labsproject" ?
[16:38:51] <paladox>	 yeh
[16:38:55] <paladox>	 im getting that too
[16:40:08] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:40:24] <wikibugs>	 10Beta-Cluster-Infrastructure, 15User-Luke081515: Admin permissions for cawiki at beta-cluster - https://phabricator.wikimedia.org/T141890#2515777 (10Luke081515) a:03Luke081515
[16:41:13] <shinken-wm>	 PROBLEM - Puppet run on integration-jessie-lego-test01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[16:42:09] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[16:42:49] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[16:44:01] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:44:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515795 (10greg)
[16:45:05] <shinken-wm>	 PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[16:46:10] <shinken-wm>	 PROBLEM - Puppet run on integration-aptly01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[16:46:36] <shinken-wm>	 PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[16:46:41] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2515799 (10greg) >>! In T141130#2515119, @Aklapper wrote: > I appreciate this as I considered this my work so far. :P (I tried more like every...
[16:47:52] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:48:35] <shinken-wm>	 PROBLEM - Puppet run on integration-raita is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[16:49:36] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515807 (10Ladsgroup) There were two bugs in the yesterday deployment. We couldn't...
[16:50:31] <greg-g>	 I just muted shinken-wm, will unmute after the labs stuff is done
[16:53:12] <ottomata>	 ah there were two!
[16:53:13] <ottomata>	 ok
[16:53:21] <Krenair>	 after the labs stuff is done I might be able to get it to show a useful hostname instead of that labs IP
[16:53:38] <paladox>	 :)
[16:53:52] <paladox>	 I think nodepool may also need to be restarted after labs upgrade
[16:54:04] <paladox>	 not sure if it will resolve its self or is stuck.
[16:54:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515821 (10Halfak) @greg, I'm not sure what you're looking for here.  We obviously...
[16:58:03] <paladox>	 ostriches: it seems this https://phabricator.wikimedia.org/T141887 has popped up
[16:58:31] <ostriches>	 Meh
[16:59:12] <paladox>	 ostriches yep, not sure why it has problems? do you see any errors in the error log even though you may get lost in there.
[17:02:44] <paladox>	 ostriches the its-base plugin builds against the gerrit 2.11.5 api.
[17:02:49] <paladox>	 it should be 2.12.*
[17:02:55] <paladox>	 https://gerrit.googlesource.com/plugins/its-base/
[17:04:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 15User-Luke081515: Admin permissions for cawiki at beta-cluster - https://phabricator.wikimedia.org/T141890#2515840 (10Luke081515) 05Open>03Resolved Done:         (change visibility) 16:45, 2 August 2016 Luke081515 (talk | contribs | block) changed group membership for Tonihe...
[17:07:31] <wikibugs>	 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Regression, 07Unplanned-Sprint-Work: [Regression] Fix browser tests for language switching on the beta cluster - https://phabricator.wikimedia.org/T141647#2515870 (10dr0ptp4kt) @phuedx, @jhobs, @bmansurov for your engine...
[17:14:16] * paladox yay windows 10 update has begun rolling out https://blogs.windows.com/windowsexperience/2016/08/02/how-to-get-the-windows-10-anniversary-update/
[17:14:23] * paladox bash now aviable worlwide
[17:19:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515946 (10greg) >>! In T141825#2515821, @Halfak wrote: > If you are asking how we...
[17:25:40] <paladox>	 ostriches hi, i still get
[17:25:40] <paladox>	 root@gerrit-test3:/home/paladox# systemctl status gerrit.service
[17:25:40] <paladox>	 ● gerrit.service - LSB: Start/stop Gerrit Code Review
[17:25:40] <paladox>	    Loaded: loaded (/etc/init.d/gerrit)
[17:25:40] <paladox>	    Active: failed (Result: exit-code) since Tue 2016-08-02 17:25:02 UTC; 23s ago
[17:25:40] <paladox>	   Process: 27252 ExecStart=/etc/init.d/gerrit start (code=exited, status=1/FAILURE)
[17:25:42] <paladox>	 Aug 02 17:25:02 gerrit-test3 gerrit[27252]: ** ERROR: GERRIT_SITE not set
[17:25:44] <paladox>	 Aug 02 17:25:02 gerrit-test3 systemd[1]: gerrit.service: control process exited, code=exited status=1
[17:25:47] <paladox>	 Aug 02 17:25:02 gerrit-test3 systemd[1]: Failed to start LSB: Start/stop Gerrit Code Review.
[17:25:49] <paladox>	 Aug 02 17:25:02 gerrit-test3 systemd[1]: Unit gerrit.service entered failed state.
[17:25:51] <paladox>	 even after updating gerrit
[17:29:02] <wikibugs>	 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Regression, 07Unplanned-Sprint-Work: [Regression] Fix browser tests for language switching on the beta cluster - https://phabricator.wikimedia.org/T141647#2515984 (10jhobs) a:05dr0ptp4kt>03jhobs A new build is runnin...
[17:29:07] <wikibugs>	 06Release-Engineering-Team, 06Operations, 06Services, 07Wikimedia-Incident: Review new service 'pre-deployment to production' checklist - https://phabricator.wikimedia.org/T141897#2515987 (10greg)
[17:30:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2516001 (10greg) >>! In T141825#2515946, @greg wrote: > I think a separate task ab...
[17:34:50] <paladox>	 ostriches ?
[17:39:27] <paladox>	 ostriches im not sure why it is not setting GERRIT_SITE
[17:39:28] <paladox>	 ?
[18:15:43] <Amir1>	 greg-g: do have some free time to talk about the deployment window?
[18:17:33] <greg-g>	 Amir1: sure, sorry, was catching up with other things
[18:18:28] <Amir1>	 greg-g: okay, it's either we should get a dedicated deployment window every day or we piggy back in services
[18:18:54] <Amir1>	 I need to mention that ORES can't be deployed by services team
[18:20:15] <Amir1>	 depends on you and Ops obliviously 
[18:22:01] <greg-g>	 why every day?
[18:23:35] <Amir1>	 we don't deploy every day
[18:23:51] <Amir1>	 but maybe we need to do a urgent fix
[18:23:59] <Amir1>	 *an
[18:24:08] <greg-g>	 urgent fixes are urgent fixes and don't need a window
[18:24:20] <greg-g>	 you just need to coordinate with me/ops when you do need to
[18:24:39] <Amir1>	 hmm, okay
[18:24:39] <greg-g>	 and if it's really urgent and you discovered it 23 hours before your next window, would you wait?
[18:24:46] <greg-g>	 the answer should be no
[18:25:00] <Amir1>	 yes, you're right
[18:25:24] <greg-g>	 so, anyways, I think for now ORES sould be a part of the Services window (which is every day, just different times)
[18:25:25] <Amir1>	 we are learning and trying to adapt a production-like deployment system 
[18:25:46] <greg-g>	 yes, thanks for learning :) it's all we all can do
[18:26:21] <Amir1>	 okay. Sounds good. Just one thing. What should we do as coordination with the services team? 
[18:27:08] <Amir1>	 like, asking if they are doing anything at that time. 
[18:27:42] <greg-g>	 good question. When the window starts jouncebot will ping you (and everyone else int eh window), those who are planning to deploy something will then say "I plan to do X" or "no X deploy today". Based on that information you all self-organize and do the needfull.
[18:28:17] <greg-g>	 honestly, I haven't taken much active management of it (maybe a mistake) as initially it didn't need it and my active management wasn't desired (back when it was just parsoid)
[18:29:07] <Amir1>	 okay. So probably I need to add my name and Aaron to that section
[18:29:18] <Amir1>	 I hope that's okay for you. 
[18:29:19] <greg-g>	 exactly, that's how you join the window :)
[18:29:26] <Amir1>	 https://wikitech.wikimedia.org/wiki/Deployments
[18:29:59] <Amir1>	 Is there any place we should add? documentations somewhere? 
[18:30:52] <greg-g>	 all of the services windows
[18:31:05] <greg-g>	 I copy/paste the last week for the next week so it'll stay in there
[18:31:47] <Amir1>	 sure
[18:42:44] <greg-g>	 Amir1: (just to be clear, are you adding your names to that wiki page or am I? I'd appreciate you :) )
[18:42:58] <Amir1>	 greg-g: editing right now
[18:43:01] <Amir1>	 it's super big
[18:44:08] <Amir1>	 https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=815996&oldid=815934
[18:44:12] <Amir1>	 greg-g: ^
[18:46:39] <greg-g>	 yeah, I added a lot of future weeks to get the new european swat window listed
[18:47:11] <Amir1>	 oh, great
[18:47:20] <Amir1>	 nice, it'll be in 20 days
[18:47:26] <greg-g>	 yup :)
[18:50:26] <andrewbogott>	 CI should be back running tests again.  Let me know if you discover otherwise.
[18:51:08] <thcipriani>	 will do, looks like some tests are starting to trickle through now
[18:52:05] <greg-g>	 I just saw a ci-jessie-wikimedia-blah spawn on the jenkins ui
[18:52:12] <greg-g>	 (a nodepool instance, that is)
[18:52:20] <greg-g>	 and a bunch more now
[18:52:22] <greg-g>	 \o/
[18:52:49] <paladox>	 :)
[19:12:03] <paladox>	 ostriches yes could you fix it in the package please
[19:12:04] <paladox>	 ?
[19:13:42] <paladox>	 ostriches could you make Letsencrypt optional on labs please
[19:13:47] <paladox>	 since im hitting a error with it
[19:14:02] <paladox>	 root@gerrit-test3:/var/lib/gerrit2/review_site/logs# journalctl -xn
[19:14:02] <paladox>	 -- Logs begin at Mon 2016-08-01 18:08:54 UTC, end at Tue 2016-08-02 18:49:43 UTC. --
[19:14:02] <paladox>	 Aug 02 18:49:25 gerrit-test3 puppet-agent[1386]: (/Stage[main]/Apache/Service[apache2]) Dependency Service[gerrit] has failures: true
[19:14:02] <paladox>	 Aug 02 18:49:25 gerrit-test3 puppet-agent[1386]: (/Stage[main]/Apache/Service[apache2]) Skipping because of failed dependencies
[19:14:02] <paladox>	 Aug 02 18:49:25 gerrit-test3 puppet-agent[1386]: (/Stage[main]/Gerrit::Proxy/Letsencrypt::Cert::Integrated[gerrit]/Exec[acme-setup-acme-gerrit]) Dependency Service[gerrit] has failures: true
[19:14:05] <paladox>	 Aug 02 18:49:25 gerrit-test3 puppet-agent[1386]: (/Stage[main]/Gerrit::Proxy/Letsencrypt::Cert::Integrated[gerrit]/Exec[acme-setup-acme-gerrit]) Skipping because of failed dependencies
[19:14:08] <paladox>	 Aug 02 18:49:25 gerrit-test3 puppet-agent[1386]: Finished catalog run in 11.10 seconds
[19:14:10] <paladox>	 Aug 02 18:49:26 gerrit-test3 sudo[1385]: pam_unix(sudo:session): session closed for user root
[19:14:12] <paladox>	 Aug 02 18:49:42 gerrit-test3 sudo[2153]: diamond : TTY=unknown ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
[19:14:15] <paladox>	 Aug 02 18:49:43 gerrit-test3 sudo[2154]: diamond : TTY=unknown ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
[19:14:18] <paladox>	 Aug 02 18:49:43 gerrit-test3 sudo[2154]: pam_unix(sudo:session): session opened for user puppet by (uid=0)
[19:14:21] <paladox>	 Aug 02 18:49:43 gerrit-test3 sudo[2154]: pam_unix(sudo:session): session closed for user puppe
[19:14:22] <greg-g>	 paladox: pastebin
[19:14:26] <paladox>	 oh sorry
[19:14:58] <paladox>	 https://phabricator.wikimedia.org/T141803#2516448
[19:15:02] <paladox>	 ostriches ^^
[19:16:57] <wikibugs>	 10Deployment-Systems, 03Scap3: More atomic directory operations - https://phabricator.wikimedia.org/T141913#2516467 (10Dereckson)
[19:20:03] <wikibugs>	 10Deployment-Systems, 03Scap3: More atomic directory operations - https://phabricator.wikimedia.org/T141913#2516495 (10Dereckson)
[19:21:25] <mutante>	 paladox: gotta find out why exactly acme-setup fails
[19:21:36] <mutante>	 there was a change yesterday
[19:21:37] <paladox>	 oh
[19:21:53] <mutante>	 like when Krenair was using it to get certs for beta
[19:22:10] <mutante>	 he pasted an error that was because LE changed something August 1
[19:22:13] <paladox>	 mutante but it seems to fail because of gerrit failing
[19:22:54] <mutante>	 oh, that is right
[19:23:13] <mutante>	 well, one by one. 
[19:23:19] <mutante>	 there is this now  https://gerrit.wikimedia.org/r/#/c/302491
[19:23:32] <paladox>	 Oh
[19:23:34] <mutante>	 that got us to the next step which you have now
[19:23:44] <paladox>	 Yep
[19:24:05] <mutante>	 there must be more in some logs
[19:24:16] <paladox>	 Oh, im not sure it dosent seem to show any more
[19:24:20] <paladox>	 maybe apache2
[19:24:35] <mutante>	 try running the init script manually
[19:24:38] <paladox>	 No nothing in apt
[19:24:42] <paladox>	 and how do i do that
[19:24:55] <Krenair>	 which system is it failing on?
[19:25:09] <mutante>	  how about  /etc/init.d/gerrit start 
[19:25:10] <paladox>	 root@gerrit-test3:/var/lib/gerrit2/review_site/logs# /etc/init.d/gerrit start
[19:25:10] <paladox>	 Starting Gerrit Code Review: FAILED
[19:25:10] <paladox>	 root@gerrit-test3:/var/lib/gerrit2/review_site/logs#
[19:25:24] <mutante>	 Krenair: a new instance where we use strictly the prod class
[19:25:29] <Krenair>	 which?
[19:25:43] <mutante>	 gerrit-test3 
[19:25:47] <mutante>	 in the project called git
[19:25:48] <Krenair>	 is this in labs?
[19:25:50] <Krenair>	 git, ok
[19:25:51] <mutante>	 yes
[19:25:57] <paladox>	 Oh
[19:26:20] <paladox>	 mutante https://phabricator.wikimedia.org/T141803#2516511
[19:26:40] <Krenair>	 I'm not in that project
[19:27:51] <mutante>	 oh, i gueess. it's because 
[19:27:53] <mutante>	 RUN_ARGS=' -Xmx28g
[19:28:03] <mutante>	 and we dont have that much memory there
[19:28:30] <mutante>	 well, it's trying to start it, and then it kills it again
[19:28:30] <paladox>	 ah ha
[19:28:35] <paladox>	 mutante there is no file
[19:28:37] <paladox>	  /var/lib/gerrit2/review_site/bin/gerrit.war
[19:28:38] <mutante>	 + start-stop-daemon -S -b -c gerrit2 -p /var/lib/gerrit2/review_site/logs/gerrit.pid -m -d /var/lib/gerrit2/review_site -a /usr/bin/perl -- -e '$x=$ENV{JAVA};exec $x @ARGV;die $!' 
[19:28:40] <paladox>	 called that
[19:28:42] <mutante>	 see the "die" 
[19:28:49] <paladox>	 yep but it calls
[19:28:50] <paladox>	  /var/lib/gerrit2/review_site/bin/gerrit.war
[19:28:55] <paladox>	 which dosent exist
[19:28:56] <mutante>	 paladox: wanna add him to project?
[19:29:03] <paladox>	 Yep
[19:29:24] <mutante>	 oh, well,, if there is no .war ...but .. you have the package installed
[19:29:35] <mutante>	 the new one
[19:29:48] <paladox>	 Yeh but you need the war
[19:29:53] <paladox>	 It runs the on the war
[19:30:21] <mutante>	 ok, for some reason i thought the package drops the .war in place
[19:30:29] <paladox>	 Me too
[19:30:46] <paladox>	 ostriches ^^
[19:31:02] <paladox>	 mutante ok ive added Alex Monk to the project
[19:31:05] <paladox>	 aka Krenair
[19:31:31] <mutante>	 paladox: the "dpkg -L gerrit" we did earlier
[19:31:35] <ostriches>	 It drops the war in place in /var/lib/gerrit2/
[19:31:36] <mutante>	 see the .war in there?
[19:31:41] <paladox>	 Oh
[19:31:46] <paladox>	 Maybe we need to change it
[19:31:47] <ostriches>	 You have to run it with init to get a fully provisioned gerrit install.
[19:31:52] <ostriches>	 Puppet will never get it right
[19:31:57] <paladox>	 Oh
[19:31:58] <ostriches>	 Because gerrit's init is braindead.
[19:32:04] <paladox>	 How do we do it through init
[19:32:06] <ostriches>	 So you gotta run it a few times, whack it with a hammer.
[19:32:14] <ostriches>	 Then puppet takes over
[19:32:15] <paladox>	 LOL
[19:32:19] <ostriches>	 (it's *always* been like this)
[19:32:22] <paladox>	 Oh
[19:32:24] <ostriches>	 (and it's annoying as fuckballs)
[19:32:29] <paladox>	 Im not sure how to start it
[19:32:30] <paladox>	 ?
[19:32:37] <paladox>	  /etc/init.d/gerrit
[19:32:40] <ostriches>	 Just have to keep trying and debugging.
[19:32:45] <paladox>	 Ok
[19:33:01] <paladox>	 Maybe we can add a new check 
[19:33:03] <paladox>	 to init
[19:33:05] <paladox>	 called
[19:33:06] <Krenair>	 Okay I don't see any acme errors
[19:33:06] <paladox>	 install
[19:33:13] <ostriches>	 Yay I made it :D
[19:33:15] <paladox>	 that does it for us
[19:33:18] <paladox>	 Oh
[19:33:49] <paladox>	 ostriches do i just keep doing /etc/init.d/gerrit start
[19:33:56] <mutante>	 alright, which part did you mean to run multiple times
[19:33:57] <paladox>	 Or how about run
[19:34:12] <ostriches>	 Whee I win :D
[19:34:20] <paladox>	 lol
[19:34:48] <Krenair>	 krenair@gerrit-test3:~$ sudo /etc/init.d/gerrit start
[19:34:50] <Krenair>	 Starting Gerrit Code Review: FAILED
[19:34:51] <ostriches>	 2 things gotta happen to make this *better*
[19:34:52] <Krenair>	 useless
[19:34:54] <Krenair>	 where's the error logs?
[19:34:56] <mutante>	 ostriches: on eventbrite ?:)
[19:35:17] <paladox>	 Krenair i think /var/log/ or /var/lib/gerrit2/review_site/logs
[19:35:20] <ostriches>	 Krenair: /var/lib/gerrit2/review_site/logs/
[19:35:23] <paladox>	 but theres nothing in there
[19:35:42] <ostriches>	 More useful though, run `/var/lib/gerrit2/review_site/bin/gerrit.sh run`
[19:35:44] <Krenair>	 Oh brilliant, so it doesn't actually log errors to the error logs?
[19:35:46] <ostriches>	 Which dumps it all to stderr :D
[19:36:00] <ostriches>	 Anyway, I dunno what all the fuss is about :)
[19:36:04] <Krenair>	 krenair@gerrit-test3:~$ sudo /var/lib/gerrit2/review_site/bin/gerrit.sh run
[19:36:04] <Krenair>	 sudo: /var/lib/gerrit2/review_site/bin/gerrit.sh: command not found
[19:36:31] <Krenair>	  /var/lib/gerrit2/review_site/bin is empty
[19:36:33] <ostriches>	 Like I told paladox, running init is brain-dead and puppet has a hard time getting it right
[19:36:44] <paladox>	 Oh
[19:36:54] <paladox>	 yeh
[19:37:00] <ostriches>	 java -jar /var/lib/gerrit2/gerrit.war -d /var/lib/gerrit2/review_site/ init --batch --no-auto-start
[19:37:04] <ostriches>	 Then deal with any problems
[19:37:05] <ostriches>	 I know of 2.
[19:37:15] <paladox>	 Oh maybe we should add that to gerrit
[19:37:18] <paladox>	 and call it
[19:37:23] <paladox>	 gerrit install
[19:37:28] <Krenair>	 krenair@gerrit-test3:~$ sudo java -jar /var/lib/gerrit2/gerrit.war -d /var/lib/gerrit2/review_site/ init --batch --no-auto-start
[19:37:28] <Krenair>	 fatal: unknown command -d
[19:37:28] <Krenair>	       (no com.google.gerrit.pgm.-d)
[19:37:35] <paladox>	 Remove the -d
[19:37:54] <paladox>	 I always find that fails using the -d
[19:38:01] <Krenair>	 where does it go without the -d?
[19:38:16] <paladox>	 It goes /var/lib/gerrit2/review_site/
[19:38:25] <paladox>	 Oh wait
[19:38:32] <paladox>	 Not sure
[19:38:33] <paladox>	 now
[19:38:40] <paladox>	 I was looking at it wrong
[19:39:35] <ostriches>	 Krenair: gerrit.war init -d
[19:39:39] <ostriches>	 Order matters :\
[19:39:44] <paladox>	 Yep lol
[19:39:48] <mutante>	 whatever you do, dont fix it manually on this specific instance.. just on the other ones :)
[19:40:28] <ostriches>	 mutante: https://gerrit.wikimedia.org/r/#/c/302497/ helps
[19:40:32] <paladox>	 ostriches great idea we create a file that run sudo java -jar /var/lib/gerrit2/gerrit.war init -d /var/lib/gerrit2/review_site/ init --batch
[19:40:56] <paladox>	 oh
[19:41:19] <mutante>	 ostriches: 'k, thanks!
[19:41:19] <ostriches>	 We already do that in puppet.
[19:41:43] <paladox>	 Yay lets try it
[19:42:14] <paladox>	 mutante ^^
[19:47:19] <paladox>	 mutante could you merge https://gerrit.wikimedia.org/r/#/c/302497/ please
[19:47:20] <paladox>	 ?
[19:47:23] <paladox>	 so we can test
[19:54:34] <wikibugs>	 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Operations, 10Ops-Access-Requests, and 3 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2458739 (10dpatrick) Because there are a number of Phabricator tickets in the Security...
[19:57:14] <paladox>	 mutante ?
[20:04:32] <paladox>	 brb
[20:04:56] <wikibugs>	 10Deployment-Systems, 03Scap3: More atomic directory operations - https://phabricator.wikimedia.org/T141913#2516619 (10bd808) This will be one of the many benefits of implementing {T114313}. I think this task can honestly be merged into that one.
[20:06:12] <wikibugs>	 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Operations, 10Ops-Access-Requests, and 3 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2458739 (10MaxSem) Cluster access means much greater degree of trust, and by running a...
[20:06:45] <wikibugs>	 10Beta-Cluster-Infrastructure: Reenable $wgMWOAuthSecureTokenTransfer=true; on the beta cluster - https://phabricator.wikimedia.org/T67421#2516623 (10AlexMonk-WMF) @csteipp, @tgr: How do you reset all consumer secrets? Once we find that out we can unstall this and do it.
[20:07:38] <mutante>	 that change makes me wonder what happens if you run the reindex command on the existing prod install and if it runs on every single puppet run
[20:07:44] <Krenair>	 Why is https://phabricator.wikimedia.org/T38891 sitting with noone currently handling it? Task number is in the 30ks and there's a trivial patch
[20:13:02] <nuria_>	 hi ! we cannot deploy , we are getting issues accessing deployment-prep 
[20:15:14] <wikibugs>	 06Release-Engineering-Team: Cannot deploy using scap  from deployment-tin - https://phabricator.wikimedia.org/T141920#2516655 (10Nuria)
[20:16:19] <wikibugs>	 06Release-Engineering-Team: Cannot deploy using scap  from deployment-tin - https://phabricator.wikimedia.org/T141920#2516672 (10Nuria)
[20:16:36] <wikibugs>	 06Release-Engineering-Team: Cannot deploy using scap  from deployment-tin - https://phabricator.wikimedia.org/T141920#2516655 (10Nuria) p:05Triage>03High
[20:17:04] <thcipriani>	 nuria_: hi! what user are you trying to deploy as?
[20:17:14] <nuria_>	 thcipriani: myself
[20:17:18] <thcipriani>	 I recognize the error message and I *think* I see what's wrong.
[20:17:53] <nuria_>	 thcipriani: aham..
[20:20:32] <thcipriani>	 nuria_: I think this is a keyholder permissions thing
[20:20:38] <thcipriani>	 can you try: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-aqs01.deployment-prep.eqiad.wmflabs
[20:20:52] <thcipriani>	 see if that lets you ssh or if it gives you an error
[20:21:02] <nuria_>	 thcipriani: on tin?
[20:21:19] <thcipriani>	 yep, on deployment-tin in beta
[20:21:38] <nuria_>	 thcipriani: no, that worked
[20:22:27] <thcipriani>	 awesome, then deploy should work for you now. You weren't in the deploy-service group, so the agent admitted failure message was just keyholder rejecting you trying to use the key.
[20:22:35] <thcipriani>	 I added you to that group
[20:22:53] <nuria_>	 thcipriani: would you be so kind to add milimetric too?
[20:22:53] <wikibugs>	 10Beta-Cluster-Infrastructure: Cannot deploy using scap  from deployment-tin - https://phabricator.wikimedia.org/T141920#2516702 (10greg)
[20:23:00] <nuria_>	 thcipriani: he is cc-ed on ticket
[20:23:26] <nuria_>	 thcipriani: many thanks for the fast response
[20:23:52] <thcipriani>	 absolutely, I'll add milimetric, lemme know if it works for you now.
[20:24:35] <thcipriani>	 milimetric added as well.
[20:25:12] <milimetric>	 thx :)
[20:28:44] <paladox>	 Im back
[20:41:53] <wmf-insecte>	 Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #104: 04FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/104/
[20:41:55] <wmf-insecte>	 Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #104: 04FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/104/
[20:46:13] <thcipriani>	 milimetric: not that I'm deployment stalking you...but I did see some errors in the aqs/deploy on deployment-tin :) ssh-host-key stuff, *should* be fixed now. Just had to accept the host keys for ssh in /etc/ssh/ssh_known_hosts or your user known_hosts file. I did the former, just now.
[20:47:50] <milimetric>	 thx thcipriani, I appreciate it :)
[20:48:51] <milimetric>	 I see, so if we had a new deployment-aqs2 or something, it would need the same treatment.  Got it
[20:49:23] <milimetric>	 thx for stalking^Wlooking out :)
[20:49:49] <thcipriani>	 :D
[20:50:07] <thcipriani>	 yeah, in prod this is taken care of in some fancy way.
[20:50:15] <thcipriani>	 in beta: less fancy :(
[21:00:11] <bd808>	 thcipriani: puppet resource collection which nobody has figured out how to setup on the self hosted puppetmasters
[21:00:28] <paladox>	 ostriches i now get error https://phabricator.wikimedia.org/T141803#2516775
[21:00:31] <wikibugs>	 10Beta-Cluster-Infrastructure: Cannot deploy using scap  from deployment-tin - https://phabricator.wikimedia.org/T141920#2516777 (10greg) 05Open>03Resolved a:03thcipriani ```lang=irc 20:13 <    nuria_> hi ! we cannot deploy , we are getting issues accessing deployment-prep  20:17 <+thciprian> nuria_: hi! w...
[21:00:37] * bd808 stalks the stalkers
[21:00:51] <thcipriani>	 :)
[21:00:52] <ostriches>	 paladox: You have to install mysql before setting up gerrit.
[21:00:58] <paladox>	 Oh
[21:01:09] <ostriches>	 The role doesn't do that for you, it assumes you have a working mysql with a valid $db_host, $db_user and $db_pass set.
[21:01:20] <paladox>	 Oh
[21:01:21] <paladox>	 ah
[21:02:51] <wikibugs>	 10Beta-Cluster-Infrastructure: Reenable $wgMWOAuthSecureTokenTransfer=true; on the beta cluster - https://phabricator.wikimedia.org/T67421#2516782 (10Tgr) Set `oarc_secret_key` to `MWCryptRand::generateHex( 32 )`, send notification to the user that they should go to `https://meta.wikimedia.org/wiki/Special:OAuth...
[21:03:23] <paladox>	 ostriches do i set password manually in the secure.config
[21:03:24] <paladox>	 file
[21:03:25] <paladox>	 ?
[21:03:46] <ostriches>	 You set it via hiera. There's already one in labs/private.git, just need to tweak it further probably :)
[21:03:57] <paladox>	 Ok
[21:04:37] <paladox>	 ostriches oh, i thought doing that will expose the password
[21:04:48] <ostriches>	 Well it does, but it's labs :p
[21:05:00] <paladox>	 oh lol
[21:05:01] <paladox>	 ok
[21:05:06] <paladox>	 what do i put in hiera
[21:05:13] <paladox>	 ostriches ^^
[21:05:36] <ostriches>	 You can put it on wiki in hiera:<projectname>
[21:05:43] <ostriches>	 Or in the appropriate place in the hiera/ directory
[21:06:04] <paladox>	 ostriches I mean what do i put in there please? configs names?
[21:09:57] <paladox>	 ostriches i doint know how to set the password in hiera?
[21:28:26] <thcipriani>	 !log disabling puppet on scap targets briefly to test scap_3.2.2-1_all.deb
[21:32:01] <thcipriani>	 !log re-enabling puppet on scap targets
[21:32:20] <thcipriani>	 huh, no logbot.
[21:36:19] <paladox>	 ostriches i carnt seem to set the db password
[21:36:28] <paladox>	 i get a random password that i didnt do
[21:36:29] <paladox>	 ?
[21:37:06] <mutante>	 paladox: is it already in labs/private repo?
[21:37:16] <paladox>	 I doint think so
[21:37:20] <paladox>	 it is in secure.config
[21:37:22] <mutante>	 and what is the problem if you already get a password
[21:37:29] <mutante>	 the puppet run finishes?
[21:37:38] <paladox>	 Oh, i thought you would want to choose your own password
[21:37:47] <paladox>	 i guess i will just use that password
[21:38:22] <mutante>	 paladox: if you remove it, and run puppet again, it re-adds it, right?
[21:38:36] <paladox>	 Oh let me try that
[21:39:46] <paladox>	 mutante nope
[21:39:53] <paladox>	 still installs a random password
[21:39:58] <paladox>	 im just going to use that
[21:39:59] <paladox>	 for now
[21:40:13] <mutante>	 i doubt it's actually random
[21:40:38] <mutante>	 the point was if puppet sets it back 
[21:40:43] <mutante>	 to whatever 
[21:40:56] <paladox>	 Yep
[21:41:20] <mutante>	 paladox: a fake password should be in labs/private
[21:41:34] <paladox>	 Oh 
[21:41:39] <paladox>	 it is a fake password then
[21:41:48] <mutante>	 in general if you have secrets that are in private repo in prod
[21:41:55] <mutante>	 then you add a fake version of the same thing
[21:41:56] <mutante>	 in labs/private
[21:42:00] <mutante>	 so that puppet works
[21:42:09] <paladox>	 Oh
[21:42:14] <mutante>	 in prod private it's in modules/passwords
[21:42:18] <paladox>	 it works just need to create the user in the db now
[21:42:23] <paladox>	 which i also wrote on task
[21:42:23] <mutante>	 ok, yea
[21:42:27] <paladox>	 we have to do that
[21:42:38] <mutante>	 yep
[21:43:05] <wikibugs>	 10Deployment-Systems, 03Scap3: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#2516893 (10thcipriani) 05Resolved>03Open Thanks for the upload @fgiunchedi   Bug was discovered on Beta this morning :(  I fixed and bumped the minor version—new version on Beta seems to have fixed th...
[21:44:30] <mutante>	 paladox: i merged the thing with the symlink to /etc/default/gerritcodereview. so that manual step should be gone
[21:44:36] <paladox>	 Oh yay
[21:44:38] <paladox>	 thanks
[21:44:44] <mutante>	 i have to go for a little while. be back soon
[21:45:21] <paladox>	 mutante i got furthur
[21:45:21] <paladox>	 root@gerrit-test3:/var/lib/gerrit2# /usr/bin/java -jar gerrit.war reindex -d review_site --threads 4
[21:45:21] <paladox>	 [2016-08-02 21:45:05,230] [main] INFO  com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /var/lib/gerrit2/review_site/cache
[21:45:21] <paladox>	 [2016-08-02 21:45:05,885] [main] INFO  com.google.gerrit.server.project.ProjectCacheWarmer : Loading project cache
[21:45:22] <paladox>	 fatal: fetch failure on changes
[21:45:24] <paladox>	 root@gerrit-test3:/var/lib/gerrit2#
[21:48:02] <paladox>	 ostriches ^^
[21:58:48] <wmf-insecte>	 Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #106: 04FAILURE in 6 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/106/
[22:21:11] <wmf-insecte>	 Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #103: 04FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/103/
[22:55:07] <wikibugs>	 10Beta-Cluster-Infrastructure: Reenable $wgMWOAuthSecureTokenTransfer=true; on the beta cluster - https://phabricator.wikimedia.org/T67421#2517366 (10AlexMonk-WMF) Let's enable secure token transfer, then if someone decides we really need to do that  in beta, it can be done afterwards?
[23:05:38] <SMalyshev>	 somebody can explain why https://gerrit.wikimedia.org/r/#/c/302424/ is not being merged despite having +2?
[23:05:55] <paladox>	 Try re doing c+1
[23:05:58] <paladox>	 c+2
[23:06:04] <paladox>	 Please
[23:06:20] <SMalyshev>	 ok will do
[23:06:20] <SMalyshev>	 ok will do
[23:06:25] <Krenair>	 SMalyshev, maybe it was +2'd while CI was down or something?
[23:06:50] <paladox>	 I was looking
[23:06:59] <paladox>	 at there was no c+2 till now
[23:07:07] <paladox>	 at 00:06 my time
[23:07:09] <paladox>	 bst
[23:07:14] <Krenair>	 hm, nope
[23:07:43] <SMalyshev>	 I had +2ed it at 2:46... about 1:15 ago. But I've retried now
[23:08:15] <paladox>	 Oh
[23:08:22] <paladox>	 ah
[23:08:22] <SMalyshev>	 doesn't seem to work... maybe it depends on some othe patch? I'm not sure I understand new gerrit interface
[23:08:24] <paladox>	 now i know
[23:08:26] <paladox>	 why
[23:08:34] <paladox>	 you carnt write a comment and do +2
[23:08:40] <paladox>	 at the same time
[23:08:43] <SMalyshev>	 uh
[23:08:44] <paladox>	 it is a known problem
[23:08:48] <SMalyshev>	 that's unexpected
[23:08:51] <greg-g>	 but why isn't it working now
[23:09:12] <paladox>	 Not sure
[23:09:47] <greg-g>	 there it is
[23:10:05] <paladox>	 Yep, i did a recheck
[23:10:18] <SMalyshev>	 looks like there's some dependency but I'm not sure where I look for it in new UI
[23:10:25] <paladox>	 Oh yeh
[23:10:33] <paladox>	 if it has dependacies that havent merged
[23:10:35] <paladox>	 it wont merge
[23:10:44] <paladox>	 until they are merged
[23:10:47] <SMalyshev>	 so where I find the dependencies?
[23:11:01] <paladox>	 By looking at parent
[23:11:15] <SMalyshev>	 parent has just commit ID
[23:11:23] <SMalyshev>	 no link to a gerrit patch
[23:11:44] <paladox>	 Yeh
[23:11:49] <paladox>	 but it is linked to those
[23:11:50] <paladox>	 https://gerrit.wikimedia.org/r/#/c/300954/
[23:11:55] <paladox>	 Thats one of them
[23:12:12] <paladox>	 and https://gerrit.wikimedia.org/r/#/c/300953/
[23:12:14] <SMalyshev>	 paladox: wait so "related changes" are actually parents?
[23:12:27] <paladox>	 Nope
[23:12:29] <paladox>	 parent is
[23:12:31] <SMalyshev>	 ah no they are the same for all... Now I'm confused
[23:12:38] <SMalyshev>	 so how I know which one is the parent?
[23:12:42] <paladox>	 yeh i think gerrit removed that support
[23:12:45] <paladox>	 making it harder
[23:12:51] <SMalyshev>	 ught that's bad
[23:12:53] <paladox>	 But parent is the commits now
[23:14:02] <paladox>	 https://gerrit.wikimedia.org/r/#/c/302424/ requires https://gerrit.wikimedia.org/r/#/c/300954/
[23:14:18] <paladox>	 which then that requires https://gerrit.wikimedia.org/r/#/c/300953/
[23:14:30] <paladox>	 which then requires
[23:14:31] <paladox>	 https://gerrit.wikimedia.org/r/#/c/298112/
[23:14:54] <paladox>	 SMalyshev ^^
[23:14:58] <SMalyshev>	 well, old gerrit told me that... I guess now I have to investigate
[23:15:01] <SMalyshev>	 paladox: thanks
[23:15:06] <paladox>	 Your welcome
[23:15:08] <SMalyshev>	 I'll submit a task about it
[23:15:14] <paladox>	 Ok
[23:15:20] <paladox>	 You can leave your +2
[23:15:30] <paladox>	 which will then merge when ever all those patches
[23:15:34] <paladox>	 are merged
[23:17:01] <SMalyshev>	 ok, thnaks, I filed T141947
[23:18:39] <paladox>	 ok
[23:18:40] <paladox>	 thanks