[00:01:12] <wikibugs_>	 06Release-Engineering-Team, 15User-greg: Setup monthly "QA Tribe" meeting - https://phabricator.wikimedia.org/T152570#2852919 (10greg) meeting help request sent to eng-admin :)
[00:35:53] <grrrit-wm>	 (03PS1) 10Legoktm: Support DESTDIR in Makefile [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325721 
[00:35:55] <grrrit-wm>	 (03PS1) 10Legoktm: Add 'make clean' [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325722 
[00:59:48] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 032 V: 032] Add 'make clean' [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325722 (owner: 10Legoktm) 
[00:59:52] <wikibugs>	 10Continuous-Integration-Config, 10uprightdiff: Add tests for integration/uprightdiff repository - https://phabricator.wikimedia.org/T152578#2853064 (10Legoktm)
[01:00:13] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 032 V: 032] Support DESTDIR in Makefile [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/325721 (owner: 10Legoktm) 
[01:02:30] <grrrit-wm>	 (03PS1) 10Legoktm: Initial debianization [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/325726 (https://phabricator.wikimedia.org/T152577) 
[01:37:59] <grrrit-wm>	 (03CR) 10Legoktm: Initial debianization (031 comment) [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/325726 (https://phabricator.wikimedia.org/T152577) (owner: 10Legoktm) 
[01:41:51] <wikibugs_>	 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development, 13Patch-For-Review: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2853177 (10scfc) Thanks, @hashar.  On closer look, these are indeed for the most part `flake8` errors.  (I find the...
[02:27:04] <matt_flaschen>	 !log Started (foreachwikiindblist flow.dblist extensions/Flow/maintenance/FlowFixInconsistentBoards.php) 2>&1 | tee FlowFixInconsistentBoards_2016-12-06.txt on deployment-tin
[02:27:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:29:16] <matt_flaschen>	 !log foreachwikiindblist FlowFixInconsistentBoards complete
[02:29:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:31:42] <matt_flaschen>	 !Log Started (mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=enwiki) 2>&1|tee FlowFixInconsistentBoards_enwiki_2016-12-06.txt
[02:43:53] <matt_flaschen>	 !Log Killed (mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=enwiki) 2>&1|tee FlowFixInconsistentBoards_enwiki_2016-12-06.txt
[03:52:04] <matt_flaschen>	 bd808, legoktm, do we really need to manually update security groups and wait up to 60 minutes when setting up a proxy (https://wikitech.wikimedia.org/wiki/Help:Proxy)?
[03:52:10] <matt_flaschen>	 I don't remember either of those being the case before.
[03:52:15] <matt_flaschen>	 Also, when I created a proxy, it shows as:
[03:52:24] <matt_flaschen>	 	deferred-changes.wmflabs.org 	http://10.68.22.160:8080 
[03:52:30] <matt_flaschen>	 The existing ones show actual hostnames on the right.
[03:52:38] <legoktm>	 I don't know about security groups stuff, but usually the proxy takes a minute or something
[03:56:33] <matt_flaschen>	 Thanks, legoktm.  Indeed, both are wrong.
[04:18:48] <wmf-insecte>	 Yippee, build fixed!
[04:18:49] <wmf-insecte>	 Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #226: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/226/
[04:24:38] <bd808>	 matt_flaschen: the ip instead of hostname is a performance improvement
[04:24:55] <bd808>	 security groups do need to allow access
[04:25:20] <bd808>	 the time thing... I think that was a bad addition, but it can take a few minutes
[04:27:53] <bd808>	 the "web" security group is only going to exist if someone already created that in the project. Its not default
[06:41:04] <wikibugs_>	 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 05Language-Engineering October-December 2016, and 4 others: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2853431 (10Arrbee)
[07:27:16] <wikibugs_>	 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 07Jenkins, 13Patch-For-Review, and 2 others: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#2853483 (10phuedx)
[08:44:56] <shinken-wm>	 PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[09:19:54] <shinken-wm>	 RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0]
[10:08:22] <wikibugs>	 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development, 13Patch-For-Review: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2853664 (10ArielGlenn) I'm taking a look.
[10:15:03] <wikibugs>	 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development, 13Patch-For-Review: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2853671 (10ArielGlenn) Looks like that fixed it.
[10:19:20] <wikibugs_>	 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development, 13Patch-For-Review: tox-jessie is failing on operations/software - https://phabricator.wikimedia.org/T152549#2853674 (10hashar) https://gerrit.wikimedia.org/r/#/c/325715/ fix flake8 newly introduced lint E305.  The other erro...
[10:51:48] <wikibugs>	 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: CI tests on wikimedia/portals repo: cache node_modules to save time - https://phabricator.wikimedia.org/T152386#2846564 (10Jdrewniak) As @hashar mentioned in his comment:   > Regarding the long build time (11 minutes), that is due to the job...
[12:14:12] <wikibugs_>	 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: CI tests on wikimedia/portals repo: cache node_modules to save time - https://phabricator.wikimedia.org/T152386#2853784 (10Jdrewniak) As @hashar mentioned in his comment:  > Regarding the long build time (11 minutes), that is due to the job c...
[12:33:41] <icinga-wm>	 PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:49:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins: Some Jenkins jobs tend to be stuck and never times out - https://phabricator.wikimedia.org/T138281#2853808 (10hashar) >>! In T140256#2827310, @MoritzMuehlenhoff wrote: >>>! In T140256#2825802, @hashar wrote: >> I haven't seen that kernel soft lock occurring f...
[13:01:41] <icinga-wm>	 RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[13:30:25] <wikibugs>	 10Continuous-Integration-Config, 10uprightdiff: Add tests for integration/uprightdiff repository - https://phabricator.wikimedia.org/T152578#2853896 (10hashar) There are a few dependencies mentioned in the README file: build-essential g++ libopencv-highgui-dev libboost-program-options-dev. We would need them i...
[13:30:44] <wikibugs>	 10Continuous-Integration-Config, 10uprightdiff: Add tests for integration/uprightdiff repository - https://phabricator.wikimedia.org/T152578#2853064 (10hashar) p:05Triage>03Normal
[13:41:02] <wikibugs_>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2853924 (10Paladox) Note: That the update to gerrit 2.13 will make some changes to the gitweb links.  1. parent links no longer have links, it was intentional but I have brought this to upstream attention and are...
[13:57:51] <hashar>	 !sal
[13:57:51] <wm-bot>	 https://tools.wmflabs.org/sal/releng
[14:02:54] <zeljkof>	 hashar: will you be around for the next puppet swat? tomorrow 18:00–19:00 UTC+1
[14:03:10] <hashar>	 nop
[14:03:20] <zeljkof>	 I will not be here, and I would like to add https://gerrit.wikimedia.org/r/#/c/324203/ to the puppet swat
[14:03:24] <hashar>	 have my 1/1 with greg then I am rushing back home
[14:03:25] <zeljkof>	 ok, next week then :)
[14:03:33] <hashar>	 my wife goes to the gym on wednesday at 19:15 
[14:03:51] <hashar>	 zeljkof: try poking operations folks directly
[14:03:58] <hashar>	 it is trivial enough and has no incident to prod
[14:04:08] <hashar>	 + we got that one cherry picked on the CI puppet master havent we?
[14:04:16] <zeljkof>	 hashar: not really sure who to ping :|
[14:04:20] <hashar>	 if so, it is all about grabbing 3 minutes of attention from someone
[14:04:30] <hashar>	 ask around in the private channel :d
[14:04:44] <zeljkof>	 #releng private?
[14:09:13] <grrrit-wm>	 (03PS1) 10Mholloway: Whitelist Kaartic [integration/config] - 10https://gerrit.wikimedia.org/r/325775 
[14:09:14] <wikibugs_>	 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ensure ChromeDriver is installed for jobs that run Selenium tests - https://phabricator.wikimedia.org/T117418#2853979 (10zeljkofilipin) 05Open>03Resolved
[14:12:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories - https://phabricator.wikimedia.org/T152604#2853984 (10hashar)
[14:12:51] <wikibugs_>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2853996 (10hashar)
[14:16:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2854016 (10hashar)
[14:46:49] <wikibugs_>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2854082 (10hashar) Went with a very lame python script: ``` lang=python from datetime import timedelta   with open('clone.txt', 'r') as...
[14:53:25] <hashar>	 zeljkof: for chrome driver PATH
[14:53:25] <hashar>	 https://gerrit.wikimedia.org/r/324203
[14:53:31] <hashar>	 got merged at 14:08 UTC
[14:53:40] <hashar>	 and nodepool refresh the snapshots at 14:14UTC
[14:53:48] <hashar>	 so potentially it is included now
[14:53:54] <zeljkof>	 well, that is convenient :)
[14:54:04] <zeljkof>	 let me test on a patch
[14:54:19] <zeljkof>	 https://gerrit.wikimedia.org/r/#/c/320829/
[14:54:36] <zeljkof>	 will change browser to chrome, let's see if it sees chromedriver
[14:54:41] <hashar>	 jessie instance: $ ssh jenkins@10.68.18.104 which chromedriver
[14:55:11] <zeljkof>	 doesn't work?
[14:55:23] <hashar>	 ah
[14:55:33] <hashar>	  /usr/local/bin/chromedriver
[14:55:40] <hashar>	 can't paste a line starting with a / :D
[14:56:05] <zeljkof>	 :D
[14:56:14] <zeljkof>	 add space
[14:56:31] <hashar>	 the trusty instance does not have it
[14:56:52] <zeljkof>	 really? didn't we tests there too?
[14:57:42] <hashar>	 on labnodepool1001 $ grep /usr/local/bin/chromedriver /var/log/nodepool/image.log*
[14:57:48] <hashar>	 .ci-jessie-wikimedia: Notice: /Stage[main]/Contint::Browsers/File[/usr/local/bin/chromedriver]/ensure: created
[14:57:58] <hashar>	 maybe the trusty one got done before the patch landed
[14:58:19] <hashar>	 yeah it is slightly older
[14:58:39] <hashar>	 zeljkof: I am refreshing the trusty snapshot
[14:58:44] <hashar>	 nodepool image-update wmflabs-eqiad ci-trusty-wikimedia
[14:58:45] <hashar>	 ;D
[14:58:52] <zeljkof>	 nice
[14:59:35] <wikibugs>	 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ensure ChromeDriver is installed for jobs that run Selenium tests - https://phabricator.wikimedia.org/T117418#2854114 (10hashar) The Nodepool Jessie image had the patch included.  I am refreshing the Trusty one.
[15:00:15] <zeljkof>	 running the job... https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm-node-4-jessie/900/console
[15:03:56] <zeljkof>	 hashar: all green :) https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm-node-4-jessie/900/console
[15:04:28] <hashar>	 \O/
[15:04:42] <hashar>	 !log Image ci-trusty-wikimedia-1481122712 in wmflabs-eqiad is ready  T117418
[15:04:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:07:00] <wikibugs_>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2854132 (10hashar) The equivalent job that runs on Jessie (mediawiki-extensions-hhvm-jessie) does load the cache for Abusefilter :( So l...
[15:12:09] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Zuul: mediawiki-extensions jobs take 4 minutes to clone repositories (with zuul-cloner) - https://phabricator.wikimedia.org/T152604#2854136 (10hashar) On Jessie sorted by time: ``` 0:00:02s INFO:zuul.Cloner:Prepared mediawiki/extensions/Babel 0:00:02s INFO:zuul.Cloner...
[15:16:38] <grrrit-wm>	 (03PS1) 10Hashar: dib: add more repositories to git cache [integration/config] - 10https://gerrit.wikimedia.org/r/325786 (https://phabricator.wikimedia.org/T152604) 
[15:27:17] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Will have to rebuild the base images." [integration/config] - 10https://gerrit.wikimedia.org/r/325786 (https://phabricator.wikimedia.org/T152604) (owner: 10Hashar) 
[15:28:23] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: add more repositories to git cache [integration/config] - 10https://gerrit.wikimedia.org/r/325786 (https://phabricator.wikimedia.org/T152604) (owner: 10Hashar) 
[15:35:47] <wikibugs_>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2854164 (10hashar) Looking at a Jessie instance: ``` # journalctl |grep 'Startup fini' Dec 07 15:01:5...
[15:39:31] <wikibugs>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2854165 (10Paladox) I managed to fix project access links in https://gerrit-review.googlesource.com/#/c/92620/ (waiting for it to be merged and then I will get it backported) :)
[15:39:49] <shinken-wm>	 PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:43:28] <grrrit-wm>	 (03PS2) 10Mholloway: Whitelist Kaartic [integration/config] - 10https://gerrit.wikimedia.org/r/325775 
[15:43:44] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] Whitelist Kaartic [integration/config] - 10https://gerrit.wikimedia.org/r/325775 (owner: 10Mholloway) 
[15:50:59] <wikibugs>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2854196 (10Paladox) That patch was abandoned in favour of https://gerrit-review.googlesource.com/#/c/92645/ since it is cherry-picked to stable-2.13 then will be merged into master.
[15:51:04] <shinken-wm>	 PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[16:03:10] <paladox>	 Yipee i fixed the project access links in gerrit 2.13. https://gerrit-review.googlesource.com/#/c/92620/ cherry picked to stable-2.13 https://gerrit-review.googlesource.com/#/c/92645/
[16:16:35] <wikibugs_>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2854299 (10Paladox) It was merged and will be released when gerrit 2.13.4 is released.
[16:30:51] <wikibugs>	 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: Add support to peridoic CI tests for exercising arbitrary revisions - https://phabricator.wikimedia.org/T152455#2854327 (10Niedzielski)
[16:31:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2854343 (10hashar) In the instance (T113342#2850982) dhclient starts straight with a DHCPDISCOVER. Th...
[16:33:54] <wikibugs>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2854358 (10Paladox) Ah, it looks like a new exclusiveGroupPermissions = read is added in project.config in refs/meta/config so it will hide the access link, it can be removed which I did and it worked.
[17:45:37] <wikibugs>	 03Scap3: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message' - https://phabricator.wikimedia.org/T152390#2854693 (10thcipriani) 05Open>03Resolved
[17:55:20] <shinken-wm>	 PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[18:35:18] <shinken-wm>	 RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:44:51] <wmf-insecte>	 Project beta-code-update-eqiad build #133424: 04FAILURE in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133424/
[18:48:50] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[18:54:47] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[18:54:51] <wmf-insecte>	 Project beta-code-update-eqiad build #133425: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133425/
[18:55:25] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[19:03:16] <shinken-wm>	 PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[19:03:22] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[19:04:48] <wmf-insecte>	 Project beta-code-update-eqiad build #133426: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133426/
[19:06:00] <shinken-wm>	 PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[19:08:40] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[19:09:51] <wikibugs>	 10Deployment-Systems, 10MediaWiki-Internationalization, 06Performance-Team, 07Performance: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#1297948 (10Gilles) p:05Low>03High
[19:11:23] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 06Performance-Team, 07HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#2854985 (10Gilles) p:05Normal>03Triage
[19:12:43] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[19:13:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Linter: Deploy Linter extension to Beta cluster - https://phabricator.wikimedia.org/T152620#2854992 (10Legoktm)
[19:14:43] <wmf-insecte>	 Project beta-code-update-eqiad build #133427: 04STILL FAILING in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133427/
[19:14:48] <legoktm>	 ^ because gerrit is down
[19:15:19] <ostriches>	 Back "soon"
[19:15:44] <TabbyCat>	 Some ads in the meanwhile
[19:15:57] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[19:16:49] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[19:17:03] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[19:17:04] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[19:24:42] <wmf-insecte>	 Project beta-code-update-eqiad build #133428: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133428/
[19:34:42] <wmf-insecte>	 Project beta-code-update-eqiad build #133429: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133429/
[19:44:43] <wmf-insecte>	 Project beta-code-update-eqiad build #133430: 04STILL FAILING in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133430/
[19:54:42] <wmf-insecte>	 Project beta-code-update-eqiad build #133431: 04STILL FAILING in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133431/
[20:04:42] <wmf-insecte>	 Project beta-code-update-eqiad build #133432: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133432/
[20:14:41] <wmf-insecte>	 Project beta-code-update-eqiad build #133433: 04STILL FAILING in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133433/
[20:24:44] <wmf-insecte>	 Project beta-code-update-eqiad build #133434: 04STILL FAILING in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133434/
[20:29:10] <RoanKattouw>	 When is Gerrit going to be back up? :/
[20:31:51] <Krenair>	 RoanKattouw, I'd watch -operations for that sort of information
[20:34:44] <wmf-insecte>	 Project beta-code-update-eqiad build #133435: 04STILL FAILING in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133435/
[20:44:42] <wmf-insecte>	 Project beta-code-update-eqiad build #133436: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133436/
[20:46:51] <wikibugs_>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2855307 (10hashar) I have looked at the jessie snapshot.  `/etc/network/interfaces.d/eth1` magically...
[20:51:14] <paladox>	 hashar the html links need updating for gerrit 2.13.
[20:51:20] <paladox>	 jenkins links please
[20:51:26] <paladox>	 see https://gerrit.wikimedia.org/r/#/c/315057/ please.
[20:52:18] <wikibugs_>	 10Gerrit: Gerrit side-by-side diff view does not mark a removed "space" in a string (example given) (intraline different doesn't work with spaces) - https://phabricator.wikimedia.org/T51006#2855320 (10demon)
[20:52:23] <wikibugs_>	 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2855317 (10demon) 05Open>03Resolved a:03demon
[20:53:34] <wikibugs>	 10Gerrit: Gerrit side-by-side diff view does not mark a removed "space" in a string (example given) (intraline different doesn't work with spaces) - https://phabricator.wikimedia.org/T51006#2855338 (10Paladox) 05Open>03Resolved This now works.  Please reopen if the problem still happends.
[20:54:44] <wmf-insecte>	 Yippee, build fixed!
[20:54:44] <wmf-insecte>	 Project beta-code-update-eqiad build #133437: 09FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/133437/
[20:59:45] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:05:24] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:08:16] <shinken-wm>	 RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:08:20] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:15:59] <shinken-wm>	 RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:17:44] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:18:40] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:20:56] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:21:50] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:22:05] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:22:05] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:23:51] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:34:19] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[22:19:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2855719 (10hashar) I create a new image and booted an instance out of it.  Deleted the lease and sync...
[22:22:05] <hashar>	 paladox: I reviewed your patch related to Gerrit pretification of the CI results ( https://gerrit.wikimedia.org/r/#/c/325826/ )
[22:22:07] <hashar>	 looks good to me
[22:22:19] <hashar>	 maybe ostriches can get it pushed
[22:22:27] <paladox>	 hashar thanks 
[22:22:31] <paladox>	 i also tested it too
[22:22:33] <paladox>	 and works
[22:22:34] <paladox>	 :)
[22:22:41] <hashar>	 only impact would be that puppet will restart Gerrit once the change land (iirc)
[22:22:45] <hashar>	 yeah!
[22:22:52] <hashar>	 test on labs is great :]
[22:23:39] <hashar>	 just found out Gerrit got upgraded :D
[22:23:50] <hashar>	 I have seen an email about it but did not connect that was for tonight !
[22:23:51] <paladox>	 Yep
[22:24:05] <paladox>	 I didnt either
[22:24:09] <hashar>	 looks like nothing exploded so far
[22:24:09] <paladox>	 didn't check my emails
[22:24:16] <hashar>	 beside the comment formatting
[22:24:20] <paladox>	 only found out the upgrade was happening when i checked -operations
[22:24:29] <paladox>	 hashar some gitweb links wont work
[22:25:11] <paladox>	 expecially our workaround for T137354
[22:25:13] <paladox>	 hashar ^^
[22:26:10] <paladox>	 but luckly i fixed it today.
[22:26:10] <hashar>	 guess we will survive :D
[22:26:17] <paladox>	 but will be released in a 2.13.4 update
[22:26:30] <paladox>	 also gerrit no longer links parent links 
[22:27:16] <hashar>	 I am away again, need to fill a bunch of papers
[22:27:22] <paladox>	 ok
[22:27:34] <hashar>	 taxes / retirements fund etc
[22:27:39] <hashar>	 it is a mess!
[22:27:48] <paladox>	 Oh
[22:34:10] <RoanKattouw>	 How long does one normally have to wait for a new repository request to be acted on?
[22:34:46] <paladox>	 RoanKattouw qchris normaly does them so not sure how often he checks.
[22:34:48] <RoanKattouw>	 https://www.mediawiki.org/wiki/Git/New_repositories/Requests shows that requests were being fulfilled daily for a while, but that seems to have stopped last week
[22:36:17] <bd808>	 thcipriani: I've got a scap3 ssh config problem you might be able to help with
[22:36:31] <thcipriani>	 sure, shoot
[22:36:41] <bd808>	 I have a scap server setup in the striker labs project
[22:36:51] <bd808>	 it used to work
[22:36:56] <thcipriani>	 :)
[22:37:13] <bd808>	 now the ssh key is getting rejected with "error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1"
[22:37:14] <thcipriani>	 only really needs apache -> /srv/deployment
[22:37:45] <thcipriani>	 when you try to deploy from it to a target?
[22:37:54] <bd808>	 yeah
[22:37:55] <bd808>	 Failed publickey for deploy-service from 10.68.23.63 port 40406 ssh2: RSA cb:43:c3:76:3a:68:b0:29:1a:8f:3f:31:1f:87:f8:7c
[22:38:18] <bd808>	 what I don't remember is if there was magic in setting that up  (nothing in my notes)
[22:38:26] <bd808>	 and why it might be broken now
[22:39:13] <thcipriani>	 hrm. I've never run into that before
[22:39:19] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:39:20] <thcipriani>	 it certainly does return 1 in beta
[22:39:45] <bd808>	 hmmm but then something lets it work anyway?
[22:40:37] <thcipriani>	 I'm checking that now for deploy-service :)
[22:40:50] <bd808>	 I think the last time this project was used was about 8 weeks ago so there could have been all kinds of puppet changes that I didn't notice
[22:42:14] <thcipriani>	 yeah, it does seem to allow login
[22:42:41] <thcipriani>	 SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-parsoid09.deployment-prep.eqiad.wmflabs
[22:42:44] <thcipriani>	 works
[22:42:59] <bd808>	 *nod* that's pretty much how I'm testing too
[22:43:12] <thcipriani>	 even though: /usr/sbin/ssh-key-ldap-lookup deploy-service
[22:43:21] <thcipriani>	 returns 1 on deployment-parsoid09
[22:43:25] <bd808>	 so I've either got the wrong key or some missing hiera stuff
[22:44:05] <thcipriani>	 or there's some other crazy problem that someone fixed beta and I didn't notice :)
[22:44:28] * thcipriani looks at puppetmaster patches
[22:46:04] <thcipriani>	 hrm, nothing looks particularly suspect as long as keyholder is armed and/or deploy-service has access to the key some other way
[22:46:33] <bd808>	 the fingerprint of the key matches one in the deployment-tin keyholder
[22:46:37] <bd808>	 so that seems good
[22:47:46] <bd808>	 I'll try to audit the hiera stuff
[22:48:07] <thcipriani>	 hrm, is it making to checking the key in /var/log/auth.log on the target?
[22:48:13] <bd808>	 yes
[22:48:20] <thcipriani>	 weird
[22:48:21] <bd808>	 and its being rejected there
[22:49:38] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Whitelist Kaartic [integration/config] - 10https://gerrit.wikimedia.org/r/325775 (owner: 10Mholloway) 
[22:50:42] <thcipriani>	 hrm, afaik it should just be checking against /etc/ssh/userkeys/[key]
[22:50:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: Whitelist Kaartic [integration/config] - 10https://gerrit.wikimedia.org/r/325775 (owner: 10Mholloway) 
[22:50:50] <bd808>	 I've got a hunch this is puppet realted
[22:50:55] <thcipriani>	 there was one thing we changed wrt to the pam module
[22:51:09] <thcipriani>	 trying to remember what it was, but it doesn't sound like your problem is pam related
[22:51:11] <bd808>	 like my project isn't using the puppet master it thinks it is possibly
[22:52:26] <thcipriani>	 oh, yeah, puppet::master::self vs puppetmaster::standalone is somewhat of a recent thing, IIRC
[22:52:32] <bd808>	 yeah
[22:53:59] <bd808>	 thanks for rubber ducking
[22:56:13] <thcipriani>	 werd
[22:56:48] <grrrit-wm>	 (03PS3) 10Hashar: dib: fix dupe sourcing in interfaces [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) 
[22:56:50] <grrrit-wm>	 (03PS2) 10Hashar: dib: ensure /etc/apt/sources.list exists [integration/config] - 10https://gerrit.wikimedia.org/r/325576 
[22:56:52] <grrrit-wm>	 (03PS1) 10Hashar: nodepool: delete DHCP leases from snapshot [integration/config] - 10https://gerrit.wikimedia.org/r/325852 (https://phabricator.wikimedia.org/T113342) 
[22:58:51] <hashar>	 bd808: thcipriani: iirc there is some weird race condition between puppet and keyholder
[22:59:06] <hashar>	 happened multiple times when we have rebuild them a few weeks ago
[22:59:31] <hashar>	 I think the fix was restart the keyholder (or maybe disarm and arm)
[22:59:55] <hashar>	 might be something else entirely though
[23:00:30] <hashar>	 ohh
[23:00:40] <hashar>	 Gerrit 2.13 detects when an updated change is just a rebase
[23:00:50] <mutante>	 oh?
[23:00:51] <hashar>	 and shows it as a comment!
[23:01:00] <hashar>	 eg https://gerrit.wikimedia.org/r/#/c/325576/
[23:01:06] <hashar>	 I did push a new chain
[23:01:17] <mutante>	 imagines jenkins-bot then doesnt have to vote a second time
[23:01:18] <hashar>	 and it commented for me:  Uploaded patch set 2: Patch Set 1 was rebased.
[23:01:38] <hashar>	 iirc we strip the Verified vote
[23:01:46] <mutante>	 that sounds fairly normal though
[23:01:49] <hashar>	 on a rebase we keep codereview +1 votes though
[23:01:51] <mutante>	 "Patch Set was rebased"
[23:01:55] <hashar>	 yeah
[23:02:10] <paladox>	 it will also tell you if your on an old patch set too
[23:02:13] <hashar>	 but I am pretty sure Gerrit used to not mention it was a rebase
[23:02:13] <paladox>	 shows orange
[23:02:26] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[23:02:28] <hashar>	 so what I did was to get in Gerrit, Press rebase button on each patch
[23:02:35] <hashar>	 then send my new change :(
[23:03:02] <paladox>	 I think gerrit also shows when you remove reviewers as comments
[23:03:11] <paladox>	 not sure if that is only for NoteDB
[23:03:18] <grrrit-wm>	 (03PS4) 10Hashar: dib: fix dupe sourcing in interfaces [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) 
[23:03:33] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "That is being upstreamed :]" [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) (owner: 10Hashar) 
[23:04:00] <grrrit-wm>	 (03PS3) 10Hashar: dib: ensure /etc/apt/sources.list exists [integration/config] - 10https://gerrit.wikimedia.org/r/325576 
[23:04:14] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: ensure /etc/apt/sources.list exists [integration/config] - 10https://gerrit.wikimedia.org/r/325576 (owner: 10Hashar) 
[23:04:32] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: fix dupe sourcing in interfaces [integration/config] - 10https://gerrit.wikimedia.org/r/325561 (https://phabricator.wikimedia.org/T113342) (owner: 10Hashar) 
[23:05:57] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: ensure /etc/apt/sources.list exists [integration/config] - 10https://gerrit.wikimedia.org/r/325576 (owner: 10Hashar) 
[23:06:13] <grrrit-wm>	 (03PS1) 10Hashar: nodepool: delete DHCP leases from snapshot [integration/config] - 10https://gerrit.wikimedia.org/r/325854 (https://phabricator.wikimedia.org/T113342) 
[23:06:22] <hashar>	 doh
[23:06:53] <hashar>	 ostriches: interestingly Gerrit failed to register a change I have sent ( was https://gerrit.wikimedia.org/r/325852 )
[23:07:08] <hashar>	 grrrit-wm did sent a notification about it at 22:56:52
[23:07:17] <hashar>	 but the change did not exist in the UI
[23:07:24] <hashar>	 resent as https://gerrit.wikimedia.org/r/325854
[23:07:38] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] nodepool: delete DHCP leases from snapshot [integration/config] - 10https://gerrit.wikimedia.org/r/325854 (https://phabricator.wikimedia.org/T113342) (owner: 10Hashar) 
[23:08:24] <grrrit-wm>	 (03Merged) 10jenkins-bot: nodepool: delete DHCP leases from snapshot [integration/config] - 10https://gerrit.wikimedia.org/r/325854 (https://phabricator.wikimedia.org/T113342) (owner: 10Hashar) 
[23:15:35] <wikibugs>	 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Eloquence)
[23:17:24] <paladox>	 ostriches https://phabricator.wikimedia.org/T152640
[23:17:58] <bd808>	 thcipriani: mystery partially solved. the key I'm sending and the key the host is expecting don't match. Now to track down the right key
[23:19:33] <wikibugs_>	 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855865 (10demon) p:05Triage>03High a:03demon
[23:26:49] <wikibugs_>	 10Gerrit: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855875 (10demon) Weird, your account_id is actually 20, and Eloquence does indeed claim erik as its shell name. Bizarre....
[23:28:17] <wikibugs>	 10Gerrit: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2855876 (10hashar)
[23:31:33] <wikibugs_>	 10Gerrit: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2855876 (10Paladox) Should this be rated as high priority since this could happen again at any time?
[23:33:53] <bd808>	 thcipriani: mystery solved. _joe_ changed the heira variable name from "scap::server::keyholder_agents" to "scap::keyholder_agents"
[23:33:57] <wikibugs>	 10Gerrit: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2855929 (10hashar) Nothing shows up in Gerrit error log as I can tell though Gerrit has been restarted just after.
[23:34:22] <hashar>	 now I am really off !
[23:34:29] <thcipriani>	 bd808: ah ha!
[23:35:03] <thcipriani>	 yeah _joe_ did a lot of work making scap fancy :)
[23:35:14] <thcipriani>	 didn't realize there was a variable rename
[23:35:17] * thcipriani squints
[23:36:08] <bd808>	 it was a long time ago apaprently
[23:37:51] <wikibugs_>	 10Gerrit: Gerrit 2.13 failed to register a change - https://phabricator.wikimedia.org/T152642#2855934 (10demon) Definitely exists in the DB: ```  change_key                                | created_on            | last_updated_on       | owner_account_id | dest_project_name  | dest_branch_name  | status | curren...