[00:12:09] <wikibugs>	 3Scap3, 6Phabricator: Deploy phabricator with scap3 - https://phabricator.wikimedia.org/T114363#1711226 (10mmodell)
[00:22:42] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[00:27:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<30.00%)
[00:36:36] <shinken-wm>	 PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[00:40:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 6Language-Engineering, 10MediaWiki-Internationalization, 7I18n: en-rtl l10n cache not being rebuilt, eventually resulting in fatal exceptions when magic words are not dealt with properly - https://phabricator.wikimedia.org/T106125#1711342 (10Krenair)
[00:57:42] <Krenair>	 bd808, thanks for committing my local hack on the puppetmaster
[00:58:01] <Krenair>	 I had it on my todo list to deal with that
[01:11:33] <shinken-wm>	 RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:29:26] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Language-Engineering, 7I18n: l10nupdate key not set up properly - https://phabricator.wikimedia.org/T114971#1711465 (10Krenair) You can run just this command to see the issue, without having to wait for l10nupdate: `sudo-withagent l10nupdate sync-dir --no-shared-authsock -D ss...
[01:36:03] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Language-Engineering, 7I18n: l10nupdate key not set up properly - https://phabricator.wikimedia.org/T114971#1711479 (10Krenair) ```krenair@deployment-mediawiki03:~$ cat /etc/ssh/userkeys/l10nupdate  ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzcA/wB0uoU+XgiYN/scGczrAGuN99O8L7m8TviqxgX...
[02:13:28] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Language-Engineering, 7I18n: l10nupdate key not set up properly - https://phabricator.wikimedia.org/T114971#1711504 (10Krenair) Setting `mediawiki::users::l10nupdate_public_key` to the real contents of `files/ssh/tin/l10nupdate/id_rsa.pub` in labs/private would be a bad idea (...
[02:16:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Language-Engineering, 7I18n: l10nupdate key not set up properly - https://phabricator.wikimedia.org/T114971#1711508 (10Krenair) I experimented with that idea for a bit but found that target hosts still reject it due to it not matching labs LDAP.
[03:04:48] <wikibugs>	 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, and 2 others: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1711537 (10faidon) I notice the lack of IPv6 everywhere on this effort (no service IP assigned, no ACLs for it etc.). We've been building everything use...
[04:16:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Parsoid, 10RESTBase, 6Services, 7Parsoid-Tests: Move Parsoid and RESTBase testing from Travis CI to our Jenkins - https://phabricator.wikimedia.org/T78410#1711565 (10jayvdb) >>! In T78410#1703790, @GWicke wrote: > .. > We are also getting coveralls code coverage...
[04:24:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 10pywikibot-core: run at least pep8 and pep257 for new changesets submitted to pywikibot/core for any user - https://phabricator.wikimedia.org/T87169#1711578 (10jayvdb) Do we need to remove `py26` from the envlist in `tox.ini`?   If we get linting working again, it soun...
[04:53:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10pywikibot-core: run at least pep8 and pep257 for new changesets submitted to pywikibot/core for any user - https://phabricator.wikimedia.org/T87169#1711598 (10jayvdb)
[04:58:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1711600 (10jayvdb) >>! In T114421#1699621, @JanZerebecki wrote: >>>! In T114421#1695935, @jayvdb wrote: >> Yes please! Pywikibot would love t...
[05:14:33] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1711603 (10jayvdb) >>! In T114421#1706281, @JanZerebecki wrote: > Also a cost of travis is that it costs money, the free offer is usually max...
[05:17:17] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-parsoid04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[05:17:17] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-mobile04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[06:05:11] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1711632 (1001tonythomas)
[06:12:02] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1711636 (1001tonythomas) Interested candidates, please take your time in finding a **co-mentor** for this project along with your work with your proposal! to **make it fea...
[06:37:29] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK
[09:02:34] <wikibugs>	 10Browser-Tests, 10Wikidata: adapt wikidata_api gem to be compatible with mediawiki_api gem version 0.4 - https://phabricator.wikimedia.org/T106811#1711841 (10WMDE-Fisch)
[09:02:36] <wikibugs>	 10Browser-Tests, 6Release-Engineering-Team, 6Security: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1711840 (10WMDE-Fisch)
[09:30:36] <shinken-wm>	 PROBLEM - Puppet failure on deployment-fluorine is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[09:33:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 10pywikibot-core: run at least pep8 and pep257 for new changesets submitted to pywikibot/core for any user - https://phabricator.wikimedia.org/T87169#1711877 (10hashar) The tox envlist has: ``` flake8 flake8-py3 flake8-docstrings-mandatory py26 py27 py34 ```  Running al...
[09:47:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 10pywikibot-core: run at least pep8 and pep257 for new changesets submitted to pywikibot/core for any user - https://phabricator.wikimedia.org/T87169#1711901 (10jayvdb) 11 minutes is IMO acceptable.  That is roughly how long the jenkins job took before the recent upgrad...
[10:04:11] <wikibugs>	 10Deployment-Systems, 3Scap3, 5Patch-For-Review: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server - https://phabricator.wikimedia.org/T104826#1711968 (10mmodell)
[10:08:52] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster to match production - https://phabricator.wikimedia.org/T64835#1711977 (10hashar)
[10:08:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#1711976 (10hashar)
[10:09:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#746426 (10hashar) I believe we would need {T64835} as a prerequisite to overhaul the Beta cluster thumbnail system.
[10:10:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#1711982 (10hashar)
[10:10:39] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#1711981 (10hashar)
[10:11:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#1711985 (10hashar)
[10:11:41] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster to match production - https://phabricator.wikimedia.org/T64835#1711986 (10hashar)
[10:11:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#746426 (10hashar)
[10:15:46] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#1711998 (10hashar) Sorry for the mess in the blocked-by/blocking. The dependency tree is now:  * This bug: {T71757} ** {T84950} *** {T64835}
[10:16:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#934455 (10hashar) Sorry for the mess in the blocked-by/blocking. The dependency tree is no...
[10:16:56] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster to match production - https://phabricator.wikimedia.org/T64835#1712006 (10hashar) Sorry for the mess in the blocked-by/blocking. The dependency tree is now:  * {T71757} ** {T84950} *** This bug: {T64835}
[10:18:38] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#1712016 (10hashar)
[10:25:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 7Epic: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge - https://phabricator.wikimedia.org/T101545#1342113 (10hashar)
[10:26:01] <wikibugs>	 10Continuous-Integration-Infrastructure, 7Epic: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge - https://phabricator.wikimedia.org/T101545#1342113 (10hashar) I have created a root tracking task for object storage on labs: T114998. #beta-cluster needs Swift.
[10:26:25] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#670674 (10hashar)
[10:27:19] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#1712050 (10hashar) I have created a root tracking task for object storage on labs: T114998. #continuous-integration-infrastructure definitely use a Swift store.
[10:27:49] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#1712054 (10hashar)
[10:27:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 7Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#1712055 (10hashar)
[10:27:54] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#1712052 (10hashar) 5Open>3stalled
[10:28:02] <wikibugs>	 10Deployment-Systems, 3Scap3, 10Analytics, 6Services, 6operations: Use Scap3 for deploying AQS - https://phabricator.wikimedia.org/T114999#1712059 (10mobrovac) 3NEW
[10:37:47] <wikibugs>	 10Browser-Tests, 10Wikidata: adapt wikidata_api gem to be compatible with mediawiki_api gem version 0.4 - https://phabricator.wikimedia.org/T106811#1712080 (10JanZerebecki) p:5Low>3Normal
[11:24:05] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1712148 (10NiharikaKohli) I wonder if @polybuildr would be interested in mentoring?
[11:49:19] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites  - https://phabricator.wikimedia.org/T115002#1712186 (10Amire80) 3NEW
[12:25:21] <CFisch_WMDE>	 zeljkof: 1st: We just had out Wikidata browsertest strategy meeting and I invited a few of my colleagues to our weekly session.
[12:25:40] <CFisch_WMDE>	 2nd can we move the meeting tomorrow till after 1:30?
[12:45:32] <wikibugs>	 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-10-13: adapt wikidata_api gem to be compatible with mediawiki_api gem version 0.4 - https://phabricator.wikimedia.org/T106811#1712269 (10Tobi_WMDE_SW)
[12:46:06] <krrrit-wm1>	 (03Abandoned) 10Hashar: DO NOT SUBMIT. Wipe /jjb/ for demo purposes [integration/config] - 10https://gerrit.wikimedia.org/r/243163 (owner: 10Hashar)
[12:48:14] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1712272 (10Aklapper) Has this been brought up on https://lists.wikimedia.org/mailman/listinfo/releng ?
[12:48:28] <krrrit-wm1>	 (03PS2) 10Hashar: Remove mediawiki-core-release job [integration/config] - 10https://gerrit.wikimedia.org/r/243924 (https://phabricator.wikimedia.org/T86661) 
[12:48:38] <krrrit-wm1>	 (03CR) 10Hashar: [C: 032] Remove mediawiki-core-release job [integration/config] - 10https://gerrit.wikimedia.org/r/243924 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[12:49:29] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1712276 (10Amire80) >>! In T115002#1712272, @Aklapper wrote: > Has this been brought up on https://lists.wikimedia.org/mailman/listinfo/rele...
[12:50:25] <krrrit-wm1>	 (03Merged) 10jenkins-bot: Remove mediawiki-core-release job [integration/config] - 10https://gerrit.wikimedia.org/r/243924 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[12:59:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1712296 (10hashar) 5Resolved>3Open We no have the redirects expansion being done. Still need to run the Apache2 configt...
[12:59:16] <krrrit-wm1>	 (03PS2) 10Hashar: Remove operations-apache-config-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243923 (https://phabricator.wikimedia.org/T72068) 
[12:59:35] <krrrit-wm1>	 (03CR) 10Hashar: [C: 032] "Thanks Jan. I have reopened T72068" [integration/config] - 10https://gerrit.wikimedia.org/r/243923 (https://phabricator.wikimedia.org/T72068) (owner: 10Hashar)
[13:01:43] <krrrit-wm1>	 (03Merged) 10jenkins-bot: Remove operations-apache-config-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243923 (https://phabricator.wikimedia.org/T72068) (owner: 10Hashar)
[13:06:20] <krrrit-wm1>	 (03PS1) 10Hashar: Drop /srv/ssd/gerrit ref for mwext jobs [integration/config] - 10https://gerrit.wikimedia.org/r/244440 (https://phabricator.wikimedia.org/T86661) 
[13:10:39] <krrrit-wm1>	 (03PS2) 10Hashar: jjb: remove defaults 'use-zuul-for-mw-ext' [integration/config] - 10https://gerrit.wikimedia.org/r/244440 (https://phabricator.wikimedia.org/T86661) 
[13:10:57] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Technical-Debt: Disable Gerrit replication to production slaves - https://phabricator.wikimedia.org/T86661#1712320 (10hashar)
[13:11:43] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1712321 (10Aklapper) (I asked because I do not think a Phabricator task is a good place to discuss such a complex topic.)
[13:13:07] <krrrit-wm1>	 (03CR) 10Hashar: [C: 032] "jjb diff confirms it is a noop" [integration/config] - 10https://gerrit.wikimedia.org/r/244440 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[13:14:52] <krrrit-wm1>	 (03Merged) 10jenkins-bot: jjb: remove defaults 'use-zuul-for-mw-ext' [integration/config] - 10https://gerrit.wikimedia.org/r/244440 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[13:18:23] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1712327 (10greg) It's fine to discuss here, or on wikitech-l, or both.  (NB: releng@ is our private team only list, qa@ is our public list,...
[13:32:41] <krrrit-wm1>	 (03PS1) 10Hashar: Drop Gerrit replicate from git-remote-zuul-no-submodules [integration/config] - 10https://gerrit.wikimedia.org/r/244444 (https://phabricator.wikimedia.org/T86661) 
[13:33:23] <krrrit-wm1>	 (03CR) 10Hashar: [C: 032] "Bonus point, I got Gerrit change number 244444" [integration/config] - 10https://gerrit.wikimedia.org/r/244444 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[13:35:14] <krrrit-wm1>	 (03Merged) 10jenkins-bot: Drop Gerrit replicate from git-remote-zuul-no-submodules [integration/config] - 10https://gerrit.wikimedia.org/r/244444 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar)
[13:36:16] <krrrit-wm1>	 (03PS1) 10Hashar: jjb: expand git macros [integration/config] - 10https://gerrit.wikimedia.org/r/244445 
[13:39:36] <krrrit-wm1>	 (03PS2) 10Hashar: jjb: expand git macros used only in defaults [integration/config] - 10https://gerrit.wikimedia.org/r/244445 
[13:53:58] <zeljkof>	 CFisch_WMDE: great! and sure, almost any time tomorrow is good for me, 1:30 and later should be fine
[13:58:22] <wikibugs>	 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1712430 (10JanZerebecki) 5Open>3Resolved I opened T114801 for that.
[15:34:50] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1712616 (10EBernhardson) I would be willing to mentor this project
[15:39:10] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1712622 (1001tonythomas) >>! In T89682#1712616, @EBernhardson wrote: > I would be willing to mentor this project  Awesome ! Thank you @EBernhardson for taking up this one!...
[15:39:55] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1712623 (1001tonythomas)
[15:54:11] <wikibugs>	 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1712667 (10Haritha28) I am interested in working for this project for the Outreacyh 11.  As I understood from the task description as given to improve the codesniffer exte...
[17:22:45] <anomie>	 grumble. phpcs doesn't work with mediawiki/vendor, while phpunit apparently doesn't work with composer-installed vendor and PHP 5.6.
[18:16:46] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1713166 (10KuboF)
[18:44:41] <krrrit-wm>	 (03PS1) 10Aude: Enable GeoData extension for Wikibase repo jobs [integration/config] - 10https://gerrit.wikimedia.org/r/244489 
[18:48:39] <chasemp>	 twentyafterfour: do you need puppet disabled on iridium?
[18:49:24] <twentyafterfour>	 oh did I leave it disabled?  No I guess not, but I do need to submit that patch to neuter the puppet deployer stuff
[18:50:16] <chasemp>	 sure I'm going to enable for now then :)
[18:50:39] <chasemp>	 btw if you set the tags to HEAD it would effetively stop messing with anything you are doing for the moment till scap
[18:50:42] <twentyafterfour>	 ok sorry about that
[18:50:44] <chasemp>	 tag => 'HEAD'
[18:50:51] <chasemp>	 I think is a hack to say, jsut f off already
[18:50:56] <chasemp>	 no worries
[18:51:06] <twentyafterfour>	 chasemp: oh? I thought it would only work with tags
[18:52:16] <chasemp>	 i'm somewhat confident
[18:52:22] <chasemp>	 ...and a shit load of things just landed w/ puppet
[18:52:43] <twentyafterfour>	 ?
[18:54:01] <chasemp>	 inherited exim changes, local.json somehow lost it's good formatting and so was swapped out, /srv/phab/phabricator/.git/config seems to have gone unmanaged?
[18:54:07] <chasemp>	 we were doing something there and I can't remember what
[18:54:23] <chasemp>	 and on top of that the changes I wanted 
[18:57:44] <chasemp>	 twentyafterfour: I made a few changes to your https://gerrit.wikimedia.org/r/#/c/237096/
[18:57:49] <chasemp>	 hope that's cool will put it up here in a few
[19:00:39] <twentyafterfour>	 chasemp: ok cool
[19:00:58] <twentyafterfour>	 I don't know why .git/config was being managed, but I don't think it was anything important
[19:01:59] <chasemp>	 I think it was to ignore hacks in the repo for redirector and something?
[19:02:02] <chasemp>	 I'll look
[19:02:05] <chasemp>	 I'm not worried atm
[19:02:53] <twentyafterfour>	 ignore rules would need to be in .git/info/exclude ..not .git/config
[19:10:10] <chasemp>	 ah
[19:10:11] <chasemp>	         filemode = false
[19:10:18] <chasemp>	 we manage the perms on bin executables
[19:10:22] <chasemp>	 to manage who can do what
[19:10:31] <chasemp>	 and you have to tell git not to show it in diff
[19:10:33] <chasemp>	 iirc
[19:19:12] <krrrit-wm>	 (03PS1) 1020after4: git-latest-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/244495 
[19:20:39] <krrrit-wm>	 (03CR) 10Hashar: [C: 032] "Noop in both JJB and Zuul." [integration/config] - 10https://gerrit.wikimedia.org/r/244445 (owner: 10Hashar)
[19:22:21] <chasemp>	 twentyafterfour: I'm going to roll on https://gerrit.wikimedia.org/r/#/c/237096/6 and we can see what sticks
[19:22:30] <krrrit-wm>	 (03Merged) 10jenkins-bot: jjb: expand git macros used only in defaults [integration/config] - 10https://gerrit.wikimedia.org/r/244445 (owner: 10Hashar)
[19:22:34] <twentyafterfour>	 ok
[19:22:42] <chasemp>	 it's not radically different from your initial, but lots of shakeout from getting teh front end stuff going
[19:22:46] <chasemp>	 (multiple ip's for ssh etc)
[19:34:51] <chasemp>	 twentyafterfour: ostriches greg-g -- I need to decide how to manage the secondary ssh process exactly but smoke testing
[19:34:54] <chasemp>	 telnet git-ssh.wikimedia.org 22
[19:34:54] <chasemp>	 Trying 208.80.154.250...
[19:34:54] <chasemp>	 Connected to git-ssh.wikimedia.org.
[19:34:56] <chasemp>	 Escape character is '^]'.
[19:34:58] <chasemp>	 SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
[19:35:00] <chasemp>	 yup
[19:35:25] <twentyafterfour>	 yay!
[19:36:20] <twentyafterfour>	 chasemp: by manage, you mean service control / monitoring / etc?
[19:36:23] <chasemp>	 right
[19:37:04] <chasemp>	 this is just /usr/sbin/sshd -f /etc/ssh/sshd_config.phabricator
[19:37:05] <chasemp>	 :)
[19:39:58] <twentyafterfour>	 well could it just be a copy of the default sshd init scripts stuff? Are we on systemd? upstart? I'm not even sure how this stuff is configured
[19:42:18] <chasemp>	 sshd is just init atm
[19:42:45] <chasemp>	 hey...did websockets start working
[19:42:46] <chasemp>	 or something
[19:43:15] <chasemp>	 ok maybe not had a weird thing cross wires w/ secure.phab
[19:43:42] <chasemp>	 still more to do
[19:43:44] <chasemp>	 echo {} | ssh vcs@git-ssh.wikimedia.org conduit conduit.ping
[19:43:44] <chasemp>	 Received disconnect from UNKNOWN: 2: Too many authentication failures for vcs
[19:43:51] <chasemp>	 maybe you could poke at that, I'll be out next week
[19:44:26] <twentyafterfour>	 too many authentication failures? is that a fail2ban type thing?
[19:44:54] <chasemp>	 no just ssh things?
[19:45:08] <chasemp>	 I am reading, is this supported over a secondary domain
[19:45:16] <chasemp>	 this would be aweful if we came this far and it's not
[19:45:20] <chasemp>	 I thought for sure we tested it
[19:45:39] <twentyafterfour>	 I don't think phabricator cares what domain it is...
[19:46:27] <chasemp>	 I wouldn't think
[19:49:25] <chasemp>	 diffusion.ssh-host
[19:51:04] <ostriches>	 Introduced in D11800 back in Feb, should be ok...
[19:52:19] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Technical-Debt: Disable Gerrit replication to production slaves - https://phabricator.wikimedia.org/T86661#1713469 (10hashar) /srv/ssd/gerrit is gone from JJB files.  There is still one reference in multigit.sh bu...
[19:52:31] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Gerrit, 5Patch-For-Review, 7Technical-Debt: Disable Gerrit replication to production slaves - https://phabricator.wikimedia.org/T86661#1713471 (10hashar)
[20:00:07] <hashar>	 ostriches: one less Gerrit replication destination :-D  https://gerrit.wikimedia.org/r/#/c/244498/
[20:00:28] <hashar>	 ostriches: I finally sprinted the removal of all references to the replica.  And nothing from CI rely on it anymore!
[20:01:06] <ostriches>	 Heh, now if only I could stop replication to the other 2 targets.... :p
[20:01:37] <hashar>	 well GitHub it is probably worth putting it back on the table
[20:01:57] <hashar>	 but i guess that would need to wait for gitblit replacement
[20:02:46] <twentyafterfour>	 when I connect to git-ssh I just get permission denied, even after setting up a certificate. chasemp:  I can't seem to reproduce the "UNKNOWN: 2: Too many authentication failures for vcs" result
[20:03:26] <chasemp>	 twentyafterfour: yeah idk yet
[20:03:32] <chasemp>	 something aint right atm
[20:03:52] <hashar>	 ostriches: would you mind reaching out to ops to get that puppet patch deployed and the replication plugin related?
[20:04:01] <hashar>	 ostriches: i will handle the cleanup on gallium tomorrow
[20:04:18] <twentyafterfour>	 chasemp: is there a log from the sshd?
[20:05:04] <ostriches>	 hashar: the plugin reload will be me. It's just the merge we need
[20:05:54] <chasemp>	 AuthorizedKeysCommand /usr/local/lib/phabricator-ssh-hook.sh returned status 127
[20:06:13] <chasemp>	 twentyafterfour: ftm just /var/log/auth
[20:06:38] <hashar>	 ostriches: i guess i will ask eu ops tomorrow morning :}
[20:07:27] <hashar>	 ostriches: on other news I also deleted the Jenkins job mediawiki-core-release  https://gerrit.wikimedia.org/r/#/c/243924/    It used to build tarball / patch etc on new tags.  But got disabled in March. 
[20:07:49] <twentyafterfour>	 AuthorizedKeysCommand /usr/local/lib/phabricator-ssh-hook.sh returned status 127
[20:07:56] <twentyafterfour>	 hmm
[20:08:33] <twentyafterfour>	 ROOT="/phabricator"
[20:08:51] <twentyafterfour>	 chasemp: shouldn't it be /srv/phab/phabricator  ?
[20:09:09] <twentyafterfour>	 (the ROOT= var in phabricator-ssh-hook.sh)
[20:14:24] <twentyafterfour>	 chasemp: that's almost certainly the problem
[20:21:04] <twentyafterfour>	 chasemp: https://gerrit.wikimedia.org/r/244506
[20:22:03] <wikibugs>	 10Continuous-Integration-Config: Reduce nosetests output in integration/config - https://phabricator.wikimedia.org/T107683#1713564 (10hashar) I have it a few test failures recently, and indeed the errors being in the middle of a long spam is not convenient.   Lets drop the verbosity.
[20:24:09] <krrrit-wm>	 (03PS1) 10Hashar: tests: descrease nose verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/244507 (https://phabricator.wikimedia.org/T107683) 
[20:24:44] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review: Reduce nosetests output in integration/config - https://phabricator.wikimedia.org/T107683#1713582 (10hashar) a:3hashar
[20:25:57] <wikibugs>	 10Continuous-Integration-Infrastructure: Package / puppetize zuul-clear-refs.py - https://phabricator.wikimedia.org/T103529#1713588 (10hashar) a:5hashar>3None
[20:34:06] <krrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "There is a different problem going on related to the Timer plugin." [integration/config] - 10https://gerrit.wikimedia.org/r/244507 (https://phabricator.wikimedia.org/T107683) (owner: 10Hashar)
[20:36:04] <wikibugs>	 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations, and 2 others: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1713598 (10mmodell) https://gerrit.wikimedia.org/r/#/c/244506/
[20:41:18] <krrrit-wm>	 (03PS2) 10Hashar: tests: restore failures report at end of run [integration/config] - 10https://gerrit.wikimedia.org/r/244507 (https://phabricator.wikimedia.org/T107683) 
[20:42:26] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review: Reduce nosetests output in integration/config - https://phabricator.wikimedia.org/T107683#1713632 (10hashar) The issues is that tests failures are being written asap instead of buffered for the end of the run.  That is caused by `nose-timer` so I am removing...
[20:42:40] <krrrit-wm>	 (03CR) 10Hashar: "Dropping nose-timer but keeping verbosity=2" [integration/config] - 10https://gerrit.wikimedia.org/r/244507 (https://phabricator.wikimedia.org/T107683) (owner: 10Hashar)
[20:55:30] <wikibugs>	 10Browser-Tests, 10Continuous-Integration-Config: Define an CI entry point for Ruby code - https://phabricator.wikimedia.org/T104024#1713688 (10zeljkofilipin)
[20:55:37] <wikibugs>	 10Browser-Tests, 10Continuous-Integration-Config: Define a CI entry point for Ruby code - https://phabricator.wikimedia.org/T104024#1406060 (10zeljkofilipin)
[20:57:53] <ostriches>	 hashar: I don't see why we couldn't go ahead and merge today :)
[21:04:52] <hashar>	 ostriches: feel free to poke ops about it :-}
[21:04:58] <hashar>	 a bit too late for me to deal with it tonight
[21:31:32] <krrrit-wm>	 (03PS1) 10Hashar: [forrestbot] experimental tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/244590 
[21:31:47] <krrrit-wm>	 (03CR) 10Hashar: [C: 032] [forrestbot] experimental tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/244590 (owner: 10Hashar)
[21:32:42] <krrrit-wm>	 (03Merged) 10jenkins-bot: [forrestbot] experimental tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/244590 (owner: 10Hashar)
[21:35:12] <krrrit-wm>	 (03PS2) 10Hashar: Migrate forrestbot to a single tox job [integration/config] - 10https://gerrit.wikimedia.org/r/242349 
[21:36:24] <krrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "Rebased, I had tox-jessie added to experimental pipeline." [integration/config] - 10https://gerrit.wikimedia.org/r/242349 (owner: 10Hashar)
[21:39:00] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-GPGMail: Add Jenkins configuration for mediawiki/GPGMail - https://phabricator.wikimedia.org/T115041#1713333 (10hashar)
[21:44:19] <krrrit-wm>	 (03PS1) 10Hashar: [GPGMail] Generic tests [integration/config] - 10https://gerrit.wikimedia.org/r/244595 (https://phabricator.wikimedia.org/T115041) 
[21:45:27] <krrrit-wm>	 (03CR) 10Hashar: [C: 032] [GPGMail] Generic tests [integration/config] - 10https://gerrit.wikimedia.org/r/244595 (https://phabricator.wikimedia.org/T115041) (owner: 10Hashar)
[21:46:24] <krrrit-wm>	 (03Merged) 10jenkins-bot: [GPGMail] Generic tests [integration/config] - 10https://gerrit.wikimedia.org/r/244595 (https://phabricator.wikimedia.org/T115041) (owner: 10Hashar)
[21:54:32] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-GPGMail, 5Patch-For-Review: Add Jenkins configuration for mediawiki/GPGMail - https://phabricator.wikimedia.org/T115041#1713955 (10hashar) There is a bunch of tests falling but nothing to worry to much about. Fixed with https://gerrit.wikimedia.org/r/#/...
[21:55:10] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-GPGMail, 5Patch-For-Review: Add Jenkins configuration for mediawiki/GPGMail - https://phabricator.wikimedia.org/T115041#1713958 (10Tgr) 5Open>3Resolved a:3Tgr Thanks @Hashar!
[21:56:50] <wikibugs>	 10Continuous-Integration-Config: replace Jenkins job mwext-testextension-zend by mwext-testextension-zend-composer - https://phabricator.wikimedia.org/T115061#1713971 (10hashar)
[22:01:36] <wikibugs>	 10Continuous-Integration-Config: replace Jenkins job mwext-testextension-zend by mwext-testextension-zend-composer - https://phabricator.wikimedia.org/T115061#1713988 (10hashar) Kunal, Jan, that has been raised on wikitech-l.  The extension WebPlatformAuth has been refactored by @Renoirb with https://gerrit.wiki...
[22:02:58] <wikibugs>	 10Continuous-Integration-Config: replace Jenkins job mwext-testextension-zend by mwext-testextension-zend-composer - https://phabricator.wikimedia.org/T115061#1713996 (10hashar) p:5Triage>3Normal
[22:03:51] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-GPGMail, 5Patch-For-Review: Add Jenkins configuration for mediawiki/GPGMail - https://phabricator.wikimedia.org/T115041#1714003 (10hashar) If you help maintains extensions, feel free to adjust them to composer/npm based testing :-}
[23:44:12] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Parsoid, 10RESTBase, 6Services, 7Parsoid-Tests: Move Parsoid and RESTBase testing from Travis CI to our Jenkins - https://phabricator.wikimedia.org/T78410#1714147 (10GWicke) @jayvdb: http://codecov.io looks interesting, thanks for the pointer! I'd really like to...