[00:03:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Wikimedia-Apache-configuration: Apache slash expansion should not redirect from HTTPS to HTTP - https://phabricator.wikimedia.org/T95164#2801161 (10Dzahn)
[00:04:03] <grrrit-wm>	 (03CR) 10Krinkle: [C: 031] delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T150727) (owner: 10Dzahn)
[00:12:22] <twentyafterfour>	 k69
[00:12:31] <twentyafterfour>	 er
[00:13:01] <greg-g>	 that's a horrible password
[00:13:05] <Reedy>	 xd
[00:16:56] <wmf-insecte>	 Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #209: 04FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/209/
[00:17:04] <wmf-insecte>	 Project selenium-Flow » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #209: 04FAILURE in 1 min 4 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/209/
[00:21:23] <wmf-insecte>	 Yippee, build fixed!
[00:21:24] <wmf-insecte>	 Project beta-update-databases-eqiad build #12853: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/12853/
[00:22:10] <matt_flaschen>	 Or 3 apparently: https://phabricator.wikimedia.org/P4456  . ebernhardson is reviewing it.  Only one of them is a global I missed... The other is an actual fix for Beta Cluster (https://gerrit.wikimedia.org/r/#/c/322022/1/addWiki.php).
[00:22:20] <wikibugs>	 06Release-Engineering-Team, 10Tool-Labs-tools-Other: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2801214 (10Zppix)
[00:48:12] <bd808>	 anybody know how to check and see if zuul is sick? It looks like there are a lot more things queued at https://integration.wikimedia.org/zuul/ than I see over on the jenkins side
[00:49:30] <bd808>	 Maybe the jessie pool is stuck?
[00:52:44] <Zppix>	 bd808 just kill jessie and restart
[00:52:50] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06WMDE-TLA-Team, 13Patch-For-Review, 15User-Ladsgroup, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Make beta German Wiktionary - https://phabricator.wikimedia.org/T150764#2801258 (10Mattflaschen-WMF) 05Open>03Resolved Done.  There's a thing with wmgExtraLanguageNames, but d...
[00:53:04] <grrrit-wm>	 (03PS1) 10Chad: make-release: Remove --gitrootext option, doesn't work [tools/release] - 10https://gerrit.wikimedia.org/r/322028 
[00:56:45] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06WMDE-TLA-Team, 13Patch-For-Review, 15User-Ladsgroup, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Make beta German Wiktionary - https://phabricator.wikimedia.org/T150764#2801270 (10Mattflaschen-WMF) 05Resolved>03Open EnableFlow isn't working, hopefully just due to https://...
[01:01:26] <twentyafterfour>	 ^ that was my new wiki password. oops
[01:01:33] * twentyafterfour lols
[01:02:33] <grrrit-wm>	 (03PS1) 10Chad: make-release: don't bother creating patch dir, who cares if it's gone [tools/release] - 10https://gerrit.wikimedia.org/r/322030 
[01:02:33] <paladox>	 bd808 im thinking it is nodepool
[01:02:37] <bd808>	 paladox: it seems to be moving, just slowly
[01:02:46] <paladox>	 Yep
[01:03:20] <paladox>	 nodepool has reached it's capacity, so once labs gets more, nodepool should run better during heavy usage
[01:03:42] <paladox>	 twentyafterfour nodepool is your password?
[01:03:46] <paladox>	 LOL
[01:03:53] <twentyafterfour>	 no but it should be
[01:03:58] <twentyafterfour>	 nobody would ever guess it
[01:04:03] <paladox>	 LOL
[01:04:42] <bd808>	 paladox: I'm not sure that labs is ever going to "get more" to fix nodepool. I think that nodepool was a neat idea that has proven itself to not work for us.
[01:04:52] <paladox>	 Oh
[01:04:57] <twentyafterfour>	 n0dePoOl <- perfect
[01:05:07] <bd808>	 the upstream usage was very very different than what we are doing
[01:05:23] <paladox>	 oh and lol
[01:06:00] <bd808>	 pretty 1337 twentyafterfour ;)
[01:06:07] <twentyafterfour>	 oO
[01:06:24] <ostriches>	 bd808: You're reading my mind ;-)
[01:06:31] <ostriches>	 re: nodepool not working for this use case for us
[01:06:36] <paladox>	 Maybe we should create an upstream task asking for some improvements in terms of get nodepool to start quicker
[01:06:49] <twentyafterfour>	 or take it out back and shoot it
[01:06:53] <bd808>	 that
[01:07:00] <twentyafterfour>	 put us out of it's misery
[01:07:00] <bd808>	 twice jut to be sure its done
[01:07:01] <paladox>	 lol
[01:07:14] <paladox>	 what about using docker?
[01:07:22] <ostriches>	 bd808: Three times, I want in!
[01:07:23] <twentyafterfour>	 that's a possibility
[01:07:29] <paladox>	 its secure
[01:07:32] <ostriches>	 Everyone gets a chance to shoot nodepool out of rage.
[01:07:37] <twentyafterfour>	 and it's web scale
[01:07:50] <bd808>	 docker vms on a kubernetes grid that is running somewhere other than Labs would be much nicer
[01:07:52] <paladox>	 yep and it can be installed mutiple times on the same system
[01:07:58] <twentyafterfour>	 (like mongodb)
[01:07:59] <bd808>	 Faidon even said maybe in prod
[01:08:57] <paladox>	 oh
[01:09:08] <bd808>	 docker and secure don't completely go together paladox. There are a ton os ways to break out of the lxc jail unless it it very closely tuned for the job being run
[01:09:24] <paladox>	 oh
[01:09:35] <bd808>	 but there are similar things that might work
[01:09:53] <bd808>	 the holy grail of running random jobs for untrusted users may be reaching too far though
[01:10:16] <bd808>	 that's what got the whole nodepool thing started
[01:10:24] <paladox>	 oh
[01:10:26] <bd808>	 but full VMs are too heavy
[01:10:38] <paladox>	 yep
[01:10:42] <bd808>	 (At least in my opinion)
[01:11:07] <Krenair>	 wasn't the idea of using VMs to keep it contained inside the labs network?
[01:11:39] <paladox>	 they should really redesgn nodepool to be able to use the same server without having to create seperate instance on the fly.
[01:11:47] <paladox>	 Would that make it insecure?
[01:12:12] <ostriches>	 Also, jobs like linting/stylechecks/etc that don't actually *execute* any user-generated code should just go to dedicated slaves and not bother with VMs or containers or any of that overhead.
[01:12:21] <ostriches>	 Static analysis doesn't need it :)
[01:12:22] <ostriches>	 Waste of time
[01:12:30] <bd808>	 the big dream was to make the jobs isolated and on disposable hosts so that a job couldn't escape and taint other jobs or do something nasty on the network
[01:12:55] <ostriches>	 bd808: Well that and some people write sloppy jobs that can pollute subsequent runs.
[01:13:03] <ostriches>	 Oh yeah, taint other jobs.
[01:13:05] <ostriches>	 You said it
[01:14:11] <paladox>	 android tests seem to be the main taking to much resources out of nodepool
[01:14:47] <paladox>	 bd808 also we found if you depend on other patches, that can slow down things (zuul).
[01:15:19] <paladox>	 we found that with android and noticed if it depends on other patches, it can take a while for it to test.
[01:21:09] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 06Performance-Team, 07HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#1401646 (10Krinkle) >>! In T103886#2125417, @ori wrote: > https://github.com/facebook/hhvm/issue...
[01:28:23] <Krinkle>	 greg-g: added "[Jump to current event] " link to the Deployments page. Hope others fine it useful as well :)
[01:32:09] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2801337 (10Krenair) task description line 4
[01:34:47] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06WMDE-TLA-Team, 13Patch-For-Review, 15User-Ladsgroup, 05WMF-deploy-2016-11-29_(1.29.0-wmf.4): Make beta German Wiktionary - https://phabricator.wikimedia.org/T150764#2801341 (10Mattflaschen-WMF) Actually it was https://gerrit.wikimedia.org/r/#/c/322033/ .   It still doesn'...
[01:37:01] <matt_flaschen>	 Can someone review a Labs Puppet config change in operations/puppet? https://gerrit.wikimedia.org/r/#/c/321817/4 
[01:37:34] <matt_flaschen>	 greg-g, that's a followup for dewiktionary (still).  Not sure who has +2 in operations/puppet and works on Labs.
[01:39:01] <matt_flaschen>	 Is that https://gerrit.wikimedia.org/r/#/admin/groups/855,members ?
[01:40:37] <Krenair>	 puppet stuff needs ops approval, even if it's a labs-only change
[01:41:42] <Krenair>	 that group contains the ops who manage the labs infrastructure
[01:42:31] <Krenair>	 I doubt they have much to do with restbase, but you could ask
[01:44:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2801389 (10Krenair) >>! In T150649#2800637, @fgiunchedi wrote: > The issue afaics is that swift on `deployment-ms-fe01` doesn't have the password for `mw:thumbor` in `/e...
[02:52:57] <wikibugs>	 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2801487 (10Krinkle)
[03:02:36] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%)
[03:03:00] <bd808>	 blerg. what blew up there?
[03:25:56] <twentyafterfour>	 :-/
[03:29:55] <twentyafterfour>	 something's using a LOT of disk space
[03:32:44] <Krenair>	 -rw-r--r-- 1 udp2log udp2log  12G Nov 17 03:32 wfDebug.log
[03:32:49] <Krenair>	 -rw-r--r-- 1 udp2log udp2log   18G Nov 16 05:30 wfDebug.log-20161116
[03:34:30] <Krenair>	 why is that archive not zipped?
[03:52:34] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[03:57:15] <Krenair>	 I ran /usr/sbin/logrotate /etc/logrotate.conf
[03:57:26] <Krenair>	 hopefully cron should take care of that in future
[05:23:36] <wikibugs_>	 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 05Language-Engineering October-December 2016, and 4 others: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2801601 (10KartikMistry) Thanks @mobrovac I'll review patches and ask Alex or s...
[06:26:39] <greg-g>	 Krinkle: nice! I like!
[06:48:57] <wmf-insecte>	 Yippee, build fixed!
[06:48:58] <wmf-insecte>	 Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #179: 09FIXED in 2 hr 8 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/179/
[08:06:54] <wmf-insecte>	 Yippee, build fixed!
[08:06:55] <wmf-insecte>	 Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #214: 09FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/214/
[08:33:00] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] [EmailAuth] add standard endpoints [integration/config] - 10https://gerrit.wikimedia.org/r/322004 (owner: 10Gergő Tisza)
[08:33:55] <grrrit-wm>	 (03Merged) 10jenkins-bot: [EmailAuth] add standard endpoints [integration/config] - 10https://gerrit.wikimedia.org/r/322004 (owner: 10Gergő Tisza)
[08:38:05] <wikibugs_>	 10Browser-Tests-Infrastructure: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2801780 (10hashar) The CI slaves have it via puppet modules/contint/manifests/browsers.pp  | Distro | Package | Binary |--|--|-- | Debian | chromedriver | /usr/lib/chromium/chromedriver | U...
[08:45:31] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2716861 (10hashar) @demon can we stick with 400GBytes disk ?  Not sure there is a point in buying 800 GBytes disk if we have 400G ones already...
[09:38:54] <wikibugs_>	 10Continuous-Integration-Infrastructure: Frivolous Jenkins failures for Selenium due to DB error - https://phabricator.wikimedia.org/T144247#2801862 (10hashar) 05Open>03Resolved a:03hashar Wikibase change a94fe6c634780cd203ea79287b61966bacfbfdae  has been reverted and eventually passed https://gerrit.wikim...
[09:45:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Phase out scandium.eqiad.wmnet - https://phabricator.wikimedia.org/T150936#2801865 (10hashar)
[09:49:06] <grrrit-wm>	 (03PS2) 10Hashar: delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[09:49:33] <grrrit-wm>	 (03PS3) 10Hashar: delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[09:50:06] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "I have attached it to T149928 and marked the dependency with the puppet patch." [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[09:51:35] <shinken-wm>	 PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[10:02:38] <shinken-wm>	 PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[10:10:50] <wikibugs_>	 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2801957 (10demon) Yeah, 400 will be fine--we currently use about 30gb. I dunno why I said 500, that's more than enough and anything in the TB+...
[10:31:24] <hashar>	 zeljkof:  Exception Caught: Cannot access the database: Access denied for user 'wikiadmin'@'10.%' to database 'centralauth' (10.68.18.35) (internal_api_error_DBConnectionError) 
[10:31:43] <hashar>	 caused by some changes in the database permissions yesterday ( poke Krenair )
[10:42:35] <shinken-wm>	 RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:54:43] <wmf-insecte>	 Yippee, build fixed!
[10:54:44] <wmf-insecte>	 Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #221: 09FIXED in 6 min 20 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/221/
[10:54:54] <wikibugs_>	 10Browser-Tests-Infrastructure: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2802061 (10hashar) On Nodepool:  Jessie ii  chromedriver     53.0.2785.143 amd64         web browser - WebDriver support ii  chromium         53.0.2785.143 amd64         web browser
[10:58:40] <wmf-insecte>	 Yippee, build fixed!
[10:58:41] <wmf-insecte>	 Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #212: 09FIXED in 37 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/212/
[10:58:57] <wmf-insecte>	 Yippee, build fixed!
[10:58:58] <wmf-insecte>	 Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #212: 09FIXED in 54 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/212/
[11:03:13] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #215: 04FAILURE in 4 min 57 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/215/
[11:05:45] <addshore>	 hashar: do things have to go through a security review to land on the beta cluster (extension wise)?
[11:08:24] <hashar>	 addshore: I guess yes
[11:21:33] <hashar>	 the selenium-Core job is all green   \o/
[11:23:23] <wikibugs_>	 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2802152 (10hashar)
[11:23:26] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Puppet: deployment-apertium01 puppet failing due to missing packages on trusty - https://phabricator.wikimedia.org/T147210#2802150 (10hashar) 05Resolved>03Open deployment-apertium01 is still around and complaining.   Maybe the deletion failed in wikitech/horizon?
[11:26:16] <wikibugs_>	 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2802174 (10hashar)
[11:26:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Puppet: deployment-apertium01 puppet failing due to missing packages on trusty - https://phabricator.wikimedia.org/T147210#2802172 (10hashar) 05Open>03Resolved I have terminated deployment-apertium01 using Horizon.
[11:34:17] <hasharLunch>	 !log Deleted instance deployment-apertium01 . Was Trusty and lacked packages, replaced by a Jessie one ages ago. T147210
[11:34:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:43:13] <wikibugs_>	 03Scap3, 10Parsoid, 06Services, 15User-mobrovac: Env vars being overwritten - https://phabricator.wikimedia.org/T150897#2802231 (10mobrovac) p:05Triage>03High Yup, the new version of Scap is definitely rewriting that setting with the production one. I have fixed the config manually on `deployment-parso...
[12:02:38] <grrrit-wm>	 (03PS1) 10Addshore: Add ElectronPdfService to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/322083 (https://phabricator.wikimedia.org/T150945) 
[12:34:52] <grrrit-wm>	 (03CR) 10WMDE-Fisch: [C: 031] Add ElectronPdfService to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/322083 (https://phabricator.wikimedia.org/T150945) (owner: 10Addshore)
[13:31:16] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2802382 (10Gilles) Nope. Thumbor's config has those values:  ``` SWIFT_HOST = 'http://deployment-ms-fe01.deployment-prep.eqiad.wmflabs' SWIFT_API_PATH = '/v1/AUTH_mw/' S...
[13:34:28] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2802385 (10Krenair) >>! In T150649#2796944, @Krenair wrote: > does prod also show u'www-authenticate': u'Swift realm="unknown"' ?
[13:40:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2802424 (10Gilles) I'm not sure what you're asking, Swift auth works in production and we don't log at debug level there. Not sure that the swiftclient library would log...
[13:46:16] <wmf-insecte>	 Yippee, build fixed!
[13:46:16] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #216: 09FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/216/
[13:46:34] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2802442 (10Gilles) I'll write some python code to mimic what Thumbor does and attempt to get the headers of the successful response in production.
[13:51:49] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2802452 (10Gilles) ``` DEBUG:swiftclient:REQ: curl -i http://ms-fe.svc.eqiad.wmnet/auth/v1.0 -X GET DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS:...
[14:16:06] <wikibugs>	 06Release-Engineering-Team, 10Tool-Labs-tools-Other, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2802503 (10Zppix) Please see  https://gerrit.wikimedia.org/r/#/c/322037/
[14:53:11] <shinken-wm>	 PROBLEM - Host deployment-pdf02 is DOWN: CRITICAL - Host Unreachable (10.68.16.129)
[14:54:29] <shinken-wm>	 PROBLEM - Host deployment-conftool is DOWN: CRITICAL - Host Unreachable (10.68.20.30)
[15:39:48] <shinken-wm>	 PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:42:57] <wmf-insecte>	 Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #231: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/231/
[15:50:44] <paladox>	 twentyafterfour hi i've noticed that on phabricator.wikimedia.org it is now taking a long time to load the open task part on the homepage
[15:50:59] <paladox>	 Used to load more like 1-3 secs now it's 1 - 30 secs
[15:51:05] <shinken-wm>	 PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:51:05] <paladox>	 this is from today
[16:00:58] <paladox>	 twentyafterfour we should update pygments to 2.* on phabricator since it does https://bitbucket.org/birkenfeld/pygments-main/commits/e1abdf91c1fe18b9ab313f3dc298d7025af5869c?at=2.0 in 2.* now which may fix some tasks in phabricator
[16:01:07] <paladox>	 We can test on phab-01 first :)
[16:21:27] <grrrit-wm>	 (03CR) 10Chad: [C: 032] make-release: don't bother creating patch dir, who cares if it's gone [tools/release] - 10https://gerrit.wikimedia.org/r/322030 (owner: 10Chad)
[16:21:35] <grrrit-wm>	 (03CR) 10Chad: [C: 032] make-release: Remove --gitrootext option, doesn't work [tools/release] - 10https://gerrit.wikimedia.org/r/322028 (owner: 10Chad)
[16:22:29] <grrrit-wm>	 (03Merged) 10jenkins-bot: make-release: Remove --gitrootext option, doesn't work [tools/release] - 10https://gerrit.wikimedia.org/r/322028 (owner: 10Chad)
[16:23:56] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Add ElectronPdfService to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/322083 (https://phabricator.wikimedia.org/T150945) (owner: 10Addshore)
[16:24:00] <grrrit-wm>	 (03Merged) 10jenkins-bot: make-release: don't bother creating patch dir, who cares if it's gone [tools/release] - 10https://gerrit.wikimedia.org/r/322030 (owner: 10Chad)
[16:24:19] <addshore>	 thanks ostriches 
[16:24:40] <addshore>	 I'm correct in thinking that wont be in a branch until the 29th?
[16:26:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add ElectronPdfService to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/322083 (https://phabricator.wikimedia.org/T150945) (owner: 10Addshore)
[16:28:26] <grrrit-wm>	 (03Abandoned) 10Chad: Bundle HitCounters extension [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) (owner: 10Nemo bis)
[16:29:17] <ostriches>	 addshore: Yeah, not next week but the week after.
[16:29:27] <ostriches>	 We could manually add it to the current branch if needed
[16:29:40] <addshore>	 ostriches: that would be great!
[16:29:52] <addshore>	 As the current plan is to get it on betawikis next week!
[16:30:13] <addshore>	 I overlooked the fact that I hadn't added it to the script & also that there would not be a branch next week
[16:31:58] <paladox>	 twentyafterfour we doint have pygments enabled on phabricator, theres a config we have to enabled. Just compared to of the same files
[16:32:04] <paladox>	 from phab-01 and phabricator
[16:32:11] <paladox>	 and both look different colour wise
[16:32:53] <Reedy>	 addshore: if it's going on beta... You don't need it branched
[16:32:54] <paladox>	 Let me submit a patch, but we should disable phabricator default syntax highlighter that will only highlight certain languages like php
[16:33:04] <paladox>	 Since pygments wont be used for php
[16:33:23] <addshore>	 Reedy: oh duh, beta runs from master...
[16:33:28] <paladox>	 if the default syntax highlighter is enabled. but i carn't seem to find a config that disabled that
[16:33:28] <Reedy>	 :D
[16:33:35] <addshore>	 Reedy: well, awesome!
[16:33:42] <ostriches>	 addshore: Actually, yeah what Reedy said.
[16:33:51] <addshore>	 :D
[16:34:02] <addshore>	 great, then everything stays on schedule! :)
[16:34:03] <ostriches>	 I was halfway through committing when I remembered :p
[16:34:18] <ostriches>	 Soooo, just add it to extension-list-labs + other wmf-config stuff you need and you should be good to go
[16:34:35] <Reedy>	 yup
[16:35:24] <Reedy>	 relatively easy
[16:37:45] <ostriches>	 Sorta related: extension-list files suck and I want them to go away
[16:37:55] <Reedy>	 ostriches: well
[16:38:01] <Reedy>	 if people would finish extension registration migrations
[16:38:06] <Reedy>	 we could remove it
[16:38:15] <Reedy>	 but some teams don't help get it done
[16:38:17] <ostriches>	 Actually we don't need that to finish :)
[16:38:30] <Reedy>	 Have all the shittly named extensions been migrated?
[16:38:31] <paladox>	 Oh with pigments disabled it disable syntax highlighting so i wonder why it looks different
[16:38:32] <ostriches>	 I might poke that
[16:38:35] <Reedy>	 And the rest are Foo/Foo.php?
[16:39:17] <paladox>	 twentyafterfour ^^
[16:40:50] <twentyafterfour>	 paladox: I don't know but we definitely use pygments ;)
[16:40:58] <paladox>	 Oh yep
[16:41:01] <twentyafterfour>	 paladox: maybe a different version of pygments
[16:41:10] <paladox>	 Oh you may be correct
[16:41:20] <paladox>	 I just updated to pygments 2.1.3
[16:41:24] <twentyafterfour>	 production is running an old version
[16:45:23] <paladox>	 Oh yep
[16:45:38] <paladox>	 twentyafterfour we could use xenial deb to update pygments?
[16:53:19] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: pbraicator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2802868 (10jcrespo)
[16:54:02] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabiicator close to saturate its database connections  - https://phabricator.wikimedia.org/T150965#2802882 (10jcrespo)
[17:01:04] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections  - https://phabricator.wikimedia.org/T150965#2802900 (10Aklapper)
[17:02:22] <paladox>	 twentyafterfour https://phabricator.wikimedia.org/T150965
[17:02:37] <paladox>	 ^^ that probaly explains the slowness of the open tasks not loading faster
[17:03:29] <twentyafterfour>	 hmmm
[17:03:47] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2802917 (10mmodell) p:05Triage>03High
[17:12:12] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2802937 (10mmodell) This seems to be related to fulltext search, I'm still investigating further.
[17:17:38] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2802956 (10jcrespo) https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=db1043&from=now-24h&to=now  This graph is intere...
[17:27:06] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803004 (10jcrespo) As I may disconnect soon, I've left a screen on db1043 killing queries running for over 600 seconds called 878.kill-long-queries as a mitiga...
[17:29:46] <wikibugs>	 03Scap3, 10Parsoid, 06Services, 15User-mobrovac: Env vars being overwritten - https://phabricator.wikimedia.org/T150897#2803021 (10thcipriani)
[17:36:02] <ebernhardson>	 wow, stat1002 has pandas 0.13.1. The latest version is 0.19.1 :P why do i get the feeling though that upgrading would break something...
[17:37:47] <ebernhardson>	 wrong room ...
[17:42:02] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803048 (10mmodell) ok so I just deployed a hotfix which should limit those token_* queries to no more than 5 tokens and I filtered short words which could have...
[17:43:07] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803056 (10mmodell) @jcrespo: The hotfix should be live, assuming it works (I'm pretty sure it will) then your mitigation script may not be needed.
[17:49:08] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803063 (10mmodell) @jcrespo: Thanks for catching this, I wouldn't have noticed until everything died.
[17:50:47] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803065 (10Paladox) I wonder what changed since the last phabricator update to cause this problem?
[17:54:20] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803071 (10jcrespo) p:05High>03Low I have changed the script to print rather than to kill, will close this tomorrow and stop the script if I get no other in...
[17:54:30] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803074 (10mmodell) @paladox: These two commits touched PhabricatorProjectQuery but I haven't figured out what might have caused the issue exactly:  * {rPHAB9a1...
[17:55:10] <wikibugs_>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803075 (10Paladox) @mmodell thanks, maybe this https://secure.phabricator.com/rPe053534c7e84b09e5f01ac3acb41352bb6a37e05 will improve things?
[17:56:08] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2803076 (10mmodell)
[18:00:19] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T150972#2803096 (10greg)
[18:02:57] <wikibugs_>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T150465#2803115 (10greg)
[18:18:22] <grrrit-wm>	 (03CR) 10Nemo bis: "What said above is incorrect." [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) (owner: 10Nemo bis)
[18:37:25] <shinken-wm>	 PROBLEM - Host deployment-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.63)
[18:54:57] <wikibugs>	 06Release-Engineering-Team: Audit of mw extensions' use of gating in CI - https://phabricator.wikimedia.org/T150701#2803356 (10greg) From the team etherpad: See https://gerrit.wikimedia.org/r/#/c/320191/1/zuul/parameter_functions.py + mediawiki/core + mediawiki/vendor
[19:00:37] <grrrit-wm>	 (03CR) 10MarkAHershberger: "Many people have asked me about this. I don't think it should be abandoned." [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) (owner: 10Nemo bis)
[19:02:49] <wikibugs>	 06Release-Engineering-Team, 07Documentation, 07Easy: Merge Wikimedia's "Deployment checklist for new extensions" doc pages - https://phabricator.wikimedia.org/T142081#2803377 (10Aklapper) a:03Aklapper @Xephyr826 : Thanks for having taken a look at this (and the compliments) and again sorry that this turned...
[19:04:48] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2803392 (10fgiunchedi) 05Open>03Resolved @krenair I don't think the realm/headers is related  The swift proxy needs restarting once credentials are in place, I've do...
[19:06:20] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2803398 (10Krenair) thanks. puppet doesn't handle that automatically?
[19:10:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2803407 (10fgiunchedi) No it doesn't because in production that'd mean uncoordinated restarts of swift-proxy. It would probably be fine I think to restart since swift-pr...
[19:13:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2803410 (10Krenair) >>! In T150649#2803407, @fgiunchedi wrote: > No it doesn't because in production that'd mean uncoordinated restarts of swift-proxy. It would probably...
[19:15:03] <godog>	 Krenair: re: swift is it still on your radar to get backend instances with smaller disks? if so happy to help with that
[19:15:44] <Krenair>	 godog, vaguely, in that I think we should do it at some stage to not waste space. I wasn't really planning to work on it because I don't know what's involved
[19:17:04] <Krenair>	 I think it would be blocked on labs ops creating a flavour
[19:17:23] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Operations, 10Thumbor: Thumbor keeps losing Swift auth on beta - https://phabricator.wikimedia.org/T150649#2803414 (10fgiunchedi) I'd say the closest to such a list is https://wikitech.wikimedia.org/wiki/Service_restarts
[19:20:44] <grrrit-wm>	 (03CR) 10Chad: "The problem is that "people have asked me" is confirmation bias, nor is it how we decide to bundle things. The download stats for this ext" [tools/release] - 10https://gerrit.wikimedia.org/r/221059 (https://phabricator.wikimedia.org/T74420) (owner: 10Nemo bis)
[19:30:32] <godog>	 Krenair: *nod* yeah in the grand scheme of things 200GB doesn't seem like a lot
[20:00:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 07Pywikibot-tests: Perform full test suite using Wikimedia CI - https://phabricator.wikimedia.org/T132138#2803620 (10Magul) a:03Magul
[20:01:00] <Krenair>	 godog, I guess if you're used to dealing with production swift systems sure
[20:04:52] <Krenair>	 most labs instances are not that lage
[20:04:54] <Krenair>	 large*
[20:11:03] <godog>	 that too yeah, I meant more in the general labs scheme of things. Anyways I can help once the instances are online, it should be enough to add the new instances to swift-ring.git, rebalance and remove the old instances once the new ones are fully in service
[20:19:28] <andre__>	 Who's into MW DB stuff nowadays? I feel bad always CC'ing aaron on such stuff but then I'd also love to see mwjames receiving an answer in https://phabricator.wikimedia.org/T147791
[20:20:47] <greg-g>	 https://www.mediawiki.org/wiki/Developers/Maintainers just has aaron and jaime :/
[20:22:54] <andre__>	 Meh
[20:23:18] <andre__>	 paladox: why do you think T147791 should block the 1.28 release?
[20:23:50] <paladox>	 Because it shows the release version number as 1.28 alpha
[20:24:01] <paladox>	 not 1.29, so it is in the 1.28 release
[20:24:19] <andre__>	 paladox: Okay. Could you re-read my question and try to answer it? :)
[20:24:29] <paladox>	 I did
[20:25:05] <wikibugs_>	 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2803734 (10RobH) >>! In T148187#2801786, @hashar wrote: > @demon can we stick with 400GBytes disk ?  Not sure there is a point in buying 800 G...
[20:25:23] <paladox>	 because it has the number 1.28 saying it is mw 1.28, and broken db again, but i just tagged it on that project not to block it, just so someone can look through it fixing it
[20:25:27] <andre__>	 paladox: you did not. Please don't add the 1.28 tag when there is no good reason.
[20:25:34] <paladox>	 even if it is in the 1.28.1 release
[20:25:36] <andre__>	 paladox, "broken DB" is not a reason.
[20:25:42] <paladox>	 ok
[20:25:58] <andre__>	 paladox: if it's broken for a lot of people and backends it's a reason. And we don't know that yet.
[20:26:10] <Reedy>	 "PS: Running the tests on a "normal" PHP 7.0.11 with Xdebug 2.4.1 works just fine."
[20:26:11] <paladox>	 ok
[20:26:12] <andre__>	 paladox: and "it shows the release version number as 1.28 alpha" is never a reason on its own.
[20:26:20] <paladox>	 ok
[20:26:41] <andre__>	 paladox: just because someone runs version XYZ-alpha does not mean that every single issue should block XYZ-final.
[20:26:53] <paladox>	 ok
[20:26:55] <andre__>	 paladox, so please provide reasons when saying that something should be a blocker
[20:27:00] <andre__>	 thanks.
[20:27:01] <paladox>	 ok
[20:27:24] <paladox>	 andre__ why did you say here https://secure.phabricator.com/T8510#200936 that we are using https://secure.phabricator.com/D16886
[20:27:28] <paladox>	 when in fact we are not
[20:27:48] <paladox>	 i carn't find that patch listed any where in https://secure.phabricator.com/D16886
[20:27:50] <paladox>	 woops
[20:27:54] <paladox>	 https://phabricator.wikimedia.org/diffusion/PHAB/history/wmf%252Fstable/
[20:28:11] <andre__>	 We do.
[20:28:18] <andre__>	 No problem :)
[20:28:21] <paladox>	 oh sorry
[20:28:22] <wikibugs_>	 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2803739 (10RobH) Chatted with Chad in IRC.  Basically means the 800GB quote (on sub task) may be overkill, and I'll get a quote for dual 400GB.
[20:28:27] <paladox>	 my mistake
[20:28:32] <paladox>	 we have it deploy
[20:28:33] <paladox>	 sorry
[20:28:38] <andre__>	 no problem :)
[20:28:42] <paladox>	 ok :)
[20:40:08] <wikibugs>	 10Browser-Tests-Infrastructure: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2803812 (10Krinkle) @hashar Looks like we probably just need to ensure [mw-set-env-mw-selenium.sh#L10](https://github.com/wikimedia/integration-jenkins/blob/ddcc42e3f6e069ba7dcf24ca07122c4e...
[20:41:51] <wmf-insecte>	 Yippee, build fixed!
[20:41:51] <wmf-insecte>	 Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #214: 09FIXED in 50 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/214/
[20:41:56] <wmf-insecte>	 Yippee, build fixed!
[20:41:57] <wmf-insecte>	 Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #214: 09FIXED in 56 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/214/
[21:20:56] <wikibugs_>	 10Browser-Tests-Infrastructure: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2803903 (10hashar) ``` # Selenium requires the chromedriver binary to be found in our PATH for path in /usr/lib/chromium-browser /usr/lib/chromium; do  if test -d "$path"; then   export PAT...
[21:37:25] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[21:37:53] <mutante>	 hashar: are you here
[21:38:05] <mutante>	 i dont have +2 on the integration/docroot repo
[21:38:14] <hashar>	 mutante: good morning
[21:38:16] <mutante>	 but it's best to merge that together
[21:38:16] <hashar>	 or whatever
[21:38:17] <hashar>	 yeah
[21:38:22] <mutante>	 good lunch time :)
[21:38:24] <mutante>	 i am at office
[21:38:28] <hashar>	 have you pushed the puppet patch? :)
[21:38:31] <mutante>	 there was special metrics meeting
[21:38:47] <hashar>	 the change in  integration/docroot has a depends-on: whatever puppet patch
[21:38:51] <mutante>	 i have merged but not ran merge on master
[21:38:55] <hashar>	 so Zuul will refuses to merge it
[21:39:07] <hashar>	 or maybe only us have +2 there
[21:39:11] <mutante>	 https://gerrit.wikimedia.org/r/#/c/322019/
[21:39:14] <mutante>	 merged
[21:39:21] <mutante>	 i could only +1 on the other one
[21:39:22] <mutante>	 yes
[21:39:26] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[21:39:36] <mutante>	 thanks
[21:39:51] <hashar>	 some jenkins job is going to fetch/checkit out
[21:39:52] <mutante>	 merging on puppetmaster, running on contint1001
[21:39:55] <hashar>	 on contint1001 under /srv
[21:39:59] <hashar>	 then we will want to force run puppet
[21:40:00] <grrrit-wm>	 (03Merged) 10jenkins-bot: delete .htaccess files for doc/integration [integration/docroot] - 10https://gerrit.wikimedia.org/r/322020 (https://phabricator.wikimedia.org/T149928) (owner: 10Dzahn)
[21:40:02] <hashar>	 and see what happens
[21:40:13] <hashar>	 done https://integration.wikimedia.org/ci/job/integration-docroot-deploy/190/console
[21:40:18] <mutante>	 running puppet ..
[21:40:24] <wikibugs>	 06Release-Engineering-Team, 10Tool-Labs-tools-Other, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2803986 (10Zppix) p:05Triage>03Normal
[21:40:44] <hashar>	 this way we dont have to enable AllowOverride 
[21:40:44] <mutante>	 applied on contint1001, service refresh.. done
[21:40:49] <hashar>	 neat
[21:41:23] <hashar>	 https://doc.wikimedia.org/cover/mediawiki-core  --> 403
[21:41:23] <mutante>	 confirmed the old .htaccess are gone from docroots
[21:41:46] <hashar>	 (I hate apache)
[21:42:42] <mutante>	 we have to add the directory and allow indexing. right
[21:43:20] <mutante>	 i dont see "cover" among the rewrite rules
[21:43:25] <wikibugs_>	 10Continuous-Integration-Infrastructure, 13Patch-For-Review: contint: move .htacess file for doc.wm into regular Apache config - https://phabricator.wikimedia.org/T149928#2803989 (10hashar) 05Open>03Resolved a:03Dzahn Files have been moved from integration/docroot.git to puppet.git by @Dzahn  Rules might...
[21:43:44] <hashar>	 trying to find another example
[21:43:59] <mutante>	 doc.wm.org is Bad request in general .. hrmm
[21:44:19] <mutante>	 integration works 
[21:44:43] <mutante>	 stopped puppet so we can try a fix
[21:44:47] <mutante>	 looking at logs
[21:45:18] <hashar>	 trying to figure out what is wrong inside https://gerrit.wikimedia.org/r/#/c/322019/5/modules/contint/templates/apache/doc.wikimedia.org.erb
[21:45:33] <mutante>	 Cannot service directory..
[21:45:37] <mutante>	 No matching DirectoryIndex
[21:46:10] <mutante>	 we dont allow indexing 
[21:46:16] <hashar>	 yeah
[21:46:44] <hashar>	 we have a rewrite rule to route requests to dir.php
[21:47:00] <hashar>	 if the requested filename is a directory  and there is no index.php or index.php in it
[21:47:12] <hashar>	 but
[21:47:32] <mutante>	  27     # DirectoryIndex would be neater, but doesn't work properly under Apache 2.2
[21:47:37] <mutante>	 we dont care about 2.2 anymore though
[21:48:07] <hashar>	 at least https://doc.wikimedia.org/cover/ works
[21:48:14] <hashar>	 partially :(
[21:48:45] <mutante>	 i tried a live hack
[21:48:58] <mutante>	 DirectoryIndex disabled
[21:49:01] <mutante>	 Options Indexes
[21:49:10] <mutante>	 in the <Directory />
[21:49:18] <hashar>	 Symbolic link not allowed or link target not accessible: /srv/org/wikimedia/doc/lib, referer: https://doc.wikimedia.org/index.php
[21:49:50] <mutante>	 yes, this changed the error
[21:49:58] <mutante>	 before it was about the index
[21:50:11] <hashar>	 maybe we had Options previously
[21:50:23] <hashar>	 or that is because the doc.wikimedia.org  virtual host now has the proper DocumentRoot
[21:51:14] <hashar>	 https://doc.wikimedia.org/cover/ now gives the directory index
[21:51:39] <mutante>	 try / now
[21:51:59] <mutante>	 that is when it indexes and i comment out the rules we moved
[21:52:02] <hashar>	 https://doc.wikimedia.org/ no more 400
[21:52:18] <hashar>	 but we dont want apache index and instead use our own custom index builder dir.php :D
[21:52:58] <hashar>	 guess our rewrite rules are doing something funky
[21:54:32] <mutante>	 reload
[21:54:33] <mutante>	 fixed?
[21:54:42] <mutante>	 i added DirectoryIndex dir.php
[21:54:47] <mutante>	 and removed what i did earlier
[21:54:48] <hashar>	 oh
[21:55:22] <hashar>	 then https://doc.wikimedia.org/cover/WrappedString/ yields a 403
[21:55:26] <hashar>	 when it has an index.html
[21:55:37] <hashar>	 (eg https://doc.wikimedia.org/cover/WrappedString/index.html )
[21:55:52] <mutante>	 should i just add both?
[21:55:55] <mutante>	 dir.php index.html
[21:56:20] <mutante>	 done
[21:56:23] <mutante>	 works better, right
[21:56:25] <hashar>	 and index.php
[21:56:46] <mutante>	 done
[21:56:55] <mutante>	 let me puppetize the live hack then
[21:57:00] <mutante>	 and re-enable it
[21:57:41] <hashar>	 then https://doc.wikimedia.org/cover/mediawiki-core/  yields a 403
[21:58:25] <hashar>	 on purpose since the directory only contains a directory './master/'
[21:58:30] <hashar>	 eg neither index.html nor index.php
[21:58:55] <hashar>	 that is what   RewriteRule .* dir.php   was for
[21:59:57] <hashar>	 so Directory index is not quite the magic solution
[22:05:08] <hashar>	 mutante: I am tempted to use the apache config that yields a 400
[22:05:11] <mutante>	 https://gerrit.wikimedia.org/r/#/c/322201/3
[22:05:14] <hashar>	 then add some mod rewrite log
[22:05:23] <hashar>	 apache doc hints at   LogLevel alert rewrite:trace3
[22:05:23] <mutante>	 that first to unbreak the main page ?
[22:05:30] <mutante>	 it fixes most things, just not all
[22:05:30] <mutante>	 right
[22:05:39] <hashar>	 yup
[22:05:54] <mutante>	 and before it also wasnt perfect about the redirects
[22:06:02] <mutante>	 (open ticket about https->http )
[22:06:09] <mutante>	 afair
[22:06:13] <mutante>	 ok
[22:07:04] <mutante>	 !log re-enabled puppet on contint1001 after live Apache fix
[22:07:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:07:11] <hashar>	 oh  doc/cover has its own dir.php
[22:07:12] <hashar>	 bah
[22:07:30] <mutante>	 heh
[22:07:40] <hashar>	 different title :D
[22:07:47] <mutante>	 ok, it's applied and running puppet as normal again
[22:08:02] <hashar>	 great
[22:09:47] <hashar>	 mutante: actually I dont think we should add dir.php to directoryindex
[22:09:48] <hashar>	 bah
[22:09:54] <mutante>	 so the .htaccess files were just not being read before or it would have broken earlier
[22:09:55] <hashar>	 https://doc.wikimedia.org/index.php  is what is supposed to be shown
[22:10:00] <mutante>	 hmm, ok
[22:10:09] <hashar>	 dir.php being the low level interface that just foreach($dirs) print $dir
[22:10:18] <mutante>	 i already like that we could remove those old rules for 2.2
[22:10:22] <mutante>	 eithe rway
[22:10:23] <hashar>	 yeah me to
[22:10:44] <hashar>	 i should have tested the patch locally really
[22:11:12] <mutante>	 do you want to see it without dir.php ?
[22:11:23] <hashar>	 yeah
[22:11:44] <hashar>	 then we will need the rewriteconde/rewrite rule that fallback to dir.php
[22:11:49] <mutante>	 ok, done
[22:11:55] <mutante>	 now it just has index.php and index.html
[22:11:58] <hashar>	 neat now https://doc.wikimedia.org/  is fine
[22:12:02] <mutante>	 :)
[22:12:24] <hashar>	 so the use case we have 
[22:12:29] <mutante>	 yea, and other links seem fine too now
[22:12:30] <hashar>	 is a directory containing just directories
[22:12:32] <hashar>	 eg https://doc.wikimedia.org/cover/mediawiki-core/
[22:13:03] <mutante>	 i think we need to list them 
[22:13:06] <hashar>	 with the rewritecond previously, since that is a directory and there is neither index.html or index.php,  we rewrite to use dir.php
[22:13:09] <mutante>	 but there are too many to list?
[22:13:34] <mutante>	 we had another case before where the 2.2->2.4 change meant we had to add Directories individually
[22:13:45] <mutante>	 and permit listing
[22:14:08] <mutante>	 a default changed there i think
[22:14:30] <hashar>	 what I am wondering is why the rewrite does not work
[22:15:46] <hashar>	 bah apache has so many things
[22:16:04] <hashar>	 https://httpd.apache.org/docs/2.4/en/rewrite/remapping.html#fallback-resource
[22:19:07] <mutante>	 hashar: should i puppetized the removal of "dir.php" really quick
[22:19:16] <hashar>	 yup
[22:19:20] <mutante>	 and is it ok to leave for then
[22:19:24] <mutante>	 to drive back to Santa Cruz
[22:19:26] <mutante>	 from office
[22:19:35] <hashar>	 yeah just drop the dir.php from directoryindex 
[22:19:38] <hashar>	 that is good enough for now
[22:19:38] <mutante>	 ok
[22:19:41] <mutante>	 great
[22:19:43] <hashar>	 at least the rules are no more in docroot
[22:19:48] <hashar>	 and we dont have AllowOverride
[22:19:51] <mutante>	 yes :) and the compat code is gone
[22:19:54] <mutante>	 yes
[22:19:55] <hashar>	 so that is a good step forwad
[22:20:12] <hashar>	 the other bug https://phabricator.wikimedia.org/T150727  is not fixed
[22:20:17] <hashar>	 but it is not so bad
[22:20:37] <hashar>	 will have to reproduce locally and figure out a solution
[22:21:16] <mutante>	 ok, good
[22:23:40] <wikibugs>	 06Release-Engineering-Team, 10DBA, 10Phabricator: phabricator close to saturate its database connections - https://phabricator.wikimedia.org/T150965#2804147 (10mmodell) 05Open>03Resolved a:03mmodell
[22:24:56] <wikibugs_>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression: doc.wikimedia.org displays "403 Forbidden" for coverage sub directories - https://phabricator.wikimedia.org/T150727#2804164 (10hashar) Status  * The rewrite rules have been moved from integration/docroot....
[22:27:53] <hashar>	 mutante: I am off to bed. Thanks for the rules migration and have a safe drive!
[22:28:08] <mutante>	 hashar: i merged the last change and puppet ran
[22:28:14] <mutante>	 no-op 
[22:28:18] <mutante>	 and off to drive 
[22:28:25] <mutante>	 thanks too. laters!
[23:15:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression: doc.wikimedia.org displays "403 Forbidden" for coverage sub directories - https://phabricator.wikimedia.org/T150727#2804351 (10Krinkle) >>! In T150727#2804164, @hashar wrote: > Status [..] >  > `/cover/me...
[23:43:45] <wikibugs_>	 06Release-Engineering-Team, 06Developer-Relations, 10Phabricator, 10Wikimedia-Blog-Content: blog.wikimedia.org post on Phabricator improvements - https://phabricator.wikimedia.org/T141457#2804499 (10Aklapper) So I assume everybody waits for someone else to rephrase. Which means this will never happen.  Hen...