[00:07:32] <TimStarling>	 apparently that previous scap I did was a no-op
[00:08:43] <Zppix>	 whats the difference?
[00:08:54] <Reedy>	 the second wasn't?
[00:10:37] <TimStarling>	 the second worked
[00:12:43] <TimStarling>	 I used scap sync-file on a directory, because apparently sync-dir is deprecated and sync-file can take a directory as its argument
[00:13:17] <TimStarling>	 but nothing happened
[00:13:50] <Reedy>	 I think some other people were complaining about that
[00:14:00] <Reedy>	 I keep using commands that work until they are forcefully broken
[00:14:16] <Zppix>	 TimStarling:  welcome to wmf development where we deprecate things and never give an alt
[00:15:02] <TimStarling>	 the second time, there were errors, apparently related to directory cleanup, I think I read an email from Chad about it
[00:15:07] <TimStarling>	 http://paste.tstarling.com/p/bAkPMz.html
[00:16:06] <Zppix>	 once upon time i had my own site then it got expensive
[00:41:09] <TimStarling>	 I don't think the problem is limited to sync-file
[00:48:46] <wikibugs>	 10Scap: [scap] sync-file/sync-dir on a directory has no effect - https://phabricator.wikimedia.org/T162220#3155953 (10tstarling)
[00:49:23] <TimStarling>	 anyone want to claim responsibility for that, or should I just do git blame?
[00:50:48] <TimStarling>	 I wonder who came up with that grey on slightly lighter grey colour scheme
[00:54:27] <TimStarling>	 git blame says chad, four months ago
[00:56:40] <Zppix>	 TimStarling:  what are you on about?
[00:57:40] <TimStarling>	 the bug I just filed, are you ignoring wikibugs or something?
[00:59:23] <Zppix>	 TimStarling:  what about it?
[00:59:37] <TimStarling>	 that is what I am on about
[00:59:44] <TimStarling>	 would you like to clarify your question?
[00:59:48] <bd808>	 TimStarling: hmmm... I think I've seen other complaints about this. I wonder if something got messed up when RainbowSprinkles merged the sync-file and sync-dir logic into one command
[01:00:13] <bd808>	 that terminal color scheme is horrid
[01:00:14] <Zppix>	 yay its finally me that didnt break something
[01:00:23] <RainbowSprinkles>	 That's been live for quite some time
[01:00:27] <RainbowSprinkles>	 And the complaints sporadic
[01:00:27] <Zppix>	 oh the terminal ewww
[01:01:02] <RainbowSprinkles>	 Hmmm, when running on a directory. That's more specific
[01:01:13] <TimStarling>	 yeah, I guess people don't notice every time when scap doesn't work at all
[01:01:26] <bd808>	 the exclude=* at the end ...
[01:01:30] <TimStarling>	 yes, if you specify a file then the --include matches
[01:01:35] <RainbowSprinkles>	 That final --exclude
[01:01:36] <RainbowSprinkles>	 Yep
[01:01:40] <Reedy>	 I wonder if this explains why portals stuff is broken
[01:01:52] <RainbowSprinkles>	 That would explain it
[01:03:25] <TimStarling>	 so you have all been putting up with this for months?
[01:03:30] <TimStarling>	 this is why you need me personally syncing things every once in a while
[01:03:36] <RainbowSprinkles>	 I only do full scaps :)
[01:03:38] <RainbowSprinkles>	     if include:
[01:03:38] <RainbowSprinkles>	         for path in include:
[01:03:38] <RainbowSprinkles>	             rsync.append('--include=/%s' % path)
[01:03:38] <RainbowSprinkles>	         # Exclude everything not explicitly included
[01:03:38] <RainbowSprinkles>	         rsync.append('--exclude=*')
[01:23:49] <bd808>	 hmmm... so why doesn't this -- https://github.com/wikimedia/scap/blob/master/scap/main.py#L571-L574 -- match and add a '--include=/php-1.29.0-wmf.18/extensions/ParserMigration/includes/***' to the sync?
[01:24:56] <TimStarling>	 os.path.isdir() on a relative path? Shouldn't that use abspath?
[01:25:16] <TimStarling>	 ah, so if your current directory is /srv/mediawiki-deployment, maybe it will work
[01:25:29] <TimStarling>	 I mean mediawiki-staging
[01:26:32] <bd808>	 the second arg makes it relative to staging
[01:26:42] <bd808>	 but I'm not sure why that's needed
[01:27:04] <TimStarling>	 second argument to relpath?
[01:27:08] <RainbowSprinkles>	 https://phabricator.wikimedia.org/D344 - line 594 is where I tweaked that logic when combining them
[01:27:17] <TimStarling>	 yeah I am right
[01:27:34] <bd808>	 "Return a relative filepath to path either from the current directory or from an optional start directory."
[01:28:01] <TimStarling>	 yeah so relpath contains php-1.29.0-wmf.18/extensions/ParserMigration/includes
[01:28:08] <TimStarling>	 it is relative
[01:28:14] <TimStarling>	 then you feed that into isdir()
[01:28:17] <bd808>	 doh. yeah
[01:28:52] <TimStarling>	 I tested a sync-file with the current directory being /srv/mediawiki-staging, it worked
[01:29:00] <bd808>	 the relpath should only happen when adding to the rsync command
[01:29:26] <bd808>	 so line 550 should be f os.path.isdir(abspath):
[01:29:37] <TimStarling>	 yes
[01:30:20] <RainbowSprinkles>	 Ah ok, I see what I did wrong
[01:31:02] <RainbowSprinkles>	 Tricky
[01:31:37] <bd808>	 that also explains all the 'works for me' attempts at recreating the bug
[01:32:01] <RainbowSprinkles>	 Soooo, scap2/mw should probably enforce being in the deploy directory like scap3 does
[01:32:11] <bd808>	 my cwd is almost always the staging dir when I'm scapping stuff
[01:32:12] <RainbowSprinkles>	 But we should also fix this
[01:35:19] <TimStarling>	 RainbowSprinkles: you mean just do os.chdir() at the start?
[01:36:20] <RainbowSprinkles>	 Well scap2 and 3 behaviors are converging. In an ideal future, `scap deploy` and `scap sync` are the same thing which means you need to know what you're deploying.
[01:36:37] <RainbowSprinkles>	 Right now, that implies being in a git repo that contains a ./scap/scap.cfg
[01:37:41] <wikibugs>	 10Scap: [scap] sync-file/sync-dir on a directory has no effect - https://phabricator.wikimedia.org/T162220#3155996 (10demon)
[01:38:07] <RainbowSprinkles>	 Just uploaded https://phabricator.wikimedia.org/D615 for the issue ^
[01:38:16] <TimStarling>	 the normal definition of "being in a git repo" is being in any descendant subdirectory
[01:38:39] <RainbowSprinkles>	 Fair enough
[01:42:20] <TimStarling>	 there is "git rev-parse --show-toplevel" for this
[01:42:58] <RainbowSprinkles>	 Yes, I get that. The point being though you have to be somewhere within the git repo at all (as scap3 understands it), it doesn't have to be toplevel
[01:43:07] <RainbowSprinkles>	 It also doesn't make any assumptions about relative/absolute paths
[01:43:16] <RainbowSprinkles>	 Remember: the ideal universe also is moving towards a "deploy everything" model. Sync-file & friends are kind of on their way out
[01:49:32] <RainbowSprinkles>	 We're actually part way there. In the masters, the staging repo with all the core sub-repos and their submodules are being flattened into a single git repo
[01:51:07] <RainbowSprinkles>	 bd808: Fun fact on `scap clean --keep-static`? An initial branch prep on tuesdays is about 5-10m shorter with the *thousands* fewer files we have to look at
[01:51:23] <RainbowSprinkles>	 (like, tens of thousands)
[01:52:11] <RainbowSprinkles>	 The savings during a full scap outweighs the weight of a new branch ;-)
[01:53:19] <Zppix>	 bye all 
[01:53:27] <RainbowSprinkles>	 TimStarling: Landed my fix for the paths in sync-dir. It'll be on beta soon. I'll do some testing tomorrow, and we can get a bugfix release out shortly after
[01:53:38] <wikibugs_>	 10Scap: [scap] sync-file/sync-dir on a directory has no effect - https://phabricator.wikimedia.org/T162220#3155999 (10demon) 05Open>03Resolved
[01:53:47] <TimStarling>	 excellent, thanks RainbowSprinkles
[01:54:10] <RainbowSprinkles>	 yw. Thanks for helping to run this down--I hadn't been able to replicate yet
[03:54:36] <wikibugs_>	 10Deployment-Systems, 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07Chinese-Sites, 07I18n: Deploy latest zh-hans/hant translations for Wikilabels on wmflabs - https://phabricator.wikimedia.org/T162108#3156050 (10Ladsgroup) I checked OAuth again (if it's broken, it would be a serious problem) a...
[03:59:22] <wikibugs>	 10Deployment-Systems, 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07Chinese-Sites, 07I18n: Deploy latest zh-hans/hant translations for Wikilabels on wmflabs - https://phabricator.wikimedia.org/T162108#3156051 (10Ladsgroup) The changes are deployed now, please check and tell me.
[04:04:31] <wikibugs_>	 (03CR) 10Krinkle: [C: 04-1] Cache node_modules (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar)
[04:37:53] <wikibugs>	 10Deployment-Systems, 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07Chinese-Sites, 07I18n: Deploy latest zh-hans/hant translations for Wikilabels on wmflabs - https://phabricator.wikimedia.org/T162108#3156077 (10Arthur2e5) 05Open>03Resolved Looks like plain old bad luck as it is working no...
[05:51:25] <wikibugs_>	 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3156224 (10Osnard) @Paladox and @Reedy: Awesome, thanks!
[06:43:33] <wmf-insecte>	 Yippee, build fixed!
[06:43:34] <wmf-insecte>	 Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #321: 09FIXED in 2 hr 3 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/321/
[08:07:30] <wikibugs_>	 10Continuous-Integration-Config, 10Deployment-Systems, 06Release-Engineering-Team: Skip running `test` pipeline for wmf branches (gate-and-submit is enough) - https://phabricator.wikimedia.org/T129719#3156467 (10hashar) 05Open>03declined Thank you Timo for the feedback.  I am declining this crazy idea of...
[08:33:56] <icinga-wm>	 PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:40:56] <icinga-wm>	 RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[09:13:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10media-storage: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247#3156650 (10hashar)
[09:15:30] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10media-storage, 13Patch-For-Review: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3156682 (10hashar) The summary...
[09:23:46] <wikibugs_>	 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3156696 (10Paladox) Your welcome :)
[09:25:46] <wikibugs>	 10Beta-Cluster-Infrastructure, 10media-storage: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247#3156702 (10hashar)
[09:40:02] <wikibugs_>	 (03PS1) 10WMDE-Fisch: Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515
[09:43:51] <wikibugs>	 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3156736 (10Paladox) @Osnard hi, the tests now get these P5209 failures.
[09:48:26] <wikibugs_>	 (03CR) 10Addshore: [C: 031] Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[09:53:35] <wikibugs>	 (03CR) 10Tobias Gritschacher: [C: 032] Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[09:55:29] <wikibugs_>	 (03PS2) 10Tobias Gritschacher: Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[09:56:47] <wikibugs_>	 (03CR) 10Tobias Gritschacher: Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[09:56:53] <wikibugs>	 (03CR) 10Tobias Gritschacher: [C: 032] Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[09:58:18] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Add BetaFeatures as a RevisionSlider dep" [integration/config] - 10https://gerrit.wikimedia.org/r/346515 (owner: 10WMDE-Fisch)
[10:11:31] <wikibugs>	 10Beta-Cluster-Infrastructure: Request for static config file on the beta cluster returns a 403 - https://phabricator.wikimedia.org/T162164#3154252 (10hashar) That is defined in `operations/mediawiki-config`.  `static/current/extensions` is a symbolic link to `/srv/mediawiki/php/extensions`  `/srv/mediawiki/php`...
[10:24:56] <paladox>	 hashar it seems java 1.8 also needs to be installed on all the slaves too (just letting you know). It may affect android tests if we were to ever do that.
[10:25:45] <andre__>	 Heads-up: There is a 404 error for the pretty common "Lock_icon_blue.gif" icon (used in CSS for external links) in https://phabricator.wikimedia.org/T162235 - not sure if that's releng or already ops.
[10:26:01] <andre__>	 ^ greg-g
[10:26:36] <andre__>	 (bringing up here as scap had some 'old stuff cleanup' changes recently, though likely unrelated?)
[10:45:14] <wikibugs>	 10Continuous-Integration-Infrastructure: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3156911 (10hashar) Digging in the Jenkins build logs: ``` $ grep -Z -l peerDependencies /srv/jenkins/builds/*npm*/*/log|xargs -0 -n1 grep 'Cloning repository'|cut -d\/ -f4-|sort|uniq -c...
[10:53:17] <hashar>	 andre__: that is probably related to the thumbnailing / Swift file storage 
[10:54:23] <hashar>	 andre__: ah no. The file does not exist on enwiki
[11:08:00] <hashar>	 andre__: found the issue :-}  bunch of css files have to be updated, a bot can probably deal with that
[11:08:05] <shinken-wm>	 PROBLEM - Puppet run on integration-c1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[11:08:54] <andre__>	 hashar: Uh! Thanks a lot!
[11:10:22] <hashar>	 basically css get copy pasted from sites to sites :D
[11:13:44] <hashar>	 going to ask en wiki sysops
[11:13:57] <hashar>	 andre__: I could restore the file right away but I will ask enwiki sysops to do it 
[11:14:46] <andre__>	 (shouldn't Commons be used as the canonical source for such files? Meh)
[11:14:48] <andre__>	 in any case thanks
[11:15:24] * andre__ unpings greg-g - sorry man!
[11:53:20] <shinken-wm>	 PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[12:13:20] <shinken-wm>	 RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0]
[12:31:41] <wikibugs>	 10Continuous-Integration-Config: some jjb inline bash snippets might miss set -eu - https://phabricator.wikimedia.org/T106384#3157099 (10hashar) All scripts under `config/dib` have the following boilerplate: ```  if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then     set -x fi set -eu set -o pipefail ```  It is enforced b...
[12:43:13] <wikibugs_>	 (03CR) 10Hashar: Cache node_modules (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar)
[12:46:07] <wikibugs>	 (03PS10) 10Hashar: (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831
[12:48:40] <wikibugs_>	 (03CR) 10jerkins-bot: [V: 04-1] (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)
[13:02:13] <wikibugs_>	 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3157146 (10Osnard) Okay... too sad. But at least tests are running. So now it is our turn to find out why they pass in our environment but fail in yours...
[13:04:42] <wikibugs_>	 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3157151 (10Paladox) Osnard do you test with mw 1.27 and master or just 1.27? I think in 1.28 they added ApiPermissionManagerTest in the tests so that may explain...
[13:09:47] <wikibugs_>	 (03PS11) 10Hashar: test gate-and-submit only has voting jobs [integration/config] - 10https://gerrit.wikimedia.org/r/173831
[13:11:36] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] test gate-and-submit only has voting jobs [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)
[13:23:15] <elukey>	 I am working on jobrunner02 atm, just trying one thing.. need to stop hhvm
[13:29:31] <wikibugs_>	 (03PS12) 10Hashar: Remove non-voting jobs from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/173831
[13:34:39] <ema>	 !log testing possible fix for T162035 on deployment-ms-fe01 
[13:34:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:34:42] <stashbot>	 T162035: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035
[13:40:38] <wikibugs>	 (03CR) 10Hashar: [C: 032] "All the original attempt eventually led to a real test suite.  What was left to do is to ensure that non-voting jobs are not in gate-and-s" [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)
[13:41:36] <wikibugs_>	 (03Merged) 10jenkins-bot: Remove non-voting jobs from gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)
[13:49:08] <shinken-wm>	 PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[14:11:50] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-General-or-Unknown, 07JavaScript, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 4 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3157335 (10zeljkofilipin)
[14:11:52] <wikibugs_>	 10Browser-Tests-Infrastructure, 07JavaScript, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests using Firefox - https://phabricator.wikimedia.org/T161697#3157334 (10zeljkofilipin) 05Open>03stalled
[14:12:40] <wikibugs_>	 (03PS3) 10Hashar: test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890
[14:14:03] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890 (owner: 10Hashar)
[14:14:36] <wikibugs_>	 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow, 07Chinese-Sites, 15User-Luke081515: Request for "Flow"ing pages on zhwp Beta-cluster - https://phabricator.wikimedia.org/T162131#3157337 (10Trizek-WMF) >>! In T162131#3155052, @hashar wrote: > @Trizek-WMF @Danny_B Chinese projects seems...
[14:24:37] <wikibugs_>	 (03PS4) 10Hashar: test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890
[14:24:39] <wikibugs>	 (03PS1) 10Hashar: [ArticleFeedback] repo has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/346551
[14:24:41] <wikibugs_>	 (03PS1) 10Hashar: Switch mw-checks-test to extension-unittests-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/346552
[14:26:08] <wikibugs_>	 (03CR) 10jerkins-bot: [V: 04-1] test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890 (owner: 10Hashar)
[14:26:52] <wikibugs>	 (03CR) 10Hashar: [C: 031] "They are probably easily fixable :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/346552 (owner: 10Hashar)
[14:27:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow, 07Chinese-Sites, 15User-Luke081515: Request for "Flow"ing pages on zhwp Beta-cluster - https://phabricator.wikimedia.org/T162131#3157389 (10vjudge404) Flow are used by more than 1500 users now so bot owners and gadget creators can't igno...
[14:27:20] <wikibugs_>	 (03CR) 10Hashar: [C: 032] [ArticleFeedback] repo has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/346551 (owner: 10Hashar)
[14:28:14] <wikibugs_>	 (03Merged) 10jenkins-bot: [ArticleFeedback] repo has been archived [integration/config] - 10https://gerrit.wikimedia.org/r/346551 (owner: 10Hashar)
[14:33:42] <wikibugs>	 10Beta-Cluster-Infrastructure: Request for static config file on the beta cluster returns a 403 - https://phabricator.wikimedia.org/T162164#3157421 (10Mholloway) 05Open>03Invalid Thank you, @hashar!  I've updated our [[ https://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Wikipedia_Android_app_hacking | team w...
[14:41:00] <paladox>	 hashar looks like all three new mac's will be added to trilead :) instead of one.
[15:08:35] <paladox>	 hashar he will merge the change in 24 hours if no review :)
[16:10:46] <icinga-wm>	 PROBLEM - jenkins_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war
[16:13:15] <icinga-wm>	 ACKNOWLEDGEMENT - jenkins_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war amusso Jenkins is not active on contint2001 yet.
[17:54:17] <wikibugs_>	 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3158160 (10thcipriani) 05Resolved>03Open Just tagged scap debian/3.5.4-1: {rMSCAf78caf877875bfc38f9dbb4a8ac67ddb969b3c78}  @fgiunchedi could you update the package on carbon, please?
[18:12:16] <SMalyshev>	 I'm having some Lua fails on CI - anybody knows what can it be? https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer-trusty/309/console
[18:12:39] <SMalyshev>	 regular tests are fine but merging fails
[18:17:28] <paladox>	 twentyafterfour checkout http://phabricator-random-wikisaur.ml :)
[18:18:48] <wikibugs>	 (03CR) 10Jforrester: "Can we put commit-message-validator back in as voting?" [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar)
[18:22:45] <SMalyshev>	 greg-g, Reedy, twentyafterfour: any ideas about Lua OOM fails above? ^
[18:23:09] <Reedy>	 Chances are if you recheck it, it'll pass
[18:24:50] <greg-g>	 from our team meeting notes this morning:
[18:24:51] <greg-g>	 * CI instances out of RAM when running Wikibase
[18:24:53] <greg-g>	 ** TLDR: https://phabricator.wikimedia.org/T125050#3153282  PHPUnit uses ~2GB ram and fork() fails due to not enough memory (instance only has 1.5G free and linux refuses to fork)
[18:26:10] <SMalyshev>	 Reedy: recheck works, but merge does not :(
[18:26:47] <SMalyshev>	 ah, nice, now recheck doesn't work either....
[18:27:00] <SMalyshev>	 see https://gerrit.wikimedia.org/r/#/c/334194/
[18:31:59] <Reedy>	 https://gerrit.wikimedia.org/r/#/c/336838/
[18:32:05] <Reedy>	 Why is jenkins running seemingly master tests?
[18:32:11] <Reedy>	 There's no MessagesTests in 1.23
[18:42:17] <wikibugs>	 10Continuous-Integration-Config, 06Labs, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154301 (10chasemp) Using swap seems like a non-starter unless we are totally devoid of RAM capacity in which case we h...
[18:47:07] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3158369 (10Paladox) The rc0 will be released on monday so getting close to a release :)
[18:48:00] <wikibugs_>	 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#3158370 (10Paladox) Upstream will merge https://github.com/jenkinsci/trilead-ssh2/pull/14 that in less then 24...
[19:19:36] <wikibugs_>	 10Scap (Scap3-Adoption-Phase1), 10RESTBase, 13Patch-For-Review, 06Services (doing), 15User-mobrovac: Deploy RESTBase with scap3 - https://phabricator.wikimedia.org/T116335#3158458 (10mobrovac) The first round of patches scheduled for PuppetSWAT on [Thursday, 2017-04-06](https://wikitech.wikimedia.org/wik...
[19:37:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Performance-Team, 07Wikimedia-Multiple-active-datacenters: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#3158549 (10Krinkle) p:05Normal>03Low a:05aaron>03None Lowering priority pending outcome of {T156938}.
[19:49:49] <paladox>	 hashar blueocean is now stable
[19:50:15] <paladox>	 it also seemed to cause jenkins website to get slower. As lots of users have been downloading it.
[19:51:07] <paladox>	 Thats what someone said in #jenkins-infra
[19:56:29] <paladox>	 https://jenkins.io/projects/blueocean/
[19:59:08] <wikibugs_>	 (03PS1) 10Chad: Fix up "make-release: remove offline mode, it's weird" [tools/release] - 10https://gerrit.wikimedia.org/r/346600
[19:59:55] <wikibugs_>	 10Continuous-Integration-Config, 06Labs, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3158650 (10hashar) >>! In T162166#3158359, @chasemp wrote: > Using swap seems like a non-starter unless we are totally...
[20:05:56] <wikibugs_>	 (03CR) 10Chad: [C: 032] Fix up "make-release: remove offline mode, it's weird" [tools/release] - 10https://gerrit.wikimedia.org/r/346600 (owner: 10Chad)
[20:06:35] <wikibugs>	 (03Merged) 10jenkins-bot: Fix up "make-release: remove offline mode, it's weird" [tools/release] - 10https://gerrit.wikimedia.org/r/346600 (owner: 10Chad)
[20:22:47] <shinken-wm>	 PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[20:24:17] <shinken-wm>	 PROBLEM - Puppet run on deployment-trending01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[20:25:17] <shinken-wm>	 PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[20:25:25] <shinken-wm>	 PROBLEM - Puppet run on deployment-puppetmaster02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[20:26:05] <shinken-wm>	 PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[20:26:40] <Zppix>	 hashar: if you have minute mind doing me a favor involving puppet?
[20:27:07] <shinken-wm>	 PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[20:29:12] <wikibugs>	 10Continuous-Integration-Config, 06Labs, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154301 (10Platonides) Given that the problem seems to be just the algorithm memory check rather than actually needing...
[20:42:24] <wikibugs>	 10Continuous-Integration-Config, 06Labs, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3158827 (10hashar) Oops I forgot about overcommit_memory which I mentioned on the parent task. From T125050#3153574 :...
[20:43:33] <Zppix>	 paladox:  the old web browser support drop whats its phab ticket?
[20:43:51] <paladox>	 What do you mean old web browser support drop?
[20:43:53] <paladox>	 ie9 ?
[20:44:10] <Zppix>	 Some older web browsers will not be able to use JavaScript on Wikimedia wikis from this week. If you have an old web browser on your computer you can upgrade to a newer version. [2] paladox 
[20:44:22] <paladox>	 Oh, that would be ie9
[20:44:36] <paladox>	 any browser that doesnt support es5 i think
[20:44:38] <paladox>	 so ie9
[20:45:42] <paladox>	 Zppix https://phabricator.wikimedia.org/T128115
[20:45:49] <Zppix>	 thanks
[20:45:58] <paladox>	 your welcome
[20:47:45] <Zppix>	 paladox:  your new offical title is Phabricator Search Bot
[20:47:59] <paladox>	 Actuallt i used gerrit
[20:48:02] <paladox>	 to find the task
[20:48:18] <paladox>	 so much easier as i remebered seeing patches being removed dropping ie9 support
[21:00:17] <shinken-wm>	 RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:00:25] <shinken-wm>	 RECOVERY - Puppet run on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:01:04] <shinken-wm>	 RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:02:06] <shinken-wm>	 RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:02:48] <shinken-wm>	 RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0]
[21:04:18] <shinken-wm>	 RECOVERY - Puppet run on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:05:06] <wikibugs_>	 (03PS1) 10Hashar: dib: allow Linux memory overcommit on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/346634 (https://phabricator.wikimedia.org/T125050)
[21:40:01] <wikibugs_>	 10Continuous-Integration-Config, 13Patch-For-Review: 'recheck' on a CR+2 patch should trigger gate-and-submit, not test - https://phabricator.wikimedia.org/T105474#1444510 (10Zppix) @hashar  was this ever implemented?
[21:50:35] <Zppix>	 i wish i could find an easy task on ci config
[22:01:46] <wikibugs_>	 10Continuous-Integration-Config, 06Release-Engineering-Team: Priortize gerrit changesets that contain the comment of "SWAT" - https://phabricator.wikimedia.org/T162316#3158988 (10Zppix)
[22:11:45] <wikibugs_>	 10Continuous-Integration-Config, 06Release-Engineering-Team: Priortize gerrit changesets that contain the comment of "SWAT" - https://phabricator.wikimedia.org/T162316#3158988 (10demon) I disagree, we should do this via branches & repos: wmf/* branches should be prioritized, wmf-config and puppet should be pri...
[22:31:13] <wikibugs_>	 10Continuous-Integration-Config, 06Release-Engineering-Team: Priortize gerrit changesets that contain the comment of "SWAT" - https://phabricator.wikimedia.org/T162316#3159031 (10Zppix) @demon I will do that but i will still keep this idea floating about and see what others say
[22:43:01] <Zppix>	 RainbowSprinkles:  are you here still
[22:43:29] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[22:54:00] <milimetric>	 so, RainbowSprinkles, I mistakenly thought Extension:Dashiki would be deployed at the same time as mediawiki-config, and dependent changes were merged for both
[22:54:28] <milimetric>	 RainbowSprinkles: or anyone else: any idea when Extension:Dashiki will be deployed?  Should I deploy it myself (I have rights now)
[22:54:50] <greg-g>	 not right now, no, it's 7 minutes until SWAT, and then it's end of day
[22:54:57] <RainbowSprinkles>	 The code has been live for...weeks?
[22:55:01] <RainbowSprinkles>	 We started branching awhile ago
[22:57:10] <RainbowSprinkles>	 demon@tin /srv/mediawiki-staging/php-1.29.0-wmf.19/extensions/Dashiki ((5176684...))$
[22:57:12] <RainbowSprinkles>	 What's missing?
[22:57:56] <milimetric>	 RainbowSprinkles: I'm not sure but all Config:Dashiki: pages show up as missing now: https://meta.wikimedia.org/wiki/Config:Dashiki:SimpleRequestBreakdowns
[22:58:01] <milimetric>	 but I see them in the database
[22:58:10] <milimetric>	 I'll check the last sha
[22:58:27] <RainbowSprinkles>	 I'm willing to bet it's a namespace issue, one sec
[22:58:54] <RainbowSprinkles>	 Hmm, or not
[22:58:58] <milimetric>	 RainbowSprinkles: 6ff5c65 is the last I have
[22:59:13] <milimetric>	 so I think I'm missing some manual step to update a pointer to that
[22:59:26] <wikibugs_>	 (03Draft2) 10Zppix: Add more jobs to test-prio [integration/config] - 10https://gerrit.wikimedia.org/r/346656
[22:59:27] <milimetric>	 the 517... one is a few revisions behind
[22:59:47] <milimetric>	 this is my fault, I merged another change for SpamBlacklist and that got deployed automatically
[22:59:50] <RainbowSprinkles>	 Ah, well the change 6ff5c655abe8ed203fdf28b929e251463fa06c98 was merged after the branch cut :)
[22:59:52] <milimetric>	 so I assumed they all work like that
[22:59:54] <greg-g>	 milimetric: 517 is what was branched for wmf.19 on tues
[23:00:09] <RainbowSprinkles>	 So we need to cherry-pick that forward
[23:00:09] <RainbowSprinkles>	 Doing
[23:00:15] <milimetric>	 right, I definitely need to understand this branching strategy better
[23:00:28] * milimetric reads docs
[23:00:36] <greg-g>	 milimetric: simple: we branch from master weekly on tuesday, that's it :)
[23:00:37] <RainbowSprinkles>	 milimetric: Branch every tuesday!
[23:00:44] <greg-g>	 jinx
[23:00:53] <milimetric>	 from master of all extensions?
[23:00:54] <RainbowSprinkles>	 Somewhere between 10am and noon Pacific time, depending on when someone gets on it :)
[23:01:00] <RainbowSprinkles>	 milimetric: Yep, from master
[23:01:01] <greg-g>	 milimetric: the ones we deploy
[23:01:02] <milimetric>	 oh, got it
[23:01:21] <RainbowSprinkles>	 (There's a few weird ones that don't branch from master, but you don't need to worry about those)
[23:01:29] <greg-g>	 details for us to worry about :)
[23:01:40] <greg-g>	 (btw, swat is right now ;) )
[23:01:42] <RainbowSprinkles>	 milimetric: wmf-config, on the other hand, is deployed as soon as someone merges it and pushes it out
[23:01:47] <RainbowSprinkles>	 (or at least that's how it's supposed to work)
[23:02:11] <milimetric>	 yep, that was my mistake: I merged a change for wmf-config and assumed that went out every tuesday too
[23:02:20] <RainbowSprinkles>	 Nope!
[23:02:22] <RainbowSprinkles>	 :)
[23:02:37] <milimetric>	 not a mistake in your docs, just probably a common thing lazy people like me do
[23:04:04] <RainbowSprinkles>	 Anyway, I'm porting that change from master to wmf.19, it'll be live shortly
[23:04:08] <Zppix>	 milimetric:  heck im in operations chan almost daily and still dont understand how things work :P
[23:04:13] <milimetric>	 thanks very much RainbowSprinkles 
[23:05:11] <milimetric>	 I'm super lucky you all are around, all the dashboards would've been out till next Tuesday otherwise
[23:05:30] <wikibugs_>	 10Continuous-Integration-Config, 06Release-Engineering-Team: Priortize gerrit changesets that contain the comment of "SWAT" - https://phabricator.wikimedia.org/T162316#3159171 (10demon) What if I put SWAT in a lot of my comments just to prioritize my commits? ;-)  Again: this is a bad idea.
[23:05:56] <RainbowSprinkles>	 milimetric: I'm always around (except when I'm not)
[23:06:02] <RainbowSprinkles>	 :)
[23:07:44] <wikibugs_>	 10Continuous-Integration-Config, 06Release-Engineering-Team: Priortize gerrit changesets that contain the comment of "SWAT" - https://phabricator.wikimedia.org/T162316#3159183 (10Zppix) @demon it would also look only in repos that are supposed to be swatted and only allow people with production access to trigg...
[23:13:11] <RainbowSprinkles>	 milimetric: Um, that made it worse....
[23:13:13] <RainbowSprinkles>	 MWUnknownContentModelException
[23:14:01] <milimetric>	 ok, so there must be some mistake
[23:14:06] <RainbowSprinkles>	 The content model 'JsonConfig.Dashiki' is not registered on this wiki.
[23:14:22] <milimetric>	 Reedy said I could move that into extension.json, which is the change he merged last night, one sec I'll link
[23:14:35] <milimetric>	 https://gerrit.wikimedia.org/r/#/c/344007/
[23:14:39] <RainbowSprinkles>	 That's what I backported to wmf.19
[23:15:12] <milimetric>	 right, and so with that I was supposed to be able to get rid of that config out of wmf-config
[23:15:17] <RainbowSprinkles>	 Hmmm
[23:15:27] <milimetric>	 apparently not :)
[23:15:27] <RainbowSprinkles>	 I think let's revert the config removal for now
[23:15:30] <RainbowSprinkles>	 Until we figure it out
[23:15:36] <milimetric>	 I'll do that now
[23:15:38] <Reedy>	 but some wikis are still on .18?
[23:16:11] <Reedy>	 meta is .19
[23:16:15] <Reedy>	 need chargr
[23:16:16] <RainbowSprinkles>	 Ahhh, I see it
[23:16:25] <RainbowSprinkles>	 Should be JsonConfig.Dashiki
[23:16:32] <RainbowSprinkles>	 Not JsonConfig[Dashiki]
[23:16:36] <RainbowSprinkles>	 You nested it
[23:16:55] <milimetric>	 aha, I see
[23:17:06] <milimetric>	 I have the revert change here: https://gerrit.wikimedia.org/r/#/c/346661/
[23:17:11] <RainbowSprinkles>	 Either way, reverting config is safest choice for now
[23:17:16] <RainbowSprinkles>	 :)
[23:17:42] <milimetric>	 ok, then I'll send another patch to fix the extension config
[23:17:56] <milimetric>	 the syntax for that seems like dark magic, is it detailed anywhere?
[23:18:10] <RainbowSprinkles>	 No idea! JsonConfig is kind of a black box to me
[23:18:20] <RainbowSprinkles>	 (other than the fact that it's always yelling at me in the error logs!)
[23:18:31] <Reedy>	  	"JsonConfigModels": {
[23:18:31] <Reedy>	  		"Dashiki": {		 		"Dashiki": {
[23:18:35] <Reedy>	 That didn't work
[23:19:01] <Reedy>	 Should both line 20 and 25 be JsonConfig.Dashiki?
[23:19:46] <Reedy>	 milimetric: You should've seen the problems with JsonConfig, Zero(Portal|Banner)... And that was with the author of the lot involved
[23:20:15] <RainbowSprinkles>	 I think so
[23:20:23] <RainbowSprinkles>	 milimetric: Ok we're back nowhttps://meta.wikimedia.org/wiki/Config:Dashiki:SimpleRequestBreakdowns
[23:20:30] <milimetric>	 oh, I don't know anything about Zero(Portal|Banner), this is my first attempt at this
[23:20:34] <RainbowSprinkles>	 nowhttps is not a protocol, keyboards are hard
[23:20:51] <milimetric>	 cool
[23:21:21] <milimetric>	 I'm not exactly clear on the proper syntax here, is it "JsonConfigModels": { "JsonConfig.Dashiki": ?
[23:21:31] <Reedy>	 FWIW. If I hadn't borked adding it to extension.json, and also reviewing your change (which you followed mine)...
[23:21:32] <RainbowSprinkles>	 I think so, based on what you had configured before
[23:21:46] <Reedy>	 The wmf-config one could've just stayed around till the updated version was on the active branch
[23:21:51] <Reedy>	 then be removed at a later date
[23:22:00] <milimetric>	 ok, I'll throw up a patch
[23:22:00] <RainbowSprinkles>	 $wgJsonConfigs['JsonConfig.Dashiki']
[23:22:32] <RainbowSprinkles>	 So yeah, we need to use 'JsonConfigs': 'JsonConfig.Dashiki' in the json
[23:22:54] <RainbowSprinkles>	 I have officially exceeded my quota of talking about JsonConfig today
[23:23:10] <milimetric>	 ok, so [ means nest and . means . and $wg means root
[23:23:22] <Reedy>	 No one actually understands it. Questionably even it's author
[23:23:23] <milimetric>	 thanks much RainbowSprinkles, that /sort of/ makes sense
[23:23:43] <RainbowSprinkles>	 Ehhhh, foo[bar] is an array where bar is a key in foo
[23:23:54] <RainbowSprinkles>	 foo[bar.baz] is an array foo where the key is bar.baz
[23:24:03] <RainbowSprinkles>	 Technically, that dot is just part of the key
[23:24:11] <milimetric>	 k, that change is here: https://gerrit.wikimedia.org/r/#/c/346663/1/extension.json
[23:24:16] <RainbowSprinkles>	 (unless it's being handled magically, which is ugly/not-intuitive)
[23:24:34] <RainbowSprinkles>	 That /looks/ correct, but I don't want to deploy again today :)
[23:24:38] <milimetric>	 nono
[23:24:57] <milimetric>	 never again, I'll just do it very slowly next time using my merge rights
[23:25:34] <Reedy>	 As I said above... Just merge that into master
[23:25:37] <Reedy>	 It'll ride the train
[23:25:50] <Reedy>	 Sometime after that's life, merge cleanup wmf-config
[23:25:53] <Reedy>	 *live
[23:25:55] <milimetric>	 yep
[23:26:02] <milimetric>	 and revert wmf-config if it goes bad again
[23:26:05] <milimetric>	 that was my plan
[23:54:38] <wikibugs>	 10Scap, 06Discovery, 10Wikimedia-Portals, 03Discovery-Portal-Sprint: Portals deployment failed - https://phabricator.wikimedia.org/T161832#3159323 (10MaxSem) @demon fixed this:  ```<RainbowSprinkles> I think we solved the portal sync issue yesterday. Issue in scap that had crept in (and why not everyone co...