[00:08:44] <Krenair>	 https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/14078/console
[00:08:53] <Krenair>	 what is this complaining about?
[00:09:18] <Krenair>	 there was an error but there also was no error?
[00:34:35] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[01:07:17] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:07:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-pdf01 - https://phabricator.wikimedia.org/T173552#3532711 (10Krenair)
[01:08:08] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-pdf01 - https://phabricator.wikimedia.org/T173552#3532726 (10Krenair) See also T120165
[01:09:36] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:15:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-sentry01 - https://phabricator.wikimedia.org/T173554#3532743 (10Krenair)
[01:15:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-sentry01 - https://phabricator.wikimedia.org/T173554#3532743 (10Krenair) Caused by https://gerrit.wikimedia.org/r/#/c/371481/4
[01:28:38] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:35:35] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[01:39:38] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[01:43:08] <matt_flaschen>	 RoanKattouw, re your question about Vagrant, I'm using master (and if it happens on master, i haven't updated for enough yet), so I'm not sure, but what I would suggest is either:
[01:43:15] <matt_flaschen>	 1. Switch to master, so at least I can help troubleshoot better.
[01:43:43] <matt_flaschen>	 2. Workaround it by doing the git operation manually (sometimes aptitude or apt-get will let you resolve the issue in a smarter way than puppet).
[02:10:36] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:59:13] <wmf-insecte>	 Yippee, build fixed!
[03:59:14] <wmf-insecte>	 Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #489: 09FIXED in 3 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/489/
[04:20:38] <wmf-insecte>	 Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #489: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/489/
[05:31:34] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[05:49:48] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[06:11:35] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:29:50] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0]
[06:39:51] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[06:54:01] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%)
[07:02:34] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:04:03] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:42:35] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:23:39] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:28] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 50495 bytes in 0.852 second response time
[09:01:55] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3533051 (10Ladsgroup)
[09:09:50] <shinken-wm>	 RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0]
[09:40:09] <wikibugs>	 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3533113 (10zeljkofilipin) @Jrbranaa Sounds good to me! I would prefer //[[ https://en.wikipedia.org/wiki/Test_automation | test automation ]]// over //a...
[10:33:35] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[10:57:46] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-jobrunner02 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner02.diskspace.root.byte_percentfree (<100.00%)
[11:07:55] <wikibugs>	 (03PS2) 10Legoktm: Add "commit-message-validator install" command [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/372424
[11:08:05] <wikibugs>	 (03CR) 10Legoktm: Add "commit-message-validator install" command (031 comment) [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/372424 (owner: 10Legoktm)
[11:13:34] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[12:53:49] <wikibugs>	 10Beta-Cluster-Infrastructure: Disk full on deployment-jobrunner02 - https://phabricator.wikimedia.org/T173571#3533364 (10fgiunchedi)
[13:43:07] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[13:46:23] <wmf-insecte>	 Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #495: 04FAILURE in 2 min 22 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/495/
[14:04:35] <shinken-wm>	 PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[14:23:09] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
[14:42:30] <wikibugs>	 (03CR) 10WMDE-leszek: [C: 031] Get phan version from composer.json [integration/config] - 10https://gerrit.wikimedia.org/r/371097 (owner: 10Addshore)
[15:09:38] <shinken-wm>	 RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:18:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3533588 (10Jdlrobson)
[15:36:22] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3533588 (10JJMC89) The [[https://en.wikipedia.org/wiki/WP:PEIS|template include size is exceeded]].  ```lang=HTML <!--  NewPP limit rep...
[16:09:13] <addshore>	 thcipriani: just a thought, it might be a nice idea while pushing forward to have the ZUUL params for the images as optional things, I mean, IU could also mnt a volume in the right place pointing to the code I have locally and get the tests to run as in CI
[16:09:19] <addshore>	 #dundundun
[16:10:48] <thcipriani>	 addshore: seems possible, would be something implemented in the container entrypoint
[16:11:12] <thcipriani>	 if there are envvars set, use those, otherwise: <something else> for local testing
[16:12:14] <addshore>	 yup
[16:12:40] <addshore>	 interesting, the puppet one doesnt work for me locally as far as I can tell
[16:13:44] <addshore>	 when running build.sh "symlink ..\tools\common.yaml C:\Users\adam\dev\git\gerrit\integration\config\dockerfiles\contint-operations-puppet\cache\puppet\hieradata\labs\toolsbeta\common.yaml: A required privilege is not held by the client."
[16:15:30] <thcipriani>	 that's a weird error :)
[16:16:19] <thcipriani>	 I don't know where it would be attempting that symlink
[16:16:37] <addshore>	 I'll have a dig into it
[16:16:45] <addshore>	 a bit later! 
[16:16:52] <thcipriani>	 so the contint-operations-puppet/cache is what's left over after the dockerfile.build step
[16:17:27] <thcipriani>	 that is then copied (via dockerfile COPY) in the contint-operations-puppet/Dockerfile to the final container
[16:17:53] <addshore>	 [INFO] ...Creating build container operations-puppet-build from wmfreleng/operations-puppet-img
[16:18:05] <addshore>	 [INFO] ...Copying operations-puppet-build:/tmp/cache artifact to 'cache'
[16:18:06] <addshore>	 then fail
[16:18:15] <thcipriani>	 no other INFO lines?
[16:18:36] <addshore>	 there are some before those, but none between or after
[16:18:42] <thcipriani>	 huh
[16:19:06] <thcipriani>	 so it's failing during creating a docker image from wmfreleng/operations-puppet-img
[16:19:07] <addshore>	 https://www.irccloud.com/pastebin/9krNGFrv/
[16:19:27] <thcipriani>	 er, actually, I guess it's failing copying out those artifacts
[16:20:52] <thcipriani>	 the actual command being run is: docker cp "operations-puppet-build:/tmp/cache" cache
[16:21:25] <thcipriani>	 I would bet that there is some relative symlink in the cache and it's making windows freakout
[16:21:36] <addshore>	 that sounds very possible
[16:22:48] <addshore>	 i wonder if...... the data could be copied into a docker volume while passed between the containers? or there could just be a presistentish docker volume for it, rather than it coming out into the system os
[16:23:48] <thcipriani>	 the persistent volume thing makes permissions a bit tricky
[16:24:03] <addshore>	 hmm, how so?
[16:24:10] <shinken-wm>	 RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0]
[16:24:27] <thcipriani>	 hrm...trying to remember...I definitely fiddled with this for a long time
[16:24:49] <thcipriani>	 one problem is mounting volumes if the directory doesn't exist in the target, it's created as root, which is pain point 1
[16:25:50] <thcipriani>	 and everything has to be chowned in the build container before it is moved to the cache, the cp side-steps all the chowning
[16:26:18] <thcipriani>	 plus you have to setup a user for the build container, rather than just use root, so it adds extra steps :(
[16:26:29] <thcipriani>	 anyway, cp seemed a lot easier
[16:26:47] <thcipriani>	 but as I'm looking at it, it seems like there are a few related docker bugs for cp and windows machines :(
[16:27:00] <thcipriani>	 https://github.com/moby/moby/issues/7296
[16:28:05] <thcipriani>	 https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288 :(
[16:28:07] <addshore>	 oooh wait a sec
[16:28:48] <thcipriani>	 I wonder if we could just tar everything up in the build container and untar it on the target
[16:28:57] <thcipriani>	 #LazySolutions
[16:29:01] <addshore>	 the copy is done in build.sh?
[16:29:18] <thcipriani>	 yeah on line 47
[16:29:35] <addshore>	 I have had an issue with paths before, but using // instead of / always seemed to fix it and not break linux either
[16:29:37] <addshore>	 eg https://github.com/addshore/mediawiki-docker-dev/blob/master/up#L30
[16:29:54] <addshore>	 *re runs*
[16:30:12] <thcipriani>	 huh
[16:31:38] <addshore>	 or not, in this case
[16:33:52] <thcipriani>	 hrm, I'm kind of interested in a docker exec that calls tar and then untaring?
[16:34:45] <thcipriani>	 from the docker-cp man page:
[16:34:51] <thcipriani>	 > It is not possible to copy certain system files such as resources under /proc, /sys, /dev, tmpfs, and mounts created by the user in the container. However, you can still copy such files by manually running tar in docker exec.
[16:37:46] <addshore>	 hmmmm, I'll give that a shot in a bit!
[16:48:32] <thcipriani>	 addshore: does this command throw any errors: docker cp operations-puppet-build:/tmp/cache - > foo.tar
[16:48:49] <thcipriani>	 I guess if you pass '-' as an arg is streams a tar archive to stdout
[16:49:25] <addshore>	 running that has no output
[16:49:46] <addshore>	 foo.tar exists
[16:55:52] <thcipriani>	 nice, ok, that may be the ticket, I guess. just 1 extra step to untar in the container after COPY
[17:24:55] <shinken-wm>	 PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[17:30:16] <addshore>	 need to figure out how we can build multiple tags of an image too, so I could make a phan:0.8 and a pahn 0.8.5. I guess that needs changes to the build.sh too
[18:04:57] <shinken-wm>	 RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:25:52] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[18:52:48] <wikibugs>	 10Beta-Cluster-Infrastructure, 10User-Addshore: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976#3534063 (10Krinkle)
[19:04:34] <wikibugs>	 10Beta-Cluster-Infrastructure: Disk full on deployment-jobrunner02 - https://phabricator.wikimedia.org/T173571#3534091 (10Krenair) krenair@deployment-jobrunner02:~$ grep "Aug 10 " /var/log/mediawiki/jobrunner.log -c 7571 krenair@deployment-jobrunner02:~$ grep "Aug 11 " /var/log/mediawiki/jobrunner.log -c 2165281...
[19:05:53] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
[19:07:54] <wmf-insecte>	 Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #83: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/83/
[19:17:39] <wmf-insecte>	 Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #83: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/83/
[20:11:12] <wikibugs>	 10Scap (Scap3-Adoption-Phase1), 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#3534263 (10Ottomata) Not really.  You get a little bit of niceness in EventLogging because each event is an Event instance (a...
[20:17:50] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Add mediawiki/libs/XMPReader config [integration/config] - 10https://gerrit.wikimedia.org/r/372222 (owner: 10Legoktm)
[20:18:44] <wikibugs>	 (03Merged) 10jenkins-bot: Add mediawiki/libs/XMPReader config [integration/config] - 10https://gerrit.wikimedia.org/r/372222 (owner: 10Legoktm)
[20:21:47] <legoktm>	 !log deploying https://gerrit.wikimedia.org/r/372222
[20:21:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:50:11] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Rewrite commit message validator [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/367907 (owner: 10XZise)
[20:50:44] <wikibugs>	 (03Merged) 10jenkins-bot: Rewrite commit message validator [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/367907 (owner: 10XZise)
[20:50:53] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3475312 (10ksmith) There was a request in the 2017-08-16 Scrum of Scrums from the Scoring...
[21:14:55] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[21:36:48] <shinken-wm>	 PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:54:52] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:59:11] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, and 2 others: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3534478 (10demon) I thought about it yesterday. We should just bite the bullet and get git-lfs support for...
[21:59:40] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3534481 (10Dzahn) ^ Merged but had to revert it as well. After merging the service would fail to start (tested on gerrit2001 before doi...