[00:00:41] * Reedy deleted some stuff [00:13:53] RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK [00:15:16] Reedy: thanks, it works again for me so far [00:15:29] ^ too :P [00:50:36] still (or again, dunno) has "00:46:32 ERROR: unknown environment 'testenv' [00:50:42] on another change though [00:50:52] waits until tomorrow [00:52:18] Different host? [00:55:43] hmm, i dont know how to tell [01:01:43] look at the console output [01:04:52] integration-slave-docker-1004 [01:05:21] docker vs jessie , but both 1004 [03:37:21] Project mediawiki-core-code-coverage build #3064: 04FAILURE in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3064/ [05:20:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [06:55:00] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:57:18] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:51:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:56:47] (03CR) 10Hashar: [C: 032] "Deployed yesterday and it generates the doc again https://doc.wikimedia.org/puppet/" [integration/config] - 10https://gerrit.wikimedia.org/r/383875 (owner: 10Hashar) [08:58:26] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3682226 (10Joe) a:03Joe [08:59:36] (03Merged) 10jenkins-bot: operations-puppet-doc change rake target [integration/config] - 10https://gerrit.wikimedia.org/r/383875 (owner: 10Hashar) [09:02:03] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:04:17] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3682232 (10Joe) Status update: I extracted the build script from `operations/docker-images/production-images` and it is able to build th... [09:09:28] (03PS3) 10Zfilipin: selenium-RelatedArticles-jessie Jenkins jobs needs a password [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) [09:09:40] (03CR) 10Zfilipin: [C: 032] selenium-RelatedArticles-jessie Jenkins jobs needs a password [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) (owner: 10Zfilipin) [09:11:22] (03Merged) 10jenkins-bot: selenium-RelatedArticles-jessie Jenkins jobs needs a password [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) (owner: 10Zfilipin) [09:12:57] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3682246 (10zeljkofilipin) Back to green! 💚 ht... [09:14:25] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3682264 (10zeljkofilipin) 05Open>03Resolved [09:23:54] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3682317 (10zeljkofilipin) p:05Normal>03Low Since this is Ruby framework that will s... [09:24:04] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3682319 (10zeljkofilipin) p:05Normal>03Low Since this is Ruby framework that wi... [09:25:00] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3268135 (10zeljkofilipin) p:05Normal>03High [09:26:17] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3682327 (10Joe) With a quick skim at the CI repo, the following things done there are not supported by the current build system: * Auto... [09:27:56] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), 10Patch-For-Review, 10User-zeljkofilipin: mwext-ruby-jessie Jenkins job runs all Ruby tas... - https://phabricator.wikimedia.org/T164479#3682346 [09:28:19] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), 10Patch-For-Review, 10User-zeljkofilipin: mwext-ruby-jessie Jenkins job runs all Ruby tas... - https://phabricator.wikimedia.org/T164479#3234955 [09:34:36] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), 10Patch-For-Review, 10User-zeljkofilipin: mwext-ruby-jessie Jenkins job runs all Ruby tas... - https://phabricator.wikimedia.org/T164479#3682374 [09:56:51] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), 10Patch-For-Review, 10User-zeljkofilipin: mwext-ruby-jessie Jenkins job runs all Ruby tas... - https://phabricator.wikimedia.org/T164479#3682398 [09:59:11] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3682416 (10zeljkofilipin) For Popups to have daily job running Selenium tests targeting... [10:05:25] (03PS1) 10Hashar: mediawiki/vendor switch to composer test [integration/config] - 10https://gerrit.wikimedia.org/r/384002 (https://phabricator.wikimedia.org/T135161) [10:07:44] (03CR) 10Hashar: [C: 032] mediawiki/vendor switch to composer test [integration/config] - 10https://gerrit.wikimedia.org/r/384002 (https://phabricator.wikimedia.org/T135161) (owner: 10Hashar) [10:09:56] (03Merged) 10jenkins-bot: mediawiki/vendor switch to composer test [integration/config] - 10https://gerrit.wikimedia.org/r/384002 (https://phabricator.wikimedia.org/T135161) (owner: 10Hashar) [10:37:09] (03PS1) 10Zfilipin: Run mwext-ruby-jessie Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384005 (https://phabricator.wikimedia.org/T177924) [10:38:43] (03CR) 10Zfilipin: [C: 032] Run mwext-ruby-jessie Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384005 (https://phabricator.wikimedia.org/T177924) (owner: 10Zfilipin) [10:40:25] (03Merged) 10jenkins-bot: Run mwext-ruby-jessie Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384005 (https://phabricator.wikimedia.org/T177924) (owner: 10Zfilipin) [10:43:40] !log Reloading Zuul to deploy 320f06501ea72cccdd2d012b06edf2dfb35cd986 [10:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:52:55] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Only run npm job on Jenkins for builds of data-values/value-view - https://phabricator.wikimedia.org/T178083#3682547 (10Addshore) [11:35:47] (03PS1) 10Hashar: docker jobs: stop passing HOME=/var/lib/jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/384013 [11:37:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 404 Not Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 392 bytes in 0.002 second response time [11:43:06] (03CR) 10Hashar: docker: clone puppet.git in a different layer (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [11:48:11] (03PS1) 10Zfilipin: Run mwext-ruby-jessie job for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384014 (https://phabricator.wikimedia.org/T164479) [11:48:49] (03PS4) 10Hashar: docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 [11:48:51] (03PS1) 10Hashar: docker: switch puppet image to wmfreleng/ci-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/384016 [11:49:07] (03CR) 10Hashar: docker: clone puppet.git in a different layer (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [11:50:05] (03PS2) 10Zfilipin: Run mwext-ruby-jessie job for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384014 (https://phabricator.wikimedia.org/T164479) [11:57:50] (03PS4) 10Hashar: docker: use nobody for operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [11:57:52] (03PS2) 10Hashar: docker jobs: stop passing HOME=/var/lib/jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/384013 [12:10:17] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3682703 (10zeljkofilipin) Now selenium-Popups-381212 is failing with `timed out after 3... [12:15:03] (03CR) 10Hashar: [C: 032] docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [12:15:07] (03CR) 10Hashar: [C: 032] docker: switch puppet image to wmfreleng/ci-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/384016 (owner: 10Hashar) [12:15:26] (03CR) 10Hashar: [C: 032] "I have uploaded the image / refreshed the job and that seems to work just fine now :)" [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [12:15:29] (03CR) 10Hashar: [C: 032] docker jobs: stop passing HOME=/var/lib/jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/384013 (owner: 10Hashar) [12:16:17] (03Merged) 10jenkins-bot: docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [12:16:19] (03Merged) 10jenkins-bot: docker: switch puppet image to wmfreleng/ci-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/384016 (owner: 10Hashar) [12:16:41] (03Merged) 10jenkins-bot: docker: use nobody for operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [12:16:48] (03Merged) 10jenkins-bot: docker jobs: stop passing HOME=/var/lib/jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/384013 (owner: 10Hashar) [12:21:12] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3682818 (10Niedzielski) Yay! Thanks @zeljkofil... [12:22:53] Yippee, build fixed! [12:22:53] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #554: 09FIXED in 52 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/554/ [12:26:03] (03CR) 10Zfilipin: [C: 04-1] Add Jenkins job for browser tests (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/375377 (https://phabricator.wikimedia.org/T160238) (owner: 10Jdlrobson) [12:31:04] (03PS1) 10Zfilipin: Popups Selenium tests should run daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/384030 (https://phabricator.wikimedia.org/T177924) [12:34:55] (03PS2) 10Zfilipin: Popups Selenium tests should run daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/384030 (https://phabricator.wikimedia.org/T177924) [12:35:16] (03PS1) 10Hashar: docker: force color in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/384033 [12:36:16] (03PS3) 10Zfilipin: Popups Selenium tests should run daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/384030 (https://phabricator.wikimedia.org/T177924) [12:37:18] (03CR) 10Zfilipin: [C: 04-1] Add Jenkins job for browser tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375377 (https://phabricator.wikimedia.org/T160238) (owner: 10Jdlrobson) [12:37:31] (03PS2) 10Hashar: docker: force color in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/384033 (https://phabricator.wikimedia.org/T175057) [12:39:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: operations-puppet-tests-docker console output lacks color - https://phabricator.wikimedia.org/T175057#3682894 (10hashar) a:03hashar [12:41:47] (03CR) 10Hashar: [C: 032] "Both have color output now :)" [integration/config] - 10https://gerrit.wikimedia.org/r/384033 (https://phabricator.wikimedia.org/T175057) (owner: 10Hashar) [12:43:05] (03Merged) 10jenkins-bot: docker: force color in operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/384033 (https://phabricator.wikimedia.org/T175057) (owner: 10Hashar) [12:43:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: operations-puppet-tests-docker console output lacks color - https://phabricator.wikimedia.org/T175057#3682911 (10hashar) 05Open>03Resolved Deployed and verified. Both tox and rspec have color output... [13:02:59] 10Scap, 10Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3682971 (10fgiunchedi) 05Open>03Resolved Yup all done @thcipriani ! [13:04:11] (03PS2) 10Umherirrender: Unwrap types in function docs from {} [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 [13:04:25] (03CR) 10Umherirrender: "Patch Set 2: Rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [13:15:55] (03PS3) 10Umherirrender: Unwrap types in function docs from {} [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 [13:16:22] (03CR) 10Umherirrender: "I would say" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [13:17:20] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/381286 (owner: 10Hashar) [13:20:30] (03PS1) 10Zfilipin: Run Selenium tests for Reading Web extension [integration/config] - 10https://gerrit.wikimedia.org/r/384041 (https://phabricator.wikimedia.org/T162256) [13:21:56] (03CR) 10Zfilipin: [C: 04-1] Add Jenkins job for browser tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375377 (https://phabricator.wikimedia.org/T160238) (owner: 10Jdlrobson) [13:22:16] (03Abandoned) 10Zfilipin: Add Jenkins job for browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/375377 (https://phabricator.wikimedia.org/T160238) (owner: 10Jdlrobson) [13:32:43] (03CR) 10Umherirrender: "In CentralAuth SpecialMultiLock::getGlobalUsers it is using" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [13:36:35] (03PS4) 10Zfilipin: Popups Selenium tests should run daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/384030 (https://phabricator.wikimedia.org/T177924) [13:38:35] (03CR) 10Zfilipin: [C: 032] Run Selenium tests for Reading Web extension [integration/config] - 10https://gerrit.wikimedia.org/r/384041 (https://phabricator.wikimedia.org/T162256) (owner: 10Zfilipin) [13:40:37] (03Merged) 10jenkins-bot: Run Selenium tests for Reading Web extension [integration/config] - 10https://gerrit.wikimedia.org/r/384041 (https://phabricator.wikimedia.org/T162256) (owner: 10Zfilipin) [13:41:41] !log Reloading Zuul to deploy b5b1dc2c580a465a3fd81f0dda6144dcb1e8f5f7 [13:41:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:46:26] (03CR) 10Zfilipin: "Tested:" [integration/config] - 10https://gerrit.wikimedia.org/r/384041 (https://phabricator.wikimedia.org/T162256) (owner: 10Zfilipin) [14:01:48] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), 10Patch-For-Review, 10User-zeljkofilipin: mwext-ruby-jessie Jenkins job runs all Ruby tas... - https://phabricator.wikimedia.org/T164479#3683107 [14:16:51] 14:13:52 ERROR: unknown environment 'testenv' [14:17:11] still same issue as last night. (example https://gerrit.wikimedia.org/r/#/c/380959/5) [14:53:53] 10Release-Engineering-Team, 10Commons, 10MediaWiki-Debug-Logger, 10monitoring, 10Performance: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3683287 (10Bawolff) > assuming that person was on IRC at the time :-) well its usually considered very bad form... [14:57:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [15:07:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [15:08:05] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:08:16] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3683325 (10zeljkofilipin) @Aleksey_WMDE @Tobi_WMDE_SW I am confused, why does sele... [15:10:10] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:10:27] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3683330 (10zeljkofilipin) Raw XML config files (easy to diff): - [[ https://integra... [15:13:42] 10Release-Engineering-Team, 10Commons, 10MediaWiki-Debug-Logger, 10monitoring, 10Performance: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3683336 (10jcrespo) > just that it should be someone's responsibility and yours was the closest team i could thi... [15:13:50] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:14:59] mutante: late response, that is weird debugging now [15:15:23] thcipriani: wait, joe just fixed it [15:15:36] mutante: ah, ok, cool :) [15:15:40] it was https://gerrit.wikimedia.org/r/384063 [15:15:47] and only affected admin module [15:15:50] it seems [15:16:29] ah ha! ok, makes sense [15:17:08] 10Release-Engineering-Team, 10Commons, 10MediaWiki-Debug-Logger, 10monitoring, 10Performance: High replication lag causing read only mode on commons - https://phabricator.wikimedia.org/T178094#3683342 (10Bawolff) One could fill a book with the areas of wikimedia that nobody or everybody jointly (which is... [15:17:26] 10Release-Engineering-Team, 10Operations, 10vm-requests, 10Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3683345 (10demon) [15:17:30] 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3683343 (10demon) 05Open>03Resolved The machine itself is up and running with... [15:21:28] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3683353 (10zeljkofilipin) I was able to reproduce the problem on my machine, target... [15:21:52] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:22:28] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:34:01] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:34:06] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:45:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T174359#3683406 (10demon) 05Open>03Resolved [16:02:28] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:11] Project mediawiki-core-code-coverage build #3065: 04STILL FAILING in 1 hr 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3065/ [16:34:08] !log ladsgroup@deployment-tin:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki (T177857) [16:34:12] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3683551 (10thcipriani) >>! In T177276#3682327, @Joe wrote: > * Tagging images with date instead of a semver versioning. I am dibated abo... [16:34:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:34:14] T177857: Test RDF exports in beta cluster - https://phabricator.wikimedia.org/T177857 [17:17:04] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3683713 (10Jdlrobson) 05Open>03declined Let's decline this then and go straight to... [17:37:08] hi, I'm is there a way to temporarily increase the max object size in gerrit? I'm trying to push a service deploy repo, and the initial commit is too big. Here's the error I'm getting: https://pastebin.com/AXwh1dsg [17:49:50] bmansurov hi, which repo? [17:50:13] objects can be increased though through the project config. [17:50:36] paladox: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/chromium-render/deploy [17:50:50] thanks [17:50:56] where it says "Maximum Git object size limit" [17:51:15] paladox: it doesn't yet, I've created a commit locally, but cannot push it to this repo [17:51:19] I get the error locally [17:51:26] Yep [17:51:58] it's because mediawiki repo is set at 20m [17:51:59] you need to rise it in https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/chromium-render/deploy [17:52:17] under "Maximum Git object size limit" [17:52:26] paladox: I see [17:52:28] thanks! [17:52:33] your welcome :) [18:02:35] unfortunately, changing the project config didn't work, which is expected according to https://gerrit-review.googlesource.com/Documentation/config-project-config.html#receive.maxObjectSizeLimit [18:04:09] (03CR) 10Gergő Tisza: mediawiki/vendor switch to composer test (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/384002 (https://phabricator.wikimedia.org/T135161) (owner: 10Hashar) [18:05:04] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3683836 (10thcipriani) 05Open>03Resolved a:03thcipriani scap 3.7.1-1 is now live. `php5-cli | hhvm | php-cli` is now part of `Suggests` ```[thcipriani@tin ~]$ apt-cache show scap Pa... [18:05:37] 10Gerrit, 10Readers-Web-Backlog: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10bmansurov) [18:10:19] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Platform-Team (MWPT-Q1-Jul-Sep-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3683867 (10CCicalese_WMF) [18:18:38] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q1-Jul-Sep-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3683896 (10CCicalese_WMF) [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:48:51] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683980 (10bmansurov) [18:49:20] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10bmansurov) [18:49:36] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10bmansurov) Pulled the task into the sprint for visibility. [18:51:23] 10Gerrit, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board, 10Unplanned-Sprint-Work: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683987 (10bmansurov) a:03bmansurov [19:56:33] (03PS4) 10Umherirrender: Unwrap types in function docs from {} [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 [19:57:12] (03CR) 10Umherirrender: "Patch Set 4: Removed () from regex, only matches [] and {}, the () is okay, see https://docs.phpdoc.org/references/phpdoc/types.html#array" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [20:03:35] 10MediaWiki-Codesniffer: Handle multiple array types in UnusedUseStatementSniff - https://phabricator.wikimedia.org/T178195#3684152 (10Umherirrender) [20:05:18] 10MediaWiki-Codesniffer: Handle multiple array types from phpdoc in FunctionCommentSniff - https://phabricator.wikimedia.org/T178196#3684164 (10Umherirrender) [20:05:42] (03CR) 10Umherirrender: "I created T178195 and T178196 for the update of the existing sniffs" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [20:06:10] 10MediaWiki-Codesniffer: Handle multiple array types from phpdoc in UnusedUseStatementSniff - https://phabricator.wikimedia.org/T178195#3684182 (10Umherirrender) [20:11:04] 10RelEng-Archive-FY201718-Q1, 10Release, 10Train Deployments: 1.30.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T170635#3684194 (10Etonkovidova) [20:42:27] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3684307 (10Krinkle) [21:15:13] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684356 (10Jdlrobson) [21:15:40] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10Jdlrobson) Joining late to the party, but if we solve this temporarily won't this still be a problem for future deploys? [21:31:58] 10MediaWiki-Codesniffer, 10MediaWiki-Platform-Team (MWPT-Q1-Jul-Sep-2017): Sniff that the class name matches the file name - https://phabricator.wikimedia.org/T174244#3684417 (10CCicalese_WMF) [21:40:17] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684445 (10Paladox) >>! In T178189#3684356, @Jdlrobson wrote: > Joining late to the party, but if we solve this temporarily won't this still be... [21:41:34] 10MediaWiki-Codesniffer, 10MediaWiki-Platform-Team (MWPT-Q1-Jul-Sep-2017), 10Patch-For-Review: Enforce one class per file in preparation for PSR-4 - https://phabricator.wikimedia.org/T173798#3684453 (10CCicalese_WMF) [22:09:27] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684525 (10bmansurov) Maybe we need to lift the limit for the first time and in subsequent pushes there won't be need because all the big files... [22:11:12] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684530 (10bmansurov) [22:11:33] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10bmansurov) [23:24:01] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3683839 (10Legoktm) Just to clarify, are you trying to commit a binary of chromium into git? I think we normally use git-fat for large files? (t... [23:37:37] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684726 (10demon) [[ http://tools.wmflabs.org/bash/quip/AU7VTzhg6snAnmqnK_pc | temporary solutions have a terrible habit of becoming permanent,...