[00:26:15] 10Scap, 10Deployments: Purge the php7 fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#3652714 (10demon) >>! In T146226#3651390, @mmodell wrote: > This is probably blocked by {T176370} Blocked, or declined as a result of? [03:26:40] 10Scap, 10Deployments: Purge the php7 fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#3652808 (10mmodell) maybe declined, I don't know? We might still want to deal with php7 opcache but I would guess that's a long way off. [04:02:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:42:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:12:46] 10Scap, 10Deployments: Purge the php7 fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#2654338 (10Joe) I don't see how this task can be blocked by to a switch to php7. I am also not completely sure it makes sense at all. I'm removing the relationship with the php7/hhvm... [05:13:15] 10Scap, 10Deployments: Purge the php7 fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#3652894 (10Joe) [05:16:31] 10Scap: Purge the hhvm fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#3652899 (10Legoktm) [05:16:58] 10Scap: Purge the hhvm fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#2654338 (10Legoktm) [05:17:04] wtf [05:17:16] I can't have it in both Deployments and Scap? [05:17:19] twentyafterfour: ^ [05:18:03] 10Scap: Purge the hhvm fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#2654338 (10Legoktm) Not sure what's wrong with the tags...but anyways, this task should still be about HHVM. *After* we've switched to PHP 7 it should be retitled if it's still relevant. [05:24:23] (03CR) 10Legoktm: [C: 032] "Deleted in jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/381812 (owner: 10Legoktm) [05:26:32] (03Merged) 10jenkins-bot: Drop mediawiki-core-phpcs-jessie, replaced by docker version [integration/config] - 10https://gerrit.wikimedia.org/r/381812 (owner: 10Legoktm) [05:29:12] (03CR) 10Legoktm: [C: 032] Add experimental "composer-package-php70-docker" job [integration/config] - 10https://gerrit.wikimedia.org/r/381378 (owner: 10Legoktm) [05:32:06] (03Merged) 10jenkins-bot: Add experimental "composer-package-php70-docker" job [integration/config] - 10https://gerrit.wikimedia.org/r/381378 (owner: 10Legoktm) [05:33:17] !log deploying https://gerrit.wikimedia.org/r/381812 https://gerrit.wikimedia.org/r/381378 [05:33:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:43:23] (03PS1) 10Legoktm: Include 'php7.0-dba' in php-mediawiki image [integration/config] - 10https://gerrit.wikimedia.org/r/381925 [05:46:35] (03CR) 10Legoktm: [C: 032] "Pushed, will rebuild composer-package image for this, none others should be affected." [integration/config] - 10https://gerrit.wikimedia.org/r/381925 (owner: 10Legoktm) [05:47:46] (03Merged) 10jenkins-bot: Include 'php7.0-dba' in php-mediawiki image [integration/config] - 10https://gerrit.wikimedia.org/r/381925 (owner: 10Legoktm) [05:49:12] (03PS1) 10Legoktm: Bump composer-package docker image to v2017.10.03.05.46 [integration/config] - 10https://gerrit.wikimedia.org/r/381927 [05:53:26] (03CR) 10Legoktm: [C: 032] Bump composer-package docker image to v2017.10.03.05.46 [integration/config] - 10https://gerrit.wikimedia.org/r/381927 (owner: 10Legoktm) [05:55:28] (03Merged) 10jenkins-bot: Bump composer-package docker image to v2017.10.03.05.46 [integration/config] - 10https://gerrit.wikimedia.org/r/381927 (owner: 10Legoktm) [05:58:26] 10Continuous-Integration-Infrastructure (shipyard): Make it easier to identify dependent images that need rebuilding when a base docker image changes - https://phabricator.wikimedia.org/T177272#3652932 (10Legoktm) [06:02:53] (03CR) 10jenkins-bot: Update PHP_CodeSniffer to 3.1.0 from 3.0.2 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/380310 (owner: 10Paladox) [06:28:24] (03PS1) 10Legoktm: Output `.env` in docker jobs for ease of debugging [integration/config] - 10https://gerrit.wikimedia.org/r/381930 [06:28:26] (03PS1) 10Legoktm: Add php-redis extension to php-mediawiki image [integration/config] - 10https://gerrit.wikimedia.org/r/381931 [06:28:28] (03PS1) 10Legoktm: Make sure `composer` is in the path in composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/381932 [06:30:50] (03CR) 10jerkins-bot: [V: 04-1] Output `.env` in docker jobs for ease of debugging [integration/config] - 10https://gerrit.wikimedia.org/r/381930 (owner: 10Legoktm) [06:31:37] (03CR) 10Legoktm: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/381930 (owner: 10Legoktm) [06:35:15] 10Scap: Purge the hhvm fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226#3652986 (10mmodell) @legoktm: scap is a subproject of deployments now. So it can't be in both the parent and the subproject. The subproject implies the parent. [06:39:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:40:03] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [06:51:06] (03CR) 10Legoktm: [C: 032] Output `.env` in docker jobs for ease of debugging [integration/config] - 10https://gerrit.wikimedia.org/r/381930 (owner: 10Legoktm) [06:53:17] (03Merged) 10jenkins-bot: Output `.env` in docker jobs for ease of debugging [integration/config] - 10https://gerrit.wikimedia.org/r/381930 (owner: 10Legoktm) [06:54:28] 10Continuous-Integration-Infrastructure (shipyard): Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3653004 (10Legoktm) [06:55:01] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:57:19] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:57:20] (03CR) 10Legoktm: [C: 032] Add php-redis extension to php-mediawiki image [integration/config] - 10https://gerrit.wikimedia.org/r/381931 (owner: 10Legoktm) [06:58:30] (03Merged) 10jenkins-bot: Add php-redis extension to php-mediawiki image [integration/config] - 10https://gerrit.wikimedia.org/r/381931 (owner: 10Legoktm) [07:00:01] (03PS2) 10Legoktm: Make sure `composer` is in the path in composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/381932 [07:00:12] (03CR) 10Legoktm: [C: 032] Make sure `composer` is in the path in composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/381932 (owner: 10Legoktm) [07:02:13] (03Merged) 10jenkins-bot: Make sure `composer` is in the path in composer-package image [integration/config] - 10https://gerrit.wikimedia.org/r/381932 (owner: 10Legoktm) [07:11:25] (03PS1) 10Legoktm: Move composer-package-php70-docker out of experimental [integration/config] - 10https://gerrit.wikimedia.org/r/381937 (https://phabricator.wikimedia.org/T144961) [07:13:34] (03CR) 10Legoktm: [C: 032] Move composer-package-php70-docker out of experimental [integration/config] - 10https://gerrit.wikimedia.org/r/381937 (https://phabricator.wikimedia.org/T144961) (owner: 10Legoktm) [07:15:02] (03Merged) 10jenkins-bot: Move composer-package-php70-docker out of experimental [integration/config] - 10https://gerrit.wikimedia.org/r/381937 (https://phabricator.wikimedia.org/T144961) (owner: 10Legoktm) [07:15:11] !log deploying https://gerrit.wikimedia.org/r/381937 [07:15:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:17:03] 10Continuous-Integration-Infrastructure, 10Tracking: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#3653034 (10Legoktm) [07:17:05] 10Continuous-Integration-Config, 10Patch-For-Review: Create composer-php70 job - https://phabricator.wikimedia.org/T144961#3653029 (10Legoktm) 05stalled>03Open a:03Legoktm All repositories except for avro-php pass the job (T177273), so it's no longer experimental. [07:32:52] 10MediaWiki-Codesniffer: Decide whether we want to move phpcs.xml to .phpcs.xml - https://phabricator.wikimedia.org/T177256#3653046 (10Addshore) Tastes good to me [07:33:34] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:38:52] 10Gerrit, 10Diffusion: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3653049 (10Pnorman) [07:39:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:40:18] 10Gerrit, 10Release-Engineering-Team, 10Diffusion: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3653061 (10Gehel) [08:13:35] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:45:11] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Migrating browser tests from Jenkins on beta - https://phabricator.wikimedia.org/T175179#3653118 (10zeljkofilipin) The only Node.js+Selenium job targeting beta cluster is [[ https://integration.wikime... [08:45:52] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Migrating browser tests from Jenkins on beta - https://phabricator.wikimedia.org/T175179#3653121 (10zeljkofilipin) Let me know as soon as you have any tests ready and we can work on making it happen. [08:46:09] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10User-zeljkofilipin: Migrating browser tests from Jenkins on beta - https://phabricator.wikimedia.org/T175179#3653122 (10zeljkofilipin) a:05zeljkofilipin>03None [08:49:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:02:03] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:09:32] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Beta-Cluster-reproducible, and 4 others: beta: Wikidata dispatchChanges.php causes a lot of "Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock" - https://phabricator.wikimedia.org/T175109#3653146 (10Ladsgr... [09:10:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Beta-Cluster-reproducible, and 4 others: beta: Wikidata dispatchChanges.php causes a lot of "Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock" - https://phabricator.wikimedia.org/T175109#3653147 (10Ladsgr... [09:10:29] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)), 10MW-1.31-release-notes (WMF-deploy-2017-10-03 (1.31.0-wmf.2)), and 2 others: mwext-ruby-jes... - https://phabricator.wikimedia.org/T164479#3653149 [09:23:40] 10Gerrit, 10Release-Engineering-Team, 10Diffusion: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3653159 (10Aklapper) ...and https://phabricator.wikimedia.org/r/branch/maps/kartotherian;refs/heads/master redirects to https://phabricator.wikimedia.org/diffusion/GMKT... [09:37:15] 10Gerrit, 10Release-Engineering-Team, 10Diffusion: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3653194 (10Pnorman) I recall it not working when I tested. https://gerrit.wikimedia.org/r/#/admin/projects/maps/maps-deploy,branches doesn't work, with the page https:... [09:49:54] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3653231 (10hashar) Sorry my bad. The saucelabs* Jenkins instances are for the daily... [09:55:55] zeljkof: i don't mind being bold and +2ing your change [09:56:02] need anyone else's review? [09:56:26] 10Gerrit, 10Release-Engineering-Team, 10Diffusion: Diffusion links are broken from within Gerrit - https://phabricator.wikimedia.org/T177277#3653049 (10Paladox) i carn't find the repo at all in diffusion for https://gerrit.wikimedia.org/r/#/admin/projects/maps/maps-deploy,branches, so i guess we need to impo... [09:56:31] phuedx: be bold! :) [09:56:48] I don't think so, I just needed somebody else to take a look, my javascript-fu is still white belt [09:57:00] thanks a lot for the review! [10:05:28] yw :) [10:14:31] zeljkof: now that it's merged, i've kicked of a relatedarticles build: https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-RelatedArticles-jessie/ [10:14:39] sorry, https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-RelatedArticles-jessie/66/ [10:18:34] phuedx: should be green now [10:18:56] zeljkof: https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-RelatedArticles-jessie/66/testReport/chrome/ReadMore2/_before_all__hook4/ [10:19:16] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [10:25:05] zeljkof: any idea what happened there? [10:35:56] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [10:38:36] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3653307 (10zeljkofilipin) It is running, but failing a lot: https://integration.wi... [10:38:55] phuedx: sorry, got distracted with other things, looking... [10:51:10] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3653317 (10zeljkofilipin) [[ https://integration.wikimedia.org/ci/view/Selenium/job... [10:54:06] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3653320 (10zeljkofilipin) 42 out of 69 failures (61%) are `The save has failed. (fa... [11:15:20] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:30:52] phuedx: I think there is something wrong with logging in at beta cluster [11:31:03] wikidata tests are also failing [11:37:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 404 Not Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 392 bytes in 0.003 second response time [11:50:22] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [12:19:54] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [12:22:39] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #544: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/544/ [12:29:57] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [12:38:35] 10Continuous-Integration-Infrastructure (shipyard), 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore, 10WMDE-QWERTY-Sprint-2017-09-19: Have CI run lintr for analytics/wmde/WDCM R files - https://phabricator.wikimedia.org/T176194#3616744 (10Addshore) 05Open>03Resolved Marking as resol... [12:38:54] (03CR) 10Hashar: "I guess the change is obsolete now?" [integration/config] - 10https://gerrit.wikimedia.org/r/371097 (https://phabricator.wikimedia.org/T174339) (owner: 10Addshore) [12:39:49] (03CR) 10Addshore: [C: 04-1] "Not really / yet." [integration/config] - 10https://gerrit.wikimedia.org/r/371097 (https://phabricator.wikimedia.org/T174339) (owner: 10Addshore) [12:42:14] (03CR) 10Hashar: [C: 031] Run WebPageTest tests from Asia to verify the new cache pop. [integration/config] - 10https://gerrit.wikimedia.org/r/362972 (https://phabricator.wikimedia.org/T168416) (owner: 10Phedenskog) [13:01:53] (03PS1) 10Aude: Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 [13:02:28] (03PS2) 10Aude: Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 [13:02:32] (03CR) 10Aude: [C: 032] Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 (owner: 10Aude) [13:04:29] Yippee, build fixed! [13:04:29] Project selenium-Math » chrome,beta,Linux,BrowserTests build #533: 09FIXED in 28 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/533/ [13:11:28] (03CR) 10Aude: Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 (owner: 10Aude) [13:11:30] (03CR) 10Aude: [C: 032] Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 (owner: 10Aude) [13:14:15] (03Merged) 10jenkins-bot: Update Wikidata [tools/release] - 10https://gerrit.wikimedia.org/r/381974 (owner: 10Aude) [13:21:25] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3653488 (10jkroll) @MaxSem: Jkroll [13:23:10] zeljkof: does check experimental run the node.js based browser tests for all extensions [13:23:14] or do i have to enable that? [13:23:47] phuedx: sorry, doing swat, but in short, yes, you have to enable it, take a look at relatedarticles setup in zuul config [13:23:56] zeljkof: ta [14:21:46] 10Release-Engineering-Team (Kanban), 10Analytics: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653631 (10fdans) [14:23:31] !log Gerrit: created analytics/wikistats2.git for fdans - T177288 [14:23:32] 10Release-Engineering-Team (Kanban), 10Analytics: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653651 (10fdans) [14:23:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:23:36] T177288: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288 [14:26:10] !log Created https://github.com/wikimedia/analytics-wikistats2 - T177288 [14:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:33:15] 10Release-Engineering-Team (Kanban), 10Analytics: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653672 (10hashar) I have copied the repository with: ``` git clone ssh://vcs@git-ssh.wikimedia.org/source/wikistats.git cd wikistats.git git remote add gerrit ssh://gerrit... [14:33:40] Yippee, build fixed! [14:33:40] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #535: 09FIXED in 1 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/535/ [14:34:28] hashar: could you look at https://gerrit.wikimedia.org/r/#/c/381520/ / https://gerrit.wikimedia.org/r/#/c/381723/ ? [14:34:40] 10Release-Engineering-Team (Kanban), 10Analytics: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653674 (10fdans) [14:35:17] 10Release-Engineering-Team (Kanban), 10Analytics: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653675 (10hashar) [14:39:18] (03PS1) 10Hashar: Add analytics/wikistats2 [integration/config] - 10https://gerrit.wikimedia.org/r/381991 (https://phabricator.wikimedia.org/T177288) [14:41:33] (03CR) 10Hashar: [C: 032] Add analytics/wikistats2 [integration/config] - 10https://gerrit.wikimedia.org/r/381991 (https://phabricator.wikimedia.org/T177288) (owner: 10Hashar) [14:42:51] (03Merged) 10jenkins-bot: Add analytics/wikistats2 [integration/config] - 10https://gerrit.wikimedia.org/r/381991 (https://phabricator.wikimedia.org/T177288) (owner: 10Hashar) [14:45:52] 10Release-Engineering-Team (Kanban), 10Analytics, 10Patch-For-Review: Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3653697 (10hashar) [15:08:14] (03Draft1) 10Hashar: docker: lame dependency tree [integration/config] - 10https://gerrit.wikimedia.org/r/381761 [15:12:44] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:20:46] (03PS2) 10Hashar: docker: lame dependency tree [integration/config] - 10https://gerrit.wikimedia.org/r/381761 [15:21:29] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 3 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3653824 (10Jdlrobson) I've kicked off a new bu... [15:31:29] polygerrit is going blue https://pbs.twimg.com/media/DLH9to8XUAEx-V8.jpg:large :). [15:33:59] 10Continuous-Integration-Infrastructure (shipyard): Make it easier to identify dependent images that need rebuilding when a base docker image changes - https://phabricator.wikimedia.org/T177272#3653867 (10hashar) I tackled that a bit yesterday and went with a very rough patch to build the dependency tree of imag... [15:34:04] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:34:19] 10Release-Engineering-Team (Kanban): Make it easier to identify dependent images that need rebuilding when a base docker image changes - https://phabricator.wikimedia.org/T177272#3653869 (10hashar) [15:49:37] (03CR) 10Hashar: [C: 032] Add Smarita to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381520 (owner: 10Gergő Tisza) [15:49:56] (03CR) 10Hashar: [C: 032] Add Neha Gupta to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381723 (owner: 10Gergő Tisza) [15:52:46] RECOVERY - Puppet errors on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:23] (03Merged) 10jenkins-bot: Add Smarita to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381520 (owner: 10Gergő Tisza) [15:54:00] (03PS2) 10Hashar: Add Neha Gupta to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381723 (owner: 10Gergő Tisza) [15:54:07] (03CR) 10Hashar: [C: 032] Add Neha Gupta to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381723 (owner: 10Gergő Tisza) [15:55:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3653921 (10Jdlrobson) @Peter it might be worth changing the article we test on if this cannot be resolved... [15:56:14] (03Merged) 10jenkins-bot: Add Neha Gupta to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/381723 (owner: 10Gergő Tisza) [15:57:09] tgr: done! sorry for the delay I was in an audio [16:08:35] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 3 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3653951 (10phuedx) Builds 66 and 67 (post merg... [16:09:03] thanks! [16:12:03] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking): Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3653955 (10Tgr) Do you mean something like https://en.wikipedia.org/w/index.php?title=Barack_Obama&action... [16:19:57] hmm it seems arcanist dosent work on windows again. [16:20:03] twentyafterfour ^^ [16:21:57] it keeps saying https://phabricator.wikimedia.org/P6069 [16:22:51] hasharAway: As you reviewed some other CI whitelist additions, might you have a look at https://gerrit.wikimedia.org/r/#/c/380989/ and https://gerrit.wikimedia.org/r/#/c/381639/ too? [16:23:15] (Once you're not "away" any more, that is.) [16:32:16] thcipriani|afk: when you have time (no hurry) another look at https://gerrit.wikimedia.org/r/#/c/381742/ would be welcomed! Please and thank you... [16:36:31] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Operations-Software-Development: New tool to track package updates/status for hosts and images (debmonitor) - https://phabricator.wikimedia.org/T167504#3653980 (10hashar) [16:46:15] gehel: sure I can take a look at that today :) [16:47:37] happends on the mac too [16:47:38] The current working directory is not part of a working copy for a supported version control system (Git, Subversion or Mercurial). [16:47:38] (Run with `--trace` for a full exception trace.) [16:49:46] (03PS3) 10Zoranzoki21: Whitelist user Jayprakash12345 on Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/381717 [16:50:11] i filled https://phabricator.wikimedia.org/T177295 [16:55:36] never mind [17:05:04] 10RelEng-Archive-FY201718-Q1, 10Release, 10Train Deployments: 1.31.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T172806#3654070 (10matmarex) [17:17:23] paladox, please consider using markup (see https://www.mediawiki.org/wiki/Phabricator/Help#Formatting and check the task preview when creating tasks. It makes things way more readable. Thanks! [17:17:36] ok sorry. [17:34:35] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:37:50] Can someone fix https://phabricator.wikimedia.org/T176104 to not be a "BUG" so we can set it as a UBN and give it some story points? [17:43:57] James_F: fixed I think. [17:44:05] I just made priority/story points visible [17:44:13] I'm not sure how to not make the task a "BUG" [17:46:02] 10Release-Engineering-Team (Kanban): Make it easier to identify dependent images that need rebuilding when a base docker image changes - https://phabricator.wikimedia.org/T177272#3654279 (10Legoktm) That looks like a good start to me. I think I want to collapse the dependency tree a bit (merge php-mediawiki, com... [17:47:04] legoktm: Thanks. That's enough for now. :-) [17:49:46] thcipriani: what was the rationale for hardcoding specific versions of the docker images in jjb? I feel like it's getting annoying but I'm also not sure exactly why it was implemented [17:52:55] legoktm: initially I did it to avoid pushing latest == deploy. Also the "latest" tag doesn't really tell you a precise version so hard to tell if a host has the version you want since it will always have a "latest" version. [17:53:02] it's annoying [17:53:24] but that was my initial thinking [17:55:15] the problem of pushing latest == deploy is kind of a non-issue at this point. But the problem where you could have 2 agents each running tests with the "latest" version of an image and having that image be different could still be problematic. [17:55:52] hmm is there a way to make docker check to see if dockerhub has a more later version? [17:58:25] https://github.com/moby/moby/issues/34394 [17:58:49] they suggest a workaround of "docker pull myimage:tag && docker run myimage:tag" [17:59:52] but 99% of the time the docker pull would be a waste of time :/ [18:01:36] yeah, was hoping to find more info https://hub.docker.com/v2/repositories/wmfreleng/operations-puppet/ [18:01:41] maybe my problem is that this is automated enough for me :) $ ./bump.py composer-package should grab the most recent date tag from dockerhub and then edit jjb to update the docker image to that and push a commit to gerrit :) [18:02:24] thcipriani: you want https://hub.docker.com/v2/repositories/wmfreleng/operations-puppet/tags/ [18:02:33] I think [18:03:55] hrm, maybe [18:09:36] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:21:49] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 3 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3654498 (10Jdlrobson) New error is: ``` Error... [18:36:48] huh https://github.com/docker/distribution/issues/628 [18:49:11] (03PS1) 10Hashar: docker: build script learned --verbose [integration/config] - 10https://gerrit.wikimedia.org/r/382032 [19:10:56] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [19:15:25] 10MediaWiki-Codesniffer: How to handle return in __construct inside codesniffer - https://phabricator.wikimedia.org/T177323#3654793 (10Umherirrender) [19:15:57] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [19:18:26] (03PS4) 10Zoranzoki21: Whitelist user Jayprakash12345 on Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/381717 [19:20:59] (03PS5) 10Zoranzoki21: Whitelist user Jayprakash12345 on Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/381717 [19:42:02] (03PS16) 10Legoktm: Phabricator/harbormaster job templates [integration/config] - 10https://gerrit.wikimedia.org/r/295396 (https://phabricator.wikimedia.org/T130950) (owner: 1020after4) [19:47:50] 10Scap (Scap3-MediaWiki-MVP), 10releng-201617-q3, 10scap2: Bring co-master / fanout capabilities to scap3 deployments - https://phabricator.wikimedia.org/T121276#3654941 (10mmodell) [19:47:53] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10WorkType-NewFunctionality: Remove apache dependency from scap3 deployment host - https://phabricator.wikimedia.org/T116630#3654940 (10mmodell) 05Open>03declined [20:01:29] heya, what's the policy on granting access to deployment-prep? am I OK to do it for https://phabricator.wikimedia.org/T176637 ? [20:08:39] 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3655005 (10Dzahn) a:05demon>03Dzahn [20:13:26] MaxSem: policy is just case-by-case. Should be fine to get them access. [20:18:34] thanks [20:21:49] 10Release-Engineering-Team (Watching / External), 10Contributors-Team, 10MobileFrontend, 10Operations, and 2 others: Diff page consistently produces 503 on beta cluster on first visit - https://phabricator.wikimedia.org/T176637#3655031 (10MaxSem) Added you to the project, now you can ssh into hosts like `d... [21:03:33] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog: Determine status of fundraising-related branches and tags - https://phabricator.wikimedia.org/T177334#3655092 (10demon) [21:06:02] fyi some things might temporarily break as i'm merging through a version upgrade to the Elastica library. Untill patches are merged for vendor, CirrusSearch and Elastica things might not pass [21:07:39] fun [21:08:13] ya know we could just have a monorepo ... :P [21:10:23] it'd be fun :) [21:11:01] let's just use perforce like all the cool kids [21:13:03] I'll wait to see how gwicke likes mercurial :) [21:13:45] I used it before git, it has a nice design [21:14:11] built a wiki on top of it back in the day [21:15:00] don't love the idea of a monorepo, though [21:15:43] it sounds reasonable for a single org, but for a collection (however tightly coupled) of floss projects it seems odd [21:35:14] yeah [21:42:34] ok it's all merged through and unrelated patches are passing jenkins again [22:03:41] Project beta-code-update-eqiad build #175321: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/175321/ [22:13:41] Project beta-code-update-eqiad build #175322: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/175322/ [22:18:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Parser, 10Readers-Web-Backlog (Tracking), 10User-Jdlrobson: Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576#3655317 (10Jdlrobson) @tgr that works perfectly. I've added an inline comment to tell... [22:20:31] (03PS3) 10Hashar: docker: lame dependency tree [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T77272) [22:21:18] (03PS4) 10Hashar: docker: lame dependency tree [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T177272) [22:23:40] Project beta-code-update-eqiad build #175323: 04STILL FAILING in 39 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/175323/ [22:25:39] who's messing around on beta? [22:25:39] 22:23:12 error: Your local changes to the following files would be overwritten by checkout: [22:25:42] 22:23:12 Hooks.php [22:25:45] 22:23:12 Please commit your changes or stash them before you switch branches. [22:25:48] 22:23:12 Aborting [22:27:18] bearND: are you? [22:27:28] greg-g: hello [22:27:32] hola! [22:27:47] you don't happen to be messing with a hooks.php file on beta, do you? [22:28:17] greg-g: I'm trying to deploy MCS to beta cluster (tried several times) but it failed. [22:28:29] so no, not that file eh? [22:28:46] greg-g: but I don't think I've touched hooks.php. At least not intentionally. I only touch JS code [22:29:05] (I can't easily tell from the jenkins output which hooks.php file in the extensions/ subdir ) [22:30:52] hmm, git status in /srv/deployment/mobileapps/deploy looks a bit unexpected: [22:30:56] cap deploy -s (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) [22:30:56] scap/log/ [22:30:56] T164033: Test size of "Reading stripped" HTML vs non-stripped HTML - https://phabricator.wikimedia.org/T164033 [22:30:56] T168848: Bootstrap an initial version of the Page Summary API in MCS - https://phabricator.wikimedia.org/T168848 [22:30:56] T169277: Investigate missing page in specific "On this day" event - https://phabricator.wikimedia.org/T169277 [22:30:57] T169274: Expand "On this day" endpoint language support - https://phabricator.wikimedia.org/T169274 [22:30:57] T167921: Support Lazy loading of page content not needed for first paint - https://phabricator.wikimedia.org/T167921 [22:30:57] T174698: Parenthetical stripping is too aggressive - https://phabricator.wikimedia.org/T174698 [22:30:57] T174808: Add swagger spec for summary endpoint - https://phabricator.wikimedia.org/T174808 [22:30:58] T162179: Extract HTML Compatibility Layer from MCS Mobile Sections API - https://phabricator.wikimedia.org/T162179 [22:31:22] oh, shoot, I didn't want to have stashbot talk this much. sorry [22:31:25] :) [22:31:50] I see "cap deploy" but I ran "scap deploy" [22:32:22] going to remove that file [22:33:39] Project beta-code-update-eqiad build #175324: 04STILL FAILING in 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/175324/ [22:35:06] 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Make it easier to identify dependent images that need rebuilding when a base docker image changes - https://phabricator.wikimedia.org/T177272#3655422 (10hashar) > That looks like a good start to me. I think I want to collapse the dependency tree a bit (m... [22:36:36] greg-g: looks like someone put some logging info into Echo [22:37:08] (03CR) 10Hashar: "That is really rough. I have wasted my time trying different recursion algo, but in the end the good old one based on a depth variable wor" [integration/config] - 10https://gerrit.wikimedia.org/r/381761 (https://phabricator.wikimedia.org/T177272) (owner: 10Hashar) [22:38:00] !log git stash /srv/mediawiki-staging/php-master/extensions/Echo to fix beta-code-update-eqiad [22:38:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:40:10] thcipriani: do you think I could try again to deploy mobileapps on beta cluster? [22:40:47] thanks thcipriani re echo [22:41:10] bearND: yeah, I'm looking at your deployment logs now, not real clear on what's failing :\ [22:41:29] try it again I'll try to catch what's happening... [22:41:59] ok [22:42:46] thcipriani: "1 of 1 default targets failed, exceeding limit" [22:42:53] yeap, I saw [22:43:36] Yippee, build fixed! [22:43:36] Project beta-code-update-eqiad build #175325: 09FIXED in 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/175325/ [22:43:56] bearND: digging now [22:45:06] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:50:50] bearND: I see the problem, new code on beta :( [22:51:05] thcipriani: ? [22:51:15] new scap code broken :( [22:51:20] thcipriani: I guess I'll roll back, right? [22:51:53] bearND: will take me a few [22:52:01] ah, i see. Luckily this worked on prod earlier ;) [22:54:47] yeah, beta gets scap in the same way it gets mediawiki code, on merge (basically) [22:55:01] continuously integrate all the things [22:55:33] good to know [23:20:04] 10Gitblit-Deprecate, 10Cleanup, 10Project-Admins: Archive #Gitblit-Deprecate - https://phabricator.wikimedia.org/T138986#3655674 (10mmodell) [23:20:05] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [23:20:06] 10Gitblit-Deprecate, 10Diffusion: Redirect git.wikimedia.org HEAD URLs to Diffusion - https://phabricator.wikimedia.org/T141965#3655672 (10mmodell) 05Open>03declined I don't actually see `tags/master/` anywhere on the page https://www.mediawiki.org/wiki/Extension:EventLogging (I opened view source and did... [23:27:02] bearND|afk: whenever you're back around, once puppet runs around beta you should be fine to deploy there, change reverted for the time being. [23:28:36] thcipriani: cool, thanks. How will I know when puppet is done? [23:30:43] bearND: I ran it on both tin and mcs01 which is (I think) the only target you're deploying to, so it should be done now [23:31:28] thcipriani: ah, ok. will try in a sec [23:33:18] thcipriani: yeah. It worked! :) [23:33:28] bearND: glad to hear it :)