[00:09:26] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [00:26:03] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10Deployments, 10WorkType-NewFunctionality: Scap3 submodule space issues - https://phabricator.wikimedia.org/T137124#3678451 (10mmodell) @halfak: That one is also on my radar and it's related to this work. [00:29:45] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3475312 (10mmodell) Relatedly, phabricator had some work done on git-lfs support upstream, however, it seems to be undocume... [00:30:28] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:04:05] Is this a harmless warning during scap, or would it explain why my files aren’t being copied? [01:04:10] > 01:03:14 Unable to find keyholder key for deploy_service [01:10:27] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:30:52] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [03:03:08] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3678578 (10Dzahn) [03:06:40] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3678591 (10Dzahn) [03:10:10] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3678593 (10Dzahn) 23:00 < MaxSem> uhh, why wold it depend on PHP? Yea, why? [03:12:12] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3678595 (10demon) PHP should probably be a Suggests, not a Depends. It's only used by the master and only used for linting. This is a packaging issue -- easily fixed. [03:49:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [04:11:00] Yippee, build fixed! [04:11:01] Project mediawiki-core-code-coverage build #3062: 09FIXED in 1 hr 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3062/ [05:11:50] PROBLEM - Free space - all mounts on deployment-sca03 is CRITICAL: CRITICAL: deployment-prep.deployment-sca03.diskspace._srv.byte_percentfree (<10.00%) [05:21:52] RECOVERY - Free space - all mounts on deployment-sca03 is OK: OK: All targets OK [06:44:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:49:01] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:12:42] (03CR) 10Legoktm: [C: 032] "This is pretty awesome. I'm going to merge this now, but ideally in the long run we can split up this sniff into some smaller ones. Would " [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383004 (owner: 10Umherirrender) [07:12:50] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3678740 (10MoritzMuehlenhoff) If the scap package itself doesn't use PHP, it should not depend/recommend/suggest it; if the person using scap needs PHP for some workflow it should be pulle... [07:13:47] (03Merged) 10jenkins-bot: Validate doc syntax [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383004 (owner: 10Umherirrender) [07:15:07] (03CR) 10Legoktm: [C: 032] "We already have similar sniffs here, so I think it's OK for now, but I do agree that in the long-run it should get split out into separate" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 (owner: 10Umherirrender) [07:15:13] (03CR) 10jerkins-bot: [V: 04-1] Add sniff for @params instead of @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 (owner: 10Umherirrender) [07:38:32] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 4703 bytes in 0.697 second response time [07:38:36] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 4827 bytes in 0.695 second response time [07:39:12] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 4703 bytes in 0.700 second response time [07:39:12] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 5254 bytes in 0.631 second response time [07:39:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [07:40:52] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 4704 bytes in 1.181 second response time [07:56:44] 10Continuous-Integration-Infrastructure, 10Nodepool: Increase Jenkins/Nodepool capacity - https://phabricator.wikimedia.org/T173047#3678780 (10hashar) @bd808 My sentence was poorly phrased and could have suggested I wanted to migrate Nodepool on a dedicated OpenStack cluster. That is not the case. The aim is... [08:14:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [08:40:04] 10MediaWiki-Codesniffer: Decide whether we want to move phpcs.xml to .phpcs.xml - https://phabricator.wikimedia.org/T177256#3678862 (10hashar) I love the upstream issue title: > Consider adding leading dot to config files for visibility When a file has a leading dot, it is hidden from ls which hmm... goes agai... [08:46:42] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#3678875 (10hashar) [08:46:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Secondary production Jenkins for CI - https://phabricator.wikimedia.org/T150771#3678873 (10hashar) 05Open>03stalled I would like to ultimately have the Jenkins in active/active. I miss time... [08:47:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Jenkins: Upgrade ci ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#3678877 (10hashar) p:05Triage>03Low [08:48:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: CI is down (jenkins) - https://phabricator.wikimedia.org/T177174#3678879 (10hashar) 05Open>03declined It happens from time to time. We would need a thread dump to figure out what is going exactly. [08:55:02] 10Continuous-Integration-Config: mw-ext-php70-phan-jessie complains about PHP temp directory not writable to composer - https://phabricator.wikimedia.org/T167969#3678885 (10hashar) Most probably the job should invoke the slave-script global-setup.sh which takes care of creating the tmp directories. mw-fetch-com... [09:23:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: CI is down (jenkins) - https://phabricator.wikimedia.org/T177174#3649345 (10Gehel) More than a thread dump, we would need GC logs. If you are interested in digging into this further, I would suggest the following config to the... [10:36:11] (03PS2) 10Hashar: Convert tabs to 4 spaces [integration/config] - 10https://gerrit.wikimedia.org/r/381286 [10:47:23] (03PS2) 10Hashar: docker: change tox example to use master [integration/config] - 10https://gerrit.wikimedia.org/r/381287 [10:49:23] 10Continuous-Integration-Infrastructure, 10Nodepool: Update Nodepool to catch up with upstream master branch - https://phabricator.wikimedia.org/T144601#3679112 (10hashar) [10:49:26] 10Continuous-Integration-Scaling, 10Release-Engineering-Team (Backlog), 10Operations, 10Nodepool, 10WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#3679110 (10hashar) 05Open>03declined We are most probably never goin... [10:49:32] 10Continuous-Integration-Infrastructure, 10Nodepool: Update Nodepool to catch up with upstream master branch - https://phabricator.wikimedia.org/T144601#2604952 (10hashar) 05Open>03declined We are most probably never going to upgrade Nodepool. [10:52:11] (03PS3) 10Hashar: docker: change tox example to use master [integration/config] - 10https://gerrit.wikimedia.org/r/381287 [10:52:25] (03CR) 10Hashar: [C: 032] docker: change tox example to use master [integration/config] - 10https://gerrit.wikimedia.org/r/381287 (owner: 10Hashar) [10:56:44] (03Merged) 10jenkins-bot: docker: change tox example to use master [integration/config] - 10https://gerrit.wikimedia.org/r/381287 (owner: 10Hashar) [11:17:02] (03Abandoned) 10Hashar: docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [12:10:20] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3679311 (10zeljkofilipin) @Jdlrobson I am work... [12:16:09] (03Restored) 10Hashar: docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [12:16:41] (03PS2) 10Hashar: docker: use nobody for operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [12:22:46] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #553: 04FAILURE in 46 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/553/ [12:23:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:27:29] 10Beta-Cluster-Infrastructure: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679383 (10zeljkofilipin) [12:30:44] 10Beta-Cluster-Infrastructure, 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679389 (10zeljkofilipin) [12:31:35] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3679392 (10zeljkofilipin) Blocked on {T178062}. [12:46:22] (03PS3) 10Hashar: docker: clone puppet.git in a different layer [integration/config] - 10https://gerrit.wikimedia.org/r/378911 [12:49:39] (03PS3) 10Hashar: docker: use nobody for operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [13:02:56] (03PS1) 10Gehel: Switch wikidata query service to use generic maven job template [integration/config] - 10https://gerrit.wikimedia.org/r/383830 [13:04:20] (03CR) 10jerkins-bot: [V: 04-1] Switch wikidata query service to use generic maven job template [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [13:04:43] Project selenium-Math » chrome,beta,Linux,BrowserTests build #542: 04FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/542/ [13:04:47] Project selenium-Math » firefox,beta,Linux,BrowserTests build #542: 04FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/542/ [13:19:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [13:51:01] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3679677 (10zeljkofilipin) Tried running tests on my machine. ``` ~/Documents/gerrit/m... [13:58:42] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679689 (10Reedy) Copy paste fail. Left it as primary, should've been preauth Reverted again in https://gerrit.wikimedia.org/r/#/c/383835/ and fixed [13:59:06] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679691 (10Reedy) [13:59:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [14:02:27] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679340 (10Mainframe98) The same error now appears for AntiSpoof: ``` MediaWiki internal error. Original exc... [14:03:45] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679710 (10Reedy) Fixes for AntiSpoof https://gerrit.wikimedia.org/r/383836 Campaigns https://gerrit.wikimedia... [14:05:46] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3679713 (10zeljkofilipin) I have manged to get tests running on my machine with this:... [14:09:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:12:48] zeljkof: want to merge https://gerrit.wikimedia.org/r/383836 to finish unbreaking beta? [14:13:38] Reedy: sure, +2d, thanks! :) [14:13:45] copy pasta fail :( [14:13:53] Reedy: happens :D [14:13:59] There's https://gerrit.wikimedia.org/r/383837 and https://gerrit.wikimedia.org/r/383838 [14:14:05] But I don't think they'll affect beta [14:19:05] zeljkof: ^ want to do those two too [14:19:09] seems they're all enabled on beta xD [14:19:31] They do: "got CampaignsSecondaryAuthenticationProvider" [14:19:32] Reedy: sure :D [14:20:08] silly things [14:20:16] +2d [14:29:10] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47368 bytes in 0.790 second response time [14:29:42] Yay, all fixed now [14:30:15] So just https://gerrit.wikimedia.org/r/#/c/383835/ to be re-reviewed and deployed [14:30:52] RECOVERY - App Server Main HTTP Response on deployment-mediawiki05 is OK: HTTP OK: HTTP/1.1 200 OK - 46802 bytes in 1.226 second response time [14:33:33] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46750 bytes in 0.730 second response time [14:33:33] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 35346 bytes in 0.668 second response time [14:34:11] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 46796 bytes in 3.544 second response time [14:36:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:42:36] PROBLEM - Puppet errors on integration-slave-docker-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:59:04] PROBLEM - Puppet errors on integration-slave-docker-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:59:47] (03CR) 10Addshore: docker: clone puppet.git in a different layer (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/378911 (owner: 10Hashar) [15:00:19] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679836 (10zeljkofilipin) a:03Reedy Assigning to @Reedy since he is fixing it. [15:00:36] 10Continuous-Integration-Infrastructure (shipyard): Provide git repositories on docker slaves to act as reference to git clone - https://phabricator.wikimedia.org/T178076#3679838 (10hashar) [15:02:32] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679851 (10zeljkofilipin) p:05Unbreak!>03Normal Beta is back, so it's no more UBN. [15:06:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:00] 10Beta-Cluster-Infrastructure, 10TitleBlacklist, 10MW-1.31-release-notes (WMF-deploy-2017-10-17 (1.31.0-wmf.4)), 10User-zeljkofilipin: Beta cluster down - https://phabricator.wikimedia.org/T178062#3679866 (10Reedy) 05Open>03Resolved It's all fixed now.. Just the revert of the revert to be merged, but t... [15:08:49] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Provide git repositories on docker slaves to act as reference to git clone - https://phabricator.wikimedia.org/T178076#3679869 (10hashar) [15:09:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Provide git repositories on docker slaves to act as reference to git clone - https://phabricator.wikimedia.org/T178076#3679838 (10hashar) p:05Triage>03Normal [15:11:14] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3679884 (10hashar) [15:11:47] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3679892 (10zeljkofilipin) Can not reproduce on... [15:11:49] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3432960 (10hashar) 05Open>03Resolved RESOLVED HUMOROUS :D //T31079// [15:22:36] RECOVERY - Puppet errors on integration-slave-docker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:11] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3679940 (10bd808) Scap [[https://phabricator.wikimedia.org/source/scap/browse/master/scap/tasks.py;96c10d0176573f19ce3beb86e24bba7ffdb29893$144-165|shells out to PHP]] for one particular w... [15:24:06] 10Release-Engineering-Team (Kanban), 10Readers-Web-Backlog, 10RelatedArticles, 10Browser-Tests, and 4 others: Automated browser tests cannot create pages on the Beta Cluster as anonymous user in RelatedArticles tests - https://phabricator.wikimedia.org/T176315#3679943 (10zeljkofilipin) Looks like the job f... [15:32:44] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3432960 (10zeljkofilipin) Wait, what!? No music in scap?! 😢 [15:34:04] RECOVERY - Puppet errors on integration-slave-docker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:19] 10Continuous-Integration-Config, 10Wikidata: Only run npm job on Jenkins for builds of data-values/value-view - https://phabricator.wikimedia.org/T178083#3680120 (10WMDE-leszek) [16:15:32] (03PS1) 10WMDE-leszek: Only run npm job for changes in data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/383872 (https://phabricator.wikimedia.org/T178083) [16:23:51] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3680162 (10thcipriani) 05Open>03Resolved [16:24:44] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3680177 (10thcipriani) 05Resolved>03Open Reopening until deployed. Landing the patch closed the task. [16:24:55] (03CR) 10Hashar: "I have pushed wmfreleng/operations-puppet:v2017.10.12.12.49 to docker hub. But haven't updated the jjb job." [integration/config] - 10https://gerrit.wikimedia.org/r/379762 (owner: 10Addshore) [16:26:01] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3680180 (10demon) 05Resolved>03Open [16:30:39] (03PS1) 10Hashar: operations-puppet-doc change rake target [integration/config] - 10https://gerrit.wikimedia.org/r/383875 [16:43:41] 10Scap, 10Deployments, 10Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3680227 (10thcipriani) 05Resolved>03Open Hiya @fgiunchedi could I trouble you to update the scap package to version 3.7.1-1? It's tagged in the repo and only contains a packaging fix for... [17:24:23] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10WorkType-NewFunctionality: Play elevator music while scap is running - https://phabricator.wikimedia.org/T170484#3680421 (10zeljkofilipin) YEAH!!!1!!1! [17:35:55] hasharAway: if you're back at any point, I'm still having trouble with JJB (https://gerrit.wikimedia.org/r/#/c/383830/). No emergency at all, but let me know if you have a some time to help me... [17:37:55] (03CR) 10Paladox: Switch wikidata query service to use generic maven job template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [17:42:01] (03PS2) 10Gehel: Switch wikidata query service to use generic maven job template [integration/config] - 10https://gerrit.wikimedia.org/r/383830 [17:42:16] (03CR) 10Gehel: Switch wikidata query service to use generic maven job template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [17:50:45] (03PS1) 10Zfilipin: WIP selenium-RelatedArticles-jessie needs passwords [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) [17:53:33] It looks like sessions are broken on beta.wp. Not blocking me personally. [17:53:37] (03PS2) 10Zfilipin: WIP selenium-RelatedArticles-jessie needs passwords [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) [17:54:17] (03CR) 10Zfilipin: "Tested here:" [integration/config] - 10https://gerrit.wikimedia.org/r/383887 (https://phabricator.wikimedia.org/T176315) (owner: 10Zfilipin) [17:54:47] (03CR) 10jerkins-bot: [V: 04-1] Switch wikidata query service to use generic maven job template [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [18:08:34] (03CR) 10Paladox: Switch wikidata query service to use generic maven job template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [18:08:42] (03CR) 10Thcipriani: "I think you're seeing test failures here because you've removed a job that is referenced by zuul/layout.yaml. Needs to be removed there as" [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [18:15:46] (03PS2) 10Umherirrender: Add sniff for @params instead of @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 [18:16:32] (03CR) 10Umherirrender: "PatchSet 2: Rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 (owner: 10Umherirrender) [18:19:06] (03CR) 10Umherirrender: "I am not working on split of the sniff." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383004 (owner: 10Umherirrender) [18:28:53] (03CR) 10Legoktm: [C: 032] Add sniff for @params instead of @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 (owner: 10Umherirrender) [18:30:07] (03Merged) 10jenkins-bot: Add sniff for @params instead of @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383168 (owner: 10Umherirrender) [18:31:59] (03CR) 10Legoktm: "Maybe instead of all punctuation just [] and {} to start with?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/383184 (owner: 10Umherirrender) [18:38:53] (03PS3) 10Gehel: Switch wikidata query service to use generic maven job template [integration/config] - 10https://gerrit.wikimedia.org/r/383830 [18:39:34] (03CR) 10Gehel: "Right! I mostly have not understood anything about zuul yet... But the suggestion seem coherent, let's try..." [integration/config] - 10https://gerrit.wikimedia.org/r/383830 (owner: 10Gehel) [18:40:01] paladox, thcipriani: thanks for the pointer (and for being patient with me :) [18:40:20] Your welcome :). [18:40:44] jjb is a twisty maze of passages all alike :) [18:48:48] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<22.22%) [19:03:49] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [19:59:16] 10Scap, 10Operations: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039#3680962 (10Dzahn) I stopped hhvm on gerrit2001 but i can't remove the package just yet because that requirement also means both hhvm and scap get removed if you attempt to remove hhvm. [20:04:24] the form for pastes looks so wierd now [20:04:25] it has a comment box [20:04:27] which wont be useful when creating pastes [20:10:38] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.4 - https://phabricator.wikimedia.org/T156120#3680998 (10Paladox) The 2.14 update succeeded again when migrating from 2.13 -> 2.14. [20:21:57] 10Continuous-Integration-Infrastructure, 10Zuul: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165#3681048 (10Paladox) [20:27:17] (03Draft1) 10Paladox: Update paramiko to 2.2 [integration/zuul] (upstream) - 10https://gerrit.wikimedia.org/r/383913 [20:27:19] (03Draft2) 10Paladox: Update paramiko to 2.2 [integration/zuul] (upstream) - 10https://gerrit.wikimedia.org/r/383913 [20:27:34] (03PS3) 10Paladox: Update paramiko to 2.2 [integration/zuul] (upstream) - 10https://gerrit.wikimedia.org/r/383913 [20:27:53] (03PS4) 10Paladox: Update paramiko to 2.2 [integration/zuul] (upstream) - 10https://gerrit.wikimedia.org/r/383913 (https://phabricator.wikimedia.org/T171165) [20:32:14] (03Draft1) 10Paladox: Update paramiko to 2.2 [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/383914 (https://phabricator.wikimedia.org/T171165) [20:32:16] (03PS2) 10Paladox: Update paramiko to 2.2 [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/383914 (https://phabricator.wikimedia.org/T171165) [20:52:55] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:05:43] Project beta-scap-eqiad build #177253: 04FAILURE in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/177253/ [21:06:07] 21:05:42 UnboundLocalError: local variable 'search_path' referenced before assignment [21:12:45] oh [21:13:05] ^ no_justification [21:15:41] Project beta-scap-eqiad build #177254: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/177254/ [21:16:29] Bleh [21:16:34] Stupid bug [21:16:49] 10Release-Engineering-Team (Kanban), 10Page-Previews, 10Readers-Web-Backlog, 10Patch-For-Review, 10User-zeljkofilipin: Run Popups Selenium tests daily targeting beta cluster - https://phabricator.wikimedia.org/T177924#3681298 (10Jdlrobson) The tests were wired up to run only for wikis where it's enabled... [21:16:55] no_justification: https://phabricator.wikimedia.org/D809 [21:16:57] I should make a default of [] in the __init__ [21:17:06] or that [21:17:16] Or D809 [21:17:16] D809: Fix local variable 'search_path' referenced before assignment - https://phabricator.wikimedia.org/D809 [21:17:18] Approved [21:18:00] landed [21:25:34] Project beta-scap-eqiad build #177255: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/177255/ [21:28:18] ^ same deal, phab jessie debs not run yet... [21:31:08] I *think* it should be fine next go-round [21:32:53] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [21:36:37] Yippee, build fixed! [21:36:37] Project beta-scap-eqiad build #177256: 09FIXED in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/177256/ [21:40:27] yippee [21:41:07] Should a new scap release be done to prevent the one know to be broken from being installed? [21:41:23] https://phabricator.wikimedia.org/T127762#3680227 [21:42:00] 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3681353 (10Dzahn) a:05Dzahn>03demon Back to Chad. Jenkins should be usable now. [21:47:41] which one? That new release only moves the current php requirement to suggests [21:47:52] doesn't include any other changes at this point. [21:48:31] completely a bugfix release for T178039 [21:48:32] T178039: scap should not pull in HHVM on stretch hosts using PHP7 - https://phabricator.wikimedia.org/T178039 [21:48:40] oh i see [21:48:52] but what's in beta tracks master [21:49:16] so it is a bit volatile [21:49:26] ...occasionally [21:50:03] but the volatility helps to insure stability in production (best case) [22:01:42] mutante ask microsoft if internet explorer 1.0 is the best browser. [22:01:43] woops [22:06:30] paladox: lol, yes, the assistant is so bad :) [22:07:16] yeh :) [22:50:18] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3681569 (10Paladox) This looks like a notedb specific feature, as when testing locally it wont work when not using notedb. Requires us to do T174034 [23:41:24] jenkins currently "23:39:20 ERROR: unknown environment 'testenv' [23:42:07] and Dereckson reported on another change it cant git clone [23:55:20] oops, jenkins just died with "no space left on device" [23:55:22] https://integration.wikimedia.org/ci/job/mwgate-composer-validate/754/console [23:56:35] heh [23:56:51] a bouqet of errors [23:57:28] /dev/mapper/vd-second--local--disk 22G 21G 0 100% /srv [23:58:56] 16G jenkins-workspace