[02:32:04] (03PS1) 10EddieGP: Add TerraCodes to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/351130 [02:37:28] !log on puppetmaster02 updated cherry pick for https://gerrit.wikimedia.org/r/#/c/347360 [02:37:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:49:07] !log on puppetmaster02 manually updating /etc/conftool/data-local [02:49:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:43:46] (03PS1) 10Thcipriani: Create SWAT gate and submit pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/351131 (https://phabricator.wikimedia.org/T160668) [03:56:12] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,BrowserTests build #378: 04FAILURE in 11 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/378/ [03:56:13] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #378: 04FAILURE in 12 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/378/ [03:56:20] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #378: 04FAILURE in 19 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/378/ [11:35:41] Where would I find stuff nowadays that once upon a time was available at https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/ ? [11:37:16] andre__: the default view has all jobs that releng still maintains I believe [11:37:39] Zppix: what is "the default view"? [11:38:05] andre__: look for a tab near the top of all the jobs that jenkins runs named default [11:38:38] I have no idea what that means [11:38:39] andre__: https://integration.wikimedia.org/ci/view/Default/ [11:38:43] Ah, thanks [11:38:47] No problem [11:39:15] andre__: sorry its hard for me to explain :/ [11:39:35] no worries :) and it's hard for me to understand because it's not my cup of tea :P [11:39:47] but that links is helpful, thanks [11:40:20] andre__: anytime have a great day :) [11:40:26] you too :) [11:41:15] If your not able to locate what you need let me know and ill look for myself andre [11:41:21] thx [11:41:33] No problem [11:48:59] Zppix those tests in the default are just some that are available. The rest are all in those tabs [11:58:10] Andre i was mistaken ^ [12:22:47] Yippee, build fixed! [12:22:47] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #384: 09FIXED in 46 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/384/ [13:46:02] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #384: 04FAILURE in 2 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/384/ [14:29:36] PROBLEM - Host deployment-phab02 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [16:00:58] 10Beta-Cluster-Infrastructure, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Revision-Scoring-As-A-Service, and 4 others: ORES errors on beta sites (dewp; zhwp) - https://phabricator.wikimedia.org/T163873#3225336 (10Etonkovidova) Re-checked the case when the beta feature "New filters for edit review" is on... [16:14:17] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 325 bytes in 0.015 second response time [16:15:19] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 325 bytes in 0.033 second response time [16:16:53] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T161733#3225365 (10mmodell) 05Open>03Resolved [16:29:20] 10Beta-Cluster-Infrastructure: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225396 (10vjudge404) [16:29:39] 10Beta-Cluster-Infrastructure: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225408 (10Paladox) p:05Triage>03High [16:30:00] 10Beta-Cluster-Infrastructure: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225396 (10Paladox) [16:30:32] 10Beta-Cluster-Infrastructure: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225396 (10Paladox) PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Wikipedia' not found on 'https://en.m.wikipedia.b... [16:37:59] thcipriani: betalabs shows "502 Bad Gateway nginx/1.11.3" for any redirection. Is it known? Just 10 min ago all was fine [16:38:29] 10Beta-Cluster-Infrastructure: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225458 (10EddieGP) p:05High>03Unbreak! The whole beta cluster is down. I know that's not production, but still something to look into asap. [16:38:50] etonkovidova: I hadn't seen that in a meeting, thank you for the ping, I'll investigate. [16:39:13] thcipriani: thanks! :)) [16:48:28] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225475 (10Luke081515) [16:48:35] Can somebody take a look at this ^? [16:48:48] nvm, I see it [16:48:51] sry [16:49:02] but now currently it is completly unreachable [16:49:42] Sagan: yes, it is being investigated right now. We (releng) were just in our weekly team meeting for the past 45 minutes (and it's the first thing most of us do Monday morning, before even checking bugmail) [16:50:02] greg-g: ok :) [16:50:57] hrm, I see a lot of Notice: Undefined property: stdClass::$ores_damaging_threshold in the hhvm logs still digging [16:56:17] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1343 bytes in 0.012 second response time [17:01:19] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46465 bytes in 2.207 second response time [17:01:29] (03CR) 10Jdlrobson: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [17:02:57] (03CR) 10jerkins-bot: [V: 04-1] Add experimental Node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [17:03:27] hrm, some weird stuff in the puppet run on deployment-cache-text04 it seems [17:10:37] well, getting closer, now just serving an error page, there was a problem with the varnish frontend [17:11:14] bingo! back [17:11:34] thcipriani: what was it? /me waits patiently [17:12:22] greg-g: digging for the root cause, the "fix" was that I had to restart the varnish frontend, digging through the puppet logs to see why that was necessary [17:14:15] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 34030 bytes in 1.291 second response time [17:15:16] * thcipriani updates task [17:15:23] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47022 bytes in 0.995 second response time [17:18:19] huh, well, ok then [17:27:09] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: 502 Bad Gateway on Beta Cluster - https://phabricator.wikimedia.org/T164203#3225595 (10thcipriani) 05Open>03Resolved a:03thcipriani For some reason https://gerrit.wikimedia.org/r/#/c/350505/3/modules/varnish/templates/text-common.inc.vcl.erb caus... [18:02:48] 10Continuous-Integration-Config, 06Reading-Web-Backlog: Proof of concept to generate coverage report of some Wikimedia mediawiki extensions - https://phabricator.wikimedia.org/T151333#3225685 (10Jdlrobson) [18:14:23] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Create "High Priority" gate-and-submit pipeline - https://phabricator.wikimedia.org/T160668#3225750 (10thcipriani) https://gerrit.wikimedia.org/r/#/c/351131/ [19:42:09] (03CR) 10Hashar: Add experimental Node selenium job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [19:50:48] (03PS1) 10Hashar: wikimedia/fundraising/crm add composer and npm [integration/config] - 10https://gerrit.wikimedia.org/r/351184 (https://phabricator.wikimedia.org/T163781) [19:52:17] (03PS2) 10Hashar: wikimedia/fundraising/crm add composer and npm [integration/config] - 10https://gerrit.wikimedia.org/r/351184 (https://phabricator.wikimedia.org/T163781) [19:53:33] (03CR) 10Hashar: [C: 032] "Patches got merged last week :]" [integration/config] - 10https://gerrit.wikimedia.org/r/351184 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [19:54:27] (03Merged) 10jenkins-bot: wikimedia/fundraising/crm add composer and npm [integration/config] - 10https://gerrit.wikimedia.org/r/351184 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [19:58:13] 10Continuous-Integration-Config, 05MW-1.27-release (WMF-deploy-2016-03-08_(1.27.0-wmf.16)), 13Patch-For-Review: Convert all repos to use npm Jenkins job with jsonlint and eslint - https://phabricator.wikimedia.org/T119973#3226073 (10hashar) [20:02:11] (03PS1) 10Hashar: Remove all jsonlint references [integration/config] - 10https://gerrit.wikimedia.org/r/351187 (https://phabricator.wikimedia.org/T119973) [20:04:00] (03PS1) 10Hashar: Remove jsonlint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/351188 (https://phabricator.wikimedia.org/T119973) [20:04:23] (03CR) 10Hashar: [C: 032] Remove all jsonlint references [integration/config] - 10https://gerrit.wikimedia.org/r/351187 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [20:04:29] (03CR) 10Hashar: [C: 032] Remove jsonlint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/351188 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [20:05:12] (03Merged) 10jenkins-bot: Remove jsonlint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/351188 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [20:05:23] hashar: \o/ [20:05:50] James_F: yeah finally the old jshint is gone \o/ [20:05:55] 10Continuous-Integration-Config, 05MW-1.27-release (WMF-deploy-2016-03-08_(1.27.0-wmf.16)), 13Patch-For-Review: Convert all repos to use npm Jenkins job with jsonlint and eslint - https://phabricator.wikimedia.org/T119973#3226080 (10hashar) 05Open>03Resolved The jobs taht ran an hardcoded/obsolete versio... [20:05:56] (03Merged) 10jenkins-bot: Remove all jsonlint references [integration/config] - 10https://gerrit.wikimedia.org/r/351187 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [20:06:02] And jsonlint [20:06:31] I wanted to finish it last week but went out of time [20:06:56] I like sprints that get rid of tech debt / legacy cruft :] [20:08:37] hashar: When are we going to get rid of the 'check' pipeline? :-) [20:09:53] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: [keyresult] Migrate as many misc CI jobs as possible to Nodepool - https://phabricator.wikimedia.org/T119140#3226099 (10hashar) [20:10:31] James_F: for mediawiki repos it was used mostly for jshint/jsonlint so yeah almost gone [20:10:52] hashar: Are there any repos with useful check pipes? [20:11:05] James_F: most probably not [20:11:22] Then we can probably kill it. Yay. [20:15:02] (03PS2) 10Hashar: [MultiBoilerplate] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/350911 (owner: 10Umherirrender) [20:15:04] (03PS3) 10Hashar: Add some composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:15:08] (03CR) 10Hashar: [C: 032] [MultiBoilerplate] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/350911 (owner: 10Umherirrender) [20:16:29] (03Merged) 10jenkins-bot: [MultiBoilerplate] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/350911 (owner: 10Umherirrender) [20:16:49] (03CR) 10jerkins-bot: [V: 04-1] Add some composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:24:19] (03CR) 10Hashar: [C: 032] "Very nice! Thanks for all the verifications :]" [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:25:54] (03Merged) 10jenkins-bot: Add some composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:26:28] !log nodepool: deleting alien instance ci-trusty-wikimedia-631443 4e66ad7e-b9d3-4af1-b559-3f54968d376e [20:26:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:28:43] (03CR) 10Hashar: [C: 032] "And I have reviewed / CR+2 all the pending patches." [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:28:50] hashar: alien instance? [20:29:18] Sagan: instances that are not known to Nodepool but present in the labs project [20:29:30] nodepool does not garbage collect them [20:31:52] (03PS2) 10Jdlrobson: Add experimental Node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) [20:32:29] (03PS3) 10Jdlrobson: Add experimental Node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) [20:32:34] (03CR) 10Jdlrobson: "https://www.mediawiki.org/wiki/Selenium/Node.js/mediawiki-core-qunit-selenium-jessie_Jenkins_job doesn't document whether it works for ext" [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson) [20:39:10] !log Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ ) [20:39:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:39:23] (03CR) 10Hashar: "And I have updated the REL1_29 branches." [integration/config] - 10https://gerrit.wikimedia.org/r/350984 (owner: 10Umherirrender) [20:53:38] restbase-dev boxen are all sending warning for using network over 80% utilization, is something going on with their setup/implementation, or are they experiencing their typical traffic? [20:55:19] Project beta-scap-eqiad build #153451: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/153451/ [21:07:12] Yippee, build fixed! [21:07:13] Project beta-scap-eqiad build #153452: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/153452/ [21:22:13] PROBLEM - Free space - all mounts on castor is CRITICAL: CRITICAL: integration.castor.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.castor.diskspace._srv.byte_percentfree (<100.00%) [21:31:00] 10Deployment-Systems, 06Operations, 10Ops-Access-Requests: Enable keyholder for ORES deployments - https://phabricator.wikimedia.org/T163939#3226339 (10RobH) 05Open>03Resolved a:03RobH It appears this is merely an issue on how to use the software, but not an access request to the software itself. @ako... [22:45:29] 10MediaWiki-Codesniffer: Release Codesniffer v0.8.0 - https://phabricator.wikimedia.org/T154715#3226502 (10Samwilson) What else is blocking a new release? [22:47:03] 10MediaWiki-Codesniffer: Release Codesniffer v0.8.0 - https://phabricator.wikimedia.org/T154715#3226537 (10Samwilson) (Sorry, meant to add: it's [[ https://www.mediawiki.org/wiki/MediaWiki_Documentation_Day_2017 | Documentation Day! ]] later this week, and it might be cool to have a more-thorough CS out by then.)