[03:49:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [04:17:23] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #328: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/328/ [06:25:51] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #298: 04FAILURE in 1 hr 45 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/298/ [06:54:31] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:58:49] PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon [07:59:49] RECOVERY - git_daemon_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [08:51:32] Creating a new SSL key for deployment-copper. [08:51:39] error handling at its finest :} [08:55:59] !log Deleting deployment-copper Fails puppet due to broken OpenStack metadata http://169.254.169.254/openstack/2015-10-15/meta_data.json (fails) and no more needed (per elukey ) [08:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:57:33] PROBLEM - Host deployment-copper is DOWN: CRITICAL - Host Unreachable (10.68.18.164) [08:58:02] ^^^ I deleted it [09:30:17] !log Removing old kernel packages from deployment-pdf01 to free up disk space [09:30:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:39:44] !log upgrading puppet on deployment-pdf01 [09:39:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:03:51] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:23:52] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [10:33:23] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late via Yahoo servers - https://phabricator.wikimedia.org/T159960#3095070 (10Aklapper) [11:18:12] (03PS2) 10Hashar: Delete integration-composer-check-php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339392 (https://phabricator.wikimedia.org/T158845) [11:20:25] (03CR) 10Hashar: [C: 032] Delete integration-composer-check-php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339392 (https://phabricator.wikimedia.org/T158845) (owner: 10Hashar) [11:21:57] (03Merged) 10jenkins-bot: Delete integration-composer-check-php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339392 (https://phabricator.wikimedia.org/T158845) (owner: 10Hashar) [11:26:56] (03PS3) 10Hashar: Delete php53lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/339377 (https://phabricator.wikimedia.org/T158652) [11:33:50] (03PS4) 10Hashar: Delete php53lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/339377 (https://phabricator.wikimedia.org/T158652) [11:35:51] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095199 (10hashar) [11:36:28] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 05MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), 05MW-1.29-release-notes, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3095201 (10zeljkofilipin) [11:37:05] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3094966 (10hashar) Either the puppet master is down on puppetmaster.thumbor.eqiad.wmflabs or some firewall rule prevents... [11:37:23] (03CR) 10Hashar: [C: 032] Delete php53lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/339377 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [11:38:20] !log Deleting php53lint jobs. Replacing them with php55 equivalents [11:38:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:38:54] (03Merged) 10jenkins-bot: Delete php53lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/339377 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [11:51:24] (03PS2) 10Hashar: Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) [11:56:44] (03CR) 10Hashar: [C: 032] Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [11:57:26] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095232 (10Gilles) The puppet master appears to be shut down at the moment. The machine uses the role::puppetmaster::sta... [11:58:11] (03CR) 10jerkins-bot: [V: 04-1] Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [11:58:48] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095233 (10Gilles) Apache startup appears to be failing, it must be where the issue started: ``` gilles@puppetmaster:~$... [11:59:53] (03PS3) 10Hashar: Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) [12:00:36] (03CR) 10Hashar: [C: 032] Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [12:00:42] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095250 (10Gilles) ``` Mar 13 11:59:29 puppetmaster apache2[11688]: AH00526: Syntax error on line 8 of /etc/apache2/sites... [12:02:14] (03Merged) 10jenkins-bot: Delete composer*php53 [integration/config] - 10https://gerrit.wikimedia.org/r/339388 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [12:06:22] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095265 (10Gilles) https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslopensslconfcmd > Available in httpd 2.4.8 and... [12:07:18] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095267 (10Gilles) Seems related to {T159254} [12:09:28] (03PS1) 10Hashar: Remove mediawiki-phpunit-php53* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/342446 (https://phabricator.wikimedia.org/T158652) [12:12:15] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095304 (10Gilles) Updating apache2 as indicated in that ticket fixed the issue. With apache restored on puppetmaster.thu... [12:12:36] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095324 (10Gilles) [12:23:11] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:27:45] (03PS2) 10Hashar: Remove mediawiki-phpunit-php53* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/342446 (https://phabricator.wikimedia.org/T158652) [12:28:07] (03PS3) 10Hashar: Remove mediawiki-phpunit-php53* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/342446 (https://phabricator.wikimedia.org/T158652) [12:29:30] hashar: has releng officially depool'd all of precise? [12:30:09] (03CR) 10Hashar: [C: 032] Remove mediawiki-phpunit-php53* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/342446 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [12:30:54] Zppix: not yet [12:30:59] but will be shutdown by end of the week [12:31:05] and I guess I will delete the instances next week [12:31:11] ack, need any help? [12:33:57] (03Merged) 10jenkins-bot: Remove mediawiki-phpunit-php53* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/342446 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [12:55:21] (03PS1) 10Hashar: Replace php53 jobs with php55 equivalents [integration/config] - 10https://gerrit.wikimedia.org/r/342450 (https://phabricator.wikimedia.org/T158652) [12:57:16] (03PS2) 10Hashar: Replace php53 jobs with php55 equivalents [integration/config] - 10https://gerrit.wikimedia.org/r/342450 (https://phabricator.wikimedia.org/T158652) [13:12:35] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3095522 (10Paladox) It dosent seem the new private edits will make it inTo the 2.14 release. Also they will cut the branch on the 20 march, see https://groups.google.com/forum/m/#!topi... [13:15:07] (03PS3) 10Hashar: Replace php53 jobs with php55 equivalents [integration/config] - 10https://gerrit.wikimedia.org/r/342450 (https://phabricator.wikimedia.org/T158652) [13:16:19] (03CR) 10Hashar: [C: 032] Replace php53 jobs with php55 equivalents [integration/config] - 10https://gerrit.wikimedia.org/r/342450 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [13:17:51] (03Merged) 10jenkins-bot: Replace php53 jobs with php55 equivalents [integration/config] - 10https://gerrit.wikimedia.org/r/342450 (https://phabricator.wikimedia.org/T158652) (owner: 10Hashar) [13:19:20] !log Depooled Precise instances from Jenkins T158652 leaving the instances up for now. [13:19:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:19:24] T158652: Depool precise jenkins instances - https://phabricator.wikimedia.org/T158652 [14:30:18] 10Beta-Cluster-Infrastructure, 06Labs, 06Performance-Team, 10Thumbor: deployment-imagescaler01 can't reach puppetmaster.thumbor.eqiad.wmflabs - https://phabricator.wikimedia.org/T160324#3095671 (10hashar) Well done. Thanks :) [14:33:08] Project selenium-QuickSurveys » chrome,beta,Linux,BrowserTests build #340: 04FAILURE in 4 min 51 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/340/ [15:03:13] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#3095762 (10hashar) 05Open>03Resolved On the images themselves there is nothing else we can do. Part of the slowness wa... [15:04:02] (03PS1) 10Hashar: mwext-testextension-php55-composer to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342464 (https://phabricator.wikimedia.org/T137199) [15:05:19] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#3095766 (10hashar) [15:07:53] (03PS1) 10Hashar: rm mwext-testextension-{phpflavor}-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/342465 (https://phabricator.wikimedia.org/T137199) [15:09:15] (03CR) 10Hashar: [C: 032] mwext-testextension-php55-composer to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342464 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [15:10:00] (03CR) 10Hashar: [C: 032] rm mwext-testextension-{phpflavor}-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/342465 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [15:11:07] (03Merged) 10jenkins-bot: mwext-testextension-php55-composer to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342464 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [15:11:09] (03Merged) 10jenkins-bot: rm mwext-testextension-{phpflavor}-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/342465 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [15:12:13] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#3095815 (10hashar) [15:15:31] (03PS1) 10Hashar: mwext-testextension-php55 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342466 [15:15:47] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#3095824 (10hashar) [15:17:05] (03PS2) 10Hashar: mwext-testextension-php55 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342466 [15:30:59] (03CR) 10Hashar: [C: 032] mwext-testextension-php55 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342466 (owner: 10Hashar) [15:32:06] (03Merged) 10jenkins-bot: mwext-testextension-php55 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342466 (owner: 10Hashar) [15:41:40] 10Browser-Tests-Infrastructure, 07Easy, 05MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3095876 (10zeljkofilipin) Looks like there is one more file left: https://phabricator.wikimedia.or... [15:46:34] 10Continuous-Integration-Config, 07Puppet: also clone submodules in operations/puppet jobs - https://phabricator.wikimedia.org/T112670#1641972 (10jcrespo) I think this was done some time ago, but I could be wrong. Could you check its validity? [15:52:48] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #357: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/357/ [15:53:57] (03PS1) 10Hashar: mwext-testextension-hhvm* to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342471 (https://phabricator.wikimedia.org/T137199) [15:56:52] (03CR) 10Hashar: [C: 032] mwext-testextension-hhvm* to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342471 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [15:58:18] (03Merged) 10jenkins-bot: mwext-testextension-hhvm* to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342471 (https://phabricator.wikimedia.org/T137199) (owner: 10Hashar) [16:01:58] 10Continuous-Integration-Config, 07Puppet: also clone submodules in operations/puppet jobs - https://phabricator.wikimedia.org/T112670#3095957 (10hashar) Status: | CI job | Submodules |--|-- | operations-puppet-tox-jessie | NO | operations-puppet-rake-jessie | YES, recursive | operations-puppet-typos | NO Th... [16:13:04] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:19:23] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#3096009 (10hashar) [16:19:30] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 07WorkType-NewFunctionality: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#3096015 (10hashar) [16:19:33] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360632 (10hashar) 05Open>03Resolved a:03hashar At last. [16:23:51] (03PS1) 10Hashar: Remove dupe job from mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/342476 [16:24:08] (03Abandoned) 10Hashar: Remove dupe job from mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/342476 (owner: 10Hashar) [16:30:35] (03PS1) 10Hashar: mediawiki-phpunit-hhvm-composer to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342478 (https://phabricator.wikimedia.org/T135001) [16:31:18] (03Restored) 10Hashar: Remove dupe job from mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/342476 (owner: 10Hashar) [16:31:24] (03CR) 10Hashar: [C: 032] Remove dupe job from mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/342476 (owner: 10Hashar) [16:32:20] (03CR) 10Hashar: [C: 032] mediawiki-phpunit-hhvm-composer to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342478 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [16:32:33] (03Merged) 10jenkins-bot: Remove dupe job from mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/342476 (owner: 10Hashar) [16:35:22] (03CR) 10jerkins-bot: [V: 04-1] mediawiki-phpunit-hhvm-composer to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342478 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [16:35:34] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#3096041 (10hashar) [16:36:03] (03CR) 10Hashar: [C: 032] mediawiki-phpunit-hhvm-composer to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342478 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [16:38:08] (03Merged) 10jenkins-bot: mediawiki-phpunit-hhvm-composer to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342478 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [16:38:14] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#3096051 (10hashar) [16:38:43] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 07WorkType-NewFunctionality: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#3096055 (10hashar) [16:38:46] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#3096054 (10hashar) 05Open>03Resolved [16:42:55] PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon [16:43:53] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 07WorkType-NewFunctionality: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#3096057 (10hashar) In the interest of cleaning up the augeas stables our project is I a... [16:43:59] 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review: [keyresult] Migrate majority of CI jobs to Nodepool (part 2) - https://phabricator.wikimedia.org/T119138#3096061 (10hashar) [16:44:02] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 07WorkType-NewFunctionality: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#3096060 (10hashar) 05Open>03Resolved [16:44:19] 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate PHP extensions building jobs to Nodepool - https://phabricator.wikimedia.org/T134381#2263851 (10hashar) [16:44:22] 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review: [keyresult] Migrate majority of CI jobs to Nodepool (part 2) - https://phabricator.wikimedia.org/T119138#1819068 (10hashar) [16:45:05] hashar PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon [16:45:14] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3096072 (10EddieGP) p:05Triage>03Normal [16:47:55] RECOVERY - git_daemon_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [16:53:06] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:30] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: [keyresult] Migrate as many misc CI jobs as possible to Nodepool - https://phabricator.wikimedia.org/T119140#3096111 (10hashar) [16:58:06] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 13Patch-For-Review, 07WorkType-NewFunctionality: [keyresult] Migrate as many misc CI jobs as possible to Nodepool - https://phabricator.wikimedia.org/T119140#1819094 (10hashar) selenium-* jobs are now on their own labe... [17:04:58] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3096149 (10hashar) 05Open>03Resolved a:03hashar Must have been a transient issue. Seems the bandwidth is... [17:09:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Operations: Update npm to 3 or 4 - https://phabricator.wikimedia.org/T155488#3096168 (10hashar) [17:14:55] PROBLEM - git_daemon_running on contint2001 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon [17:15:55] RECOVERY - git_daemon_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon [17:19:18] 10Continuous-Integration-Config, 07Composer: integration/composer need a integration-composer-check-hhvm job - https://phabricator.wikimedia.org/T158845#3096227 (10hashar) [17:26:16] 10Gerrit, 06Developer-Relations, 10Developer-Wishlist (2017): Add a welcome bot to Gerrit for first time contributors - https://phabricator.wikimedia.org/T73357#3096256 (10Qgil) Can we have both? A bot welcome to everybody as soon as they show up plus a human welcome if/when an assigned human volunteers for... [17:29:46] 10Browser-Tests-Infrastructure, 07Easy, 05MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), 13Patch-For-Review: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#3096265 (10Harjotsingh) @zeljkofilipin The patch was reviewed but it's not getting verified. https:... [17:57:05] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 05MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), 05MW-1.29-release-notes, and 3 others: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3096352 (10zeljkofilipin) [19:38:49] 10Gerrit, 06Developer-Relations, 10Developer-Wishlist (2017): Add a welcome bot to Gerrit for first time contributors - https://phabricator.wikimedia.org/T73357#745463 (10Tgr) Past discussions about welcome messages on Wikipedia might hold some insight: * https://meta.wikimedia.org/wiki/Research:New_editor_w... [19:56:57] 10Gerrit, 10MediaWiki-extensions-General-or-Unknown, 06Repository-Admins, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3096625 (10SamanthaNguyen) [21:01:05] (03PS1) 10Hashar: Migrate mw-tools-scap-tox-doc-publish to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342508 (https://phabricator.wikimedia.org/T119140) [21:01:22] (03CR) 10Hashar: [C: 032] "Already deployed and tested." [integration/config] - 10https://gerrit.wikimedia.org/r/342508 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:02:22] (03Merged) 10jenkins-bot: Migrate mw-tools-scap-tox-doc-publish to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342508 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:06:57] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late via Yahoo servers - https://phabricator.wikimedia.org/T159960#3096772 (10Paladox) @valhallasw Hi, yahoo tells me that they doint throttle emails but they do have an internal thing that can block spec... [21:24:22] (03PS1) 10Hashar: Migrate apps-android-java-mwapi to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342515 (https://phabricator.wikimedia.org/T119140) [21:24:54] (03CR) 10Hashar: [C: 032] Migrate apps-android-java-mwapi to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342515 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:25:57] (03Merged) 10jenkins-bot: Migrate apps-android-java-mwapi to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/342515 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:30:54] hashar: is zuul on a general strike? It seems to be ignoring gerrit patches in the mediawiki/vagrant.git repo. example: https://gerrit.wikimedia.org/r/#/c/342514 [21:31:05] bd808: good holiday! :] [21:31:12] guess I broke it :( [21:31:15] let me debug ti [21:31:30] iohhhooooh [21:31:38] I'm "volunteering" and cleaning up a bunch of stuff that has been on my plate for a while :) [21:32:01] so yeah hmm [21:32:08] the job has a file filter to prevent it from running [21:32:25] it probably only trigger when a .rb or .pp file is modified [21:32:54] Generally zuul does go on strike... alot... [21:33:27] 10Gerrit, 06Release-Engineering-Team, 10Mail, 06Operations: Gerrit emails are showing up as being sent late via Yahoo servers - https://phabricator.wikimedia.org/T159960#3096821 (10Paladox) I have filled T160381 as wikimedia will need to fill out a form to get yahoo investigating it further. [21:33:47] hashar: is that new? I'm sure we've had lots of little fixes before that weren't .pp or .rb [21:34:32] na been the case for ages [21:34:38] I can v+2 and merge things. it won't hurt my feelings [21:34:40] will change it so that 'rake test' is always run [21:35:31] ah... wasn't switching everything to rake test new'ish? [21:36:58] hashar: what we need is a existing job (template) list [21:38:29] (03PS1) 10Hashar: MediaWiki Vagrant: always trigger rake job [integration/config] - 10https://gerrit.wikimedia.org/r/342520 [21:39:01] bd808: maybe yes [21:39:17] previously we used jobs like pplint-HEAD / erblint-HEAD [21:39:26] I moved all of them to a single entry point that just runs "rake test" [21:39:39] but the default job is "optimized" to only run on some file changes. [21:39:49] the change above create a more specific job that will always trigger will +2 it [21:39:56] I did the same for operations/puppet [21:40:04] (CI is a spaghetti mess) [21:40:21] (03CR) 10Hashar: [C: 032] MediaWiki Vagrant: always trigger rake job [integration/config] - 10https://gerrit.wikimedia.org/r/342520 (owner: 10Hashar) [21:40:31] that will also move mediawiki/vagrant out of mediawiki queue [21:40:35] which is probably a good thing [21:40:49] so your change will no more wait for mediawiki related changes to merge [21:41:18] (03Merged) 10jenkins-bot: MediaWiki Vagrant: always trigger rake job [integration/config] - 10https://gerrit.wikimedia.org/r/342520 (owner: 10Hashar) [21:42:01] bd808: it is running https://integration.wikimedia.org/ci/job/mediawiki-vagrant-rake-jessie/1/console [21:42:21] arggg got aborted [21:42:45] ah you mass rebased [21:43:49] bd808: once a change get merged, the gems will be saved in a central cache [21:43:56] and then the next builds will benefit from that cache [21:44:09] based on a rsync system you advised :] [21:44:31] hashar: I'm so smart I don't even remember that ;) [21:44:38] ;] [21:46:49] bd808: .rubocop.yml: - 'puppet/modules/mariadb/**/*' [21:46:49] :] [21:46:56] it is harmless really [21:46:58] ah [21:47:04] I'll follow up [21:47:16] or you can cr-2 to prevent it from lmerging :] [21:47:22] Gerrit will reject the submit action [21:48:02] 10Gerrit, 06Developer-Relations, 10Developer-Wishlist (2017): Add a welcome bot to Gerrit for first time contributors - https://phabricator.wikimedia.org/T73357#3096831 (10Aklapper) Uhm, thanks Tgr! That's research I was unaware of when I [[ https://phabricator.wikimedia.org/T85601#2164498 | edited the mw.or... [21:48:12] "WMF CI from the trenches" TM [21:49:16] most of the 'ci' in that repo is a waste of electricity [21:49:52] maybe we should try to boot the vagrant box [21:50:09] enable roles and run a bunch of smoke tests against it [21:50:28] as long as "we" doesn't include me ;) [21:50:33] obviously [21:50:43] Dan had grand plans at some point for end-to-end testing [21:50:57] manager delegation! wooooo [21:51:10] yeah well we found other plans for Dan :] [21:51:15] Gabriel just wants this all to be replaced with docker swarm or something [21:51:20] similar but on a larger scale! [21:51:30] yeah that is the plan [21:51:38] docker composer or something equivalen [21:51:40] I'll believe it when I see it [21:51:46] A) go buy more RAM [21:51:56] B) run docker compose or equivalent [21:52:10] C) grab a coffee while the internet is being download and dozens of containers spawn [21:52:16] mw-vagrant does a hell of a lot more than provision a couple of nodejs services [21:52:26] D) enjoy your own little full stack mediawiki install [21:52:52] Docker is the worst config management system I've seen in quite a long time [21:53:08] it's just a fancy little tarball geenerator [21:53:22] I think most of us pretty much agree that Docker is not going to be the solution [21:53:26] and honestly, not that fancy [21:53:48] as i got it Dockerfile might be attractive because it is well known and its it the hipster choice of the moment [21:53:58] but beside that, the rest doesnt look seducting :/ [21:54:32] its fine for stateless services that take little or no config [21:54:59] but in the MW ecosystem that's a pretty small slice [21:55:10] a friend of mine ended up rewriting his own docker like with plain bash and chef for provisioning [21:55:17] (and straight calls to LXC) [21:55:32] There are several cases where Docker is useful, for example to package complex dependencies for large Node and Ruby applications [21:55:38] in our case I dont know what we will end up with :/ it is being experimented [21:55:47] vagrant can manage LXC containers ;) we do it on Labs already [21:56:07] From a vendor point of view, to ship a Docker container = every user has a known config and a known dependencies set, that's for example what Discourse do, a forum in Ruby [21:56:12] hey maybe we will end up with vagrant =) [21:57:08] For PHP, deps are more or less sane with Composer, so there are less incentives to ship a container [21:57:10] my worry is how crappy the containerized OS really is [21:57:23] aye, that's a valid concern [21:57:46] well ruby has bundle, node has npm etc.. all of them come with solutions to freeze dependencies one way or another [21:57:47] hacked supervisor or no init process, minimal OS [21:57:49] Php tests dont really need much after all its not a hugely complicated test [21:58:01] this topic is a bit of a sore spot for me. I've spent 3+ years helping maintain mw-v and Gabriel has 1 commit to it. I doubt he uses it or understands the use cases it fufills [21:58:10] Zppix: go read mediawiki/core.git tests/phpunit/**.php files and come back :] [21:58:22] but he has tried a couple of times now to claim that replacing it is 'easy' [21:58:58] I will bring Vagrant to our closed-source-kabal-internal-meeting [21:59:43] but for now we are looking at the CI step where given a patch we want a system to build the container regardless of the provisioning system (Dockerfile, Vagrant or whatver) [21:59:47] meh. find the 'right' thing for packaging but don't get bullied into pretending that Docker will make it easy to ignore 3rd party use cases [22:00:23] well [22:00:29] we would first want to know about those use cases [22:00:33] get a feature set [22:00:40] hashar: i meant to process [22:00:42] and make the mediawiki for 3rd party a product [22:00:57] with hmm... a product manager? [22:01:27] bd808: I think the key point is to make it easy to install both a php app and a nodejs app (typically parsoid) with little tech knowledge [22:01:34] which imoh vagrant pretty much solve [22:01:56] then WMF also wants to solve the issue of deploying those services with bunch of dependencies [22:02:12] and we have some kubernetes cluster which we might well use for prod (maybe?) [22:02:29] then maybe vagrant could be use to provision container images that are then run on k8s [22:03:35] I thought k8s was more optimized for web related things [22:05:04] Zppix: Kubernetes is a general purpose container management system [22:05:32] and all the things we have are 'web' things (mediawiki, parsoid, restbase, citoid, mathoid, ...) [22:06:40] hashar: the PM for MediaWiki job is actually posted :) Now they just need to find a 3-legged unicorn to take the position [22:07:39] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3096862 (10demon) >>! In T156120#3095522, @Paladox wrote: > It dosent seem the new private edits will make it inTo the 2.14 release. > > Also they will cut the branch on the 20 march,... [22:07:41] wwhh? [22:07:51] hahh yeah [22:08:01] I got a few names in my mind :] [22:08:27] https://boards.greenhouse.io/wikimedia/jobs/613548?gh_src=1u385n1 [22:09:07] oh and you have a slot for an ops engineer for k8s great [22:09:57] yeah. so far the resumes for that one aren't too exciting [22:10:10] but we will keep looking [22:56:07] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3096905 (10Paladox) >>! In T156120#3096862, @demon wrote: >>>! In T156120#3095522, @Paladox wrote: >> It dosent seem the new private edits will make it inTo the 2.14 release. >> >> Als...