[00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:33:17] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:12] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48094 bytes in 3.793 second response time [02:44:17] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:14] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48084 bytes in 5.283 second response time [04:04:40] To get sql prompt on deployment-tin, [04:05:02] I'm running: [04:05:03] kartik@deployment-tin:~$ mwscript sql.php --wiki=enwiki [04:05:03] [Mon Mar 19 04:04:00 2018] [hphp] [2813:7f229a244200:0:000001] [] [04:05:06] Fatal error: unexpected St13runtime_error: locale::facet::_S_create_c_locale name not valid [04:05:11] Weird? [04:06:42] OK. export LC_ALL=C [04:06:44] worked. [04:47:33] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [04:51:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [05:22:33] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [06:07:28] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:11:37] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:43:15] !log Add cx_version to cx_translations table (T190009) [07:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:43:20] T190009: Beta: Deploy "Add cx_version to cx_translations table" - https://phabricator.wikimedia.org/T190009 [08:49:06] 10Scap, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.7-1 - https://phabricator.wikimedia.org/T189306#4060337 (10fgiunchedi) >>! In T189306#4058067, @mmodell wrote: > @fgiunchedi I guess we should do this Monday? Yes Monday works, ping me on IRC when good to go [09:21:42] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [10:05:41] (03CR) 10Hashar: [C: 032] Add TemplateWizard dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/419967 (owner: 10Samwilson) [10:08:10] (03Merged) 10jenkins-bot: Add TemplateWizard dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/419967 (owner: 10Samwilson) [10:16:12] (03CR) 10Hashar: [C: 032] "It is not perfect but it passes and upload screenshots :]" [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) (owner: 10Hashar) [10:17:51] (03Merged) 10jenkins-bot: Migrate VisualEditor screenshot job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) (owner: 10Hashar) [10:18:08] (03CR) 10Hashar: [C: 032] "And it now passes on https://gerrit.wikimedia.org/r/#/c/417234/ :)" [integration/config] - 10https://gerrit.wikimedia.org/r/419967 (owner: 10Samwilson) [10:19:52] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4060587 (10hashar) [10:40:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060622 (10hashar) p:05Triage>03High [10:46:40] (03PS7) 10Addshore: Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [10:46:43] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [10:46:53] (03CR) 10Addshore: [C: 032] Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [10:48:10] (03Merged) 10jenkins-bot: Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [10:50:02] !log reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/ [10:50:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:54:08] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:02:42] Project beta-mediawiki-config-update-eqiad build #10492: 04FAILURE in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10492/ [11:03:55] 11:02:42 error: no such remote ref 2823a787e8fbeb4255273b27ce68652d8a2ee77a [11:03:55] 11:02:42 Fetched in submodule path 'portals', but it did not contain 2823a787e8fbeb4255273b27ce68652d8a2ee77a. Direct fetching of that commit failed. [11:03:55] 11:02:42 Build step 'Execute shell' marke [11:03:56] :/ [11:06:42] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<55.56%) [11:15:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060694 (10hashar) [11:45:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060769 (10hashar) [11:47:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060622 (10hashar) [11:52:00] (03PS1) 10Hashar: Use a full python for xgboost [integration/config] - 10https://gerrit.wikimedia.org/r/420311 (https://phabricator.wikimedia.org/T190032) [11:53:46] (03CR) 10Hashar: [C: 032] Use a full python for xgboost [integration/config] - 10https://gerrit.wikimedia.org/r/420311 (https://phabricator.wikimedia.org/T190032) (owner: 10Hashar) [11:55:06] (03Merged) 10jenkins-bot: Use a full python for xgboost [integration/config] - 10https://gerrit.wikimedia.org/r/420311 (https://phabricator.wikimedia.org/T190032) (owner: 10Hashar) [11:56:01] !log Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1 | https://gerrit.wikimedia.org/r/#/c/420311/ | T190032 [11:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:56:04] T190032: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032 [12:04:12] RECOVERY - Puppet errors on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:19:12] Project beta-mediawiki-config-update-eqiad build #10493: 04STILL FAILING in 4.6 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10493/ [12:19:55] 10Project-Admins, 10OCG-General, 10Proton: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312#4060845 (10Aklapper) @ovasileva: Any opinions / feedback on how to proceed here? [12:21:46] 10Continuous-Integration-Infrastructure, 10Lexicographical data, 10Wikidata: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4060862 (10WMDE-leszek) [12:41:16] 10Project-Admins, 10OCG-General, 10Proton: Decide what to do with open OCG tasks in Phabricator - https://phabricator.wikimedia.org/T161312#4060900 (10ovasileva) @Aklapper - we're currently identifying next steps on OCG tasks/book creation. We're planning on making an announcement this week after which we'l... [12:47:28] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search, 10Patch-For-Review: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060907 (10hashar) 00:09:17.674 XGBoostDFSuite: 00... [12:48:37] (03PS1) 10Hashar: Migrate xgboost maven job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420316 (https://phabricator.wikimedia.org/T190032) [12:48:47] (03CR) 10Hashar: [C: 032] Migrate xgboost maven job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420316 (https://phabricator.wikimedia.org/T190032) (owner: 10Hashar) [12:50:04] (03Merged) 10jenkins-bot: Migrate xgboost maven job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420316 (https://phabricator.wikimedia.org/T190032) (owner: 10Hashar) [12:51:51] Project beta-mediawiki-config-update-eqiad build #10494: 04STILL FAILING in 5.8 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10494/ [13:03:17] 10Continuous-Integration-Infrastructure, 10Lexicographical data, 10Wikidata: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4060931 (10hashar) I guess @zeljkofilipin can look into it and offer some recomm... [13:05:24] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4060938 (10hashar) [13:05:26] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery-Search, 10Patch-For-Review: Migrate CI job search-xgboost-maven to use a Docker container - https://phabricator.wikimedia.org/T190032#4060935 (10hashar) 05Open>03Resolved a:03hash... [13:05:28] 10Continuous-Integration-Infrastructure, 10Lexicographical data, 10Wikidata: CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions - https://phabricator.wikimedia.org/T190039#4060939 (10WMDE-leszek) [13:05:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4003449 (10hashar) [13:09:15] Project beta-mediawiki-config-update-eqiad build #10495: 04STILL FAILING in 5.1 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10495/ [13:19:30] Project beta-mediawiki-config-update-eqiad build #10496: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10496/ [13:39:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061015 (10hashar) p:05Triage>03High [13:45:10] 10Gerrit: Experiment switching gc back on in gerrit - https://phabricator.wikimedia.org/T190045#4061062 (10Paladox) [13:48:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061078 (10hashar) [13:51:41] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [13:59:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [14:00:15] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061113 (10hashar) [14:00:17] deployment-mediawiki04 is out of space ^^ [14:02:03] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061015 (10hashar) [14:04:42] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) WARN: deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<10.00%) [14:06:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061127 (10hashar) [14:07:40] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<55.56%) [14:09:47] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [14:10:49] 10Release-Engineering-Team, 10Operations, 10Puppet: puppetdb4: use postgres db backend in puppet-compiler - https://phabricator.wikimedia.org/T187258#4061133 (10herron) Ready to begin upgrading the puppet compiler now (in preparation for the prod puppetdb upgrade). Here's the process I have in mind: # disa... [14:12:45] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2018: Write Selenium tests in JavaScript/Node.js - https://phabricator.wikimedia.org/T190046#4061137 (10zeljkofilipin) p:05Triage>03Normal [14:13:36] hello! could someone please help me with the first step in https://phabricator.wikimedia.org/T187258#4061133 today? it’s a disable of the jenkins puppet compiler project worker compiler03.puppet3-diffs.eqiad.wmflabs [14:14:38] hashar ^^ [14:19:40] (03PS1) 10Hashar: Use proper container for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/420340 (https://phabricator.wikimedia.org/T190042) [14:22:43] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [14:24:14] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2018, 10User-zeljkofilipin: Write Selenium tests in JavaScript/Node.js - https://phabricator.wikimedia.org/T190046#4061204 (10zeljkofilipin) [14:25:15] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691#4061219 (10zeljkofilipin) a:05zeljkofilipin>03None [14:25:58] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188#3716038 (10zeljkofilipin) a:03zeljkofilipin [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:01:02] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061385 (10hashar) The job had the wrong container and... [15:02:13] Project mwext-phpunit-coverage-publish build #2307: 04FAILURE in 3.7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2307/ [15:04:52] (03PS2) 10Hashar: Use proper container for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/420340 (https://phabricator.wikimedia.org/T190042) [15:04:54] (03PS1) 10Hashar: wikidata/query/rdf site goal requires tests [integration/config] - 10https://gerrit.wikimedia.org/r/420359 (https://phabricator.wikimedia.org/T190042) [15:05:16] Yippee, build fixed! [15:05:16] Project mwext-phpunit-coverage-publish build #2308: 09FIXED in 2 min 48 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2308/ [15:06:56] (03CR) 10Hashar: [C: 032] Use proper container for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/420340 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:07:06] (03CR) 10Hashar: [C: 032] wikidata/query/rdf site goal requires tests [integration/config] - 10https://gerrit.wikimedia.org/r/420359 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:08:13] (03Merged) 10jenkins-bot: Use proper container for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/420340 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:08:17] (03Merged) 10jenkins-bot: wikidata/query/rdf site goal requires tests [integration/config] - 10https://gerrit.wikimedia.org/r/420359 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:08:39] (03PS1) 10Hashar: Migrate wikidata query rdf publish job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420362 (https://phabricator.wikimedia.org/T190042) [15:09:08] (03CR) 10Hashar: [C: 032] Migrate wikidata query rdf publish job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420362 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:10:29] Project beta-mediawiki-config-update-eqiad build #10497: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10497/ [15:10:33] (03Merged) 10jenkins-bot: Migrate wikidata query rdf publish job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/420362 (https://phabricator.wikimedia.org/T190042) (owner: 10Hashar) [15:11:23] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4061412 (10Paladox) With https://gerrit.wikimedia.org/r/#/c/410245/ this fixes stretch support, we need to re arange loa... [15:13:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4061417 (10hashar) [15:13:31] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042#4061415 (10hashar) 05Open>03Resolved a:03hashar [15:14:09] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4005023 (10hashar) [15:20:44] !log reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad [15:20:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:49:06] Yippee, build fixed! [15:49:07] Project beta-mediawiki-config-update-eqiad build #10498: 09FIXED in 7.4 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/10498/ [16:06:24] hi, pinging again about depooling compiler03 from jenkins [16:09:01] herron i think you can disable that project from jenkins ui [16:09:07] if your an admin [16:09:30] actually [16:09:38] herron you can switch of the slave it runs on [16:10:32] herron https://integration.wikimedia.org/ci/computer/compiler02.puppet3-diffs.eqiad.wmflabs/ [16:10:37] herron: you just need compiler03.puppet3-diffs.eqiad.wmflabs marked offline? [16:10:46] herron there's a disconnect button [16:10:50] bd808: yes please [16:11:04] paladox: hmm afact I’m not an admin [16:11:13] oh [16:11:58] !log jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258 [16:12:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:12:12] herron: should be done [16:12:20] bd808: awesome! thanks! [16:19:36] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4061727 (10Paladox) I've fixed upped https://gerrit.wikimedia.org/r/#/c/410245/ now, and now it works on stretch and je... [16:23:48] @seen Hauskatze [16:24:11] re: the ping for securedirexperiment: i have no idea why but: [16:24:12] The last Puppet run was at Mon Dec 11 18:20:54 UTC 2017 (141002 minutes ago). [16:24:15] so.. no idea [16:24:34] looking at the puppet failure though [16:25:10] Failed to parse template role/secureredir/nginx.conf.erb [16:25:18] Detail: undefined local variable or method `nginx_cfg' for # [16:27:17] so.. nginx config error in experimental setup .. [16:27:38] (03PS1) 10Addshore: Revert "Run node selenium tests as part of WikibaseLexeme CI build" [integration/config] - 10https://gerrit.wikimedia.org/r/420373 [16:27:41] (03PS2) 10Addshore: Revert "Run node selenium tests as part of WikibaseLexeme CI build" [integration/config] - 10https://gerrit.wikimedia.org/r/420373 [16:27:44] (03CR) 10Addshore: [C: 032] Revert "Run node selenium tests as part of WikibaseLexeme CI build" [integration/config] - 10https://gerrit.wikimedia.org/r/420373 (owner: 10Addshore) [16:27:48] gotta find out who was experimenting [16:28:54] (03Merged) 10jenkins-bot: Revert "Run node selenium tests as part of WikibaseLexeme CI build" [integration/config] - 10https://gerrit.wikimedia.org/r/420373 (owner: 10Addshore) [16:29:11] mutante seems that template does not exist [16:29:15] at least for that role [16:29:16] https://github.com/wikimedia/puppet/tree/production/modules/role/templates [16:29:40] !log reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/ [16:29:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:32:19] paladox: can we find out who was using the test setup most? [16:32:37] mutante not sure what you mean? [16:32:53] who created the instance and used it [16:33:01] you mean something like https://tools.wmflabs.org/openstack-browser/project/ [16:33:02] ? [16:33:03] who wrote the nginx config that is gone [16:33:05] mutante ^^ [16:33:59] mutante [16:34:01] aha [16:34:13] mutante it was cherry picked, here's the commit still pending review https://gerrit.wikimedia.org/r/#/c/317450/ [16:34:40] ok, see last comment on that [16:34:47] and ping the owner of it plz [16:35:00] also in meeting and phone just crashed [16:36:29] ok [16:36:31] done [16:37:03] ah found it [16:37:10] <%- nginx_cfg[domain].each do |path, target| -%> should be <%- @nginx_cfg[domain].each do |path, target| -%> [16:37:43] ok, cool, if it's a quick fix let's just do that, because it will be easier than all the other options [16:38:20] dont have time, so just up for quick fixes [16:40:11] mutante fixed it on https://gerrit.wikimedia.org/r/#/c/317450/ though it is untested so i could be wrong [16:44:51] paladox: well.. that's kind of the opposite of quick fix :) [16:45:11] oh [16:45:15] that's a new PS for the old unmerged change that i already said i'm not a good reviewer for [16:45:19] as opposed to a one-line fix [16:46:10] i cant help here then. i'll have to change my original answer if it's fixable to "not by me at least" [16:46:28] someone will have to cherry pick the new change onto the puppet master [16:46:35] (in deployment prep) [16:46:43] where i think the instance is hosted [16:48:03] here's a quicker fix.. ask the traffic team if they want to keep that instance, ask Alex Monk if he wants to keep it, if they both say no then remove it because it's called an experiment [16:48:24] that doesn't mean the gerrit change cant be useful [16:48:33] and might still be reviewed in the future [16:48:52] 10Continuous-Integration-Config: Wrong output for stylelint - https://phabricator.wikimedia.org/T190072#4061790 (10Daimona) [16:49:36] Krenair ^^ [16:49:39] i dont know who cherry-picks [16:49:45] for this project [16:50:52] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<55.56%) [16:56:31] greg-g: so when someone is asking for deployment access, I know we need to get release engineering sign off. Do you want us to just ping in here, or assign to someone specifically in your team, or assign a project tag? [16:56:48] (asking due to https://phabricator.wikimedia.org/T189414) [17:01:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery, 10Wikimedia-Portals: Migrate Jenkins job wikimedia-portals-build to Docker and to use an entry point (eg: npm builddeploy) - https://phabricator.wikimedia.org/T190073#4061823 (10hashar... [17:01:36] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4061842 (10hashar) [17:07:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Add blubber to docker integration agents - https://phabricator.wikimedia.org/T186548#4061855 (10dduvall) a:05dduvall>03thcipriani [17:19:01] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team: ores-beta grafana has been broken since January - https://phabricator.wikimedia.org/T190075#4061915 (10awight) [17:25:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [17:28:35] PROBLEM - Puppet errors on integration-cumin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:35:49] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [17:39:17] does anyone know if a `recheck` comment on Gerrit will restart the mwext-phpunit-coverage-patch job as well? [17:40:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4061966 (10Smalyshev) Seems to be still happening. See: https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer-jessie/9226/console [18:03:36] RECOVERY - Puppet errors on integration-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [18:09:08] 10Release-Engineering-Team (Kanban), 10Operations, 10Ops-Access-Requests, 10User-greg: Requesting deployment access for samwilson - https://phabricator.wikimedia.org/T189414#4062061 (10RobH) a:03greg @greg, I'm assigning this to you for your approval, as release engineering has to sign off on new deploy... [18:25:01] (03PS1) 10WMDE-leszek: Add experimental job for WikibaseLexeme to run node selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/420398 [18:28:29] 10Release-Engineering-Team (Kanban), 10Operations, 10Ops-Access-Requests, 10User-greg: Requesting deployment access for samwilson - https://phabricator.wikimedia.org/T189414#4062113 (10greg) a:05greg>03RobH Approved. [18:28:34] robh: the assigning to me works fine, I bet. [18:28:43] heh, cool [18:28:47] thx =] [18:31:10] (03CR) 10Addshore: [C: 032] Add experimental job for WikibaseLexeme to run node selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/420398 (owner: 10WMDE-leszek) [18:32:16] Project mwext-phpunit-coverage-publish build #2326: 04FAILURE in 3.3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2326/ [18:32:36] (03Merged) 10jenkins-bot: Add experimental job for WikibaseLexeme to run node selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/420398 (owner: 10WMDE-leszek) [18:33:51] Yippee, build fixed! [18:33:52] Project mwext-phpunit-coverage-publish build #2327: 09FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2327/ [18:33:59] !log reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/ [18:34:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:34:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.004 second response time [18:36:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4062127 (10greg) [18:49:40] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<30.00%) WARN: deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<10.00%) [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:16:35] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4062249 (10RobH) Chris swapped the bad disk out this AM: ``` Record: 6 Date/Time: 03/14/2018 14:35:50 So... [19:22:48] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<10.00%) [19:24:18] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4062290 (10awight) @mmodell Could you let me know, am I waiting for a beta scap upgrade, or is there anything I can continue checking? [19:29:41] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [19:32:51] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [19:35:03] (03CR) 10Chad: [C: 032] Remove ArticleCreationWorkflow, undeployed [tools/release] - 10https://gerrit.wikimedia.org/r/420136 (owner: 10MaxSem) [19:35:34] (03Merged) 10jenkins-bot: Remove ArticleCreationWorkflow, undeployed [tools/release] - 10https://gerrit.wikimedia.org/r/420136 (owner: 10MaxSem) [19:37:09] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4062317 (10awight) The catch seems to be that tin and deployment-tin are running jessie, so the git-lfs package isn't easily available. [19:37:52] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#3764041 (10Dzahn) tin will be replaced very soon [19:41:07] thcipriani: I'm pretty sure that scap doesn't actually deploy to labweb1001 and 1002. Do you have a moment to look or should I make a ticket? [19:42:42] https://github.com/wikimedia/puppet/blob/3ca48af991dc0767551f24b81828436e7205a8d6/hieradata/common/scap/dsh.yaml#L45-L46 [19:42:46] It's in the dsh group... [19:42:50] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#4062342 (10Andrew) [19:43:03] Reedy: I agree! But I just did a sync-file and the change didn't arrive [19:43:27] And it's not the first time I've done a 'scap pull' to get a change deployed [19:44:09] Fun... [19:44:13] They're in the dsh list on tin too [19:44:50] caching? [19:45:01] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#4062363 (10bd808) [19:45:28] andrewbogott: I can take a look. It's the wikitech file you just syncd? [19:45:37] thcipriani: correct [19:45:42] and, thank you! [19:45:50] * thcipriani checks it out [19:45:53] I haven't done a scap pull on either host yet [19:51:13] andrewbogott: wait you said you haven't done a scap pull? I see "This next conditional bit is meant to select wikitech on labweb but" on labweb1002 when I grep the wikitech file which looks like the last change. [19:51:42] thcipriani: last change is [19:51:43] https://gerrit.wikimedia.org/r/#/c/420406/3/wmf-config/wikitech.php [19:51:48] oh wait [19:52:00] I did my sync-file wrong [19:52:03] ignore me for a bit :) [19:52:18] * andrewbogott forget the fetch on tin [19:54:00] thcipriani: and, it worked. So nevermind! [19:54:13] :) [19:54:48] all works: best outcome [19:55:16] and nothing was broken! [19:56:35] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4062409 (10awight) [20:28:31] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #151: 04FAILURE in 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/151/ [20:35:30] 10Release-Engineering-Team, 10Operations, 10Puppet: puppetdb4: use postgres db backend in puppet-compiler - https://phabricator.wikimedia.org/T187258#4062512 (10herron) Compiler02 has been upgraded to puppetdb 4 and populate-puppetdb kicked off. The first few dozen hosts have compiled/populated successfully... [20:37:00] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4062513 (10mmodell) @awight: It looks to me like the gerrit commit message hook is breaking? I think we are really close, let me see if I can figure out what could cause the commit-msg hook to break like that. [20:57:15] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4062601 (10mmodell) >>! In T180627#4055517, @awight wrote: > I'm trying to check that in, and run: > ``` > cd submodules/assets > git config -f .lfsconfig lfs.url https://gerrit.wikimedia.org/r/scoring/ores/assets/info/lfs... [21:02:13] 10Continuous-Integration-Config, 10AbuseFilter: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4062632 (10hashar) [21:06:52] 10Continuous-Integration-Config, 10AbuseFilter: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4062645 (10hashar) The CI infrastructure basically runs `npm install && npm test`. We surely added some basic configuration to all MediaWiki... [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:13:35] (03CR) 10Krinkle: [C: 031] "I agree abstract methods in abstract classes should not need @covers. However, implemented methods in abstract classes should have @covers" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420259 (https://phabricator.wikimedia.org/T187738) (owner: 10MaxSem) [21:20:12] (03CR) 10Krinkle: [C: 031] Add checks for invalid annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/420159 (https://phabricator.wikimedia.org/T182057) (owner: 10MaxSem) [21:30:48] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) [21:31:05] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:40:46] RECOVERY - Free space - all mounts on deployment-mediawiki05 is OK: OK: All targets OK [21:52:03] (03CR) 10Hashar: [V: 032 C: 032] "Got deployed as is" [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/411372 (https://phabricator.wikimedia.org/T187567) (owner: 10Thcipriani) [21:52:15] (03CR) 10Hashar: [V: 032 C: 032] "Got deployed as is" [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/420001 (owner: 10Hashar) [21:54:43] 10Project-Admins: create new component project MediaWiki-extensions-AddPersonalUrls - https://phabricator.wikimedia.org/T190095#4062822 (10RV1971) [21:57:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: integration-slave-jessie-XXXX is out of disk space - https://phabricator.wikimedia.org/T189365#4062840 (10greg) a:05greg>03None [21:58:26] (03Abandoned) 10Hashar: docker image for commited node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/402876 (owner: 10Hashar) [21:58:50] (03Abandoned) 10Hashar: Integration tests with make-wmf-branch [integration/config] - 10https://gerrit.wikimedia.org/r/256413 (owner: 10Hashar) [22:01:05] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:02:29] 10Gerrit, 10Upstream: Polygerrit Outgoing reviews, Incoming reviews and Recently closed headings should be more different from commits - https://phabricator.wikimedia.org/T186406#4062859 (10Paladox) Upstream have redesgned the dashboard too, which looks very different in 2.16 / 3.0 then 2.15 or lower. [22:02:46] (03PS3) 10Hashar: Delete wikimedia-fundraising-civicrm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/414703 (https://phabricator.wikimedia.org/T187797) [22:03:08] (03CR) 10Hashar: [C: 032] Delete wikimedia-fundraising-civicrm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/414703 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:04:27] (03Merged) 10jenkins-bot: Delete wikimedia-fundraising-civicrm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/414703 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:06:59] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T183964#4062872 (10greg) 05Open>03Resolved [22:08:28] (03Abandoned) 10Hashar: (WIP) massage php-compile [integration/config] - 10https://gerrit.wikimedia.org/r/342638 (https://phabricator.wikimedia.org/T134381) (owner: 10Hashar) [22:12:36] 10Continuous-Integration-Config, 10AbuseFilter: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4062887 (10Volker_E) @hashar Eurgh, we've got reports upstream about this, you're the first one in our environment though (which might make... [22:13:25] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062894 (10Dzahn) 05Open>03Resolved a:03Dzahn Thank you for your work on this, Paladox. I'll call it resolved the... [22:13:29] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4062897 (10Dzahn) [22:13:33] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062899 (10Dzahn) a:05Dzahn>03Paladox [22:14:14] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062905 (10Paladox) @Dzahn though the change wasen't merged yet https://gerrit.wikimedia.org/r/#/c/410245/ [22:18:08] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062926 (10Dzahn) Eh.. then please define "stretch support is working" [22:18:13] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062927 (10Dzahn) 05Resolved>03Open [22:18:16] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4062928 (10Dzahn) [22:18:43] 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4062929 (10Volker_E) [22:19:08] 10Phabricator, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#4062930 (10Paladox) @Dzahn i cherry picked this https://gerrit.wikimedia.org/r/#/c/410245/ onto my local puppet master a... [22:20:02] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4062932 (10awight) Wow, so commit-msg hook was indeed corrupted, thank you! It was a 12kB HTML file rather than a script. And that fixes the commit error! Nice remote debugging. Unfortunately, I'm hitting an issue with... [22:20:40] Project mwext-phpunit-coverage-publish build #2359: 04FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2359/ [22:22:13] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4062940 (10mmodell) @awight: yeah, git-lfs only uses https, afaik. But I'm not sure why it would need to authenticate anyway? Seems strange. [22:22:49] Yippee, build fixed! [22:22:50] Project mwext-phpunit-coverage-publish build #2360: 09FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2360/ [22:25:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3: Migrate PHPUnit Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#3855599 (10greg) [22:25:52] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4005401 (10greg) [22:39:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)), 10Patch-For-Review, and 2 others: Create selenium-MediaWiki-jessie daily Jenkins job - https://phabricator.wikimedia.org/T185011#4062999 (10greg) p:05High>03Normal [22:49:07] (03PS18) 10Zoranzoki21: Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [22:50:19] (03CR) 10jerkins-bot: [V: 04-1] Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [23:52:28] 10Release-Engineering-Team (Watching / External), 10Operations: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4063260 (10RobH) a:05Cmjohnson>03None [23:55:13] 10Release-Engineering-Team (Watching / External), 10Operations: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4063265 (10Dzahn) a:03Dzahn