[01:05:06] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Post mortem for T139740 Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T188740#4034013 (10Jrbranaa) Created and sent out invites. Also decided to call this a "retrospective" vs "post mortem" :-) [01:08:00] Since T172165 implies we will require PHP 7 for 1.31 release, and we're still running lint on 5.5 on CI, is there a change planned to bump the lint version? At least to 5.6? [01:08:01] T172165: Require either PHP 7.0+ or HHVM in MW 1.31 - https://phabricator.wikimedia.org/T172165 [01:29:36] 10Phabricator (2018-03-07): Phame blog posts don't have 'published' metadata, only 'updated' - https://phabricator.wikimedia.org/T188890#4034020 (10mmodell) 05Open>03Resolved a:03mmodell Confirmed that the feeds now have both and elements. [01:43:04] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:44:51] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:45:31] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:46:10] (03CR) 10Chad: [V: 032 C: 032] Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [01:46:25] (03CR) 10Chad: [V: 032 C: 032] enable lfs on scoring/ores/* [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416775 (owner: 10Paladox) [01:47:15] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:47:38] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:47:42] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:52:43] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:53:10] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:53:42] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:55:35] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:57:45] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:57:54] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:59:12] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:00:19] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:00:55] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [02:04:26] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:04:30] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:04:48] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:04:53] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:04:58] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:08:45] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:09:51] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:12:54] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:13:50] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:09:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [03:35:51] 10Release-Engineering-Team, 10Developer-Relations, 10TechCom-RFC, 10WMF-Legal, 10User-bd808: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#4034158 (10Krinkle) [03:36:14] 10Release-Engineering-Team, 10Developer-Relations, 10WMF-Legal, 10TechCom-RFC (TechCom-Approved), 10User-bd808: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2426190 (10Krinkle) [04:02:45] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [04:03:41] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:20] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:36] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:56] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:07:52] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:09:14] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:25] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:32] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:50] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:52] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:52] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [04:14:54] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:17:54] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:18:44] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [04:18:50] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:22:39] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:23:04] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [04:24:50] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:25:32] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:27:16] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [04:27:37] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [04:27:39] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:28:12] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [04:53:30] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:13:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:35:55] 10Phabricator, 10Community-Tech: Herald rule for Community Tech - https://phabricator.wikimedia.org/T178649#4034222 (10Samwilson) 05Resolved>03Open Can #mediawiki-extensions-templatewizard please be added to H260? [07:09:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:01:17] zeljkof: good morning :D [08:01:35] zeljkof: do you have any merge rights on https://github.com/amire80/commons_upload ? [08:02:00] hashar: I think so [08:02:26] I was testing them this morning, they are absolutely trivial [08:02:31] I'll take a look at your patches in 30-60 minutes, ok? [08:02:32] though one depend on the other :] [08:02:34] ok [08:02:43] need to merge one, then rebase the other [08:02:51] (and tweak the other while it is rebased) [08:03:06] If you have tested them, I can just merge and release new version of the gem [08:03:06] but that would fix the lib when uploading a file that already exist \o/ [08:03:23] well I need one to be merged first, then the second would have to be adjusted [08:03:33] so yeah in 30-60mins that works for me [08:04:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4034335 (10hashar) [08:05:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, 10Patch-For-Review: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4034338 (10hashar) And eventually I h... [08:12:37] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-PoolCounter, 10Patch-For-Review: Fix tests of PoolCounter extension - https://phabricator.wikimedia.org/T178517#4034341 (10hashar) I kept trying to figure out a solution yesterday night, but eventually I give up. I am not familiar enough with socket... [08:13:25] (03PS3) 10Hashar: Drop mwext-PoolCounter-build-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/416913 (https://phabricator.wikimedia.org/T187797) [08:13:35] (03CR) 10Hashar: [C: 032] Drop mwext-PoolCounter-build-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/416913 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [08:14:06] (03CR) 10Hashar: [C: 032] "I have force merged https://gerrit.wikimedia.org/r/#/c/416907/ since the tests are broken/racing (T178517)" [integration/config] - 10https://gerrit.wikimedia.org/r/416913 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [08:14:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4034346 (10hashar) [08:15:01] (03Merged) 10jenkins-bot: Drop mwext-PoolCounter-build-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/416913 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [08:50:11] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:50:59] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:17:34] 10Gerrit: Incorrect gitiles git clone url - https://phabricator.wikimedia.org/T189182#4034375 (10TerraCodes) [09:30:02] (03PS1) 10Thiemo Kreuz (WMDE): Remove warning about "missing" @param comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/417214 (https://phabricator.wikimedia.org/T184650) [09:41:00] (03PS1) 10Hashar: docker: add /deploy repo support to npm-test [integration/config] - 10https://gerrit.wikimedia.org/r/417217 (https://phabricator.wikimedia.org/T187797) [09:43:00] (03PS2) 10Hashar: docker: add /deploy repo support to npm-test [integration/config] - 10https://gerrit.wikimedia.org/r/417217 (https://phabricator.wikimedia.org/T187797) [10:07:33] (03CR) 10Hashar: [C: 032] "Seems good locally :)" [integration/config] - 10https://gerrit.wikimedia.org/r/417217 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [10:08:45] (03Merged) 10jenkins-bot: docker: add /deploy repo support to npm-test [integration/config] - 10https://gerrit.wikimedia.org/r/417217 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [10:09:37] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T183963#4034562 (10jcrespo) [10:21:50] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T183963#4034595 (10Peachey88) [10:27:21] !log Deploy docker images for /deploy repositories | https://gerrit.wikimedia.org/r/#/c/417217/ [10:27:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:43:37] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, 10Patch-For-Review: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4034658 (10Deskana) Thank you for you... [10:48:30] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#4034671 (10zeljkofilipin) p:05Triage>03Low [10:56:14] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, 10Patch-For-Review: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4034685 (10hashar) You are welcome :]... [11:00:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#3893742 (10hashar) The bot user page [[ https://commons.wikimedia.org/wiki/User:LanguageScreenshotBot | User:LanguageScreenshotBot ]] has all t... [11:40:36] 10Phabricator, 10Community-Tech: Herald rule for Community Tech - https://phabricator.wikimedia.org/T178649#4034830 (10Aklapper) 05Open>03Resolved Sure! Done. [11:55:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 2 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4034883 (10zeljkofilipin) [11:59:09] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#3893742 (10zeljkofilipin) a:03zeljkofilipin [12:12:20] (03PS1) 10Samwilson: Add QUnit to TemplateWizard [integration/config] - 10https://gerrit.wikimedia.org/r/417243 [12:23:00] (03PS2) 10Samwilson: Add QUnit to TemplateWizard [integration/config] - 10https://gerrit.wikimedia.org/r/417243 (https://phabricator.wikimedia.org/T188466) [12:32:29] 10Gerrit: "git clone" URL shown on gitiles is incorrect, triggers "Permission denied (publickey)" error - https://phabricator.wikimedia.org/T189182#4034940 (10Aklapper) [12:42:31] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:42:41] Project beta-scap-eqiad build #198715: 04FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198715/ [12:44:44] Project beta-scap-eqiad build #198716: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198716/ [12:54:34] Project beta-scap-eqiad build #198717: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198717/ [13:04:39] Project beta-scap-eqiad build #198718: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198718/ [13:07:40] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [13:14:44] Project beta-scap-eqiad build #198719: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198719/ [13:17:31] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [13:24:42] Project beta-scap-eqiad build #198720: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198720/ [13:34:38] Project beta-scap-eqiad build #198721: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198721/ [13:44:36] Project beta-scap-eqiad build #198722: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198722/ [13:54:40] Project beta-scap-eqiad build #198723: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198723/ [13:57:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035043 (10hashar) [13:58:38] 10Phabricator: When navigating from a ticket to a tag, highlight the ticket in the workboard - https://phabricator.wikimedia.org/T189207#4035044 (10Pginer-WMF) [14:00:56] (03PS1) 10Hashar: Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) [14:01:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035060 (10hashar) [14:02:03] (03CR) 10jerkins-bot: [V: 04-1] Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:02:05] !log deployment-tin is out of disk space on /srv [14:02:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:03:43] !log deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc # 1.4GBytes [14:03:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:04:19] bah it keeps coming back :) [14:07:40] (03PS2) 10Hashar: Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) [14:09:56] Yippee, build fixed! [14:09:57] Project beta-scap-eqiad build #198724: 09FIXED in 6 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198724/ [14:11:49] (03PS3) 10Hashar: Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) [14:12:17] !log deployment-tin: rm -fR /srv/ocg [14:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:16:12] (03CR) 10Hashar: [C: 032] Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:16:30] (03PS1) 10Hashar: beta-scap-eqiad: clear HHVM byte cache on each run [integration/config] - 10https://gerrit.wikimedia.org/r/417263 [14:16:56] (03CR) 10Hashar: "deployment-tin /srv went full today :o\" [integration/config] - 10https://gerrit.wikimedia.org/r/417263 (owner: 10Hashar) [14:17:44] (03Merged) 10jenkins-bot: Experimental Docker jobs for /deploy repositories [integration/config] - 10https://gerrit.wikimedia.org/r/417260 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:22:41] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%) [14:29:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035140 (10hashar) [14:38:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035155 (10hashar) [14:44:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035175 (10hashar) [14:47:12] (03PS1) 10Hashar: Use Docker for change-propagation-deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417279 (https://phabricator.wikimedia.org/T187797) [14:47:32] (03CR) 10Hashar: [C: 032] Use Docker for change-propagation-deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417279 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:48:59] (03Merged) 10jenkins-bot: Use Docker for change-propagation-deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417279 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:50:19] (03PS1) 10Hashar: Add experimental kartotherian-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/417280 [14:51:41] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [14:52:51] (03PS1) 10Hashar: Migrate kartotherian-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417281 (https://phabricator.wikimedia.org/T187797) [14:53:47] (03CR) 10Hashar: [C: 032] Add experimental kartotherian-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/417280 (owner: 10Hashar) [14:54:55] (03PS2) 10Hashar: Migrate trending-edits-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417281 (https://phabricator.wikimedia.org/T187797) [14:54:58] (03CR) 10Hashar: [C: 032] Migrate trending-edits-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417281 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:55:20] (03Merged) 10jenkins-bot: Add experimental kartotherian-deploy-npm-node-6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/417280 (owner: 10Hashar) [14:56:33] (03Merged) 10jenkins-bot: Migrate trending-edits-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417281 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [14:58:23] (03PS1) 10Hashar: Migrate 3d2png-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417282 (https://phabricator.wikimedia.org/T187797) [14:59:23] (03PS1) 10Hashar: Migrate citoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417283 (https://phabricator.wikimedia.org/T187797) [15:00:15] (03PS1) 10Hashar: Migrate cxserver-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417284 (https://phabricator.wikimedia.org/T187797) [15:01:15] (03PS1) 10Hashar: Migrate mathoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417285 (https://phabricator.wikimedia.org/T187797) [15:02:18] (03PS1) 10Hashar: Migrate graphoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417286 (https://phabricator.wikimedia.org/T187797) [15:04:04] (03PS1) 10Hashar: Drop kartotherian-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417288 (https://phabricator.wikimedia.org/T187797) [15:04:54] (03PS1) 10Hashar: Migrate mobileapps-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417289 (https://phabricator.wikimedia.org/T187797) [15:05:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4035253 (10hashar) [15:07:39] (03CR) 10Hashar: [C: 032] Migrate 3d2png-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417282 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:07:57] (03CR) 10Hashar: [C: 032] Migrate citoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417283 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:08:27] (03CR) 10Hashar: [C: 032] Migrate cxserver-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417284 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:08:28] (03CR) 10Hashar: [C: 032] Migrate mathoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417285 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:08:41] (03CR) 10Hashar: [C: 032] Migrate graphoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417286 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:08:50] (03CR) 10Hashar: [C: 032] Drop kartotherian-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417288 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:09:03] (03CR) 10Hashar: [C: 032] Migrate mobileapps-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417289 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:09:31] thcipriani: so eventually I have migrated most mediawiki services to Docker containers. No need for Blubber yet as discussed on tuesday :] [15:11:06] (03CR) 10Hashar: [C: 032] Migrate 3d2png-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417282 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [15:11:39] (03PS2) 10Hashar: Migrate 3d2png-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417282 (https://phabricator.wikimedia.org/T187797) [15:11:41] (03PS2) 10Hashar: Migrate citoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417283 (https://phabricator.wikimedia.org/T187797) [15:11:43] (03PS2) 10Hashar: Migrate cxserver-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417284 (https://phabricator.wikimedia.org/T187797) [15:11:45] (03PS2) 10Hashar: Migrate mathoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417285 (https://phabricator.wikimedia.org/T187797) [15:11:47] (03PS2) 10Hashar: Migrate graphoid-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417286 (https://phabricator.wikimedia.org/T187797) [15:11:49] (03PS2) 10Hashar: Drop kartotherian-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417288 (https://phabricator.wikimedia.org/T187797) [15:11:50] stupid ci [15:11:51] (03PS2) 10Hashar: Migrate mobileapps-deploy-npm-node-6 to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/417289 (https://phabricator.wikimedia.org/T187797) [15:16:29] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#4035302 (10zeljkofilipin) [[ https://commons.wikimedia.org/w/index.php?title=Template:Wikipedia-screenshot&redirect=no | Template:Wikipedia-scr... [15:21:46] hashar: thcipriani the DB behind most openstack components ((cc andrewbogott)) is going to be switched between hosts in 10m, we need to stop nodepool [15:21:55] sorry for the lack of notification, didn't really occur to us until now [15:23:16] thcipriani: I'm also going to need to merge a mediawiki config patch as part of this failover, https://gerrit.wikimedia.org/r/#/c/417290/. Is that going to interfere with other things you're doing? [15:25:52] andrewbogott: I'm not deploying anything currently. I think Europe Mid-Day SWAT is done, so there shouldn't be anything really happening for the next 2 hours. [15:26:16] great, that's what I thought/hoped [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:32:44] 10Phabricator: When navigating from a ticket to a tag, highlight the ticket in the workboard - https://phabricator.wikimedia.org/T189207#4035340 (10Aklapper) If you are after moving one specific ticket on workboard, the recommended way would be to use the {nav name=Add Action... > Move on Workboard} dropdown ite... [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:34:58] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:35:39] PROBLEM - Puppet errors on integration-slave-docker-1005 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:35:51] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:35:56] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:36:27] 10Phabricator (Upstream), 10Upstream: When navigating from a ticket to a tag, highlight the ticket in the workboard - https://phabricator.wikimedia.org/T189207#4035348 (10Aklapper) [15:37:04] PROBLEM - Puppet errors on deployment-cumin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:37:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:38:21] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:38:23] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:38:30] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:38:57] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:39:33] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:39:43] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:39:51] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:39:51] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:41:05] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:41:34] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:42:12] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:43:26] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:50:33] paladox: Whee https://gerrit.wikimedia.org/g/operations/software/gerrit/gerrit/+/refs/heads/wmf/stable-2.14 [15:53:40] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: LanguageScreenshotBot uploads files to Commons without a license - https://phabricator.wikimedia.org/T184732#4035380 (10zeljkofilipin) There is no way to //change// text of a file via the API?! From https://commons.wikimedia.org/w/api.php?action=help&... [16:04:27] no_justification: yay :) [16:09:02] thcipriani: I have more patches coming later in the day but the actual bit where nodepool and wikitech are down should be done for the day [16:09:10] (Of course I'm going to break wikitech again tomorrow :) ) [16:09:18] :) [16:09:42] thanks for the notification and good luck with later patches :) [16:14:19] no_justification: actually that would be easier to maintain too :) [16:14:36] RECOVERY - Puppet errors on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:36] We won’t need to go through archiva to store plugins now [16:14:44] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:58] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:41] RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:47] But aren't bundled plugins extracted during `init` into ./plugins? [16:15:51] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:57] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:21] no_justification: only if you press y on each plugin [16:16:36] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:43] Put a n if you doint want the plugins to be replaced with the one in the war [16:16:44] But if they're not extracted they won't run? [16:17:02] RECOVERY - Puppet errors on deployment-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:15] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:21] Plus, the point of plugins is that you can deploy them without rebuilding/restarting core gerrit ;-) [16:17:30] Yeh :) [16:17:31] The main benefits here [16:17:36] a) tracking what the heck we have deployed [16:17:42] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:45] Yeh [16:17:46] b) Allowing us to more easily bring in non-standalone extensions [16:18:14] Yep [16:18:22] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:24] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:24] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:33] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:55] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:19:51] RECOVERY - Puppet errors on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:19:53] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:21:07] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:09] (03PS3) 10Umherirrender: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 [16:42:12] (03CR) 10BearND: [C: 04-1] Add mobileapps-diff-test periodic job (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/415886 (owner: 10Mholloway) [16:42:35] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:44:30] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [16:54:21] thcipriani: where is the list of targets that get a mediawiki scap deploy? It appears that my labweb1001/1002 hosts aren't in it. [16:55:06] on tin it's at /etc/dsh/groups/mediawiki-installation + a few other mediawiki-XXX files in there in puppet it's... [16:55:56] https://github.com/wikimedia/puppet/blob/production/hieradata/common/scap/dsh.yaml#L3-L45 [16:56:21] + https://github.com/wikimedia/puppet/blob/production/hieradata/common/scap/dsh.yaml#L56-L70 [16:58:25] cat /etc/dsh/group/mediawiki-* /etc/dsh/group/scap-* on tin gets you everything [16:58:48] thcipriani: so all I need is https://gerrit.wikimedia.org/r/#/c/417309/ ? [16:59:04] or is there a parallel thing in the config files? [17:00:29] do you pool and depool these via conftool? If so you can ensure that scap doesn't try to hit them when you've depooled for maintenance by adding a {cluster: xxx, service: xxx } thing above. [17:00:48] let me check on other stuff, you'll need to add...some role to the hosts if it's not already there. [17:02:00] thanks [17:02:11] I'm not using conftool for these currently [17:03:46] probably should though [17:03:49] ok, so I'm adding - {'cluster': 'wikitech', 'service': 'apache2'} [17:03:57] but I also need to tell someone which hosts are in that cluster right? [17:05:16] I don't know how the conftool initial setup works. _joe_ would probably know off the top of his head. [17:05:19] (03CR) 10Hashar: [C: 032] "For sure!" [integration/config] - 10https://gerrit.wikimedia.org/r/417243 (https://phabricator.wikimedia.org/T188466) (owner: 10Samwilson) [17:06:31] andrewbogott: for each target it seems like folks are using profile::mediawiki::common to ensure that the firewall gives scap access, users, and directories. [17:06:54] (03Merged) 10jenkins-bot: Add QUnit to TemplateWizard [integration/config] - 10https://gerrit.wikimedia.org/r/417243 (https://phabricator.wikimedia.org/T188466) (owner: 10Samwilson) [17:07:34] * andrewbogott wonders if that's included in ::mediawiki [17:07:43] oh, I guess not as it's a profile [17:07:55] this is from looking at snapshots. You could probably find what silver does (e.g., if it includes mediawiki or mediawiki::common) but it was a deeper puppet rabbit hole than snapshots ended up being. [17:08:37] but you're probably better able to navigate that rabbit hole than I am [17:08:39] most of the things in that profile are included in profile::openstack::base::wikitech::web [17:08:44] so we're probably mostly good [17:09:03] silver definitely doesn't do those things but I'm trying to standardize a bit [17:11:56] hm, now it looks like something is broken in CI [17:11:59] ah yeah profile::openstack::base::wikitech::web has {mediawiki:} so that's probably about everything [17:12:01] https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/1123/console [17:12:03] oh good :) [17:12:12] could be transient [17:12:56] yeah, seems better now [17:13:04] !log deleting a few nodepool instances that are no more registered in Jenkins [17:13:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:16:50] (03PS4) 10Umherirrender: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 [17:17:04] (03CR) 10Umherirrender: "Added call to isDeprecatedByIdentifier" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [17:17:34] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:10] back shortly meeting [17:32:17] woops [18:14:02] I have a patch that refuses to merge (https://gerrit.wikimedia.org/r/#/c/415514/). Unrelated tests are failing: jquery.wikibase.edittoolbar. Has anyone seen that? [18:20:55] stephanebisson: happened on https://gerrit.wikimedia.org/r/c/416765/ as well [18:20:59] after two rechecks we got lucky and it went away [18:21:14] but if that issue now affects CI of unrelated repositories, we definitely need to look into that :/ [18:29:06] Lucas_WMDE: It's been affecting some MediaWiki core and VisualEditor-MediaWiki patches today. [18:34:08] I’ve opened a task T189228 [18:34:09] T189228: Spurious CI failures for jquery.wikibase.edittoolbar in mediawiki-extensions-qunit-jessie - https://phabricator.wikimedia.org/T189228 [18:36:00] !log Update mobileapps to afb0167 [18:36:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:40] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<22.22%) [18:57:40] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [19:20:39] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins, 10Patch-For-Review: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4036144 (10TJones) Gerrit has indeed now replicated the repo to GitHub. Thanks! [19:30:52] I just noticed, is https://github.com/wikimedia/mediawiki/blob/REL1_30/RELEASE-NOTES-1.30#L3 intended? [19:31:00] 1.30 has definitely been a release for a while now :P [19:39:55] (03PS1) 10Legoktm: Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) [19:40:25] (03PS1) 10Umherirrender: Archive PHPExcel [integration/config] - 10https://gerrit.wikimedia.org/r/417344 (https://phabricator.wikimedia.org/T189238) [19:40:30] Reception123: the current branch/point release hasn't been released yet [19:41:03] (03CR) 10jerkins-bot: [V: 04-1] Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) (owner: 10Legoktm) [19:43:04] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10PHP 7.0 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7 - https://phabricator.wikimedia.org/T144962#4036236 (10Umherirrender) [19:46:25] legoktm: Oh, ok. That will be 1.30.1, right? [19:54:06] (03CR) 10Umherirrender: Archive PHPExcel (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/417344 (https://phabricator.wikimedia.org/T189238) (owner: 10Umherirrender) [19:55:42] Reception123: likely, yes [19:57:12] 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster Obama page often responds with 503 - https://phabricator.wikimedia.org/T188913#4036318 (10Niedzielski) This seems to always happen on deployment-cache-text04: ``` Request from 73.252.38.252 via deployment-cache-text04 deployment-cache-text04, Varnish X... [20:00:08] 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster Obama page often responds with 503 - https://phabricator.wikimedia.org/T188913#4036348 (10Niedzielski) https://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page seems to always work while the Obama article fails: ``` Request from 73.252.38.252 via deploym... [20:09:09] 10Continuous-Integration-Infrastructure: Install php-yaml for Translate - https://phabricator.wikimedia.org/T189244#4036384 (10Reedy) [20:09:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Support for Blubber defaults and/or policies - https://phabricator.wikimedia.org/T174631#4036394 (10dduvall) 05declined>03Open p:05Triage>03High a:03dduvall Blubber now has a more secure default permissions scheme (see {T187372}) but... [20:29:07] Yippee, build fixed! [20:29:07] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #139: 09FIXED in 42 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/139/ [20:39:02] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:48:41] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<22.22%) [20:55:17] PROBLEM - Puppet errors on deployment-maps01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:02:39] (03PS2) 10Legoktm: Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) [21:03:51] (03CR) 10jerkins-bot: [V: 04-1] Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) (owner: 10Legoktm) [21:14:43] (03PS3) 10Legoktm: Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) [21:35:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4036639 (10hashar) [21:37:14] (03CR) 10Legoktm: [C: 032] "Awesome :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [21:38:11] (03Merged) 10jenkins-bot: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [21:38:30] (03PS1) 10Hashar: Remove tilerator-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417390 (https://phabricator.wikimedia.org/T187797) [21:38:42] (03CR) 10jenkins-bot: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [21:39:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4036649 (10hashar) [21:40:29] (03CR) 10Hashar: [C: 032] Remove tilerator-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417390 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [21:42:01] (03Merged) 10jenkins-bot: Remove tilerator-deploy-npm-node-6-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/417390 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [21:47:57] Project mwext-phpunit-coverage-publish build #1906: 04FAILURE in 5.4 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1906/ [21:48:35] Yippee, build fixed! [21:48:35] Project mwext-phpunit-coverage-publish build #1907: 09FIXED in 37 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1907/ [21:49:57] Project mwext-phpunit-coverage-publish build #1910: 04FAILURE in 5.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1910/ [21:50:18] Yippee, build fixed! [21:50:19] Project mwext-phpunit-coverage-publish build #1911: 09FIXED in 21 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1911/ [21:50:23] no_justification you forgot to submit https://gerrit.wikimedia.org/r/c/416357/ :) [21:51:28] "Cannot Merge" [21:51:32] I didn't feel like rebasing [21:51:49] no_justification ah i will rebase then [21:51:51] somehow [21:52:37] Project mwext-phpunit-coverage-publish build #1914: 04FAILURE in 4.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1914/ [21:52:59] Yippee, build fixed! [21:52:59] Project mwext-phpunit-coverage-publish build #1915: 09FIXED in 21 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1915/ [21:53:59] (03PS1) 10Hashar: Experimental docker jobs for parsoid/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) [21:54:32] no_justification when pushing to the meta branch do i do git push origin HEAD:refs/for/meta/config? [21:54:39] or git push origin HEAD:refs/for/refs/meta/config? [21:54:47] (03PS3) 10Paladox: Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 [21:54:50] ah worked [21:54:52] git push origin HEAD:refs/for/refs/meta/config [21:55:52] legoktm: for mwext-testextension-php70-jessie-non-voting is that new? [21:56:16] legoktm: please drop it from the test pipeline, that consumes Nodepool instances and i would rather not have new jobs on nodepool [21:56:20] ;D [21:56:26] hasharDinner: I'm making it voting in a few hours [21:56:33] no i mean [21:56:39] dont add it to the test pipeline [21:56:45] it is too many builds being added [21:56:57] ah [21:57:00] ok [21:57:09] i am almost done moving the rest of the jobs, will start tackling the migration of the mediawiki jobs [21:57:20] and will come with some kind of proposal / rfc for people to shime in [21:57:27] I'll do that as part of making it voting then in a little bit [21:57:32] Project mwext-phpunit-coverage-publish build #1917: 04FAILURE in 4.9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1917/ [21:57:40] I am eager to see us running tests under php7, but not on Nodepool :] [21:57:41] (rename php5 queue to php, add php70 there) [21:57:51] well it's going to be in gate regardless [21:57:58] Yippee, build fixed! [21:57:59] Project mwext-phpunit-coverage-publish build #1918: 09FIXED in 25 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1918/ [21:58:04] well I would rather delay that by at least a week [21:58:29] I might have a good straight forward way to migrate all of them to Docker [21:58:50] but I need to write my thoughts down. Hopefully we can start migrating soonish (less than a couple weeks) [21:58:59] I think we're far behind enough that we need to get php 7 voting basically now [21:59:10] hopefully it provides enough motivation to dockerize them [21:59:37] but adding 500 builds or so to Nodepool when I am working on removing builds from it is counterproductive [21:59:45] !log legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space [21:59:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:00:14] Not testing against PHP 7 is even worse IMO [22:00:34] well I am asking to delay that by at least a week [22:00:41] so as to not more pressure on nodepool [22:01:13] It can't add any more pressure to nodepool [22:01:32] It's already been running in test for 2 weeks and moving it to gate is going to reduce the number of builds overall [22:06:42] legoktm: AHHHHHHH I havent noticed it being in test for 2 weeks :D [22:06:47] so yeah i guess it is fine [22:07:05] lol [22:07:15] hasharDinner: it's quicker than hhvm :D [22:07:26] oh that is not the problem ;D [22:07:38] No, it's a benefit though :P [22:07:39] the issue is that it takes 3 minutes to respawn a Nodepool instance [22:07:50] so each additional one being consumed slow down the whole stack [22:08:41] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [22:09:43] (03PS4) 10Legoktm: Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) [22:09:47] (03CR) 10Hashar: [C: 04-1] "The script jobs are missing --entrypoint=/run-oid.sh" [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:09:58] hasharDinner: ^ should have moved everything out of test and into php5/gate [22:10:03] (I'll rename the queue later) [22:10:14] really we should just have a "check gate" option I feel [22:10:58] "gate is closed" [22:11:06] (03PS5) 10Legoktm: Run MediaWiki tests against PHP 7.0 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/417343 (https://phabricator.wikimedia.org/T144962) [22:13:24] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<44.44%) [22:14:48] (03PS2) 10Hashar: Experimental docker jobs for parsoid/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) [22:17:05] bah [22:19:48] !log cleaned up /srv on integration-slave-jessie-1001 . Upgrade packages and reboot. [22:19:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:20:19] (03CR) 10Hashar: [C: 032] Experimental docker jobs for parsoid/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:21:42] (03CR) 10jerkins-bot: [V: 04-1] Experimental docker jobs for parsoid/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:22:37] legoktm i think you may like this feature https://groups.google.com/forum/#!topic/repo-discuss/I3Wxo3Belwc (not that actual thing but how it does it) [22:23:14] (03CR) 10Hashar: [C: 032] "I forgot to deploy the job parsoidsvc-deploy-npm-node-6-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:23:19] paladox: !!! [22:23:25] very cool [22:23:25] :) [22:23:31] yeh. [22:23:40] detects whitespace and replys as a robot comment [22:24:14] so it's just an account leaving comments? [22:24:37] (03Merged) 10jenkins-bot: Experimental docker jobs for parsoid/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/417453 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [22:24:37] yeh [22:24:47] it's a bot behind the scenes [22:25:57] mmm [22:28:24] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [22:34:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday): Install php-yaml for Translate - https://phabricator.wikimedia.org/T189244#4036859 (10hashar) We would need Debian packages for each of the PHP version we support: * for php 5.5 we would need it to be packaged for Jessie like I did... [22:35:17] Reedy: can you shoot more details on https://phabricator.wikimedia.org/T189244 (php-yaml for Translate). It is a bit scarse as it is :] [22:36:28] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4036867 (10hashar) [22:40:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10MW-1.31-release-notes (WMF-deploy-2018-03-13 (1.31.0-wmf.25)), 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4036894 (10hashar) parsoid... [22:46:15] (03Abandoned) 10Paladox: Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/413005 (owner: 10Paladox) [22:49:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday): Install php-yaml for Translate - https://phabricator.wikimedia.org/T189244#4036384 (10Legoktm) the YAML extension is bundled with HHVM already. `php-yaml` is only in Debian Stretch, so we'd need to backport it to Jessie for 5.5. [22:50:01] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4036925 (10Dzahn) I have tried with manual partitioning, i have tried with jessie instead of stretch, i have trie... [23:20:13] Project beta-update-databases-eqiad build #23996: 04FAILURE in 12 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/23996/ [23:21:22] oojs bump [23:36:43] 10Beta-Cluster-Infrastructure: Create Fatal-Monitor dashboard in logstash-beta - https://phabricator.wikimedia.org/T185974#4037095 (10greg) p:05Triage>03Normal