[00:59:10] (03CR) 10Chad: "Ideally we would know what their previous reference was so we could log between the two :\" [tools/release] - 10https://gerrit.wikimedia.org/r/231313 (owner: 10Florianschmidtwelzow) [01:16:46] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1894141 (10demon) Files posted for 1.24.6, 1.25.5 and 1.26.2. Waiting on 1.23.13 until we figure out the unit test questions on the blocked ta... [01:23:44] ostriches: regarding your comment on https://gerrit.wikimedia.org/r/#/c/231313/2 I agree, that this would be the ideal solution. But I must say, that I'm not very familar with the proces of generating a wmf branch, maybe you can help me? :D Would it be possible to autogenerate a JSON with old refs (commit sha1 e.g.) for these extensions in https://github.com/wikimedia/mediawiki-tools-release/blob/c896ee256290c5be6f166567bbaf [01:23:44] c8795c43fc4b/make-wmf-branch/MakeWmfBranch.php#L260-L265 ? [01:23:50] https://github.com/wikimedia/mediawiki-tools-release/blob/c896ee256290c5be6f166567bbafc8795c43fc4b/make-wmf-branch/MakeWmfBranch.php#L260-L265 [01:25:02] In theory you could grab the sha1 from the submodule since we already know about wmf branch -1. In practice make-wmf-branch is a terrible script and it's probably not worth thattttt much effort. [01:40:39] Yippee, build fixed! [01:40:39] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #477: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/477/ [01:55:42] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1894162 (10demon) 1.23.13 up as well. https://releases.wikimedia.org/mediawiki/1.23/mediawiki-1.23.13.patch.gz https://releases.wikimedia.org... [01:57:15] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1894165 (10demon) Git tagging was already done too. [02:48:02] Project beta-scap-eqiad build #83084: 04FAILURE in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83084/ [03:15:31] Yippee, build fixed! [03:15:32] Project beta-scap-eqiad build #83086: 09FIXED in 9 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83086/ [06:04:40] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [06:05:50] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [07:12:43] RECOVERY - Free space - all mounts on deployment-fluorine is OK: OK: All targets OK [07:54:18] PROBLEM - puppet last run on scandium is CRITICAL: CRITICAL: Puppet last ran 4 days ago [07:56:10] RECOVERY - puppet last run on scandium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:35:19] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #820: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/820/ [09:58:21] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.02 ms [10:00:11] (03CR) 10Hashar: "Did the same for bin/mw-selenium-teardown.sh (i.e. keep directory around)" (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:00:20] (03PS2) 10Hashar: Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:00:50] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [10:01:34] (03CR) 10Hashar: [C: 032] Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:01:48] (03CR) 10Hashar: [C: 032] Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:03:03] (03CR) 10Hashar: Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:03:10] (03CR) 10Hashar: [C: 032] Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:06:30] (03Merged) 10jenkins-bot: Workaround for tmpdir being created with a different user [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260174 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [10:11:12] (03PS1) 10Hashar: Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) [10:11:19] (03PS2) 10Hashar: Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) [10:14:29] (03CR) 10Hashar: [C: 032] Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) (owner: 10Hashar) [10:14:49] (03CR) 10Hashar: "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) (owner: 10Hashar) [10:14:54] 6Release-Engineering-Team, 3releng-201516-q3, 10Wikimedia-Developer-Summit-2016: Code-review migration to Differential status/discussion - https://phabricator.wikimedia.org/T114320#1894366 (10Qgil) I have proposed the session for Monday at 10am: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016#... [10:15:21] (03PS3) 10Hashar: Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) [10:16:09] (03CR) 10Hashar: Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) (owner: 10Hashar) [10:16:17] (03CR) 10Hashar: [C: 032] Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) (owner: 10Hashar) [10:18:14] (03Merged) 10jenkins-bot: Revert "Tie mwext-mw-selenium* jobs to specific slaves" [integration/config] - 10https://gerrit.wikimedia.org/r/260335 (https://phabricator.wikimedia.org/T120824) (owner: 10Hashar) [10:19:04] grmblbllblbl [10:19:40] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [10:21:02] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1894372 (10hashar) Following the discussion on Friday evening with Dan, the workaround would be to only delete f... [10:21:19] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [10:31:11] 10Beta-Cluster-Infrastructure, 6Discovery, 10Wikimedia-Portals: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1894379 (10hashar) [10:31:29] 10Beta-Cluster-Infrastructure, 6Discovery, 10Wikimedia-Portals: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1893907 (10hashar) Looks the `portals` submodule is not properly updated for some reason. [10:35:50] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.36 ms [10:37:25] 10Beta-Cluster-Infrastructure, 6Discovery, 10Wikimedia-Portals: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1894391 (10hashar) a:3hashar The `portals` is part of `operations/mediawiki-config.git` which is updated via the job [... [10:38:20] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [10:43:47] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 6Discovery, 10Wikimedia-Portals, and 2 others: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1894421 (10hashar) [10:48:17] !log Banned testing-shinken- bot (useless duplicate notifications) [10:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:48:56] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 6Discovery, 10Wikimedia-Portals, and 2 others: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1894426 (10hashar) I have cherry picked https://gerrit.wikimedia.org/r/... [10:51:50] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 6Discovery, 10Wikimedia-Portals, and 2 others: 'portals' git submodule does not update on beta ( under /srv/mediawiki-staging ) - https://phabricator.wikimedia.org/T122018#1894429 (10hashar) p:5Triage>3Normal Should be good now. Pending m... [10:53:29] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [11:16:39] 10Deployment-Systems, 6Release-Engineering-Team: scap-purge-l10n-cache hanging - https://phabricator.wikimedia.org/T122008#1894495 (10hashar) 5Open>3Resolved a:3hashar `/usr/bin/touch: cannot touch ‘/srv/mediawiki/wmf-config/InitialiseSettings.php’: Read-only file system` `/srv` is a symlink to `/mnt` w... [11:18:37] 10Deployment-Systems, 6Release-Engineering-Team: scap-purge-l10n-cache hanging - https://phabricator.wikimedia.org/T122008#1894505 (10Reedy) Why've you resolved it? :P It's only "fixed" because the mw app server with the r/o filesystem was pooled yesterday Don't think it's a labs issue either... [11:19:12] 10Continuous-Integration-Config, 6Release-Engineering-Team, 6Collaboration-Team-Backlog, 10Flow, 5MW-1.25-release: Flow (or tests) broken in REL1_25 - https://phabricator.wikimedia.org/T122025#1894507 (10hashar) Looks like some QUnit test is falling with: `'Could not find .flow-board', Object{0: {0: 10Deployment-Systems, 6Release-Engineering-Team: scap-purge-l10n-cache hanging - https://phabricator.wikimedia.org/T122008#1894561 (10hashar) 5Resolved>3Open Sorry I though it was on beta for some reason. Maybe the task should be rephrased as `mw1228.eqiad.wmnet has read only /srv/mediawiki-staging` ? [11:49:12] lunch [11:50:53] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [11:52:09] 10Deployment-Systems, 6Release-Engineering-Team: scap-purge-l10n-cache hanging - https://phabricator.wikimedia.org/T122008#1894569 (10Reedy) T122005 is the ops ticket This is for scap-purge-l10n-cache handling it badly and apparently hanging in infinitum! [12:12:52] * aude *sighs* ... more permission issues [12:12:54] https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/16060/consoleFull [12:27:04] 7Browser-Tests, 5Patch-For-Review, 3Reading Web Sprint 62 - DJ-Jazzy-Jeff-and-the-Fresh-Sprints: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1894596 (10bmansurov) 5Open>3Resolved Closing since the last QuickSurvey browser test failure happened on December 1... [12:46:50] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3181: 04FAILURE in 49 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3181/ [13:04:26] !log restarting cxserver on deployment-cxserver03 [13:04:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:04:49] (03PS1) 10Robert Vogel: Added dependency for 'NSFileRepo' extension [integration/config] - 10https://gerrit.wikimedia.org/r/260355 [13:11:51] kart_: hello. cxserver on beta cluster deployment-cxserver03 reports a 404 when trying to reach / or /_version [13:12:02] kart_: no idea what is going. I tried to restarted it a couple time but hasn't fixed it [13:12:49] HTTP WARNING: HTTP/1.1 404 Not Found - 756 bytes in 0.037 second response time [13:12:50] 0.037378s [13:12:56] oh been going on for 4 days.. [13:21:06] hashar: needs to be changed to / [13:21:14] hashar: to /_info [13:21:21] hashar: ^^ [13:21:34] mobrovac: also, fix for registry is set tomorrow. [13:21:38] ahh [13:21:48] $ curl http://localhost:8080/_info [13:21:48] {"name":"mediawiki-contenttranslation","version":"1.0.0","description":"cxserver - The Content Translation Server","home":"https://mediawiki.org/wiki/CX"}hashar [13:21:55] no clue how Shinkens generates it probes though [13:21:58] its probes [13:22:24] the error shows up on left bar of http://shinken.wmflabs.org/service/deployment-cxserver03/Content%20Translation%20Server [13:22:29] and is being reported here by shinken-wm [13:22:50] Maybe Yuvi has more idea. [13:24:54] (03CR) 10Hashar: [C: 04-1] "If the QUnit test is flappy, the easiest action would be to remove it from VE and have a task to restore / fix it." [integration/config] - 10https://gerrit.wikimedia.org/r/259959 (https://phabricator.wikimedia.org/T116258) (owner: 10Legoktm) [13:35:57] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [13:53:27] (03CR) 10Hashar: "NSFileRepo uses a specific job "mwext-NSFileRepo-testextension-zend" which does inject LockDown as a dependency. The CI loader loads the" [integration/config] - 10https://gerrit.wikimedia.org/r/260355 (owner: 10Robert Vogel) [13:56:58] (03CR) 10Hashar: [C: 04-1] "Sorry it is not the SetupAfterCache hook but $wgExtensionFunctions which should be used. That is run late in MediaWiki initialization at " [integration/config] - 10https://gerrit.wikimedia.org/r/260355 (owner: 10Robert Vogel) [13:58:30] (03PS3) 10Hashar: [ShoutWikiAPI] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/260248 (owner: 10Paladox) [14:05:45] (03CR) 10Hashar: [C: 032] [ShoutWikiAPI] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/260248 (owner: 10Paladox) [14:06:38] (03Merged) 10jenkins-bot: [ShoutWikiAPI] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/260248 (owner: 10Paladox) [14:27:50] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #730: 04FAILURE in 1 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/730/ [14:32:24] 10Continuous-Integration-Infrastructure, 10MediaWiki-API, 10MediaWiki-Unit-tests, 10MediaWiki-extensions-CentralAuth: Fix ApiDocumentationTest on CentralAuth - https://phabricator.wikimedia.org/T122007#1894737 (10Anomie) It looks like something in the CI infrastructure rather than the test itself, since th... [14:35:58] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.90 ms [14:38:24] 5Continuous-Integration-Scaling, 7Tracking: [tracking] Disposable VMs need a cache for package managers - https://phabricator.wikimedia.org/T112560#1894755 (10hashar) [14:38:25] 5Continuous-Integration-Scaling, 5Patch-For-Review, 7Tracking: Puppetize / play out with devpi - https://phabricator.wikimedia.org/T117207#1894753 (10hashar) 5Open>3Resolved Finished experimentation [14:38:35] (03Abandoned) 10Hashar: (WIP) use pmcache has a pypi cache [integration/config] - 10https://gerrit.wikimedia.org/r/250009 (https://phabricator.wikimedia.org/T117207) (owner: 10Hashar) [14:39:16] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Tracking: tracking: Cleanup contint puppet manifests so they are easier to reuse / split slaves by functions - https://phabricator.wikimedia.org/T110864#1894757 (10hashar) 5Open>3Resolved a:3hashar The few immediate blockers we... [14:39:43] 10Continuous-Integration-Infrastructure: Update jobs to use zuul-cloner with git cache - https://phabricator.wikimedia.org/T97098#1894761 (10hashar) [14:39:45] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: Nodepool images need Gerrit mirror for git-clone performance - https://phabricator.wikimedia.org/T87294#1894760 (10hashar) 5Open>3Resolved [14:39:46] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10releng-201415-Q3, 10releng-201415-Q4, 7Epic: [EPIC] Run CI jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#1894762 (10hashar) [14:41:36] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 6Labs, 5Patch-For-Review: Instance creation fails - https://phabricator.wikimedia.org/T120586#1894763 (10hashar) So this task is about Nova getting weird back on Sunday December 6th. Got fixed by Yuvi by restarting `nova-conductor` a... [14:42:05] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 6Labs, 5Patch-For-Review: Instance creation fails - https://phabricator.wikimedia.org/T120586#1894765 (10hashar) p:5High>3Normal Lowering priority since the immediate issue has been fixed. [14:43:45] 5Continuous-Integration-Scaling: Figure out how to inject facts in the diskimage-builder chroot - https://phabricator.wikimedia.org/T110737#1894769 (10hashar) 5Open>3declined a:3hashar I worked around facts entirely. No need for this task anymore. [14:43:52] (03CR) 10Robert Vogel: "Hashar, thanks for your the reply and the detailed explanation! I will fix NSFileRepo in the recommended way." [integration/config] - 10https://gerrit.wikimedia.org/r/260355 (owner: 10Robert Vogel) [14:44:37] (03Abandoned) 10Robert Vogel: Added dependency for 'NSFileRepo' extension [integration/config] - 10https://gerrit.wikimedia.org/r/260355 (owner: 10Robert Vogel) [14:46:12] (03CR) 10Hashar: "Glad you like it. Sorry for the annoyance, but in the end that means people can load both extensions in whatever order and it will just wo" [integration/config] - 10https://gerrit.wikimedia.org/r/260355 (owner: 10Robert Vogel) [14:47:34] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: tox-jessie job is failing randomly on integration/config with could not install deps, because of throttling - https://phabricator.wikimedia.org/T114650#1894772 (10hashar) 5Open>3Resolved a:3hashar Transient issue due to Pypi. Will... [14:59:35] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1894803 (10zeljkofilipin) [15:03:48] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1894811 (10zeljkofilipin) [15:12:52] Yippee, build fixed! [15:12:53] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3182: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3182/ [15:26:03] PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:36:01] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1894877 (10zeljkofilipin) [15:36:37] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [15:47:38] 10Continuous-Integration-Infrastructure: Investigate installing php5.3 on trusty and/or debian instance - https://phabricator.wikimedia.org/T103786#1894882 (10hashar) >>! In T118932#1831937, @bd808 wrote: > > Semi-confusingly the version of phpenv at https://github.com/CHH/phpenv is the better version to use. T... [15:51:20] 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-MediaWiki: Please remove VisualEditor from 'mediawiki-extensions-qunit' check for MediaWiki core - https://phabricator.wikimedia.org/T122055#1894893 (10matmarex) 3NEW [15:52:41] 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-MediaWiki: Please remove VisualEditor from 'mediawiki-extensions-qunit' check for MediaWiki core - https://phabricator.wikimedia.org/T122055#1894901 (10matmarex) I really don't want to be annoyed by this continuously for the next two or three we... [15:56:24] 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-MediaWiki: Please remove VisualEditor from 'mediawiki-extensions-qunit' check for MediaWiki core - https://phabricator.wikimedia.org/T122055#1894907 (10hashar) We can't remove VisualEditor. A few other extensions rely on it for Javascript testin... [15:56:40] 10Continuous-Integration-Config, 10VisualEditor, 10VisualEditor-MediaWiki: Please remove VisualEditor from 'mediawiki-extensions-qunit' check for MediaWiki core - https://phabricator.wikimedia.org/T122055#1894908 (10hashar) [16:11:45] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1894947 (10zeljkofilipin) [17:14:15] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1895194 (10zeljkofilipin) [18:04:20] Can someone help me? I saw that you use unit tests from arcanist, like here: https://phabricator.wikimedia.org/harbormaster/unit/238/. Can someone tell me, how I can install a UnitTest engine for PHP? [18:08:50] (03CR) 10Krinkle: "@Hashar: If it were that easy, I would've done that 2 months ago." [integration/config] - 10https://gerrit.wikimedia.org/r/259959 (https://phabricator.wikimedia.org/T116258) (owner: 10Legoktm) [18:13:05] looks like jenkins slaves are running short on space? been getting intermittent odd failures of parsoid jobs (see failures on PS2 and PS3 of https://gerrit.wikimedia.org/r/#/c/260395/ ) [18:17:32] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 6Labs, 5Patch-For-Review: Instance creation fails - https://phabricator.wikimedia.org/T120586#1895375 (10coren) a:5coren>3None [18:20:55] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/260248 (owner: 10Paladox) [18:21:37] hmm [18:21:50] none 512M 512M 0 100% /mnt/home/jenkins-deploy/tmpfs [18:22:20] that's on 1015 [18:22:35] 1017 as well [18:23:00] it may be related to the recent TMPDIR trouble [18:24:07] legoktm@integration-slave-trusty-1017:/mnt/home/jenkins-deploy/tmpfs$ du -hs * [18:24:07] du: cannot read directory ‘jenkins-0/pulse-Wu3v9nzr3huT’: Permission denied [18:24:07] 32M jenkins-0 [18:24:07] 344M jenkins-1 [18:24:07] 76M jenkins-2 [18:24:08] 62M jenkins-3 [18:25:03] 134556 npm-8813-c848bd41 [18:25:03] 134920 npm-7495-ffbd2472 [18:25:42] eh [18:25:48] the npm directories look reasonable [18:26:34] hi folks, if no one complained already, let it be known that jenkins is failing for all commits to mediawiki/core (again) [18:26:45] is it the disk space issue? [18:26:52] I'm looking into that right now [18:28:18] !log deleted some large npm directories from tmpfs on 1017 due to tmpfs being full [18:28:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:28:31] none 512M 241M 272M 48% /mnt/home/jenkins-deploy/tmpfs [18:28:33] >.> [18:29:47] legoktm: it doesn't look like global-teardown is actually deleting files in TMPDIR [18:30:00] according to the job output anyway [18:30:36] legoktm: could be related to https://gerrit.wikimedia.org/r/#/c/260174/ [18:30:36] uhh [18:30:42] that would cause problems. [18:31:01] werd [18:31:18] !log same thing on 1015 [18:31:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:31:42] uh, the globs in that patch are quoted [18:32:12] `rm -v -rf "${TMPDIR_FS}/*"` [18:32:45] uh, my bash-foo has no idea what that means [18:33:30] jzerebecki: ^ [18:34:27] legoktm: it means bash won't expand the glob, it will take it as literally part of the filename [18:34:54] i.e. no files are named "*" in that directory, so nothing is deleted [18:35:04] lol [18:35:11] so can we remove the quotes then? [18:35:20] yep yep [18:35:24] pushing a patch now [18:38:50] (03PS1) 10Dduvall: Fix globs in global and mw-selenium teardown scripts [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260407 (https://phabricator.wikimedia.org/T120824) [18:41:59] (03CR) 10Dduvall: [C: 032] Fix globs in global and mw-selenium teardown scripts [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260407 (https://phabricator.wikimedia.org/T120824) (owner: 10Dduvall) [18:43:01] (03Merged) 10jenkins-bot: Fix globs in global and mw-selenium teardown scripts [integration/jenkins] - 10https://gerrit.wikimedia.org/r/260407 (https://phabricator.wikimedia.org/T120824) (owner: 10Dduvall) [18:43:34] !log Updating slave scripts on all integration slaves to deploy I4edf7099acfeb0f06ea2042902bef03097137d6e [18:43:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:49:42] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3184: 04FAILURE in 3 min 41 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3184/ [19:01:01] !log Purging TMPDIR contents on idle integration slaves [19:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:02:42] subbu, legoktm, MatmaRex: we should be in the clear now [19:03:11] marxarelli, thanks! [19:03:31] no problem! [19:49:13] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1895637 (10Reedy) [20:06:04] !log Downgrading Jenkins plugin from 1.24 to 1.21 [20:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:12:01] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1895797 (10Reedy) [20:12:59] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1894018 (10Reedy) [20:22:38] ostriches, me and MatmaRex would like a couple of VE JS/CSS fixes to go out to fix UI regressions that users have been complaining about [20:22:46] is that ok? [20:23:04] Yeah, I lost that part of the scrollback but Tyler caught me up. That's fine :) [20:38:51] 6Release-Engineering-Team, 5MW-1.23-release, 5MW-1.25-release, 5MW-1.26-release: Release 1.23.13, 1.24.6, 1.25.5, 1.26.2 - https://phabricator.wikimedia.org/T122023#1895894 (10demon) 5Open>3Resolved a:3demon https://lists.wikimedia.org/pipermail/mediawiki-announce/2015-December/000187.html [20:50:04] why is the diffusion link for this rev not working? https://gerrit.wikimedia.org/r/#/c/241690/ [20:50:33] diffusion link? [20:59:16] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:03:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39456 bytes in 0.716 second response time [21:11:00] robla: There's a few kinks we're trying to work out with displaying non-merged changes in diffusion, known problem. [21:11:25] tldr: refs that aren't refs/heads/* don't get indexed properly. [21:37:49] Project beta-scap-eqiad build #83176: 04FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83176/ [21:41:15] 7Browser-Tests, 5Patch-For-Review, 3Reading Web Sprint 62 - DJ-Jazzy-Jeff-and-the-Fresh-Sprints: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1896185 (10Jdlrobson) I see fields of green..... :) https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-e... [21:45:41] marxarelli|lunch: I noticed in SAL you purged the TMPDIR on slaves at 19:00 UTC. Was something going wrong? [21:45:57] oh [21:46:03] https://gerrit.wikimedia.org/r/#/c/260407/ :-D [21:48:37] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1896204 (10hashar) @dduvall Thank you! I noticed some left over before our weekly meeting and checked again a m... [21:48:45] stupid glob patterns and quotes :-/ [22:43:19] Yippee, build fixed! [22:43:19] Project beta-scap-eqiad build #83179: 09FIXED in 7 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/83179/ [23:04:12] 10Deployment-Systems, 10Architecture, 10Wikimedia-Developer-Summit-2016-Organization, 7Availability: WikiDev 16 working area: Software engineering - https://phabricator.wikimedia.org/T119032#1896437 (10bd808) [23:30:53] 10Continuous-Integration-Config, 6Release-Engineering-Team, 10Flow, 3Collaboration-Team-Current, 5MW-1.25-release: Flow (or tests) broken in REL1_25 - https://phabricator.wikimedia.org/T122025#1896566 (10Mattflaschen) [23:46:31] 7Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: More Flow no-JS browser tests - https://phabricator.wikimedia.org/T114729#1896619 (10Mattflaschen)