[00:13:27] (03CR) 10Paladox: [C: 031] Update composer to 1.0.0-alpha11 [integration/composer] - 10https://gerrit.wikimedia.org/r/258933 (owner: 10Legoktm) [00:19:37] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1879646 (10kaldari) > A second concern is that changes are being pushed directly to Gerrit bypassing review entirely. If t... [00:34:11] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1879675 (10kaldari) @hashar: Whenever I try to force push the reset, I get the error: ! [remote rejected] master -> mas... [00:41:46] 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should be aware of security patches - https://phabricator.wikimedia.org/T118477#1879697 (10thcipriani) [00:42:09] And then ignore them anyway? :D [00:43:42] heh, well, first step is to reliably detect them. [00:47:42] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:46] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:06] jenkins seems rather broken https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/42948/console [00:48:13] idk if it's just me [00:48:37] my patch has nothing to do with visual editor [00:48:47] lol [00:52:35] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39418 bytes in 2.096 second response time [00:52:39] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39745 bytes in 1.356 second response time [00:54:12] aude: not sure what the problem is exactly, seems to be happening quite a bit, but only on integration-slave-trusty-1012, I'm going to take it offline for the time being. [00:54:25] thcipriani: ok :/ [00:54:36] maybe hashar knows (when he's online) [00:57:01] !log marking integration-slave-trusty-1012 offline, strange zuul.cloner behavior. [00:57:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:57:15] the job was probably aborted mid git pull so the repo is busted, and all jobs will continue to fail until someone rm -rf's it [01:01:33] legoktm: just rm rf /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm on 1012? [01:02:02] thcipriani: yeah [01:02:05] as sudo probably [01:02:24] legoktm: kk, lemme give that a try and then I'll bring the done back online. [01:04:15] *bring it back online when I'm done. [02:11:26] 10Continuous-Integration-Infrastructure: Zuul-cloner fails in mediawiki-extensions-hhvm job due to dirty VisualEditor submodule - https://phabricator.wikimedia.org/T121479#1879906 (10Krinkle) 3NEW [02:18:59] 10Continuous-Integration-Infrastructure: Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1879929 (10Krinkle) 3NEW [02:19:11] 10Continuous-Integration-Infrastructure, 7Regression: [Regression] Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1879936 (10Krinkle) [02:25:26] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1879943 (10Krinkle) Still happening, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1879944 (10Krinkle) p:5High>3Unbreak! [02:34:55] !log Ran 'sudo rm -rf /mnt/home/jenkins-deploy/tmpfs/jenk*' on ci slaves via salt [02:35:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:38:16] !log beta-mediawiki-config-update-eqiad jobs have been stuck on 'queued' for the past 3 hours [02:38:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:38:35] greg-g: thcipriani|afk: ^ FYI, beta is not being updated. queue filling in Zuul. [02:38:53] Perhaps time to fix that job The Right Way. It's been over a year now. We know the solution. [02:55:57] Krinkle: g.reg-g's on leave. ostriches is his ping-man right now. [02:56:35] Also, more important than the beta update job, CI for mediawiki/phpunit currently has three unbreak-now regressions causing things to fail 8/10 times. [02:57:03] https://phabricator.wikimedia.org/T121479 , https://phabricator.wikimedia.org/T121481, https://phabricator.wikimedia.org/T120824 [02:57:31] * James_F sighs. :-( [02:59:19] * ostriches feels pinged [02:59:43] Sorry o–striches. [03:14:12] 10Deployment-Systems, 10Architecture, 10Wikimedia-Developer-Summit-2016-Organization, 7Availability: WikiDev 16 working area: Software engineering - https://phabricator.wikimedia.org/T119032#1879994 (10RobLa-WMF) [04:06:25] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:15] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39403 bytes in 0.790 second response time [05:08:16] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:43:11] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [06:04:14] and jenkins is broken again :( [06:05:31] is anyone in sf still awake? [06:11:18] aude: what's broken? [06:17:46] tmp stuff [06:18:09] legoktm: https://integration.wikimedia.org/ci/job/mwext-Wikibase-client-tests-sqlite-hhvm/7170/console [06:18:45] looks like some patches got merged though, maybe if those tests were run on a different host [06:19:02] !log marked integration-slave-trusty-1017 as offline due to tmpfs issue [06:19:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [06:19:08] :/ [06:21:22] not much else I can do... [06:21:47] ok [06:22:04] thanks [06:56:45] PROBLEM - Host angry-caching-proxy is DOWN: CRITICAL - Host Unreachable (10.68.19.184) [07:30:51] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1880105 (10Nikerabbit) [07:34:10] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1880113 (10Nikerabbit) Strict mode is really useful, because it helped me to find a data loss iss... [08:09:53] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1880130 (10jcrespo) > currently trying to do development with strict mode enabled is difficult @... [08:13:01] zeljkof: good morning. Can you cut the new mediawiki_selenium release for Android/IE ( https://phabricator.wikimedia.org/T114362 )? [08:13:10] MultimediaViewer has some IE related tests [08:13:18] and I am sure others do as well :D [08:24:43] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1880136 (10Nikerabbit) Correct. I am using strict mode locally to make sure we don't introduce an... [08:26:50] 10Continuous-Integration-Infrastructure: Zuul-cloner fails in mediawiki-extensions-hhvm job due to dirty VisualEditor submodule - https://phabricator.wikimedia.org/T121479#1880139 (10hashar) The root cause is a canceled build right in the middle of the submodule update operation: https://integration.wikimedia.o... [08:29:10] !log stopping zuul-merger on gallium for maintenance [08:29:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:33:58] 10Continuous-Integration-Infrastructure, 7Regression: [Regression] Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1880144 (10hashar) I guess the repository on gallium has grown out of control. I did a `git gc` on it which removed a bunch of o... [08:34:42] PROBLEM - zuul_merger_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [08:44:42] 10Continuous-Integration-Infrastructure, 7Regression: [Regression] Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1880148 (10hashar) [08:45:24] 10Continuous-Integration-Infrastructure, 7Regression: [Regression] Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1879929 (10hashar) And I just remembered that cloning the whole mediawiki/core on Precise never really worked. It takes ages to... [08:47:04] !log restarted zuul-merger on gallium [08:48:32] RECOVERY - zuul_merger_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [08:51:48] 10Continuous-Integration-Infrastructure, 7Regression: [Regression] Git clone times out for php-composer-package-validate on mediawiki-core - https://phabricator.wikimedia.org/T121481#1880156 (10hashar) fetch has completed and took way more than 10 minutes to write the pack file :( [08:56:59] hashar: will do [09:03:35] zeljkof: came to it overnight looking at MultimediaViewer failures :D [09:04:03] zeljkof: also I fixed a browser test all by myself for the first time ever [09:04:35] zeljkof: a Math browser tests could not find the 'Create source' tab because it has been renamed, so I switched the logic to use a css selector https://gerrit.wikimedia.org/r/#/c/259075/ [09:08:20] hashar: great! :) [09:08:33] so would need some review / +2 :-} [09:08:41] it worked for me locally [09:08:43] and [09:09:07] pass! [09:10:55] hashar: looking [09:20:56] hashar: +2d [09:21:31] zeljkof: thx, will rety the job [09:21:59] great, just wait for the merge to complete [09:30:31] https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/749/console [09:30:36] And I type 3 + 2 # features/step_definitions/math_steps.rb:25 [09:30:36] 09:25:18 unable to locate element, using {:id=>"wpTextbox1", :tag_name=>"textarea"} [09:30:39] WTF [09:36:31] zeljkof: "Could not find link to Sauce Labs job URL." is that a known issue ? :D [09:45:59] hashar: it is a feature :P (looking for phab task) [09:46:20] (03CR) 10Thiemo Mättig (WMDE): "I fixed the issue discussed above, added PHPDoc comments for everything and removed a not needed return. See my diff here: https://gerrit." [tools/code-utils] - 10https://gerrit.wikimedia.org/r/190825 (owner: 10Daniel Kinzler) [09:46:29] zeljkof: the repo uses mediawiki_selenium 1.5.0 [09:47:07] hashar: it is an old issue [09:47:12] probably also in 1.5 branch [09:47:22] https://phabricator.wikimedia.org/T105589 [09:47:23] apparently that is the bug [12:40:42] hashar: working on https://phabricator.wikimedia.org/T114362 [12:41:53] zeljkof: working? isn't it solved already? [12:42:05] hashar: I mean, releasing the gem :) [12:42:12] OHHHHHHH [12:42:26] apologies for the delay [12:50:23] zeljkof: you should be able to rake gem:release [12:50:23] would be neat :-} [12:51:03] hashar: I did a clean install of mac os recently, had to configure this and that before :) [12:53:47] hashar: Could i have help to fix https://gerrit.wikimedia.org/r/#/c/258676/ test. The normal one passes but the test extension composer test dosen't it keeps failing at database error. [12:55:07] It should pass the experimental test for test extension unit composer. Since if it dosen't it would indicate an error in this extension or the required extension. [13:02:16] hashar: looks like it is just "rake release" :) [13:02:24] will test after lunch [13:22:21] 7Browser-Tests, 5Patch-For-Review, 3Reading Web Sprint 60 - Boom Headshot!: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1880623 (10phuedx) 5Open>3stalled @Jdlrobson: The commit message of [246801](https://gerrit.wikimedia.org/r/246801) needs rewording and t... [13:36:48] (03CR) 10Hashar: [C: 032 V: 032] "Lets go and see what happens :-)" [integration/composer] - 10https://gerrit.wikimedia.org/r/258933 (owner: 10Legoktm) [13:37:15] !log bumping composer on CI to 1.0.0-alpha11 https://gerrit.wikimedia.org/r/#/c/258933/ [13:37:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:37:23] legoktm: composer updating :-} [13:39:11] (03CR) 10Hashar: "Mass pulled on slaves via:" [integration/composer] - 10https://gerrit.wikimedia.org/r/258933 (owner: 10Legoktm) [13:44:25] (03PS1) 10Hashar: Revert "Update composer to 1.0.0-alpha11" [integration/composer] - 10https://gerrit.wikimedia.org/r/259241 [13:44:58] (03CR) 10Hashar: [C: 032] "Causes operations/mediawiki-config.git to fail on composer validate." [integration/composer] - 10https://gerrit.wikimedia.org/r/259241 (owner: 10Hashar) [13:45:08] (03CR) 10Hashar: [V: 032] "Causes operations/mediawiki-config.git to fail on composer validate." [integration/composer] - 10https://gerrit.wikimedia.org/r/259241 (owner: 10Hashar) [13:45:43] !log reverted composer upgrade on CI with https://gerrit.wikimedia.org/r/#/c/259241/ [13:45:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:48:45] hashar: ha! i think i found the problem with running npm tests for deploy repos :) [13:49:02] and it has nothing to do with the dev modules, they are being installed correctly [13:53:08] (03CR) 10Paladox: "This error should be reported to composer so it can be fixed." [integration/composer] - 10https://gerrit.wikimedia.org/r/259241 (owner: 10Hashar) [14:01:02] (03PS1) 10Hashar: Fill required fields in /composer.json [integration/composer] - 10https://gerrit.wikimedia.org/r/259242 [14:02:33] (03PS1) 10Hashar: Validate integration/composer with the shipped version [integration/config] - 10https://gerrit.wikimedia.org/r/259243 [14:03:41] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/#/c/259242/ make composer.json validate" [integration/composer] - 10https://gerrit.wikimedia.org/r/258933 (owner: 10Legoktm) [14:04:10] (03PS1) 10Zfilipin: Release patch version 1.6.3 [selenium] - 10https://gerrit.wikimedia.org/r/259244 (https://phabricator.wikimedia.org/T114362) [14:04:11] mobrovac: great [14:04:27] zeljkof: shouldn't it be 1.7.0 ? [14:04:56] hashar: why? "just" one bug (regression) was fixed [14:05:15] it is now actually doing what it used to do [14:05:17] or it can be a new feature of the environment rewrite :-D [14:05:50] I can change to 1.7.0, but I think 1.6.3 is good to [14:07:14] zeljkof: would repo having gem 'mediawiki_selenium', '~> 1.6.2' get it ? [14:07:24] I have no clue what ~> means [14:07:25] hashar: yes [14:07:35] it means change the last digit at will :) [14:07:41] looks like >1.6.0 =< 1.7 [14:07:53] yeah so that would save us from bumping ver everywhere [14:08:12] true [14:08:21] merging 1.6.3? [14:08:25] (03CR) 10Hashar: [C: 032] Release patch version 1.6.3 [selenium] - 10https://gerrit.wikimedia.org/r/259244 (https://phabricator.wikimedia.org/T114362) (owner: 10Zfilipin) [14:08:30] zeljkof: yup [14:08:33] tag it and push! [14:08:56] rake release actually does all the work! [14:09:04] this will be the perfect test [14:10:34] (03Merged) 10jenkins-bot: Release patch version 1.6.3 [selenium] - 10https://gerrit.wikimedia.org/r/259244 (https://phabricator.wikimedia.org/T114362) (owner: 10Zfilipin) [14:19:49] (03PS1) 10Zfilipin: Simplify instructions on releasing the gem [selenium] - 10https://gerrit.wikimedia.org/r/259247 (https://phabricator.wikimedia.org/T114362) [14:33:42] zeljkof: guess you can override the tag and get this one in as well [14:34:01] hashar: there is no rush on this one [14:34:07] it can go into the next release [14:34:12] (03CR) 10Hashar: [C: 032] Simplify instructions on releasing the gem [selenium] - 10https://gerrit.wikimedia.org/r/259247 (https://phabricator.wikimedia.org/T114362) (owner: 10Zfilipin) [14:34:16] go go go [14:34:23] rerelease time :-} [14:34:35] I have just realized we have added the rake target, but forgot to update the docs [14:34:38] same old story [14:34:48] yeah [14:34:54] {bug|1} [14:36:08] but there is no rush in getting the docs updated with the release [14:36:14] it will get into the next on [14:36:15] one [14:36:49] also, I am in the middle of testing if the new gem actually fixes builds ;) https://integration.wikimedia.org/ci/view/BrowserTests/view/MultimediaViewer/ [14:37:10] oh the IE ones [14:37:35] safari too [14:37:49] (03PS1) 10Mobrovac: Bug fix: Node services: fix up the module path in the config [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 [14:38:02] hashar: ^^ [14:38:38] zeljkof: seems head has 1.6.2 already so just retriggering a job would be enough isn't it ? [14:38:44] (03CR) 10jenkins-bot: [V: 04-1] Bug fix: Node services: fix up the module path in the config [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [14:38:58] samn you tox [14:39:00] mobrovac: oh well hmm [14:39:02] hashar: hm, good idea, will try [14:39:03] (03Merged) 10jenkins-bot: Simplify instructions on releasing the gem [selenium] - 10https://gerrit.wikimedia.org/r/259247 (https://phabricator.wikimedia.org/T114362) (owner: 10Zfilipin) [14:39:04] s/samn/damn/ [14:39:35] hashar: what? [14:40:12] those relatives paths are killing me [14:40:27] hashar: no, still locked on 1.6.2, will update Gemfile.lock https://integration.wikimedia.org/ci/view/BrowserTests/view/MultimediaViewer/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/660/console [14:40:34] mobrovac: I think each of parsoid / cxserver have a slightly difference :-/ [14:40:55] that is why we had to add a NPM_SET_PATH variable or something and cd $NPM_SET_PATH [14:41:07] then have parsoid use src/ but cxserver uses . [14:41:10] or something along those lines [14:41:11] :-/ [14:41:47] hashar: services based on service-runner have all that standardised and the idea is to move parsoid to that eventually as well [14:42:01] hashar: both have "src" as their submodules [14:42:07] zeljkof: I guess it is because of the .lock file having mediawiki_selenium (1.6.2) [14:42:17] hashar: yes [14:42:47] mobrovac: ok ok [14:42:55] mobrovac: will look at it tomorrow :-/ [14:42:58] :P [14:43:03] mobrovac: gotta flip papers / accounting [14:43:09] i see [14:43:31] or we can merge/deploy it and pray [14:46:17] hashar: mobrovac: Christmas time is the perfect time for prayer [14:46:53] hashar: i'm preparing PS2 for it which should work for parsoid as well, hang on [14:47:42] zeljkof: nah, too many people praying this time of year, god might miss ours :D [14:48:39] mobrovac: excuses, excuses.. it's like saying this is a bad time to send letter to Santa! ;) [14:48:49] hahaha [14:48:52] touché zeljkof [14:49:37] hashar: quick q, NPM_SET_PATH env var is exported, right, so it should be available when running npm-install-dev.py, right ? [14:49:46] s/right// [14:53:32] mobrovac: I have recently done a clean install of mac os, and now I have no problems with the recommended vagrant setup :/ [14:53:43] :) [14:53:46] cool! [14:53:54] looks like the old machine had something strange, but I could not figure out what [14:54:04] anyway, just tested it, works fine [14:54:28] mobrovac: in theory yeah [14:54:43] mobrovac: originally the job was created for the parsoid repo iirc [14:54:57] jk [14:55:04] mobrovac: then kart_ wanted the same for cxserver and since the repos slightly varied we added a few hacks in the CI jobs :-/ [14:55:07] now [14:55:17] we have a nice messy job runner :D [14:55:43] mobrovac: probably the easiest is to look at the console output for parsoid and cxserver /deploy runs [14:55:53] I think most shell scripts have set -x [14:56:01] sorry :-( [14:56:09] but willing to get them easier [14:56:49] no worries, hashar, i'll manage :) [14:57:08] I am sure you will [14:57:25] I am just pissed up that the lame tech debt I have helped create consume smart brains cycles [15:00:08] mobrovac: so I think the trick was to run the server after doing a cd to the proper path [15:00:18] so the config stanza ends up point to the proper place [15:00:20] hashar: When running the composer extension unittests where is the files located for example is it in vendor. so for example in extension that use the test it would look like example/vendor or is it located in mediawiki core. I need help to fix the failing of the extension unit test composer at https://gerrit.wikimedia.org/r/#/c/258676/ [15:00:29] https://integration.wikimedia.org/ci/job/mwext-testextension-zend-composer/507/console [15:00:42] hashar: that's a parsoid quirk i think, but that's manageable [15:01:04] hashar: for service-runner services, where you run npm from is irrelevant [15:01:58] hashar: IE green so far :D https://integration.wikimedia.org/ci/view/BrowserTests/view/MultimediaViewer/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/661/console [15:07:11] mobrovac: good to know ! [15:07:38] paladox: composer is run from the root of mediawiki/core workspace [15:08:08] paladox: in mediawiki/core the composer.json has the 'composer-merge-plugin' which traverse the extensions/ directory and look for composer.json files [15:08:23] paladox: the plugin grab all dependencies and merge them with the ones in mediawiki/core [15:08:25] then composer run [15:09:04] Hashar: Ok thanks for explaining that [15:09:06] Hashar: Ok thanks. Do you know how i can fix the test since keeps failing at database because of required semantic mediawiki. [15:09:38] paladox: no idea :(( maybe the extension miss the proper install magic to add the missing table [15:09:48] paladox: or the smw_object_ids is installed by some other extension [15:10:23] hashar: Oh ok. Yes they say when using composer you can skip to using the update script or if you do it manualy you have to go through the special page. [15:10:48] paladox: apparently the installer first create a default page "Creating main page with default content" [15:11:12] hashar: Would that mean it skips the update process making the test fail. [15:11:13] paladox: but the extension seems to hook into the page creation process , and its code is executed before install.php had a chance to populate the table [15:11:35] paladox: then, I am not sure why the non composer job works just fine [15:12:14] zeljkof: still looks good so far [15:12:17] hashar: Oh, How would i get install.php to be executed first. Thats because the non composer dosen't execute the composer.json file. So it isen't installing semantic mediawiki. [15:12:38] zeljkof: maybe we can restore IE / Android browser tests jobs that have been deleted? [15:12:48] hashar: yes, all green so far, I have started other IE/Safari MMV jobs [15:13:01] hashar: only if the repos ask for them :D [15:14:15] zeljkof: then you want to announce somewhere that Android / IE are supported so dev ask for them :-D [15:14:43] hashar: writing mail to qa list as I type this ;) [15:14:53] paladox: I have no idea :-( [15:15:02] hashar: Ok. [15:15:11] paladox: probably want to reproduce locally first [15:15:14] that would ease debugging [15:15:24] Ok. [15:16:33] (03PS2) 10Mobrovac: Bug fix: Node services: fix up the module path in the config [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 [15:16:45] hashar: i checked ^^ locally with both parsoid and cxserver deploy repos [15:17:12] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1880804 (10hashar) In https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/PageAssessments,access I have g... [15:17:45] now [15:19:04] mobrovac: ahh I got it, so the service-node magic always uses node_modules/.. right ? [15:19:11] regardless of current working directory? [15:20:00] hashar: it uses the node_modules/ dir as the ref point, and assumes the service is located in node_modules/../ which is true for src repos, but not deploy ones [15:20:07] hashar: so, yes, exactly that :) [15:20:13] /srv/deployment/integration/slave-scripts/bin/npm-install-dev.py [15:20:13] [15:20:13] cd "$NPM_SET_PATH" # for deploy: ./src [15:20:13] npm test [15:20:29] so maybe nom-install-dev should be run after cd "$NPM_SET_PATH" ? [15:20:36] no no [15:20:38] so it will be run from /src/ [15:20:55] it doesn't matter, what matters is the location of node_modules, not the CWD [15:21:30] and for deploy jobs we use: [15:21:31] ln -fs "$WORKSPACE/node_modules" "$NPM_SET_PATH/node_modules" [15:21:38] (yeah more complications) [15:22:58] hashar: running the script from deploy's root dir is ok now with this patch, no reshuffling needed :) [15:23:23] ok ok [15:23:25] well [15:23:29] lets deploy that [15:23:43] let's! [15:23:45] (03CR) 10Hashar: [C: 032] "Moaaar tech debt :-}" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [15:23:56] zeljkof: stand by for praying [15:23:57] :P [15:24:06] so for reference, the deployment is done on integration-saltmaster [15:24:14] hashar: :D [15:24:14] ah kk [15:24:35] (03Merged) 10jenkins-bot: Bug fix: Node services: fix up the module path in the config [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [15:24:35] and you use [15:24:36] root@integration-saltmaster:~# salt '*slave*' cmd.run 'cd /srv/deployment/integration/slave-scripts && git pull' [15:24:54] mobrovac: :D [15:24:59] up-to-date [15:25:14] mobrovac: can you retry some Parsoid / cxserver and whatever patchsets on /deploy ? [15:25:16] k, will send a recheck for cxserver [15:25:24] yup [15:25:25] * mobrovac doing it [15:25:27] should be all about commenting 'recheck' on the latest merged change of each /deploy.git repos [15:25:35] if it is screwed up, gotta revert / rinse repeat [15:25:42] revert is a button click, then +2 [15:25:53] then salt '*slave*' cmd.run 'cd /srv/deployment/integration/slave-scripts && git pull' [15:25:59] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5Patch-For-Review: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1880820 (10zeljkofilipin) [15:27:02] mobrovac: gotta rush out, be back in half an hour [15:27:12] kk [15:27:20] Yippee, build fixed! [15:27:20] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #814: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/814/ [15:29:56] hashar: build of https://gerrit.wikimedia.org/r/#/c/258435/ succeeded :))) [15:30:27] goood [15:30:30] zeljkof: congrats [15:30:57] mobrovac: you probably want to copy paste the validation results on the task [15:31:03] hashar: sorry, lost the conversation, congrats on what? [15:31:12] I am heading outside for a few, be back in half an hour [15:31:19] zeljkof: -os_x_10.9-safari-sauce build #814: FIXED [15:31:31] hashar: :D [15:31:34] did not even notice [15:31:35] grat [15:31:36] great [15:31:44] brb [15:32:17] (03CR) 10Hashar: "Forced update with:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [15:39:16] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5Patch-For-Review: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1880847 (10zeljkofilipin) [15:47:14] (03CR) 10Mobrovac: "Tested mathoid and cxserver and both deploy repos' test jobs succeed:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [16:26:00] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1880923 (10hashar) a:3NiharikaKohli @NiharikaKohli phased out the whole history in favor of a single change introducing... [16:26:03] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1880925 (10hashar) 5Open>3Resolved [16:26:39] (03CR) 10Hashar: "Great thank you to have tested!" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/259251 (owner: 10Mobrovac) [16:33:39] !log cleared /mnt/home/jenkins-deploy/tmpfs/jenkins-2 from integration-slave-trusty-1017 and added it back to the pool [16:33:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:35:09] (03CR) 10Hashar: [C: 032] "Feel free to poke #wikimedia-releng to get them deployed faster :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/259146 (owner: 10Gilles) [16:36:42] (03Merged) 10jenkins-bot: Add thumbor/svg-engine [integration/config] - 10https://gerrit.wikimedia.org/r/259146 (owner: 10Gilles) [16:39:29] (03CR) 10Hashar: "Left to do is to create both tmp directories to make sure they belong to jenkins-deploy user." (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:39:33] (03PS2) 10Hashar: Don't leave one of the TMPDIRs around when it is switched inbetween [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:47:45] (03PS3) 10Hashar: Create both possible TMPDIR so they belong to jenkins-deploy [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:48:10] (03CR) 10Hashar: [C: 032] "Made it to always create both directories" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:49:32] (03Merged) 10jenkins-bot: Create both possible TMPDIR so they belong to jenkins-deploy [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:50:26] (03CR) 10Hashar: "Force updated with:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258634 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki) [16:50:51] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1880981 (10hashar) p:5Unbreak!>3High https://gerrit.wikimedia.org/r/#/c/258634/ should fix it. [17:09:28] (03PS1) 10Mobrovac: Mathoid: Use mathoid-deploy-npm for the deploy repo patches [integration/config] - 10https://gerrit.wikimedia.org/r/259282 [17:10:31] (03CR) 10jenkins-bot: [V: 04-1] Mathoid: Use mathoid-deploy-npm for the deploy repo patches [integration/config] - 10https://gerrit.wikimedia.org/r/259282 (owner: 10Mobrovac) [17:12:13] (03PS2) 10Mobrovac: Mathoid: Use mathoid-deploy-npm for the deploy repo patches [integration/config] - 10https://gerrit.wikimedia.org/r/259282 [17:21:31] hashar: ooops. I'll look into why the composer update totally failed [17:50:36] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 6Discovery, 3Discovery-Cirrus-Sprint: ElasticSearch taking 17% of RAM on integration slaves - https://phabricator.wikimedia.org/T89083#1881301 (10Deskana) From what I can see, #discovery has resolved this on our end as we said that ElasticSearch can... [17:50:40] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 6Discovery: ElasticSearch taking 17% of RAM on integration slaves - https://phabricator.wikimedia.org/T89083#1881302 (10Deskana) [18:18:50] 6Release-Engineering-Team, 6Phabricator: "Discovery" project does not show up in the list of projects in "Projects" field - https://phabricator.wikimedia.org/T121551#1881514 (10Smalyshev) 3NEW [18:29:33] Any idea what's wrong with our repo? https://gerrit.wikimedia.org/r/#/c/258920/ [18:29:40] > This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. [18:29:57] There's no submodule bump, and nothing to rebase... [18:30:20] legoktm: no worries about composer update. It is hard to figure out what is going to break [18:30:40] legoktm: for ops/mediawiki-config , I guess it has some Symphony console that conflicts with what is shipped by composer itself :-\ [18:31:00] it definitely does not [18:31:09] just my 30'000 view of ants [18:31:13] I think I botched the update somehow [18:31:17] 30'000 feet [18:31:43] anyway, I proposed a Jenkins job that would run /vendor/bin/composer from integration/composer https://gerrit.wikimedia.org/r/#/c/259243/ [18:31:45] might help [18:32:06] we would need to update /composer.json to add some required fields which is https://gerrit.wikimedia.org/r/#/c/259242/ [18:32:38] and finally, for the tmpfs issue you observed yesterday, Jan has a fix and I deployed it a few hours ago. Should be fine now (ref is https://phabricator.wikimedia.org/T120824 ) [18:33:10] merely brain dumping for the whole audience. I am heading out for rest of evening [18:33:37] http://fpaste.org/301249/raw/ totally broken :P [18:33:46] hashar: yay about tmpfs :D [18:34:56] PROBLEM - Puppet failure on deployment-puppetmaster is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:35:44] legoktm: yup :-} good luck with the trace! [18:35:50] I am off! [18:36:03] o/ [18:37:18] * hashar waves [18:44:59] RECOVERY - Puppet failure on deployment-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [18:47:01] 6Release-Engineering-Team, 10Phabricator-Upstream: Exact matches should always win when suggesting/auto-completing - https://phabricator.wikimedia.org/T76732#1881635 (10Smalyshev) [20:29:37] 6Release-Engineering-Team: Where should we branch for Wikimedia wikis? - https://phabricator.wikimedia.org/T121570#1882301 (10Reedy) 3NEW [20:30:25] 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1882314 (10kaldari) @NiharikaKohli: Feel free to go ahead and a submit a patch with the rest of the (pre-Job Queue) code. [21:16:15] hello [21:16:21] wanted to notify that I'm removing the puppetmaster LDAP variable [21:16:27] and role::puppet::self LDAP class [21:16:31] from all deployment-prep instances [21:16:36] because they are already set in hiera [21:16:40] so this is a noop [21:16:42] and cleans up LDAP [21:17:05] in the future, please do not select role::puppet::self when creating instances in deployment-prep [21:17:08] as that is not needed [21:17:10] thank you [21:17:23] ShiveringPanda: noted, thanks! [21:17:48] for further queries, please send a postcard to: Depressing dark corner, 149 New Montgomery St, SF, USA [21:18:56] done [21:19:03] I'm doing the same for integration project as well [21:21:36] Yippee, build fixed! [21:21:37] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #100: 09FIXED in 5 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/100/ [21:22:33] I was wondering why it was set in both places [21:22:45] historical raisins. LDAP was before Hiera's time [21:23:12] https://phabricator.wikimedia.org/T101447 is tracking ticket for the boring work of killing them all [21:26:16] oh, of course [21:36:13] krenair@tools-bastion-01:~$ ldaplist -l hosts | grep puppetVar | grep -v instancename= | grep -v instanceproject= | wc -l [21:36:13] 238 [21:38:11] Krenair: how many unique puppetvars tho? [21:38:55] krenair@tools-bastion-01:~$ ldaplist -l hosts | grep puppetVar | grep -v instancename= | grep -v instanceproject= | grep -o "puppetVar: .*=" | sort -d | uniq | wc -l [21:38:55] 36 [21:39:10] that isn't too bad. can you pastebin them to me? [21:40:44] ShiveringPanda, http://paste.ubuntu.com/14037495/ [21:41:04] that's just the output of ldaplist -l hosts | grep -o "puppetVar: .*=" | sort -d | uniq [21:41:14] which tbh you could've done yourself :p [21:41:33] indeed [21:41:37] but you were already there :P [21:46:20] lazy :P [21:46:51] best kind of 'zy' [21:47:02] I have gotten rid of 'puppetmaster' from about 60 hosts tho [21:58:07] Krenair: lol, 'download as text' rquires I login with my ubuntu account?! [21:58:14] wut [21:58:22] ShiveringPanda, honestly I just used the pastebinit command [21:58:35] it ended up going to ubuntu [21:58:38] yeah [21:58:43] I was lolling at ubuntu [22:24:49] Krenair: down to 21 uniqs now [22:28:17] 10Deployment-Systems, 3Scap3: sync-wikiversions not syncing wikiversions.json with mira - https://phabricator.wikimedia.org/T121585#1883019 (10thcipriani) 3NEW [22:55:03] PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:57:41] down to 18! [22:57:47] aaand break [22:57:52] !log On scandium created /srv/ssd/zuul/git/wikimedia/fundraising/crm repo manually. Namespace conflict with wikimedia/fundraising/crm/civicrm.git which prevented zuul-merger to clone the arm repo [22:57:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:02:58] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1883118 (10Krinkle) [23:07:34] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1883140 (10Krinkle) This is blocking various things: * Updating MediaWiki core to use PHPUnit specified in composer.json. * Which in turn blocks o... [23:10:41] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1883162 (10Krinkle) [23:10:52] 10Continuous-Integration-Infrastructure: make CI able to run PHPUnit from composer instead of a system wide installation - https://phabricator.wikimedia.org/T112867#1883166 (10Krinkle) [23:10:53] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1649314 (10Krinkle) [23:10:57] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1649314 (10Krinkle) [23:31:05] 10Deployment-Systems, 3Scap3: Require sanity test to pass before syncing files to all web servers - https://phabricator.wikimedia.org/T121597#1883295 (10Krinkle) 3NEW [23:31:09] Reedy: ^ [23:35:01] RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0] [23:57:10] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#1883389 (10bd808) >>! In T112895#1710002, @bd808 wrote: > > The only way I know to pick and choose updates is with the `composer update vendor/pack... [23:58:34] 10Continuous-Integration-Infrastructure, 7HHVM, 5Patch-For-Review: "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2 - https://phabricator.wikimedia.org/T98876#1883394 (10Krinkle)