[00:15:07] PROBLEM - Host deployment-bastion is DOWN: CRITICAL - Host Unreachable (10.68.16.58) [00:28:53] (03CR) 10Krinkle: "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/270545 (https://phabricator.wikimedia.org/T99982) (owner: 10Krinkle) [00:34:37] Whoa what does it mean when Jenkins reports a test result as "UNSTABLE" ?! [00:34:39] https://gerrit.wikimedia.org/r/#/c/269305/ [00:35:58] RoanKattouw: Interesting [00:36:32] Krenair: I think I may've just fixed a 4 year old by accident [00:36:35] Which made that happen [00:36:46] > 19:59:03 Tests: 10785, Assertions: 68434, Incomplete: 15, Skipped: 103. [00:37:48] Krinkle, you accidentally fixed a 4-year-old? [00:37:50] :) [00:38:26] bug [00:38:30] 4 year old bug [00:38:52] swapping phpunit 3.7.17 from our git install to that of composer may've updated some indirect dependency [00:39:02] which made junit recognise something as unstable e.g. incomplete tests [00:39:04] I'll check [00:40:08] Test report file /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/log/junit-phpunit-allexts.xml was length 0 [00:40:13] That made it untable [00:40:20] K, not caused by me I think [00:40:56] I feel like I'm missing a lot of context here, but.. ok [00:41:53] Krenair: I meant to ping Roan but instead pinged myself, but failed to do that and pinged you [00:42:57] ah :) [00:44:14] lol [00:44:35] Well that's nice, but the incomplete tests are in Wikidata [00:44:43] I'll try +2ing it again to see if it's intermittent [00:44:52] But if not, all merging is broken until Wikidata removes their 0-assertion tests [00:46:46] incomplete isn't the issue [00:46:50] the issue is the junit file is empty [00:46:55] instead of containing proper summary [00:47:16] looking into it [00:55:50] RoanKattouw: seems it's not deterministically failing [00:55:54] did it work after retry? [00:56:03] the hhvm job junit fail [00:56:15] I'll retry now [01:00:08] Well, something else failed alright [01:00:11] This one is even more fun: https://integration.wikimedia.org/ci/job/mwext-Echo-testextension-php55/86/console [01:00:27] I'm gonna take a break, but I'll be back in 1-2 hours and I'll file bugs for all the weird intermittent CI issues I've seen today [01:00:46] It's hard to get anything to V+2 and the error is different every time [01:01:05] Yeah [01:01:09] Can't help you there :/ [01:01:18] No worries [01:01:21] Doesn't seem to be caused by my recent change so I'll leave it for now [01:01:35] Between that and all the ecmabot-wm errors spamming my inbox I feel like all of our infrastructure is falling apart around us [01:01:41] So I'm gonna go outside in the last hour of daylight and come back later [01:09:52] 10Continuous-Integration-Infrastructure: Remove Junit plugin from Jenkins installation - https://phabricator.wikimedia.org/T94685#2030105 (10Krinkle) It's also used by mediawiki extension jobs that run mw-run-phpunit-allexts. The following is a frequently used pattern: ``` - mw-run-phpunit-allexts pub... [01:12:32] RoanKattouw_away: oh, you get those ecmabot failures too? [01:12:39] I just noticed it was down so I instructed labs to reboot it [01:12:42] but it ain't working [01:13:54] disabled for now [01:19:59] 10Continuous-Integration-Infrastructure: mediawiki-extensions-php55 broken for Echo (and possibly others) with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2030115 (10Krinkle) [01:20:52] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2030120 (10Krinkle) p:5Triage>3Normal [01:45:27] (03PS1) 10Krinkle: Convert mediawik-core-php53lint to a shallow clone [integration/config] - 10https://gerrit.wikimedia.org/r/270885 [01:46:22] (03CR) 10Krinkle: [C: 032] "Fixed REL1_25 mediawiki core passing." [integration/config] - 10https://gerrit.wikimedia.org/r/270885 (owner: 10Krinkle) [01:47:48] (03Merged) 10jenkins-bot: Convert mediawik-core-php53lint to a shallow clone [integration/config] - 10https://gerrit.wikimedia.org/r/270885 (owner: 10Krinkle) [01:50:43] (03PS1) 10Siebrand: Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 [01:51:50] (03CR) 10jenkins-bot: [V: 04-1] Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [01:53:47] (03PS2) 10Siebrand: Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 [01:53:54] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [01:54:49] (03PS3) 10Siebrand: Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 [01:54:51] (03CR) 10jenkins-bot: [V: 04-1] Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [01:54:53] RoanKattouw_away: One of those bugs was actually mine. The others I don't know. Fixed that one just now :) [01:55:07] (the one that made mwext-Echo-textexensions-* fail) [01:55:14] o/ [01:55:39] (03CR) 10jenkins-bot: [V: 04-1] Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [03:19:59] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [04:31:21] Whoa WTF, it gets crazier [04:31:21] 04:30:07 ..................................[Tue Feb 16 04:30:07 2016] [hphp] [3849:7f5c3ec79d00:0:000001] [] [04:31:21] 04:30:07 Warning: File not found: /mnt/jenkins-workspace/workspace/mwext-Echo-testextension-hhvm/src/vendor/phpunit/phpunit/PHPUnit/Util/ErrorHandler.php in /mnt/jenkins-workspace/workspace/mwext-Echo-testextension-hhvm/src/vendor/composer/ClassLoader.php on line 412 [04:31:21] 04:30:07 [Tue Feb 16 04:30:07 2016] [hphp] [3849:7f5c3ec79d00:0:000002] [] [04:31:23] 04:30:07 Warning: Invalid argument: function: class not found in /mnt/jenkins-workspace/workspace/mwext-Echo-testextension-hhvm/src/vendor/composer/ClassLoader.php on line 412 [04:31:24] 04:30:07 [Tue Feb 16 04:30:07 2016] [hphp] [3849:7f5c3ec79d00:0:000003] [] [04:31:26] 04:30:07 Fatal error: Class undefined: PHPUnit_Framework_SkippedTestError in /mnt/jenkins-workspace/workspace/mwext-Echo-testextension-hhvm/src/vendor/phpunit/phpunit/PHPUnit/Framework/Assert.php on line 2821 [04:31:37] Krinkle: Did you do something that could have caused that by any chance? [04:42:25] 10Continuous-Integration-Infrastructure: mediawiki-extensions-php55 broken for Echo (and possibly others) with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2030213 (10Catrope) Happened for MW core too: https://integration.wikimedia.org/ci/job/mediawiki-extension... [04:42:36] 10Continuous-Integration-Infrastructure: mediawiki-extensions-php55 fails intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2030214 (10Catrope) [04:45:31] 10Continuous-Integration-Infrastructure: composer-hhvm fails intermittently with "composer.json not found" - https://phabricator.wikimedia.org/T127031#2030215 (10Catrope) 3NEW [04:48:59] 10Continuous-Integration-Infrastructure: phpunit jobs fail intermittently but frequently with "unknown database" errors - https://phabricator.wikimedia.org/T127032#2030223 (10Catrope) 3NEW [05:20:02] Project beta-update-databases-eqiad build #6512: 04FAILURE in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6512/ [08:50:27] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: "Exception encountered, of type "Exception"" on (Beta) Special:CentralAuth/Riley_Huntley - https://phabricator.wikimedia.org/T127039#2030461 (10Josve05a) 3NEW a:3Riley_Huntley [08:50:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: "Exception encountered, of type "Exception"" on (Beta) Special:CentralAuth/Riley_Huntley - https://phabricator.wikimedia.org/T127039#2030469 (10Josve05a) a:5Riley_Huntley>3None [08:50:59] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: "Exception encountered, of type "Exception"" on (Beta) Special:CentralAuth/Riley_Huntley - https://phabricator.wikimedia.org/T127039#2030461 (10Josve05a) [08:54:05] 7Browser-Tests, 10Math, 5Patch-For-Review: Math Selenium test fails with unable to locate element, using {:id=>"wpTextbox1", :tag_name=>"textarea"} (Watir::Exception::UnknownObjectException) - https://phabricator.wikimedia.org/T126463#2030473 (10hashar) The patch has fixed it apparently. Both jobs are still... [08:56:10] 10Continuous-Integration-Infrastructure, 10Mathoid, 5Patch-For-Review: Enable node4 for mathoid tests - https://phabricator.wikimedia.org/T124447#2030475 (10hashar) https://gerrit.wikimedia.org/r/#/c/270547/1..2/package.json !! Congratulations on fixing the git URL :-} [09:00:08] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: rebuild deployment-bastion on trusty - https://phabricator.wikimedia.org/T126537#2030482 (10hashar) Looks like this task is a success. The Jenkins jobs updating the beta cluster seems to be working more or less properly. [09:15:27] 10Beta-Cluster-Infrastructure: Output of wmf-beta-update-databases.py is not clear on errors - https://phabricator.wikimedia.org/T110407#2030488 (10hashar) ``` if p.wait() > 0: raise Exception("command: ", cmd, "output: ", f.read()) f.seek(0) print f.read().strip() f.c... [09:30:15] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: Output of wmf-beta-update-databases.py is not clear on errors - https://phabricator.wikimedia.org/T110407#2030497 (10hashar) a:3hashar [09:31:42] (03PS1) 10Florianschmidtwelzow: Remove ApiSandbox from branching configuration [tools/release] - 10https://gerrit.wikimedia.org/r/270903 [09:33:20] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #520: 15ABORTED in 9 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/520/ [09:33:24] 10Beta-Cluster-Infrastructure: Beta cluster update database is broken - https://phabricator.wikimedia.org/T127040#2030500 (10hashar) 3NEW [09:33:47] 7Browser-Tests, 10Wikidata, 5Patch-For-Review: Sitelink browser test sometimes fails with firefox - https://phabricator.wikimedia.org/T126585#2030508 (10hoo) Change deployed, please verify. [09:41:50] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10DBA, 10Flow: Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql - https://phabricator.wikimedia.org/T127040#2030517 (10hashar) [09:42:39] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10DBA, 10Flow: Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql - https://phabricator.wikimedia.org/T127040#2030522 (10hashar) Caused by change https://gerrit.wikimedia.org/r/#/c/270755/ for... [09:45:27] (03PS1) 10Hoo man: Add Capiunto to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/270904 (https://phabricator.wikimedia.org/T126399) [09:47:17] (03PS2) 10Hoo man: Add Capiunto to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/270904 (https://phabricator.wikimedia.org/T126399) [09:49:31] (03CR) 10Hashar: [C: 032] "Been on beta for a while ( T93418 )" [tools/release] - 10https://gerrit.wikimedia.org/r/270904 (https://phabricator.wikimedia.org/T126399) (owner: 10Hoo man) [09:50:27] (03Merged) 10jenkins-bot: Add Capiunto to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/270904 (https://phabricator.wikimedia.org/T126399) (owner: 10Hoo man) [09:52:38] !log will cut the wmf branches this afternoon starting around 14:00 CET [09:52:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:53:47] 7Browser-Tests, 10Math, 5Patch-For-Review: Math Selenium test fails with unable to locate element, using {:id=>"wpTextbox1", :tag_name=>"textarea"} (Watir::Exception::UnknownObjectException) - https://phabricator.wikimedia.org/T126463#2030556 (10zeljkofilipin) @hashar: I have noticed that the tests are still... [09:54:41] 6Release-Engineering-Team, 5WMF-deploy-2016-02-16_(1.27.0-wmf.14): MW 1.27.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T125597#2030558 (10hashar) I will cut the wmf branches this afternoon starting around 13:00 UTC (14:00 CET). [09:56:48] 10Browser-Tests-Infrastructure: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2030560 (10zeljkofilipin) 3NEW a:3zeljkofilipin [09:59:41] (03PS1) 10Zfilipin: Logging in via LoginPage#login_with is a security risk [selenium] - 10https://gerrit.wikimedia.org/r/270905 (https://phabricator.wikimedia.org/T127042) [10:01:07] 10Browser-Tests-Infrastructure, 5Testing-Initiative-2015, 7JavaScript, 5Patch-For-Review: Experiment with browser testing in other software languages - https://phabricator.wikimedia.org/T108874#2030584 (10zeljkofilipin) a:5zeljkofilipin>3None [10:02:21] 10Browser-Tests-Infrastructure, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#2030587 (10zeljkofilipin) a:5zeljkofilipin>3None [10:02:44] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 7Blocked-on-Operations, and 2 others: Beta: submodule update reverts new portals commits - https://phabricator.wikimedia.org/T126061#2030588 (10hashar) Thanks @ksmith and @debt. Looks that is a pragmatic compromise. In the first place I... [10:04:03] 10Browser-Tests-Infrastructure, 5Patch-For-Review: New version of page-object Ruby gem is breaking CI - https://phabricator.wikimedia.org/T126354#2030590 (10zeljkofilipin) a:5zeljkofilipin>3None [10:04:35] 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#2030593 (10zeljkofilipin) a:5zeljkofilipin>3None [10:09:55] 6Release-Engineering-Team, 5WMF-deploy-2016-02-16_(1.27.0-wmf.14): MW 1.27.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T125597#2030611 (10hashar) [10:18:16] PROBLEM - salt-minion processes on scandium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [10:20:30] Yippee, build fixed! [10:20:30] Project beta-update-databases-eqiad build #6520: 09FIXED in 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/6520/ [10:24:32] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2030643 (10zeljkofilipin) [10:29:02] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms [10:30:44] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [10:31:16] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2030669 (10zeljkofilipin) [10:52:22] hashar: do you remember why we run core selenium tests in a special way? [10:52:45] using @clean cucumber tag instead of the usual @en.wikipedia.beta.wmflabs.org [10:53:01] zeljkof: which tests ? [10:53:03] example: https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/903/consoleFull [10:53:06] @clean I have no idea [10:53:23] I am cleaning up something (no pun intended) and puzzled with it [10:53:25] maybe it was a way to flag tests known as good for daily run [10:53:30] ok, will try to clean up @clean tag [10:53:38] well, all of them are fine... [10:53:45] just run them on my machine [10:53:49] will test [10:54:00] you can browse in the integration/config repo maybe [10:54:54] will do [10:55:22] zeljkof: 76ef47a416bc09355601baf213d3a5520bced258 [10:55:22] zeljkof: that points to you :-} [10:55:27] Run only scenarios tagged both @clean and @phantomjs [10:55:32] This change will make the job pass and it will also make it way faster, [10:55:32] since just a few scenarios are tagged @clean at the moment. [10:55:32] [10:55:40] hm [10:55:45] that is for the old qa-browsertests stuff [10:55:55] but why didn't we just tag it the normal way...? [10:55:57] git log -S@clean [10:56:16] imho the idea is that you add tests for local testing / or not ready yet [10:56:24] I know! [10:56:25] and @clean was a way to flag them as ready for daily runs [10:56:32] so maybe that can be removed now [10:56:32] people from core were complaining [10:56:46] that the test are containing wikipedia specific code :P [10:57:17] e11a02056f19dbc16a4c786418a05d1ab6806c8b [10:58:02] how do I find bugzilla bug 67616 in phab? [10:58:20] hm, or is it phab task? [10:58:53] ok, it is not phab task [10:59:27] found it https://phabricator.wikimedia.org/T69616 [11:00:40] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms [11:23:06] zeljkof: I just https://bugzilla.wikimedia.org/ [11:23:09] and that redirect [11:23:13] or add 2000 [11:23:15] ;D [11:23:17] I am out [11:23:22] luncchhh [11:23:25] thanks [11:23:35] remembered that myself after a while [11:28:09] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [11:30:54] !log T117710 integration-saltmaster:~# salt -v '*slave-trusty*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm-composer/src/skins/BlueSky' [11:30:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:30:57] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: some tests run from mwext-testextension-hhvm will pick up files from extensions that were not checked out for this job - https://phabricator.wikimedia.org/T117710#2030783 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.o... [11:43:33] (03PS2) 10Zfilipin: Logging in via LoginPage#login_with is a security risk [selenium] - 10https://gerrit.wikimedia.org/r/270905 (https://phabricator.wikimedia.org/T127042) [11:44:38] (03CR) 10Zfilipin: "Patch set 2 removes the code that required the deleted file." [selenium] - 10https://gerrit.wikimedia.org/r/270905 (https://phabricator.wikimedia.org/T127042) (owner: 10Zfilipin) [11:51:00] Project beta-scap-eqiad build #90201: 04FAILURE in 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90201/ [12:01:55] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2030833 (10zeljkofilipin) [12:03:31] hashar: It seems permission denied for mysql. [12:03:32] 11:55:47 Function: MediaWikiTestCase::listTables [12:03:32] 11:55:47 Error: 1049 Unknown database 'jenkins_u0_mw' (127.0.0.1:3306) [12:03:37] https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/745/console [12:03:50] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/31053/console [12:04:11] or legoktm or jzerebecki or krinkle ^ [12:04:11] Yippee, build fixed! [12:04:11] Project beta-scap-eqiad build #90203: 09FIXED in 6 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90203/ [12:07:19] (03PS1) 10Paladox: [translatewiki] Switch from php53lint to php55lint [integration/config] - 10https://gerrit.wikimedia.org/r/270937 [12:14:03] (03CR) 10Nikerabbit: "See T97889" [integration/config] - 10https://gerrit.wikimedia.org/r/270937 (owner: 10Paladox) [12:17:15] (03PS1) 10Paladox: Add new hhvm lint and use it [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) [12:17:22] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #521: 04STILL FAILING in 2 hr 43 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/521/ [12:18:04] (03Abandoned) 10Paladox: [translatewiki] Switch from php53lint to php55lint [integration/config] - 10https://gerrit.wikimedia.org/r/270937 (owner: 10Paladox) [12:18:45] (03CR) 10jenkins-bot: [V: 04-1] Add new hhvm lint and use it [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) (owner: 10Paladox) [12:21:16] (03PS2) 10Paladox: Add new hhvm lint and use it [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) [12:24:13] 7Browser-Tests, 10Wikidata, 5Patch-For-Review: Sitelink browser test sometimes fails with firefox - https://phabricator.wikimedia.org/T126585#2030866 (10adrianheine) Not fixed :/ [12:25:34] (03PS3) 10Paladox: Add new hhvm lint and use it [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) [12:27:12] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2030874 (10Aklapper) [12:27:51] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1336469 (10Aklapper) [12:28:46] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#2030888 (10zeljkofilipin) [12:29:53] 10Continuous-Integration-Config, 7Easy, 5Patch-For-Review: translatewiki.net phplint job should use HHVM to lint (that is what prod is using) - https://phabricator.wikimedia.org/T97889#2030889 (10Paladox) Hi instead of using php5.4 features i think it will either use php5.5 or 5.6 features but not sure which... [12:39:05] (03PS1) 10JanZerebecki: Wikidata wmf branch to wmf/1.27.0-wmf.14 [tools/release] - 10https://gerrit.wikimedia.org/r/270941 [12:49:55] (03CR) 10JanZerebecki: [C: 032] Wikidata wmf branch to wmf/1.27.0-wmf.14 [tools/release] - 10https://gerrit.wikimedia.org/r/270941 (owner: 10JanZerebecki) [12:51:16] (03Merged) 10jenkins-bot: Wikidata wmf branch to wmf/1.27.0-wmf.14 [tools/release] - 10https://gerrit.wikimedia.org/r/270941 (owner: 10JanZerebecki) [14:10:47] Project beta-scap-eqiad build #90216: 04FAILURE in 6 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90216/ [14:13:21] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10DBA, 10Flow: Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql - https://phabricator.wikimedia.org/T127040#2031185 (10matthiasmullie) I guess patch-reference_wiki.sql didn't fully make it to... [14:15:00] o/ [14:22:12] 10Continuous-Integration-Infrastructure: composer-hhvm (intermittently) failing on Echo: "./composer.json not found." - https://phabricator.wikimedia.org/T127031#2031231 (10Krinkle) [14:23:59] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10DBA, 10Flow: Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql - https://phabricator.wikimedia.org/T127040#2031237 (10hashar) You should be able to reach the beta cluster databases via: ```... [14:24:17] 10Continuous-Integration-Infrastructure: mediawiki phpunit jobs frequently fail with "Unknown database 'jenkins_u1_mw'" - https://phabricator.wikimedia.org/T127032#2031238 (10Krinkle) [14:24:53] 10Continuous-Integration-Infrastructure: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2031241 (10Krinkle) [14:26:33] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-02, 3Wikidata-Sprint-2016-02-16: Support PHP 5.5 in CI for Wikidata stuff - https://phabricator.wikimedia.org/T126441#2031247 (10Lydia_Pintscher) [14:30:20] 10Continuous-Integration-Infrastructure: Tune MySQL innodb settings on CI slaves - https://phabricator.wikimedia.org/T96250#2031267 (10Krinkle) 5Open>3declined a:3Krinkle MySQL performance doesn't seem to be an issue at the moment. tmpfs cut it down by an order of magnitude. For simplicity sake, probably b... [14:30:22] 10Continuous-Integration-Infrastructure, 7Tracking: MySQL tuning on CI slaves (tracking) - https://phabricator.wikimedia.org/T96249#2031270 (10Krinkle) [14:30:31] 10Continuous-Integration-Infrastructure, 7Tracking: MySQL tuning on CI slaves (tracking) - https://phabricator.wikimedia.org/T96249#2031272 (10Krinkle) 5Open>3Resolved a:3Krinkle [14:30:33] 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10MediaWiki-Installer: Creating MySQL tables for MediaWiki sometimes stalled on I/O for several minutes - https://phabricator.wikimedia.org/T96229#2031274 (10Krinkle) [14:30:35] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 7JavaScript: Apache on Jenkins slave takes over 30s to respond (QUnit/AJAX "Test timed out") - https://phabricator.wikimedia.org/T95971#2031275 (10Krinkle) [14:37:26] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10DBA, 10Flow: Beta cluster update database is broken because of Flow db patch db_patches/patch-ref_target_not_null.sql - https://phabricator.wikimedia.org/T127040#2031294 (10matthiasmullie) Ha, deployment-tin! Dear god, been awhile since I ssh'ed... [14:37:29] 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#2031299 (10hashar) [14:39:22] zeljkof: I am closing the rubocop tasks [14:39:35] it is {done} imho, no need to keep pushing further [14:39:36] hashar: go ahead [14:39:43] agreed [14:40:42] Yippee, build fixed! [14:40:43] Project beta-scap-eqiad build #90219: 09FIXED in 6 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90219/ [14:41:04] 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#2031317 (10hashar) [14:41:20] 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#2031319 (10hashar) a:3zeljkofilipin [14:42:25] 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#2031324 (10hashar) 5Open>3Resolved Rubocop is enabled on the repositories we care of. Most of them are 100% compliant, some might have some leftover todo. The task was about fixi... [14:42:32] zeljkof: rubocop is done ;-} [14:43:26] hashar: great, thanks! [14:46:08] hashar: If you have a minute, I can brief you on vendor require-dev change? [14:46:22] cutting wmf branches :D [14:46:37] k :) [14:46:47] Krinkle: maybe write down something to qa list / wikitech-l for the general audience ? [14:47:03] as I understand it the task is mediawiki/vendor not including composer require-dev [14:47:19] which we need to run phpunit from the version provided by composer [14:47:31] so end up doing git clone && composer install --only-require-dev-something [14:47:38] which sounds all good to me :-} [14:47:42] Yeah, and we don't want to include it in vendor, either. It changes very often and we need immediate preview at run time as part of the test tools [14:47:57] hashar: yep, almost like that [14:48:09] This allows us to deprecate integration/phpunit [14:48:19] and apparently nothing broke :-} [14:48:20] and then, in a few days, we can easily update phpunit in core by simply changing the value in composer.json [14:48:25] \O/ [14:48:50] that definitely deserve some kind of short mail / announce to wikitech-l. I am sure dev around the planet are going to rejoice [14:48:51] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-GettingStarted, 6Operations: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#2031357 (10Aklapper) Any news here? Still happening, still "high priority"? [14:48:57] It also unblocks us from using other dev packages inside test suite. E.g. vfsStream php package for mocking file system [14:48:59] (if we want to) [14:49:04] might want to sync/double check with Jan [14:49:24] he did most/all of the composer/phpunit work iirc (at least I haven't been involved) [14:49:37] and potentially we can merge phpunit/phpcs into one job if we want to [14:49:45] we don't need to change anythning (I will not) [14:49:49] but it's a possibility [14:49:51] mediawiki-core-phpcs iskilling me [14:50:05] oh? [14:50:12] it reanalyze the whole list of php files which takes a while (6-7 minutes?) [14:50:19] Right [14:50:39] I thought about how grunt let you define target files such as '**/*.js', [14:51:12] and maybe one might find the list of php files changes (like our shell script bin/git-changed-in-head and inject that in phpcs as the list of files to lint [14:51:27] phpcs has a --cache= option [14:51:44] But I don't think we should use it. [14:51:46] It doesn't scale [14:51:48] same with grunt [14:51:55] it's not worth the optimisation imho [14:52:04] There's bigger fish to catch. [14:52:28] we'd need to store it per repo+branch somewhere, and store it there in gate jobs, and fetch from it in test jobs [14:52:50] we have all the logic for that [14:52:59] which doesn't work well if you want to separate concerns, which means Zuul will not know which things are run (e.g. composer-hhvm is just 'composer test', it doesn't now phpunit or phpcs is used) [14:53:08] but I dont think there is any need to cache when we can figure out a way to only lint HEAD files [14:53:52] Maybe, but it will require maintenance and it's going to break things and cause bugs and it'll make things pass that should fail. [14:54:01] For one, most linters I know bypass ignore options when passing files directly [14:54:16] e.g. jshintignore: libs. $ jshint src/foo.js libs/bar.js will fail [14:54:29] and there'll be bugs in git-changed-in-head [14:54:33] I'm glad it's gone [14:54:48] nobody else does it in the industry. We just need to make it run faster and make the repos faster. [14:54:53] smaller* [14:55:01] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [15:20:42] !log integration-make-wmfbranch : change tmpfs to /mnt/make-wmf-branch (from /var/make-wmf-branch ) [15:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:21:34] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2031498 (10zeljkofilipin) [15:23:25] !log integration-make-wmfbranch : /mnt/make-wmf-branch mount now has gid=wikidev and group setuid (i.e. mode 2775) [15:23:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:27:12] apologies to today’s swatters for my cryptic (yet somewhat urgent) swat request coming up [15:39:12] hashar: There is a db access denied happening at https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/2740/console [15:39:17] 15:36:15 DB connection error: Access denied for user 'jenkins_u0'@'localhost' (using password: YES) (127.0.0.1:3306) [16:09:17] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2031690 (10zeljkofilipin) [16:20:29] Hello, i'm comming from -labs, i have a pb with gerrit. When i git review i obtain : [16:20:42] ! [remote rejected] master -> master (prohibited by Gerrit) [16:20:58] error: failed to push some refs to 'ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index' [16:22:10] when i git push i have Had trouble running git log --color=always --decorate --oneline HEAD --not remotes/gerrit/ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index -- [16:22:59] fatal: bad revision 'remotes/gerrit/ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index' [16:24:03] sorry i invert push and review but in both cases i can't basically copy the work i recently put on tools.vocabulary-index [16:25:29] 6Release-Engineering-Team: MW 1.27.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T127086#2031747 (10greg) 3NEW a:3demon [16:26:11] 6Release-Engineering-Team: MW 1.27.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T127086#2031758 (10greg) [16:36:24] 10Beta-Cluster-Infrastructure, 5Patch-For-Review, 7Puppet, 7Tracking: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2031799 (10greg) Please do not change to a goal. The reverse dependencies, as @aklapper pointed out, are very important. [16:38:33] 10Continuous-Integration-Infrastructure: Investigate installing php5.3 on trusty and/or debian instance - https://phabricator.wikimedia.org/T103786#2031808 (10JanZerebecki) Recently mentioned for solving this: https://github.com/php-build/php-build/ (one needs to use master as there was no release for quite some... [16:38:35] Youni: errm, could you explain why you run "git review" without any parameters? See the "(prohibited by Gerrit)" error you get. [16:40:58] andre__: that's normal (no arguments to git-review): https://www.mediawiki.org/wiki/Gerrit/Getting_started [16:45:23] Sigh, thanks. I checked https://www.mediawiki.org/wiki/Gerrit/Tutorial which I normally follow when I don't want to use my brain. Too many guides again. :( [16:46:45] gerrit keeps timing out on me :( [16:53:10] * greg-g reluctantly pings ostriches re gerrit ^ [16:53:42] thcipriani: Web? SSH? Works for me.... [16:53:59] ostriches: web, might be local to me. [16:54:10] just retry [16:54:15] I can't replicate. [16:54:19] been going on for a few months [16:55:02] * ostriches has never replicated this supposed failure [16:55:09] * ostriches blames everyone's ISP and local PC. [16:55:17] 10Browser-Tests-Infrastructure, 5Patch-For-Review: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2031916 (10zeljkofilipin) [16:55:42] (03PS1) 10Siebrand: Skip puppet-lint max. 80 char/line check [integration/config] - 10https://gerrit.wikimedia.org/r/270991 [16:57:15] (03CR) 10jenkins-bot: [V: 04-1] Skip puppet-lint max. 80 char/line check [integration/config] - 10https://gerrit.wikimedia.org/r/270991 (owner: 10Siebrand) [16:57:35] "just get gigabit net with 8ms latency with a <1 year old macbook, you'll be fine" [16:59:30] ususal suspects are ISP peering politics and http://www.bufferbloat.net/ [17:01:17] (03CR) 10Siebrand: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/270991 (owner: 10Siebrand) [17:01:50] greg-g: my gigabit wired connection as 105ms vs 108 ms for the wireless one :-} hardly a difference [17:02:08] (03CR) 10Siebrand: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [17:02:25] hashar: :) [17:02:35] oh sorry [17:02:47] that is over ipv4 [17:02:47] jzerebecki: good ole bufferbloat [17:02:56] with v6 I only have 88ms !!!! :D [17:20:13] hassent [17:20:19] hashar: sent [17:27:56] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [17:33:21] 10Deployment-Systems, 6Operations, 6Performance-Team, 10Traffic, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032126 (10Krinkle) [17:33:28] 10Deployment-Systems, 6Operations: error on tin:/srv/mediawiki-staging: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T127093#2032127 (10thcipriani) 3NEW [17:33:44] 10Continuous-Integration-Infrastructure: Support installing composer require-dev packages together with mediawiki/vendor - https://phabricator.wikimedia.org/T112895#2032135 (10Ricordisamoa) [17:35:35] 10Deployment-Systems, 6Operations, 6Performance-Team, 10Traffic, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032143 (10Krinkle) `$wgLocalStylePath` was using `/w/static/{wmfbranch}` instead o... [17:46:21] 10Deployment-Systems, 6Operations: error on tin:/srv/mediawiki-staging: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T127093#2032183 (10thcipriani) [17:57:25] releng folks: is this error known? https://integration.wikimedia.org/ci/job/parsoidsvc-php-parsertests/6851/console [17:57:52] 17:47:02 - This package requires php >=5.5.9 but your PHP version does not satisfy that requirement. [17:58:44] git review ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index [17:58:50] returns: [17:59:02] I haven't seen that one yet, but I've seen all sorts of bogus CI failures the past week [17:59:06] Some of them related to composer [17:59:19] fatal: bad revision 'remotes/gerrit/ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index [17:59:41] Had trouble running git log --color=always --decorate --oneline HEAD --not remotes/gerrit/ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index -- [17:59:53] Youni: You can't pass a URL as the first argument to git review [17:59:56] What are you trying to do? [18:00:19] i try to fill my gerrit repo for the first time [18:01:20] 10Deployment-Systems, 6Operations, 6Performance-Team, 10Traffic, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032268 (10Krinkle) Rollout to group1 and group2 is blocked on Apache config being... [18:03:08] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:36] what would be the right command? [18:03:56] You're trying to clone it? [18:04:02] 10Deployment-Systems, 6Operations, 6Performance-Team, 10Traffic, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032287 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmfl... [18:04:03] That would be `git clone ` [18:05:17] I saw a bunch of noise on UW jobs over the weekend, zeljkof and marxarelli I assume that's good news for us [18:05:36] but it is empty i can clone but it comes an empty directory from gerrit [18:06:10] MarkTraceur: which jobs? [18:06:18] The UW browser test ones? [18:06:29] Maybe it wasn't you working on it, I can look at my logs more carefully [18:06:54] Youni: it should have been initialized with an empty initial commit [18:07:59] marxarelli: I guess it was zeljkof https://gerrit.wikimedia.org/r/270720 [18:08:01] so what is the way to put the tool's git content to gerrit? [18:09:13] i can clone from tools account to my station the directory download right with the files [18:09:35] but if i clone from gerrit it comes empty [18:11:21] hiyyyyyy thcipriani :) [18:11:32] is it possible to make scap to service reload (SIGHUP) instead of restart? [18:11:39] ottomata: howdy [18:12:17] MarkTraceur: ah, i see. he deleted the MW-Selenium tests and the jobs it looks like [18:12:34] than if i try this : [18:12:52] (03PS1) 10Legoktm: Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 [18:13:02] we're pairing tomorrow morning. i'll bring up UW tests and see how soon we can get going on a proof of concept for the JS framework/lib [18:13:12] ottomata: currently we're just calling service [name] restart https://github.com/wikimedia/scap/blob/master/scap/tasks.py#L569-L570 [18:13:31] sorry if i try git review +ssh_url [18:13:44] (03CR) 10jenkins-bot: [V: 04-1] Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 (owner: 10Legoktm) [18:13:45] ok, I'm going to make a feature request then! [18:13:53] eventlogging-service can SIGHUP to reload configs and schemas [18:14:03] it shows 6 commit [18:14:09] hmm [18:14:15] i guess on code deploy i'd want restart though [18:14:16] hm [18:14:46] but when i submit: [18:15:12] error: src refspec HEAD:refs/publish/ssh://y-verciti@gerrit.wikimedia.org does not match any. [18:15:36] error: failed to push some refs to 'ssh://y-verciti@gerrit.wikimedia.org:29418/labs/tools/vocabulary-index.git' [18:19:48] thcipriani: https://phabricator.wikimedia.org/T127097 [18:20:16] 3Scap3: Custom command for service restarts during scap deploy - https://phabricator.wikimedia.org/T127097#2032351 (10Ottomata) 3NEW [18:20:17] (03PS2) 10Legoktm: Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 [18:20:19] ottomata: you can, FWIW, run a command at the end of any stage. [18:20:45] oh?! [18:21:00] reading harder... [18:21:01] ottomata: yup, looking for the documentation... [18:21:54] Iiinteresting, so thcipriani i could do [18:21:58] checks: ... [18:22:05] command: service eventlogging-service-eventbus reload [18:22:06] ? [18:22:42] ottomata: https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html#additional-checks [18:22:58] ja was reading that [18:23:13] ottomata: you should give it a stage, too. I think the default is to run after every stage [18:25:24] aye ja, promote [18:25:24] ok [18:25:35] thcipriani: i think i'm going to rearrange how we deploy the scheams, that's been a todo anyway [18:25:37] thanks [18:26:05] ottomata: yup, lmk if you run into any weirdness. [18:26:16] 3Scap3: Custom command for service restarts during scap deploy - https://phabricator.wikimedia.org/T127097#2032386 (10Ottomata) 5Open>3Invalid a:3Ottomata This is already supported via custom checks, whoops! [18:31:27] than i do: git-review ssh_url -x 4770065 (first initial chek-in) [18:31:35] but returns: [18:31:59] Could not fetch review information for change 4770065 [18:36:08] 10Continuous-Integration-Infrastructure: Deal with bitbucket.org rate limit for downloading phantomjs (npm jobs intermittently fail) - https://phabricator.wikimedia.org/T127100#2032437 (10Krinkle) 3NEW [18:49:28] 10Continuous-Integration-Infrastructure: Deal with bitbucket.org rate limit for downloading phantomjs (npm jobs intermittently fail) - https://phabricator.wikimedia.org/T127100#2032482 (10Krinkle) [18:51:18] 10Continuous-Integration-Infrastructure: Deal with bitbucket.org rate limit for downloading phantomjs (npm jobs intermittently fail) - https://phabricator.wikimedia.org/T127100#2032437 (10Krinkle) [18:53:56] for god sake php [18:54:09] Warning: symlink(): File exists in /srv/mediawiki-staging/multiversion/checkoutMediaWiki.php on line 210 [19:00:54] !log tin: checking out mw 1.27.0-wmf.14 [19:00:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:17:57] 6Release-Engineering-Team, 10Flow, 3Collaboration-Team-Current, 5MW-1.25-release: Flow (or tests) broken in REL1_25 - https://phabricator.wikimedia.org/T122025#2032597 (10Umherirrender) [19:17:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 10Wikibase-DataModel-JavaScript, 10Wikidata: REL1_25 fails shared tests due to Wikidata: The module 'wikibase' required by 'wikibase.datamodel.__namespace' must exist - https://phabricator.wikimedia.org/T125965#2032594 (10Umherirrender) 5Ope... [19:32:51] thcipriani: hey when was the .14 branch cut? [19:33:56] thcipriani: nm, 4h ago [19:33:57] :) [19:35:36] yesh. :) [19:38:52] can't remember but why does scap sync-masters rebuild all the cdb on the other master ? [19:39:25] hashar: fundamentally so that the co-master is ready to run scap or l10nupdate [19:39:38] both of those do things based on the cdb state [19:43:48] bd808: since it takes 40 secs we could have the co master to run it on scp /l10nupdtae? [19:44:19] or maybe it will take longer because it has to catch up [19:44:28] possibly, except neither host knows that it is the "co-master" so it would have to happen on both sides [19:45:05] bah states .. [19:45:25] another question while you are around bd808 , have we considered the whole l10n stuff as plain php in HHVM ? [19:45:47] speculating that we dont have enough RAM to hold all the l10n for two versions [19:45:49] yes. there's even code for it [19:46:03] but HHVM crashed when we tried it last [19:46:16] due to lack of garbage collection in the TC cache [19:46:23] there's a bug somewhere ... [19:46:41] ;-( [19:46:53] https://phabricator.wikimedia.org/T99740 [19:47:11] and last question again: is there some kind of way to state I want a 4 GBytes memory that has 256 MB in RAM and the rest on disk ? :D [19:47:24] with entries being moved magically depending on usage count [19:47:37] I mean in C / the kernel whatever low level language [19:48:02] a custom slab allocator of some sort would be possible [19:48:15] but yeah complicated [19:48:19] * hashar needs to buy a book about low system programming [19:49:35] hashar: this is *the* book https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Unix_Environment [19:49:39] the complete newbie I am would let the linux kernel handle it via the swap [19:49:49] swap is gross [19:49:54] * hashar looks up for "101 Programing in the Unix Environment" [19:49:58] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-02: [Task] use generic composer based Jenkins jobs for WikibaseJavaScriptApi CI - https://phabricator.wikimedia.org/T100654#2032699 (10JanZerebecki) https://gerrit.wikimedia.org/r/270761 [19:50:01] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-02: [Task] use generic composer based Jenkins jobs for WikibaseJavaScriptApi CI - https://phabricator.wikimedia.org/T100654#2032702 (10JanZerebecki) [19:50:04] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review: [Task] change phpunit CI job for Wikidata.org to use composer - https://phabricator.wikimedia.org/T107834#2032706 (10JanZerebecki) https://gerrit.wikimedia.org/r/#/c/270765/ [19:50:24] anyway we have a task about using .php files. That is good for tonight ;-) [19:50:24] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review: [Task] use generic composer based Jenkins jobs for WikibaseJavaScriptApi CI - https://phabricator.wikimedia.org/T100654#1317486 (10JanZerebecki) [19:50:30] hashar: Or read any code written by bblack and then ask him how it works :) [19:50:37] oh yeah [19:50:45] Hey all — is it ok if I cause some downtime for deployment-logstash2? [19:51:00] andrewbogott: it won't end the world [19:51:19] Brandon has nice explanations. But yesterday I was wondering why strcopy() does not fail when copying a string that is too long [19:51:20] (or at least shouldn't) [19:51:48] andrewbogott: yeah can do just !log here on start and end ;) [19:52:37] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-16: [Task] use generic composer based Jenkins jobs for WikibaseJavaScriptApi CI - https://phabricator.wikimedia.org/T100654#2032711 (10JanZerebecki) [19:52:50] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-16: [Task] change phpunit CI job for Wikidata.org to use composer - https://phabricator.wikimedia.org/T107834#2032714 (10JanZerebecki) [19:53:19] thing I love with Unix is that a 1992 book is still relevant [19:53:19] hashar: fundamentally, because C does not try to pretend to be smarter than then developer. It lets you do very low level things rather than aiming for safety [19:53:33] yeah figured that out eventually [19:53:50] C is assembly in a nice dress [19:54:11] and I went to read an interview by one of the early C adopters that stated at one point they pondered between null terminated strings and strings having an explicit size [19:54:26] went with \0 because it was simpler to implement leading to decades of buffer overflow [19:57:06] (03PS1) 10JanZerebecki: Remove unused WikibaseJavaScriptApi jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271045 [19:58:36] !log migrating deployment-logstash2 to labvirt1010 [19:58:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:01:23] PROBLEM - Host deployment-logstash2 is DOWN: CRITICAL - Host Unreachable (10.68.16.147) [20:06:38] wonderful people of rellang, would anyone help with deploying kartographer on beta cluster? https://gerrit.wikimedia.org/r/#/c/270441/ [20:07:13] i could deploy it myself, but hashar might be unhappy [20:09:19] yurik: ah will do it is straightforward :D [20:09:28] the 1am CET saturday deploy wasn't my kind of thing ;) [20:09:39] yeah yeah [20:09:44] its 2300 here :-P [20:10:15] thx hashar :) [20:10:19] stuff going to trigger on https://integration.wikimedia.org/zuul/ under postmerge [20:10:21] do you need to do anything other than +2? [20:10:36] Jenkins does all the magic [20:10:49] then gotta check the logs / make sure stuff works [20:10:57] hashar, and you make jenkins do magic. [20:11:02] hashar is magical [20:11:02] and you definitely want to try out Kartographer ;) [20:11:07] hmm no [20:11:11] (03CR) 10JanZerebecki: [C: 032] Remove unused WikibaseJavaScriptApi jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271045 (owner: 10JanZerebecki) [20:11:11] I am more like a show organizer [20:11:24] there is a large audience, I just pick the best magicians around and make sure they show up on stage [20:11:37] but since I am lazy, it is Jenkins picking the magician and making sure they perform [20:11:47] if something wrong Jenkins just ring. [20:11:52] hashar, since noone other than you knows about those magicians, you in turn become one [20:11:54] magic == not having to do stuff ;) [20:12:00] yeah maybe [20:12:19] well lot of folks knows about Jenkins! it used to be mostly me but that was 3 years ago [20:12:19] magic == not understanding [20:12:34] we have learnt to hate it since then [20:12:39] not much more :-P [20:12:41] (03Merged) 10jenkins-bot: Remove unused WikibaseJavaScriptApi jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271045 (owner: 10JanZerebecki) [20:13:15] well,ok, a bit more than that [20:15:12] yurik: and jenkins report back on the change i.e. https://gerrit.wikimedia.org/r/#/c/270441/ [20:15:21] well I think [20:15:43] https://integration.wikimedia.org/ci/view/Beta/ should give a good overview [20:15:57] aaa, there are blinkies there [20:17:50] and the deploy host is https://integration.wikimedia.org/ci/computer/deployment-tin.eqiad/ [20:17:56] there is some jobs running on it [20:17:56] hashar, ooops, forgot to +2 - a tiny change, can i merge it with the branch? https://gerrit.wikimedia.org/r/#/c/270892/ [20:18:04] (03PS3) 10JanZerebecki: Do not run rake-jessie for UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/270764 (https://phabricator.wikimedia.org/T94161) (owner: 10Zfilipin) [20:18:08] not related to kartographer [20:18:35] yurik: no clue for Graph. Guess you want the review on master then cherry pick to wmf.14 [20:18:41] .14 is not deployed yet [20:18:45] hashar, cool, i will do that [20:18:49] but has been cut [20:19:05] (03CR) 10JanZerebecki: [C: 032] Do not run rake-jessie for UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/270764 (https://phabricator.wikimedia.org/T94161) (owner: 10Zfilipin) [20:19:06] PROBLEM - Puppet failure on deployment-analytics01 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [20:19:07] the change on master will magically deploy on beta as well if you want to test it before cherry picking to wmf.14 [20:19:23] hashar, so if i cherry pick and merge to 14, do i need to sync it on tin? [20:19:25] i will [20:20:26] yurik: yeah anything merging in a wmf branch is to be deployed on spot [20:20:29] on prod [20:20:34] err no [20:20:49] yurik: errrrrr not for wmf.14 it is not synced yet ;-) [20:20:54] (03Merged) 10jenkins-bot: Do not run rake-jessie for UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/270764 (https://phabricator.wikimedia.org/T94161) (owner: 10Zfilipin) [20:21:01] so just merge and when we sync .14 for the first time your patch will land in prod [20:21:02] hashar, right, that's what i wondered about [20:21:06] cool [20:21:07] thx [20:42:43] PROBLEM - Puppet failure on deployment-analytics02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:46:42] PROBLEM - Puppet failure on deployment-analytics03 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [20:49:57] !log reloading zuul for 3bf7584..67fec7b [20:50:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:54:14] RECOVERY - Puppet failure on deployment-analytics01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:37] (03PS4) 10JanZerebecki: Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [20:56:56] (03CR) 10jenkins-bot: [V: 04-1] Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [20:57:47] RECOVERY - Puppet failure on deployment-analytics02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:00:59] 10Deployment-Systems, 5Patch-For-Review: l10nupdate syncs not working after rebuilding tin due to ssh-agent errors - https://phabricator.wikimedia.org/T125992#2033016 (10thcipriani) Now `/var/log/l10nupdatelog/l10nupdate.log` seems to be updating. The output, however, doesn't inspire confidence that it's worki... [21:01:47] RECOVERY - Puppet failure on deployment-analytics03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:08:40] RECOVERY - Host deployment-logstash2 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [21:10:13] (03PS5) 10JanZerebecki: Update test runners for translatewiki repo [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [21:14:37] !log deployment-logstash2 migration finished [21:14:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:17:52] (03CR) 10JanZerebecki: "The Depends-On needs to be removed, or that patch needs to be merged. E.g. composer-test can be first only added to the experimental pipel" [integration/config] - 10https://gerrit.wikimedia.org/r/270887 (owner: 10Siebrand) [21:19:30] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-02, 3Wikidata-Sprint-2016-02-16: Support PHP 5.5 in CI for Wikidata stuff - https://phabricator.wikimedia.org/T126441#2033106 (10JanZerebecki) 5Open>3Resolved [21:22:05] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-16: [Task] change phpunit CI job for Wikidata.org to use composer - https://phabricator.wikimedia.org/T107834#2033115 (10JanZerebecki) a:3JanZerebecki [21:22:14] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2016-02-16: [Task] use generic composer based Jenkins jobs for WikibaseJavaScriptApi CI - https://phabricator.wikimedia.org/T100654#2033116 (10JanZerebecki) a:3JanZerebecki [21:50:45] 10Browser-Tests-Infrastructure, 10Reading-Web, 5Patch-For-Review, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13), 5WMF-deploy-2016-02-23_(1.27.0-wmf.15): Fix failed MobileFrontend browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94156#2033185 (10Jdlrobson) [21:51:34] 10Browser-Tests-Infrastructure, 10Reading-Web, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13), 5WMF-deploy-2016-02-23_(1.27.0-wmf.15): Fix failed MobileFrontend browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94156#1156443 (10Jdlrobson) [22:16:25] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [22:24:45] Project beta-scap-eqiad build #90267: 04FAILURE in 8 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90267/ [22:29:58] 10Browser-Tests-Infrastructure, 10Reading-Web, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13), 5WMF-deploy-2016-02-23_(1.27.0-wmf.15): Fix failed MobileFrontend browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94156#2033335 (10Jdlrobson) The remaining tests are failing with "Net::ReadTimeout (Net::Re... [22:38:33] legoktm: Could you review https://gerrit.wikimedia.org/r/#/c/270938/ please. [22:38:56] it seems that https://integration.wikimedia.org/zuul/ has frozen again. [22:40:06] krinkle: It seems phpunit isent working here https://integration.wikimedia.org/ci/job/mwext-testextension-php55-non-voting/4/console [22:40:15] 22:37:20 Couldn't find a usable PHPUnit. [22:40:22] legoktm hashar ^^ [22:47:10] (03PS1) 10Paladox: [SolrStore] Add SemanticMediaWiki dependacy support [integration/config] - 10https://gerrit.wikimedia.org/r/271137 [22:52:48] Yippee, build fixed! [22:52:49] Project beta-scap-eqiad build #90270: 09FIXED in 7 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90270/ [22:56:24] hashar: [22:56:25] 22:54 < legoktm> Feb 16 22:52:09 labnodepool1001 nodepoold[1596]: JenkinsException: Error in request.Possibly authentication failed [500] [22:56:34] !log contint: Nodepool instances pool exhausted [22:56:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:56:37] heh, sorry [22:56:41] beeh [22:56:42] I can see it building more slaves right now [22:56:47] must be some labs issue [22:57:11] hashar: journald has a bunch of exceptions, I think jenkins was returning 500 errors to nodepool? [22:57:48] looking at /var/log/nodepool/nodepool.log on labnodepool1001.eqiad.wmnet [22:58:12] yeah apparently Nodepool could not authenticate with Jenkins [22:58:32] first event on 21:18 UTC [22:59:48] it would be cool if `nodepool list` had a web dashboard [23:00:17] I dont think it has any embedded server [23:00:19] zuul does [23:01:17] so why the hell does nodepool cant authenticate with Jenkins [23:02:50] !log Nodepool can not authenticate with Jenkins anymore. Thus it can not add slaves it spawned. [23:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:11:27] I am gonna [23:11:31] nuke Jenkins [23:14:38] !log Jenkins: Could not create rootDir /var/lib/jenkins/config-history/nodes/ci-jessie-wikimedia-34969/2016-02-16_22-40-23 [23:14:39] hehe [23:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:14:47] CAUSE THERE IS ONLY 32K INODES PER DIR !!!!!!!!!!!!! [23:15:07] found via https://integration.wikimedia.org/ci/log/Warnings/ [23:15:26] hahahahaha [23:15:47] I mean, that's terrible :( [23:16:11] yeah [23:16:17] that has hit me a few time [23:16:24] not this particular occurence [23:16:28] but the 32 k inode limit [23:17:13] !log Jenkins accepting slave creations again. Root cause is /var/lib/jenkins/config-history/nodes/ has reached the 32k inode limit. [23:17:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:17:34] 2016-02-16 23:17:23,775 INFO nodepool.NodePool: Need to launch 3 ci-jessie-wikimedia nodes for gallium.wikimedia.org on wmflabs-eqiad [23:17:34] :D [23:17:41] 2016-02-16 23:16:40,691 INFO nodepool.NodeLauncher: Node id: 35052 added to jenkins [23:18:17] !log jenkins@gallium find /var/lib/jenkins/config-history/nodes -maxdepth 1 -type d -name 'ci-jessie*' -exec rm -vfR {} \; [23:18:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:19:08] 10Beta-Cluster-Infrastructure, 6Labs: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#2033534 (10yuvipanda) [23:19:09] hashar: also, should rake-jessie only be triggered if a .rb file changes (or something like that?) or does it need to run on every patchset? [23:19:11] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2033533 (10yuvipanda) [23:19:17] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1992944 (10yuvipanda) This is unrelated to swift, since this is only removing the mounts from instances that do not use them at all and is effectively a n... [23:19:58] legoktm: can be used for other changes as well not neccesarly just .rb files [23:20:04] like one can change a gemfile [23:20:12] or want to run browser tests via the rake test command [23:20:15] ok [23:22:56] !log new instances on deployment-prep no longer get NFS because of https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=311783&oldid=311781 [23:22:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:24:31] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2033562 (10yuvipanda) https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=311783&oldid=311781 turns off NFS for ne... [23:27:49] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2033587 (10yuvipanda) I've set `mount_nfs: true` on all the instances I listed earlier. These are all still no-ops, since I actually need to unmount them... [23:31:17] legoktm: kudos on the first diagnostic ! [23:34:59] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2033610 (10yuvipanda) I've unmounted /data/project on deployment-poolcounter01 and verified that puppet doesn't bring it back. I'll unmount it on all the... [23:36:14] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Jenkins, 7Nodepool: Postmortem: Nodepool can't add slaves to Jenkins due to config plugin directory reaching 32k inodes - https://phabricator.wikimedia.org/T127131#2033616 (10hashar) 3NEW [23:36:25] legoktm: greg-g postmortem at https://phabricator.wikimedia.org/T127131 [23:36:40] legoktm: I caught it via https://integration.wikimedia.org/ci/log/Warnings/ [23:36:59] which had a super nice stacktrace with the message Caused by: java.lang.RuntimeException: Could not create rootDir /var/lib/jenkins/config-history/nodes/ci-jessie-wikimedia-34969/2016-02-16_22-40-23 [23:37:29] legoktm: thank you again for the quick reaction [23:37:43] hashar: legoktm thanks both [23:38:26] it will catch up eventually [23:38:36] though the Nodepool allocation algorithm is bugged :/ [23:38:47] it does not replenish the pool fast enough [23:44:48] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Jenkins, 7Nodepool: Postmortem: Nodepool can't add slaves to Jenkins due to config plugin directory reaching 32k inodes - https://phabricator.wikimedia.org/T127131#2033663 (10Paladox) [23:50:12] (03PS1) 10Paladox: [XMLContentExtension] Replace jslint test with jshint and jsonlint test [integration/config] - 10https://gerrit.wikimedia.org/r/271154 [23:57:08] (03PS1) 10Paladox: [ReplaceSet] Replace jslint test with jsonlint and jshint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271156