[00:01:22] here's the full meal deal description of the jobs that fire -- https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L1261-L1312 [00:02:03] (03PS1) 10Ejegg: fundraising/REL_X tests = REL_X tests [integration/config] - 10https://gerrit.wikimedia.org/r/289975 [00:02:15] and these are excluded on your branches -- https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L691-L702 [00:02:17] ^^ that may be completely wrong [00:02:47] (03CR) 10jenkins-bot: [V: 04-1] fundraising/REL_X tests = REL_X tests [integration/config] - 10https://gerrit.wikimedia.org/r/289975 (owner: 10Ejegg) [00:03:10] guess so! [00:03:18] yeah... that's hashar territory [00:03:56] wat--- AssertionError: mediawiki/core rake job must NOT run on fundraising/REL1_42 [00:04:02] ok, that test is bs [00:04:04] there are tests [00:04:27] we can totally take your rake tests! (once we merge this 1.25.6 stuff) [00:04:58] tests/test_zuul_scheduler.py [00:05:06] https://gerrit.wikimedia.org/r/#/c/288554/ [00:05:17] confirmed that before this merge we were passing against trusty/5.5 [00:05:26] (03PS2) 10Ejegg: fundraising/REL_X tests = REL_X tests [integration/config] - 10https://gerrit.wikimedia.org/r/289975 [00:06:01] but.... we're not running on trusty/5.5 in prod, so the REL1_25 tests make more sense [00:07:17] ya...but makes it seem more likely to me they will bomb against the older tests too [00:07:23] all of these "session_regenerate_id(): Cannot regenerate session id - headers already sent [00:07:37] that looks like rogue whitespace in the patch somewhere [00:07:40] cwd you think? I figure there's something in the test setup [00:08:06] bd808: whitespace? which patch? [00:08:30] dunno. gerrit is shit for showing a diff on a merge [00:08:30] PS2 of that zuul layout change is v+2... [00:09:32] bd808: I tried a git diff between REL1_25 and fr/REL1_25 with the merge, and there was nothing besides what I expected [00:09:53] deleted readmes, different submodules [00:11:29] hmm, actually, let's not test our branch against hhvm after all [00:15:41] cwd: you remember the instructions for testing jjb locally? [00:16:13] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder [00:16:17] ugh it has been awhile [00:17:03] nvm, found em [00:17:08] thanks bd808 ! [00:17:33] your boxes still run on precise hosts right? You will probably need those 2 extra patches from Max [00:17:55] si [00:19:09] blarg... value error in jenkins_jobs/modules/project_matrix.py [00:19:18] Only ['groovy', 'tox', 'slave', 'label-expression', 'jdk', 'python', 'dynamic', 'user-defined'] axes types are supported [00:19:47] lemme see if that was my change [00:20:49] nope... [00:21:02] probably broke on some os update [00:23:10] oh hey, jjb needs update [00:24:08] ejegg: since our last core merged passed against trusty/5.5, then we merge 1.25 and it breaks, if it wasn't actual regressions what would that imply? code that worked under 5.3 and doesn't under 5.5? [00:24:16] i feel like the inverse is more likely to be true [00:24:21] no idea [00:24:51] I have no idea what's the difference in the test setup for those two suites [00:25:05] I mean, they're gonna have different LocalSettings and stuff, right? [00:25:33] well yeah, but that was fine before we merged 1.25 [00:25:56] cause I ran the API tests which had tons of errors in CI locally under hhvm and they almost all passed [00:26:33] yeah i am not in a state where i can get any of the tests to pass locally :( [00:27:08] feck feck feck [00:27:19] do your donation interface tests pass locally with the updated mediawiki? [00:27:26] yep [00:27:42] forcemerge, deploy, rollback if busted? [00:27:53] i'm practically there [00:27:56] figure the rest out monday? [00:27:56] it's so drastic [00:28:06] but i really don't know what else to do [00:28:23] tumbleweeds round these parts... [00:28:33] well, here goes... [00:28:40] * cwd buckles in [00:28:53] legoktm: around? cwd and ejegg could use help from somebody who understands the various phpunit test setups [01:03:02] bd808, cwd, ejegg: hi, what's up? [01:03:23] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:04:20] hi legoktm ! [01:04:31] We kinda gave up and forcemerged [01:04:39] and it's looking ok in prod so far... [01:05:45] ok, I just got home, haven't read scrollback yet [01:05:48] anyway, somewhere between the last core update on payments and 1.26, mediawiki-phpunit-php55-trusty started failing hard [01:05:55] legoktm: hi! merging REL1_25 into fundraising REL1_25 caused CI meltdown [01:06:06] we noticed that wasn't the test suite running on normal REL1_25 [01:06:33] also the last couple patches to REL1_25 were force merge [01:06:53] but cherry picking around those didn't help so probably not the culprit [01:08:24] iirc one of the security patches broke tests plus fundraising has a different phpunit setup [01:08:56] legoktm: does this change make fundraising branches get the same tests as the matching REL branch: https://gerrit.wikimedia.org/r/289975 [01:09:10] or does it just break stuff? [01:09:25] I'm having a hard time getting jjb to run again locally [01:10:53] yes, but I'm not sure if that's what you want [01:10:58] well, I don't even know what the problem is yet [01:11:30] is there a bug for this? [01:12:35] no we've just been flailing [01:14:47] ok, umm, how urgent is this? I kinda need to write a paper due on monday >.> [01:16:04] legoktm: we 'thereifixedit' for now [01:16:10] legoktm: i think we are okay for now, the force merge state is deployed and not exploding so we'll clean it up on monday [01:16:21] thanks and good luck! [01:17:00] file a bug and cc me on it? and thanks :) [01:18:09] will do [02:38:21] PROBLEM - Parsoid on deployment-parsoid06 is CRITICAL: Connection refused [06:31:17] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [06:57:16] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [11:50:45] legoktm: Im having a look at the failures but the failures were introduced after bumping to 1.25.6 [11:51:19] I also found fundraising/REL1_25 to have been testing with php55 tests when in fact they should be using php53. [12:31:19] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 tests are broken - https://phabricator.wikimedia.org/T135906#2314939 (10Paladox) [12:32:29] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 tests are broken - https://phabricator.wikimedia.org/T135906#2314953 (10Paladox) Adding @Legoktm since he said file a bug and cc him. [12:33:08] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 tests are broken - https://phabricator.wikimedia.org/T135906#2314956 (10Paladox) This also includes fundraising/REL1_25 which is also broken. [14:43:36] Maybe someone can add Urbanecm to the zuul-whitelisted users? He's doing a lot if site-request work, and I think it would be helpful, if I don't have to recheck ;) [14:49:13] Luke081515 i can upload the patch that whitelists him. [14:49:19] thx [14:49:22] Do you have his username and email [14:49:31] And also what should i say in the description [14:50:36] paladox: I think mentioning site-requests is enough. Mail is here: https://gerrit.wikimedia.org/r/#/c/290000/ [14:50:43] Ok thanks [14:50:47] nice patch number btw :D [14:53:54] Yep [14:58:58] (03PS1) 10Paladox: Whitlist user Urbanecm [integration/config] - 10https://gerrit.wikimedia.org/r/290001 [14:59:06] Luke081515 ^^ [15:00:16] (03CR) 10Paladox: "Per request by Luke081515." [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:00:30] thanks :) [15:01:02] (03CR) 10Luke081515: [C: 031] Whitlist user Urbanecm [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:01:10] Your welcome :) [15:01:58] (03CR) 10Urbanecm: "What does it mean for me?" [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:02:29] (03PS2) 10Urbanecm: Whitelist user Urbanecm [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:04:32] (03CR) 10Paladox: "@Urbanecm it means you can use the more advanced tests. Since currently your using basic tests due to you not being whitelisted which mean" [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:04:36] Luke081515 ^^ [15:04:53] yep, I explained some stuff to him at -operations were he actually is [15:05:00] (03CR) 10Luke081515: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:05:10] Ok thanks [15:05:40] And thanks for rechecking ^^ [15:05:41] :) [15:06:13] np ;) [15:09:30] ostriches: Hi when do you plan to turn gitblit off. Since i was thinking that we can add [15:09:31] [remote "https://github.com/wikimedia/mediawiki"] [15:09:31] url = https://github.com/wikimedia/mediawiki [15:09:31] push = +refs/heads/*:refs/heads/* [15:09:31] push = +refs/tags/*:refs/tags/* [15:09:31] threads = 3 [15:09:37] (03CR) 10Urbanecm: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/290001 (owner: 10Paladox) [15:09:44] to config in the mw core repo on the phabricator server [15:10:38] Hi, can somebody make new deployments table? I want to schedule one patch for May 23 Morning SWAT and I can't. Thanks! [15:10:39] then we can use the download code i had help with by twentyafterfour to use gitlibt to download commits for mw core and operations-puppet [15:10:54] Luke081515 ^^ [15:11:37] It is a way around so at one point in the near future we will manually add refs/changes to github for both of those repos [15:15:53] Luke081515 can i decline https://phabricator.wikimedia.org/T135909 please. [15:16:01] he was using the wrong branch. [15:17:07] Plus he was using https://phabricator.wikimedia.org/rEABF2807b5883a9c4f7bee11ea4e7992c63dd28c0d9e [15:17:19] which has not been merged yet. Still in refs/changes/ [15:20:08] Oh wait no that change was merged [15:20:15] But was not supported in REL1_26 [15:34:32] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2315107 (10Paladox) [15:35:39] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2314939 (10Paladox) [15:36:41] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2315110 (10Paladox) [15:37:26] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2314939 (10Paladox) p:05Triage>03High Setting as high because the tests are broken and we need the tests to verify we doin... [15:39:06] I was away, sry [15:39:28] Yippee, build fixed! [15:39:28] Project selenium-MobileFrontend ยป chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #22: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/22/ [15:39:55] Ok [15:40:10] Luke081515 [15:40:10] https://phabricator.wikimedia.org/T135906 [15:40:27] guess that was right [15:40:29] REL1_25, REL1_26 and fundraising tests are broken [15:40:34] Yep, thanks [15:43:48] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2315114 (10Paladox) [15:55:56] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:38] Luke081515 i found the course of the errors in REL1_25 and REL1_26 [15:56:39] https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm-trusty/747/console [15:56:45] ^^ passes now [15:59:44] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2315162 (10Paladox) I have confirmed that https://phabricator.wikimedia.org/rMWdfd1256837ef7b259880b6cd470ce17487ac9379 is ind... [15:59:58] :) [16:01:19] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2314939 (10Paladox) MobileFrontend is now failing REL1_26 and REL1_27 See https://integration.wikimedia.org/ci/job/mediawiki-... [16:01:44] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MobileFrontend: MediaWiki core REL1_25 and fundraising/REL1_25 and REL1_26 tests are broken - https://phabricator.wikimedia.org/T135906#2315170 (10Paladox) [16:03:42] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MobileFrontend: MediaWiki core and MobileFrontend break branches REL1_25 and fundraising/REL1_25 and REL1_26 and REL1_27 tests - https://phabricator.wikimedia.org/T135906#2314939 (10Paladox) [16:08:51] Luke081515 https://secure.phabricator.com/T10691 [16:08:57] thats alot of hours [16:09:00] to work on it [16:09:45] I don't think we really need this. But this is just my opinion [16:11:08] I mean you can clone a repo and upload with a different remote, is very similar [16:13:10] Yep i just want to be able to use git if i want to [16:13:28] Since arc support is new on windows [16:13:41] And windows isent a high priority [16:13:42] for them [16:13:53] and git supports windows [16:13:59] and has done for a long time [16:51:27] Luke081515 want to switch some more repos onto diffusion mirror [16:51:33] please [16:51:38] Do you have some repos that you want switching on [16:51:44] Ican get the link [16:51:49] For you [17:35:46] ostriches: You frogot to tag mw core 1.23 lts [17:38:45] Woops my mistake it has sorry about that [22:13:03] 05Gerrit-Migration, 10Diffusion, 10GitHub-Mirrors, 06Repository-Admins: Have Phabricator take over replication to Github - https://phabricator.wikimedia.org/T115624#2315438 (10Paladox) These repos have been switched on for diffusion mirroring. * mediawiki extension VisualEditor * visualeditor/visualedito... [22:16:18] 05Gerrit-Migration, 10Diffusion, 10GitHub-Mirrors, 06Repository-Admins: Have Phabricator take over replication to Github - https://phabricator.wikimedia.org/T115624#2315440 (10Paladox) For mw core and operations-puppet we can do this mw core [remote "git@github.com:wikimedia/mediawiki.git"] url =... [23:59:12] 10Continuous-Integration-Infrastructure, 05Gerrit-Migration, 06Release-Engineering-Team, 10Differential, and 3 others: Investigate migrating the Wikipedia Android App to Differential - https://phabricator.wikimedia.org/T134505#2315512 (10Niedzielski) @dbrant, is this something we could squeeze in a little...