[00:01:04] done? [00:01:28] AndyRussG|a-whey: forcing a submit despite jenkins, I believe [00:01:38] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi Prospect Field - Saving Error - https://phabricator.wikimedia.org/T107087#1764368 (Eileenmcnaughton) I think it's worth retesting in 4.6 - I couldn't replicate it on 4.6 - but I may have not understood the steps [00:01:47] Weeee [00:01:52] Good luck!! [00:03:52] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249652 (owner: Eileen) [00:04:46] awight: we have some greenery https://gerrit.wikimedia.org/r/#/c/249654/ [00:08:09] (Abandoned) Eileen: Merge branch 'civi-4.6.9' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/249589 (owner: Eileen) [00:08:38] eileen: Hi-five! [00:08:49] And that's the head of the new branch? fantastic. [00:08:57] yes [00:09:15] I feel like I've eyeballed the merge 'reasonably well' [00:09:17] want me to create a deployment-plus-4.6.9 as well, or perhaps we should use civi-4.6.9-deployment? [00:11:44] I'm not sure what's best there... [00:12:31] do you want to eyeball the 4 fix-merge-mess commits … https://gerrit.wikimedia.org/r/#/c/249650/1 [00:13:14] awight, eileen: Hey, so... I'm going to sign out for a while. If you need me for anything (though at this point I honestly don't know what you could possibly need me for), go ahead and use the secret code to blow up my phone. [00:13:48] (CR) Awight: [C: 2] "I think that's right!" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249650 (owner: Eileen) [00:13:48] K4-713: cool - we are just waiting now … patiently [00:13:59] That's the "fun" part, right? [00:14:02] :p [00:14:06] K4-713: thanks! [00:14:12] (CR) jenkins-bot: [V: -1] Merge branch 'civi-4.6.9' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249650 (owner: Eileen) [00:14:13] up to Running task: Create financial records for contributions [00:14:54] you are champions [00:15:06] i will also be aroundish tonight if there's anything i can help with [00:15:07] eileen: WOOHOOO [00:15:08] ...seriously. [00:15:10] right on time [00:15:14] CHAMPIONS. [00:15:34] I wondered if prod would be faster than staging [00:16:36] it's the same box, but doesn't have to do its own DB work [00:16:47] I guess we're limited by the db server anyway [00:17:12] ok [00:17:48] so I guess there are another 4 gerrits to be reviewed before it will merge [00:18:24] I'll kill out the others against the other branches [00:19:55] (Abandoned) Eileen: Use Civi method to cancel recurring via qc [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249564 (https://phabricator.wikimedia.org/T110367) (owner: Eileen) [00:20:28] (Abandoned) Eileen: Reintroduce minor change lost in merge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/249597 (owner: Eileen) [00:20:42] (Abandoned) Eileen: Revert "Revert "4.6 upgrade preparation, replace contribution_type with financial_type"" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/249594 (owner: Eileen) [00:20:56] (Abandoned) Eileen: Revert "Revert commits on wrong branch" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/249595 (owner: Eileen) [00:21:13] (Abandoned) Eileen: Remove trailing spaces [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/249596 (owner: Eileen) [00:22:57] (CR) Eileen: [C: 2] "no change in this" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249653 (owner: Eileen) [00:23:26] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249653 (owner: Eileen) [00:23:47] (Abandoned) Eileen: Revert commits on wrong branch [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249607 (owner: Eileen) [00:23:59] (Abandoned) Eileen: Refactor BaseAuditProcessor main loop [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249613 (https://phabricator.wikimedia.org/T111425) (owner: Eileen) [00:24:12] (Abandoned) Eileen: Lean on the convenience fn even more [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249622 (owner: Eileen) [00:24:25] (Abandoned) Eileen: File-extension-proof WP audit filename parsing. [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249630 (owner: Eileen) [00:24:34] (Abandoned) Eileen: Tighten worldpay®ex_for_recon for WPG [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249631 (owner: Eileen) [00:24:47] (Abandoned) Eileen: Move Context/Logger init to wmf_audit_main [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249632 (owner: Eileen) [00:25:00] (Abandoned) Eileen: More updates to the French thank-you template [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249633 (https://phabricator.wikimedia.org/T115496) (owner: Eileen) [00:25:10] (Abandoned) Eileen: Update SmashPig lib [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249634 (owner: Eileen) [00:25:24] (Abandoned) Eileen: Update DonationInterface lib [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249635 (https://phabricator.wikimedia.org/T116199) (owner: Eileen) [00:26:00] (Abandoned) Eileen: Remove bad code [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249636 (https://phabricator.wikimedia.org/T116817) (owner: Eileen) [00:26:07] (Abandoned) Eileen: Fail gracefully if campaigns were added using the native Civi interface [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249637 (https://phabricator.wikimedia.org/T116423) (owner: Eileen) [00:26:16] (Abandoned) Eileen: Revert "Revert "4.6 upgrade preparation, replace contribution_type with financial_type"" [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249638 (owner: Eileen) [00:26:30] (Abandoned) Eileen: Revert "Revert commits on wrong branch" [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249639 (owner: Eileen) [00:27:12] (Abandoned) Eileen: Revert "4.6 upgrade preparation, replace contribution_type with financial_type" [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249608 (owner: Eileen) [00:27:26] (Abandoned) Eileen: Fix JPMorgan Importer [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249609 (https://phabricator.wikimedia.org/T113381) (owner: Eileen) [00:27:40] (Abandoned) Eileen: Add Direct Mail Appeal column to Trilogy Importer [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249610 (https://phabricator.wikimedia.org/T114155) (owner: Eileen) [00:28:22] (Abandoned) Eileen: Remove recurring clause from Bengali thank-you letter [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249611 (owner: Eileen) [00:28:36] (Abandoned) Eileen: Remove unused recurring info template partial [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249612 (owner: Eileen) [00:29:02] (Abandoned) Eileen: Fix audit error handling [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249614 (owner: Eileen) [00:29:26] (Abandoned) Eileen: Audit: stop prepending 'RFD ' to gateway refund IDs [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249621 (https://phabricator.wikimedia.org/T115511) (owner: Eileen) [00:29:43] (Abandoned) Eileen: Update name pattern for Amazon audit to find logs [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249623 (https://phabricator.wikimedia.org/T111425) (owner: Eileen) [00:29:50] (Abandoned) Eileen: Make refund import logic reusable [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249624 (https://phabricator.wikimedia.org/T110367) (owner: Eileen) [00:29:57] (Abandoned) Eileen: Use random temp dirs in unit tests [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249625 (owner: Eileen) [00:30:06] (Abandoned) Eileen: Update French thank-you [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249626 (https://phabricator.wikimedia.org/T115496) (owner: Eileen) [00:30:15] (Abandoned) Eileen: Tests for Amazon audit processor [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249627 (https://phabricator.wikimedia.org/T111425) (owner: Eileen) [00:30:23] (Abandoned) Eileen: Don't move "empty" recon files in test mode [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249628 (owner: Eileen) [00:30:29] (Abandoned) Eileen: update SmashPig to ignore blank lines in worldpay audit [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249629 (owner: Eileen) [00:30:53] (Abandoned) Eileen: Amazon audit processor [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249615 (https://phabricator.wikimedia.org/T111425) (owner: Eileen) [00:31:05] (Abandoned) Eileen: Set subscr_id on Amazon recurring if missing [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249616 (https://phabricator.wikimedia.org/T111431) (owner: Eileen) [00:31:15] (Abandoned) Eileen: Make smashpig settings file optional, fix comment [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249617 (owner: Eileen) [00:31:48] (Abandoned) Eileen: WPG RECON [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249618 (owner: Eileen) [00:31:54] (Abandoned) Eileen: Allow for multiple audit files per date [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249619 (https://phabricator.wikimedia.org/T111425) (owner: Eileen) [00:32:01] (Abandoned) Eileen: Respect individual gateway test_mode settings [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249620 (owner: Eileen) [00:32:21] (Abandoned) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into HEAD [wikimedia/fundraising/crm] (civi-4.6.9) - https://gerrit.wikimedia.org/r/249640 (owner: Eileen) [00:33:03] phew that's them all gone! [00:33:48] so, I don't fully understand the right branching etc - but we should probably finalise the code we want to put on master & put that on staging [00:35:52] +1, that sounds like the right approach! [00:36:52] This is the correct head? https://gerrit.wikimedia.org/r/#/c/249654/ [00:37:51] yes [00:38:20] mmm. I thought we had green light [00:38:43] on that gerrit - are you scrolled down? [00:40:03] ooh of course, we only have green lights at the end. [00:40:03] k [00:40:36] (CR) Awight: [V: 2] Merge branch 'civi-4.6.9' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249650 (owner: Eileen) [00:40:50] (CR) Awight: [C: 2 V: 2] Revert "Revert "4.6 upgrade preparation, replace contribution_type with financial_type"" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249651 (owner: Eileen) [00:41:08] (CR) Awight: [C: 2 V: 2] Revert "Revert commits on wrong branch" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249652 (owner: Eileen) [00:41:26] (CR) Awight: [C: 2 V: 2] Remove trailing spaces [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249653 (owner: Eileen) [00:41:40] (CR) Awight: [C: 2 V: 2] Reintroduce minor change lost in merge [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249654 (owner: Eileen) [00:42:16] eileen: That branch looks right, feel free to deploy to staging [00:42:40] ok - at what point do we create the deploy branch of it? [00:42:53] * Jeff_Green back in a minute [00:43:10] we can go ahead and test that now, that would be smart [00:43:20] * awight flounders for a branch name [00:43:33] deployment-prep-4.6.9 [00:43:36] donno [00:43:52] just to make sure we don't have anything that looks like a standard to be followed in the future ;) [00:44:47] ... and lesse, we should create it by merging master-plus-4.6.9 onto deployment [00:45:01] so I'll create those branches in gerrit, on crm and crm/civicrm [00:45:56] but wait--for the crm/civicrm repo I should definitely base the deployment-prep-4.6.9 branch off of civi-4.6.9 rather than master, eh? [00:46:55] awight: so I did a plus branch for both crm & crm/civcirm [00:47:58] cool-i'll base crm/civicrm#deployment-prep-4.6.9 off of master-plus-4.6.9, then? [00:48:13] I think that is === civi-4.6.9, still [00:48:18] yep - I just realised the civicrm submodule version has changed slightly [00:48:52] k [00:49:47] (PS1) Eileen: Adjust civicrm submodule to reflect merged version [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249664 [00:50:25] (CR) Eileen: [C: 2] "just a submodule change to reflect the merge" [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249664 (owner: Eileen) [00:50:54] (CR) Awight: [C: 2 V: 2] Adjust civicrm submodule to reflect merged version [wikimedia/fundraising/crm] (master-plus-4.6.9) - https://gerrit.wikimedia.org/r/249664 (owner: Eileen) [00:52:17] ok, created crm#deployment-prep-4.6.9 = crm#deployment, and now we want to crm#deployment-prep-4.6.9 += crm#master-plus-4.6.9 [00:52:25] ah! hold on [00:52:59] *now* I did that [00:53:05] doing the merge now... [00:53:10] on staging it's 'sort-of' clean agains the +one branch [00:54:11] okay, only conflicts are tests [00:55:14] (PS1) Awight: Merge master-plus-4.6.9 into deployment-prep-4.6.9 [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/249665 [00:55:23] done [00:57:09] Now, creating crm/civicrm#deployment-prep-4.6.9 = master-plus-4.6.9, and that won't need to be changed. I'm just creating it for clarity [00:57:10] well staging 'loads' still [00:57:21] done [00:58:03] lemme mess with staging for a second... try to make it look like we deployed deployment-prep-4.6.9 using the production script [00:58:36] cool [00:58:47] then we can stop & take stock of where we are [00:59:13] ejegg|away: I'm gonna stash this (on staging): sites/all/modules/wmf_audit/astropay/AstroPayAuditProcessor.php [00:59:33] ahh permission denied. Whatev [00:59:45] yeah - I stashed some other stuff - but couldn't stash that [00:59:48] eileen: okay, I've deployed the deployment-prep-4.6.9 branch to staging. Still look good? [01:00:10] awight: no [01:00:17] it's got the old civicrm code! [01:00:40] awight: please do, don't know why theres an uncommittes change there [01:00:54] oh,permissions issues? [01:01:30] ejegg|away: yeah, just rm it when you have a minute [01:01:44] dang,not near a machine i can get to the cluster on [01:02:12] ok no worries, it's not blocking any git stuff [01:02:43] awight: the code on staging now doesn't look like the plus branch [01:03:14] ah, never committed my merge. [01:03:21] Want to take a look? https://gerrit.wikimedia.org/r/249665 [01:03:32] wrong branch! [01:03:43] (PS1) Awight: Merge master-plus-4.6.9 into deployment-prep-4.6.9 [wikimedia/fundraising/crm] (deployment-prep-4.6.9) - https://gerrit.wikimedia.org/r/249667 [01:03:57] There, now it's in the right place. [01:04:36] (CR) jenkins-bot: [V: -1] Merge master-plus-4.6.9 into deployment-prep-4.6.9 [wikimedia/fundraising/crm] (deployment-prep-4.6.9) - https://gerrit.wikimedia.org/r/249667 (owner: Awight) [01:06:24] V-1 doesn't mean anything cos this branch has no tests [01:07:17] I was wondering about that! [01:07:35] & all the conflicts were on the tests - so that's cool [01:12:10] well - I guess that merge all makes sense.. [01:13:56] (CR) Eileen: [C: 2] "This seems OK - the only non-upgrade related stuff seems to be the queue2civicrm improvements" [wikimedia/fundraising/crm] (deployment-prep-4.6.9) - https://gerrit.wikimedia.org/r/249667 (owner: Awight) [01:14:24] (CR) Awight: [V: 2] Merge master-plus-4.6.9 into deployment-prep-4.6.9 [wikimedia/fundraising/crm] (deployment-prep-4.6.9) - https://gerrit.wikimedia.org/r/249667 (owner: Awight) [01:15:03] eileen: Can you rm sites/all/modules/wmf_reports/tests/phpunit/WMFReportsTest.php [01:15:21] or just chmod -R g+w the whole tree [01:16:15] both done [01:17:59] Okay, the tree is updated [01:18:09] civicrm looks right [01:18:42] the web interface loads, but I'm missing the admin menu n stuff [01:18:54] maybe cos my dashboard is crazy [01:19:06] you know a quick way to clear my dashboard items? [01:20:26] okay, I have the menus now [01:20:31] How does it look to you? [01:21:15] checking [01:22:05] I see a few differences against civi-4.6.9 [01:22:05] click around is all good! [01:22:36] what like? [01:22:39] ah, all tests so far [01:22:49] I was thrown off by a .info diff, but it was for test classes [01:23:03] ok [01:23:29] "trailing spaces" it seems did not carry over [01:23:36] whatev [01:23:57] yeah - it might have - that was all about trying to get merge comparisons to make sense [01:24:06] & I think it WAS in civi-4.6.9 [01:24:24] oo [01:24:28] oops [01:24:34] wmf_civicrm/bootstrap.inc has differences [01:24:55] how about I just checkout -p that file... [01:25:26] wmf_civicrm/wmf_civicrm.instal as well... starting to look like a problem [01:25:30] where should I look for those - I did have to edit those files [01:26:03] I'm doing a diff between civi-4.6.9 and deployment-prep-4.6.9, should have only been vendor/ and tests/ changes [01:26:56] err. [01:27:07] Actually, that's wrong. [01:27:28] maybe there's no easy way to check our work... [01:27:54] civi-4.6.9 doesn't have the merge from master in it [01:28:12] so both those files DO have changes from master in the deployment repo [01:28:27] right [01:29:03] I'm thinking about... taking a short walk around the block. This a decent time? [01:29:10] yeah good idea! [01:29:13] :) [01:35:25] diff against master-plus-4.6.9 looks obviously correct, at least [01:53:10] hi! just emailed a small thing I found [01:57:36] Jeff_Green: I don't think we'll have more fr-tech-ops things until tomorrow! Thanks for all the help! [01:57:57] ok great. what time are you planning to start tomorrow? [01:58:14] lessee... [01:58:36] I think the schema migration is done around 10 UTC [01:59:55] so like 6AM my time [02:00:14] in 8 hours... whatever that is [02:00:38] alright [02:00:53] unless you tell me otherwise, I'm going to guess you aren't getting up that early [02:00:57] I don't think I'd be productive at that hour, yeah [02:01:19] ha [02:01:48] ok, so I'll just be around by 9AM my time [02:02:15] great! [02:04:46] eileen: What hours are you planning to work today? [02:05:14] that crazy talk, nvm [02:05:57] I'll do some basic testing maybe 6AM Pacific, and will decide whether we can reenable or rollback [02:06:17] Anything specific you want me to look out for? [02:07:09] Jeff_Green: oh, a thing! Can you disable the listeners somehow? [02:07:23] which ones? [02:07:25] yeesh, I'm creeped out that they haven't been failmailing all day [02:07:36] smashpig or the legacy one? [02:07:38] awight: yeah - I'll check in in 8 hours before I go bed [02:07:41] the PaymentsListeners and SmashPig stuff that gets web routed [02:08:04] really?? [02:08:09] eileen: please don't fall in any rabbitholes! [02:08:23] Jeff_Green: No idea why it's not exploding [02:08:23] i didn't think those cared about civi/drupal backend [02:08:28] why would it be? [02:08:29] hunh [02:08:33] lemme check [02:09:06] *ok* ignore me [02:09:13] ok [02:09:16] The only thing that connects to civi is an audit processor [02:09:23] wait should I ignore you now? or before? [02:09:25] :-P [02:09:30] hehe [02:09:40] probably a permanent thing [02:38:22] (CR) AndyRussG: [C: -1] "Yay, almost there!!! :D (Apologies for the -1... See inline...)" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [06:22:28] Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising-Backlog, Patch-For-Review: Add "direct mail appeal" column to Trilogy import - https://phabricator.wikimedia.org/T114155#1764743 (awight) @XenoRyet: Nice one! [06:36:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Coinbase import fails with database error - https://phabricator.wikimedia.org/T116445#1764757 (awight) @LeanneS: Please do upload to the fileshare when you get the chance! It sounds like tech should take a look at the new file before you import it, so... [11:01:31] Wikimedia-Fundraising, WMF-Design, Wikimedia-General-or-Unknown, Easy, Mobile: [[wmf:Benefactors]] not displaying well on Mobile - https://phabricator.wikimedia.org/T87805#1764962 (Pcoombe) Okay @CaitVirtue, I made some mobile improvements. How does it look now? [13:17:47] Jeff_Green: ohai! [13:18:21] A thing that's been bugging me: or do you know of a way to restart the Jenkins UI with all the jobs still disabled? [13:18:39] I'd like to slow-start them one at a time [13:21:59] good morning awight [13:22:10] hmm, offhand no, but there must be a way [13:22:26] lemme see [13:22:38] Looks like the big bad schema migration has blown over without a hitch. [13:23:06] wait, now I'm looking at an exception [13:45:55] Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Tech Backlog, Fundraising-Backlog, and 2 others: Civi upgrade drush script should return non-zero error code on failure - https://phabricator.wikimedia.org/T117040#1765170 (awight) NEW [13:46:46] I open to anything, here. If we do the migration on staging, I think it only means we have to copy the db over when complete. [15:03:12] Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Tech Backlog, Fundraising-Backlog, and 3 others: Stop calling CREATE FUNCTION from application code - https://phabricator.wikimedia.org/T117044#1765342 (awight) NEW [15:10:53] awight, anything I can do to help? Looks like a permissions error stymied the schema upgrade attempt? [15:11:09] ejegg: Not sure what caused it yet... [15:11:17] but yeah, FUNCTION-related [15:11:26] I'll have a patch to review in a few seconds! [15:12:37] (PS1) Awight: Don't create SQL function at runtime [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) [15:13:49] going straight to civi-4.6.9-depoyment with that, cos the cherry-pick into new master and deployment is going to be indirect no matter what. [15:14:21] (CR) jenkins-bot: [V: -1] Don't create SQL function at runtime [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) (owner: Awight) [15:14:38] V-1 cos no tests. [15:17:22] Fundraising Sprint William Shatner, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Staging must have the same db permissions as production for the civi user - https://phabricator.wikimedia.org/T117047#1765395 (awight) NEW a:Eileenmcnaughton [15:19:20] awight: did you sleep at all? [15:19:55] a few hours. Planning on an afternoon nap while a bunch of hdds heat up [15:20:02] ... one way or another. [15:20:11] heh [15:20:38] so there is some function/trigger rebuilding that it's choking on? [15:20:43] cwd: ejegg: I could use review of the steps in https://etherpad.wikimedia.org/p/Civi_upgrade_4.6__running_notes #RECOVERY ALTERNATIVES #option 3 [15:20:46] yeah [15:21:10] sure thing [15:21:12] i remember that stuff having to be done in a very specific order. is that the problem? [15:21:16] and yeah checking it out [15:21:56] Long story short, I'm taking a minute to go for the spliced migration. [15:22:16] I could also use a cross-check of my judgement call, to attempt repair at all. [15:22:29] awight: the log is so short - is there usually more info in it before the point where the error occurred? [15:22:55] Jeff_Green estimates 2-5 hours to roll back, so we sort of have until Pacific afternoon to make the final do/die call. [15:23:03] ejegg: yeah, different file-- [15:23:18] see https://phab.wmfusercontent.org/file/data/y4ww26oee73vuykozswu/PHID-FILE-6mfqyl3o7afuy5ngp7hg/hp7sbl6hv2ihysop/civi-4.6-upgrade-20151024.log [15:23:46] thanks! [15:24:17] ejegg: That's on the filesystem also, under sites/default/files/civicrm/ConfigAndLog/*.log [15:24:23] ah, cool [15:24:54] Jeff_Green: all that happened was, I'm leaning on cwd and ejegg's shoulders to get review of my patch, the plan, and the overall judgement call to even attempt this splice job. [15:25:10] so it did all the schema updates from the .sql.tpl files, but one of the php upgrade tasks bombed out after that [15:25:29] awight: great [15:25:33] ejegg: Yeah, in ./CRM/Upgrade/Incremental/php/FourThree.php: [15:25:42] (nothing follows my ":") [15:26:30] I'm gonna soothe my soul by checking on family awakeness for a few minutes. [15:27:20] :) and shoveling poopy duck water [15:34:22] what exactly failed? guessing it's: CRM_Core_DAO::executeQuery(CRM_Contact_BAO_Contact::DROP_STRIP_FUNCTION_43); [15:34:26] but what did it complain about? [15:35:34] (CR) Ejegg: "Also need to comment it out in the upgrade script." [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) (owner: Awight) [15:39:12] (CR) Ejegg: "oops, nvm, missed the bit where you plan to tell Civi it finished that part of the upgrade" [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) (owner: Awight) [15:40:08] cwd: permissions error "alter routine command denied to user ..." [15:42:45] weird! that can't have been the first destructive edit, what is different about dropping a function? [15:43:07] or is that the open question [15:43:38] "Drop if exists", maybe it's the exists condition that avoid the failure [15:44:20] ejegg: yes, agreed about the oversight, I might as well comment the other usage out in case we forget about it later. [15:45:41] ejegg: actually, I'm gonna leave those in--you'll need to run the create function as root anyway to do a successful migration, so it *should* break at that point if the function doesn't exist... [15:45:53] grr. [15:46:06] cwd: I think we're being pretty conservative in granting permissions, and that's the first we've seen it need to do anything with functions [15:46:41] awight|soul: yeah, actually leaving it in the upgrade script is the right thing to do for all the dev environments [15:46:53] ah. good argument. [15:47:28] Am I needed for anything at the moment, or should I drift away for another 15m? [15:47:42] Please take your well-needed break [15:47:45] :p [15:47:48] don't worry [15:47:53] well-deserved too [15:48:47] Anyway, the 'pick up where we left off' plan is looking good to me, as long as you can tell it not to try to re-apply all the .sql.tpl files from 4.3a1 - 4.6 [15:49:29] i doubt anything is corrupted given where it died [15:49:42] https://phabricator.wikimedia.org/T117052 [15:49:53] this looks fun [15:50:36] oh waitaminute, that log with all the SQL in it is an example of a test migration that worked - that's why it's got the rest of the sql [15:50:58] so, yeah, I think those steps ought to be just right [15:51:50] PPena: anything i should know ^ [15:54:05] ejegg: "an example of a test migration that worked" yes [15:55:29] awight|soul: looks like there is more trigger creation happening in 4.6 - are we granting those permissions? [15:55:49] ejegg: That should all be commented out. Where is this? [15:56:00] cos: no, we don't have the permissions [15:56:13] CRM_Core_DAO::triggerRebuild [15:57:35] ah, OK, only called when that function doesn't exist [15:57:47] wat. ok thx [15:58:18] so we might want to create those triggers as root too, I think... [16:00:06] hmm, ok, it's just updating the last_modified date on civicrm_contact when any of the related records in address/email/phone change or are deleted [16:00:32] ejegg: good find [16:00:41] or rather, modified_date [16:01:00] not sure if that's important for e.g. merge logic [16:01:05] What a crappy test: if !function create trigger [16:01:14] yarp [16:02:20] awight|soul: wow, especially 'cause the upgrade script creates the function but not the triggers, as far as I can tell! [16:03:08] awight|soul: assuming you're not coming to tech priorities :) [16:03:13] ejegg: wait, we've commented out all the base stuff to create triggers, right? [16:03:28] atgomez: I think it's better to focus on the fix [16:03:37] we only commented out the bit that tests for trigger permission! [16:03:56] ejegg: but in previous patches, we should have taken out anything with the potential to CREATE/DROP triggers [16:04:05] awight|soul: word. [16:04:34] awight|soul: well, there are a couple uncommented places left. I'll submit a patch to snip em [16:04:48] ! ok that's scary to hear, thanks for finding them! [16:05:15] atgomez: Anything you want to share with the group, that I can provide? [16:06:04] awight|soul: submit on civic-4.6.9-deployment? [16:06:26] ejegg: I'm thinking so, yeah. [16:06:30] shortest path... [16:06:37] k [16:07:55] oh hey, that branch has stuff commented out! [16:08:14] sorry, didn't fetch all the latest before crying wold [16:08:16] *wolf [16:09:05] hehe wasn't the last patch yours, too? I hate doing that to myself [16:09:55] heh, somehow I was on a revision that didn't have the bit in DAO::createTriggers commented [16:10:14] I think it came after we cut the branch [16:10:33] ok, plan still looks solid [16:14:13] It's batshit... The plan is always to rollback in case of anything, I'm just being greedy at this point. [16:14:55] awight|soul: naah, that's the beauty of scripting the upgrades as incremental! [16:15:06] cwd MBeat will be filling out that ticket [16:15:35] sorry on a Zendesk billing call [16:16:53] Jeff_Green: are you on board with the repair plan? Etherpad should be up to date. [16:17:19] looking [16:17:22] ejegg: it's a pretty huge cost difference, to rollback and reschedule. [16:18:01] Currently, we're looking at a 5-hour timeframe to repair, plus followup testing, which is a clear win over rollback if that takes 5 hours. [16:18:28] yeah, sounds especially hairy with the one db that we want to stay live on the same replicated server as a db we'd want to roll back [16:19:05] awight|soul: I'm not sure I'm reading the right section here [16:19:18] RECOVER ALTERNATIVES #3? [16:19:18] The outline under "alternative" #3 [16:19:20] yeah [16:19:22] ok [16:19:33] "magic function" is part of that approach [16:19:58] * awight|soul shoves a freshman into a locker then immediately feels remorse [16:20:33] awight|soul: it's ok with me. if i understand correctly, the subsequent fail plan would be to go back to #1 right? [16:20:39] yes [16:20:50] no questions asked: at that point, it didn't work. [16:21:04] my main concern is that we would somehow lose track of peripheral foo (mainly triggers) in the shuffle on db1025 [16:21:21] We can live w/o the triggers, if that happens [16:21:27] they are confirmed non-critical [16:21:57] also i don't entirely understand mysql "functions", not confident they're making it into dumps etc. [16:22:21] Ino, I'm sad about that one. [16:22:22] function == stored procedure, right? [16:22:25] yeah [16:22:52] actually--we don't have to worry about the function too much cos after upgrade, it's only called from a trigger--which won't exist :) [16:23:23] awight|soul: are the stored procedures being restored when you drop/restore the test civi from the dump we made a few weeks ago? [16:23:55] Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Revisit triggers. We probably want them. - https://phabricator.wikimedia.org/T117055#1765636 (awight) NEW [16:24:21] Jeff_Green: it's just this one, and wan't introduced until 4.3 (our dumps are from 4.2) [16:24:22] awight|soul: should we be compensating in code? Not that we're going to be automatically calling or texting anyone, which is the only reason I can thing we'd need a pure numeric phone number [16:24:30] awight|soul: ok [16:24:34] ejegg: I can't even fathom it [16:24:46] i'm on board with plan #3 [16:24:59] ok. Shall we begin [16:25:20] sure [16:25:55] Gimme 5 minutes. I'm playing with tanks of propane and oxygen while I watch progress bars, it's somehow relaxing compared to the db stuff [16:26:26] hopefully you aren't chain smoking at the same time [16:26:28] get some hydrazine in there and you'll really chill out [16:40:38] Wikimedia-Fundraising, WMF-Design, Wikimedia-General-or-Unknown, Easy, Mobile: [[wmf:Benefactors]] not displaying well on Mobile - https://phabricator.wikimedia.org/T87805#1765684 (CaitVirtue) Looks great! These improvements make the page good enough for Q2. I think there's more we can do here... [16:43:24] hehe. family existed, that was nice [16:44:54] Jeff_Green: I'm gonna start running the #3 repair on production:civicrm [16:45:23] etherpad is down 8) [16:45:36] hehe [16:46:02] awight tell me when you want me to run those queries [16:46:29] Now is good [16:47:13] ejegg: you think https://gerrit.wikimedia.org/r/#/c/249749/ is what we should merge? [16:47:56] mmm, sure, that will do it. K, will +2 [16:48:01] awight here goes [16:48:05] thanks x2! [16:48:12] done [16:48:14] (CR) Ejegg: [C: 2 V: 2] "Should do the trick!" [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) (owner: Awight) [16:48:23] k, now I'm gonna run the update [16:48:31] (update civicrm_phone statement) [16:48:33] how do you show stored procedures? [16:48:43] donno! [16:48:46] show function status? [16:48:52] yeah [16:49:00] sort of. I don't see the body [16:49:01] show function code is a thing too [16:49:15] (CR) jenkins-bot: [V: -1] Don't create SQL function at runtime [wikimedia/fundraising/crm/civicrm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249749 (https://phabricator.wikimedia.org/T117044) (owner: Awight) [16:49:27] ok it's there [16:49:33] there's also one stored procedure [16:49:53] what is it? [16:49:55] awight: I didn't get a 'submit' button on that patch [16:50:10] you have to kill jenkins [16:50:26] a, thx [16:50:39] merged [16:50:40] Jeff_Green: looks like I don't even have the perms to run that function. Can you: ERROR 1370 (42000): execute command denied to user 'civicrm'@'10.64.40.%' for routine 'civicrm.civicrm_strip_non_numeric' [16:50:44] gah! [16:50:44] awight it's silverpop export [16:51:00] oh. ok no dependencies with civi [16:51:04] awight which user are you using? [16:51:13] UPDATE civicrm_phone SET phone_numeric = civicrm_strip_non_numeric(phone); [16:51:19] the application's db user [16:51:33] why not civiadmin? [16:51:44] ok [16:51:59] on production? [16:52:05] same pw? [16:52:16] good question. lemme see if I have that somewhere [16:53:53] see boron:mysql [16:54:30] grrr. that doesn't work for me. hold a sec [16:54:39] in. [16:54:59] running the phone update [16:55:09] ERROR 1290 (HY000): The MariaDB server is running with the --read-only option so it cannot execute this statement [16:55:16] yeah, works from barium [16:55:22] you must be on the wrong server [16:55:25] right! [16:55:29] blaaargh [16:55:37] db1025 :-P [16:56:31] OK the update query is running [16:56:36] done! [16:56:55] now, to jury rig the civicrm_domain.version [16:57:18] we should update the 'howto" doc to "USE CIVIADMIN FOR UPGRADES" and look at making sure that user has adequate privs in the future [16:57:27] YES [16:57:31] yes and yes [16:57:44] it might already, I think that's what we did historically [16:57:57] not positive about the functions/procedures though [16:58:29] I'd be pretty happy if replication could be compatible with triggers and functions. [16:58:44] I would just uncomment all that stuff and let it rip [16:59:13] deploying patched code... [17:01:09] (PS1) Awight: Bump civicrm submodule (T117044) [wikimedia/fundraising/crm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249768 [17:01:22] (CR) Awight: [C: 2 V: 2] Bump civicrm submodule (T117044) [wikimedia/fundraising/crm] (civi-4.6.9-deployment) - https://gerrit.wikimedia.org/r/249768 (owner: Awight) [17:01:57] !log update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78 [17:02:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:02:21] We're ready to restart the schema migration, now. [17:02:28] doing so... [17:04:38] !log restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC [17:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:04:53] Jeff_Green: ejegg: we're back on the road. [17:05:17] yay [17:05:18] It's clunking through the next steps [17:06:27] nice! [17:06:51] ccccccddfrjhgrjnkcbvtjefrebbbjhchgbjeghvcjhi [17:06:55] blargh [17:06:59] yubispam [17:08:00] uhh [17:08:08] the force merge broke zuul again. [17:08:49] argh, f'real? [17:13:04] legoktm: sorry, it's hard to manage new deployment branches. Maybe we should be matching a pattern like branch_name ~= /deployment/ to determine which branches get the unit test treatment. [17:14:15] yes, that's very easy to do actually [17:16:04] Fundraising-Backlog, Continuous-Integration-Config: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1765798 (awight) NEW [17:16:18] legoktm: Thanks! made a note for us to look into it [17:24:09] awight: ejegg: cwd: XenoRyet: hmmm looks like I'm very late to the party... anything I can help with? Testing, looking at code, anything u can think of? [17:24:46] Fundraising-Backlog, Continuous-Integration-Config: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1765837 (Legoktm) The "something in zuul falls over dead" is {T93812} for reference. [17:25:01] ResourceLoader question: there's no way to specify globals (like you specify messages) in the module registration, is there? [17:25:58] AndyRussG: The upgrade is at a pretty calm spot, happily... We're just watching a log file now. [17:26:08] ejegg: nope :( [17:26:15] awight: cool beans!! Many congratulations!!!!!! [17:26:22] legoktm: ah, bummer [17:26:41] ejegg: you can create a "data" module that just has the contents mw.foobar.set( ... ) and construct the JS code in PHP. [17:26:43] AndyRussG: sort of :) the baby looks a little different... [17:26:48] I think VE does that [17:27:35] Ehh, not worth it for my purposes, but good suggestion! [17:27:57] ejegg: out of curiosity, what's the use case? [17:28:12] (PS9) Ejegg: Large banner limit mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) [17:28:16] awight: switched a birth! [17:28:29] wanted to pass $wgCentralNoticeCookieDomain to the large banner limit mixin, but only for campaigns where it's in use [17:28:41] $wgNoticeCookieDomain, rather [17:31:12] (CR) Ejegg: "Thanks for pointing out mw.cookie!" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [17:34:00] fr-tech: noting that the upgrade (take 2) will email all of us when it completes [17:35:28] augh, that's what I get for buying stuff from Harbor Freight: O2 tank is utterly empty [17:35:50] Already been through a return for the $40 regulator cos it was... missing the handle [17:41:27] Gonna pace around [17:48:20] I might go take a bike ride for an hour. Any loose ends? [17:49:09] Civi is currently at: Upgrade DB to 4.4.4 [17:50:10] Call if anything happens! [18:32:22] Fundraising Tech Backlog, Team-Practices-This-Week: Measure Maintenance Fraction for FR-Tech - https://phabricator.wikimedia.org/T114328#1766028 (DStrine) I believe I added the correct information for fr-tech to the wiki page above. We started logging new vs maintenance on resolved tasks starting on 9/30. [18:34:06] (PS1) Cdentinger: refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 [18:34:47] (CR) jenkins-bot: [V: -1] refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 (owner: Cdentinger) [18:36:48] (PS2) Cdentinger: refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 [18:52:49] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Coinbase import fails with database error - https://phabricator.wikimedia.org/T116445#1766075 (LeanneS) @awight I've added this latest file to the same folder on the server. [18:56:49] cwd: wow, 3 minutes? That's crazy! [18:58:12] ejegg: well they haven't said anything yet but i've seen it that low in the log [18:58:20] unless it's non-linear somehow [18:58:29] (CR) Ejegg: [C: -1] "Needs i18n. Also, 3 minute expiration is nuts! They really have to fix that." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 (owner: Cdentinger) [19:00:05] ejegg: doh, yeah this is france after all [19:05:11] awight: hi [19:16:52] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Coinbase import fails with database error - https://phabricator.wikimedia.org/T116445#1766174 (Ejegg) The file from the 19th looks like it's valid UTF-8 too. The problematic character noted in the error message was encoded as C3A4, which is correct for... [19:22:21] ejegg: dang, i added the msg to the i18n files but it's just showing up as the key: [19:22:31] is there something i have to do to refresh the cache? [19:23:04] cwd: did you add it to the message list in the RL module declaration? [19:23:57] nope! thanks [19:25:02] Eileen: hey, I'll be at a computer in 30 min [19:25:11] cool [19:28:09] ejegg: thank fixed it! [19:28:23] nice [19:28:32] (PS3) Cdentinger: refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 [19:29:15] anyone who speaks french want to see if this is an ok translation? https://gerrit.wikimedia.org/r/#/c/249797/3/gateway_common/i18n/interface/fr.json [19:29:16] (CR) jenkins-bot: [V: -1] refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 (owner: Cdentinger) [19:29:26] cwd: Do we want to wait for an email reply before imposing that kind of rapid refresh? [19:30:28] good question, PPena what's your preference? [19:45:44] cwd nope, if you feel like you have enough data that shows its 3 min lets go for it [19:45:55] once they confirm, if they can extend, then we extend on our side too :) [19:58:41] (PS9) Ejegg: Add banner_history module to consume BH log id associations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/246425 (https://phabricator.wikimedia.org/T112022) (owner: AndyRussG) [20:00:33] eileen: hey, hopefully you got some sleep [20:01:21] awight: yeah - I should probably have done more checking before going to bed [20:01:25] eileen: Most of the kerfuffle is documented in https://etherpad.wikimedia.org/p/Civi%20upgrade%204.6:%20running%20notes [20:01:35] naw, there was nothing to check by that time [20:01:48] I was fooled by the "status 0" email at first--that was a cruel letdown [20:02:08] found a big old stack trace during FourThree.php, when trying to "create function" [20:02:25] yeah - I saw that & had a brief go at tunnellingthgouth & decided I was too tired to figure out why the tunnel didn't work [20:02:29] (PS4) Cdentinger: refresh page after 3 minutes (token expiry) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 [20:02:52] eh tunneling won't do much until the schema migration is complete, anyway [20:03:15] lessee... we're currently at "Running task: Upgrade DB to 4.5.beta9: SQL" [20:03:26] yep [20:03:41] 1.5hr to go [20:04:16] sweet [20:04:26] I can see why you wanted those timings! [20:07:09] it is helping quite a bit! [20:07:18] At least with the anxiety [20:07:25] :-) [20:09:34] (CR) Ejegg: [C: 2] "Looks like this will do the job!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/246425 (https://phabricator.wikimedia.org/T112022) (owner: AndyRussG) [20:12:22] Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Tech Backlog, Fundraising-Backlog, and 4 others: Stop calling CREATE FUNCTION from application code - https://phabricator.wikimedia.org/T117044#1766426 (Ejegg) p:Unbreak!>Triage [20:36:50] (CR) Ejegg: "Think it's worth adding a config variable for the refresh timeout? This is good behavior to have, but I hope they'll be able to give us l" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/249797 (owner: Cdentinger) [20:38:27] Wheee, we're in the 4.6s ! [20:39:34] nice! [20:41:17] what's the deal with deployment-prep-4.6.9? [20:41:31] oh, that's just a topic, branch is deployment... nvm [20:42:48] ejegg: deployment-prep-4.6.9 is ridiculous. That's civi-4.6.9 + master + deployment, it's meant to be the new deployment branch. [20:43:16] Currently, we have civi-4.6.9-deployment on production, that was just civi-4.6.9 + patch for deploymentizing [20:43:34] k [20:43:42] we're not going to build on that branch, though, it gets tossed w/o merging into anything [20:45:06] we'll merge that to deployment now to update prod, then turn master-plus-4.6.9 into master? [20:46:33] ah, confusing, there are two versions of that patch. I see one has deployment-prep as the actual branch, not the topic [20:47:32] ouch! I see [20:47:39] I just added some notes at the bottom of the etherpad [20:48:58] eileen: ^ please adjust my descriptions of each branch if you remember differently... [21:19:16] Jeff_Green: I think the odds are low and decreasing that we'll have to do a rollback, btw. [21:20:01] If this finishes successfully, would you mind snapshotting the civicrm database at that point? [21:20:01] dang my internet is being super flakey! [21:20:02] (CR) AndyRussG: [C: -1] "Cool! Just one more detail: this module needs to depend on mediawiki.cookie now. :)" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [21:20:11] awight yup [21:20:19] ejegg: It's quiet in here, you haven't missed much! btw, /topic [21:20:30] there's a public log fwiw [21:20:55] awight it will have to be the master that we dump, it'll take maybe 20 min [21:21:20] log is where? [21:21:29] Jeff_Green: That's worth it [21:21:36] k [21:21:42] ejegg: It's in the topic: http://bit.ly/1eJqqCc [21:21:58] oh, heh [21:22:08] I thought you meant the db upgrade log [21:22:40] that was eileen complaining of netflakes... [21:22:43] oh joy, that would have been crazy [21:22:46] yeah, exactly [21:37:38] Just a couple of minutes left. [21:38:42] eileen: please don't log in or do anything to the production civi when it's done--JGreen is gonna start a backup of the fresh 4.6.9 data [21:38:54] That should take c. 20 min [21:39:40] awight: cool [21:39:40] damn, it feels good to be a gangsta [21:39:58] Jeff_Green: the migration is complete, ready to make that dump! [21:40:25] ok here goes [21:40:55] (PS10) Ejegg: Large banner limit mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) [21:41:43] (CR) Ejegg: "Ah, so it does. I guess mw.util is the only one we can take for granted" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [21:46:12] hey cwd! on that task you can unfortunately view edit history… mind removing the comment & recommenting if you want to keep it? https://phabricator.wikimedia.org/T117052 [21:47:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Staging must have the same db permissions as production for the civi user - https://phabricator.wikimedia.org/T117047#1766824 (awight) [21:47:53] Fundraising Sprint Vengaboys, Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Civi upgrade drush script should return non-zero error code on failure - https://phabricator.wikimedia.org/T117040#1766826 (awight) [21:48:37] atgomez: sorry about that! removing now [21:50:16] Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spot-check that all core and custom data is preserved during the migration - https://phabricator.wikimedia.org/T115427#1766848 (awight) p:Normal>U... [21:51:15] thanks cwd [21:52:27] Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Corrupt dashboard items are throwing warnings during Civi upgrade - https://phabricator.wikimedia.org/T116521#1766863 (awight) [21:52:29] Fundraising Sprint UB40, Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Orphaned action is emitting warnings during Civi upgrade - https://phabricator.wikimedia.org/T116522#1766864 (awight) [21:53:23] Fundraising-Backlog, Continuous-Integration-Config: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1766875 (hashar) Yup force merge is not supported by Zuul and that has the side effect of dead locking it. The workflow is: ``` - name: wikimedia/fundraising/crm check:... [21:54:04] Fundraising-Backlog, Continuous-Integration-Config: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock of Zuul) - https://phabricator.wikimedia.org/T117062#1766887 (hashar) [22:05:44] !log CiviCRM upgrade to 4.6.9 succeeded; new data is backed up [22:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:05:55] Enabling the Drupal modules... [22:06:33] cool [22:06:50] ooh, tingling with excitement! [22:07:00] :p [22:07:07] It was the anticipation [22:07:29] Jeff_Green: feel free to "Unkill the slow db query killer" any time [22:07:50] eileen: let us know if the tunneling is burdensome [22:07:51] awight: that's not a thing anymore afaik [22:07:57] ooh, okay then [22:08:03] awight: I can't see it on the list but drush updb [22:08:16] to grab those few updates in wmf_civicrm [22:08:30] Jeff_Green: also, please enable Jenkins but w/o running any jobs, if possible... [22:08:40] the tunnelling might not have been that bad but I was quite tired at the time [22:08:47] awight before we do that.... [22:08:56] we still have replication disabled [22:09:05] eileen: sure! we're on civi-4.6.9-deployment, that includes the modules you're thinking about? [22:09:08] Jeff_Green: k [22:09:20] awight: yes it does [22:09:27] cool [22:09:27] and it's going to take many hours for the slaves to catch up with all these queries [22:09:29] although poss better to switch over first [22:09:41] but they are just a couple of minor upgrade actions [22:09:43] Jeff_Green: okay, we can leave the jobs down for that long [22:09:50] does anything of import use the read-only instance on db1008? [22:09:58] Jeff_Green: it's still fine if we're doing low-traffic testing stuff, though? [22:10:01] If I had slaves I'd make them work faster [22:10:06] Jeff_Green: I don't think so [22:10:08] O_O [22:10:17] It always comes to this [22:10:29] are we ready to start replication? [22:11:07] yes [22:11:38] Jeff_Green: can fr-tech poke at it before replication catches up? [22:11:47] All we're risking is loss of test data, right? [22:12:00] poke at what? [22:12:14] It would be great to start testing the web UI and drush commands [22:12:28] We don't need real jenkins jobs until the very end, so that's fine [22:12:35] But it would be a shame to lose the testing time now [22:13:14] err: [poke at production Civi's] web UI and drush commands [22:13:55] awight: lemme start replicaiton and then we'll talk [22:15:17] (CR) AndyRussG: [C: 2] "Excellent!! :) Thanks!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [22:17:04] Fundraising Sprint Queen, Fundraising Sprint Rowlf the Dog, Fundraising Sprint Snoop (Dogg|Lion), Fundraising Sprint The Pogues, and 6 others: Log DonationInterface fatal errors to syslog - https://phabricator.wikimedia.org/T107918#1766980 (atgo) This task was in the done column in Venga Boys. I... [22:26:12] (Merged) jenkins-bot: Large banner limit mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247460 (https://phabricator.wikimedia.org/T90919) (owner: Ejegg) [22:29:19] awight ok replication is going for db1008 and lutetium. I'm going to leave fdb2001 off for now [22:31:42] awight: i didn't manage to research how to start jenkins with the jobs disabled, there's probably a way but... [22:34:10] i f'ing hate jenkins. [22:34:33] +1 [22:35:55] everything it does uses drush right? [22:36:01] Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, Fundraising Tech Backlog, Fundraising-Backlog, and 4 others: Stop calling CREATE FUNCTION from application code - https://phabricator.wikimedia.org/T117044#1767038 (awight) a:awight [22:36:12] it's sooo tempting to move drush aside long enough to get jenkins up and go in and disable every job [22:36:13] Jeff_Green: not quite [22:36:22] dstrine: hi! looking for some guidance about what task to take on next :) For stuff that I'm familiarish with currently in the sprint, there's T97676 (Kafkatee) and T113013 (ie8 and 9), though I think maybe ejegg was interested in, or at least was thinking about taking on, the latter? I could also try my hand at a task in some the DI or Civi codebases, though I'd probably be slower at those, since I'm still quite [22:36:22] new with them. For now I can also just try to plow through my to-do list of minor stuff (but that certainly won't last the rest of the sprint...) [22:36:25] I was thinking about the "disable real fast" approach, too [22:36:41] It's not the end of the world if a job twitches for a second. [22:37:00] We could all LOIC dogpile on the web UI [22:37:20] ugh--if it didn't take me twenty tries to log in, using the correct password [22:37:25] heh [22:37:49] ejegg: do you have opinions about AndyRussG comment above on T113013 [22:38:11] I think ejegg wasn't super into it lol [22:38:21] just checking whether I still have a functioning IE VM at all [22:38:39] we could do some fun with sed on false :-P [22:39:53] https://wiki.jenkins-ci.org/display/JENKINS/Configuration+Slicing+Plugin [22:40:19] there. pages and pages of googling to reveal a plugin. kill me now. [22:41:58] which should have been "edit puppet vars generating /etc/crontab; deploy" [22:42:21] ejegg: If you happen to be into it ;p it's pretty straightforward: https://dev.modern.ie/tools/vms/linux/ [22:42:30] no pressure! heh :) [22:42:40] I do have those already set up [22:42:46] Jeff_Green: whaddayu bet we can't even install the plugin without starting the service and enabling all the jobs anyway [22:43:00] * Jeff_Green likes the sed -i approach, but I'm only 39% confident it will work [22:43:27] every jenkins job has a config.xml file which has that false [22:43:35] yep, just hoping I could avoid downloading them all again. My XP image is refusing to do anything without a serial number - though I had made a snapshot on initial install [22:43:37] the question is whether it's also stored elsewhere [22:44:07] we have jenkins-cli fwiw [22:44:43] harr: "You can also go to http://your-jenkins:8080/cli for some of the available commands." [22:44:50] that pretty much sums it up [22:44:59] go to --hellp [22:45:08] does jenkins-cli work offline? [22:45:28] probably not. [22:45:49] jenkins-cli login [22:46:00] uhh that was supposed to be at the shell [22:46:18] nope, the cli works through an URL [22:46:23] IDIOTS [22:48:09] ok well here's a question for you [22:48:26] if I'm reading the XML right, I think only one of all these jobs is disabled [22:48:48] does that fit your memory of what jenkins looks like when it's up? [22:48:50] where is this xml btw? [22:48:52] yeah it does [22:49:01] the "one-offs" job is the only one, right? [22:50:42] ejegg: dstrine: ^ soo.... ? [22:51:46] AndyRussG: If you've already got those set up, you have a head start on me! [22:52:10] Now that the Civi DB is back up I can start looking at the silverpop dedupe [22:52:27] ejegg: K I'll grab that saliva-ed IE cookie then! :) [22:52:36] ejegg: note, we're still waiting to do anything write-y to the civi db [22:52:40] Mmm spit + bad software [22:52:49] Jeff_Green: ^ should we resolve that before tackling Jenkins? [22:53:08] ejegg: and AndyRussG thanks guys. I was about to say "it's up to you guys" [22:53:08] awight: for sure, that ticket only needs selects [22:53:49] Fundraising Sprint Tom Waits, Fundraising Sprint UB40, Fundraising Sprint Vengaboys, Fundraising Sprint William Shatner, and 3 others: Fix banners on IE8 and IE9 - https://phabricator.wikimedia.org/T113013#1767131 (AndyRussG) a:AndyRussG [22:53:54] Kafkatee, the ie bug and silverpop are all important. It's better then are done before scare week. [22:54:36] or "freak week" whatever K4 calls it [22:54:37] awight resolve which? [22:54:40] dstrine: yeah we really are down to the gold at the bottom of the pan [22:54:55] Jeff_Green: We're waiting for your OK to start testing the new production Civi [22:55:08] You said, don't do anything until replication catches up? [22:55:11] oh. sorry I lost that fact. sure why not? [22:55:15] hehehe [22:55:18] I like it [22:55:20] naah, replication is going to take many hours [22:55:21] right? [22:56:01] Jeff_Green: so in effect you're saying: eileen: ejegg: game on, right? [22:56:12] yes [22:56:29] i'd rather know soon that it's a go [22:56:36] me too [22:56:44] fr-tech: GAME ON! [22:56:49] we can test the new Civi. [22:56:58] you want me to take down the 503 wall? [22:57:06] mmm. sure [22:59:36] ok here goes. leaving jenkins down of course. [23:00:01] should be back online [23:02:26] site is taking its time loading [23:02:42] yeah [23:02:42] awight: staging? [23:02:48] hey, it loaded! [23:02:50] ah, there it is! [23:02:52] cwd: prod [23:03:09] w/ tunnel? [23:03:15] Ouch, reloading the page takes just as long [23:03:25] there are horror queries happening [23:03:35] cwd: JGreen just put the proxy back inplace [23:03:37] Jeff_Green: nooo [23:03:52] eileen: ^ [23:03:52] aw jeez, like what? [23:03:54] they're not long, there just...there [23:04:12] Is this from junk we have on our dashboards? [23:04:20] dashboards should all die. [23:04:28] I might just drop that table [23:04:43] * awight 's moment passes [23:04:51] removed my dashlets [23:05:08] it does look dashboardy [23:05:16] seriously? SELECT count(civicrm_contribution.id) as DATAOBJECT_NUM FROM civicrm_contribution WHERE ( is_test = 0 ) [23:05:40] aw man, why for they wanna do that? [23:05:45] where is that? [23:05:51] lots of them 'Sending data' [23:06:21] looks like its happening for many civi uids [23:06:28] eileen: Is there a way to remove all users' dashlets, without damaging the dashboard system? [23:06:33] ok, I'll stop hammering on it till some of that stuff dies down [23:06:55] Even without the dashlets, it's taking 15s to load [23:07:10] awight: I think you can truncate civicrm_dashboard_contact [23:07:40] also - you can set a cache time on them https://civicrm.wikimedia.org/civicrm/admin/setting/misc?reset=1 [23:07:48] I'll hold off until we figure out what's happening [23:08:04] Seems to be slow on every page [23:09:15] There's an issue that keeps regressing back in, where some Drupal requests try to do some statsy thing with a connection to self, but that isn't possible with our proxy setup, so the backend has delay about this long. I wonder... [23:09:48] !log set dashboard cache interval to 60m [23:09:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:09:56] oops. in the CRM. [23:10:53] awight: i can't remember how we discovered that stats-self-reconnect thing [23:13:02] Jeff_Green: it was so evil. We slew it with a settings.php thing, but went back and forth about the right boolean to cause disablement. [23:14:12] Hey, the site is responding normally now. [23:15:38] Wikimedia-Fundraising-CiviCRM: Slow query on deduping 2 contacts - https://phabricator.wikimedia.org/T116886#1767224 (Eileenmcnaughton) I tested this on live & got 10 sec to load this page (4.6) https://civicrm.wikimedia.org/civicrm/contact/merge?reset=1&cid=2713553&oid=4961909 [23:18:49] awight: that bug I just commented on probably wants to be given some priority - based on my digging on it so far (not necessarily today but it does mean deduping 2 contacts is doing stuff it shouldn't) [23:19:41] Wikimedia-Fundraising-CiviCRM: Slow query on deduping 2 contacts - https://phabricator.wikimedia.org/T116886#1767247 (awight) [23:19:42] Fundraising Tech Backlog, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, Patch-For-Review: [epic] Upgrade Civi to 4.6 & integrate new reporting - https://phabricator.wikimedia.org/T77910#1767246 (awight) [23:19:51] also doesn't look like civicrm_wmf install updates ran? [23:20:04] eileen: It does look scary. They use 2-contact dedupe pretty regularly. [23:20:16] eileen: oh, thx! lemme run the updatedb [23:20:41] awight: ok - I discovered it earlier in the week when trying to add hooks [23:21:03] I Think this was harmless: Missing argument 6 for CRM_Core_BAO_Navigation::createOrUpdateReportNavItem(), called in [warning] [23:22:15] awight: hmm - I should look but yeah - won't be a major [23:22:39] ok, the wmf_civicrm migrations are done. [23:22:43] through wmf_civicrm_update_7063 [23:22:56] 7061 doesn't seem to have worked [23:23:22] eileen: no warnings. hmm [23:24:09] I think I can just run it [23:24:28] not good practice but very safe sql I think [23:26:19] I'm gonna stick my teeth in a food, will be semipresent for a few [23:27:13] food sounds like a good plan right now... [23:51:49] hmm, whitescreen on queue2civicrm config page [23:52:18] ooh [23:53:41] PHP Fatal error: Unsupported operand types in /srv/org.wikimedia.civicrm/drupal/modules/system/system.module on line 2134