[11:11:57] Fundraising Sprint Homebrew Hadron Collider, Fundraising-Backlog, FR-Paypal, Recurring-Donations, Patch-For-Review: Paypal recurring donations missing contribution_tracking link since 2017-04-10 - https://phabricator.wikimedia.org/T163443#3205468 (Pcoombe) Thanks @Ejegg. If we can backfill th... [14:39:11] Wikimedia-Fundraising-Banners: Remove IE9 specific code from banners once JS is disabled for it - https://phabricator.wikimedia.org/T160945#3206184 (Pcoombe) Open>Resolved [16:07:56] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3206804 (Nuria) Ping @LilyOfTheWest please let me know if you can provide more info. We can certainly compile a dataset of banners/c... [16:18:35] fundraising-tech-ops: replace lutetium with new hardware running debian/jessie - https://phabricator.wikimedia.org/T145110#3206861 (Jgreen) Open>Resolved lutetium was shut down this AM [16:18:40] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3206863 (Jgreen) [16:19:47] fundraising-tech-ops, Operations, ops-ulsfo, Patch-For-Review: rack/setup frbackup2001 - https://phabricator.wikimedia.org/T162469#3206869 (Jgreen) Open>Resolved this is done! [16:19:56] fundraising-tech-ops: replace backup4001 with a new machine at codfw named frbackup2001 - https://phabricator.wikimedia.org/T161903#3206874 (Jgreen) Open>Resolved a:Jgreen This is done! [16:20:40] Fundraising Sprint Far Beer, fundraising-tech-ops, Epic: EPIC: build fundraising civicrm (barium) replacement server on Debian Jessie, with HHVM or PHP5.5 - https://phabricator.wikimedia.org/T136959#3206885 (Jgreen) [17:00:19] fr-tech: As I was passing Project MAC, [17:00:19] I met a Quux with seven hacks. [17:00:19] Every hack had seven bugs; [17:00:19] Every bug had seven manifestations; [17:00:19] Every manifestation had seven symptoms. [17:00:19] Symptoms, manifestations, bugs, and hacks, [17:00:19] How many losses at Project MAC? [17:00:20] -- discuss. [17:01:06] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3206992 (Nuria) >Pitching in from WMDE Fundraising. Our tracking requirements are two-fold: @gabriel-wmde :Let's please not mix use... [17:30:29] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042#3207100 (DStrine) @Nuria I don't think WMDE has access to pivot or any LDAP related systems. That may be very complicated to setup. [17:33:08] good morning fr-tech! [17:49:57] wassup ejegg [17:50:40] oh, not a lot just yet [17:50:52] how's things with you? [17:51:36] oh fine [17:51:47] just been getting the civi staging db restored [17:52:07] there is an annoying mysql bug that has followed us across versions [17:52:32] is that what's been crashing the restore in the middle of the c_t table? [17:54:07] yep [17:54:16] mysqldump outputs a syntax error [17:54:27] have to trim it out of the 45G file [17:54:35] oh man, that's no fun to edit [17:54:57] is it because of some crazy data in that table we should delete anyway? [17:55:12] nah it's just the triggers [17:55:15] https://bugs.mysql.com/bug.php?id=57516 [17:55:24] notabug wontfix [17:55:45] * Jeff_Green wonders if there's an option to just use a different delimiter for the triggers that would work around this? [17:56:23] it's gotta be something like that [17:57:52] i'm thinking Notabug Wontfix would be a pretty good name for a fictional, british programmer [17:58:01] hah [17:58:18] with a detective hat [18:02:10] ahh, so it changes delimiter fine, it's just using the current delimiter breaks things? [18:03:01] i have not tried changing it [18:04:19] ejegg something about the syntax around specifying that delimiter trips up mysqldump vs mysql client [18:05:05] but, yeah there's no problem with the delimiter when loaded the way we normally add triggers [18:05:44] hmm [18:06:07] we've just been working around it by excluding triggers from the restores, and then manually adding them later [18:06:14] (when we remember they exist) [18:06:28] man, that's a pain [18:06:58] i don't understand why it's wontfix, seems like an obvious bug [18:08:17] hmm, when I try dumping, I get DELIMITER ;; before the trigger and DELIMITER ; [18:08:20] after [18:09:35] yeah, i think the gist is that, since ; normally terminates a SQL statement, having it as a delimiter makes it look to the parser like two statements, one of which is empty [18:09:59] like there should be an escape character before the delimiter there [18:10:24] this thread seems to say that there is no such thing as a bad delimiter, just depends what the client expects [18:10:59] right, so mysqldump should quote or escape the delimiter and all would be dandy [18:11:19] trying locally [18:16:27] i don't see an option for specifying the delimiter? [18:22:06] weird, seems to have restored the triggers just fine for me [18:22:33] stupid expected behavior [18:22:44] it's so weird that we just upgraded the server and the client and this still happens [18:24:03] How are you running the restore? [18:24:14] cat file | mysql [18:24:16] I just did mysql civirestore < civicrm.sql [18:24:30] so effectively the exact same [18:24:36] sure seems like it... [18:25:36] Jeff_Green: did the dev_civicrm one somehow work?? or is that still old data? [18:25:57] i didn't restore triggers [18:26:10] figured we'd do all that as part of getting civi up to the right version and whatnot [18:27:21] ah, but you ran the strip delimiters script? [18:27:26] yeah [18:32:28] (CR) Ejegg: [C: 2] Fix civibuild options for updated amp [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349357 (owner: Ejegg) [18:36:53] (Merged) jenkins-bot: Fix civibuild options for updated amp [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349357 (owner: Ejegg) [18:38:23] (PS4) Ejegg: Always update c_t table for initial recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349254 (https://phabricator.wikimedia.org/T163443) [18:38:25] (PS3) Ejegg: Backfill missing contribution_tracking links [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349495 (https://phabricator.wikimedia.org/T163443) [18:38:47] XenoRyet: those two ought to pass V+1 now. Mind taking a look? [18:39:06] Yea, give me one minute and I'll check 'em out. [18:39:12] thanks! [18:50:16] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349254 (https://phabricator.wikimedia.org/T163443) (owner: Ejegg) [18:51:39] Seddon, the FDC banner to comment on plans is showing one sentence in English [18:56:35] well i can log into dev civi now [18:56:41] triggers still aren't there and skin is messed up [18:57:25] ejegg: we were thinking about doing a process control release [18:57:47] does that sound like a fun way tospend the afternoon? [18:59:34] (CR) XenoRyet: [C: 2] Always update c_t table for initial recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349254 (https://phabricator.wikimedia.org/T163443) (owner: Ejegg) [18:59:45] cwd sure! [19:00:04] I am doing another couple hours at the homework help thing, starting in 3 hrs [19:00:14] cool [19:00:19] i'm guessing it won't take that long [19:00:23] right on [19:00:29] the only thing is we don't have a clean rollback procedure [19:00:33] easier to roll forward [19:00:42] but that means building the deb which is time consuming [19:00:52] however i could script it, i was thinking about wasting somet ime on that [19:01:52] in fact it might be worth doing that first, unless the release is urgent [19:02:01] just so we have a quick way to bail out if necessary [19:04:54] (Merged) jenkins-bot: Always update c_t table for initial recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349254 (https://phabricator.wikimedia.org/T163443) (owner: Ejegg) [19:06:39] I'm looking at archiving p-c logs again, and I'm not sure what to do about current activity [19:07:51] Jeff_Green: current activity? [19:08:06] i don't know of an obvious way to determine whether p-c is actively writing to a file [19:08:15] We're only going to archive stuff older than a couple days, right? [19:08:20] well [19:08:33] nothing is going to get backed up until it enters the archive pipeline [19:08:43] ah [19:08:49] because we can't really rsync files that may or may not be actively being written to [19:08:55] this was the same problem with jenkins [19:09:02] how long is the longest job? [19:09:24] but jenkins adjusted a symlink after it moved past a particular job build, so I could look for where that symlink was pointing [19:09:32] it must be the silverpop export right? [19:09:52] cwd yeah, or sometimes the gc audit parser takes stupid long [19:10:13] does seem like a bad idea to wait several hours for them to get backed up [19:11:09] this is just logs about the job, not the job itself [19:11:31] yeah but they seem important to debugging no? [19:11:40] api stuff [19:12:20] I dunno [19:12:31] (CR) XenoRyet: [C: 2] Backfill missing contribution_tracking links [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349495 (https://phabricator.wikimedia.org/T163443) (owner: Ejegg) [19:12:37] seems like if they're important they should be central logged in the first place [19:13:01] yeah indeed [19:13:04] that's probably the right fix [19:13:52] backups are daily [19:14:35] syslog is automatically duplicated on-host and off-host, in roughly realtime [19:15:52] what we did with jenkins was compress completed jobs to a local archive dir (hourly?), and copy that archive dir offhost daily [19:17:01] i was just going to reuse that logic here, if I can figure out how to determine "job is done" [19:17:33] is there worry about this being too much data for syslog? [19:17:33] (Merged) jenkins-bot: Backfill missing contribution_tracking links [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349495 (https://phabricator.wikimedia.org/T163443) (owner: Ejegg) [19:18:22] I think it's probably OK, but it's definitely a large amount of data [19:18:50] it's kind of duplicative, since a lot of the jobs log to syslog too [19:19:14] maybe we could cut down on the duplicated stuff? [19:19:16] yeah we would need to address that [19:19:52] Hmm, sometimes you /want/ console + syslog output though [19:20:04] that's when you use a -v flag :-) [19:20:39] heh, we actually use two of 'em to invoke drush [19:20:52] -vv ? [19:21:44] yep [19:22:53] --debug -vvvv++ --loglevel=∞ [19:25:35] i think the best I can do for now is ignore stuff less than a day old, and archive the rest [19:28:13] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349720 (owner: Ejegg) [19:31:00] if that's easy it's probably good enough for the short term [19:31:05] better than no archiving at all :P [19:31:17] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/347796 (owner: Eileen) [19:31:47] i just found a horrible hack that will sort of work, we seem to write a consistent last line at the end of job [19:32:17] so I'll look for that, and we'll archive only runs that are >1d or have that completed last line [19:33:38] that seems pretty good [19:36:04] (PS1) Ejegg: Use : after host in mysql dsn [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350012 [19:40:51] (PS25) AndyRussG: [WIP] Custom campaign mixin param handlers [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/343953 (https://phabricator.wikimedia.org/T144453) [19:41:14] (PS18) AndyRussG: [WIP] Banner sequence campaign mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/344988 (https://phabricator.wikimedia.org/T144453) [19:50:14] PROBLEM - check_procs on civi1001 is CRITICAL: PROCS CRITICAL: 39139 processes [19:55:14] RECOVERY - check_procs on civi1001 is OK: PROCS OK: 266 processes [20:12:57] (CR) Ejegg: [C: 2] Use : after host in mysql dsn [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350012 (owner: Ejegg) [20:13:31] (CR) jerkins-bot: [V: -1] Use : after host in mysql dsn [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350012 (owner: Ejegg) [20:21:09] (CR) Ejegg: [C: 2] Use : after host in mysql dsn [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350012 (owner: Ejegg) [20:26:36] (Merged) jenkins-bot: Use : after host in mysql dsn [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350012 (owner: Ejegg) [20:26:54] (PS3) Ejegg: Add de Bijenkorf card [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349720 [20:27:16] fr-tech can anybody review ^^^ and https://gerrit.wikimedia.org/r/349721 ? [20:27:31] Trivial patches, but needed to get the adyen audit to stop failing [20:28:09] I'd like to deploy them with the recurring c_t fix if possible [20:32:23] Heh, there's today's audit failmail, right on time [20:40:13] (CR) Ejegg: [C: 2] Use defaults that will play nicer with the .deb [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/348132 (owner: Awight) [20:41:11] (Merged) jenkins-bot: Use defaults that will play nicer with the .deb [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/348132 (owner: Awight) [21:14:33] (CR) XenoRyet: [C: 2] Add de Bijenkorf card [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/349721 (owner: Ejegg) [21:15:04] ty XenoRyet [21:15:26] Spacing looks weird on that case statement you added to the other one, but otherwise looks fine. [21:15:39] Oh, lemme fix that [21:15:56] (Merged) jenkins-bot: Add de Bijenkorf card [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/349721 (owner: Ejegg) [21:17:23] (PS4) Ejegg: Add de Bijenkorf card [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349720 [21:18:13] ok, spacing fixed [21:18:36] (CR) XenoRyet: [C: 2] Add de Bijenkorf card [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349720 (owner: Ejegg) [21:19:03] There ya go [21:19:57] rockin. Lemme get that deployed, then how bout we do a call to figure out a few things about paypal and the 600-status cc donations? [21:20:50] (PS1) Ejegg: Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350083 [21:22:41] (Merged) jenkins-bot: Add de Bijenkorf card [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/349720 (owner: Ejegg) [21:22:50] (CR) jerkins-bot: [V: -1] Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350083 (owner: Ejegg) [21:25:18] ejegg: I've got an OB appointment at 3:00, but I could probably do a call when I get back from that. [21:25:38] k, great! [21:27:46] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350083 (owner: Ejegg) [21:27:52] (CR) Ejegg: [C: 2] Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350083 (owner: Ejegg) [21:29:34] (PS1) Ejegg: Update SmashPig library [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/350084 [21:30:07] (CR) Ejegg: [V: 2 C: 2] Update SmashPig library [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/350084 (owner: Ejegg) [21:32:26] (Merged) jenkins-bot: Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350083 (owner: Ejegg) [21:32:47] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350086 [21:32:51] (CR) jerkins-bot: [V: -1] Update SmashPig library [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/350084 (owner: Ejegg) [21:33:13] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350086 (owner: Ejegg) [21:33:51] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350086 (owner: Ejegg) [21:41:36] !log updated civicrm from 51dbbad9f7822a7b3df730a2bd92ee3fb176b3ec to a11c108961cf49afea7d446f83332d0e9ce3eb0c [21:41:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:56:33] (PS1) Ejegg: Remove 'Module missing message fixer' module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350091 [22:56:59] (CR) Ejegg: [C: 2] Remove 'Module missing message fixer' module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350091 (owner: Ejegg) [22:57:49] (CR) jerkins-bot: [V: -1] Remove 'Module missing message fixer' module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350091 (owner: Ejegg) [22:59:49] (CR) Ejegg: [C: 2] "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350091 (owner: Ejegg) [23:03:42] (Merged) jenkins-bot: Remove 'Module missing message fixer' module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350091 (owner: Ejegg) [23:06:40] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350092 [23:06:43] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350092 (owner: Ejegg) [23:06:49] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350092 (owner: Ejegg) [23:08:56] !log updated civicrm from a11c108961cf49afea7d446f83332d0e9ce3eb0c to 40d88c055bbfc2a46097c50c1250829aecd8d789 [23:09:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:10:55] (PS1) Ejegg: init Civi before backfill [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350093 [23:11:01] wb XenoRyet [23:11:17] I forgot a line in that civi upgrade ^^^ [23:11:25] looking [23:11:52] also, fyi, civi CI is kinda precarious right now [23:11:58] (PS5) Eileen: 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 [23:12:06] Yea? [23:12:24] (CR) XenoRyet: [C: 2] init Civi before backfill [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350093 (owner: Ejegg) [23:12:30] eileen1: civi CI is kinda janky right now [23:12:46] It's pointing to upstream civicrm-buildkit and amp [23:13:15] but upstream amp is still missing the pre-created db feature [23:13:31] I thought I might get this merged over the weekend: https://github.com/amp-cli/amp/pull/46 [23:13:52] but for right now, I just copied a homebuilt phar onto the CI servers [23:14:15] unfortunately, they get recycled every now and then, and will re-download the upstream amp after that [23:14:33] so, if I can't get that PR merged, I'll have to revert it all to point back to the local fork [23:16:16] (CR) jerkins-bot: [V: -1] 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 (owner: Eileen) [23:17:18] (Merged) jenkins-bot: init Civi before backfill [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/350093 (owner: Ejegg) [23:18:28] (PS6) Eileen: 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 [23:21:00] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350094 [23:21:05] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350094 (owner: Ejegg) [23:21:12] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/350094 (owner: Ejegg) [23:22:03] (PS7) Eileen: 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 [23:22:07] (CR) jerkins-bot: [V: -1] 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 (owner: Eileen) [23:22:38] !log updated civicrm from 40d88c055bbfc2a46097c50c1250829aecd8d789 to 061cd61cb0e475b607a53e6183331ff7abe691c3 [23:22:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:25:21] whew, finally got that backfill running [23:25:33] (CR) jerkins-bot: [V: -1] 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 (owner: Eileen) [23:25:49] (PS8) Eileen: 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 [23:28:33] XenoRyet: want to join me in the -talk hangout? [23:28:54] Sure, one sec. [23:29:07] (CR) jerkins-bot: [V: -1] 4.7.19 upgrade (WIP) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/348659 (owner: Eileen) [23:29:08] Only got about half an hour, but hopefully we can figure something out in that time. [23:32:18] Fundraising Sprint Homebrew Hadron Collider, Fundraising-Backlog, FR-PayPal-ExpressCheckout, Patch-For-Review, Unplanned-Sprint-Work: Paypal audit parser still not working for new recurring express checkout - https://phabricator.wikimedia.org/T162975#3208337 (Ejegg) Open>Resolved [23:33:02] Fundraising-Backlog, FR-PayPal-ExpressCheckout, Epic: Epic: Distinguish PayPal legacy vs. Express Checkout transactions - https://phabricator.wikimedia.org/T138013#3208340 (Ejegg) [23:33:04] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, Epic: Paypal Express checkout campaign ready - https://phabricator.wikimedia.org/T131817#3208341 (Ejegg) [23:33:06] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, FR-WMF-Audit, Patch-For-Review: Update audit parser for PayPal Express Checkout - https://phabricator.wikimedia.org/T134605#3208339 (Ejegg) Open>Resolved