[00:01:43] awight: 406e48fff4447c799303c4b067a2cce72f87fb73 details some changes [00:02:18] removes a bunch of stuff, but we might not have to [00:04:01] one more thing that would be annoying about 1.28ing [00:04:06] remaking this branch every time [00:04:50] yeah I like that patch. [00:05:04] could just cherry pick that every time [00:05:08] But I don't think we necessarily have to remake it every time, cos master is actually linear [00:05:24] we would just merge the new tag in, theoretically [00:05:47] ah, just name it fr-vendor or something? [00:05:49] I'm more worried about the extensions submodules, though... [00:06:03] ooh this is vendor. [00:06:18] jesus I accidentally used Differential and couldn't make any sense of the UI [00:06:36] the dystopian future [00:07:20] awight: regarding strategy, maybe there will be a way to have both queues on even if they don't do exactly the same thing, and then compare results? Might be complicated, also maybe not worth it, I guess... [00:07:38] cwd: huh. yeah I hope and believe we can use vanilla mw-core/vendor [00:07:50] AndyRussG: comparing results would be especially difficult [00:07:55] cos it would require forking the consumers [00:08:01] hmmm [00:08:10] Interesting idea, though... maybe we could pull the data into the staging environment or something? [00:08:21] So what would be the point of turning both queues on simultaneously? to allow rollback? [00:08:31] yes [00:08:37] hmmm [00:09:02] why would it require forking the consumers? we could just have an old version of them on one box, and a new version on another? [00:09:10] but it might rather involve forking the db.... [00:09:42] the old version will still require maintenance though, which seems like a fork--totally something we can do if it makes sense though [00:10:17] I like this direction though, really interesting idea [00:12:16] Fundraising Sprint Killing Time, Fundraising Sprint Muggle Baiting, Fundraising-Backlog, FR-Adyen, and 3 others: IL donation form: CVV field only allows three digits, blocks AmEx donors - https://phabricator.wikimedia.org/T137811#2467602 (Dzahn) p:Unbreak!>Triage [00:12:21] Maybe we could dump a bunch of real data from the new queue and use it to test consumers in a staging area? [00:12:31] database errors loading payments [00:12:36] on vagrant [00:13:07] this hole is too deep for friday past 6pm [00:13:23] crazy. I'll rm my vagrant in solidarity [00:13:31] don't do it! [00:13:33] see you next week! [00:14:12] > VirtualBox is complaining that the kernel module is not loaded. [00:14:28] bewildering [00:14:29] sudo /sbin/rcvboxdrv setup [00:14:33] Bad argument setup [00:14:40] just merged the rtbt thing [00:14:49] sorry for taking so fucking long [00:14:54] ttyl! [00:15:00] nice! [00:15:14] no worries, it wasn't going anywhere fast [00:16:19] me too. see ya! [00:42:05] Fundraising-Backlog, fundraising-tech-ops, MediaWiki-extensions-DonationInterface, Epic, Performance: [Epic] Payments performance tuning - https://phabricator.wikimedia.org/T140311#2467614 (Danny_B) [08:23:35] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, Unplanned-Sprint-Work: Donations message with empty body! - https://phabricator.wikimedia.org/T140515#2468099 (awight) I've been using this query to check Ingenico volume: ``` sel... [08:26:23] Fundraising-Backlog, Phabricator: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2468102 (awight) [10:30:25] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, Unplanned-Sprint-Work: Donations message with empty body! - https://phabricator.wikimedia.org/T140515#2468238 (awight) p:High>Unbreak! This patch could use some review...... [11:04:47] Fundraising-Backlog, Phabricator: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2468245 (Aklapper) Please elaborate what it's for and see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Proposal_format for required data. [11:41:57] Fundraising-Backlog, Project-Admins: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2468255 (Danny_B) [11:42:18] Fundraising-Backlog, Project-Admins: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2468102 (Danny_B) p:Triage>Low [12:45:17] Fundraising-Backlog, Project-Admins: Please create a #signal-to-noise tag - https://phabricator.wikimedia.org/T140531#2468102 (Luke081515) Maybe kind of a subtask of T31079? :) [17:10:28] (PS3) Awight: Log completed donations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) [17:10:30] (PS1) Awight: Remove a Worldpay test [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299287 [17:10:32] (PS1) Awight: Wrap test global manipulations in setMwGlobals [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299288 [17:12:56] (CR) jenkins-bot: [V: -1] Log completed donations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [17:18:28] (CR) Cdentinger: [C: 2] Remove a Worldpay test [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299287 (owner: Awight) [17:21:39] (Merged) jenkins-bot: Remove a Worldpay test [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299287 (owner: Awight) [17:22:30] (CR) Cdentinger: [C: 2] "I sort of understand, just gonna take your word for it." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299288 (owner: Awight) [17:25:36] (Merged) jenkins-bot: Wrap test global manipulations in setMwGlobals [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299288 (owner: Awight) [17:27:01] (CR) Cdentinger: "Couldn't find one of the test response files, but only under 5.3" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [17:31:38] (CR) Awight: "I hate myself for writing tests at all..." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [17:33:18] (CR) Awight: "Okay I won't lie, I saw that error on the console, coming from some other test, but it didn't harm phpunit." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [17:42:21] (PS4) Awight: Log completed donations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) [17:46:34] cwd|afk: ^ at your leisure [17:50:07] (PS1) Awight: Fix typo masking gateway validation tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299289 [17:52:48] cwd|afk: You might be interested, the 1.27 tests sometimes take exactly the same amount of time ^ [17:53:52] Meanwhile, on the other patchset it took 3x longer. [17:54:33] The phpunit tests themselves took exactly the same time, c. 30 sec [17:55:36] k: a minute and a half to clone mw-core. That makes me cry. Not the fault of our code, either way. [17:58:03] hmm isn't that interesting [17:58:21] some thing that we do performs badly under 1.27 [17:59:34] (CR) Cdentinger: [C: 2] Log completed donations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [18:03:29] (Merged) jenkins-bot: Log completed donations [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299280 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [18:10:17] cwd|afk: ty x2 [18:10:20] deploying... [18:15:58] (PS1) Awight: Log completed donations [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299290 (https://phabricator.wikimedia.org/T140515) [18:16:10] (CR) Awight: [C: 2] Log completed donations [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299290 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [18:17:44] (Merged) jenkins-bot: Log completed donations [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299290 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [18:20:42] (PS1) Awight: update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299291 [18:20:54] (CR) Awight: [C: 2] update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299291 (owner: Awight) [18:21:31] np! assume these were cherry picked around the mustache stuff? [18:22:42] very much so, thanks for checking! [18:23:11] This is all a timely reminder of how much weekend instability sucks... [18:24:51] yeah indeed. do you know what the reason is for the blank messages? or is that what you are looking for? [18:26:15] It's terrifying and mysterious [18:26:22] (Merged) jenkins-bot: update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299291 (owner: Awight) [18:26:25] My only guess so far is the internals of getStompTransaction [18:27:07] so we are accidentally dropping the body on our end somewhere? [18:27:19] but only in a specific situation [18:27:44] !log update paymentswiki from 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c to 25c97ba0f27b61859f90fd205c53d587c2838fec [18:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:28:08] yep. empty body, maximum creep factor [18:28:23] sucks [18:29:16] gargh. rsync_blaster is copying the whole repo. scary [18:29:31] too much green? [18:30:45] hehe exacto [18:31:02] !log enable LogCompleted for Ingenico [18:31:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:31:13] did permissions change or something? [18:31:33] on activemq? [18:31:48] It's only one gateway, which just increases creep [18:32:19] i mean why did rsync copy more stuff than you expected? [18:32:53] oh, I was imagining it was a new box maybe. Was > 10k lines of backscroll so my buffer didn't catch the hostname [18:33:09] cool. logging fired without taking the site down. [18:33:15] \o/ [18:52:18] Of course, now the bug is afraid to rear its beady eyes [19:12:25] AHA [19:14:17] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, and 2 others: Donations message with empty body! - https://phabricator.wikimedia.org/T140515#2468490 (awight) Well. The debug logging just captured: > Jul 16 18:55:37 payments1002 gl... [19:21:39] (PS1) Awight: Logging to debug my logging [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299295 [19:21:47] cwd|afk: The sad. [19:25:18] awight|mope: it goes deeper? [19:25:33] turtle shells with nothing inside. [19:25:42] json_encode is failing. [19:25:57] something in the request break the parser? [19:26:10] * awight|mope briefly ponders 2nd brewski [19:26:29] That patch above will at least give us the errno [19:27:55] The sick part is that I'm logging more of the transaction data than usual, so this would be sort of expected--but I'm sure getStompTransaction is having the same fail, with data in the regular normalized message. [19:28:24] I'm tempted to blame php5.5 features, we'll see... [19:29:48] FWIW, this can wait. It's more morbid fascination than obligation of fiducial duty that's motivating me at this point. [19:30:07] It's juuust borderline enough volume that I wouldn't mind capping the oil well though [19:31:40] oh hold on with that CR... [19:38:26] it's good that they will get resolved through the audit anyway [19:39:16] weird that they are so few and far between, can you think of a reason for that? [19:41:48] naw it will take a little backfill prance, by scraping gc comm logs for I_OWP contact details [19:41:57] Hopefully that can be scripted. [19:43:14] ah bummer [19:43:58] don't know why the hhvm test fails and not the others [19:44:13] looks like last_error is implemented: https://github.com/facebook/hhvm/blob/master/hphp/runtime/ext/json/ext_json.php [19:44:37] awight|mope: i have to take off for awhile but happy to merge something in the next few minutes [19:46:14] (PS1) Awight: [WIP] Log json_encode errors [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299297 (https://phabricator.wikimedia.org/T140515) [19:46:46] that's not mergeable, holdon [19:48:02] (CR) jenkins-bot: [V: -1] [WIP] Log json_encode errors [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299297 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [19:49:17] (PS2) Awight: Log json_encode errors [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299297 (https://phabricator.wikimedia.org/T140515) [19:49:19] (PS1) Awight: [WIP] Test for logCompleted [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299298 [19:50:37] that's totally cheating. [19:52:49] (CR) jenkins-bot: [V: -1] [WIP] Test for logCompleted [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299298 (owner: Awight) [19:53:23] ergo, this patch is dastardly ready to go: https://gerrit.wikimedia.org/r/#/c/299297/ [19:56:59] (CR) Cdentinger: [C: 2] Fix typo masking gateway validation tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299289 (owner: Awight) [19:58:26] (Merged) jenkins-bot: Fix typo masking gateway validation tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299289 (owner: Awight) [20:00:04] Fundraising Sprint Muggle Baiting, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review, and 2 others: Donations message with empty body! - https://phabricator.wikimedia.org/T140515#2468533 (awight) @MBeat33 I'm realizing that we can detect this error while the donor is... [20:00:39] awight|mope: interesting that json encode failure is not a fatal error [20:01:20] this is why non-deterministic software happens [20:02:09] (CR) Cdentinger: [C: 2] Log json_encode errors [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299297 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [20:02:17] ok i gotta bounce, will check in later! [20:03:39] (Merged) jenkins-bot: Log json_encode errors [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299297 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [20:05:42] ttyl! [20:14:30] (Abandoned) Awight: Logging to debug my logging [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299295 (owner: Awight) [20:19:58] (PS1) Awight: Log json_encode errors [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299302 (https://phabricator.wikimedia.org/T140515) [20:20:19] (CR) Awight: [C: 2] Log json_encode errors [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299302 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [20:21:19] (PS1) Awight: update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299303 [20:21:34] (CR) Awight: [C: 2] update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299303 (owner: Awight) [20:22:02] (Merged) jenkins-bot: Log json_encode errors [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/299302 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [20:28:01] (Merged) jenkins-bot: update DonationInterface submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/299303 (owner: Awight) [20:32:42] !log update paymentswiki from 25c97ba0f27b61859f90fd205c53d587c2838fec to 0c14940f4930e94a9287acae978cc6e661e54ee1 [20:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:46:36] way: 37123219:9800384548 Completed donation: {"json_encode_error":"5: Malformed UTF-8 characters, possibly incorrectly encoded"} [20:46:46] There ye are [20:47:48] Okay, well I can sleep tight at that one. [22:42:15] (PS1) Awight: [WIP] Reproduce Ingenico name encoding bug [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299350 (https://phabricator.wikimedia.org/T140515) [22:43:59] (CR) jenkins-bot: [V: -1] [WIP] Reproduce Ingenico name encoding bug [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299350 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [23:35:58] (PS1) Awight: Force bad encodings to approximate UTF-8 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299352 (https://phabricator.wikimedia.org/T140515) [23:36:08] (Abandoned) Awight: [WIP] Reproduce Ingenico name encoding bug [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299350 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [23:37:36] (CR) jenkins-bot: [V: -1] Force bad encodings to approximate UTF-8 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299352 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [23:49:25] (PS2) Awight: Force bad encodings to approximate UTF-8 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299352 (https://phabricator.wikimedia.org/T140515) [23:51:08] (CR) jenkins-bot: [V: -1] Force bad encodings to approximate UTF-8 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299352 (https://phabricator.wikimedia.org/T140515) (owner: Awight) [23:52:34] (PS3) Awight: Force bad encodings to approximate UTF-8 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/299352 (https://phabricator.wikimedia.org/T140515)