[00:14:52] #1454: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1454 [00:14:52] #1446: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1446 [00:14:52] #1438: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1438 [00:14:52] #1432: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1432 [00:14:52] #1421: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1421 [00:15:51] #1440: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1440 [00:17:52] #1419: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1419 [00:17:52] #1437: (AG) TA:C|tmgia:a"s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1437 [00:20:52] #1401: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1401 [00:20:52] #1402: (AG) TA:C -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1402 [00:39:51] #1466: (AG) H:N|T:S Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1466 [01:08:51] #1467: (S) O:S|TS:B|P:MH|TT:CT|IU:N|TA:DI|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1467 [01:12:56] (PS2) Katie Horn: Edits to the wr1 parser so it can handle the oid refactor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119228 [01:14:19] (PS3) Katie Horn: Edits to the wr1 parser so it can handle the oid refactor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119228 [01:14:29] (PS1) Adamw: Fix the test for contribution tags during Thank You generation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119430 [01:16:27] (PS4) Katie Horn: Edits to the wr1 parser so it can handle the oid refactor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119228 [01:17:02] (CR) Katie Horn: "Derp." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119228 (owner: Katie Horn) [01:18:48] (PS1) Adamw: Fix the restarted recurring conditional [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119431 [01:18:51] #1468: (S) AT:S|O:S|TS:B|P:MH|TT:CT|IU:N|TA:DI|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1468 [01:19:51] #1467: (S) N:DWFDTDA -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1467 [01:29:51] #1469: (S) AT:S|O:S|TS:ID|P:NtH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1469 [01:34:52] #1470: (S) AT:S|O:S|TS:B|P:SH|TT:D|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1470 [01:40:25] (CR) Adamw: [C: 2 V: 2] "What the, I thought I..." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119228 (owner: Katie Horn) [01:41:51] #1472: (S) O:S|TS:B|P:NtH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1472 [01:44:52] #1472: (S) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1472 [01:45:14] (PS1) Adamw: fix typo [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119433 [01:47:51] #1473: (S) O:S|TS:B|P:NtH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1473 [01:54:51] #1474: (S) O:S|TS:B|P:NtH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1474 [01:55:51] #1475: (S) O:S|TS:B|P:NtH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1475 [06:45:02] (CR) Mwalker: [C: 2] Fix the test for contribution tags during Thank You generation [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119430 (owner: Adamw) [10:54:51] #1476: (PC) O:PC|T:TI Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [15:41:03] (CR) Pcoombe: [C: 2] "Looks good, thanks Adam!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119431 (owner: Adamw) [16:39:28] * K4-713 whistles in the dark [16:40:00] Hey the-wub: Are we running anything deliberate in RU at the moment? [16:44:06] K4-713: yeah, we have our standard low-level English campaign [16:44:15] But that's it? [16:44:47] I'm seeing some strange things in the error log. Not very many strange things, though. [16:45:32] the-wub: B14_0317_enWW_txt_hr look like a proper campaign to you? [16:45:46] Or, wait. No, that's a banner. Dur. [16:46:06] yeah, that's one that's up at the moment [16:46:07] We may be having problems with yandex. [16:47:00] Yeah, if that's a valid banner at the moment, it seems like yandex is having issues. [16:47:08] Do we have somebody who can test? [16:47:19] I never had yandex test account information. [16:47:30] (I think they told me it doesn't exist) [16:47:55] I don't know. ppena might [16:48:10] I can disable it for now if you like? [16:48:26] Yeah, that's probably the best thing for the moment. [16:48:35] I have enough in the error log to track down the issue, I think. [16:49:10] Do we have any ewallet types through globalcollect out there? I can search the error logs for those as well. [16:51:06] okay yandex is disabled [16:51:07] Sorry: I meant to say "other" ewallet types out there. [16:51:11] webmoney? [16:51:13] Coffee is still kicking in. [16:52:29] the-wub: Yep... there's a bunch of yandex, and, like... two webmoney in there with the same issue. [16:53:33] K4-713: want me to disable webmoney as well then? [16:53:45] the-wub: Yeah, we probably should also disable webmoney until I sort this out. [16:53:48] Thanks. [16:54:24] I'll, ah, tell everybody at standup in a minute, and send an email when we have fixed the issue (to the best of our knowledge). [16:55:01] Oh, wait. We have new account people at GC again. Remind me to have Pats ask them about test accounts with ewallet types. [16:55:06] okay K4-713 it's done. [16:55:14] thanks - if you can update people that would be great, I have to dash off now [16:55:16] I like asking this question every time I get a new audience. :) [16:55:22] later! [16:59:56] * rdwrer hi5s pizzzacat [17:07:45] rdwrer: what is this high five for?!! [17:07:51] Puns [17:07:54] In general [17:07:55] oh haha [17:15:51] !log updated crm from f8b2dab4689ae096898a21f9b3b57d505aa23f5e to 8c8f0de23baa800c2d3ad47056bdb011b1beb9d3 [17:15:56] K4-713: deployed. [17:15:56] Logged the message, Master [17:16:35] awight: woot. I'll kick off a wr1 job. [17:33:51] #1434: (AW) AT:AW|TS:ID -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1434 [17:41:51] #1262: (AW) *Deployed* -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1262 [17:42:05] !log tools updated from 87dbe60d2557a38553ae2bb602f777b4096915b8 to de586ae4ad03be52da1bb24177ea4bf7cdbd2c94 [17:42:10] Logged the message, Master [18:13:50] awight, pizzzacat: jfyi, I'm railing the queue consumer and reconciling the delayed globalcollect data. [18:14:05] So, if you get a bunch of alert emails that are like "OMG MELTING" that's probably me. [18:14:41] Kicked qc up to 600 messages per run, from 250. [18:15:38] woot. [18:15:43] * rdwrer scared that he understands these things [18:15:52] #1434: (AW) TS:PCR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1434 [18:15:52] #1434: (AW) TS:PD -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1434 [18:15:52] #1434: (AW) *Deployed* -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1434 [18:16:02] okie [18:16:03] K4-713: err, permission to work on my "duty cycle" improvement to the QC [18:16:19] awight: oooh. What did you have in mind? [18:16:28] K4-713: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [18:16:30] Just "run for x seconds and bail"? [18:16:37] oh sorry that is not clear [18:16:45] K4-713: yes, exactly what you said. [18:17:04] Okay. There's a decision in there that I'm not sure we made yet. [18:17:15] And that is: recurring vs one-time queues. [18:17:29] hmm. Right. [18:17:32] If one-time is full, recurring might not get any love until one-time empties. [18:17:42] That's kind of ok, actually... [18:17:46] Is it? [18:17:48] but only by coincidence [18:17:51] ha [18:17:51] #1436: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [18:18:00] yeah cos recurring is more offline-y [18:18:19] As far as I know, all the ones we do ourselves still come through one-time, if it's the first donation, yes. [18:18:25] * K4-713 thinks [18:18:27] ...okay. [18:18:31] We could round-robin on queues [18:18:34] eh. [18:18:57] I'm okay with one-time (perhaps we should start calling it "realtime"?) being a deliberate priority. [18:20:02] I want to make them separate jobs, ideally. I know the limitation is our fixed number of job runners, though. [18:20:10] Any idea how expensive those are? [18:21:45] How expensive job runners are? [18:21:49] I think it depends on the job. [18:22:30] K4-713: oh wow! Those are almost free. [18:22:53] I'm looking at the pstree, and the only process spawned is sh->drush [18:23:11] Let's just crank to 6 runners, and let each queue have its own job? [18:23:40] I think we tried that once, and it sort of... bogs down like crazy. [18:23:51] #1477: (S) AT:S|O:S|TS:B|P:NtH|TT:CT|IU:N|TA:BID|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1477 [18:24:08] Particularly if some jerk is constantly re-running audit scripts on a free runner... [18:24:10] <-- [18:24:17] looks like the qc uses only 23MB of physical memory [18:24:21] lol [18:24:32] It's not so much that, as the db-hammering, I think. [18:24:38] aaah hrm [18:25:00] I wonder if the transactions and moving long reads onto slave dbs help with that at all... [18:25:04] All the other jobs slow down when I run audit scripts. [18:25:08] You can see it in the trends. [18:25:18] where do I see that? [18:25:47] btw, the wr1 parser is a lot heavier than the qc, it's using about 500MB of memory [18:25:47] In jenkins, if you go to the job you're interested in, there's a "trend" link on the... history pane? [18:25:51] oh! [18:26:44] Of course, the trend situation right now is not indicative of anything, as I increased the message count pretty drastically. [18:27:04] Well, I found it but the qc trend pre-knob-cranking is not showing evidence that wr1 is slowing it down [18:27:56] Anyway. I guess this is secondary to what I'm trying to do. I'll look at the duty cycle fix, and the medium-load bug. [18:29:51] #1436: (AW) AT:AW|TS:ID -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [18:32:51] #1436: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [19:01:53] awight: Argh. Something tells me that something is still broken with the script. I don't think it's accurately determining if the record exists for the new style merchanterference. [19:02:09] d'oh. [19:02:33] K4-713: rats. unit tests! [19:02:46] * awight runs out of the building [19:02:53] Yeah... this one would be tricky to test anyway. [19:03:08] Units wouldn't have gotten it. [19:03:11] exactly why it needs a test :p [19:03:17] indeed. [19:03:31] * awight calls self black [19:04:49] hm. I am going to try something silly that involves pausing qc for a few minutes. [19:04:54] k [19:05:38] seeing if these really are dupes... [19:06:59] yep. Dupes. [19:07:12] okay, doing that for a bit. Probably a simple thing. [19:07:21] Or... maybe not. [19:09:49] ugh. [19:12:20] awight: Do we have a wmf_common function somewhere that grabs a civi contribution by the contribution_tracking id? [19:13:40] err... looking. [19:13:53] It seems like something that should exist. Somewhere. [19:13:58] :/ [19:15:39] K4-713: I don't think it exists yet. IMO it should live in wmf_civicrm/tracking.inc [19:15:50] Crazy. [19:15:51] Okay. [19:16:06] That's mildly astonishing, that we haven't needed to do that yet. [19:16:40] We only check for uniqueness using (gateway,gateway_txn_id) [19:16:58] guess so. [19:17:03] I mean, that is the only other option. [19:17:27] Is it possible to stick to that in the gc_wr1 code? [19:17:32] nope [19:17:34] ok [19:17:39] wr1 only has merchantref. [19:17:49] right [19:18:30] * awight sticks another piece of glass into my globe-shaped voodoo ball [19:18:44] This thing needs **such a rewrite**. [19:19:00] hehe. /me makes everyone nervous by moving towards that [19:19:13] I have lost track of the number of fundamental assumptions that have changed since it... congealed. [19:20:16] Whether or not to use the queue is an interesting one. I still think decoupling is a good answer there, cos it speeds things up so much. You should feel welcome to dump dupes into the queue as well, IMO. [19:21:05] Yeah... I'm still pretty sure that we should just dump things in the queue and let qc sort it all out in one place. [19:21:21] The problem with wr1, is that it's literally grepping through server logs. [19:21:25] ...for things that it needs. [19:21:29] It makes a lot of sense to do that, and in the distant future we could glean small bits of extra information from even dupes [19:21:40] sorry about the server logs. WTF [19:22:05] Yeah, so the idea there, is to cut down the grep work from "everything ever" to "just the missing stuff". [19:22:08] Can we send that information from the server in some sort of noop queue message? [19:22:16] not really. [19:22:25] And by that I mean "maybe". [19:22:33] remind me--cc_limbo? [19:22:54] ...should be killed with fire, and won't help us here because it assumes that everybody has order_id. [19:23:15] ok what I mean is, where are we getting this PII which you need to weld to the WR1? [19:23:27] server logs. [19:23:32] argh? [19:23:35] yes. [19:23:38] what is happening to write that? [19:23:44] this is DI, right? [19:23:53] Kind of? [19:24:00] * awight twitches a bit [19:24:22] But in all honesty, unless it's burning us down, I don't have time for that kind of thing until after WorldPay. [19:25:17] So, I'm going to add another band-aid to the band-aid ball, and bring this up with atgomez for wider prioritization. [19:25:30] I get it. But going forward, I like the idea of not... grepping logs... [19:25:36] I mean, if it were up to me, we wouldn't even be doing WP. [19:25:40] <_< [19:25:42] >_> [19:25:45] hehe. Nor GC. [19:25:58] Nah, that can stay until we freaking stabilize some other things. [19:26:10] Elliott mentioned he has done a WP integration... [19:26:12] Like... fundamental data movement through systems things. [19:26:28] awight: That's nice. Was it Silent Post? [19:26:31] donno [19:27:22] Okay, I need to disappear for a bit and think about how I'm going to spend all day not doing WP again. [19:27:29] lol [19:27:52] I'm not the only person who isn't surprised, am I? [19:28:06] I mean... I couldn't possibly be the only unsurprised... [19:28:06] I think you're not supposed to open the doors when there is a house fire. [19:28:20] BACKDRAFT. [19:28:22] oops [19:28:33] like, let's rush all the neighbors into this inferno [19:28:44] Oh, I was just thinking "Oxygen". [19:28:51] right [19:28:54] argh [19:29:11] * awight goes back to pretending this queue duty cycle thing is extremely important [19:29:24] har [19:39:38] (PS1) Adamw: Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [19:39:54] (PS2) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [19:40:35] K4-713: want a distraction? ^^ [19:40:51] #1436: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [19:40:51] #1436: (AW) TS:PCR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1436 [20:45:56] (CR) Adamw: "Woohoo!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/119337 (owner: Katie Horn) [22:03:51] #1260: (AG) N:SPME(i -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1260 [22:05:51] #676: (AG) Last year but not this year... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/676 [22:06:39] mwalker|away: for your amusement, mediawiki-core/includes/db/DatabaseSqlite.php -> replaceVars() [22:06:51] #676: (AG) Request from Major Gifts -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/676 [22:15:51] #1476: (AG) T:TT -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [22:15:52] #1476: (AG) P:NtH -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [22:15:52] #1476: (AG) TT:CT -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [22:15:52] #1476: (AG) TA:BID -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [22:15:52] #1476: (AG) BG:#Maef|BC:#Iit -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1476 [22:19:51] #1033: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1033 [22:19:52] #1035: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1035 [22:19:52] #1139: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1139 [22:19:52] #1140: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1140 [22:19:52] #1141: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1141 [22:19:52] #1142: (AG) H:Y -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1142 [22:22:51] #1439: (AG) ->Sprint 45 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1439 [22:23:52] #1445: (AG) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1445 [22:23:52] #1445: (AG) Anne to follow up with Caitlin about process on... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1445 [22:25:51] #1360: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1360 [22:26:52] #1265: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1265 [22:27:51] #1272: (AG) This is probable deployed? -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1272 [22:27:52] #1272: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1272 [22:28:51] #1358: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1358 [22:30:51] #1376: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1376 [22:31:51] #1454: (AG) P:SH -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1454 [22:32:52] #1389: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1389 [22:32:52] #1390: (AG) P:MH -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1390 [22:33:51] #1391: (AG) P:SH -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1391 [22:33:52] #1391: (AG) O:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1391 [22:34:52] #1406: (AG) ->Sprint 45 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1406 [23:34:51] #1361: (AG) AT:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1361 [23:38:51] #945: (AG) ->Sprint 45 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/945