[18:58:20] !log updated fraud rules on payments [18:58:39] Logged the message, Master [19:20:41] K4-713: how about, we use the local drush repo but I checkout the exact release we have been using? and make the timestamp patch at that branch? [19:21:10] I have some serious reservations about changing drush in the next... two or so days. [19:21:24] Or, really any days in which we make more than 1M. [19:21:46] But, please feel free to get it ready for our first 800k day. [19:21:52] okok [19:21:56] ...whenever that ends up being. [19:22:10] I don't think we'll get any resolution on the transaction lock issue without this instrumentation, tho [19:22:48] The other approach is to ask Jeff_Green to write a monitoring process which dumps all open transactions every... 20 seconds or so. [19:23:12] And, high-traffic is exactly the scenario we need in order to debug. [19:23:22] uh oh [19:23:27] :p [19:23:32] yeah it's an issue. [19:23:46] especially since, I cannot enable transactions on the queue consumer until we figure this out. [19:24:00] so we're slowly inserting corrupted data until that time [19:24:14] how does the drush timestamp question relate to the open transactions? [19:24:27] I needa figure out where the slow transaction is coming from. [19:24:47] would it help to make the stupid shell script log? because that's trivial [19:24:54] the wrapper I mean [19:25:25] or does drush not take all of its input by argument? [19:25:45] nah the resolution is much finer than script start/stop [19:25:57] We need timestamps on each line of drush output [19:25:58] oic [19:26:21] ok well we can properly stage things now, how about we start by getting it all working in staging? [19:26:24] anything else running on the -write db needs to be logging as well [19:26:34] shore [19:27:11] we could enable write-db query logging for a bit but that probably can't stay on long term [19:27:18] that can be toggled on the fly [19:27:29] That... might cause K4 grey hairs more than even twiddling drush would [19:27:35] ha [19:27:56] i think we'd want to supervise it while its on [19:28:06] Jeff_Green: what about passively monitoring long transactions? [19:28:21] basically, tweak the long-query-killer to log 15sec+ open transactions [19:28:32] sure [19:28:32] I think we're set to timeout at 50sec, fwiw [19:28:39] really? that would be peachy [19:28:59] well, maybe sure? [19:29:03] lol [19:29:28] by transaction do you mean essentially query? [19:29:43] transaction, methink. [19:30:01] But, the issue is something holding open a row lock [19:30:25] I read somewhere that innodb makes every statement inside a transaction... [19:31:36] show engine innodb status; seems to give us some of the infos i want, under "TRANSACTIONS" [19:31:54] however those are query ids and we want the query itself [19:31:58] so... [19:32:09] I will not kibbitz [19:32:12] atm I'm failing to find the query slayer [19:32:16] where the hell does that run? [19:32:18] hrm [19:33:15] oh there it is. stupid_query_slayer on db1008 [19:33:27] well there's the script anyway [19:34:10] the only good thing about this script are the comments at the top [19:34:28] # awjr: there was a script on.. db9 maybe that ran as root [19:34:28] # awjr: periodically [19:34:28] # awjr: looking for long running queries and killing them [19:34:28] # Jeff_Green: lofl [19:34:28] # ***Jeff_Green cries lot [19:34:28] # awjr: i want to murder civicrm in the face [19:34:49] BAHAHA [19:34:52] BLAWRharhar [19:34:54] Jeff_Green: ah yeah, the watchdog [19:34:55] * K4-713 wipes tear from eye [19:35:04] Ah, the good old days. [19:35:06] i remember that fondly [19:35:13] :-) [19:35:26] i remember weighing the pros/cons of fixing queries in civicrm vs writing that script [19:35:30] well this thing h'aint been running for a long time [19:36:14] how about I fire it up in a non-slaying mode on db1025 for the moment [19:37:10] once i started looking at how civicrm constructed those queries… the choice was clear. [19:37:50] but you couldn't yourself to murder it in the face? [19:38:10] there were some strong arms holding me back [19:38:59] Don't look at me. [19:39:08] (maybe it was James A?) [19:39:11] :p [19:39:48] as it was last time we ran it, it logged at 60s and killed at 300s [19:39:57] awjr: hey, hopefully I don't cause painful flashbacks, but check out this new direction for Civi: https://civicrm.org/blogs/peter-haight/persistence-refactoring [19:40:19] With a real ORM, the fucking queries will not be constructed via spaghetti string manipulation... [19:43:33] is doctrine what zend uses, awight? [19:44:22] awjr: I'm not sure how tightly they're coupled. The trend tho has been towards Doctrine, if anything. Symfony also recommends it. [19:45:16] oh yeah, i think i used it once back in my contracting days - i think i recall liking it. and without a doubt i liked it more than civicrm's bao/dao crap [19:46:12] that is not hyperbole. It's currently impossible to customize anything in the entity persistence or presentation layers... [20:08:24] alright, the query logger-slayer now runs on db1025, you can see logs on indium/loudon [20:10:06] it doesn't look for transactions yet, that's a whole different basket of poodles [21:18:35] (CR) Adamw: [C: 2] "Merging following Katie's approval of https://gerrit.wikimedia.org/r/#/c/94476/" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/94475 (owner: Adamw) [21:19:02] (CR) jenkins-bot: [V: -1] Merge remote-tracking branch 'origin/contrib' into HEAD [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/94475 (owner: Adamw) [21:21:04] (CR) Adamw: [V: 2] Merge remote-tracking branch 'origin/contrib' into HEAD [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/94475 (owner: Adamw) [21:39:06] (PS1) Adamw: fix a typo in Civi release [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/98699 [21:39:19] K4-713: por fa: ^^ [21:39:32] typo, huh? [21:40:15] btw, I'm totally onboard with requiring more rigorous testing before deploying updates. [21:40:19] (CR) Katie Horn: [C: 2 V: 2] fix a typo in Civi release [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/98699 (owner: Adamw) [21:40:26] I found a huge breaking thing in Civi 4.3... [21:40:28] * K4-713 nods [21:40:41] K4-713: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1230?version=1 [21:40:42] * K4-713 nods more [21:41:01] argh [21:41:05] ...it burns [21:41:10] We would have blowed up so freeking hard. With no recovery... [21:41:27] No way the migrations are reversible. [21:41:35] We'd just have to eat a few minutes of lost data. [21:41:49] * K4-713 takes more headache pills [21:47:34] (PS1) Adamw: Revert "FR #988 - Allow ThankYou generator to know about all translations" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98701 [21:47:35] (PS1) Adamw: Revert "Process each message in a DB transaction" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98702 [21:47:36] (PS1) Adamw: Update civicrm submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98703 [21:47:46] K4-713: meep. [21:50:31] mwalker: you in time-sensitive land still? [21:50:34] ^^^ [21:50:44] ya; for a couple more minutes [21:50:48] no worries. [21:53:12] (PS1) Mwalker: New thank you message [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98705 [21:53:31] awight, K4-713: one of you please :) ^ [21:53:36] onit [21:53:47] why the... [22:00:34] K4-713: moment for code review? https://gerrit.wikimedia.org/r/98701 https://gerrit.wikimedia.org/r/98702 https://gerrit.wikimedia.org/r/98703 [22:00:43] awight: What for? [22:00:54] (CR) Adamw: [C: 2 V: 2] Update civicrm submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98703 (owner: Adamw) [22:00:56] I'm trying to juggle, ah, conflisting priorities. [22:01:01] *conflicting [22:01:02] gotcha [22:01:52] mwalker: problem. I think you're relying on reverted code. [22:01:55] awight: seriously though, what is it for? [22:02:00] eh?? [22:02:09] the CR you just asked me to do. [22:02:19] There are two commits I reverted in -deployment [22:02:29] I'm also reverting in master so we don't all go crazy. [22:02:34] what? [22:02:42] why? we should be able to just reapply? [22:02:43] awight: Oh, okay. [22:02:44] for example, matt just wrote a patch relying on code we reverted in deployment [22:03:05] mwalker: you want to reapply the commit we reverted last week? [22:06:31] (Abandoned) Katie Horn: Revert "FR #988 - Allow ThankYou generator to know about all translations" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98701 (owner: Adamw) [22:09:08] awight: rebase for https://gerrit.wikimedia.org/r/#/c/98702/ ? [22:09:44] (PS1) Adamw: Revert "Process each message in a DB transaction" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98707 [22:09:49] K4-713: that one ^^ [22:10:06] new one, eh? [22:10:11] aaargh [22:10:13] a new revert [22:10:17] with _exactly the same code_. [22:10:22] Way to go, gerrit. [22:10:22] hehe [22:10:42] I spose I could have rebased but then you would have to mop up brain fragments [22:10:55] Mine or yours? [22:11:02] (CR) Katie Horn: [C: 2 V: 2] Revert "Process each message in a DB transaction" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98707 (owner: Adamw) [22:11:11] Reversion Done! [22:11:23] ...hehe. [22:11:26] Reversion Aversion. [22:12:00] (CR) Adamw: [C: 2] New thank you message (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98705 (owner: Mwalker) [22:12:50] (PS1) Mwalker: Add the List-Unsubscribe Header [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98708 [22:13:10] (PS2) Adamw: Update civicrm submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98703 [22:17:50] (CR) Adamw: [C: 2] Add the List-Unsubscribe Header (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98708 (owner: Mwalker) [22:30:57] !log updating civicrm from 82af6188f5e019ab23636e78719e17c4cb838859 for new thankyou message [22:31:14] Logged the message, Master [22:38:40] Jeff_Green: where in the heck does https://civicrm.wm.o log to? [22:40:31] errors or access logs? [22:40:41] errors go to syslog, thus central logs [22:40:59] access logs are at /var/log/apache2/somethingerother.ssl.somethingerother.log [22:41:12] eh; both I guess -- I'm getting a whitescreen in a particular part of civi: https://civicrm.wikimedia.org/admin/config/thank_you/test_email [22:41:29] but I don't see myself in the access logs of civicrm.access.log or civicrm-ssl.access.log [22:41:35] K4-713: see link abobe [22:41:43] oh, and then there's php which might do something entirely 'other' [22:44:15] well; katie just got it to work [22:44:19] so... apparently civi just hates me [23:17:38] * K4-713 is crazy-happy with the quiet of the error logs [23:18:18] (CR) Adamw: [C: 2 V: 2] Update civicrm submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/98703 (owner: Adamw) [23:33:08] crazy + happy, I think, is the right construction [23:33:54] Happy like a berzerker, OKAY [23:36:10] Jeff_Green: QA box cannot pull from here, right? tcp://silicon.frack.eqiad.wmnet:61613 [23:36:34] argggh security breach [23:36:38] * awight vaporizes [23:36:44] awight: should be disallowed by firewall rules, is it not? [23:36:49] o i hope [23:36:56] try telnetting