[00:35:15] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1237 processes [00:40:15] RECOVERY - check_procs on frdb1001 is OK: PROCS OK: 486 processes [00:46:03] that is me running a query I think - will kill if more alerts [01:24:28] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, and 2 others: Add yearly totals to... - https://phabricator.wikimedia.org/T186165#4177110 [01:28:20] (PS1) Eileen: Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430528 (https://phabricator.wikimedia.org/T186165) [01:31:37] eileen: if you are around want to try the new bastion per an email i just sent? [01:31:47] probably would have been good to ask before i sent it to fr-all [01:34:35] :-) reading now [01:35:43] is rigel dead already (just comparing my file with the proposed) [01:36:08] eileen: nope rigel is still working [01:36:21] that's the codfw one [01:37:01] well i just tried & it didn't work - not sure if that's my bug tho [01:37:25] might be -hand on [01:38:03] hang not hand :-) [01:38:14] ssh config can be horrible [01:38:23] cwd yep looks like it worked - in frdev again [01:38:31] great [01:38:34] & I had to accept a key [01:38:36] asked for the new key and you said yes? [01:38:43] super [01:38:55] yep [01:39:06] it is actually the same key as tellurium, was an experiment to see if we could actually switch the server w/o interruption [01:39:14] & I see the new name in the request for key so it's not some random hacker box [01:39:20] :) [01:39:44] thanks for testing [01:42:46] nw [01:43:26] cwd btw I found a small change for the trigger script but it will be a case of we approve & deploy & tell you where the file is & you can reload at your convenience [01:43:44] (it will catch a very small number of rows that were being missed ) [01:44:02] ok sounds good [05:45:41] (PS1) Eileen: Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430541 (https://phabricator.wikimedia.org/T186165) [05:49:58] (CR) jerkins-bot: [V: -1] Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430541 (https://phabricator.wikimedia.org/T186165) (owner: Eileen) [06:46:55] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: assess GDPR civi extension and try to export all of a donor's data from Civi - https://phabricator.wikimedia.org/T192677#4177555 (Eileenmcnaughton) I installed the GDPR extension... [07:09:19] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: assess GDPR civi extension and try to export all of a donor's data from Civi - https://phabricator.wikimedia.org/T192677#4177590 (Eileenmcnaughton) Also - for this use AND for th... [11:45:43] (PS9) Jgleeson: WIP -Ported Mapper Transformer behaviour from DI. -Added AmountToCents Transformer. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/427723 [14:54:36] hi jgleeson, mepps, AndyRussG ! [14:54:55] hellloooo ejegg! [14:55:48] ejegg: booo! [14:56:18] K from now on, all my greetings will be prefixed by "sies" [14:56:23] so hellosies and boosies [14:56:58] Also it's Fabricator, and our two-week planning periods are sprinks, not sprints [14:57:09] not shrimps? [14:57:20] no, that was last fiscal decade [14:57:43] lol(sies) [14:57:53] ;p -sies [14:59:25] ejegg, do you have a minute to chat about your comment here https://gerrit.wikimedia.org/r/#/c/427723/8/Core/Mapper/Mapper.php@138 [15:00:54] i am going to be rebooting servers for new kernels [15:01:07] shouldn't hurt nothin but might cause an alert or two [15:03:27] jgleeson: sure! [15:03:54] So, I was just wanting to make sure I understood why you would catch, log, and re-throw [15:03:56] ejegg, great. I just wanted to get your thoughts on how you see smashpig reporting exceptions [15:04:09] instead of just logging at the original place you throw the exception [15:04:40] hmm I think we don't actually throw the exceptions [15:04:46] that's internal to the YAML component [15:05:02] we could wrap the usage of the component [15:05:12] I'll double check [15:05:26] ah, ok, just the first one is our own exception then [15:06:05] actually I misspoke, we do control a lot of them [15:06:34] pcoombe Seddon hi!!! Do you or anyone else, as far as you know, use the fields "country_matches_geocode" or "region" fields in Pivot? [15:09:38] ejegg, so instead we would do something like this? https://glot.io/snippets/f0p0s6zv22 [15:15:02] AndyRussG: not the former [15:15:31] AndyRussG: is region the sub national level one? [15:15:38] yep [15:15:39] jgleeson: yeah, that was my thought [15:15:52] AndyRussG: Yeah I sometimes use that [15:16:01] ejegg, cool, will update [15:16:02] Seddon: okok gotcha [15:16:24] yeah just looking at how to transition that to EventLogging (T186048) [15:16:24] T186048: Adapt Druid banenr_activity jobs to EventLogging-based impression recording - https://phabricator.wikimedia.org/T186048 [15:16:39] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Adapt Druid banenr_activity jobs to EventLogging-based impression recording - https://phabricator.wikimedia.org/T186048#4178632 (AndyRussG) (Note that Pivot is not working properly now for Banner impressions, due to some... [15:16:58] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Adapt Druid banenr_activity jobs to EventLogging-based impression recording - https://phabricator.wikimedia.org/T186048#4178642 (AndyRussG) [15:17:12] AndyRussG: mainly if I'm trying filter out anomalous traffic [15:17:22] ah yeah that makes sense [15:18:06] Seddon: btw Pivot is poo now because upstream made it closed source, but there's a new replacement (backed by the same datastore): https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset [15:18:12] Just trying to get it working [15:19:35] bye all, off to the dentist! [15:20:47] AndyRussG: Yeah I took a gander at that yesterday with JAllemandou [15:20:53] Looks interesting [15:21:12] ah cool [15:21:25] Not working for me just yet [15:22:10] hmmm looks like now I'm finding it [15:23:57] silly closed-source pivot folks [15:51:13] fr-tech config change to activate EventLogging for banner data, for an initial test and to get some initial content into Hive: https://gerrit.wikimedia.org/r/430613 [15:56:23] AndyRussG: great! [15:58:14] ejegg: we could put it on the morning SWAT at 11 am Pacific, then turn off EL at the evening SWAT at 16:00 Pacific [15:58:29] Just a wee test to get our toes wet [15:58:54] at least it's not the SWAT of dearth today [15:58:59] (dearth of deployers) [15:59:41] dearthly hollows? [16:43:09] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Kafkatee changes - https://phabricator.wikimedia.org/T183978#4178979 (AndyRussG) [17:02:57] hey cwd I messed up on the server update instructions :| and now I can't access it [17:03:33] ccogdill: no worries, it can be fiddly [17:03:38] do you get an error? [17:03:59] I think I responded wrong to this question [17:04:07] Are you sure you want to continue connecting (yes/no)? [17:04:28] re: The authenticity of host 'frbast-eqiad.wikimedia.org (XXXXXXXX)' can't be established.... No matching host key fingerprint found in DNS. [17:04:43] wasn't sure if I should share the server address so deleted a few things there [17:05:07] now I just get a "Permission denied (publickey)." error when I try to ssh [17:06:51] ccogdill: did you say yes or no? [17:06:56] I said yes [17:07:03] that's right [17:07:09] something else must be up [17:07:11] checking [17:07:26] this is what followed after I said yes [17:07:41] "Warning: Permanently added 'frbast-eqiad.wikimedia.org,xxxxxxxx' (ECDSA) to the list of known hosts. [17:07:41] Permission denied (publickey)." [17:10:05] ccogdill: you are typing ssh frdev1001 ? [17:10:17] ssh frdev1001 [17:10:27] just copy, pasted so I couldn't fool myself [17:10:35] yep :) [17:11:13] well huh [17:11:24] can you try this: ssh frbast.wikimedia.org [17:12:06] oh I think it worked [17:12:23] I got a wolf/dog with a chat bubble. is that expected? :P [17:12:30] hehe yep [17:12:46] you are on the bastion server now... so there must be something in ssh config [17:13:02] https://wikitech.wikimedia.org/wiki/Fundraising/tech/ssh_config [17:13:04] okay yeah and this isn't letting me into mysql or anything... [17:13:38] yeah, it's a different server [17:13:45] you go through this one to get to the other ones [17:13:53] e.g. frdev [17:14:19] ohh I think I did this before you edited the page! [17:14:20] ccogdill: did you edit the file using the command on the page? [17:16:28] ugh I'm still getting the error after updating my ssf config again... [17:16:31] *ssh [17:17:14] can't get into frbast.wikimedia.org anymore either, though! [17:17:22] uh oh [17:17:45] ccogdill: can you paste the contents of .ssh/config here: https://tools.wmflabs.org/paste/ [17:17:53] nothing in there should be sensitive [17:18:44] rnn [17:18:50] oops cwd [17:19:03] I figured it out :\ [17:19:52] cool [17:20:02] ccogdill: did you just have auth failures by any chance? [17:21:02] haha oh man, yeah I'm actually still not getting in. operation timed out. okay okay pasing my thing in there. sorry, I am being a pain! [17:21:22] haha not at all [17:21:22] https://tools.wmflabs.org/paste/view/d78c0eb7 [17:21:32] yeah there is a thing that will ban after a couple tries [17:21:35] i can clear it [17:22:00] i think your ssh config is good [17:22:03] were you pressing the yubikey? [17:23:07] it's not giving me the chance [17:23:42] didn't ever get far enough for the yubikey prompt [17:31:43] ccogdill: ok sorry had to figure out how to unban on the new system [17:31:47] try again? [17:33:21] weee! [17:33:23] I'm n [17:33:25] *in [17:33:32] and proud to be the first banned user 8) [17:33:40] yay! [17:33:42] hehe [17:33:47] thanks so much [17:34:04] no problem! [17:36:55] !log updated CiviCRM from 401344fb30 to 0fdef242a3 [17:36:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:34:52] mepps I'll be on in 3 min if you still want to talk stuff through [19:10:29] fr-tech default EL banner impression setting now at 0.01 sample rate [19:10:48] so that's what we should get for everything for the next little while [19:26:54] We'll turn it back off at 16:00 hrs... https://gerrit.wikimedia.org/r/#/c/430649/ [19:27:03] (16:00 Pacific time, evening SWAT) [19:28:38] thanks AndyRussG [19:38:51] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Adapt Druid banenr_activity jobs to EventLogging-based impression recording - https://phabricator.wikimedia.org/T186048#4179742 (AndyRussG) There are two fields that in Druid that we can't currently get with the pure eve... [19:39:06] ejegg: :) [19:39:15] back in a bit, kid pickup time... [19:40:47] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Patch-For-Review: Refund the u... - https://phabricator.wikimedia.org/T190854#4179755 [19:45:08] heading to the library, back on in 20 [20:13:36] (PS6) Mepps: Start query for updating contributions mistakenly marked refunded Query from XenoRyet added Bug: T190098 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:13:58] (PS7) Mepps: Query for updating contributions mistakenly marked refunded [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:14:16] XenoRyet does this query still look right? ^^ [20:14:23] looking [20:14:47] i may want to turn this into a batch too but i want to make sure it works correctly first [20:15:24] Yea, query still looks good. [20:25:26] (PS8) Mepps: Query for updating contributions mistakenly marked refunded [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:27:22] (PS9) Mepps: Query for updating contributions mistakenly marked refunded [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) [20:27:53] ejegg|afk might take a look at that when you get back ^^ [20:29:12] mepps: hi! [20:29:45] sorry, been at the lib a bit, just my normal workaround for their http(s) only net access wasn't working [20:30:09] back on the cellular net, will look at that query [20:31:37] no worries ejegg, just posted it [20:31:43] ah nice, using the API to swap the statuses! [20:31:56] hehe, so many 'c' tables [20:34:28] mepps have you tried that query on staging? [20:34:48] the 'not like %-%' bit seems kindof expensive [20:34:53] not yet ejegg but that's a good way to test it [20:37:05] ejegg: The thinking with that one is to identify the initial transactions since time recieved alone wasn't enough to do it. If there's a better way to identify those I'm all for changing it. [20:39:00] XenoRyet: so the link to contribution_tracking isn't enough? [20:39:18] i thought that dct.contribution_id always pointed to the first [20:39:25] I want to say it wasn't, but I don't remember why now. [20:40:47] I might have been wrong though. [20:41:56] fr-tech: i'm thinking about imaging rigel real quick [20:42:02] the codfw bastion [20:42:16] anyone think they're gonna need that soon? [20:42:48] cwd not unless we have an emergency failover [20:48:31] XenoRyet: looks like you can just add a condition to the join on ccr [20:48:39] I'll put it on the ticket [20:48:45] 10-4 [20:49:36] XenoRyet i'm signing off for the weekend shortly, can you carry thsi forward? [20:49:47] next step is testing on staging but i wasn't sure how to confirm we had some test data [20:49:52] https://www.theverge.com/2018/5/3/17316684/twitter-password-bug-security-flaw-exposed-change-now [20:50:04] (CR) Ejegg: [C: -1] "Think we can make the trxn_id conditions faster" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [20:50:31] mepps: Yea, I'll put it on my list. [20:53:59] (PS1) Mepps: WIP Move errored messages to damaged queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430800 (https://phabricator.wikimedia.org/T191654) [20:54:17] (PS2) Mepps: WIP Move errored messages to damaged queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430800 (https://phabricator.wikimedia.org/T191654) [20:58:25] (CR) Ejegg: [C: -1] "Might want to make the batch smaller, if you're going to use it at all. I take back the comments on the filters, though. Just tried the qu" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429097 (https://phabricator.wikimedia.org/T190098) (owner: Mepps) [20:58:41] XenoRyet: ^^^ that query seems fine, actually [20:58:50] Ok cool. [20:59:00] Was just about to push a patch, but we'll leave it as is. [21:03:32] brb, gotta go be a dad for about half an hour. [21:07:31] (PS3) Mepps: WIP Move errored messages to damaged queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430800 (https://phabricator.wikimedia.org/T191654) [21:10:38] (CR) jerkins-bot: [V: -1] WIP Move errored messages to damaged queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430800 (https://phabricator.wikimedia.org/T191654) (owner: Mepps) [21:56:08] (PS2) Ejegg: Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430541 (https://phabricator.wikimedia.org/T186165) (owner: Eileen) [21:56:15] XenoRyet: mind cross-checking that? ^^^ [21:56:25] looking [21:56:31] Turns out the test was a depending on the old broken behavior [21:57:01] so I just fixed the test in PS2 to match eileen's new null trxn_id handling [21:57:12] I hope it passes [21:57:20] still trying to get my local triggers updated [21:59:30] Yea, looks ok to me. [21:59:38] Thanks! Mind +2ing it? [21:59:57] Yea, no worries. [22:00:52] (CR) XenoRyet: [C: 2] Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430541 (https://phabricator.wikimedia.org/T186165) (owner: Eileen) [22:05:10] (Merged) jenkins-bot: Tweak totals calculation to handle records with NULL trxn_id. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430541 (https://phabricator.wikimedia.org/T186165) (owner: Eileen) [22:15:58] (CR) Ejegg: "Dang, just realized this isn't giving us the original crypto-currency! BTC / BCH doesn't show up anywhere in the csv, just in the json exp" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430286 (https://phabricator.wikimedia.org/T192953) (owner: Eileen) [22:17:10] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430822 [22:17:21] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430822 (owner: Ejegg) [22:19:52] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430822 (owner: Ejegg) [22:21:07] !log updated CiviCRM from 0fdef242a3 to 9752607052 [22:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:35:49] (CR) Ejegg: [C: 1] "This looks like the about the best we can do with the csv - sorry I didn't notice that was missing the source cryptocurrency!" (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430286 (https://phabricator.wikimedia.org/T192953) (owner: Eileen) [23:08:08] (CR) Eileen: Add bitpay import (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430286 (https://phabricator.wikimedia.org/T192953) (owner: Eileen)