[00:00:25] cwd so, we'll want to turn of queues -> deploy stuff -> regenerate trigger script -> apply new triggers -> turn queues on [00:01:27] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Edit bitpay page inside their console - https://phabricator.wikimedia.org/T192790#4149925 (Ejegg) p:Triage>Normal a:Ejegg [00:13:19] ejegg: cwd if we know when we'll do it I'll send a quick email out notifying. But if we don't I'll just send when we do it, or not [00:20:46] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4163494 (Ejegg) Thanks for the 'Beste' update @jrobell, but it looks like we lost the 'if' logic around it!... [00:24:51] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4170050 (Ejegg) @jrobell, Looks like we lost a LOT of the tags in the various translations of the latest tha... [00:45:51] eileen: i can do it now-ish [00:46:00] or in a little bit [00:46:03] cwd OK - ejegg are you about? [00:46:19] yep, now is good for me [00:46:28] want me to turn stuff off? [00:47:32] yep - should we flick out an email or is this below the threshold? [00:47:38] go for it [00:47:46] never hurts to communicate [00:47:46] i can wait a few minutes [00:47:48] yeah [00:47:55] that unspellable word again - my T-conspiracty will be all about spelling threshhol [00:48:33] it is a weird one [00:48:38] tempting to pronounce that H [00:50:42] I have so many PRs that are basically 'fix how spelliing of threshhold now I have googled how to spell it' [00:50:44] !log disabled fundraising jobs for db update [00:50:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:03] !log email sent advising of minor interuption [00:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:06] ok eileen, ready to deploy the code [00:51:19] (PS1) Eileen: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430014 [00:51:26] ejegg: ok - just getting it merged to deploy [00:51:35] (CR) Eileen: [C: 2] Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430014 (owner: Eileen) [00:53:31] ejegg: here is the deploy list [00:53:31] + f5cb025b533e0227d80e2c1e16a0bad5395c317a Reformat rgc.module and tests to drupal standard [00:53:32] + 9712cbdca200744be3ba620661299798be48187e Recurring GC job sends queue message [00:53:32] + 6bcf4863a29cf0a447355b49a775f816d31b8484 Use post hook rather than triggers for original amount / currency [00:53:32] + 68f588095c9305886e28f1b73d5e5397926f3bfd Revert change to exclude contribution extra from trigger join [00:53:32] + 8794ceb40918ef3b14448a9a2d29d300f7b9453d Pass skipRecentView from a couple of places for (v minor) speed up [00:53:44] Are those top 2 good to go? [00:53:44] yep, that's what we want [00:54:46] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430016 [00:55:38] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430016 (owner: Eileen) [00:55:53] go zuul go [00:56:10] (Merged) jenkins-bot: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430014 (owner: Eileen) [00:56:36] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430016 (owner: Eileen) [00:56:51] that was nice & speedy! [00:58:07] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430017 [00:58:13] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430017 (owner: Eileen) [00:58:58] OK - we are merged - so we need 2 thngs to happen in quick sequence 1) ejegg push the code 2) I'll regenerate the triggers 3) cwd can run trigger script [00:59:22] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Edit bitpay page inside their console - https://phabricator.wikimedia.org/T192790#4170158 (Ejegg) I added the privacy summary in the footer, and that might be all I can do. There's no option for a header image, and when... [00:59:35] eileen: cool, hit me with the path when ready [00:59:51] OK ejegg are you in frpm ? If so you can push now [00:59:56] eileen: ok, will do [01:00:43] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430017 (owner: Eileen) [01:01:38] !log updated CiviCRM from 47197006d5 to 46883844a3 [01:01:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:01:53] ejegg: so I should hit the command now? [01:02:02] should be good to go [01:02:36] cwd you can access my homedir? I can copy the file there [01:03:00] sure that's fine [01:09:02] cwd ~/may1trigges.mysql [01:11:29] eileen: about to run [01:11:37] cwdent@frdb1001:~$ #sudo mysql civicrm < may1trigges.mysql [01:11:43] look sane? [01:12:16] cwd yep [01:12:37] done [01:13:02] that's good - guess we need to test somehow now :-) [01:13:25] cool, I'll send a test few through the pipeline [01:13:46] ooh, we'll need to manually run the wx audit, it just missed its trigger time [01:14:55] ok, 2 new donations, let's see how the calculated fields look [01:15:26] (PS1) Eileen: Commit triggers as run on live on 1 May 2018 after updates/ removal of contribution_extra triggers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430020 [01:16:18] totals look good! [01:16:18] ejegg: cwd here is the updated trigger mysql for committing - https://gerrit.wikimedia.org/r/#/c/430020/ - since it's mostly a record of what we just did we should probably quick-+2 it & then follow up in panic if we spot any issues [01:16:28] yep [01:18:03] (CR) Ejegg: [C: 2] "Looks sane, but I'd love to get rid of those extra modified_date triggers soon" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430020 (owner: Eileen) [01:18:19] sorry phone call [01:18:22] ok, I'll turn the jobs back on [01:20:11] !log re-enabled fundraising jobs [01:20:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:20:27] OK, now i'll manually run the ingenico audit [01:21:12] And we'll see if the new behavior gives us any errors! [01:21:13] opps I said it was up in an email - too early [01:21:24] cwd update on how to revert if you need to…. [01:21:57] basically from drupal dir [01:22:05] drush cvapi System.flush triggers=1 [01:22:25] (Merged) jenkins-bot: Commit triggers as run on live on 1 May 2018 after updates/ removal of contribution_extra triggers [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430020 (owner: Eileen) [01:22:38] & a new file should be generated in drupal/sites/default/files/civicrm/ConfigAndLo [01:22:40] g [01:22:58] ok, finding a good number of donations! [01:23:09] this should be a good volume test [01:24:38] 5,015 coming through... [01:26:41] lol [01:26:48] can we get a timing? [01:27:49] hmm, that first batch was a little slow, but let's see what the average is [01:28:09] https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=22&fullscreen&from=now-30m&to=now [01:29:29] if you look at the time period right after 01:00 for the last couple weeks, that'll give you the heavy loads [01:30:46] e.g. https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&panelId=22&fullscreen&from=1524099825204&to=1524100690746 [01:33:15] eileen darn, that is a bit of a perf hit for sure [01:34:54] from doing this in php instead of db trigger? [01:35:51] cwd yeah [01:35:54] ejegg: arg [01:36:11] on the bright side, no failmails yet! [01:36:16] :) [01:39:00] ejegg: so drawing out further https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&panelId=22&fullscreen&from=1516317333000&to=1524107182311 [01:40:09] there is no clear pattern on the impact of the triggers being added I guess since not really enough data [01:43:13] hmm, different campaigns are going to have different profiles [01:43:20] What is it processing more of per second? [01:43:33] that should be donations per second [01:44:04] 4? [01:44:27] yeah, that's about what we're getting now [01:44:43] 3.5 ish [01:48:40] cwd: weird, why does the queue graph say no data points for the last few days? [01:48:56] https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&panelId=9&fullscreen&from=1524757778813&to=1525139326812 [01:50:06] seems to stop right after 15:30 on 4/26: https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&panelId=9&fullscreen&from=1524753679965&to=1524759524370 [01:51:41] huh, checking [01:51:52] there has been a lot of puppet refactoring for stretch [01:52:36] ejegg: right - but absolute peak of 6.54 this year seems low https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&panelId=22&fullscreen&from=1516317333000&to=1524107182311 [01:55:28] ah I guess I have 180 per min in mind from earlier - so that's about 3 per sec [01:55:50] ejegg: shall we maybe plan to review performance after a week? [01:56:07] yeah, sounds good [01:56:59] for now, I'm happy we've gotten through the audit batch without failmail! [01:57:08] going to sign off for the evening [01:57:15] have a good one! [01:59:49] Fundraising-Backlog, fundraising-tech-ops: Fundraising queues missing prometheus data since 4/26 - https://phabricator.wikimedia.org/T193456#4170230 (cwdent) [02:15:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Do some queue performance analysis - https://phabricator.wikimedia.org/T193457#4170244 (Eileenmcnaughton) [02:32:54] (PS1) Eileen: Add extra conditions to drush script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430022 [02:49:42] (PS1) Eileen: Prevent error on logging report for custom table disabled for logging (here wmf_donor) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/430023 (https://phabricator.wikimedia.org/T192808) [11:11:51] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4170746 (Pcoombe) It looks like marking the page up for translation wasn't done properly. I'll try to fix it. [12:47:56] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4170951 (jrobell) Thank you @Pcoombe ! [14:38:06] (CR) Ejegg: [C: 2] Prevent error on logging report for custom table disabled for logging (here wmf_donor) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/430023 (https://phabricator.wikimedia.org/T192808) (owner: Eileen) [14:42:30] (Merged) jenkins-bot: Prevent error on logging report for custom table disabled for logging (here wmf_donor) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/430023 (https://phabricator.wikimedia.org/T192808) (owner: Eileen) [14:46:47] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4171105 (Pcoombe) Okay, I believe the translations are now correctly tagged and published for nl, plus da, i... [14:55:01] (PS2) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [14:55:35] (PS3) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [14:58:08] (CR) jerkins-bot: [V: -1] Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) (owner: Mepps) [15:02:06] (PS4) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [15:04:48] (PS5) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [15:06:42] (CR) jerkins-bot: [V: -1] Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) (owner: Mepps) [15:08:55] (PS1) Mepps: Composer fix for failing build [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/430074 [15:11:36] (PS6) Mepps: Log error and move to damaged on ValidationException in orphan rectifier, but continue Bug: T191654 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428940 (https://phabricator.wikimedia.org/T191654) [15:11:57] (PS2) Mepps: Composer fix, small [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/430074 [15:15:32] (CR) Ejegg: [C: 2] Composer fix, small [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/430074 (owner: Mepps) [15:16:04] (PS2) Ejegg: Add extra conditions to drush script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430022 (owner: Eileen) [15:16:09] (CR) Ejegg: [C: 2] Add extra conditions to drush script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430022 (owner: Eileen) [15:18:10] (Merged) jenkins-bot: Composer fix, small [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/430074 (owner: Mepps) [15:20:28] mepps, don't you think the same component should have the responsibility for popping orphans off the pending table to rectify, and moving those messages to the damaged table? [15:20:54] (Merged) jenkins-bot: Add extra conditions to drush script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430022 (owner: Eileen) [15:21:25] hmm ejegg that makes sense to me, i guess i wasn't sure how easy that would be in crm, does it make sense to do it in paymentswiki? [15:21:30] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/430076 [15:21:55] mepps can't we just put a try / catch in the drush wrapper script? [15:22:11] the DamagedDatabase::storeMessage function is part of SmashPig [15:22:17] so it's available in CRM as well [15:22:34] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/430076 (owner: Ejegg) [15:22:55] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/430076 (owner: Ejegg) [15:33:55] cwd I'd like to deploy a thing that lets the Adyen capture job read risk scores from the payments fraud table [15:34:17] I just need to add something to /etc/smashpig/main.yaml [15:34:34] ok [15:35:00] oh hey, it may actually already be there [15:37:44] !log updated SmashPig standalone from a4de12d415 to 99585c8084 [15:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:43:07] jgleeson|gym: ping me when you're back, and I can walk you through the thank you letter update [15:45:37] (PS2) Ejegg: build: Updating jakub-onderka/php-parallel-lint to 1.0.0 [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/417769 (owner: Libraryupgrader) [15:53:54] (PS3) Ejegg: build: Updating jakub-onderka/php-parallel-lint to 1.0.0 [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/417769 (owner: Libraryupgrader) [15:55:19] (CR) Ejegg: [C: 2] build: Updating jakub-onderka/php-parallel-lint to 1.0.0 [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/417769 (owner: Libraryupgrader) [16:00:42] (PS2) Ejegg: Replace usage of User::EDIT_TOKEN_SUFFIX with Wikimedia\Session\Token:SUFFIX [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/423182 (owner: Florianschmidtwelzow) [16:01:05] (PS3) Ejegg: Replace usage of User::EDIT_TOKEN_SUFFIX with Wikimedia\Session\Token:SUFFIX [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/423182 (owner: Florianschmidtwelzow) [16:01:28] (CR) Ejegg: [C: 2] Replace usage of User::EDIT_TOKEN_SUFFIX with Wikimedia\Session\Token:SUFFIX [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/423182 (owner: Florianschmidtwelzow) [16:08:50] (Merged) jenkins-bot: Replace usage of User::EDIT_TOKEN_SUFFIX with Wikimedia\Session\Token:SUFFIX [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/423182 (owner: Florianschmidtwelzow) [16:12:55] hi XenoRyet ! [16:13:56] ejegg, back, just gonna grab a coffee [16:13:59] 'mornin [16:14:16] hi jgleeson [16:14:48] XenoRyet: lmk if you want to go through anything with the ct_id bug [16:15:05] Yea, let me get booted up and see where I am, then I'll shoot you some questions. [16:16:03] The basic gist of where I was getting stuck though is that I'm not seeing how the old GC stuff doesn't also run into this bug. I was planning on dealing with it the same way we did it there, but I can't find where it's being handled. [16:21:24] hi fr-tech! [16:21:31] ejegg, hit me up when you're free [16:21:53] jgleeson: right now works! [16:22:07] ok great, see you in the usual place [16:22:11] cool [16:22:38] Oh, XenoRyet or dstrine, did you just change some meetings? looks like they ended up as multi-day affairs [16:22:54] I haven't changed anything today. [16:23:07] Heh, look at that though. [16:23:21] ejegg: XenoRyet sorry one second [16:23:43] No worries. [16:24:54] I mangled a few meetings yesterday and then I had no idea where they went. I guess now I found them :P [18:47:17] PROBLEM - check_gcsip on payments2003 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:50:17] RECOVERY - check_gcsip on payments2003 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.211 second response time [18:52:18] PROBLEM - check_gcsip on payments1003 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:57:18] PROBLEM - check_gcsip on payments1001 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:57:19] PROBLEM - check_gcsip on payments1003 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:57:20] PROBLEM - check_gcsip on payments1004 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:57:21] PROBLEM - check_gcsip on payments1002 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:57:23] PROBLEM - check_gcsip on payments2001 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [18:58:03] cwd, anything to be worried about above? [19:00:14] RECOVERY - check_gcsip on payments1004 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.116 second response time [19:00:15] RECOVERY - check_gcsip on payments1003 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.118 second response time [19:00:16] RECOVERY - check_gcsip on payments1001 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.124 second response time [19:00:18] RECOVERY - check_gcsip on payments1002 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.116 second response time [19:00:19] RECOVERY - check_gcsip on payments2001 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.205 second response time [19:08:54] jgleeson: nah it's just some puppet updates [19:09:26] ah ok cool [19:09:42] what is gcsip out of interest? [19:09:49] global collect? [19:09:54] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4172133 [19:10:57] i think so? [19:12:00] https://ps.gcsip.com/ [19:21:22] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression dat... - https://phabricator.wikimedia.org/T192839#4172265 [19:39:24] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4172510 (Ejegg) Hi @jrobell or @pcoombe, I had to make one tiny correction in the unsubscribe tag - could on... [19:52:18] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Update silverpop export to use new do_not_... - https://phabricator.wikimedia.org/T192517#4141682 [20:15:13] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:15:14] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:20:13] PROBLEM - check_puppetrun on frbackup2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:20:13] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:20:14] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 7 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/etc/network/if-up.d/wikimedia-lvs-realserver],File[/etc/motd.tail],File[/usr/bin/check-raid.py],File[/usr/share/wikimedia-lvs-realserver/sysctl.conf] [20:20:15] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[kafkatee] [20:25:13] PROBLEM - check_puppetrun on frbackup2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:25:14] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:25:14] RECOVERY - check_puppetrun on rigel is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:25:15] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 41 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[kafkatee] [20:30:13] RECOVERY - check_puppetrun on frbackup2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [20:30:13] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:35:13] RECOVERY - check_puppetrun on alnitak is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [20:41:44] sorry about missing stand up - had some connection probs. I put up a patch on that log tab issue & I'll start on bitpay today [20:55:00] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4172756 (Pcoombe) @Ejegg Done! [21:02:55] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Patch-For-Review: Refund the u... - https://phabricator.wikimedia.org/T190854#4172773 [21:15:03] (PS1) Ejegg: Update NL thank you letter [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430256 (https://phabricator.wikimedia.org/T193228) [21:18:09] ejegg, AndyRussG o/ [21:18:19] hi cwd [21:18:24] just talking to Jeff_Green about kafkastuff [21:18:31] yah? [21:18:41] there is a kafkacat candidate for stretch [21:18:54] kafkacat vs. kafkatee ... you can't just do kafkacat | tee ? [21:19:30] i think they're not comparable tools really [21:20:01] kafkatee has service/daemon oriented features [21:20:12] ah i see it's a user and everything [21:20:51] right, and I think it has some degree of buffering, daemonizes itself, that kind of thing [21:21:57] kafkacat seemed to dump the stuff just fine [21:22:06] but yeah, you'd have to keep track of how far it got [21:22:27] to decide what offset to use for the subsequent run [21:22:56] guessing that's the extra stuff -tee adds? [21:23:12] true [21:23:34] also keeps metrics we probably drop on the floor [21:26:29] we are going to be consuming a different stream now yeah? [21:29:33] yeah [21:31:12] woohoo, puppet is finally "probably mostly" v4.8 compatible [21:32:10] and with that, I hereby punt for the day. have a good evening all [21:37:52] cwd so... is there are kafkatee candidate for stretch, too? [21:38:18] nope :( [21:39:23] we can build it [21:39:32] but the puppet stuff is a problem too [21:47:20] nothing insurmountable just wanted to explore options [22:11:43] (CR) Ejegg: [C: 2] Update NL thank you letter [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430256 (https://phabricator.wikimedia.org/T193228) (owner: Ejegg) [22:14:38] (PS1) Ejegg: Update CiviCRM [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430265 [22:14:42] (CR) Ejegg: [C: 2] Update CiviCRM [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430265 (owner: Ejegg) [22:16:14] (Merged) jenkins-bot: Update NL thank you letter [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430256 (https://phabricator.wikimedia.org/T193228) (owner: Ejegg) [22:19:29] (Merged) jenkins-bot: Update CiviCRM [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/430265 (owner: Ejegg) [22:21:29] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430266 [22:21:33] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430266 (owner: Ejegg) [22:23:51] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/430266 (owner: Ejegg) [22:25:54] ejegg: just realised you already deployed that civi -fix - thanks [22:26:03] (just doing a test for upstreaming it now) [22:26:08] eileen: deploying it now actually [22:26:16] looked safe enough! [22:28:06] ejegg: cool [22:28:32] I am having to do a bit of work on upstreaming the test because I had to refactor a little on the report to make it testable [22:28:41] but nearly done [22:29:05] !log updated CiviCRM from 46883844a3 to 401344fb30 [22:29:07] https://github.com/civicrm/civicrm-core/pull/12065 [22:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:44:50] (CR) Eileen: [C: 2] Catch rollbacks between contrib and post hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429987 (https://phabricator.wikimedia.org/T192977) (owner: Ejegg) [22:45:17] (CR) Eileen: [C: 2] "There is no downside to this IMHO but I also don't believe it will ever be triggered based on my testing" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429987 (https://phabricator.wikimedia.org/T192977) (owner: Ejegg) [22:48:06] Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Patch-For-Review: Help to deploy new Thank you email in Dutch (with updated tax copy) - https://phabricator.wikimedia.org/T193228#4163494 (Ejegg) OK, this is deployed and a test send looked good [22:48:47] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog: Give John Bennet a full tour of fr-tech systems - https://phabricator.wikimedia.org/T191833#4173120 (Ejegg) Open>Resolved p:Tria... [22:48:59] Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising Sprint Unbreaking Now, Fundraising Sprint Value Subtracting, and 4 others: Capture Adyen payments without pending messages - https://phabricator.wikimedia.org/T149861#4173124 (... [22:49:08] (Merged) jenkins-bot: Catch rollbacks between contrib and post hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/429987 (https://phabricator.wikimedia.org/T192977) (owner: Ejegg) [22:53:01] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Civi change log yielding server error 4/23... - https://phabricator.wikimedia.org/T192808#4150242 [23:18:55] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising-Backlog, and 2 others: Add signature images... - https://phabricator.wikimedia.org/T189535#4044915 [23:33:21] ejegg: looking at bitpay - we have the column headers but not sample data? I see 'date' but not sure format, ditto currency etc [23:36:39] eileen: oh sorry, there was sample data in that pdf too [23:36:53] will send it as well [23:37:44] cool [23:38:50] eileen: just forwarded you a thing - did the attachments come with it ? [23:40:50] yes [23:41:18] cool [23:41:34] oh dang - this looks painful - there is also a reconciliation report with totals [23:41:47] that don't necessarly match [23:41:57] eileen: right, and that one's only in json format [23:42:27] but we generally don't use recon reports for civi [23:44:13] eileen: I think we'll be fine using the csv ledger [23:45:31] ejegg: ok - why don't we pull these by api - because the import is less work? [23:45:55] yeah, basically [23:46:07] dang - it's an image! [23:46:16] whaaa [23:46:21] is it mostly less work cos we already have a structure in place? [23:46:26] right [23:46:36] api needs some kind of auth, etc [23:47:26] & we haven't got a structure in place for storing all those codes? Or possibly they might all be weird & whacky? I guess the latter [23:47:48] which codes? [23:48:23] credentials [23:48:44] argh it's a screenshot of part of the csv - still I'll do best guess [23:48:49] yeah, we do... [23:49:12] wait, sorry, there's better example data [23:55:41] eileen: ok, copypasted another bit of example data [23:57:12] cool [23:58:48] looks like dates are UTC