[00:14:11] I just pushed a patch to turn off rml fetch per converstation today - if someone wants to check I'll deploy [00:23:40] I've also added the changes to thank you main so that it only runs for the past day on most runs and once an hour does up to 60 days (dwisehaupt you might like to check that one) [00:30:21] Wikimedia-Fundraising-Banners: [CANDIDATE] Mobile large and small w/ old 5 option ask string - https://phabricator.wikimedia.org/T269306 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr Signing off on QA for these variants now @spatton No blockers found, these banners are now READY TO TEST! [00:33:09] Wikimedia-Fundraising-Banners: [CANDIDATE] Mobile large and small w/ old 5 option ask string - https://phabricator.wikimedia.org/T269306 (TSkaff) Thanks again! [00:39:22] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, Android-app-Bugs, and 6 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (MattCleinman) From https://developer.apple.com/library/archive/d... [00:49:59] Wikimedia-Fundraising-Banners: [CANDIDATE] Mobile large and small w/ old 5 option ask string - https://phabricator.wikimedia.org/T269306 (spatton) Thank you @jbolorinos-ctr! [01:00:50] Wikimedia-Fundraising-Banners: [Candidate] M Lg: Payment methods 1st in order - https://phabricator.wikimedia.org/T269289 (jbolorinos-ctr) Open→Resolved Ok signing off on QA for these now @TSkaff!! Everything looks good! No blockers found! I do like the cadence of Payment Methods > Amounts > Frequen... [01:10:07] (PS1) Cstone: Remove the for your record section when it is a stock. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644951 [01:27:53] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (AndyRussG) //Note: edited my last comment to fix some typos and unclear bits. :)// From [[ https://phabricator.wikimedia.org/T268685#6661852... [01:39:46] ejegg|afk: if you are around at some point I put up changes on frpm process control [01:43:23] hi eileen ! [01:43:35] will take a look, one sec [01:43:37] ejegg: hey [01:43:44] there are 2 commits [01:43:51] both things we have discussed [01:46:13] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (AndyRussG) Again, replying to part of [[ https://phabricator.wikimedia.org/T268685#6653580 | this ]] comment: > 2) We wouldn't be able to ma... [01:49:00] eileen: i guess if you want the _60_days command to have a slow-start version it should probably use --days=60 as well [01:49:11] right now it's just copy-pasted from the other with --days=1 [01:49:22] ejegg: arg did I miss that - ok will fix [01:49:39] but besides that everything looks good! [01:49:55] oh hey, RML import was every 8 minutes? [01:50:57] ejegg: yeah looks like it [01:52:06] ok deploying now - lint didn't like my line length [01:52:39] yah, I wonder if we can relax that [01:52:49] I think they did add some kind of lintrc files [01:52:52] might help on missing stuff [01:53:17] aha, they have a .yamllint file there [01:53:26] and it DOES have a rule for the line length [01:53:45] I guess we just extend that to something silly long like 240 chars [01:53:56] new tweet length [01:54:16] !log process-control config revision is f863b32627 [01:54:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:54:30] ok deployed now to check it went [01:55:05] check queue is still going [01:55:10] or are they 280 chars now? [01:55:37] ok, ty sender should hit the :56 one in a minute! [01:56:52] oh right yeah [01:57:16] yep it's running [01:57:34] we have a bit of drama here [01:57:53] neighbour came by & asked to borrow Richard's weedeater. He gave her 4 sheep instead [01:57:55] IRL ? [01:57:59] oh hah [01:58:00] yep [01:58:11] but, from the sounds he separated some mums & lambs [01:58:20] awwwww [01:58:24] & there is bereft baaing [02:01:39] ejegg: so we could reduce the sleep on donation processing [02:01:51] hmm, donations queue is up a ways [02:01:56] but we might need to do it on most runs not all [02:02:29] ahh, recurrings have started for the utc-day [02:02:54] ah thash it [02:03:05] oh right eileen, we'd need a :56 dq run too [02:04:31] yep [02:04:59] oh nice, the process control jobs chart works again [02:05:55] and seems to have already added a bar for your new 60 days job - Jeff_Green or dwisehaupt must have figured out a smarter way to generate it [02:07:10] ok so I think to make it work for the dq I'd need to change it to run on 2 min past the hour for the slower one [02:07:23] that way it's easier to make the dq one work [02:12:45] ejegg: I've pushed up no-sleep for normal run, 15 sec once an hour - wanna check? [02:14:08] sure, one sec! [02:14:42] latest fail mail is an emoji one [02:15:12] I think https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/644394 would fix [02:16:50] yeah that whole chain is emojis eileen as soon as i finish testing this adyen one eileen im gona test the emoji one [02:17:03] cool [02:17:19] or should I say .... 🆒 [02:18:22] 😂 [02:19:20] ok, got it, I should have fully read your irc comments first [02:20:22] shoot, now we need to make the donations queue consumer take the seconds on the command lines [02:20:36] huh? [02:21:04] so we can extend the timeout on the non-sleepy runs and squeeze a few more seconds out of it [02:22:01] I'll just do that up quick [02:23:21] we seem to call it batch_time [02:24:48] ehh, I like the SmashPig time-limit better [02:25:27] oh right, you used time_limit for the TY sender [02:25:38] that's the one to be consistent with [02:26:29] cool [02:28:29] ejegg: I thought I'd look at that phab to make fredge less wierd - do you know where the repo is that does the db create for the docker build? [02:29:35] I guess the goal is that we can just create the dbs through buildkit now [02:29:39] eileen: you mean for CI? [02:29:47] or AndyRussG's new one? [02:30:02] yep ci - see https://phabricator.wikimedia.org/T267970 [02:30:30] drat, i don't know off hand [02:31:17] ejegg eileen one sec I can find the CI docker images repo, if that's useful? [02:31:33] AndyRussG: yeah I think so [02:31:39] you found the place once before I think [02:31:42] but I lost it [02:32:47] Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (jbolorinos-ctr) Was there a drop in MC on any banner type in particular? Mobile vs Desktop? I'll take a look at this first thing tomorrow morning [02:32:52] eileen ejegg https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/dockerfiles/civicrm/ [02:32:56] what was lost is found again?? [02:33:02] thanks [02:33:26] np, I'm around but doing non-work for a bit, but feel free to ping 4 anything.... :) [02:36:20] Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (jbolorinos-ctr) Ok well I just tried making a test monthly donation on Paypal and got this error, so that does explain it: {F33933383} [02:39:57] (PS1) Ejegg: Stop logging normalized contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644953 [02:39:58] thx [02:39:59] Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (jbolorinos-ctr) Steps to reproduce: 1. Open https://en.wikipedia.org/wiki/Wikipedia?banner=B2021_1006_en6C_dsk_p1_lg_template&country=US 2. Select any donation amount 3. Select... [02:40:00] (PS1) Ejegg: Autoformat queue2civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644954 [02:40:02] (PS1) Ejegg: Add time_limit drush option for donations qc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 [02:40:25] eileen with ^^^ we can have a longer time limit for the non-sleepy QC runs [02:40:43] Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (jbolorinos-ctr) @spatton @DStrine any ideas as to why this may be happening? Is there any further information I can provide to help investigate this issue? [02:41:34] ooh, monthly donations on paypal coming up with errors? [02:41:43] hmmm [02:44:57] no errors in the logs for the token in that screenshot [02:45:44] Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (spatton) I followed those steps above and was able to complete a recurring paypal donation, arriving at the TY page as expected. I'm using Windows 10, Chrome 86.0. @jbolorin... [02:47:39] (CR) Eileen: "one question" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 (owner: Ejegg) [02:48:24] (CR) Eileen: [C: +2] Autoformat queue2civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644954 (owner: Ejegg) [02:49:05] (CR) Eileen: [C: +2] "yep - will reduce disk churn & improve readability" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644953 (owner: Ejegg) [02:50:18] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (spatton) [02:54:07] ejegg: I just had one thought on that drush one [02:56:01] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (spatton) p:High→Unbreak! [02:56:18] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Investigate low rate of paypal monthly convert donations - https://phabricator.wikimedia.org/T269303 (spatton) Bumping the priority to Unbreak Now because I've been able to confirm this on multiple devices. TY fr-tech! [03:01:12] oops, just saw that. looking [03:01:57] oh, good catch on that time limit [03:02:17] yah, we don't use zero but it would be nice to be consistent [03:03:28] (PS2) Ejegg: Add time_limit drush option for donations qc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 [03:04:30] (CR) Ejegg: "Thanks Eileen" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 (owner: Ejegg) [03:05:34] (CR) Eileen: [C: +2] Add time_limit drush option for donations qc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 (owner: Ejegg) [03:06:21] (CR) Cstone: [C: +2] "Looks good!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/644260 (https://phabricator.wikimedia.org/T268654) (owner: Ejegg) [03:06:57] (Merged) jenkins-bot: Adyen: detect more recurring auth messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/644260 (https://phabricator.wikimedia.org/T268654) (owner: Ejegg) [03:10:47] (CR) jerkins-bot: [V: -1] Autoformat queue2civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644954 (owner: Ejegg) [03:11:29] silly timeout [03:11:42] (CR) Ejegg: [V: +2] Autoformat queue2civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644954 (owner: Ejegg) [03:11:48] forced [03:16:36] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/644956 [03:16:44] hmm, does that last one on master need a jiggle? [03:17:23] no, zuul's just taking its time [03:20:25] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (spatton) [03:23:46] hey ejegg! just looking for confirmation that y'all saw this issue: https://phabricator.wikimedia.org/T269303 [03:25:29] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1359 [03:27:42] ug [03:27:52] replag too? [03:27:54] all the fun [03:28:35] well no queries running at the moment [03:28:46] also the dulcet tones of traumatised sheep [03:30:31] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1658 [03:31:44] replag getting worse [03:33:47] the heck could that be? [03:34:48] OK on the paypal I might have seen something - when logging in (incognito) I saw 'validate captcha' fly past but no captcha showed up [03:35:01] & it failed - that's ringing some bells [03:35:17] dwisehaupt: do you know re replag? [03:35:21] no hung query [03:37:02] captcha patch I've seen related to a / in the url https://lab.civicrm.org/dev/financial/-/issues/153 [03:37:12] https://github.com/civicrm/civicrm-core/pull/18993 [03:37:29] note I didn't wind up logged in [03:37:49] oh yeah, i just notice that capcha 404 in the network log too [03:38:27] but it would let non incognitos through? [03:38:28] OK - I literally cannot log into paypal on my incognito chrome [03:38:45] huh [03:38:49] regardless of whether I start from our page - or just plain paypal [03:38:58] oh i got it in normal chrome [03:39:19] actually - it probably still has remnants of the request - so ignore that [03:39:31] spatton so it's only broken on incognito mode? [03:39:38] it just broke for me on normal ejegg [03:39:43] oh hmm [03:40:24] cstone did you get to the screen where you review the purchase and it asks you to click 'agree and pay'? [03:40:30] or did it break before that? [03:40:35] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 1408730 Threads: 19 Questions: 172253539 Slow queries: 809352 Opens: 47096627 Flush tables: 1 Open tables: 200 Queries per second avg: 122.275 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [03:40:53] it asked for my password [03:41:10] then broke [03:41:19] but i was logged into paypal earlier [03:41:19] in my incognito window I logged in & back out & then it worked [03:41:39] no it didn't [03:42:24] I got a failure code in the url VFJBTlNBQ1RJT05fUkVGVVNFRA [03:42:30] hmm, it's working for me [03:42:36] I think we need to email paypal [03:42:42] same eileen [03:43:30] ok - 26th time lucky [03:46:25] cstone did you save the token from the paypal URL in your attempt? [03:46:33] I'm writing an email [03:46:37] yeah I can get three [03:46:43] that thank you mail fail is new - gonna check it out since it could hold up further ones [03:46:43] and would like to include whatever I can from the log for a failed one [03:47:34] ooh, tricksy email badness [03:47:38] thanks eileen [03:47:47] worth fixing by hand for now, I guess? [03:47:57] yep - email starts with a - [03:48:06] EC-4B833170TM2653305 [03:48:12] EC-5WC815227K1336457 [03:48:30] EC-0C834453SM730661K [03:48:53] Cannot GET /webapps/hermes/api/checkout/EC-4B833170TM2653305/session/create [03:48:57] JS error [03:51:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: thank you mail tripped on cid 48633919 - https://phabricator.wikimedia.org/T269322 (Eileenmcnaughton) [03:51:43] thanks! [03:51:44] ok - https://phabricator.wikimedia.org/T269322 covers the email one [03:52:08] oh, is the replag silverpop-related? [03:52:39] being that it's just on frdev [03:52:42] timing feels too late [03:52:46] and silverpop is running [03:52:49] hmm, ok [03:52:59] ah if it's running they yeah - likely [03:53:09] & if it's only frdev which it seems to be [03:53:30] also silverpop will slow down as 7 days becomes a lot more [03:58:04] when I look up the rfc for email I wind up being .... asked to donate to wikipedia... [04:02:39] ^ I appreciate all y'all working on that paypal weirdness, cstone ejegg eileen [04:05:42] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (Ejegg) I searched our server logs for the EC- token in the screenshot and don't see any errors. I turned on debug logging for PayPal and @cstone wa... [04:07:52] (Merged) jenkins-bot: Add time_limit drush option for donations qc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644955 (owner: Ejegg) [04:07:56] hah eileen [04:08:21] i was thinking I'd deploy that ^^^ but it's gotten awfully late here [04:08:56] eileen if you'll be around to monitor I'll push it up to prod [04:09:15] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (MBeat33) Thanks @Ejegg We've had one confirmed ticket so far, and I got EC-8A2275024W502191E when replicating this. [04:09:32] ejegg: yeah ok [04:09:59] are you tweaking the job - or just deploying that [04:10:46] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/644956 (owner: Ejegg) [04:11:02] think I'll just deploy the patches for now [04:11:06] ok cool [04:13:00] !log updated fundraising CiviCRM from a2979cbba1 to 913ccdfd2b [04:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:28:17] good night all! [04:29:00] ok, replag looks better and donations queue is back down [04:29:02] phew [04:32:08] huh, strange dip in traffic for a half hour around 3:00 UTC [05:14:39] eileen: sorry yes. the frdev replag is the silverpop job. it is known at that time daily. [05:15:10] got sucked into kid/dinner routine. [12:40:30] jgleeson: Just saw your mention. I got curious and looked around some more, the dockerized FR stuffs seems great. I left a minor comment https://phabricator.wikimedia.org/T262971#6661449 [12:40:54] I'm wondering how you'll manage the many components, basically. I see it's already an active conversation :-) [12:41:57] & I linked to a similar but much smaller thing we've been doing, which encapsulates each component in an arbitrary way, so it can provision itself within an existing container, or as a new container in a docker-compose cluster. Not sure if this is going to work well in the long run. [13:03:54] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (jgleeson) hey @AndyRussG. I've read your replies and although I could reply to each point further explaining and answering your questions,... [13:04:20] Hi awight, thanks I'll check it out [13:09:37] awight: yeah we different views on how we can do that basically [13:13:02] I personally like the idea of each application/service/project having its own docker image (which could extend from a common base image if it makes sense) and then we pull those in via docker compose while also incorporating a shared directory of cross-app stuff that makes sense to be disseminated across the apps from one place for the purposes of using the stack together in development [13:15:57] awight: do the modules in that project represent different configurations for mediawiki (a little bit like roles?) [13:24:52] what AndyRussG has done so far is definitely a great achievement! [14:06:49] Hi jgleeson! Noticed I'm not seeing fresh results on frdev1001 for the past 30 mins or so. Grafana looks mostly okay, but it looks like replication is lagging: https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?viewPanel=2&refresh=1m&orgId=1&from=now-6h&to=now [14:11:09] oh rly pcoombe lemme check thast out [14:11:12] that* [14:11:54] ty! [14:15:07] pcoombe: I see data timestamped: 2020-12-03 14:12:57 [14:15:34] the latest contribution at the point I ran that query [14:15:41] on frdev1001 [14:16:42] I also see the rep lag creeping up [14:17:13] looks like the same thing happened yesterday too [14:18:23] ah no, it was earlier this morning. [14:19:10] just checked again, donation stamped 2020-12-03 14:18:13 [14:19:54] although if you are relying on data from stored in frdb1003 then I can see that might be the issue [14:20:36] yeah, the last I can see is 2020-12-03 13:22:38 [14:20:42] I'm not sure how the db cluster relates to the apps we run. What is it you're running on frdev1001 pcoombe ? [14:22:01] ah yeah, I am using databases on frdb1003 [14:22:23] results scripts, and the mysql prompt [14:23:11] ah. I wonder what's causing that to backup so much. Jeff_Green you around? [14:23:26] hey, yes, just getting oriented! [14:24:36] usually the lag on 1003 is from the scripts that generate the cubes, checking though [14:28:38] there are queries from both superset and from the cube scripts in the slow log [14:29:46] ah [14:30:09] Okay, grafana says lag is back to 0 now and I'm seeing fresh results [14:30:30] ah great! [14:32:54] looking at replication lag for the past 7 days today stands out as different, not why yet [14:37:28] cool. I've gotta drop off for a little bit to go pickup my son from school but mepps is around if you need anyone to help test stuff. catch you all later! [14:37:44] thanks jgleeson! [14:38:09] hey Jeff_Green--i'm seeing some mysql errors in failmail over the last 12 hours--could that have anything to do with it? [14:39:43] I was just looking at that, the only place I'm seeing replication lag is on frdb1003, which should only affect analytics-related stuff. Which job was spewing errors? [14:43:07] it looks like something in wmf_common [14:43:12] let me see if there's more in logs [14:43:22] hmm it's an ingenico donation [14:43:51] these? Subject: Fail Mail : UNKNOWN ERROR: civi1001 (wmf_common) [14:43:54] hmm i suspect this might the emoji thing [14:44:00] yeah Jeff_Green [14:44:09] what's the emoji thing? [14:44:22] we've been getting errors when people put emojis in forms [14:44:26] and they get to the db layer [14:44:33] ha! [14:44:43] db does not have a sense of humor [14:45:46] oh hmm, it looks like we got these wmf common errors even after the emoji failmails stopped [14:46:40] but i am seeing the phrase "illegal" in there [15:02:29] Fundraising-Backlog: Emojis in firstname/lastname causing names to be too long for Ingenico - https://phabricator.wikimedia.org/T241757 (mepps) Hmm there was a patch here but we're still seeing db errors as of today caused by someone putting an emoji into last name. [15:24:17] Wikimedia-Fundraising-Banners: [Candidate] M Lg: Payment methods 1st in order - https://phabricator.wikimedia.org/T269289 (HNordeenWMF) Hi @jbolorinos-ctr. One more variant we'd like to test along side these when you have a moment! Frequency > Amounts > Payment Methods https://en.m.wikipedia.org/wiki/Wikipe... [15:49:45] Fundraising-Backlog: Emojis in firstname/lastname causing names to be too long for Ingenico - https://phabricator.wikimedia.org/T241757 (Cstone) @mepps the last name causing the donations queue consumer to fail is this task https://phabricator.wikimedia.org/T268356 [15:50:11] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (DStrine) [16:01:43] (CR) Ejegg: [C: +1] "This sure does fix errors searching for emojis on staging. I'm still trying to understand how it does that without also neutering searches" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [16:06:47] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (MBeat33) [16:27:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: mailing events count really high for 10/6 email send - https://phabricator.wikimedia.org/T265073 (MBeat33) Open→Resolved a:MBeat33 [16:41:40] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, fr-donorservices: Civi: Recur fail message sent to donor w/non-failed donation - https://phabricator.wikimedia.org/T269372 (MBeat33) [16:59:45] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (EMartin) @MBeat33 Update: I had Ingenico look into this timeout and they report: There have been no issues reported on our end at the moment. I have reached out... [17:02:57] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (MBeat33) many thanks, @EMartin ! I asked the donor to retry and if they (or anyone else) report it recurring, I'll add here. [17:15:28] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (Pcoombe) Removed monthly paypal from donatewiki until this is confirmed fixed. [Diff](https://donate.wikimed... [18:04:28] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (jbolorinos-ctr) Weird...I just tested this on Windows 10/Firefox 83 and was able to get the form to load {F33934718} Did the user mention which OS they're using... [18:32:15] Fundraising-Backlog: Emojis in firstname/lastname causing names to be too long for Ingenico - https://phabricator.wikimedia.org/T241757 (Eileenmcnaughton) The patch is here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/644394 [18:37:19] (CR) Eileen: "@ejegg here is the explanation given" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [18:37:42] Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (DStrine) [18:39:14] (CR) Eileen: "whereas I see on Kanji" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [18:42:44] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/645132 [18:43:43] Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (DStrine) I was able to load the iframe on firefox, chrome and safari. There was a bit of a slow load but nothing out of the ordinary. I was able to make... [18:44:06] Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (MBeat33) [18:44:11] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (DStrine) p:Triage→Unbreak! [18:53:27] Wikimedia-Fundraising-Banners: [Candidate] Foundation logo - https://phabricator.wikimedia.org/T269247 (jbolorinos-ctr) @scervantes @spatton I think this one looks good, my one item of feedback is we should make the logo more salient...because it looks a little weird having full white text and then a faded l... [18:54:24] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (MBeat33) [18:56:46] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 1 - https://phabricator.wikimedia.org/T269248 (jbolorinos-ctr) Screenshot Test Results - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zd2dd176e23d1060e5d5 [18:57:43] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (MBeat33) [18:57:54] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 2 - https://phabricator.wikimedia.org/T269249 (jbolorinos-ctr) @scervantes @spatton why does this version not include Amazon Pay? [18:58:01] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Adyen: Process Adyen IPN messages in parallel (improve TY delivery times) - https://phabricator.wikimedia.org/T233251 (DStrine) p:Triage→High [18:58:34] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 2 - https://phabricator.wikimedia.org/T269249 (jbolorinos-ctr) Screenshot Test Results - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z9afab321f41ce509cbb [19:00:15] Wikimedia-Fundraising-Banners: Missing Amazon Pay option for US - https://phabricator.wikimedia.org/T269379 (jbolorinos-ctr) [19:01:41] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (DStrine) @MBeat33 and @EMartin this looks like ingenico getting a little wonky for a short period of time. We can... [19:06:27] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (MBeat33) Will do, thank you. [19:13:53] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (EMartin) @MBeat33 I was on the phone with Ingenico and they acknowledge they have an issue in flight right now.... [19:16:32] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (EMartin) Ingenico is having an intermittent problem today and it is being triaged. [19:21:24] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Minor improvements and doc following initial feedback - https://phabricator.wikimedia.org/T268954 (DStrine) p:Triage→Low [19:21:34] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Turn off import of RML during peak load - https://phabricator.wikimedia.org/T269185 (DStrine) p:Triage→Medium [19:21:53] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we speed up thank you query? - https://phabricator.wikimedia.org/T269043 (DStrine) p:Triage→Medium [19:22:03] Fundraising Sprint We all meet again, Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Add 'Stock Information' field to thank you letter - https://phabricator.wikimedia.org/T268081 (DStrine) p:Triage→High [19:29:26] AndyRussG: just gonna finish dinner then when I'm back we might catch up to talk about civi docker? [19:29:53] jgleeson|dinner2: ok cool thanks! [19:35:45] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (Eileenmcnaughton) @AndyRussG @jgleeson what strikes me reading this is that we are all on a learning curve with docker. A lot of work has gon... [19:49:21] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (Ejegg) For the ones with ct_ids, they seem to be getting the redirect to the iframe just fine, so we're not seein... [19:49:57] ejegg: did we do the final job schedule changes? [19:51:00] not yet [19:51:56] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (MBeat33) Thanks @Ejegg. After checking w/ Evelyn, we're inviting these donors to retry, hopefully they get throug... [19:58:33] eileen: thanks for the ping on the replag with the silverpop export. it's a known issue and i'm looking at a way we can hopefully handle that better. [20:04:51] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (jgleeson) Thanks, @Eileenmcnaughton. I do think it's unlikely we'd revise the approach to building images later on using the current approach... [20:05:47] dwisehaupt: ok - we can reduce the size of the export further if needed - it uploads all contacts changed within 7 days - which is pretty conservative [20:06:17] i think we won't need to do that, but thanks for letting me know that's an option. [20:14:03] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (spatton) [20:14:15] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (spatton) p:Triage→High [20:16:44] AndyRussG: is now a good time to talk about the civi docker stuff for you? [20:16:59] I'm around for another 30 minutes of so if that works for a quick intro [20:17:21] jgleeson: hey! I'm in a meeting, but I expect it'll end in a 5-10 minutes [20:17:45] is after that ok, at least just to touch base on this? [20:18:06] ahh looks like it might end early actually [20:18:08] ah sorry AndyRussG I forgot about the analytics sync up [20:18:16] sure I'll be around [20:21:07] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog: Is rsyslog dropping messages that audit processor needs? - https://phabricator.wikimedia.org/T269269 (DStrine) p:Triage→Medium [20:21:18] Fundraising Sprint We all meet again, Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Patch-For-Review: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (DStrine) p:Triage→Medium [20:22:07] Wikimedia-Fundraising-Banners: Dsk Lg - Adyen Hosted variant - https://phabricator.wikimedia.org/T269389 (jbolorinos-ctr) [20:22:29] jgleeson: hey meeting ended [20:22:41] Wikimedia-Fundraising-Banners: Dsk Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269390 (jbolorinos-ctr) [20:22:43] so anytime is cool [20:23:12] Wikimedia-Fundraising-Banners: M Lg Adyen Hosted variant - https://phabricator.wikimedia.org/T269391 (jbolorinos-ctr) [20:23:20] cool should we jump into today's standup link? [20:23:24] AndyRussG: ^ [20:23:35] jgleeson: sure! [20:23:41] Wikimedia-Fundraising-Banners: M Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269392 (jbolorinos-ctr) [20:24:07] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) Thanks Sam! Getting started on these now, will let you know if any issues are found [20:28:04] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) [20:56:16] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (EMartin) Ingenic has routed us out of their Miaimi data center they report an issue with their Internet Service P... [21:01:34] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (EMartin) see task: https://phabricator.wikimedia.org/T269378 [21:40:38] (PS1) Ejegg: Restore log level filtering [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645177 (https://phabricator.wikimedia.org/T269371) [21:41:17] (CR) jerkins-bot: [V: -1] Restore log level filtering [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645177 (https://phabricator.wikimedia.org/T269371) (owner: Ejegg) [21:43:29] Wikimedia-Fundraising-Banners: Dsk Lg - Adyen Hosted variant - https://phabricator.wikimedia.org/T269389 (jbolorinos-ctr) Screenshot Test Results - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zc1df51cea1a9e7d2f15 [21:44:47] (CR) Ejegg: [C: +2] "Nice, thanks for the explanation. So this IS a reliable filter for just those 4 byte characters which would break the DB queries." [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [21:45:04] Wikimedia-Fundraising-Banners: Dsk Lg - Adyen Hosted variant - https://phabricator.wikimedia.org/T269389 (jbolorinos-ctr) Confirmed the following MC options redirect to Special:AdyenGateway ✅ Yes, I'll give monthly ✅ No thanks, I'll give once ✅ Yes I'll donate monthly, but for a different amount Status: ✅... [21:47:46] Wikimedia-Fundraising-Banners: Dsk Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269390 (jbolorinos-ctr) Screenshot Test Results - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z5107460e3e681d51e3d [21:52:07] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) Just going to mention this suggestion you had here @spatton which would be nice to have T266183 The payments page indicates it's a monthly donation, bu... [21:52:49] Wikimedia-Fundraising-Banners: Dsk Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269390 (jbolorinos-ctr) Confirmed the following MC options redirect to Special:AdyenGateway ✅ Yes, I'll give monthly ✅ No thanks, I'll give once ✅ Yes I'll donate monthly, but for a different amount Status: ✅ PA... [21:53:28] Wikimedia-Fundraising-Banners: Dsk Lg - Adyen Hosted variant - https://phabricator.wikimedia.org/T269389 (jbolorinos-ctr) Also confirmed MC is turned OFF for Paypal [21:53:57] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) [21:54:01] Wikimedia-Fundraising-Banners: Dsk Lg - Adyen Hosted variant - https://phabricator.wikimedia.org/T269389 (jbolorinos-ctr) Open→Resolved No blockers found, signing off on QA for this variant [21:54:35] Wikimedia-Fundraising-Banners: Dsk Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269390 (jbolorinos-ctr) Also confirmed MC is turned off for the Paypal option [21:55:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, fr-donorservices: Civi: Recur fail message sent to donor w/non-failed donation - https://phabricator.wikimedia.org/T269372 (MBeat33) Amber spotted two more cases where the Status was still In Progress, like cid=29132710 or F... [22:15:15] (CR) jerkins-bot: [V: -1] dev/core#2165 Test & fix for Handle emojis less fatally where not supported [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [22:15:59] (CR) Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/644394 (https://phabricator.wikimedia.org/T261319) (owner: Eileen) [22:40:14] okok, somehow looking at https://en.wikipedia.org/wiki/UTF-8#Encoding made me less confused about that emoji patch [22:40:34] i had some doubts after +2ing whether I actually understood it [22:40:50] but i'm back to +2 land now [22:47:57] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) [22:47:59] Wikimedia-Fundraising-Banners: Dsk Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269390 (jbolorinos-ctr) Open→Resolved No blockers found. Closing as resolved. [22:48:13] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/645132 (owner: Ejegg) [22:48:44] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/645132 (owner: Ejegg) [22:50:22] !log updated standalone SmashPig IPN listener from 63dffcb11f to 3029b07004 [22:50:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:51:10] Wikimedia-Fundraising-Banners: M Lg Adyen Hosted variant - https://phabricator.wikimedia.org/T269391 (jbolorinos-ctr) Screenshot Test Results - https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zafedcf9c2a752d12d1b [22:51:12] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Smashpig: Adyen listener sending ProcessCaptureRequestJob for some recurring donations - https://phabricator.wikimedia.org/T268654 (Ejegg) a:Ejegg [22:51:45] Wikimedia-Fundraising-Banners: M Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269392 (jbolorinos-ctr) Screenshot Test Results - Mobile: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zc4ebe7d881fb0b93ee1 [22:52:59] Wikimedia-Fundraising-Banners: M Lg Adyen Hosted variant - https://phabricator.wikimedia.org/T269391 (jbolorinos-ctr) Verified Visa/Mastercard payment MC redirects to Special:AdyenGateway on the following options: ✅ Yes, I'll give monthly ✅ No thanks, I'll give once ✅ Yes I'll donate monthly, but for a diff... [22:59:55] eileen: I just pushed a change to process-control to add those different time_limit params to the qc jobs. Want to take a peek? [23:00:03] ejegg: sure [23:00:29] thanks! [23:02:16] (PS1) Ejegg: DO NOT MERGE Output module name before enabling [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/645195 [23:02:24] ejegg: so time limit 158 vs 173 [23:02:41] how long do we have to play with ? 180? [23:04:01] also - the - before the command - is that only needed when multiple commands? [23:05:52] I can see 158 is 173 -15 so that makes sense [23:06:27] Wikimedia-Fundraising-Banners: M Lg Adyen Hosted variant - https://phabricator.wikimedia.org/T269391 (jbolorinos-ctr) Verified MC is turned OFF for Paypal [23:06:38] yah, the - before the command is yaml syntax for an array [23:06:42] so only needed for multiples [23:06:56] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) [23:06:58] Wikimedia-Fundraising-Banners: M Lg Adyen Hosted variant - https://phabricator.wikimedia.org/T269391 (jbolorinos-ctr) Open→Resolved No blockers found. Closing task as Resolved [23:07:13] let's see, the 158 is what we have in the db setting, so what it's using now [23:07:20] with the 15 sec sleep [23:07:35] and it seems to have gone over that limit once yesterday [23:08:14] Wikimedia-Fundraising-Banners: M Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269392 (jbolorinos-ctr) Verified Visa/Mastercard payment MC redirects to Special:AdyenGateway on the following options: ✅ Yes, I'll give monthly ✅ No thanks, I'll give once ✅ Yes I'll donate monthly, but for a diff... [23:08:25] oh hey, looks like it's the wmf_civicrm module that takes so long to enable: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/3929/console [23:08:31] zip codes? [23:08:40] Wikimedia-Fundraising-Banners: M Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269392 (jbolorinos-ctr) Verified Paypal MC is turned OFF [23:08:56] Wikimedia-Fundraising-Banners: M Sm Adyen Hosted variant - https://phabricator.wikimedia.org/T269392 (jbolorinos-ctr) Open→Resolved No blockers found. Closing as Resolved [23:08:58] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) [23:09:35] wait no, they're part of something else now [23:10:08] Wikimedia-Fundraising-Banners: [Candidate] Adyen Hosted Redirect banners for low-level campaign - https://phabricator.wikimedia.org/T269387 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr Alright signing off on QA for these now @spatton. No blockers found, and the only issue is the slight lack o... [23:12:14] phew, though even before that it takes a good 4 minutes to get all the repos cloned and dependencies downloaded [23:24:57] hole crap, 12 minutes to enable wmf_civicrm! [23:25:03] *holy [23:27:22] ok, next step I guess is to log between each update in the _install fn [23:27:38] ooh nice ejegg with that extra logging [23:28:52] hehe those coding 101 debugging techniques [23:30:28] ejegg: it could be the triggers... [23:30:47] ejegg: gtg on the ccccccknfuvhifhiltjvkuleghcfrjgvtccdjdhetbdf [23:30:51] ccccccknfuvhtcjibcnjjkjiieiivtujjibiergurdku [23:30:52] dang [23:31:01] well you know what my yubikey means [23:31:03] hah [23:35:14] (PS1) Ejegg: DO NOT MERGE Logging for wmf_install functions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645204 [23:35:31] hah, ok, I'll deploy those process-control updates [23:45:33] ooh, so we DO enable all the civicrm extension in the middle of installing wmf_civicrm [23:45:46] and that's the update which is taking forever [23:45:55] i'm starting to thing it IS the zip codes [23:46:07] so can we omit those for CI? [23:46:37] well, I'll go one step further with that logging and try to narrow it down to the specific civi extn [23:47:08] !log adjusted timings for donations queue consumer and thank you mailer [23:47:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:20] (PS2) Ejegg: DO NOT MERGE Logging for wmf_install functions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645204 [23:57:36] hmm, 10-15 sec delay on each civi module install https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/3931/console [23:58:02] so maybe just like eileen says, it's trying to do some http communication [23:58:38] hmm - just thinking - I possibly had some things to prevent that in the buildkit that aren't happening [23:58:50] geocoder only took 14 sec to install [23:59:07] nice [23:59:37] so the ping backs could be to blame - I think setting communityMessageUrl to FALSE might be the fix there but I'll have to dig