[00:10:51] !log also speed the repair process-control config revision is c4ad2f5990 [00:10:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:50:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [02:00:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 241254 Threads: 6 Questions: 145571781 Slow queries: 8493 Opens: 7201 Flush tables: 1 Open tables: 200 Queries per second avg: 603.396 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [03:15:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [03:25:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 246354 Threads: 6 Questions: 150838053 Slow queries: 8704 Opens: 7210 Flush tables: 1 Open tables: 200 Queries per second avg: 612.281 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [06:00:13] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline - https://phabricator.wikimedia.org/T236834 (AndyRussG) Hi! Here are some results: - Over three samp... [14:01:30] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog: Add elements to monthly convert thank you email - https://phabricator.wikimedia.org/T237578 (Pcoombe) Open→Resolved [17:06:00] Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (KHaggard) [17:06:08] Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (KHaggard) p:Triage→High [17:30:00] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (DStrine) [17:44:00] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (RobH) p:Triage→Normal [17:44:08] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (RobH) [17:45:00] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (RobH) @jgreen: Can you provide the hostname and network info for these? I think we want to have the network ports, one each, plugged into the fasw (so the single server... [17:56:50] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (Jgreen) [17:57:25] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (Jgreen) [18:00:10] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (Jgreen) [18:03:35] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install fundraising db system - https://phabricator.wikimedia.org/T239139 (Jgreen) >>! In T239139#5690558, @RobH wrote: > @jgreen: Can you provide the hostname and network info for these? I think we want to have the network ports, one each, plu... [18:05:37] fundraising-tech-ops, Operations, ops-eqiad: (Need By: IMMEDIATE) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) [18:05:55] fundraising-tech-ops, Operations, ops-eqiad: (Need By: IMMEDIATE) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) p:Normal→High [18:06:49] fundraising-tech-ops, Operations, ops-eqiad: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) [18:13:24] fundraising-tech-ops, Operations, ops-eqiad: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) [18:42:08] (PS2) Cstone: Add width to monthly convert modal and donate monthly button. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552341 (https://phabricator.wikimedia.org/T238785) [19:02:13] PROBLEM - check_mysql on frdb1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [19:07:13] PROBLEM - check_mysql on frdb1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [19:12:13] PROBLEM - check_mysql on frdb1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [19:14:43] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (wiki_willy) @Jgreen - I gave @Jclark-ctr a heads up on this, so he'll starting working on it, when he gets in a bit lat... [19:22:14] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 541 Threads: 2 Questions: 1331940 Slow queries: 187 Opens: 577 Flush tables: 1 Open tables: 570 Queries per second avg: 2461.996 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39 [19:25:00] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (Jgreen) >>! In T239139#5690948, @wiki_willy wrote: > @Jgreen - I gave @Jclark-ctr a heads up on this, so he'll starting... [20:29:26] (PS4) Jgleeson: WIP: add contact_id filter to wmf_eoy_receipt module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/552089 (https://phabricator.wikimedia.org/T238691) [20:30:24] fr-tech be there shortly! [20:30:48] Me too, need two more minutes [20:43:41] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog: Get paypal to prevent older paypal recurring from sending inaccurate cancel email - https://phabricator.wikimedia.org/T236506 (XenoRyet) a:XenoRyet→None [20:59:13] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: AVS score in fraud filters - https://phabricator.wikimedia.org/T238352 (XenoRyet) @EMartin Sorry I didn't get to this on Friday. I think using the score country map could be a good idea. In my research on the problem it looks like AVS is... [21:02:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=267 [critical = 100] [21:05:25] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/552891 [21:06:33] (PS2) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/552891 [21:07:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=148 [critical = 100] [21:07:49] cstone: + others if you check there it's what will go out with a merge to deploy [21:09:24] it might be easier just to double check the patches i there [21:09:33] because the mailing one really should go ou [21:09:48] & the ternary one is just https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/550996/ [21:10:40] the scariest looking one is removing an unused function [21:12:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=208 [critical = 100] [21:12:30] eileen: if you feel comfortable with yours then I think they are okay to go out, XenoRyet jgleeson others options? [21:12:43] the others are mine [21:13:35] yeah hah sorry I failed at typing [21:14:05] people I tried to tag what do you think of the above [21:14:22] the scariest looking change in there is this -https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/552891/-1..2/sites/all/modules/wmf_civicrm/wmf_civicrm.module - but it just removes a never-called function & tests would have spat on that [21:17:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:18:16] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (Jclark-ctr) [21:22:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:23:15] cstone: just looking [21:27:13] ejegg|away's ternary double assignment here hurt my brain a little at first https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/548572/2/sites/all/modules/wmf_civicrm/db_switcher.inc [21:27:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:27:45] ternaries almost always hurt my brain a little. [21:27:51] :) [21:27:52] hah agree [21:28:04] they all look good to me [21:32:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:37:22] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:37:44] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (Jclark-ctr) Racked Server Connected to port 26 on fasw-c1a & fasw-c2a Entered production idrac password @Jgreen... [21:40:36] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (Jclark-ctr) [21:42:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:46:10] (CR) XenoRyet: [C: +2] Add width to monthly convert modal and donate monthly button. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552341 (https://phabricator.wikimedia.org/T238785) (owner: Cstone) [21:46:56] XenoRyet: don't review my code then - I'm a ternary fan [21:47:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:47:39] Heh, I like writing them, I just have a hard time reviewing them. ;-) [21:47:44] (CR) Eileen: [C: +2] "Discussed on IRC & no-one had concerns despite this being late in the piece from code freeze pov" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/552891 (owner: Eileen) [21:47:58] (Merged) jenkins-bot: Add width to monthly convert modal and donate monthly button. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552341 (https://phabricator.wikimedia.org/T238785) (owner: Cstone) [21:48:01] XenoRyet: I'm a total fan of the new ?? syntax [21:48:23] fr-tech, Jeff_Green, dwisehaupt I'm wondering how up to date the silverpop_* tables are on frdev1001? Are these pretty close to live like the rest? [21:48:24] cstone: I gave the merge commit a +2 based on the above [21:48:29] eileen: +1 ?? [21:48:40] Oh yea, ?? is my favorite provided I can remember to use it. [21:48:46] i like ?? too [21:48:57] jgleeson: as in the silverpop DB? [21:49:00] I never made much use of ?: [21:49:03] yeah eileen [21:49:08] that IS the live one - on frdev [21:49:18] oh... [21:49:18] ie we run the export from frdev not from live [21:49:30] ah I was guessing it was a copy like the others [21:49:32] ok then.. [21:49:35] the numbers dont add up [21:49:39] dang [21:49:42] in what way.... [21:51:22] jgleeson: we run the silverpop export on frdev db to save slow queries hanging live [21:52:16] so the unsubscribes list (silverpop_excluded) numbers are roughly the same as the ones posted by Katie but the _all_ list is out by just over 1mil [21:52:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:52:58] where did her numbers come from? [21:53:10] https://phabricator.wikimedia.org/T239126 [21:53:20] watson I guess [21:54:39] * jgleeson 's brain immediately thinks of ways to introduce the word elementary [21:56:24] jgleeson: oh yeah that sort of thing hurts brains [21:56:57] jgleeson: so you are saying there are MORE contacts in our 'all' table than in her all count? [21:57:05] Yep [21:57:22] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [21:57:47] eileen: silverpop_export_view contains 17052039 [21:57:54] as of now [21:58:15] but that report on the ticket shows ~1.6mln [21:58:29] which is what prompted katie to raise it I guess [21:58:43] I'm just trying to find the CSV [21:58:51] not sure where they live on the server [21:59:49] sorry, not ~1.6mln... that should have been ~16mln [22:00:12] I think the ticket would be in caps if it dropped to 1.6mln [22:00:38] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/552891 (owner: Eileen) [22:01:11] jgleeson: I see an upload job just possibly timed out too - I'm not 100% sure [22:01:47] the plot thickens [22:02:28] PROBLEM - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] [22:04:37] !log civicrm revision changed from 852c4a36bd to 5cf2d2713f, config revision is c4ad2f5990 [22:04:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:04:44] cstone: ^^ your deploy [22:04:50] thanks eileen !!! [22:06:07] ACKNOWLEDGEMENT - check_rsyslog_backlog on frdb1002 is CRITICAL: CRITICAL frlog1001=207 [critical = 100] Dallas_Wisehaupt remote syslog queue backlog processing. - The acknowledgement expires at: 2019-11-26 18:04:44. [22:07:41] (PS1) Cstone: Merge branch 'master' into deployment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552900 [22:08:39] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) >>! In T239139#5691375, @Jclark-ctr wrote: > Racked Server > > Connected to port 26 on fasw-c1a & fasw-c2a >... [22:08:41] actually it was fundraiser_public_data_export that timed out - I'm not sure what that is [22:09:20] I'm trying to track down the path of the process-control config file to locate the CSV path on the box [22:09:48] it looks like the process-control jobs for silverpop are in the civi1001 folder on frpm1001 [22:10:16] and I can't see a config file for process-control that gets deployed with it [22:10:22] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) forgot to remove from disabled group: ` robh@fasw-c-eqiad# show | compare [edit interfaces interface-range dis... [22:10:40] (CR) Cstone: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552900 (owner: Cstone) [22:13:40] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (Jgreen) [22:15:57] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) [22:17:22] jgleeson: so yeah it calls upload.py - but the path is ' config = process.globals.get_config()' [22:17:27] in there [22:20:01] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: AVS score in fraud filters - https://phabricator.wikimedia.org/T238352 (EMartin) As agreed in chat, we'll go with the score country map approach for the 4 countries so we don't take unnecessary risk. I'll report back on what occurs on the f... [22:20:29] yeah eileen I tracked down the config file for silverpop and have located the csv files [22:20:43] ohh where? [22:20:45] current issue is the _all_ list is 3.7gb [22:20:50] lol [22:20:53] difficult to play with lol [22:21:05] they live at /var/lib/silverpop_export [22:21:08] you could probably re-import it into a temporary table [22:22:16] on civi1001 * [22:23:30] ah [22:24:04] I guess it could be scp'd onto frdev [22:24:08] dstrine: current status on the silverpop list discrepancy is that the output katie is seeing doesn't match the source data used to popualte the files which are ultimately uploaded to silverpop [22:24:40] source data is showing a count higer by about 1mil and change [22:25:02] which roughly matches the drop she's highlighting [22:26:47] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Update English TY email text based on donor feedback - https://phabricator.wikimedia.org/T238684 (Cstone) This has been deployed and is now live. [22:32:22] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/552900 (owner: Cstone) [22:33:29] jgleeson: the file has 17052083 lines in it [22:34:24] that sounds like the rows we expected in it - but for some reason on import there are less? [22:35:24] yep feels that way [22:36:57] eileen: I just checked the process-control logs for the jobs to build the files and upload them and they both look good [22:37:13] line count as you say and the uploaded file looks correct [22:37:19] yep [22:37:52] (Abandoned) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552577 (owner: Cstone) [22:37:53] feels like something might be happening in the stages after? i.e the import to watson (wherever that happens) [22:38:22] jgleeson: here's a theory [22:38:42] there was some server work around then? What about if it was most of the way through the upload at the time? [22:38:55] & so the file was fine - just truncated [22:39:31] fundraising-tech-ops, Operations, ops-eqiad: (ESCALATED Need By: 11/26/19) rack/setup/install frdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T239139 (RobH) a:Jclark-ctr→Jgreen [22:39:49] the server work was on thursday night. so the job should have run since them. [22:40:12] but if it hasn't run since then, that's a possibility. [22:40:36] eileen: checkout /var/log/process-control/silverpop_emails_upload_files/silverpop_emails_upload_files-20191125-085053.log it looks like the upload was successful without any hicups [22:40:52] hmm [22:41:14] We need to sftp in & see if the file is still there but I think it gets deleted on upload [22:42:17] (PS1) Cstone: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 [22:42:27] 2019-11-25 09:03:46,831 INFO Ran into trouble: (, PermissionError(13, 'Permission denied'), ) [22:42:43] ok - that's from 13 hours ago [22:42:49] in that log file. [22:43:20] Isn't the phab 14 hours ago - ie. before that file [22:44:08] ah I guess that was when it was generated [22:45:52] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [22:45:58] where was the config? [22:46:43] /etc/silverpop_export.yaml on civi1001 eileen [22:48:26] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (jgleeson) It looks like the latest export contained within the file //DatabaseUpdate-20191125082239.csv// contains 17,052,039 records which... [22:50:05] eileen: dstrine I've left some notes on the ticket about what we've looked at so far and asked if we have any visibility to logs on the silverpop side. I've gotta jump off now but I'll check in again tomorrow on it, who knows tonights upload might fix it? [22:50:24] jgleeson: ok thanks [22:50:41] g'night all o/ [22:50:57] file size in sftp is 3885445068 Nov 25 09:18 DatabaseUpdate-20191125082239.csv [22:51:40] & locally 3885445068 [22:53:03] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Eileenmcnaughton) I checked the file that has been sftp'd to silverpop and the file size is 3885445068 - which is the same as our one local... [23:05:14] (PS1) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 [23:05:52] (PS2) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 [23:09:34] (CR) XenoRyet: "recheck" [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [23:09:43] (CR) jerkins-bot: [V: -1] Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 (owner: Cstone) [23:10:42] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (CCogdill_WMF) And the file we upload has no duplicate email addresses? [23:11:01] (PS3) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 [23:11:23] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Eileenmcnaughton) The code should have pre-merged those [23:12:27] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Eileenmcnaughton) The file is still present in the sftp folder - I thought they were deleted on upload? [23:14:30] (CR) jerkins-bot: [V: -1] Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 (owner: Cstone) [23:14:49] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (CCogdill_WMF) They should be... we aren't able to access the files after upload. [23:15:32] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Eileenmcnaughton) hmm I can definitely see it there now... is it accessible now? [23:16:23] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: Huge decrease in contacts from the main database update - https://phabricator.wikimedia.org/T239126 (Eileenmcnaughton) Does the upload start automatically? Could it be a timing issue - e.g the file was incompletely uploaded when the silverpo... [23:17:35] (PS1) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552924 [23:17:48] (Abandoned) Cstone: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552918 (owner: Cstone) [23:21:57] (CR) XenoRyet: "recheck" [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [23:23:11] (CR) Cstone: [C: +2] Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552924 (owner: Cstone) [23:27:34] (Merged) jenkins-bot: Update SmashPig to v0.5.11.1 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552924 (owner: Cstone) [23:28:04] (CR) XenoRyet: "recheck" [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [23:29:16] dstrine: do you know if WMF has some office licenses we can use (for testing opening UTF8 on mac excel) [23:33:20] (CR) Cstone: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [23:36:09] Fundraising Sprint A Wrinkle in Timezones, Fundraising-Backlog: AVS score in fraud filters - https://phabricator.wikimedia.org/T238352 (XenoRyet) The 'score country map' values for IE, CA, AU, and NZ have been reduced to -25. The logs look reasonable so far, and overall negative fraud scores don't appea... [23:37:07] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/552907 (owner: Cstone) [23:39:51] !log payments-wiki revision changed from e4d51fe247 to 2eb54fd6ef [23:39:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:48] (PS1) Eileen: [WIP] test adding a UTF-8 sequence. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/552929 [23:50:48] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Usual Subscripts, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog, FR-Civi-Dedupe: Middle name and initial appearing in first or last name fields causing lots of manual dedupe - https://phabricator.wikimedia.org/T230954 (... [23:57:31] eileen: yeah you just need mepps to approve [23:57:55] ah ok - cool - I guess it might take a few days too huh? [23:58:33] it's just an OIT request