[00:17:04] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Conflicting fields in some Acoustic contact records - https://phabricator.wikimedia.org/T254304 (Eileenmcnaughton) @KHaggard Looking at 1764053 - this is a contact who is no longer being exported & is on the suppression list. So it's a different issue... [00:17:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Excessive modified_date triggers locking civicrm_contact - https://phabricator.wikimedia.org/T193446 (Eileenmcnaughton) @Ejegg I think we fixed this? [00:44:45] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: retrieve the contacts who have signed up for email without going through our donor system - https://phabricator.wikimedia.org/T161759 (Eileenmcnaughton) I feel like this is done [00:46:43] Fundraising-Backlog, FR-Email: Spike: Investigate Silverpop invalid rows - https://phabricator.wikimedia.org/T152448 (Eileenmcnaughton) Open→Resolved a:Eileenmcnaughton I think we should close this - if it was still an issue there will be more recent tickets [01:56:58] (PS1) Eileen: WIP - am guessing tests won't cope... [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605716 [01:58:11] (CR) jerkins-bot: [V: -1] WIP - am guessing tests won't cope... [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605716 (owner: Eileen) [02:06:05] (PS2) Eileen: WIP - am guessing tests won't cope... [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605716 [02:06:55] (CR) jerkins-bot: [V: -1] WIP - am guessing tests won't cope... [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605716 (owner: Eileen) [05:05:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:15:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:25:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:30:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:35:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:40:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:45:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:50:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [05:55:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:00:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:05:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:10:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:15:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:17:13] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Q4 19/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (Eileenmcnaughton) I did a bit of a 'literature search' (trawled phab) & foun... [06:20:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:25:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:30:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:35:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:40:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:45:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:50:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:54:57] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Prepare CN deploy 2020-06 - https://phabricator.wikimedia.org/T255476 (AndyRussG) a:AndyRussG [06:55:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [06:55:50] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/605810 (https://phabricator.wikimedia.org/T255476) [06:57:12] (CR) AndyRussG: [C: +2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/605810 (https://phabricator.wikimedia.org/T255476) (owner: AndyRussG) [06:58:53] (CR) jerkins-bot: [V: -1] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/605810 (https://phabricator.wikimedia.org/T255476) (owner: AndyRussG) [07:00:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:05:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:10:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:13:52] (CR) AndyRussG: [V: +2 C: +2] "+2 verified because the only failure is a warning about code that hasn't changed in several years." [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/605810 (https://phabricator.wikimedia.org/T255476) (owner: AndyRussG) [07:15:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:21:53] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Q4 19/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (Eileenmcnaughton) So reading up on materialized views here is what I understand... [07:22:10] (PS1) Eileen: Add modified date to the silverpop_export_staging table. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605816 (https://phabricator.wikimedia.org/T253152) [07:23:13] (CR) jerkins-bot: [V: -1] Add modified date to the silverpop_export_staging table. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/605816 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [07:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:40:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:50:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [07:55:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:00:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:05:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:15:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:20:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:25:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:30:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:35:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:40:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:45:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:50:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:53:03] Fundraising-Backlog: pgehres.bannerimpressions not updating - https://phabricator.wikimedia.org/T255544 (Pcoombe) [08:55:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:00:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:10:06] Fundraising-Backlog: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (Pcoombe) [09:10:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:15:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:20:51] (CR) Jforrester: [C: +2] ESLint: Enforce jQuery variable pattern [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/603631 (owner: Esanders) [09:21:35] (CR) Jforrester: [C: +1] Speed up BannerMessageGroup's getKeys and getDefinitions. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/598470 (https://phabricator.wikimedia.org/T221119) (owner: Nikerabbit) [09:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:50:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [09:50:13] (CR) jerkins-bot: [V: -1] ESLint: Enforce jQuery variable pattern [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/603631 (owner: Esanders) [09:55:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:00:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:05:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:15:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:20:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:25:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:30:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:35:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:40:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:45:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [10:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:15:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:50:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [11:55:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:00:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:05:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:10:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:15:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:20:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [12:25:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14564 [12:30:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11999 [12:34:16] PROBLEM - Host payments1004 is DOWN: PING CRITICAL - Packet loss = 100% [12:34:32] PROBLEM - Host pay-lvs1002 is DOWN: PING CRITICAL - Packet loss = 100% [12:35:30] RECOVERY - Host payments1004 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [12:37:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Adyen_endpoint_critical 1 [=1],Astropay-DLocal_endpoint_critical 3 [=1] [12:37:10] RECOVERY - Host pay-lvs1002 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [12:37:18] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 1 minute ago with 10 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/etc/update-motd.d/99-footer],File[/etc/motd.tail],File[/usr/local/bin/package_update_check] [12:38:34] fr-tech the puppet alerts seem to be related to a network event they're sorting out in #wikimedia-operations [12:40:14] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 1888658 Threads: 2 Questions: 62133957 Slow queries: 133953 Opens: 1263332 Flush tables: 1 Open tables: 200 Queries per second avg: 32.898 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:42:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [12:42:14] PROBLEM - check_puppetrun on frdb1003 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 6 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/etc/update-motd.d/99-footer],File[/etc/motd.tail] [12:42:24] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 6 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/etc/update-motd.d/99-footer],File[/etc/motd.tail],File[/usr/local/bin/package_update_check] [12:45:24] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 9 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/etc/rssh.conf],File[/etc/vim/vimrc.local],File[/etc/update-motd.d/99-footer],File[/etc/motd.tail] [12:47:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [12:47:14] RECOVERY - check_puppetrun on frdb1003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [12:47:18] RECOVERY - check_puppetrun on frdb1002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [12:50:18] RECOVERY - check_puppetrun on civi1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:52:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [12:57:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [13:02:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [13:06:00] Jeff_Green: yeah sorry for the impact to fr hosts, we had a slightly wrong procedure for the router upgrade, shouldn't happen again when we do the second router in ~20 minutes [13:07:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [13:09:46] cdanis: ok, yeah it knocked out payments processing for about 4 minutes with some kind of campaign up, I'm not sure yet how significant that is [13:12:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [13:16:50] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (Pcoombe) [13:17:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: All_endpoint 0 [=10] [13:22:08] RECOVERY - check_log_messages on frav1002 is OK: OK [14:13:59] Fundraising-Backlog, FR-Civi-Dedupe: Estimate amount of not-yet Deduped Civi Records - https://phabricator.wikimedia.org/T247343 (MBeat33) Open→Resolved a:MBeat33 **"answer is 800k+"** Thank you so much, @Eileenmcnaughton this is great information! email: Eileen McNaughton 12:00 AM (10... [14:15:54] Jeff_Green: okay, final router reboot is happening now and looks like no pfw impact this time, sorry again :( [14:17:36] cdanis: cool, thank you! [14:18:57] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (Jgreen) Open→Resolved p:Triage→Medium a:Jgreen Done! It should be on the list of available sources in Superset. [14:54:15] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog: Flowchart all incoming data from non fundraising sources - https://phabricator.wikimedia.org/T254783 (mepps) By non-fundraising sources @dstrine, do you mean HEP data, minfraud, TS, etc? [15:26:05] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (EYener) Thanks, Jeff! [15:35:52] Fundraising-Backlog, fundraising-tech-ops: Upgrade mariaDB on frdb1003 to >= 10.2 - https://phabricator.wikimedia.org/T255066 (Jgreen) [15:36:00] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (EYener) Hmm. @Jgreen , I'm not seeing it there. I added a table, so the DB is not empty, if that helps. [15:37:59] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog: Flowchart all incoming data from non fundraising sources - https://phabricator.wikimedia.org/T254783 (DStrine) I'm referring to any WMF data that is being pulled in or referenced in some way in our systems, databases etc. [15:39:34] Fundraising-Backlog: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (DStrine) p:Triage→Unbreak! [15:42:37] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (DStrine) [15:45:06] Fundraising Sprint Lazy Loading Life, fundraising-tech-ops: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (Jgreen) a:Jgreen [15:49:15] Fundraising Sprint Lazy Loading Life, fundraising-tech-ops: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (Jgreen) This was a replication timeout due to an extremely longrunning query. I restarted replication at about 12:30 UTC and it caught up quickly after that. There sho... [15:58:00] Fundraising Sprint Lazy Loading Life, fundraising-tech-ops: Tables on frdev1001 not updating - https://phabricator.wikimedia.org/T255544 (Jgreen) Open→Resolved [16:16:28] Fundraising-Backlog, fundraising-tech-ops: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster - https://phabricator.wikimedia.org/T255456 (Jgreen) [16:18:36] Fundraising-Backlog, fundraising-tech-ops: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster - https://phabricator.wikimedia.org/T255456 (Jgreen) [16:22:52] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (Jgreen) >>! In T255559#6228407, @EYener wrote: > Hmm. @Jgreen , I'm not seeing it there. I added a table, so the DB is not empty, if that helps. Fixed. I had added it to the superset config but fo... [16:42:06] (CR) Cstone: [C: +2] "Looks good, let lowercase through." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/605698 (https://phabricator.wikimedia.org/T255171) (owner: Ejegg) [16:46:15] (Merged) jenkins-bot: Allow lowercase letters in India PAN [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/605698 (https://phabricator.wikimedia.org/T255171) (owner: Ejegg) [17:05:00] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising Sprint YAMLton, the Musical, Fundraising-Backlog: Special Characters do not export well from civi - https://phabricator.wikimedia.org/T237564 (LeanneS) @Eileenmcnaughton Here is an example: 2155287 [17:32:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Dedupe suggestion: track down OG contact records for "new" donors - https://phabricator.wikimedia.org/T238156 (KHaggard) @DStrine Sorry for the delay in getting back to you. I checked civi again and it seems like my two contact records did not merge at... [17:36:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Dedupe suggestion: track down OG contact records for "new" donors - https://phabricator.wikimedia.org/T238156 (CCogdill_WMF) I can't think of anything else! If fr-tech thinks this still works, can y'all perform your own test or jump on a screenshare wit... [18:06:15] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-AutoTY-Email: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) I spent some time... [18:11:22] hey jgleeson mepps quick question, any special reason fgetcsv was not used in the matching gifts API to parse the file? [18:11:25] also helloooooo [18:12:45] AndyRussG: can you link to the code you're looking at so I can refresh pls? [18:14:14] jgleeson: yee one sec [18:15:34] jgleeson: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/DonationInterface/+/master/gateway_common/employerSearch.api.php#78 [18:15:51] I think we already parse the whole file there with the call to fgetcsv? [18:15:59] maybe I'm missing something though? [18:16:55] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-AutoTY-Email: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) So one thing of no... [18:17:29] Ah no it's just the first line I guess [18:18:00] yeah AndyRussG it looks like that was the most efficient way to validate that the file actually was a csv [18:18:17] file(..) then reads the whole thing into an array [18:18:23] jgleeson: ah ok got it [18:19:13] jgleeson: but so there was no special issue with fgetcsv as a possible way to parse the file, also, correct? [18:19:51] AndyRussG: $employers = array_map( 'str_getcsv', $employersFile ); [18:20:02] that's the current thingy parsing the file [18:20:38] I guess we don't want to parse the file upfront in case it's either not a valid csv or it's something we don't want to open [18:21:34] jgleeson: okok gotcha [18:22:25] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-AutoTY-Email: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) Steps to add this... [18:25:32] Fundraising-Backlog, fundraising-tech-ops: New DB for Superset - https://phabricator.wikimedia.org/T255559 (EYener) Thank you! It's working great now. [18:26:11] (PS1) Mepps: Adding tax deductibility language for English, Spanish, and Dutch templates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605985 (https://phabricator.wikimedia.org/T242001) [18:31:52] Fundraising-Backlog: Close filehandle in MW API code - https://phabricator.wikimedia.org/T255593 (AndyRussG) [18:32:13] Fundraising-Backlog: Close filehandle in MW API code - https://phabricator.wikimedia.org/T255593 (AndyRussG) [18:34:02] (CR) jerkins-bot: [V: -1] Adding tax deductibility language for English, Spanish, and Dutch templates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605985 (https://phabricator.wikimedia.org/T242001) (owner: Mepps) [18:48:29] Wikimedia-Fundraising-Banners: Banner remains visible after user enters a valid email and clicks submit - https://phabricator.wikimedia.org/T253147 (jbolorinos-ctr) Hey @EWilfong_WMF, I'm not able to get the link you provided to load, would you happen to have an updated link for the most recent changes on t... [18:51:45] Wikimedia-Fundraising-Banners: Banner remains visible after user enters a valid email and clicks submit - https://phabricator.wikimedia.org/T253147 (jbolorinos-ctr) [18:53:11] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) The test for eoysummary h... [18:55:13] (PS2) Mepps: Adding tax deductibility language for English, Spanish, and Dutch templates [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605985 (https://phabricator.wikimedia.org/T242001) [18:56:03] Wikimedia-Fundraising-Banners: [Enhancement] Auto-select email input field after user clicks RML link - https://phabricator.wikimedia.org/T255598 (jbolorinos-ctr) [19:03:05] Wikimedia-Fundraising-Banners: [Enhancement] Create a copy test for RML that includes implementation intentions - https://phabricator.wikimedia.org/T255599 (jbolorinos-ctr) [19:03:56] Wikimedia-Fundraising-Banners: [Enhancement] Update "Submit" button to - https://phabricator.wikimedia.org/T255600 (jbolorinos-ctr) [19:05:34] Wikimedia-Fundraising-Banners: [Enhancement] Update "Submit" button to "Continue" or something more appropriate - https://phabricator.wikimedia.org/T255600 (jbolorinos-ctr) [19:05:53] Wikimedia-Fundraising-Banners: Banner remains visible after user enters a valid email and clicks submit - https://phabricator.wikimedia.org/T253147 (EWilfong_WMF) The link in my comment loads for me. Can't think of why it would not load for you unless there's a client-side issue. [19:10:03] Wikimedia-Fundraising-Banners: Unnecessary Back button included in nag - https://phabricator.wikimedia.org/T255602 (jbolorinos-ctr) [19:12:43] Wikimedia-Fundraising-Banners: Banner remains visible after user enters a valid email and clicks submit - https://phabricator.wikimedia.org/T253147 (jbolorinos-ctr) Weird, yeah it loads for me now too...might just be having one of those days I guess. Thanks @EWilfong_WMF! [19:15:14] Wikimedia-Fundraising-Banners: QA for 'RML options' banner - https://phabricator.wikimedia.org/T253054 (jbolorinos-ctr) Hey @EWilfong_WMF I've added several bugs and enhancement suggestions as subtasks to this task, just let me know if you have any questions about any of these! [19:53:14] fundraising-tech-ops: improve banner logger and kafkatee monitoring - https://phabricator.wikimedia.org/T254382 (Jgreen) [19:54:10] fundraising-tech-ops: improve banner logger and kafkatee monitoring - https://phabricator.wikimedia.org/T254382 (Jgreen) * added check_kafkatee to frack nagios/icinga, monitoring broker state in kafkatee status logs [19:54:24] fundraising-tech-ops: improve banner logger and kafkatee monitoring - https://phabricator.wikimedia.org/T254382 (Jgreen) Open→Resolved [19:54:28] fundraising-tech-ops, observability, Epic: [Epic] overhaul fundraising cluster monitoring - https://phabricator.wikimedia.org/T91508 (Jgreen) [19:55:34] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) a:mepps [19:55:54] fundraising-tech-ops: Reconfigure fundraising public-reporting server role to fundraising-data - https://phabricator.wikimedia.org/T255435 (DStrine) [19:55:56] fundraising-tech-ops: fundraising tech operations GDPR tracking task - https://phabricator.wikimedia.org/T226749 (DStrine) [19:57:03] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Contact Source job failed this morning 6/10/2020 - https://phabricator.wikimedia.org/T255033 (mepps) Open→Resolved a:mepps [19:57:05] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Contact Source job failed this morning 6/10/2020 - https://phabricator.wikimedia.org/T255033 (DStrine) a:mepps→None [19:57:18] Wikimedia-Fundraising-Banners: [Enhancement] Update "Submit" button to "Continue" or something more appropriate - https://phabricator.wikimedia.org/T255600 (EWilfong_WMF) We left this as "Submit" to keep it consistent with the control that does not have a second step. The idea being we want the initial sign... [19:58:12] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog: Investigate drupal upgrade - https://phabricator.wikimedia.org/T251914 (DStrine) [20:01:45] Fundraising-Backlog: Rebellious colon displaying within ingenico iframe - https://phabricator.wikimedia.org/T255477 (DStrine) [20:05:39] Fundraising Sprint Just space out, Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create a new CiviCRM extension with an API action to fetch data from SSBI... - https://phabricator.wikimedia.org/T249922 [20:05:41] Fundraising-Backlog, Epic: Epic: matching gifts HEP integration (was employer field V2) - https://phabricator.wikimedia.org/T228765 (Ejegg) [20:15:58] Wikimedia-Fundraising-Banners: QA for 'RML options' banner - https://phabricator.wikimedia.org/T253054 (EWilfong_WMF) Just noting on this main task that this is the current candidate being QA'ed: https://en.wikipedia.org/wiki/Wikipedia?banner=Trilogy_mlWW_dsk_p1_lg_RMLTiming1&country=US [20:21:07] Wikimedia-Fundraising-Banners: [Enhancement] Auto-select email input field after user clicks RML link - https://phabricator.wikimedia.org/T255598 (EWilfong_WMF) I am seeing the email input being auto-selected (put in focus) when the user clicks the RML link. This is the code in this banner and the control:... [20:29:53] Fundraising-Backlog, fundraising-tech-ops: payments-listener server high availability - https://phabricator.wikimedia.org/T170318 (Jgreen) [20:39:57] Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint X-rays, Fundraising Sprint YAMLton, the Musical, Fundraising-Backlog: Special Characters do not export well from civi - https://phabricator.wikimedia.org/T237564 (Eileenmcnaughton) OK last time around I could replicate the issue on Exc... [20:40:16] fr-tech this is some good news for a change https://www.bbc.co.uk/news/health-53061281 [20:55:41] Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising Sprint Kavascript: the chill new language, and 4 others: generate generic email for failed recurring donations - https://phabricator.wikimedia.org/T230064 (... [20:57:41] Fundraising Sprint Kavascript: the chill new language, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (mepps) @Eileenmcnaughton and I a... [20:59:52] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog: Investigate drupal upgrade - https://phabricator.wikimedia.org/T251914 (Eileenmcnaughton) I have doubts about the usefulness of wmf_campaign but digging it looks like MG used it as recently as 2019 select campaign_key FROM wmf_campaigns_campaig... [21:11:04] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Conflicting fields in some Acoustic contact records - https://phabricator.wikimedia.org/T254304 (KHaggard) Ok, it's good to know that cid 1764053 will never be mailed to. Would it be helpful if I included the CIDs from the other 19 contacts from my quer... [21:18:50] (PS1) AndyRussG: Add object caching for API for employer data [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/606016 (https://phabricator.wikimedia.org/T255593) [21:20:56] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Patch-For-Review: Close filehandle in MW API code - https://phabricator.wikimedia.org/T255593 (AndyRussG) [21:21:07] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Patch-For-Review: Close filehandle in MW API code - https://phabricator.wikimedia.org/T255593 (AndyRussG) a:AndyRussG [21:21:11] (CR) jerkins-bot: [V: -1] Add object caching for API for employer data [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/606016 (https://phabricator.wikimedia.org/T255593) (owner: AndyRussG) [21:26:38] (PS2) AndyRussG: Add object caching for API for employer data [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/606016 (https://phabricator.wikimedia.org/T255593) [21:29:01] (CR) jerkins-bot: [V: -1] Add object caching for API for employer data [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/606016 (https://phabricator.wikimedia.org/T255593) (owner: AndyRussG) [21:37:23] (CR) Ejegg: [C: -1] "Oops, this is a silly way to do it" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605340 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [21:37:56] Wikimedia-Fundraising-Banners: Unnecessary Back button included in nag - https://phabricator.wikimedia.org/T255602 (EWilfong_WMF) Open→Resolved Agreed with this assessment that the "back" button is unnecessary/confusing on the RML options step in the nag. I've modified the banner to hide that on thi... [21:37:58] Wikimedia-Fundraising-Banners: QA for 'RML options' banner - https://phabricator.wikimedia.org/T253054 (EWilfong_WMF) [21:44:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Conflicting fields in some Acoustic contact records - https://phabricator.wikimedia.org/T254304 (Eileenmcnaughton) @KHaggard yes - please provide the others - I've spotted something in the silverpop script that looks a bit wrong to me & I'm wondering i... [22:03:29] ejegg|food: are you joining tech talk or off lunching? [22:03:54] oops, joining now [22:04:01] forgot to take the |food flag down [22:43:24] (PS1) Eileen: [WIP] Limit the email update [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/606026 (https://phabricator.wikimedia.org/T253152) [22:43:26] (PS1) Eileen: Random WIP [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/606027 [22:44:27] (CR) jerkins-bot: [V: -1] [WIP] Limit the email update [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/606026 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [22:44:33] (CR) jerkins-bot: [V: -1] Random WIP [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/606027 (owner: Eileen) [22:55:35] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (Eileenmcnaughton) @KHaggard just noting on the removals that - latest_curre... [23:23:31] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (KHaggard) Thanks @Eileenmcnaughton for surfacing those phab tasks and lettin... [23:31:17] AndyRussG: did you want a hand debugging those test failures on the frontend caching? [23:31:44] ejegg: hey thanks! Mmmm I think I have it mostly under contro [23:31:49] *control [23:32:09] ejegg: one question I do have, is setup run once per test class, or once per test? [23:32:23] I think it's once per test but now I'm not sure [23:35:11] setup is once per test, yep [23:35:23] Wikimedia-Fundraising-Banners: [BETA] QA Touch Target Size Guidelines variant - https://phabricator.wikimedia.org/T255636 (jbolorinos-ctr) [23:35:24] there's another method you can use to run things one per class if you want [23:35:52] ejegg: ah okok yeah once per test is what I need [23:36:01] I'm able to run them locally on vagrant, so all good there :) [23:43:58] Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (Eileenmcnaughton) @KHaggard given we need to co-ordinate the timing for dele... [23:53:44] ejegg: any tips on getting more informative test failure results? It just says: --- Expected [23:53:46] +++ Actual [23:53:48] @@ @@ [23:53:50] Array ( [23:53:52] - 0 => Array (...) [23:53:54] + 1 => Array (...) [23:55:11] AndyRussG: let's see what assertion you're using [23:55:36] there's an assertArrayEquals that knows a bit more about arrays