[00:13:06] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [01:16:13] (PS7) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [01:22:53] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [02:52:58] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [04:13:00] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [04:14:34] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [04:15:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4957 [04:16:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5016 [04:17:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5074 [04:17:15] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5079 [04:18:15] ACKNOWLEDGEMENT - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5079 Dwisehaupt Replication lag from large alters. - The acknowledgement expires at: 2020-08-08 06:17:29. [04:18:15] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5074 Dwisehaupt Replication lag from large alters. - The acknowledgement expires at: 2020-08-08 06:17:29. [04:18:15] ACKNOWLEDGEMENT - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5016 Dwisehaupt Replication lag from large alters. - The acknowledgement expires at: 2020-08-08 06:17:29. [04:20:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5257 [04:23:53] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5257 Dwisehaupt Replication lag from large deletes - The acknowledgement expires at: 2020-08-08 06:23:26. [04:26:08] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) >>! In T257232#6367340, @Eileenmcnaughton wrote: > @Jgreen I think we should drop the triggers & tab... [04:27:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2022 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 73 days 12 hours - memory use is 12.44M (peak 59.11M, 0.22% of max, fragmentation 1.47%), connected_slaves is 2, donations is 2159, jobs is 0, jobs-adyen is 2, jobs-paypal is 109, payments-antifraud is 4584, payments-init is 1132, recurring is 0, refund is 0, unsubscribe is 19 [04:28:57] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Eileenmcnaughton) Cool - & that was 85 GB? [04:32:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2147 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 73 days 12 hours - memory use is 12.94M (peak 59.11M, 0.23% of max, fragmentation 1.45%), connected_slaves is 2, donations is 2311, jobs is 0, jobs-adyen is 2, jobs-paypal is 111, payments-antifraud is 4857, payments-init is 1212, recurring is 0, refund is 0, unsubscribe is 19 [04:32:55] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2147 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 73 days 12 hours - memory use is 12.94M (peak 59.11M, 0.23% of max, fragmentation 1.45%), connected_slaves is 2, donations is 2311, jobs is 0, jobs-adyen is 2, jobs-paypal is 111, payments-antifraud is 4857, payments-init is 1212, recurring is 0, refund is 0, unsubscribe [04:32:55] ueue backup during civi upgrade. [04:38:32] (PS8) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [04:41:09] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [04:58:07] (PS9) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [05:04:39] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:27:15] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 6355074 Threads: 1 Questions: 1347060190 Slow queries: 7680 Opens: 2597150 Flush tables: 1 Open tables: 856 Queries per second avg: 211.966 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:30:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 6355658 Threads: 1 Questions: 1343341292 Slow queries: 298269 Opens: 2911406 Flush tables: 8 Open tables: 196 Queries per second avg: 211.361 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:31:11] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 7405212 Threads: 1 Questions: 1370834760 Slow queries: 4551 Opens: 2612642 Flush tables: 1 Open tables: 979 Queries per second avg: 185.117 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:32:11] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 7404747 Threads: 1 Questions: 1370809277 Slow queries: 4451 Opens: 2612341 Flush tables: 1 Open tables: 881 Queries per second avg: 185.125 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:37:15] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 73 days 13 hours - memory use is 3.68M (peak 59.11M, 0.11% of max, fragmentation 2.67%), connected_slaves is 2, donations is 2236, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 9, payments-init is 0, pending is 1, recurring is 6, refund is 0, unsubscribe is 26 [05:45:48] (PS10) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [05:48:27] Fundraising-Backlog, fundraising-tech-ops: Plan for FY20-21 Q1 civicrm outage to upgrade civi1001 to Debian Buster - https://phabricator.wikimedia.org/T257985 (Dwisehaupt) Open→Resolved a:Dwisehaupt civi1001 has been upgraded and rebooted. process-control jobs are rolling, mail flowing out, [05:53:35] (PS1) Eileen: Remove function moved to deduper [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618886 (https://phabricator.wikimedia.org/T256316) [05:54:51] (PS1) Eileen: Remove all remaining location handlinng from wmf_civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618887 [06:07:46] (CR) jerkins-bot: [V: -1] Remove function moved to deduper [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618886 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:10:14] (PS2) Eileen: Remove function moved to deduper [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618886 (https://phabricator.wikimedia.org/T256316) [06:11:56] (CR) jerkins-bot: [V: -1] Remove all remaining location handlinng from wmf_civicrm.module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618887 (owner: Eileen) [06:51:12] (CR) AndyRussG: "> AndyRuss - check if you have triggers in your DB? (SHOW TRIGGERS) They need to be loaded up manually I think - but not if the setting lo" (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/614853 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [06:52:29] (CR) Eileen: "It's not actually a stored procedure - it's a view. We use a prepare statement to concatenate the variable into the statement" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/614853 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [06:56:06] (PS11) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [06:56:08] (PS1) Eileen: Superficial cleaup [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618922 (https://phabricator.wikimedia.org/T256316) [06:57:59] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:59:21] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [10:51:20] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [13:45:31] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Nyan cats for everyone, Fundraising Sprint Octopus hugs, and 5 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KH... [15:07:32] fr-tech looks like we're seeing some rounding weirdness with the adyen failmail [15:08:27] although the error being reported is that the job can't find a corresponding entry in the pending table [15:20:29] jgleeson: ooh, is that new with the PHP upgrade? [15:25:02] https://stackoverflow.com/questions/42981409/php7-1-json-encode-float-issue [15:25:29] Ah, I guess that's specific to json encoding [15:32:02] ah [15:41:00] Wikimedia-Fundraising-Banners: Wikipedia fundraiser box show incorrect locality - https://phabricator.wikimedia.org/T259675 (Pcoombe) Hi, thanks for reporting this. Our geolocation system uses an IP database which is normally quite accurate at the country level, but very occasionally can give the wrong locat... [15:49:46] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) >>! In T257232#6367747, @Eileenmcnaughton wrote: > Cool - & that was 85 GB? Should have been, yes.... [16:37:51] Wikimedia-Fundraising-Banners: Wikipedia fundraiser box show incorrect locality - https://phabricator.wikimedia.org/T259675 (MBeat33) We saw 8 geolocation errors related to Japan this week, @Pcoombe which is within the ordinary volume we would expect given our transaction volume. [17:03:47] (PS1) Pcoombe: Clarify banner *edit* history, not the other banner history [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/619014 [17:22:42] omg - stack overflow's random 'network questions' section at the side of that SQL answer just let me to learn the Michael Jackson holds a patent: https://worldbuilding.stackexchange.com/questions/183231/how-to-keep-my-bipedal-mech-from-falling-over-on-its-side-every-time-i-take-a-s [17:29:13] jgleeson|away: want to look at the localsettings repo when you're back? [17:29:30] ejegg: :O [17:29:30] the latest change ought to turn off MinFraud for paypal express checkout [17:31:16] patent is for the shoes that enable this move: https://youtu.be/h_D3VFfhvs4?t=423 [17:34:10] hmmm [17:34:22] fr-tech here are my show triggers in vagrant: https://paste.toolforge.org/view/5f74dd3c [17:35:29] AndyRussG: so some of those are definitely logging-related [17:35:58] I'm curious to check on prod for contacts deleted before two days ago [17:38:35] ejegg: hmmm yeah I have a bit of cooking and stuff to do and can look on prod for that in about 1.5 hrs! [17:41:49] ejegg: also thanks for looking at the triggers! [17:47:04] (PS1) Pcoombe: Show banner edit history link for everyone [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/619020 [17:52:46] AndyRussG: ok, I see 586 rows on prod where the contact_Id in silverpop export points to a contact deleted more than 2 days ago [17:53:18] still mostly recently updated ones [17:53:28] i.e. from 7/27 till 8/05 [17:54:24] hmm, so probably a change that was deployed along with the change to stop fully rebuilding that table [17:54:33] ALSO started pulling in deleted contacts [17:56:26] oh right, or it just doesn't delete them when they get deleted [17:56:53] yep yep, looks like that would need to be another step [17:57:09] that's what you were getting at I think? [18:04:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Deadlocks every 20 mins when recurring q consumer starts - https://phabricator.wikimedia.org/T259770 (Ejegg) I'm guessing this is because the recurring queue handles both payments and new/cancel subscription events, and when it's inserting payments it c... [18:24:51] (CR) Ejegg: Add modified_date to silverpop_export and filter final view on it. (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/614853 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [19:07:36] ejegg: yeah that's what I thought was happening [19:09:03] makes sense AndyRussG - filter up front for both is_deleted and modified date makes sense it would leave those un-cleaned-up [19:09:14] urf [19:09:28] grammar is hard [19:09:58] ejegg: heheh sql grammar fun fun [19:10:06] ejegg: K I'll make a bug I guess? [19:19:45] fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (Dwisehaupt) This will be of good use when we do the update to pull in this code: https://github.com/civicrm/civicrm-core/pull/17706 [19:56:33] ejegg: come to think of it I don't see anywhere where anything is ever removed from silverpop_export [19:56:56] so, as far as a persistent table goes, it'll only ever grow, in any case...? What am I missing? [19:56:56] good call AndyRussG [19:58:20] ejegg: so the way it was made persistent was just making it not deleted? maybe something's different about it 'cause it's a view? [19:59:47] AndyRussG: before, silverpop_export was deleted entirely and rebuild entirely every night [19:59:57] *rebuilt [20:00:11] ejegg: right so what changed other than just not deleting it? [20:00:12] the view has always been on top of silverpop_export [20:00:32] ahhh ok maybe I'm confusing the two [20:00:34] AndyRussG: yah, not deleting it and then only adding things that were changed recently [20:01:00] ejegg: but isn't the only adding things that were changed recently the change that isn't merged yet? [20:08:29] ejegg: ahhh ok, woops I was confusing stuff [20:08:41] so only the view is what's being filtered in the recent patch [20:09:23] but then why is that even useful, if the table it's based on is only supposed to have what's recent anyway? [20:10:00] K I think I'm gonna have to re-read the code... 8p sorry for the confusion there [20:10:53] Wikimedia-Fundraising-Banners: QA for en6C pre-test on August 5, 2020 - https://phabricator.wikimedia.org/T259297 (spatton) Open→Resolved a:spatton Thanks much @jbolorinos-ctr! [20:11:17] Wikimedia-Fundraising-Banners: QA for en6C pre-test on August 5, 2020 - https://phabricator.wikimedia.org/T259297 (spatton) a:spatton→jbolorinos-ctr [20:13:18] Wikimedia-Fundraising-Banners: QA for en6C pre-test on August 12, 2020 - https://phabricator.wikimedia.org/T259918 (spatton) [20:13:27] Wikimedia-Fundraising-Banners: QA for en6C pre-test on August 12, 2020 - https://phabricator.wikimedia.org/T259918 (spatton) p:Triage→High [20:14:55] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T259919 (spatton) [20:15:45] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T259920 (spatton) [20:16:56] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T259921 (spatton) [20:18:20] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T259922 (spatton) [20:18:34] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T259919 (spatton) p:Triage→High [20:18:39] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T259920 (spatton) p:Triage→High [20:18:41] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T259921 (spatton) p:Triage→High [20:18:45] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T259922 (spatton) p:Triage→High [20:50:54] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T259919 (jbolorinos-ctr) Screenshot Test Results - Desktop: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z307a60c1135fd0b24d2 - Variant: https://app.crossbro... [20:54:08] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T259921 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z5be54077f0379fb586b - Variant: https://app.crossbrowse... [20:56:13] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T259920 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z9cd6b79ab04348d7cda - Variant: https://app.crossbrow... [20:59:26] Wikimedia-Fundraising-Banners: 2020-08-12 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T259922 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z20b5d30f09d30b0556d - Variant: https://app.crossbrowse... [21:03:28] Wikimedia-Fundraising-Banners, fr-donorservices: India banner: extra decimals on Other amount field gets PTF riled up - https://phabricator.wikimedia.org/T259762 (jbolorinos-ctr) I believe this is due to the fact that some places '.' or ',' is used for the thousand mark, so after adding 3 chars after the... [21:07:56] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Nyan cats for everyone, Fundraising Sprint Octopus hugs, and 5 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KH... [21:51:17] Fundraising Sprint Nyan cats for everyone, Fundraising-Backlog, fundraising-tech-ops: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Eileenmcnaughton) @Jgreen that is about 200GB less than is was at the start I think (+ 50GB off the drupal D... [21:58:07] Fundraising-Backlog, fr-donorservices: Unsubscribe footer copy error in ES + email question - https://phabricator.wikimedia.org/T259935 (MBeat33) [21:58:25] (PS5) Eileen: Add modified_date to silverpop_export and filter final view on it. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/614853 (https://phabricator.wikimedia.org/T253152) [21:58:30] (CR) Eileen: Add modified_date to silverpop_export and filter final view on it. (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/614853 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [21:59:03] (PS3) Eileen: Add code comments [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/618410 (https://phabricator.wikimedia.org/T253152) [21:59:50] (CR) jerkins-bot: [V: -1] Add code comments [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/618410 (https://phabricator.wikimedia.org/T253152) (owner: Eileen) [22:01:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Deadlocks every 20 mins when recurring q consumer starts - https://phabricator.wikimedia.org/T259770 (Eileenmcnaughton) That would be consistent with it struggling to get the INSERT lock. It might be the other approach is to address the deadlock handlin... [22:02:39] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Traffic: Geoip lookup - Misidentifying country due to travelling - https://phabricator.wikimedia.org/T175691 (Tgr) This also prevents me from making a card donation (via the donation link in the sidebar menu, but I imagine click... [22:04:40] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Traffic: Geoip lookup - Misidentifying country due to travelling - https://phabricator.wikimedia.org/T175691 (Tgr) Related: {T122097} [22:05:57] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097 (Tgr) This can be quite confusing. I still have a US geoip cookie, even though I haven't been in the US for almost five months now, and have restarted the browser and the... [22:13:23] Wikimedia-Fundraising-Banners: [Enhancement] Rearrange Visa payment logos to match other payment option button - https://phabricator.wikimedia.org/T259937 (jbolorinos-ctr) [22:14:15] Wikimedia-Fundraising-Banners: [Enhancement] Rearrange Visa payment logos to match other payment option button - https://phabricator.wikimedia.org/T259937 (jbolorinos-ctr) [22:20:22] Fundraising-Backlog, fr-donorservices: Unsubscribe footer copy error in ES + email question - https://phabricator.wikimedia.org/T259935 (CDenes_WMF) Hi Michael! I unfortunately do //not// have access to unsubscribe pages. Perhaps @Pcoombe could point you in the right direction? [22:26:17] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Traffic: Geoip lookup - Misidentifying country due to travelling - https://phabricator.wikimedia.org/T175691 (Platonides) It could go both ways. If as an Hungarian with only Hungarian credit card, and temporarily visiting the US... [22:31:45] Fundraising-Backlog, fr-donorservices: Unsubscribe footer copy error in ES + email question - https://phabricator.wikimedia.org/T259935 (MBeat33) Thanks, @CDenes_WMF I'll make a note of it! [23:04:54] (PS4) Eileen: Add code comments [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/618410 (https://phabricator.wikimedia.org/T253152)