[00:13:19] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11364949 (10andrea.denisse) Hi @Dwisehaupt, >>! In T367370#11361652, @Dwisehaupt wrote: > @fgiunche... [00:47:15] (03PS1) 10Eileen: Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) [00:59:20] (03PS2) 10Eileen: Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) [01:03:29] (03Abandoned) 10Eileen: Set batch status to verified_total if it matches the expected settlement [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203593 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [01:18:57] (03CR) 10CI reject: [V:04-1] Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [01:24:34] (03PS3) 10Eileen: Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) [01:40:43] (03CR) 10CI reject: [V:04-1] Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [02:51:19] (03PS1) 10Eileen: Handle invoicededuction [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1204102 (https://phabricator.wikimedia.org/T152249) [02:51:47] (03CR) 10CI reject: [V:04-1] Handle invoicededuction [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1204102 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [02:52:31] (03PS2) 10Eileen: Handle invoicededuction [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1204102 (https://phabricator.wikimedia.org/T152249) [03:49:34] (03PS1) 10Eileen: Fix a few recurring gifts that got the wrong channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204103 (https://phabricator.wikimedia.org/T406193) [03:51:55] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [04:23:00] (03PS1) 10Eileen: Insert gift data rows for subsequen recurrings without them [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204105 (https://phabricator.wikimedia.org/T406193) [04:50:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Duplicate donations made via Apple Pay and Google Pay from the WP app - https://phabricator.wikimedia.org/T407383#11365050 (10KHancock99) Another donor using Apple Pay on the Wikipedia App has also experienced a duplicate donation,... [07:46:26] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1204147 (owner: 10L10n-bot) [10:55:21] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1204147 (owner: 10L10n-bot) [13:16:45] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/1204457 (owner: 10L10n-bot) [14:17:22] (03CR) 10Ejegg: [C:03+2] "Here's the documentation on that type: https://docs.adyen.com/reporting/settlement-reconciliation/transaction-level/settlement-details-rep" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1204102 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [14:17:56] (03Merged) 10jenkins-bot: Handle invoicededuction [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1204102 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [14:18:53] (03CR) 10Ejegg: "Dang. Could we just make it fit into the WmfTransaction scheme until we abandon that? I.e. Transaction_Fees ?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203554 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [14:19:12] (03PS3) 10Eileen: Address deprecation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203560 (https://phabricator.wikimedia.org/T399782) [14:19:18] (03CR) 10Ejegg: [C:03+2] Address deprecation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203560 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [14:19:46] (03PS3) 10Eileen: Add fail information to test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203561 [14:19:48] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366762 (10tappof) [14:19:50] (03CR) 10Ejegg: [C:03+2] Add fail information to test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203561 (owner: 10Eileen) [14:36:07] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366819 (10Jgreen) [14:36:44] (03PS6) 10Eileen: Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) [14:37:03] (03CR) 10Ejegg: [C:03+2] "Hooray for proper decimal precision!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [14:37:58] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366822 (10Jgreen) [14:41:04] (03Merged) 10jenkins-bot: Address deprecation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203560 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [14:41:05] (03Merged) 10jenkins-bot: Add fail information to test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203561 (owner: 10Eileen) [14:48:48] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366880 (10Jgreen) [14:50:35] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366894 (10Jgreen) [14:52:46] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366915 (10Jgreen) [14:54:46] (03Merged) 10jenkins-bot: Fix comparison to ignore float crazies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203558 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [15:05:10] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11366982 (10Jgreen) [15:07:17] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11367000 (10Jgreen) [15:53:21] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11367306 (10Jgreen) [16:28:11] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11367462 (10Jgreen) [16:31:58] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11367496 (10Jgreen) [17:09:14] wfan I've got a few other donor portal patches I'd like to get merged before we get too many merge conflicts: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1198995 [17:09:30] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1203588 [17:09:51] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1200198 [17:09:58] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1200199/2 [17:10:10] would you mind taking a look at those? [17:15:34] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11367736 (10Jgreen) [17:27:30] (03PS2) 10Lars SG: Add updateAnnualDonors API Action This ended up being too complex to handle with an API call directly (let alone from cv), so I've added an API action to handle it. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203508 (https://phabricator.wikimedia.org/T377438) [17:28:28] (03CR) 10Lars SG: Add updateAnnualDonors API Action This ended up being too complex to handle with an API call directly (let alone from cv), so I've added an (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203508 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [18:06:24] 06Fundraising-Analysis, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, 07fr-acoustic: Discrepancy between Acoustic “Active Recurring” counts and Metabase/Civi (blocking 6ENC sends) - https://phabricator.wikimedia.org/T409958 (10AKanji-WMF) 03NEW [18:20:32] (03CR) 10Eileen: "Well ejegg ideally we would save ourselves the query too...." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203554 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [18:24:00] (03PS4) 10Eileen: Update to cope with 2 files with the same batch in a run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204097 (https://phabricator.wikimedia.org/T399782) [18:29:27] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11368058 (10Jgreen) [18:34:51] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1204643 [18:35:05] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1204643 (owner: 10Eileen) [18:35:30] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1204643 (owner: 10Eileen) [18:57:42] (03PS2) 10Eileen: Fix a few recurring gifts that got the wrong channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204103 (https://phabricator.wikimedia.org/T406193) [18:57:44] (03CR) 10Ejegg: [C:03+2] Fix a few recurring gifts that got the wrong channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204103 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [18:58:05] (03PS2) 10Eileen: Insert gift data rows for subsequen recurrings without them [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204105 (https://phabricator.wikimedia.org/T406193) [18:58:06] (03CR) 10Ejegg: [C:03+2] Insert gift data rows for subsequen recurrings without them [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204105 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [19:17:37] (03Merged) 10jenkins-bot: Fix a few recurring gifts that got the wrong channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204103 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [19:17:38] (03Merged) 10jenkins-bot: Insert gift data rows for subsequen recurrings without them [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204105 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [19:27:53] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204656 [19:30:41] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204656 (owner: 10Eileen) [19:33:42] !log config revision changed from 6877a9ba to 4abcd79b [19:33:42] disabling process control while I run update (only takes 3-4 minutes but out of caution...) [19:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:34:38] !log civicrm upgraded from a519e045 to 04649670 [19:34:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:36:29] hmm, 500 error loading API4 explorer in prod? [19:36:32] ah, worked on reload [19:37:46] ejegg: yeah the upgrade is running now - should not take long [19:38:06] finished [19:38:08] !log config revision changed from 4abcd79b to 0c731fd4 [19:38:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:58:15] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-init is 1873 1500, pending is 2493 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 33 keys, up 9 days 1 hours - memory use is 272.25M (peak 518.90M, 3.52% of max, fragmentation 1.06%), connected_slaves is 3, donations is 1555, jobs is 0, jobs-adyen is 37, jobs-dlocal is 0, jobs-paypal is 1052, payments-antifraud is 3798, recurring is 22, refund is 7, un [19:58:15] is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:58:41] hmm, queue backup [19:59:09] plenty of payments-wiki traffic [20:03:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-init is 2181 2000, pending is 3116 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 34 keys, up 9 days 1 hours - memory use is 274.77M (peak 518.90M, 3.55% of max, fragmentation 1.06%), connected_slaves is 3, donations is 1837, jobs is 0, jobs-adyen is 50, jobs-dlocal is 0, jobs-paypal is 1243, payments-antifraud is 4445, recurring is 28, refund is 10, u [20:03:17] e is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:08:15] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 5118 5000, payments-init is 2493 2000, pending is 3751 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 36 keys, up 9 days 1 hours - memory use is 277.19M (peak 518.90M, 3.58% of max, fragmentation 1.06%), connected_slaves is 3, donations is 2123, jobs is 0, jobs-adyen is 52, jobs-dlocal is 0, jobs-paypal is 1411, recurring is 29, refund is [20:08:15] scribe is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:09:17] i'm going to ack that alert since we know about it and are investigating. [20:09:52] ACKNOWLEDGEMENT - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 5118 5000, payments-init is 2493 2000, pending is 3751 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 36 keys, up 9 days 1 hours - memory use is 277.19M (peak 518.90M, 3.58% of max, fragmentation 1.06%), connected_slaves is 3, donations is 2123, jobs is 0, jobs-adyen is 52, jobs-dlocal is 0, jobs-paypal is 1411, recurring is 29, re [20:09:52] 0, unsubscribe is 0 Dwisehaupt known - investigating https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:11:06] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11368418 (10Dwisehaupt) @andrea.denisse Thanks for the feedback. We want to mimic what we currently... [20:11:53] looks like a bunch of paypal entries. [20:35:20] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Send 'decline-recurring' messages to a different queue from donations - https://phabricator.wikimedia.org/T409966 (10Ejegg) 03NEW [20:36:06] (03open) 10dwisehaupt: Upgrade civi to 6.8.0 [repos/fundraising-tech/community-civicrm] - 10https://gitlab.wikimedia.org/repos/fundraising-tech/community-civicrm/-/merge_requests/26 [20:39:17] (03PS1) 10Eileen: Update smashPig to v0.9.12.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204667 [20:42:00] (03CR) 10Ejegg: [C:03+2] Update smashPig to v0.9.12.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204667 (owner: 10Eileen) [20:44:50] eileen: ejegg: was there a change in the config/civi updates just over an hour ago that would have stopped queues or affected processing? [20:45:23] or did queues not get turned back on yet? [20:45:30] should be back on [20:45:33] let me check [20:45:51] I didn't press Y [20:45:59] ha! that'll do it. :) [20:48:16] (03CR) 10Ejegg: [C:03+2] Fix url to remove listId repetition [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202876 (https://phabricator.wikimedia.org/T379702) (owner: 10Eileen) [20:50:16] (03CR) 10Wfan: [C:03+2] Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 (owner: 10Ejegg) [20:50:18] (03CR) 10Ejegg: [C:03+2] Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 (owner: 10Ejegg) [20:50:23] (03PS2) 10Ejegg: Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 [20:50:30] (03PS3) 10Ejegg: Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 [20:50:43] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Observability-Alerting, 13Patch-For-Review: Shift frack alerting to use prometheus-alertmanager instead of icinga - https://phabricator.wikimedia.org/T367370#11368540 (10andrea.denisse) >>! In T367370#11368418, @Dwisehaupt wrote: > @andrea.denisse Thanks for... [20:51:07] (03CR) 10Ejegg: [C:03+2] Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 (owner: 10Ejegg) [20:53:16] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 25 keys, up 9 days 2 hours - memory use is 262.54M (peak 518.90M, 3.41% of max, fragmentation 1.06%), connected_slaves is 3, donations is 4424, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 39, payments-antifraud is 12, payments-init is 6, pending is 1, recurring is 2, refund is 0, unsubscribe is 0 https://icinga.wiki [20:53:16] /cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:54:34] (03CR) 10CI reject: [V:04-1] Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 (owner: 10Ejegg) [20:56:39] (03Merged) 10jenkins-bot: Reformat JS tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200198 (owner: 10Ejegg) [20:58:56] (03CR) 10Eileen: [C:03+2] Add updateAnnualDonors API Action This ended up being too complex to handle with an API call directly (let alone from cv), so I've added an [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203508 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [20:59:58] (03Merged) 10jenkins-bot: Update smashPig to v0.9.12.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204667 (owner: 10Eileen) [21:03:32] (03PS5) 10Eileen: Declare entity ref, allowing SK consent join [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202843 (https://phabricator.wikimedia.org/T379702) [21:06:40] (03CR) 10Ejegg: [C:03+2] "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 (owner: 10Ejegg) [21:06:48] (03PS2) 10Ejegg: Add test for activity tracking parameter [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200200 (https://phabricator.wikimedia.org/T407644) [21:06:53] (03Merged) 10jenkins-bot: Fix url to remove listId repetition [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202876 (https://phabricator.wikimedia.org/T379702) (owner: 10Eileen) [21:07:10] (03PS1) 10Eileen: Update to latest 6.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204672 [21:07:42] (03PS3) 10Ejegg: Fix undefined keys on portal login page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1198995 [21:08:22] (03CR) 10Wfan: [C:03+2] Fix undefined keys on portal login page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1198995 (owner: 10Ejegg) [21:08:28] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi much slower than usual! - https://phabricator.wikimedia.org/T402837#11368586 (10SHust) Flagging that Civi has been super slow again today. [21:08:41] (03Abandoned) 10Ejegg: Acoustic export:ignore changes from the last 3 minutes [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1200402 (owner: 10Ejegg) [21:20:47] (03Merged) 10jenkins-bot: Fix 'missing required prop' warnings in tests [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200199 (owner: 10Ejegg) [21:20:47] (03Merged) 10jenkins-bot: Add test for activity tracking parameter [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1200200 (https://phabricator.wikimedia.org/T407644) (owner: 10Ejegg) [21:20:49] (03Merged) 10jenkins-bot: Fix undefined keys on portal login page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1198995 (owner: 10Ejegg) [21:21:36] (03PS1) 10Eileen: Run recurring upgrade again with UPDATE ON DUPLICATE [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204675 (https://phabricator.wikimedia.org/T406193) [21:31:44] (03PS1) 10Eileen: Update SmashPig to v0.9.12.5 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1204680 [21:31:55] (03CR) 10Eileen: [C:03+2] Update SmashPig to v0.9.12.5 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1204680 (owner: 10Eileen) [21:32:19] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204681 [21:32:49] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204681 (owner: 10Eileen) [21:38:30] (03PS9) 10Wfan: Modify recurring amount [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1201197 (https://phabricator.wikimedia.org/T388755) [21:38:30] (03PS5) 10Wfan: Donor portal link to donation if restart donation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203140 (https://phabricator.wikimedia.org/T409590) [21:39:02] (03PS4) 10Wfan: Add coming soon model [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203180 (https://phabricator.wikimedia.org/T396271) [21:40:44] (03CR) 10Ejegg: [C:04-1] "That tracking field doesn't feel like the right way to do it - were you thinking it would have '1st', '2nd', etc? I think we can just loo" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1203196 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone) [21:41:32] (03CR) 10CI reject: [V:04-1] Donor portal link to donation if restart donation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203140 (https://phabricator.wikimedia.org/T409590) (owner: 10Wfan) [21:41:53] (03PS2) 10Eileen: Run recurring upgrade again with UPDATE ON DUPLICATE [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204675 (https://phabricator.wikimedia.org/T406193) [21:41:54] (03CR) 10Ejegg: [C:03+2] Run recurring upgrade again with UPDATE ON DUPLICATE [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204675 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [21:42:58] (03PS6) 10Wfan: Donor portal link to donation if restart donation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203140 (https://phabricator.wikimedia.org/T409590) [21:43:08] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07Wikimedia-production-error: BannerExistenceException: Banner by that name already exists! - https://phabricator.wikimedia.org/T286673#11368695 (10Reedy) ==== Error ==== * mwversion: 1.46.0-wmf.2 * timestamp: 2025-11-12T20:34:08.676Z * phpversio... [21:43:38] (03PS5) 10Wfan: Add coming soon model [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203180 (https://phabricator.wikimedia.org/T396271) [21:43:38] (03PS2) 10Wfan: add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) [21:44:53] (03CR) 10CI reject: [V:04-1] Donor portal link to donation if restart donation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203140 (https://phabricator.wikimedia.org/T409590) (owner: 10Wfan) [21:50:31] !log civicrm upgraded from 04649670 to 2265451b [21:50:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:52:32] (03PS2) 10Ejegg: Add trackingParams to packageFiles [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203588 (https://phabricator.wikimedia.org/T407644) [21:52:32] (03PS4) 10Ejegg: Add properties for DonorPortal amount validation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203587 (https://phabricator.wikimedia.org/T388753) [21:55:31] 06Fundraising-Analysis, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Send App `app_install_id` value to FR Tech data to improve app donation tracking - https://phabricator.wikimedia.org/T405536#11368733 (10Lars) @HNordeenWMF @SNowic... [22:08:18] (03Merged) 10jenkins-bot: Run recurring upgrade again with UPDATE ON DUPLICATE [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204675 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [22:08:48] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07Wikimedia-production-error: BannerExistenceException: Banner by that name already exists! - https://phabricator.wikimedia.org/T286673#11368792 (10Ejegg) This code seems unrelated to the change that's just gone out. Seems like it is just thrown wh... [22:16:42] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204686 [22:16:54] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204686 (owner: 10Eileen) [22:17:43] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204686 (owner: 10Eileen) [22:20:35] !log civicrm upgraded from 2265451b to c98db6d4 [22:20:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:31:41] (03CR) 10Lars SG: [C:03+2] Declare entity ref, allowing SK consent join [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202843 (https://phabricator.wikimedia.org/T379702) (owner: 10Eileen) [22:39:05] 06Fundraising-Analysis, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Send App `app_install_id` value to FR Tech data to improve app donation tracking - https://phabricator.wikimedia.org/T405536#11368875 (10HNordeenWMF) Hi @Lars! I'l... [22:47:07] (03CR) 10CI reject: [V:04-1] Declare entity ref, allowing SK consent join [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202843 (https://phabricator.wikimedia.org/T379702) (owner: 10Eileen) [22:48:58] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202843 (https://phabricator.wikimedia.org/T379702) (owner: 10Eileen) [22:53:36] (03PS1) 10Eileen: Kick start the upgrader [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204693 (https://phabricator.wikimedia.org/T406193) [22:54:49] (03CR) 10Lars SG: [C:03+2] Kick start the upgrader [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204693 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [23:10:00] (03CR) 10CI reject: [V:04-1] Kick start the upgrader [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204693 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [23:10:58] (03CR) 10Eileen: [V:03+2] "inconsistent verified is those flaky tests" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1204693 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [23:12:43] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204694 [23:12:52] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204694 (owner: 10Eileen) [23:13:46] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1204694 (owner: 10Eileen) [23:15:58] !log civicrm upgraded from c98db6d4 to 90beebc7 [23:15:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:31:40] (03PS1) 10Reedy: SpecialCentralNoticeBanners: Don't directly throw an exception in a try {} [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1204697 [23:31:40] (03PS1) 10Reedy: SpecialCentralNoticeBanners: Various cleanup [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1204698 [23:35:00] (03CR) 10CI reject: [V:04-1] SpecialCentralNoticeBanners: Various cleanup [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1204698 (owner: 10Reedy) [23:35:26] (03PS2) 10Reedy: SpecialCentralNoticeBanners: Various cleanup [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1204698 [23:57:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [23:58:11] ^^ that was me applying updates that just happened to be caught before the db could restart. [23:59:03] (03PS7) 10Wfan: Donor portal link to donation if restart donation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203140 (https://phabricator.wikimedia.org/T409590)