[01:12:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2185 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 75 days 9 hours - memory use is 16.30M (peak 59.11M, 0.19% of max, fragmentation 0.97%), connected_slaves is 2, donations is 19, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1711, payments-init is 62, recurring is 0, refund is 0, unsubscribe is 0 [01:17:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3832 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 9 hours - memory use is 13.37M (peak 59.11M, 0.23% of max, fragmentation 1.31%), connected_slaves is 2, donations is 9, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2247, payments-init is 106, recurring is 0, refund is 0, unsubscribe is 1 [01:22:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3741 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 9 hours - memory use is 13.94M (peak 59.11M, 0.21% of max, fragmentation 1.19%), connected_slaves is 2, donations is 28, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1270, payments-init is 61, recurring is 0, refund is 56, unsubscribe is 1 [01:27:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3930 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 9 hours - memory use is 14.10M (peak 59.11M, 0.23% of max, fragmentation 1.27%), connected_slaves is 2, donations is 97, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2554, payments-init is 63, recurring is 0, refund is 0, unsubscribe is 1 [01:32:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3638 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 9 hours - memory use is 14.39M (peak 59.11M, 0.21% of max, fragmentation 1.10%), connected_slaves is 2, donations is 58, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 874, payments-init is 62, recurring is 0, refund is 0, unsubscribe is 1 [01:37:15] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 75 days 9 hours - memory use is 2.24M (peak 59.11M, 0.10% of max, fragmentation 3.63%), connected_slaves is 2, donations is 28, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 91, payments-init is 120, pending is 0, recurring is 0, refund is 0, unsubscribe is 2 [04:59:07] (PS6) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [05:05:41] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:29:11] (PS7) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [05:36:38] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:08:21] (PS8) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [06:15:48] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:34:58] (PS9) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [06:42:15] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:53:01] (PS10) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [07:00:15] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [08:46:26] (PS14) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [08:46:28] (PS11) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [08:53:05] (CR) jerkins-bot: [V: -1] Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [08:54:06] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [09:24:32] (PS1) Eileen: Minor test cleanup [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619149 [10:47:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3449 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 18 hours - memory use is 12.94M (peak 59.11M, 0.22% of max, fragmentation 1.38%), connected_slaves is 2, donations is 71, jobs is 0, jobs-adyen is 0, jobs-paypal is 3, payments-antifraud is 895, payments-init is 0, recurring is 343, refund is 0, unsubscribe is 6 [10:52:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3806 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 18 hours - memory use is 16.57M (peak 59.11M, 0.23% of max, fragmentation 1.09%), connected_slaves is 2, donations is 25, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-antifraud is 2068, payments-init is 72, recurring is 538, refund is 0, unsubscribe is 6 [10:57:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3619 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 14.83M (peak 59.11M, 0.22% of max, fragmentation 1.16%), connected_slaves is 2, donations is 109, jobs is 0, jobs-adyen is 0, jobs-paypal is 3, payments-antifraud is 1294, payments-init is 73, recurring is 704, refund is 0, unsubscribe is 8 [11:02:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 4085 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 19 hours - memory use is 14.96M (peak 59.11M, 0.24% of max, fragmentation 1.28%), connected_slaves is 2, donations is 59, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1641, payments-init is 89, recurring is 373, refund is 0, unsubscribe is 0 [11:07:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3836 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 16.20M (peak 59.11M, 0.25% of max, fragmentation 1.23%), connected_slaves is 2, donations is 29, jobs is 0, jobs-adyen is 0, jobs-paypal is 4, payments-antifraud is 4766, payments-init is 88, recurring is 569, refund is 0, unsubscribe is 1 [11:12:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6058 5000, pending is 3858 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 17.74M (peak 59.11M, 0.25% of max, fragmentation 1.13%), connected_slaves is 2, donations is 82, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-init is 72, recurring is 741, refund is 0, unsubscribe is 2 [11:13:41] (PS3) Eileen: Remove function moved to deduper [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618886 (https://phabricator.wikimedia.org/T256316) [11:13:41] (PS2) Eileen: Superficial cleaup [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618922 (https://phabricator.wikimedia.org/T256316) [11:13:43] (PS3) Eileen: Remove blank address test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619078 [11:13:45] (PS15) Eileen: Move cleanup on email addresses to deduper as a sort of pre-hook [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/618856 (https://phabricator.wikimedia.org/T256316) [11:17:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7303 5000, pending is 3929 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 19 hours - memory use is 20.24M (peak 59.11M, 0.27% of max, fragmentation 1.05%), connected_slaves is 2, donations is 77, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 92, recurring is 926, refund is 0, unsubscribe is 3 [11:22:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 11287 10000, pending is 3238 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 75 days 19 hours - memory use is 19.03M (peak 59.11M, 0.26% of max, fragmentation 1.09%), connected_slaves is 2, donations is 45, jobs is 0, jobs-adyen is 1, jobs-paypal is 1, payments-init is 76, recurring is 159, refund is 0, unsubscribe is 3 [11:27:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2722 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 12.63M (peak 59.11M, 0.22% of max, fragmentation 1.35%), connected_slaves is 2, donations is 75, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 4301, payments-init is 82, recurring is 352, refund is 0, unsubscribe is 3 [11:27:51] (PS12) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [11:32:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2786 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 19 hours - memory use is 13.77M (peak 59.11M, 0.20% of max, fragmentation 1.17%), connected_slaves is 2, donations is 73, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 3177, payments-init is 81, recurring is 533, refund is 0, unsubscribe is 5 [11:34:30] (CR) jerkins-bot: [V: -1] Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [11:37:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3390 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 15.08M (peak 59.11M, 0.21% of max, fragmentation 1.10%), connected_slaves is 2, donations is 54, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 1247, payments-init is 121, recurring is 721, refund is 0, unsubscribe is 5 [11:42:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2997 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 12.52M (peak 59.11M, 0.20% of max, fragmentation 1.24%), connected_slaves is 2, donations is 95, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 1069, payments-init is 72, recurring is 153, refund is 0, unsubscribe is 6 [11:43:11] (PS13) Eileen: Move email handling to the duper and address bug where email data can be lost [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/619075 (https://phabricator.wikimedia.org/T256316) [11:47:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 2390 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 19 hours - memory use is 9.94M (peak 59.11M, 0.19% of max, fragmentation 1.50%), connected_slaves is 2, donations is 85, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 2023, payments-init is 87, recurring is 316, refund is 0, unsubscribe is 7 [11:52:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3530 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 75 days 19 hours - memory use is 14.04M (peak 59.11M, 0.22% of max, fragmentation 1.23%), connected_slaves is 2, donations is 44, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 3231, payments-init is 80, recurring is 488, refund is 0, unsubscribe is 7 [11:57:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3803 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 20 hours - memory use is 14.53M (peak 59.11M, 0.24% of max, fragmentation 1.29%), connected_slaves is 2, donations is 101, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 2242, payments-init is 110, recurring is 661, refund is 0, unsubscribe is 7 [12:02:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3970 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 20 hours - memory use is 15.76M (peak 59.11M, 0.23% of max, fragmentation 1.24%), connected_slaves is 2, donations is 44, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2268, payments-init is 81, recurring is 259, refund is 0, unsubscribe is 0 [12:07:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3833 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 20 hours - memory use is 14.92M (peak 59.11M, 0.24% of max, fragmentation 1.24%), connected_slaves is 2, donations is 36, jobs is 0, jobs-adyen is 2, jobs-paypal is 1, payments-antifraud is 2954, payments-init is 33, recurring is 409, refund is 0, unsubscribe is 0 [12:12:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3897 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 20 hours - memory use is 17.80M (peak 59.11M, 0.25% of max, fragmentation 1.09%), connected_slaves is 2, donations is 67, jobs is 0, jobs-adyen is 0, jobs-paypal is 2, payments-antifraud is 4388, payments-init is 75, recurring is 547, refund is 0, unsubscribe is 3 [12:17:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 5942 5000, pending is 4108 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 20.15M (peak 59.11M, 0.26% of max, fragmentation 1.01%), connected_slaves is 2, donations is 80, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 104, recurring is 640, refund is 0, unsubscribe is 3 [12:22:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6167 5000, pending is 3559 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 17.09M (peak 59.11M, 0.25% of max, fragmentation 1.16%), connected_slaves is 2, donations is 44, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 72, recurring is 33, refund is 0, unsubscribe is 3 [12:27:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6314 5000, pending is 3987 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 19.92M (peak 59.11M, 0.25% of max, fragmentation 1.00%), connected_slaves is 2, donations is 61, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 45, recurring is 64, refund is 0, unsubscribe is 3 [12:32:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6278 5000, pending is 3823 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 15.21M (peak 59.11M, 0.25% of max, fragmentation 1.25%), connected_slaves is 2, donations is 65, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 91, recurring is 90, refund is 0, unsubscribe is 3 [12:37:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6539 5000, pending is 3806 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 20.04M (peak 59.11M, 0.25% of max, fragmentation 1.06%), connected_slaves is 2, donations is 41, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 78, recurring is 116, refund is 0, unsubscribe is 4 [12:42:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7002 5000, pending is 3520 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 18.55M (peak 59.11M, 0.25% of max, fragmentation 1.13%), connected_slaves is 2, donations is 80, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 72, recurring is 31, refund is 0, unsubscribe is 5 [12:47:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 5808 5000, pending is 3762 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 16.09M (peak 59.11M, 0.24% of max, fragmentation 1.19%), connected_slaves is 2, donations is 69, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 41, recurring is 51, refund is 0, unsubscribe is 5 [12:52:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 5444 5000, pending is 3503 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 20 hours - memory use is 16.23M (peak 59.11M, 0.24% of max, fragmentation 1.22%), connected_slaves is 2, donations is 45, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 86, recurring is 74, refund is 0, unsubscribe is 6 [12:57:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3512 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 21 hours - memory use is 15.40M (peak 59.11M, 0.23% of max, fragmentation 1.21%), connected_slaves is 2, donations is 94, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 4453, payments-init is 0, recurring is 101, refund is 0, unsubscribe is 10 [13:02:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 4187 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 21 hours - memory use is 17.82M (peak 59.11M, 0.23% of max, fragmentation 1.04%), connected_slaves is 2, donations is 41, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2654, payments-init is 89, recurring is 23, refund is 0, unsubscribe is 0 [13:07:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3664 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 12.74M (peak 59.11M, 0.22% of max, fragmentation 1.30%), connected_slaves is 2, donations is 43, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1120, payments-init is 102, recurring is 37, refund is 0, unsubscribe is 3 [13:12:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3806 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 21 hours - memory use is 13.32M (peak 59.11M, 0.22% of max, fragmentation 1.31%), connected_slaves is 2, donations is 91, jobs is 0, jobs-adyen is 2, jobs-paypal is 0, payments-antifraud is 1296, payments-init is 77, recurring is 63, refund is 0, unsubscribe is 5 [13:17:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3157 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 14.44M (peak 59.11M, 0.20% of max, fragmentation 1.15%), connected_slaves is 2, donations is 71, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 649, payments-init is 45, recurring is 90, refund is 0, unsubscribe is 7 [13:22:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3282 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 13.61M (peak 59.11M, 0.21% of max, fragmentation 1.21%), connected_slaves is 2, donations is 38, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1697, payments-init is 75, recurring is 19, refund is 0, unsubscribe is 8 [13:27:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3569 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 14.09M (peak 59.11M, 0.21% of max, fragmentation 1.27%), connected_slaves is 2, donations is 94, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1144, payments-init is 38, recurring is 28, refund is 0, unsubscribe is 9 [13:32:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3712 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 14.40M (peak 59.11M, 0.22% of max, fragmentation 1.16%), connected_slaves is 2, donations is 56, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1983, payments-init is 84, recurring is 39, refund is 0, unsubscribe is 10 [13:37:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3377 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 14.61M (peak 59.11M, 0.21% of max, fragmentation 1.18%), connected_slaves is 2, donations is 38, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1619, payments-init is 77, recurring is 50, refund is 0, unsubscribe is 12 [13:42:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3363 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 13.00M (peak 59.11M, 0.21% of max, fragmentation 1.26%), connected_slaves is 2, donations is 78, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1343, payments-init is 77, recurring is 16, refund is 0, unsubscribe is 14 [13:47:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3228 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 21 hours - memory use is 12.55M (peak 59.11M, 0.20% of max, fragmentation 1.30%), connected_slaves is 2, donations is 67, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 800, payments-init is 15, recurring is 30, refund is 0, unsubscribe is 14 [13:52:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3431 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 75 days 21 hours - memory use is 14.55M (peak 59.11M, 0.22% of max, fragmentation 1.20%), connected_slaves is 2, donations is 28, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-antifraud is 2830, payments-init is 76, recurring is 43, refund is 0, unsubscribe is 14 [13:57:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3553 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 22 hours - memory use is 14.76M (peak 59.11M, 0.22% of max, fragmentation 1.19%), connected_slaves is 2, donations is 83, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 2777, payments-init is 150, recurring is 50, refund is 0, unsubscribe is 14 [14:02:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3126 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 75 days 22 hours - memory use is 11.75M (peak 59.11M, 0.20% of max, fragmentation 1.43%), connected_slaves is 2, donations is 43, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 987, payments-init is 86, recurring is 10, refund is 0, unsubscribe is 0 [14:07:15] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3262 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 22 hours - memory use is 12.62M (peak 59.11M, 0.21% of max, fragmentation 1.30%), connected_slaves is 2, donations is 38, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1594, payments-init is 1, recurring is 25, refund is 0, unsubscribe is 1 [14:11:14] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: pending is 3262 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 75 days 22 hours - memory use is 12.62M (peak 59.11M, 0.21% of max, fragmentation 1.30%), connected_slaves is 2, donations is 38, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1594, payments-init is 1, recurring is 25, refund is 0, unsubscribe is 1 Dw [14:11:14] donation volume. [15:05:13] PROBLEM - check_disk on civi1001 is CRITICAL: DISK CRITICAL - free space: /dev 7957 MB (100% inode=99%): /run 1437 MB (90% inode=99%): / 5855 MB (10% inode=93%): /dev/shm 7974 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7974 MB (100% inode=99%): /boot 141 MB (56% inode=99%): /srv 601495 MB (72% inode=99%): /srv/archive/banner_logs 4133080 MB (54% inode=99%): /run/user/573 1594 MB (100% inode=99%): /run/u [15:05:13] (100% inode=99%) [15:10:15] PROBLEM - check_disk on civi1001 is CRITICAL: DISK CRITICAL - free space: /dev 7957 MB (100% inode=99%): /run 1445 MB (90% inode=99%): / 5720 MB (10% inode=93%): /dev/shm 7974 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7974 MB (100% inode=99%): /boot 141 MB (56% inode=99%): /srv 601492 MB (72% inode=99%): /srv/archive/banner_logs 4133062 MB (54% inode=99%): /run/user/573 1594 MB (100% inode=99%): /run/u [15:10:15] (100% inode=99%) [15:15:15] PROBLEM - check_disk on civi1001 is CRITICAL: DISK CRITICAL - free space: /dev 7957 MB (100% inode=99%): /run 1445 MB (90% inode=99%): / 5590 MB (10% inode=93%): /dev/shm 7974 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7974 MB (100% inode=99%): /boot 141 MB (56% inode=99%): /srv 601492 MB (72% inode=99%): /srv/archive/banner_logs 4133093 MB (54% inode=99%): /run/user/573 1594 MB (100% inode=99%): /run/u [15:15:15] (100% inode=99%) [15:20:15] PROBLEM - check_disk on civi1001 is CRITICAL: DISK CRITICAL - free space: /dev 7957 MB (100% inode=99%): /run 1433 MB (89% inode=99%): / 5459 MB (10% inode=93%): /dev/shm 7974 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7974 MB (100% inode=99%): /boot 141 MB (56% inode=99%): /srv 601492 MB (72% inode=99%): /srv/archive/banner_logs 4133076 MB (54% inode=99%): /run/user/573 1594 MB (100% inode=99%): /run/u [15:20:15] (100% inode=99%) [15:21:39] ACKNOWLEDGEMENT - check_disk on civi1001 is CRITICAL: DISK CRITICAL - free space: /dev 7957 MB (100% inode=99%): /run 1433 MB (89% inode=99%): / 5459 MB (10% inode=93%): /dev/shm 7974 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7974 MB (100% inode=99%): /boot 141 MB (56% inode=99%): /srv 601492 MB (72% inode=99%): /srv/archive/banner_logs 4133076 MB (54% inode=99%): /run/user/573 1594 MB (100% inode=99%) [15:21:39] 1594 MB (100% inode=99%): Dwisehaupt looking at log bloat. [16:20:17] Fundraising-Backlog, fundraising-tech-ops: Investigate increase syslog volume on civi1001 after buster upgrade - https://phabricator.wikimedia.org/T260000 (Dwisehaupt) [21:29:25] (PS1) Ejegg: Donation API checks for gateway enablement [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/619162 [21:30:41] AndyRussG: ^^^ [21:31:37] (CR) jerkins-bot: [V: -1] Donation API checks for gateway enablement [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/619162 (owner: Ejegg) [21:31:45] you can see a similar check here in the routing page: https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/special/GatewayFormChooser.php$486 [21:31:50] hmm, let's see [21:32:49] oh hmm, maybe some test setup needs updating [21:36:20] (PS2) Ejegg: Donation API checks for gateway enablement [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/619162 [21:40:38] (CR) AndyRussG: [C: +2] ":)" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/619162 (owner: Ejegg) [21:42:45] (Merged) jenkins-bot: Donation API checks for gateway enablement [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/619162 (owner: Ejegg) [21:43:53] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/619164 [21:43:59] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/619164 (owner: Ejegg) [21:44:46] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/619164 (owner: Ejegg) [21:44:51] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/619165 [21:45:11] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/619165 (owner: Ejegg) [21:49:24] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/619165 (owner: Ejegg) [21:58:46] !log updated payments-wiki from cd012f37f1 to 932aacde54 [21:58:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:20:29] Fundraising-Backlog, Operations, User-Urbanecm, User-dancy, Wiki-Setup (Create): New wiki for fundraising Thank You pages with similar config as donatewiki - https://phabricator.wikimedia.org/T259002 (DStrine) @Pcoombe or @spatton will probably want to be the "bureaucrats and admins"