[00:00:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1498 [00:10:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1202 [00:15:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1260 [00:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1313 [00:25:18] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 2285180 Threads: 1 Questions: 78937184 Slow queries: 79438 Opens: 29080 Flush tables: 1 Open tables: 616 Queries per second avg: 34.543 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:25:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1369 [00:35:18] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 1590492 Threads: 1 Questions: 66534499 Slow queries: 113456 Opens: 25548 Flush tables: 1 Open tables: 1060 Queries per second avg: 41.832 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [01:29:11] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3620193 (Jgreen) nginx and memcache metric collectors are done [01:31:45] fundraising-tech-ops, monitoring: fundraising postfix metrics into prometheus - https://phabricator.wikimedia.org/T176293#3620194 (Jgreen) [01:32:05] Fundraising-Backlog, fundraising-tech-ops: One hour test Wed 20 Sep - https://phabricator.wikimedia.org/T176294#3620207 (Eileenmcnaughton) [01:36:18] fundraising-tech-ops, monitoring: fundraising recurring_gc* queues metrics into prometheus - https://phabricator.wikimedia.org/T176295#3620222 (Jgreen) [01:39:29] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3620235 (Jgreen) @fgiunchedi as of now we're done with ganglia! I shut down the aggregators yesterday. I'm going to leave this task open while we finish some minor cle... [01:40:42] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3620237 (Jgreen) p:Normal>Low [02:56:35] (PS1) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:02:19] (CR) jerkins-bot: [V: -1] Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:11:06] (PS2) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:14:01] (CR) jerkins-bot: [V: -1] Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:15:29] (CR) Eileen: Minor refactor on Omnimail classes. (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:16:26] (PS3) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:19:45] (CR) jerkins-bot: [V: -1] Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:22:43] (PS4) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:26:07] (CR) jerkins-bot: [V: -1] Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:28:53] (PS5) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:31:08] (PS1) Eileen: Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) [03:34:36] (CR) jerkins-bot: [V: -1] Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:35:09] (CR) jerkins-bot: [V: -1] Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [03:37:48] (PS6) Eileen: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) [03:38:08] (PS2) Eileen: Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) [03:54:44] (PS1) Eileen: Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 [03:58:09] (CR) jerkins-bot: [V: -1] Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (owner: Eileen) [05:08:41] (PS2) Eileen: Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 [05:14:43] (CR) jerkins-bot: [V: -1] Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (owner: Eileen) [05:17:33] (PS3) Eileen: Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (https://phabricator.wikimedia.org/T176255) [07:18:13] (PS1) Eileen: Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) [07:24:13] (CR) jerkins-bot: [V: -1] Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [07:30:49] (PS2) Eileen: Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) [07:32:14] Fundraising-Backlog, fundraising-tech-ops: Frack replag master thread - https://phabricator.wikimedia.org/T173472#3620406 (Jeff_G) >>! In T173472#3531968, @Marostegui wrote: > I remember that last week when troubleshooting it, I did mention the amount of writes as the first possible explanation after che... [07:34:22] (CR) jerkins-bot: [V: -1] Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [07:53:43] (PS3) Eileen: Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) [08:36:39] (PS1) Eileen: Update Omnimail recipient load to support parameterised batching. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379184 (https://phabricator.wikimedia.org/T176255) [10:01:28] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3620676 (fgiunchedi) @Jgreen that's awesome news! I think we can finally shut down ganglia for good ! [10:06:48] fundraising-tech-ops, monitoring: fundraising postfix metrics into prometheus - https://phabricator.wikimedia.org/T176293#3620194 (fgiunchedi) FWIW we'll need to do sth similar in production too, likely by parsing the logs and extracting metrics with https://github.com/google/mtail and as mentioned in {T... [12:01:25] Fundraising-Backlog, fundraising-tech-ops: Frack replag master thread - https://phabricator.wikimedia.org/T173472#3620908 (Jgreen) [12:01:26] Fundraising-Backlog, fundraising-tech-ops: Network errors and MySQL replication recovery - https://phabricator.wikimedia.org/T176266#3620906 (Jgreen) Open>Resolved I adjusted slave_net_timeout from the <=MariaDB 10.2.3 default 3600s to new default 60s and we haven't seen this since. Closing task,... [12:31:02] fundraising-tech-ops: build & deploy latest pybal for frack - https://phabricator.wikimedia.org/T176260#3620954 (Jgreen) [12:31:46] fundraising-tech-ops: build & deploy latest pybal for frack - https://phabricator.wikimedia.org/T176260#3619209 (Jgreen) >>! In T176260#3619457, @Framawiki wrote: > @Jgreen it's about #operations ? Please tags to this task :) (and a description to your account can be good too) Thanks, I added fundraising-te... [14:13:44] fundraising-tech-ops, Operations, netops: remove fundraising firewall rules related to ganglia - https://phabricator.wikimedia.org/T176319#3621226 (Jgreen) [14:15:04] fundraising-tech-ops, Operations, netops: remove fundraising firewall rules related to ganglia - https://phabricator.wikimedia.org/T176319#3621259 (Jgreen) [14:15:07] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3621258 (Jgreen) [14:15:24] fundraising-tech-ops, monitoring: fundraising postfix metrics into prometheus - https://phabricator.wikimedia.org/T176293#3620194 (Jgreen) [14:15:27] fundraising-tech-ops, Operations, netops: remove fundraising firewall rules related to ganglia - https://phabricator.wikimedia.org/T176319#3621226 (Jgreen) [14:17:57] fundraising-tech-ops, Operations: Long term storage for frack prometheus data - https://phabricator.wikimedia.org/T175738#3621268 (Jgreen) [14:38:20] (PS7) Mepps: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 [14:47:13] (CR) jerkins-bot: [V: -1] Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [14:49:30] (PS8) Mepps: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 [14:56:41] (CR) jerkins-bot: [V: -1] Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [15:07:50] Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Fix warnings in Amount staging helper - https://phabricator.wikimedia.org/T160962#3621446 (Ejegg) a:Ejegg [15:11:48] (PS1) Ejegg: adyen should be available for CA/CAD [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379243 [15:11:55] (CR) Ejegg: [C: 2] adyen should be available for CA/CAD [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379243 (owner: Ejegg) [15:14:31] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/379244 [15:15:33] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/379245 [15:16:13] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/379244 (owner: Ejegg) [15:16:31] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/379245 (owner: Ejegg) [15:24:36] (Merged) jenkins-bot: adyen should be available for CA/CAD [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379243 (owner: Ejegg) [15:26:20] Fundraising-Backlog, fundraising-tech-ops: Get Kaitlin Thaney access to CiviCRM - https://phabricator.wikimedia.org/T176177#3621500 (cwdent) I have emailed Lisa about this (does she have a Phab account?) and will generate the keys when I hear back from her, first thing we need for new frack accounts is e... [15:33:05] !log limited donations queue consumer to 200 per run [15:33:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:35:23] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/379244 (owner: Ejegg) [15:35:28] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/379245 (owner: Ejegg) [15:36:34] !log updated payments-wiki from 896f16ac380aaa6022749b395f9187ecf4bc802b to 9da84823ff4ac3071d218ec211fb59c054de9e15 [15:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:40:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1241 [15:45:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1428 [15:45:19] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1372 [15:46:13] !log limited donations queue consumer to 100 per run [15:46:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:50:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1662 [15:50:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1561 [15:55:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10255 10000, payments-antifraud is 1049 1000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 76 days 16 hours - memory use is 12.76M (peak 25.49M, 0.19% of max, fragmentation 1.24%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 486, payments-init is 731, pending is 105, recurring is 110, refund is 0, unsubscribe [15:55:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1755 [15:55:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [16:00:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 12304 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 16 hours - memory use is 13.91M (peak 25.49M, 0.22% of max, fragmentation 1.28%), connected_slaves is 2, jobs is 0, jobs-adyen is 109, jobs-paypal is 496, payments-antifraud is 622, payments-init is 545, pending is 396, recurring is 131, refund is 0, unsubscribe is [16:00:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2001 [16:00:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1917 [16:02:58] Fundraising-Backlog, fundraising-tech-ops: Get Kaitlin Thaney access to CiviCRM - https://phabricator.wikimedia.org/T176177#3621624 (CaitVirtue) @cwdent She doesn't have a phab account, but email is a good way to reach her. @LeanneS C-level approval isn't listed as required in our "how to get access t... [16:05:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 14445 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 16 hours - memory use is 15.00M (peak 25.49M, 0.24% of max, fragmentation 1.30%), connected_slaves is 2, jobs is 0, jobs-adyen is 275, jobs-paypal is 522, payments-antifraud is 187, payments-init is 234, pending is 372, recurring is 24, refund is 0, unsubscribe is 0 [16:05:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2207 [16:05:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2094 [16:10:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 16306 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 16 hours - memory use is 17.08M (peak 25.49M, 0.26% of max, fragmentation 1.22%), connected_slaves is 2, jobs is 0, jobs-adyen is 169, jobs-paypal is 524, payments-antifraud is 951, payments-init is 827, pending is 364, recurring is 60, refund is 0, unsubscribe is 0 [16:10:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2370 [16:10:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2279 [16:15:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 18564 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 16 hours - memory use is 18.50M (peak 25.49M, 0.28% of max, fragmentation 1.24%), connected_slaves is 2, jobs is 0, jobs-adyen is 71, jobs-paypal is 495, payments-antifraud is 628, payments-init is 543, pending is 409, recurring is 109, refund is 1, unsubscribe is 0 [16:15:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2396 [16:15:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2563 [16:17:24] fundraising-tech-ops: frack syslog collection running 1/2 hour behind - https://phabricator.wikimedia.org/T176326#3621652 (Ejegg) [16:18:27] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, FR-Ingenico, and 3 others: spike: investigate creating an ingenico form with no city and state - https://phabricator.wikimedia.org/T151769#3621666 (Pcoombe) Awesome, thanks @Ejegg! If this has been successf... [16:20:08] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2686 [16:20:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 20733 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 19.77M (peak 25.49M, 0.30% of max, fragmentation 1.26%), connected_slaves is 2, jobs is 0, jobs-adyen is 199, jobs-paypal is 517, payments-antifraud is 213, payments-init is 274, pending is 343, recurring is 148, refund is 1, unsubscribe is [16:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2549 [16:25:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 22934 10000, payments-antifraud is 1133 1000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 22.41M (peak 25.49M, 0.33% of max, fragmentation 1.19%), connected_slaves is 2, jobs is 0, jobs-adyen is 108, jobs-paypal is 506, payments-init is 887, pending is 425, recurring is 33, refund is 1, unsubscrib [16:25:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2819 [16:25:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2683 [16:30:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 25021 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 23.76M (peak 25.49M, 0.35% of max, fragmentation 1.20%), connected_slaves is 2, jobs is 0, jobs-adyen is 162, jobs-paypal is 517, payments-antifraud is 647, payments-init is 562, pending is 349, recurring is 60, refund is 1, unsubscribe is 0 [16:30:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2933 [16:30:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2693 [16:35:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 27216 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 25.08M (peak 25.49M, 0.37% of max, fragmentation 1.20%), connected_slaves is 2, jobs is 0, jobs-adyen is 321, jobs-paypal is 490, payments-antifraud is 235, payments-init is 299, pending is 372, recurring is 96, refund is 1, unsubscribe is 0 [16:35:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2933 [16:35:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2742 [16:40:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 29371 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 27.27M (peak 27.27M, 0.39% of max, fragmentation 1.18%), connected_slaves is 2, jobs is 0, jobs-adyen is 232, jobs-paypal is 497, payments-antifraud is 916, payments-init is 871, pending is 315, recurring is 126, refund is 2, unsubscribe is [16:40:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2985 [16:40:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2880 [16:45:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 31648 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 28.37M (peak 28.38M, 0.41% of max, fragmentation 1.20%), connected_slaves is 2, jobs is 0, jobs-adyen is 16, jobs-paypal is 519, payments-antifraud is 631, payments-init is 547, pending is 343, recurring is 16, refund is 2, unsubscribe is 0 [16:45:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3174 [16:45:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3102 [16:50:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 33598 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 29.48M (peak 30.02M, 0.43% of max, fragmentation 1.20%), connected_slaves is 2, jobs is 0, jobs-adyen is 206, jobs-paypal is 539, payments-antifraud is 236, payments-init is 259, pending is 179, recurring is 51, refund is 2, unsubscribe is 0 [16:50:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3389 [16:50:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3295 [16:52:55] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Registering test in impressions - https://phabricator.wikimedia.org/T176334#3621826 (DStrine) [16:55:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 35653 10000, payments-antifraud is 1057 1000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 31.91M (peak 31.94M, 0.45% of max, fragmentation 1.16%), connected_slaves is 2, jobs is 0, jobs-adyen is 149, jobs-paypal is 482, payments-init is 795, pending is 332, recurring is 79, refund is 2, unsubscrib [16:55:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3593 [16:55:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3513 [17:00:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 37740 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 33.79M (peak 33.87M, 0.48% of max, fragmentation 1.16%), connected_slaves is 2, jobs is 0, jobs-adyen is 158, jobs-paypal is 520, payments-antifraud is 665, payments-init is 558, pending is 429, recurring is 102, refund is 3, unsubscribe is [17:00:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3803 [17:00:19] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3715 [17:05:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 39242 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 32.25M (peak 34.85M, 0.49% of max, fragmentation 1.24%), connected_slaves is 2, jobs is 0, jobs-adyen is 250, jobs-paypal is 219, payments-antifraud is 91, payments-init is 87, pending is 132, recurring is 47, refund is 1, unsubscribe is 0 [17:05:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4010 [17:05:19] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3929 [17:06:25] !log removed limit from donation queue consumer [17:06:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:10:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 37960 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 17 hours - memory use is 30.69M (peak 34.85M, 0.49% of max, fragmentation 1.31%), connected_slaves is 2, jobs is 0, jobs-adyen is 46, jobs-paypal is 67, payments-antifraud is 120, payments-init is 93, pending is 33, recurring is 72, refund is 1, unsubscribe is 0 [17:10:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4139 [17:10:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4210 [17:15:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 36180 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 17 hours - memory use is 29.18M (peak 34.85M, 0.49% of max, fragmentation 1.36%), connected_slaves is 2, jobs is 0, jobs-adyen is 6, jobs-paypal is 24, payments-antifraud is 68, payments-init is 39, pending is 0, recurring is 96, refund is 1, unsubscribe is 0 [17:15:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4435 [17:15:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4364 [17:20:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 34078 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 18 hours - memory use is 27.63M (peak 34.85M, 0.48% of max, fragmentation 1.42%), connected_slaves is 2, jobs is 0, jobs-adyen is 22, jobs-paypal is 30, payments-antifraud is 2, payments-init is 9, pending is 0, recurring is 104, refund is 1, unsubscribe is 0 [17:20:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4636 [17:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4551 [17:20:35] Fundraising Sprint RadioActivewear, Fundraising-Backlog: Investigate PayPal-EC related FailMail - https://phabricator.wikimedia.org/T176339#3622000 (XenoRyet) [17:21:37] Fundraising Sprint RadioActivewear, Fundraising-Backlog: Investigate PayPal-EC related FailMail - https://phabricator.wikimedia.org/T176339#3622000 (XenoRyet) p:Triage>High [17:25:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 32858 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 6 keys, up 76 days 18 hours - memory use is 26.47M (peak 34.85M, 0.48% of max, fragmentation 1.47%), connected_slaves is 2, jobs is 0, jobs-adyen is 7, jobs-paypal is 19, payments-antifraud is 2, payments-init is 0, pending is 0, recurring is 11, refund is 1, unsubscribe is 0 [17:25:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4846 [17:25:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4762 [17:30:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 30756 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 18 hours - memory use is 24.93M (peak 34.85M, 0.46% of max, fragmentation 1.52%), connected_slaves is 2, jobs is 0, jobs-adyen is 1, jobs-paypal is 16, payments-antifraud is 17, payments-init is 22, pending is 14, recurring is 13, refund is 1, unsubscribe is 0 [17:30:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4971 [17:30:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5047 [17:35:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 28884 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 18 hours - memory use is 23.58M (peak 34.85M, 0.44% of max, fragmentation 1.54%), connected_slaves is 2, jobs is 0, jobs-adyen is 6, jobs-paypal is 15, payments-antifraud is 7, payments-init is 9, pending is 10, recurring is 14, refund is 1, unsubscribe is 0 [17:35:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5261 [17:35:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5178 [17:40:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 26640 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 18 hours - memory use is 22.02M (peak 34.85M, 0.45% of max, fragmentation 1.66%), connected_slaves is 2, jobs is 0, jobs-adyen is 4, jobs-paypal is 22, payments-antifraud is 17, payments-init is 30, pending is 11, recurring is 80, refund is 8, unsubscribe is 0 [17:40:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5444 [17:40:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5394 [17:45:08] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5660 [17:45:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 24769 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 18 hours - memory use is 20.52M (peak 34.85M, 0.43% of max, fragmentation 1.72%), connected_slaves is 2, jobs is 0, jobs-adyen is 4, jobs-paypal is 18, payments-antifraud is 11, payments-init is 18, pending is 0, recurring is 2, refund is 8, unsubscribe is 0 [17:45:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5608 [17:45:46] Wikimedia-Fundraising: Store link on thank you page in US/CA - https://phabricator.wikimedia.org/T176342#3622112 (Pcoombe) [17:50:08] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5856 [17:50:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 22645 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 18 hours - memory use is 19.00M (peak 34.85M, 0.39% of max, fragmentation 1.67%), connected_slaves is 2, jobs is 0, jobs-adyen is 7, jobs-paypal is 24, payments-antifraud is 11, payments-init is 14, pending is 0, recurring is 2, refund is 8, unsubscribe is 0 [17:50:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5813 [17:55:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 21627 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 18 hours - memory use is 18.03M (peak 34.85M, 0.39% of max, fragmentation 1.76%), connected_slaves is 2, jobs is 0, jobs-adyen is 7, jobs-paypal is 14, payments-antifraud is 19, payments-init is 24, pending is 0, recurring is 4, refund is 8, unsubscribe is 0 [17:55:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6070 [17:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5978 [18:00:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6169 [18:00:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 19472 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 76 days 18 hours - memory use is 16.51M (peak 34.85M, 0.39% of max, fragmentation 1.92%), connected_slaves is 2, jobs is 0, jobs-adyen is 1, jobs-paypal is 22, payments-antifraud is 13, payments-init is 21, pending is 16, recurring is 4, refund is 8, unsubscribe is 0 [18:00:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6053 [18:05:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 17502 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 76 days 18 hours - memory use is 15.01M (peak 34.85M, 0.39% of max, fragmentation 2.12%), connected_slaves is 2, jobs is 0, jobs-adyen is 7, jobs-paypal is 11, payments-antifraud is 4, payments-init is 10, pending is 0, recurring is 14, refund is 0, unsubscribe is 0 [18:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6274 [18:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6175 [18:10:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 15225 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 76 days 18 hours - memory use is 13.38M (peak 34.85M, 0.35% of max, fragmentation 2.14%), connected_slaves is 2, jobs is 0, jobs-adyen is 2, jobs-paypal is 11, payments-antifraud is 10, payments-init is 24, pending is 12, recurring is 19, refund is 0, unsubscribe is 0 [18:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6337 [18:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6292 [18:15:09] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 13290 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 76 days 18 hours - memory use is 11.95M (peak 34.85M, 0.35% of max, fragmentation 2.40%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 22, payments-antifraud is 15, payments-init is 21, pending is 8, recurring is 20, refund is 0, unsubscribe is 0 [18:15:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6517 [18:15:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6486 [18:20:18] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 11059 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 6 keys, up 76 days 19 hours - memory use is 10.28M (peak 34.85M, 0.35% of max, fragmentation 2.79%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 11, payments-antifraud is 7, payments-init is 7, pending is 0, recurring is 21, refund is 0, unsubscribe is 0 [18:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6700 [18:20:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6709 [18:25:01] (PS4) Mepps: WIP Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 [18:25:03] (PS1) Mepps: WIP approvePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 [18:25:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6892 [18:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6896 [18:26:46] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Update CiviCRM ganglia bits to point to Prometheus - https://phabricator.wikimedia.org/T171524#3622256 (Ejegg) duplicate>Open [18:30:07] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7105 [18:30:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7118 [18:30:29] (CR) Mepps: [C: 2] Don't send empty or '0' address fields to minFraud [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379120 (owner: Ejegg) [18:31:11] (PS2) Mepps: Fix empty( $thing->doStuff() ) use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379138 (owner: Ejegg) [18:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7304 [18:35:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7261 [18:40:17] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 76 days 19 hours - memory use is 4.80M (peak 34.85M, 0.28% of max, fragmentation 4.86%), connected_slaves is 2, donations is 3746, jobs is 0, jobs-adyen is 4, jobs-paypal is 11, payments-antifraud is 10, payments-init is 24, pending is 17, recurring is 5, refund is 5, unsubscribe is 3 [18:40:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7485 [18:40:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7520 [18:45:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7560 [18:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7673 [18:47:46] (PS34) Mepps: Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 [18:50:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7669 [18:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7790 [18:51:54] (CR) jerkins-bot: [V: -1] WIP Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 (owner: Mepps) [18:55:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7786 [18:55:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7907 [18:55:42] (CR) jerkins-bot: [V: -1] WIP approvePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [18:56:51] (PS5) Mepps: WIP Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 [18:57:27] (PS6) Mepps: WIP Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 [18:57:47] (CR) Mepps: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [18:58:11] ejegg ^^ hoping that's close so we can close it out! [18:58:35] cool! [18:58:37] will take a look [19:00:07] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7853 [19:00:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8019 [19:01:22] mepps looks like the getStatus tests just need phpcs fixes [19:01:32] oh, it's the commented out lines [19:02:08] ahh [19:02:20] should i return those to being skipped? [19:02:24] Maybe /* */ comments instead of per-line? [19:02:35] or just delete 'em ? [19:02:51] and we can copy the code over as needed [19:05:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8028 [19:05:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8120 [19:06:21] (PS9) Mepps: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 [19:07:39] (CR) Mepps: [C: 2] Fix empty( $thing->doStuff() ) use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379138 (owner: Ejegg) [19:10:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8292 [19:10:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8207 [19:14:41] (Merged) jenkins-bot: Fix empty( $thing->doStuff() ) use [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379138 (owner: Ejegg) [19:15:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8270 [19:15:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8399 [19:17:44] (CR) jerkins-bot: [V: -1] Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [19:20:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8278 [19:20:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8511 [19:20:44] (CR) Ejegg: [C: -1] "One function call needs rename, suggestion to log more" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [19:25:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1489 [19:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8651 [19:30:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) [19:30:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8757 [19:35:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) [19:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8793 [19:35:35] (PS35) Mepps: Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 [19:38:22] (CR) jerkins-bot: [V: -1] Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [19:39:23] (PS36) Mepps: Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 [19:40:17] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 301 Threads: 2 Questions: 9554 Slow queries: 168 Opens: 81 Flush tables: 1 Open tables: 144 Queries per second avg: 31.740 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [19:40:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8876 [19:45:07] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8933 [19:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9007 [19:55:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9171 [20:00:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9260 [20:05:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9342 [20:10:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9466 [20:15:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9555 [20:16:54] ejegg: are you able to look at some of these https://gerrit.wikimedia.org/r/#/dashboard/self- not 100% sure it matters to have them by the big email but would give a few more options [20:17:17] eileen: yep, looking at the first one now! [20:20:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9630 [20:22:20] (CR) Ejegg: [C: 2] "Let's take this for a test drive!" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [20:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9753 [20:28:23] (Merged) jenkins-bot: Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [20:30:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9848 [20:31:43] (PS7) Ejegg: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:32:02] (CR) Ejegg: [C: 2] Minor refactor on Omnimail classes. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:32:18] (PS3) Ejegg: Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:32:49] (CR) Ejegg: [C: 2] Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:33:47] eileen: just a head up, when you deploy this stuff please do a drush cc all - a file for the orphan rectifier was moved in a patch I merged a little bit ago [20:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9923 [20:36:16] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: New payments-wiki errors: We don't have an order status after doing a GET_ORDERSTATUS - https://phabricator.wikimedia.org/T176355#3622667 (Ejegg) [20:38:54] (Merged) jenkins-bot: Minor refactor on Omnimail classes. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379156 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:38:56] (Merged) jenkins-bot: Update Omnimail vendor dir to support offset on recipients. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379158 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:40:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10013 [20:40:34] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Form chooser short term fix: retry without 'gateway' param - https://phabricator.wikimedia.org/T176356#3622687 (Ejegg) [20:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10067 [20:50:08] (PS4) Ejegg: Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:50:16] (CR) Ejegg: [C: 2] Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [20:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10195 [20:50:59] (PS10) Mepps: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 [20:55:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10282 [20:56:15] who thought to put phpcs on donationinterface? so tricky when it keeps failing builds, oh wait... [20:57:30] hehe [20:59:41] (CR) Mepps: "Yay!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 (owner: Mepps) [21:00:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10366 [21:01:42] (Merged) jenkins-bot: Update Omnimail Recipient.load to support offset & limit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379161 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:04:23] (PS4) Ejegg: Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:04:36] (CR) Ejegg: [C: 2] Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:05:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10412 [21:09:06] (CR) Mepps: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [21:10:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10465 [21:10:27] (PS2) Mepps: Remove old test harness [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379123 (owner: Ejegg) [21:10:30] (CR) Mepps: [C: 2] Remove old test harness [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379123 (owner: Ejegg) [21:11:04] (Merged) jenkins-bot: Update Omnimail jobs to support storing settings about multiple jobs. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379174 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:15:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10529 [21:15:34] (PS2) Ejegg: Update Omnimail recipient load to support parameterised batching. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379184 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:16:08] (CR) Ejegg: [C: 2] "Nice way of going about the batching!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379184 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:17:45] (Merged) jenkins-bot: Remove old test harness [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379123 (owner: Ejegg) [21:19:44] (Merged) jenkins-bot: Update Omnimail recipient load to support parameterised batching. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379184 (https://phabricator.wikimedia.org/T176255) (owner: Eileen) [21:20:13] ejegg: thanks - I'll get those deployed now [21:20:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10669 [21:20:28] eileen: great! [21:20:38] you saw my note about a drush cc all? [21:20:48] ejegg: nope not yet [21:21:38] Fundraising Sprint RadioActivewear, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Investigate PayPal-EC related FailMail - https://phabricator.wikimedia.org/T176339#3622914 (XenoRyet) [21:21:58] ejegg: on one of the gerrits? [21:22:07] in IRC [21:22:26] I just merged another thing which moves an orphan-rectifier file [21:22:38] so we should drush cc all as soon as we deploy [21:22:48] ah ok [21:23:36] does it make sense to have our drupal cron clear /tmp? I kinda like having those files hang around a little bit for checking & retrying but not forever [21:24:44] eileen: wouldn't that mean it might delete the CSVs generated on import before the user could download errored rows etc? [21:24:54] Maybe delete files over a certain age? [21:25:05] yeah I guess so - perhaps one day? [21:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10748 [21:25:38] k, we'll want to add a 'download these soon warning' to that screen [21:26:54] (PS11) Ejegg: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [21:27:10] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/379416 [21:27:40] (CR) Ejegg: [C: 2] "Looks good to me locally! Still might be nice to break those big response arrays out into separate files later." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [21:28:08] ejegg: merge commit is here https://gerrit.wikimedia.org/r/#/c/379416/ - the only one I don't know if orphan slayer which i guess is the one you were saying would deploy [21:28:43] eileen: yep, it does change the name of the old file, but the rest is just adding a new drush command [21:28:55] should be fine to deploy as long as we cc all [21:29:11] (CR) Eileen: [C: 2] "If jenkins likes this then I do!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/359233 (owner: Ejegg) [21:30:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10827 [21:33:16] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/379416 (owner: Eileen) [21:34:18] (Merged) jenkins-bot: Remove unnecessary trigger refresh [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/359233 (owner: Ejegg) [21:34:23] (Merged) jenkins-bot: Adding getStatus tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378257 (owner: Mepps) [21:34:27] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/379416 (owner: Eileen) [21:35:16] (PS1) Eileen: Vendor update, omnimail-silverpop. [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/379419 [21:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10893 [21:36:54] (CR) Eileen: [C: 2] Vendor update, omnimail-silverpop. [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/379419 (owner: Eileen) [21:40:17] (Merged) jenkins-bot: Vendor update, omnimail-silverpop. [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/379419 (owner: Eileen) [21:40:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10957 [21:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11009 [21:45:41] !log update civicrm from 0fb46767f0c86b3fc85c44e228fd7c1a7ae75e46 to da833ac561f7586a0b64fbef2cdf2963bc4055c3 [21:45:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:50:07] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11068 [21:55:07] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11122 [21:58:47] (PS13) Ejegg: Optionally send more Minfraud parameters [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/275043 (https://phabricator.wikimedia.org/T128902) [21:59:40] (CR) Ejegg: "@mepps I split that response out into a separate file" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/275043 (https://phabricator.wikimedia.org/T128902) (owner: Ejegg) [22:00:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10700 [22:00:58] crap, orphan rectifier failmail [22:01:04] mepps ^^ [22:05:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10830 [22:10:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10974 [22:15:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11114 [22:17:32] Fundraising-Backlog, fundraising-tech-ops: Get Kaitlin Thaney access to CiviCRM - https://phabricator.wikimedia.org/T176177#3623008 (cwdent) @CaitVirtue thank you! From: Lisa Gruwell Date: Wed, 20 Sep 2017 10:37:57 -0700 Subject: Re: Kaitlin Thaney CiviCRM access To: Cas... [22:19:38] ejegg i see orphan-rectify being available but the log is saying it's not--could this be a cache issue? [22:19:55] mepps I just enabled orphan_slayer [22:20:01] ejegg, ahh [22:20:03] that should do it, I think [22:20:15] ejegg phew [22:20:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11242 [22:20:17] I forgot you moved the old rectifier into the new module [22:20:35] gotcha [22:22:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Engage seeing warning message when editing in Civi. - https://phabricator.wikimedia.org/T176371#3623036 (LeanneS) [22:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11381 [22:30:07] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11519 [22:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11655 [22:37:00] Fundraising-Backlog, fundraising-tech-ops: Get Kaitlin Thaney access to CiviCRM - https://phabricator.wikimedia.org/T176177#3623092 (LeanneS) @CaitVirtue I've updated the Collab page with this step. [22:37:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Engage seeing warning message when editing in Civi. - https://phabricator.wikimedia.org/T176371#3623100 (Eileenmcnaughton) Thanks for logging it - worth looking into but not screaming at me [22:40:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11789 [22:42:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Engage seeing warning message when editing in Civi. - https://phabricator.wikimedia.org/T176371#3623109 (LeanneS) Sounds good. Thanks @Eileenmcnaughton [22:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10737 [22:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10627 [22:55:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10733 [23:00:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10864 [23:03:07] XenoRyet: if you get some time for CR, these two would be great to close out: https://gerrit.wikimedia.org/r/379079 https://gerrit.wikimedia.org/r/275043 [23:03:38] Fundraising-Backlog, fundraising-tech-ops: Get Kaitlin Thaney access to CiviCRM - https://phabricator.wikimedia.org/T176177#3623146 (CaitVirtue) Thank you! [23:03:52] Next on my wishlist would be https://gerrit.wikimedia.org/r/377903 and https://gerrit.wikimedia.org/r/378356 [23:05:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10971 [23:07:20] Sure, I can take a look at those. [23:07:29] thanks, man [23:09:14] (PS1) Ejegg: Add 'ingenico' to error form definitions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379438 [23:10:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11071 [23:11:09] (CR) XenoRyet: [C: 2] Record the form variant in contribution tracking [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379079 (https://phabricator.wikimedia.org/T173869) (owner: Ejegg) [23:11:46] Fundraising-Backlog, fundraising-tech-ops: One hour test Wed 20 Sep - https://phabricator.wikimedia.org/T176294#3623164 (Eileenmcnaughton) Outcome was crazy lag but, as it turns out, crazy volume [23:15:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11216 [23:19:38] (Merged) jenkins-bot: Record the form variant in contribution tracking [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379079 (https://phabricator.wikimedia.org/T173869) (owner: Ejegg) [23:20:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11348 [23:25:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11469 [23:25:46] ejegg: I've introduced some bug getting credentials in the omnimail stuff - weirdly on live only .... [23:26:04] "Request failed: Unable to login user - username is missing" [23:26:12] (CR) XenoRyet: Optionally send more Minfraud parameters (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/275043 (https://phabricator.wikimedia.org/T128902) (owner: Ejegg) [23:26:17] digging - but I don't think it's a big issue if we are not getting atm [23:26:50] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface: Generic orphan rectifier needs to handle contribution tracking - https://phabricator.wikimedia.org/T176376#3623195 (Ejegg) [23:27:18] very weird indeed! [23:30:07] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11019 [23:32:48] (CR) XenoRyet: [C: 2] Better language fallback for unsupported locales [extensions/LandingCheck] - https://gerrit.wikimedia.org/r/378356 (https://phabricator.wikimedia.org/T174693) (owner: Ejegg) [23:34:16] (Merged) jenkins-bot: Better language fallback for unsupported locales [extensions/LandingCheck] - https://gerrit.wikimedia.org/r/378356 (https://phabricator.wikimedia.org/T174693) (owner: Ejegg) [23:34:49] (PS1) Ejegg: WIP shouldRectifyOrphan for Ingenico [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379440 (https://phabricator.wikimedia.org/T176376) [23:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11111 [23:36:40] (CR) jerkins-bot: [V: -1] WIP shouldRectifyOrphan for Ingenico [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379440 (https://phabricator.wikimedia.org/T176376) (owner: Ejegg) [23:38:50] (CR) XenoRyet: [C: 2] Retry deadlocked inserts, take two [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/377903 (https://phabricator.wikimedia.org/T118487) (owner: Ejegg) [23:40:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11242 [23:42:21] (Merged) jenkins-bot: Retry deadlocked inserts, take two [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/377903 (https://phabricator.wikimedia.org/T118487) (owner: Ejegg) [23:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11374 [23:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11508 [23:55:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11643 [23:56:00] (PS1) Ejegg: WIP: Prometheus metrics reporting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524)