[00:00:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2071 [00:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2023 [00:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2228 [00:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2182 [00:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2385 [00:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2332 [00:15:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2492 [00:15:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2539 [00:18:15] Fundraising-Backlog: Move tracking of silverpop out of settings table - https://phabricator.wikimedia.org/T177451#3659301 (Eileenmcnaughton) [00:20:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2697 [00:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2651 [00:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2745 [00:25:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2848 [00:30:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2945 [00:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2811 [00:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2896 [00:35:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3025 [00:40:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3131 [00:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2990 [00:45:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3242 [00:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3153 [00:50:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3396 [00:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3317 [00:55:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3542 [00:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3479 [01:00:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3707 [01:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3647 [01:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3866 [01:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3809 [01:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3984 [01:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4044 [01:15:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4207 [01:15:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4155 [01:20:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4363 [01:20:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4332 [01:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4498 [01:25:13] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4525 [01:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4679 [01:30:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4693 [01:35:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4849 [01:35:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4862 [01:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5040 [01:40:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5022 [01:45:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5200 [01:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5222 [01:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5383 [01:50:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5345 [01:55:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5483 [01:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5494 [02:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5624 [02:00:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5629 [02:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5807 [02:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5799 [02:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5977 [02:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5980 [02:15:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6160 [02:15:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6156 [02:20:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6336 [02:20:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6335 [02:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6508 [02:25:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6514 [02:30:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6692 [02:30:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6682 [02:35:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6867 [02:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6849 [02:40:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7045 [02:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7011 [02:45:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7226 [02:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7151 [02:50:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7247 [02:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7182 [02:55:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7226 [02:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7230 [03:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7048 [03:00:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7035 [03:05:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7113 [03:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7061 [03:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7075 [03:10:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7176 [03:15:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7220 [03:15:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7087 [03:20:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7271 [03:20:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7080 [03:25:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7319 [03:25:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7097 [03:30:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6898 [03:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7364 [03:35:09] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6238 [03:35:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7075 [03:40:19] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6151 [03:40:19] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6543 [03:45:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6112 [03:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6538 [03:50:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6009 [03:50:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6512 [03:55:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6049 [03:55:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6522 [04:00:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5857 [04:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6539 [04:05:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5780 [04:05:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6584 [04:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6349 [04:10:13] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5794 [04:13:26] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5794 Casey Dentinger Collecting replag data [04:14:19] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6349 Casey Dentinger Collecting replag datums [04:19:42] (CR) jenkins-bot: Merge branch 'master' into deployment 77a7ef7 Short array syntax 1e695c9 Handle additional paypal refund reason codes. [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/381353 (owner: XenoRyet) [04:53:39] Fundraising Sprint RadioActivewear, Fundraising-Backlog: Persistent colors for labels in Grafana - https://phabricator.wikimedia.org/T176173#3616043 (Ejegg) @mepps There's a way to do this (if tedious) Under the 'Display' tab for your graph, choose 'Series overrides'. Enter an alias, then press the plus... [05:47:25] (CR) jenkins-bot: Merge branch 'master' into deployment 77a7ef7 Short array syntax 1e695c9 Handle additional paypal refund reason codes. [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/381353 (owner: XenoRyet) [05:49:45] (PS1) Ejegg: Update links in donation age widget [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/382362 [07:40:13] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 743827 Threads: 64 Questions: 80637213 Slow queries: 83693 Opens: 9685 Flush tables: 1 Open tables: 663 Queries per second avg: 108.408 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 486 [08:00:05] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 745628 Threads: 2 Questions: 80425591 Slow queries: 55093 Opens: 1683 Flush tables: 1 Open tables: 985 Queries per second avg: 107.862 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 402 [10:20:06] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 87% free (6594 MB out of 7629 MB) [10:25:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 87% free (6600 MB out of 7629 MB) [10:30:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 87% free (6608 MB out of 7629 MB) [10:35:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 88% free (6705 MB out of 7629 MB) [10:40:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 88% free (6709 MB out of 7629 MB) [10:45:06] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 88% free (6710 MB out of 7629 MB) [10:50:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 88% free (6711 MB out of 7629 MB) [10:55:06] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6714 MB out of 7629 MB) [11:00:06] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6716 MB out of 7629 MB) [11:05:06] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6719 MB out of 7629 MB) [11:10:15] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6720 MB out of 7629 MB) [11:15:12] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6723 MB out of 7629 MB) [11:20:12] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6726 MB out of 7629 MB) [11:25:12] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6728 MB out of 7629 MB) [11:30:12] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6731 MB out of 7629 MB) [11:35:12] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6734 MB out of 7629 MB) [11:40:22] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6737 MB out of 7629 MB) [11:45:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6739 MB out of 7629 MB) [11:45:27] Wikimedia-Fundraising-Campaigns (FY Q1 Jul-Sep 2017), Wikimedia-CentralNotice-Administration (Q3-2017): WMF Fundraising 2017-18 - Japan (5 September - 3 October) - https://phabricator.wikimedia.org/T170255#3660358 (Pcoombe) Open>Resolved [11:50:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6741 MB out of 7629 MB) [11:55:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6743 MB out of 7629 MB) [12:00:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6745 MB out of 7629 MB) [12:00:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1308 [12:05:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6747 MB out of 7629 MB) [12:05:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1456 [12:10:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1573 [12:10:14] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6749 MB out of 7629 MB) [12:15:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1729 [12:15:13] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6750 MB out of 7629 MB) [12:20:09] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6751 MB out of 7629 MB) [12:20:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1886 [12:25:09] PROBLEM - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6752 MB out of 7629 MB) [12:25:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2030 [12:29:59] ACKNOWLEDGEMENT - check_swap on frdb1002 is CRITICAL: SWAP CRITICAL - 89% free (6752 MB out of 7629 MB) Jeff_Green not critical, plenty of RAM available now, investigating... [12:30:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2182 [12:35:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2234 [12:40:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2218 [12:45:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2253 [12:50:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2156 [12:55:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2091 [13:00:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2064 [13:05:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2181 [13:10:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2280 [13:15:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2422 [13:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2552 [13:25:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2684 [13:30:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2795 [13:35:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2905 [13:40:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3030 [13:40:34] Jeff_Green: i did ack that last night but it must be a new alert [13:45:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3119 [13:50:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3265 [13:55:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3385 [14:00:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3495 [14:05:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3620 [14:10:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3745 [14:15:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [14:20:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3465 [14:25:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3457 [14:30:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3574 [14:35:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3684 [14:40:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3779 [14:45:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3918 [14:50:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4003 [14:55:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4104 [15:00:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4225 [15:05:18] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4303 [15:10:08] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4391 [15:13:47] cwd: yup [15:14:17] i'm going to restart it in a minute, gonna reduce innodb_buffer_pool_size a little [15:15:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4514 [15:18:39] fundraising-tech-ops, Operations, netops, ops-eqiad: Interface error on fasw-c-eqiad:vcp-255/1/0 - https://phabricator.wikimedia.org/T177333#3661112 (ayounsi) Open>Resolved a:ayounsi Chris replaced c1a-0 <-> c1b-1; interface errors are gone. [15:19:02] cool cool [15:20:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4593 [15:25:09] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4381 [15:26:53] mbeat mnoor ccogdill jessicarobell : heads up I'm going to restart mysql on frdev1001 to see if I can clear up the replication lag, I see you guys are logged in--nothing you need to do, this is just a heads up [15:27:11] ty Jeff_Green [15:27:25] thanks! [15:28:29] ok done [15:30:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2558 [15:35:09] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 417 Threads: 2 Questions: 1538261 Slow queries: 30 Opens: 334 Flush tables: 1 Open tables: 246 Queries per second avg: 3688.875 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:21:33] fundraising-tech-ops, Operations, netops, ops-codfw: vcp port down on fasw-c-codfw - https://phabricator.wikimedia.org/T177332#3661454 (ayounsi) Open>Resolved a:ayounsi Papaul reseated the cable, no more issues. [16:54:48] (PS8) Ejegg: Log client-side errors on payments forms [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/259795 (https://phabricator.wikimedia.org/T121800) [18:38:15] (PS1) Mepps: Handle payment not initiated [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 [18:38:42] hey ejegg let me know if ^^ is headed in the right direction [18:39:01] sure thing, taking a look! [18:43:27] (PS2) Mepps: Handle payment not initiated [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 [18:43:46] made some quick fixes--i was testing over in di so i had debugged there [18:44:55] (CR) Ejegg: "This looks like a pretty clean way of implementing it! We might need to fix up the factory method on PaymentResult to do something meanin" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 (owner: Mepps) [18:45:06] I like the direction! [18:45:28] great! yeah going to do more on it (like tests!) just wanted to make sure it was on the right track [18:45:38] Just need to make sure the pendingness gets communicated back out to the thing calling the gateway [18:45:59] hmm should i make the status PENDING? [18:46:09] well, i guess it /could/ call gateway->getFinalStatus() after calling that [18:46:11] i'm thinking of it in context of orphan_rectiifer [18:46:21] but if we could do it more functionally , that would be great [18:46:36] hmm so in process donor return it does go into the payment result [18:46:54] from the adapter's point of view, all it should care about is that the donor hasn't taken the necessary steps to do anything more [18:46:58] just not in processResponse [18:47:16] the decision to call it cancelled should be up to the orphan rectifier logic [18:47:38] mepps it returns a PaymentResult::fromResults using the finalStatus [18:48:08] but unfortunately that method doesn't do anything with the finalStatus except check whether it's 'FAILED' [18:55:21] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3661982 (Pcoombe) @spatton To clarify: the only thing stored in the `pgehres` database for banners is the impression numbers we pul... [18:58:07] (CR) XenoRyet: [C: 2] Report average consumed message age [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381263 (https://phabricator.wikimedia.org/T176920) (owner: Ejegg) [19:01:38] (Merged) jenkins-bot: Report average consumed message age [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381263 (https://phabricator.wikimedia.org/T176920) (owner: Ejegg) [19:04:03] (CR) XenoRyet: [C: 2] Report average thank you mail delay [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381264 (https://phabricator.wikimedia.org/T176920) (owner: Ejegg) [19:06:24] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3662016 (Jgreen) Here's the update--I think we've finally narrowed this down to a problem at kafka or possibly with kafka<->librdka... [19:07:34] (Merged) jenkins-bot: Report average thank you mail delay [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381264 (https://phabricator.wikimedia.org/T176920) (owner: Ejegg) [19:20:37] (CR) jerkins-bot: [V: -1] Handle payment not initiated [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 (owner: Mepps) [19:29:24] (PS3) Mepps: Handle payment not initiated [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 [19:53:02] (CR) jerkins-bot: [V: -1] Handle payment not initiated [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/382513 (owner: Mepps) [20:26:48] (PS1) Eileen: Remove temp tables pro-actively. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/382533 [20:29:45] (CR) jerkins-bot: [V: -1] Remove temp tables pro-actively. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/382533 (owner: Eileen) [21:17:19] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382607 [21:17:56] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382607 (owner: Ejegg) [21:19:29] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382607 (owner: Ejegg) [21:22:33] !log updated CiviCRM from 59e3e00ecbaa3557fd32dc49259a1ae12edf7414 to cca1a9d90ca41fe9fc75c12095225220dcc43226 [21:22:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:43:12] (PS2) Eileen: Remove temp tables pro-actively. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/382533 [22:19:28] (PS2) Ejegg: T177230 Thank you letter update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382034 (owner: Mepps) [22:19:47] (CR) Ejegg: [C: 2] T177230 Thank you letter update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382034 (owner: Mepps) [22:26:03] (Merged) jenkins-bot: T177230 Thank you letter update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382034 (owner: Mepps) [23:02:44] (PS1) Ejegg: Update CiviCRM submodule [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382621 [23:02:45] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382622 [23:03:19] (CR) Ejegg: [C: 2] Update CiviCRM submodule [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382621 (owner: Ejegg) [23:03:37] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382622 (owner: Ejegg) [23:04:03] (Merged) jenkins-bot: Update CiviCRM submodule [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382621 (owner: Ejegg) [23:04:14] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/382622 (owner: Ejegg) [23:10:25] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3662878 (Nuria) @Ottomata can you confirm that last statement, I thought we looked at this before and there was no lost data on ana... [23:40:44] !log updated CiviCRM from cca1a9d90ca41fe9fc75c12095225220dcc43226 to 197b22dce5a6c60a6e3697f5b9ba49de536d0bbb [23:40:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:54:27] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: PayPal EC cURL timeouts - https://phabricator.wikimedia.org/T177438#3663070 (Ejegg) Setting $wgDonationInterfaceTimeout seems to have worked, still not sure why the original didn't. I only saw a si... [23:55:11] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Patch-For-Review: Update postal address in Thank You email - https://phabricator.wikimedia.org/T177230#3651273 (Ejegg) OK @jrobell , this is deployed!