[00:08:11] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: codfw:frack:rack/install/configuration new firewalls - https://phabricator.wikimedia.org/T374176#10123906 (10Papaul) [00:31:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [00:36:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [00:36:50] ^^^ checking on that. [00:37:40] ugh. [00:38:43] ACKNOWLEDGEMENT - check_mysql on frdb2004 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt replication error - investigating https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [00:46:09] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 13145 Threads: 4 Questions: 3393002 Slow queries: 0 Opens: 1147 Open tables: 1141 Queries per second avg: 258.121 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [01:27:15] 06Fundraising-Backlog, 06Editing-team, 06SRE, 06Traffic-Icebox, and 5 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085#10123988 (10Pppery) 05Open→03Stalled [02:22:41] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1071057 [02:23:40] eileen: I'm going to deploy those currency changes. I can stop the donations queue and do a slow-start to check a few for starters [02:24:00] err, the civicrm changes including the original_currency patches [02:24:05] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1071057 (owner: 10Ejegg) [02:24:52] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1071057 (owner: 10Ejegg) [02:29:32] !log disabled donations queue consumer for civi deploy [02:29:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:31:56] !log fundraising civicrm upgraded from 67ee99ce to 5dd4edc1 [02:31:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:36:22] ok, all those contribution sources look fine [02:36:27] restarting the qc [02:37:53] !log restarted donations queue consumer [02:37:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:38] PROBLEM - Host frdb2004 is DOWN: PING CRITICAL - Packet loss = 100% [10:36:10] RECOVERY - Host frdb2004 is UP: PING OK - Packet loss = 0%, RTA = 30.47 ms [10:43:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:48:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:53:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:58:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:03:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:08:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:13:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:18:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:23:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:28:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:33:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:38:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:43:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:44:31] ACKNOWLEDGEMENT - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] Jeff_Green what. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:53:16] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:23:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2518 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:23:21] 03Fundraising Sprint: quietlyBreakingThings(), 06Fundraising-Backlog, 07payments-orchestration, 07Unplanned-Sprint-Work: Make gravy form match adyen form - https://phabricator.wikimedia.org/T373557#10125981 (10Ejegg) a:03Ejegg [14:28:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 1433601 Threads: 4 Questions: 144598493 Slow queries: 683 Opens: 3087 Open tables: 1174 Queries per second avg: 100.863 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:50:51] (03PS1) 10Ejegg: Reformat Gravy CSS [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1071224 (https://phabricator.wikimedia.org/T373557) [14:51:53] (03PS1) 10Ejegg: Fix field heights in Gravy form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1071225 (https://phabricator.wikimedia.org/T373557) [15:15:45] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: host won't boot lists backplane error for pay-lb2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T374054#10126212 (10Jhancock.wm) looks like us shutting down the server to move it fixed the error. Can you take a look and co... [15:52:07] PROBLEM - Host frav1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:54:20] 06Fundraising-Backlog, 10FR-donorservices: Possible email search bug - https://phabricator.wikimedia.org/T374260 (10SHust) 03NEW [15:57:09] RECOVERY - Host frav1003 is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [17:27:27] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install frban2002 - https://phabricator.wikimedia.org/T369931#10126728 (10Dwisehaupt) Host OS installed and built out with basics. Awaiting the completion of T374269 to finish config and testing. [17:28:18] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q#:rack/setup/install payments200[456] - https://phabricator.wikimedia.org/T369942#10126731 (10Dwisehaupt) payments2006 built out and mariadb cloned out. Awaiting completion of T374269 to finish config and testing. [18:32:36] whew, more nagios spam [18:36:07] ejegg: ? oh, the ones from early this morning. [18:41:28] 03Fundraising Sprint: quietlyBreakingThings(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Import migration - add support for currency conversion - https://phabricator.wikimedia.org/T368998#10126948 (10MDemosWMF) @Eileenmcnaughton I tried to test this out and setup the mapping, but I got the err... [18:57:47] oh i see [19:13:10] 03Fundraising Sprint: quietlyBreakingThings(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Import migration - add support for currency conversion - https://phabricator.wikimedia.org/T368998#10127042 (10Ejegg) Hi @MDemosWMF , I just moved this one to 'done' because I deployed the one code change I... [19:15:23] 03Fundraising Sprint: quietlyBreakingThings(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Import migration - add support for currency conversion - https://phabricator.wikimedia.org/T368998#10127044 (10MDemosWMF) @Ejegg yes, I think once we have that hopefully it will work! [19:50:07] ejegg: was just about to ping you on here, but all good now? [19:50:54] I am not sure why an-druid1001 would work better than 1005, but druid clusters are weird sometimes [19:55:36] oh perhaps I hadn't tried the full url with 1005 [19:55:51] I think I only added the eqiad.wmnet when I swapped to 1001 [19:56:04] and thank you, I do have it working! [20:20:46] (03CR) 10Samiqussairen: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) (owner: 10Damilare Adedoyin) [20:20:48] (03CR) 10Samiqussairen: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) (owner: 10Damilare Adedoyin) [20:20:49] (03CR) 10Samiqussairen: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) (owner: 10Damilare Adedoyin) [20:20:58] (03CR) 10Samiqussairen: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) (owner: 10Damilare Adedoyin) [20:22:25] (03CR) 10Samiqussairen: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) (owner: 10Damilare Adedoyin) [20:57:57] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install frlog2002 - https://phabricator.wikimedia.org/T369935#10127280 (10Dwisehaupt) 05Open→03Resolved Host is built and config will continue in T372933 [20:59:08] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: host won't boot lists backplane error for pay-lb2002.frack.codfw.wmnet - https://phabricator.wikimedia.org/T374054#10127286 (10Dwisehaupt) 05Open→03Resolved Thanks. It's back online and up. Hopefully it has a transient error. [22:05:42] 06Fundraising-Backlog, 10FR-donorservices: Possible email search bug - https://phabricator.wikimedia.org/T374260#10127364 (10SHust) [22:13:20] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Adyen:1, Paypal:1) 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [22:14:51] 06Fundraising-Backlog, 10FR-donorservices: Fundraising access request for Doris Morgan (no rush since she'll start working on Sept 10th) - https://phabricator.wikimedia.org/T374287 (10SHust) 03NEW [22:18:20] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Adyen:1, Paypal:1) 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [22:23:16] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages