[02:18:58] (PS1) Eileen: Drupal update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/642212 [02:20:06] (PS1) Eileen: Re-apply WMF patches [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/642213 [02:35:20] (CR) jerkins-bot: [V: -1] Drupal update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/642212 (owner: Eileen) [03:25:38] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1256 [03:30:32] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 284930 Threads: 13 Questions: 23793920 Slow queries: 163705 Opens: 9435730 Flush tables: 1 Open tables: 200 Queries per second avg: 83.507 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [06:20:32] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, FR-Email: Investigate email preference center prototype - https://phabricator.wikimedia.org/T266607 (MSuijkerbuijk_WMF) Thanks @Eileenmcnaughton. I'll discuss this with the team and review next s... [09:57:30] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1003:down [09:57:40] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1003:down [09:58:12] ah nice I have highlights for kafka and saw these alerts too :) [09:58:26] I am doing a roll restart of kafka brokers for openjdk upgrades [10:01:09] ACKNOWLEDGEMENT - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1003:down Dwisehaupt Dont wake me in the middle of the night. [10:01:09] ACKNOWLEDGEMENT - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1003:down Dwisehaupt Dont wake me in the middle of the night. [10:12:34] RECOVERY - check_kafkatee on frban2001 is OK: OK: brokers:9 topics:2 [10:12:40] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 [10:52:29] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1005:down [10:52:35] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1005:down [10:57:35] RECOVERY - check_kafkatee on frban2001 is OK: OK: brokers:9 topics:2 [10:57:35] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 [11:27:42] are those pages --^ ? [11:28:07] if so let's sync about them, we do regularly upgrade kafka jumbo, I can alert you beforehand etc... [11:57:14] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Tim Otten - https://phabricator.wikimedia.org/T252694 (Aklapper) @Dwisehaupt: Which other task or which person is this task [stalled](https://www.mediawiki.org/wiki/Bug_management/Bug_report_life_cycle) on? [11:57:16] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Coleman Watts - https://phabricator.wikimedia.org/T252693 (Aklapper) @Dwisehaupt: Which other task or which person is this task [stalled](https://www.mediawiki.org/wiki/Bug_management/Bug_report_life_cycle)... [15:40:30] fr-tech, I was thinking about possible power savings using docker vs vagrant/other for the times we're working outside of the house (not like we'll be doing that much atm) as that is one of my real gripes with vagrant, it drains the battery! [15:45:44] jgleeson: ahh good point! [16:06:02] yep yep, mine too [17:28:17] Fundraising-Backlog: Emojis in names breaking things - continued - https://phabricator.wikimedia.org/T268356 (Cstone) [17:42:37] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Tim Otten - https://phabricator.wikimedia.org/T252694 (Dwisehaupt) @Aklapper This is stalled on an external contractor. I have set up our initial portions but can't proceed until they follow up with additio... [17:43:05] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Coleman Watts - https://phabricator.wikimedia.org/T252693 (Dwisehaupt) @Aklapper This is stalled on an external contractor. I have set up our initial portions but can't proceed until they follow up with add... [18:02:31] dstrine: sorry, trying from other lappy and the page is just not loading :( [18:02:41] ejegg: ok [18:14:34] !log shifting 100% of thank_you mail through frmxs ahead of tomorrow's banner test - T267259 [18:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:41] T267259: Create new job to send Civi thank you emails over smtp through frmx hosts - https://phabricator.wikimedia.org/T267259 [18:42:56] Fundraising-Backlog, fundraising-tech-ops: Create new job to send Civi thank you emails over smtp through frmx hosts - https://phabricator.wikimedia.org/T267259 (Dwisehaupt) [19:47:19] (PS1) Jgleeson: WIP: Add in check for additional recurring cotributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/642517 [19:49:28] (PS2) Jgleeson: WIP: Add in check for additional recurring cotributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/642517 (https://phabricator.wikimedia.org/T260910) [19:50:45] fr-tech I'm gonna call it a day. have a great weekend all o/ [19:51:04] see ya jgleeson|away [20:07:40] (CR) jerkins-bot: [V: -1] WIP: Add in check for additional recurring cotributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/642517 (https://phabricator.wikimedia.org/T260910) (owner: Jgleeson) [20:20:29] fr-tech i'm going to delete those gdpr job rows in the civi db - we verified that the email doesn't exist in the acoustic dbs that are targeted [20:21:52] ok, deleted [20:22:06] that'll quiet down the failmail a bit for the weekend [21:09:11] oh darn, still getting that failmail - was there another job? [21:11:33] It's super nice out here. Going to take the dog on a decent walk. be back in a bit. [23:41:20] (CR) Ejegg: [C: -1] "That check methods looks pretty good, though we might want to ask for clarification on the best conditions to add. We'll want to call it f" (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/642517 (https://phabricator.wikimedia.org/T260910) (owner: Jgleeson) [23:58:03] Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, Android-app-Bugs, and 6 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (MattCleinman) Could we try copying the thankyou site file to `ht...