[01:13:43] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (minFraud:2) 2 [=1], check_endpoints_warning (minFraud:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:18:43] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (minFraud:2) 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:23:43] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [07:10:19] (03CR) 10Krinkle: [C:03+2] Replace uses of mw.Uri with browser native URL [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1073275 (https://phabricator.wikimedia.org/T374314) (owner: 10Ebrahim) [08:10:02] (03Merged) 10jenkins-bot: Replace uses of mw.Uri with browser native URL [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1073275 (https://phabricator.wikimedia.org/T374314) (owner: 10Ebrahim) [10:08:10] anilk: cstone, wow... that's a real cae of low-house-number-ophobia. [10:08:49] I like how that street intersects with Punkeydoodles Ave [10:51:39] (03PS8) 10Jgleeson: Add Gravy Audit Processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1068813 (https://phabricator.wikimedia.org/T367788) [12:24:22] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 07payments-orchestration: Gravy Audit/Settlement Multi-Processor strategy - https://phabricator.wikimedia.org/T371812#10152819 (10jgleeson) [12:26:34] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy Audit/Settlement Processing (donations) - https://phabricator.wikimedia.org/T367788#10152825 (10jgleeson) [12:26:37] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943 (10jgleeson) 03NEW [12:26:38] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943#10152837 (10jgleeson) [12:26:56] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy Audit/Settlement Processing (donations) - https://phabricator.wikimedia.org/T367788#10152849 (10jgleeson) [12:26:57] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943#10152850 (10jgleeson) [12:27:26] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943#10152851 (10jgleeson) [12:27:27] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Integration - https://phabricator.wikimedia.org/T364501#10152852 (10jgleeson) [12:35:13] 06Fundraising-Backlog, 07payments-orchestration: Gravy Audit/Settlement Processing (refunds) - https://phabricator.wikimedia.org/T374947 (10jgleeson) 03NEW [12:35:37] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943#10152905 (10jgleeson) [12:35:38] 06Fundraising-Backlog, 07payments-orchestration: Gravy Audit/Settlement Processing (refunds) - https://phabricator.wikimedia.org/T374947#10152904 (10jgleeson) [12:36:43] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 07payments-orchestration: Gravy Audit/Settlement Processing (refunds) - https://phabricator.wikimedia.org/T374947#10152909 (10jgleeson) [12:39:35] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy Audit/Settlement Processing (donations) - https://phabricator.wikimedia.org/T367788#10152915 (10jgleeson) earlier related SmashPig patch for completeness is here https://gerrit.wikimedia.o... [12:41:19] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 07payments-orchestration: Gravy Audit/Settlement Multi-Processor strategy - https://phabricator.wikimedia.org/T371812#10152921 (10jgleeson) [12:41:21] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Audit/Settlement Processing - https://phabricator.wikimedia.org/T374943#10152922 (10jgleeson) [13:11:20] (03CR) 10Jgleeson: [C:04-1] "Code looks good but I wonder if we could reorder it slightly just keep preserve the cancel method behaviour." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1070075 (https://phabricator.wikimedia.org/T366769) (owner: 10Cstone) [13:12:53] (03CR) 10Jgleeson: [C:04-1] "just to* preserve" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1070075 (https://phabricator.wikimedia.org/T366769) (owner: 10Cstone) [13:22:53] (03CR) 10Jgleeson: [C:03+2] "LGTM! thanks for the test to make review much easier!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1057929 (https://phabricator.wikimedia.org/T223330) (owner: 10Wfan) [13:41:25] (03Merged) 10jenkins-bot: Use epc as unsubscribe url for thank you email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1057929 (https://phabricator.wikimedia.org/T223330) (owner: 10Wfan) [13:53:43] oho, the unsub URL is almost switched to the prefs center [13:54:02] lemme see if I can get that functionality for resending a new link going [14:32:45] lol, paralyzed by the decision of where to put the new API action and what to name it [14:53:21] ah dang, we'll need to deal with merged contacts too [14:58:39] at least that is a solvable problem, unlike naming [15:04:58] (03PS1) 10Ejegg: Special return for valid but expired prefs link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073470 (https://phabricator.wikimedia.org/T352337) [15:06:13] 06Fundraising-Backlog, 10FR-email-preferences, 10Recurring-Donations, 13Patch-For-Review: For donor prefs pages, offer to send a new link when checksum is expired - https://phabricator.wikimedia.org/T352337#10153661 (10Ejegg) This is newly urgent because we're moving to the Email Prefs page for unsubscribe... [15:18:51] (03PS2) 10Ejegg: Special return for valid but expired prefs link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073470 (https://phabricator.wikimedia.org/T352337) [16:11:22] (03PS1) 10Jgleeson: Populate gravy 'backend processor' field in audit-generated messages. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073485 (https://phabricator.wikimedia.org/T371812) [16:27:54] (03PS3) 10Ejegg: Special return for valid but expired prefs link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073470 (https://phabricator.wikimedia.org/T352337) [16:34:00] 10fundraising-tech-ops: Fundraising access request for hnordeen - https://phabricator.wikimedia.org/T374127#10154156 (10Dwisehaupt) [16:36:10] 10fundraising-tech-ops: Fundraising access request for hnordeen - https://phabricator.wikimedia.org/T374127#10154173 (10Dwisehaupt) SSL client cert created and sent via email. Password sent via SMS. Superset account re-enabled and offered assistance to reset the password if needed. [16:37:12] (03PS9) 10Jgleeson: Add Gravy Audit Processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1068813 (https://phabricator.wikimedia.org/T367788) [16:43:39] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154187 (10Dwisehaupt) Ran into something odd in the logs when building out hosts. The pay-... [16:56:58] macbooks cursed time really messes up slack haha [17:16:55] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154354 (10cmooney) >>! In T373942#10154187, @Dwisehaupt wrote: > Ran into something odd in... [17:49:20] gonna relocate home for the next string of meetings [17:52:40] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154484 (10Dwisehaupt) @cmooney Thanks for that info. I think we may have multiple things going on here, ie: the... [18:26:16] huh, mosh doesn't wanna reconnect on network change for some reason [18:39:31] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154700 (10Jgreen) I think the modsec noise localized to one webserver is explained by pay-lb*:/etc/anycast-healt... [18:49:44] (03PS1) 10Ejegg: Simplify unsubscribe token using prefs crt link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073512 (https://phabricator.wikimedia.org/T223330) [18:53:13] just rolled a change for the puppet server in codfw from frpm2001 to frpm2002. there is the chance that some hosts may report a transient warning as they reconfigure. [18:53:44] (03PS2) 10Ejegg: Simplify unsubscribe token using donor prefs link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073512 (https://phabricator.wikimedia.org/T223330) [19:01:15] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154772 (10Dwisehaupt) Good find @Jgreen. This makes more sense knowing there are similar checks in multiple places. [19:04:50] 10fundraising-tech-ops: install/configure frpm2002 - https://phabricator.wikimedia.org/T373629#10154821 (10Dwisehaupt) [19:07:10] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154830 (10ssingh) You can find similar `check_cmds` (healthchecks) in a bunch of places that use bird/anycast-hc... [19:07:48] (03PS2) 10Cstone: Add expire [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073293 [19:13:22] (03CR) 10CI reject: [V:04-1] Simplify unsubscribe token using donor prefs link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1073512 (https://phabricator.wikimedia.org/T223330) (owner: 10Ejegg) [19:15:53] (03PS1) 10Umherirrender: specials: Get global via IContextSource::getConfig [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/1073519 [19:16:19] (03PS3) 10Cstone: Ignore expire [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073293 [19:17:00] (03PS4) 10Cstone: Add expire message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073293 [19:18:19] (03CR) 10Ejegg: [C:03+2] Add expire message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073293 (owner: 10Cstone) [19:18:47] (03Merged) 10jenkins-bot: Add expire message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1073293 (owner: 10Cstone) [19:34:58] (03PS1) 10Umherirrender: specials: Get global via IContextSource::getConfig [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1073527 [19:43:14] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10154906 (10Jgreen) >>! In T373942#10154830, @ssingh wrote: > The `durum.yaml` one has HTTP checks (while the othe... [20:16:58] 10fundraising-tech-ops: lightweight high-availability for payments-listener and frdata services - https://phabricator.wikimedia.org/T170318#10155039 (10Jgreen) This fixed the modsecurity log madness: ` check_http -S -I 208.80.152.227 -H payments.wikimedia.org -k 'Accept: text/*' -u https://payments.wikimedia.or... [20:27:11] (03PS1) 10Umherirrender: specials: Get global via IContextSource::getConfig [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1073547 [20:34:22] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 10API-Portal, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-Site-requests: API portal showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T270308#10155068 (10Ejegg) OK, this is finally deployed [20:41:03] (03CR) 10Krinkle: [C:03+2] Replace uses of mw.Uri with browser native URL [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1073271 (https://phabricator.wikimedia.org/T374314) (owner: 10Ebrahim) [21:03:10] 10fundraising-tech-ops: Investigate metabase packaging, install, security monitoring. - https://phabricator.wikimedia.org/T375030 (10Jgreen) 03NEW [21:03:32] 10fundraising-tech-ops: Investigate metabase packaging, install, security monitoring. - https://phabricator.wikimedia.org/T375030#10155301 (10Jgreen) [21:03:34] 06Fundraising-Backlog: Connect current data set to Metabase instance - https://phabricator.wikimedia.org/T373631#10155302 (10Jgreen) [21:03:55] 10fundraising-tech-ops: Investigate metabase packaging, install, security monitoring. - https://phabricator.wikimedia.org/T375030#10155303 (10Jgreen) [21:16:09] (03PS1) 10Jdlrobson: Track bundlesizes for modules loaded on page load [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1073561 (https://phabricator.wikimedia.org/T373017) [21:52:40] (03Merged) 10jenkins-bot: Replace uses of mw.Uri with browser native URL [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1073271 (https://phabricator.wikimedia.org/T374314) (owner: 10Ebrahim) [22:38:40] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 10API-Portal, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-Site-requests: API portal showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T270308#10155514 (10apaskulin) 05Open→03Resolved Confirmed that... [22:39:26] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10155516 (10Dwisehaupt) I believe [[ https://phabricator.wikimedia.org/T170318#10155039 | this comment ]] was supp... [22:51:39] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install civi2002, frpig2002, frpm2002 - https://phabricator.wikimedia.org/T369937#10155528 (10Dwisehaupt) 05Open→03Resolved All three hosts are built. Any follow on configuration or adjustments will be in new tasks. [22:54:39] I'm having trouble tagging Noah in phab - does anyone know his handle? [22:55:18] unless he is just Noah [23:02:25] 10fundraising-tech-ops: Migrate backup civi host from civi2001 to civi2002 - https://phabricator.wikimedia.org/T375038 (10Dwisehaupt) 03NEW [23:07:46] I think nisreal https://phabricator.wikimedia.org/project/board/41/?filter=vgzBGtWGp6lW [23:07:49] eileen: [23:08:17] thanks [23:08:38] anilk: that will save some rando from getting the ping :-) [23:28:43] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 10FR-donorservices: Civi "thank you letter summary" feature request - https://phabricator.wikimedia.org/T299096#10155563 (10Eileenmcnaughton) We discussed this on a hangout just now - [[ https://docs.google.com/document/d/1kkk9LpgnxhPFqv8jBTX... [23:29:30] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Imports: Import migration - add support for currency conversion - https://phabricator.wikimedia.org/T368998#10155565 (10Eileenmcnaughton) [23:29:32] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Imports: Import migration - add support for currency conversion - https://phabricator.wikimedia.org/T368998#10155566 (10Eileenmcnaughton) 05Open→03Resolved [23:30:44] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 07FR-Imports, 07Unplanned-Sprint-Work: Add support to create organization when importing fidelity - https://phabricator.wikimedia.org/T375039 (10Eileenmcnaughton) 03NEW [23:32:36] 03Fundraising Sprint: reasonablyAccurate(), 06Fundraising-Backlog, 07FR-Imports, 07Unplanned-Sprint-Work: Add support to create organization when importing fidelity - https://phabricator.wikimedia.org/T375039#10155579 (10Eileenmcnaughton)