[00:00:02] re-enabled [00:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:30:49] (PS1) Ammarpad: Use IPUtils to check IP [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/637583 (https://phabricator.wikimedia.org/T248059) [00:34:58] (CR) Ammarpad: Use IPUtils to check IP (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/637583 (https://phabricator.wikimedia.org/T248059) (owner: Ammarpad) [01:32:43] PROBLEM - check_swap on frav1002 is CRITICAL: SWAP CRITICAL - 1% free (0 MB out of 7628 MB) [01:34:12] ACKNOWLEDGEMENT - check_swap on frav1002 is CRITICAL: SWAP CRITICAL - 1% free (0 MB out of 7628 MB) Dwisehaupt Known [03:38:42] (PS1) Eileen: Fix join to exclude any emails deleted since update_world was created [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/637586 [14:52:27] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [14:52:37] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [14:57:31] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [14:57:37] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [14:58:01] this is probably elukey and cmjohnson1 doing work on kafka-jumbo1006 ^ [14:58:10] should not cause a real problem if kafkatee is working properly [15:02:33] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [15:02:37] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [15:07:33] PROBLEM - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [15:07:37] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down [15:08:41] ACKNOWLEDGEMENT - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down Jeff_Green Known. Workers are investigating! [15:08:41] ACKNOWLEDGEMENT - check_kafkatee on frban2001 is CRITICAL: CRITICAL: kafka-jumbo1006:down Jeff_Green Known. Workers are investigating! [15:10:47] fwiw, the kafkatee alert is due to SRE working on the server [15:35:47] (CR) Ejegg: [C: +2] Fix join to exclude any emails deleted since update_world was created [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/637586 (owner: Eileen) [15:36:25] Wikimedia-Fundraising-Banners, MediaWiki-extensions-General, JavaScript, Wikimedia-production-error: Large error spike suggesting breakage of Vector JavaScript for certain clients - https://phabricator.wikimedia.org/T266852 (Jdlrobson) Thanks for confirming @spatton. Now I've taken a closer look... [15:37:11] (Merged) jenkins-bot: Fix join to exclude any emails deleted since update_world was created [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/637586 (owner: Eileen) [15:37:29] Wikimedia-Fundraising-Banners, MediaWiki-extensions-General, JavaScript, Wikimedia-production-error: Large error spike suggesting breakage of Vector JavaScript for certain clients - https://phabricator.wikimedia.org/T266852 (Jdlrobson) [16:07:24] fundraising-tech-ops: Prometheus/Grafana snapshots for long term storage - https://phabricator.wikimedia.org/T198876 (Jgreen) >>! In T198876#6590252, @Dwisehaupt wrote: > We are currently storing ~765 days of metrics with our current retention settings. They are currently only present that far back on frmon1... [16:11:49] Fundraising-Backlog: Apple Pay Spec - ingenico - https://phabricator.wikimedia.org/T228944 (EMartin) Some added insight from Ingenico about their Apple pay support when we are ready : We do support the native in-app integrations, sorry for not confirming earlier. We can set this up in the pre-production en... [16:19:58] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test: desktop small batch - https://phabricator.wikimedia.org/T266806 (jbolorinos-ctr) Screenshot Test Results - Desktop: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zcbd96487e21975d5876 - We wont lie: https://app.cross... [16:22:33] RECOVERY - check_kafkatee on frban2001 is OK: OK: brokers:9 topics:2 [16:22:33] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 [16:25:53] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test mobile large batch - https://phabricator.wikimedia.org/T266808 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z0e1b570670de09be24d - Annual Donation: https://app.cro... [16:29:05] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test mobile small batch - https://phabricator.wikimedia.org/T266809 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/za84fd22a6bd26ac463a - We wont lie: https://app.crossbr... [16:31:33] Wikimedia-Fundraising-Banners: QA for en6c pre-test Saturday 10/31 - https://phabricator.wikimedia.org/T266803 (jbolorinos-ctr) [16:31:35] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test desktop large batch - https://phabricator.wikimedia.org/T266804 (jbolorinos-ctr) Open→Resolved Everything looks good @HNordeenWMF! Status: ✅ PASS Closing this task as Resolved [16:31:54] Fundraising-Backlog, FR-Email: Civi to IBM export: No file on FTP - https://phabricator.wikimedia.org/T266665 (DStrine) Open→Resolved [16:34:28] Wikimedia-Fundraising-Banners: QA for en6c pre-test Saturday 10/31 - https://phabricator.wikimedia.org/T266803 (jbolorinos-ctr) [16:34:30] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test: desktop small batch - https://phabricator.wikimedia.org/T266806 (jbolorinos-ctr) Open→Resolved Status: ✅ PASS @HNordeenWMF signing off on QA for these also, everything looks good! Closing task as Resolved [16:38:04] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: fr-tech to look into supporting mail fail over for civi - https://phabricator.wikimedia.org/T264663 (Jgreen) >>! In T264663#6590462, @Eileenmcnaughton wrote: > I di... [16:58:43] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Met with fr-tech team today! @Ejegg found that the "uploader" is being kicked off too soon before the... [16:59:28] Wikimedia-Fundraising-Banners: 2020-10-31 en6c pre-test mobile large batch - https://phabricator.wikimedia.org/T266808 (jbolorinos-ctr) Open→Resolved Status: ✅ PASS Everything looks good for these also @HNordeenWMF! Closing task as Resolved. [16:59:30] Wikimedia-Fundraising-Banners: QA for en6c pre-test Saturday 10/31 - https://phabricator.wikimedia.org/T266803 (jbolorinos-ctr) [17:16:55] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Open→Resolved Acoustic import ran successfully and looks good to me. Thanks for the help! Resol... [17:49:00] fundraising-tech-ops, FR-Q2-FY2020-21-cleanup-list, Patch-For-Review: Upgrade payments to MediaWiki 1.35 - https://phabricator.wikimedia.org/T254298 (Dwisehaupt) Open→Resolved a:Dwisehaupt The shift has been made to make the payments and payments-staging targets to use the 1.35 release br... [18:32:06] fundraising-tech-ops, Operations: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T265086 (Jgreen) [18:44:31] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Conflicting fields in some Acoustic contact records - https://phabricator.wikimedia.org/T254304 (KHaggard) Adding in some new mystery contacts from another phab (T265700) into this one to investigate why a small number of contacts did not get modified b... [18:46:13] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog: all_Wikimedia database shows both_funds_donation_count as a decimal - https://phabricator.wikimedia.org/T265700 (KHaggard) The conversation about the few un-updated contacts has been moved to this phab T254304 so we can "officially" close this on... [18:46:56] fr-tech I can't figure out why an uncaught exception in python is still exiting with code 0 [18:49:17] process-control definitely should not continue with a multi-step job if the first step fails: https://phabricator.wikimedia.org/diffusion/WFPC/browse/master/processcontrol/runner.py$63 [18:52:05] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (MBeat33) @Eileenmcnaughton just to follow up from IR... [19:27:45] (PS1) Cstone: Monthly Convert: Variant 011, yes button text with ask [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/637767 [19:36:03] fundraising-tech-ops, Operations: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T265086 (Dwisehaupt) [19:38:59] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (bsisolak) No logs are stored more than four days (as... [19:46:23] fundraising-tech-ops, Operations: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T265086 (Dwisehaupt) [19:52:36] RECOVERY - check_swap on frav1002 is OK: SWAP OK - 33% free (2497 MB out of 7628 MB) [20:30:10] (PS1) Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/637775 [20:30:41] (CR) Eileen: [C: +2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/637775 (owner: Eileen) [20:31:48] (Merged) jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/637775 (owner: Eileen) [20:42:08] Fundraising-Backlog: Docker dev setup: Add log filters and separate files to emulate production - https://phabricator.wikimedia.org/T266616 (AndyRussG) A small improvement that we might make along the way: filter out rsyslogd messages (like startup info) sent from payments container. [21:33:29] Fundraising-Backlog: Docker dev setup: Add log filters and separate files to emulate production - https://phabricator.wikimedia.org/T266616 (AndyRussG) Also seeing this sometimes in syslog messages on the logger container... So we should fix the issue that causes it, but also not filter out warnings about su...