[01:20:35] Holy failmail, batman. [01:20:50] fr-tech: Anybody still around? [01:21:51] i'm a few blocks from my computer, checking email [01:22:52] ejegg|away: paypal timing out? [01:34:18] !log disabled IPN messages at the PayPal console [01:34:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:36:17] Fundraising Sprint Quill Pencil, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18, Patch-For-Review: Remove invalid City data from DB - https://phabricator.wikimedia.org/T174985#3583029 (Eileenmcnaughton) [04:38:03] Fundraising Sprint Quill Pencil, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18: Data clean up - remove double spaces html ampersands from first & last names - https://phabricator.wikimedia.org/T175107#3583030 (Eileenmcnaughton) [04:56:50] (PS6) Eileen: Resolve diacritic merge conflicts. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) [04:56:52] (PS3) Eileen: Do not save 0 or other invalid values to city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375945 (https://phabricator.wikimedia.org/T174980) [04:56:54] (PS2) Eileen: Remove void data from city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) [05:12:15] (PS3) Eileen: Remove void data from city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) [05:13:37] (PS1) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:13:48] (PS2) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:18:55] (PS3) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:21:44] (PS4) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:29:49] (PS5) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:34:45] (PS6) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [07:37:16] (PS7) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [08:20:12] (CR) Eileen: [C: -1] "There is some issue with the whitespace replacements on the last 28 contacts on staging" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) (owner: Eileen) [10:03:46] Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: Deploy pt-br and ja thank you letter - https://phabricator.wikimedia.org/T173809#3583660 (jrobell) Thank you @Ejegg !! I have sent the final versions around to nat... [10:24:25] Wikimedia-Fundraising-Campaigns (FY Q1 Jul-Sep 2017), Wikimedia-CentralNotice-Administration (Q3-2017): WMF Fundraising 2017-18 - Malaysia (8 August - 5 September) - https://phabricator.wikimedia.org/T170253#3583907 (Pcoombe) Open>Resolved [10:24:32] Wikimedia-Fundraising-Campaigns (FY Q1 Jul-Sep 2017), Wikimedia-CentralNotice-Administration (Q3-2017): WMF Fundraising 2017-18 - South Africa (8 August - 5 September) - https://phabricator.wikimedia.org/T170254#3583909 (Pcoombe) Open>Resolved [10:41:39] (PS8) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [10:45:34] (PS9) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [10:52:44] (PS10) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [11:47:55] (Restored) Hashar: Jenkins job validation (DO NOT SUBMIT) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [11:47:59] (PS2) Hashar: Jenkins job validation (DO NOT SUBMIT) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 [11:48:05] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [11:48:32] (PS1) Hashar: Jenkins job validation (DO NOT SUBMIT) [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/376225 [11:48:44] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/324066 (owner: Hashar) [11:48:48] (CR) Hashar: "check experimental" [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/376225 (owner: Hashar) [12:56:18] (PS1) Hashar: Jenkins job validation (DO NOT SUBMIT) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376243 [12:56:51] (CR) Hashar: "check experimental" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376243 (owner: Hashar) [15:00:08] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[generate-kafkatee.pyconf] [15:05:07] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:16:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, MediaWiki-extensions-DonationInterface: Epic: DonationInterface should not be a dependency of our CiviCRM install - https://phabricator.wikimedia.org/T174691#3584650 (Ejegg) p:Triage>Low [15:55:12] !log re-enabled IPN notification at PayPal console (48 minutes ago) [15:55:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:04:22] Hi MBeat, just to confirm - you'll give us the signal when Ingenico console is back up and we're safe to try Big English? No particular rush. [16:04:33] I think it’s good now spatton [16:04:54] so whenever makes sense for you [16:04:55] pcoombe re: site loading ... what's your opinion? Should we keep holding? [16:04:58] Thanks MBeat! [16:07:12] spatton: I think we're good to go now [16:07:18] Woot! [16:07:37] Gonna do 4:10 UTC to 5:10 [16:08:31] Actually going to double check w/ Seddon bc he's got his own test planned [16:09:53] Seddon says we're good as well [16:09:59] (PS1) Ejegg: Don't fail transaction on SuccessWithWarning [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376289 (https://phabricator.wikimedia.org/T165635) [16:10:24] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Quill Pencil, Fundraising-Backlog, and 3 others: Deal with paypal EC error 11607, show thank you page - https://phabricator.wikimedia.org/T165635#3272299 (Ejegg) a:Ejegg [16:15:26] fr-tech ^^^ would be really nice to get reviewed [16:15:42] I think it's going to prevent us showing fail pages to some donors with successful txns [16:15:50] I'll take a look [16:24:15] (PS7) Ejegg: Resolve diacritic merge conflicts. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) (owner: Eileen) [16:26:52] (PS8) Ejegg: Resolve diacritic merge conflicts. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) (owner: Eileen) [16:27:18] (CR) Ejegg: [C: 2] Resolve diacritic merge conflicts. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) (owner: Eileen) [16:27:56] (CR) Ejegg: [C: 2] "(wmf_civicrm.module fix in PS5 seems to have been undone in PS6 - I just changed the file back to how it was in PS5" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) (owner: Eileen) [16:29:20] (PS4) Ejegg: Do not save 0 or other invalid values to city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375945 (https://phabricator.wikimedia.org/T174980) (owner: Eileen) [16:31:20] (CR) XenoRyet: [C: 2] Don't fail transaction on SuccessWithWarning [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376289 (https://phabricator.wikimedia.org/T165635) (owner: Ejegg) [16:31:43] (CR) Ejegg: [C: 2] "A lot of these seem to come from PayPal, so we can also clean up the data at the first point of entry. No objection to checking again here" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375945 (https://phabricator.wikimedia.org/T174980) (owner: Eileen) [16:37:55] (Merged) jenkins-bot: Resolve diacritic merge conflicts. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374951 (https://phabricator.wikimedia.org/T149763) (owner: Eileen) [16:37:57] (Merged) jenkins-bot: Don't fail transaction on SuccessWithWarning [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376289 (https://phabricator.wikimedia.org/T165635) (owner: Ejegg) [16:38:36] (CR) Ejegg: [C: -1] "civicrm_ prefix is getting doubled. Maybe drop the prefixing in the function in case we want to clean up e.g. wmf_contribution_extra?" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) (owner: Eileen) [16:43:24] (Merged) jenkins-bot: Do not save 0 or other invalid values to city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375945 (https://phabricator.wikimedia.org/T174980) (owner: Eileen) [16:55:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1381 [17:00:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1551 [17:00:43] yaaaay [17:00:59] cwd dedupe still off? [17:01:04] yep [17:01:15] dang. i saw a deadlock failmail too [17:01:19] https://grafana.wikimedia.org/dashboard/db/frack-db?orgId=1&from=1504708538870&to=1504716060661&panelId=3&fullscreen [17:01:24] want me to turn off civimail records? [17:01:33] ejegg: sure give it a shot [17:01:57] that chart *could* mean that something is hammering the read server [17:02:10] but i have no idea anymore [17:02:18] !log disabled creating CiviMail records for Thank You emails [17:02:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:05:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1712 [17:07:29] cwd I don't think anything reads from frdb1002 [17:07:43] people do some reporting from frdev1001 [17:07:54] ejegg: 1002 is the read server [17:07:56] with the fqdn [17:08:00] oh, ok [17:08:13] in my imagination i could see excessive reads contributing to replag [17:08:19] but i am grasping at straws at this point [17:08:23] read server for what exactly? [17:08:29] dashboard? [17:08:37] hmmm, good question [17:09:00] Oh, I guess smashpig has a couple of things pointing to the read server [17:09:17] but it shouldn't be doing anything too crazy [17:09:24] fundraisingdb-read [17:09:28] anything that points to that [17:10:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1920 [17:11:46] well, nothing really to be done about that i guess, but more to talk about this afternoon :) [17:12:02] i have to run out for awhile [17:12:10] k, see ya [17:12:29] got to resolve this phone situation [17:12:30] later! [17:12:34] good luck! [17:12:36] ty [17:15:08] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2095 [17:20:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2274 [17:22:41] fr-tech any news for scrum of scrums? [17:23:30] Nothing here [17:24:00] ejegg: nothing here, thx! [17:25:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2398 [17:26:23] fr-tech dstrine pcoombe Got an ipad simulator with dev tools running finally, but there are no active ipad campaigns anymore! Gonna set up a test campaign on aa.wikibooks, and will include the current banner lib (https://meta.wikimedia.org/wiki/MediaWiki:FundraisingBanners/CoreJS-2017.js) in a banner. Might get some garbage requests to payments, then. Assuming that's OK! [17:28:39] sure, no problem here AndyRussG [17:30:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2504 [17:35:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2628 [17:36:31] pcoombe: thx :) [17:40:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2716 [17:45:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2797 [17:49:47] (PS1) Ejegg: Fix a couple auto-inserted braces, break lines [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376306 [17:50:11] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2927 [17:52:42] (PS1) Ejegg: Short array syntax [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 [17:52:46] (CR) jerkins-bot: [V: -1] Short array syntax [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 (owner: Ejegg) [17:54:59] (PS2) Ejegg: Short array syntax [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 [17:55:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2991 [18:00:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3113 [18:01:43] Hi ejegg, did you deploy the continue button fix yesterday? I thought you did but am still seeing the bug [18:02:12] pcoombe: oh, I thought I did too! [18:02:16] let me see [18:04:19] pcoombe: sorry, looks like it didn't get out there. re-doing the deploy! [18:05:08] ejegg: that's fine. No need to rush it, just checking the status so I can update the rest of the team [18:05:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3217 [18:05:30] !log updated payments-wiki from bf97cd236bdec421082db8a8a782965de9c98358 to 691115837b3bbd07a53248ca46a7bd80d2a3970e [18:05:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:43] heh, no worries, actually deployed now [18:08:39] crap, looks like that broke paypal ec [18:08:41] reverting [18:09:28] !log rolled payments-wiki back to bf97cd236bdec421082db8a8a782965de9c98358 [18:09:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:49] pcoombe: not sure what would have done that, but I'll test more locally [18:10:07] ok, going to relocate before they shut down all the streets for the pope's visit [18:10:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2968 [18:10:29] okay, thanks [18:15:10] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1315 [18:20:10] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 1140080 Threads: 3 Questions: 25247401 Slow queries: 7856 Opens: 11525 Flush tables: 1 Open tables: 607 Queries per second avg: 22.145 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [18:37:32] i swear the more i learn about this replag the less i understand it [18:39:05] https://grafana.wikimedia.org/dashboard/db/frack-db?panelId=1&fullscreen&orgId=1 [19:35:35] (PS1) Ejegg: Only uncheck card on load when shown [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376335 [19:35:42] pcoombe ^^^ should fix it [19:36:43] oops, the returnURL is missing too... [19:37:00] damn, I thought we had a test for that [20:06:13] (CR) XenoRyet: [C: 2] Only uncheck card on load when shown [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376335 (owner: Ejegg) [20:08:53] (Merged) jenkins-bot: Only uncheck card on load when shown [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376335 (owner: Ejegg) [20:12:47] (PS1) Ejegg: Fix RETURNURL for PayPal EC, add tests for API params [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376349 [20:13:24] OK, that should let us deploy again ^^^ [20:13:55] (PS2) Ejegg: Fix RETURNURL for PayPal EC, add tests for API params [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376349 [20:44:26] (CR) XenoRyet: [C: 2] Fix RETURNURL for PayPal EC, add tests for API params [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376349 (owner: Ejegg) [20:48:20] (Merged) jenkins-bot: Fix RETURNURL for PayPal EC, add tests for API params [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376349 (owner: Ejegg) [20:51:03] thanks XenoRyet [20:51:08] ok, I'll try that deploy again [20:55:37] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/376410 [20:55:49] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/376410 (owner: Ejegg) [21:00:00] hmm, suppose I should deploy CRM too [21:00:35] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/376410 (owner: Ejegg) [21:03:34] ejegg: there are 2 v minor patches which were from your feedback https://gerrit.wikimedia.org/r/#/c/375721/ [21:04:40] (PS2) Ejegg: Follow patch from QA on find duplicates task [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375720 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:04:48] (CR) Ejegg: [C: 2] Follow patch from QA on find duplicates task [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375720 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:04:57] (PS2) Ejegg: Fix spelling error. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375721 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:05:02] (CR) Ejegg: [C: 2] Fix spelling error. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375721 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:05:08] Thanks eileen! [21:10:51] thanks! [21:15:29] ejegg: on this one - https://gerrit.wikimedia.org/r/#/c/375948/3/sites/all/modules/wmf_civicrm/wmf_civicrm.install was 'drop the prefix' for a specific reason or just form? [21:16:12] (Merged) jenkins-bot: Follow patch from QA on find duplicates task [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375720 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:16:17] (the next commit in the line does weird & wonderful things to that fn but I didn't merge them because of the commit message being lost https://gerrit.wikimedia.org/r/#/c/376169/ [21:16:58] (PS4) Eileen: Remove void data from city field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) [21:17:00] (PS11) Eileen: Strip double whitespace & html ampersands from first_name, last_name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376169 (https://phabricator.wikimedia.org/T175107) [21:17:52] eileen - you were calling the function with a prefixed table name civicrm_address [21:17:59] (Merged) jenkins-bot: Fix spelling error. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/375721 (https://phabricator.wikimedia.org/T151270) (owner: Eileen) [21:18:08] but then inside the function it was prepending another civicrm_ to the table name [21:18:22] ah yeah - not in the latest version of that patch I don't think [21:18:30] I ditched the prepending [21:18:32] cool, I'll take a look in a sec [21:18:36] (not pushing you to do it now) [21:18:53] I'm still tinkering away & in no hurry - just wanted to clarify [21:19:05] I ran it on staging last night [21:20:50] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/376434 [21:20:57] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/376434 (owner: Ejegg) [21:24:09] eileen I'm going to deploy your merge from last night, including the civicrm submodule updates [21:24:18] ejegg: thanks [21:24:44] ejegg: remenber to disable one-offs first [21:24:52] yep, done [21:24:56] cool [21:24:58] !log updated CiviCRM from f9c373ec42f0a7b078a3eb7b31385183dab79bdf to 764bfe1f92cc8bbf8d7536744fb47b34ca7a4365 [21:25:10] now I'll prep another deploy with what's merged today [21:25:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:26:02] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/376434 (owner: Ejegg) [21:26:14] or... I'll wait till after the meeting [21:28:15] !log updated payments-wiki from bf97cd236bdec421082db8a8a782965de9c98358 to ed2e4811067ad3d74433e8d7f4944ae838af6e28 [21:28:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:30:02] ok, that continue button stuff looks good now [21:30:08] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[generate-kafkatee.pyconf] [21:30:34] fr-tech: well it finally happened, google 2fa'd me at the phone number they stole [21:30:46] allow me a few minutes to find a backup code [21:30:52] argh! [21:32:58] cwd want us to dial you in? [21:33:25] ah i found one, be there momentarily [21:35:08] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [21:58:33] Fundraising Sprint Quill Pencil, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3436052 (greg) >>! In T170591#3501942, @AndyRussG wrote: > Hi... I'... [22:01:57] Fundraising Sprint Quill Pencil, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3586745 (greg) (I look at the list of UBN! tasks a few times a week... [22:46:07] ok, the pope is passing even closer to my apartment than i thought [22:46:16] :-) [22:47:26] like 200 ft. it's a mob scene at the end of my block [22:55:08] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1921 [22:58:07] err, was that induced? ^^^ [22:58:41] cwd / eileen - someone running something big? [23:00:18] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 465300 Threads: 1 Questions: 10144133 Slow queries: 2419 Opens: 5656 Flush tables: 1 Open tables: 604 Queries per second avg: 21.801 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:01:33] how did that come from nowhere to be 1921 seconds behind? [23:01:40] isn't the threshold 1000? [23:02:13] and doesn't it check every 5 minutes / 300 seconds? [23:02:24] is that measure just totally bogus? [23:05:16] ha [23:05:23] i'm voting for that number being pretty bogus [23:05:29] *and* it's the other replica [23:05:34] * cwd shakes fist at sky [23:05:40] ooooh [23:14:15] (CR) Ejegg: [C: -1] "SQL still seems to have civicrm_{$tableName}" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) (owner: Eileen) [23:15:41] (CR) Ejegg: "ah, never mind, I see that's fixed in the following commit" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/375948 (https://phabricator.wikimedia.org/T174985) (owner: Eileen) [23:21:52] gotta go pick up the wife, back in a while to torture the dbs [23:29:02] :-)